Lies, Damn Lies, and Financial StatisticsIssue 04-10-17 |
In his recent presidential address to the American Finance Association, professor Campbell Harvey said that to get published in journals, there's a powerful temptation to torture the data until it confesses—that is, to conduct round after round of tests in search of a finding that can be claimed to be statistically significant. This technique is also known as "p-hacking," a reference to the p-value measure of statistical significance.
The problem, Harvey said, is that "our standard testing methods are often ill-equipped to answer the questions that we pose." He has also said that most of the empirical research in finance is likely false, and that consequently, "this implies that half the financial products (promising outperformance) that companies are selling to clients are false."
An abundance of computing power makes it possible to test thousands of trading strategies to see how each strategy would have done if it had been used during the ups and downs of the market over some past period. This is called backtesting. As a quality check, the technique is then tested on a separate set of data that wasn't used to create the technique. This method can produce some nonsensical but statistically valid relationships, particularly in finance, because researchers have more variables to manipulate in search of a subtle pattern in the data that looks like it could be a moneymaker.
Positive results of the application of this questionable research are lacking, and the big money is being made by firms that ignore finance theory. Many are loaded with mathematicians and physicists but shun finance Ph.Ds. A 2014 essay in the Journal of the American Mathematical Society referred to backtest overfitting as "pseudo-mathematics and financial charlatanism."