Are You Smarter than an Index Fund?

Its one of the most counterintuitive laws of finance: active portfolio management will not improve performance enough to beat the returns of a simple index fund.

This claim is predicted by market efficiency theory, which argues investors cannot consistently beat the market, even with insider information, because stock price movements are too unpredictable. As one popular publication famously stated: markets essentially move in a 'random walk'. As a result, investment advice has historically discouraged active trading in favour of passive strategies, instead urging investors to buy and hold in pursuit of long-term gains.

These theories won widespread respect and were awarded multiple Nobel Prizes. But they were also developed more than 50 years ago, and a lot has changed since then. Now, financial information is more widely accessible, with much of it available for free. Online brokerages have reduced the need for stockbrokers, which has increased speed and lowered transaction costs. Traditional 'buy low, sell high' approaches are no longer the only way to win, as sophisticated investment strategies now make it possible to profit when asset prices decline or don't move at all, and boost these gains even further using leveraged margin transactions. With all these advancements, the traditional pessimistic logic that it's not possible to beat a market index through active trading can't still be true. Or can it?

Surprisingly, one of the biggest challenges is not developing a theory on how to 'beat the market', because anyone can do that, with varying degrees of success. Testing these theories objectively to prove they actually work is surprisingly difficult, because of a major data problem. Broadly, there are two approaches, and both have significant flaws.

One method involves looking backward using historical market information. While one gains access to real market data, it also comes with extra knowledge that would not have been known at the time. This can result in distortions from confirmation bias, which is similar to reading the last chapter of a mystery novel, and then feeling like you knew what would happen all along. In short: the market information is valid, but investor perspective is not.

A second approach is to look ahead, using market forecasts or by generating random data. This fixes the previous problem by offering a closer approximation of the real-world uncertainty investors face. The catch is: there's no guarantee that a winning strategy proved in theory will actually work in the real world. Investor perspective may be valid, but the market data is not.

In Can Long-Run Dynamic Optimal Strategies Outperform Fixed-Mix Portfolios? Evidence from Multiple Data Sets (European Journal of Operational Research, Vol. 36, Issue 1, July 2014, doi: 10.1016/j.ejor.2014.01.030)
Daniele Bianchi (Warwick Business School, formerly Bocconi's Department of Finance) and Massimo Guidolin (Department of Finance) used a hybrid method offering the best of both worlds. Their solution was to use historical market information, but divide it into two sets. An 'in sample' set was used to guide trade decisions, while a separate 'out of sample' batch was reserved for testing these models later. This approach enabled Bianchi and Guidolin to compare performance among several portfolios representing multiple active management strategies based on selected predictors of future asset returns and two kinds of passively managed benchmarks indexes. Overall, their objective was to seek out answers to two major questions of interest.

One: is there a big difference between the composition of an optimal portfolio, and the market benchmark? Even though actively managed portfolios can benefit from frequent trading and dynamic rebalancing, if active and passive portfolios ultimately end up with a similar asset mix, one might wonder what is the point of all the extra effort? Two: when market movements are not known in advance, can these lessons and benefits still be used in the real world to achieve higher gains from investment? Combining what was learned through these research questions, did Bianchi and Guidolin's find evidence to finally overturn this theory? In short, the answer is yes and no.

Large differences were observed between the active portfolios and passive benchmark indexes. This confirmed the intuition that dynamic investment is better, at least in theory. However, using this information to profit was a different story. The researchers found no evidence that one can use this information to financially benefit in real world situations. Although results indicated that active portfolios did outperform indexes on some occasions, there was no magic formula to predict which strategies would be the successful ones, and when their highest performance periods would happen. This finding might not be the one that investors had hoped for, but it does offer new insight, and helps explain a longstanding paradox. The intuitive view that one can beat the markets is correct, but because only a small minority of the top performing funds will accomplish this, indexes remain a smart and safe investment.