The Needle and the Haystack

The Problem with Skill

One of the most difficult areas for small investors and professionals alike is the statistical definition of skill. Say a manager beats her peers by 5% for one year. Just what does that mean? How about over 5 years? 15 years?

The answer lies in the variation of manager returns. Consider the Morningstar large growth fund cohort. In 1998 it contained 558 funds, with an average return of 34.24% for that year. (Lest you you're too impressed with this, in 1998 the unmanaged Vanguard Growth Index Fund returned 42.21%.) Of course, not even one of the funds returned exactly 34.24%; there was fair amount of scatter about that figure. One can measure that degree of scatter, or variation, by calculating the standard deviation (SD), of the returns of all 558 funds. For 1998 the SD was 15.85%. In other words, by pure chance we might expect 16% of funds to perform at 1 SD above the mean, or 34.24% + 15.85% = 50.05%. In fact, only 7.5% of funds managed this. 415 large growth funds were extant in 1997, and in that year the SD of annual returns was 7.61%.

As you can see the SD for each category does bounce around, so it is best to average over many years. For the 1989-1998 10-year period, the average annual fund return SD for the large growth category was 8.33%. If the returns for each fund were due purely to skill, one would expect that the SD of annualized 10-year returns would also be about 8.33%, since the skill factor should not change from year to year. And if the returns were purely random, then the SD over longer periods would decrease over longer periods, since the variation from year to year would tend to cancel out. In that case the laws of statistics tell us that the 10 year SD should be 8.33%/sqrt(10), or 2.63%.

Let's see how things actually turned out for the large growth, large value, small growth, small value, and precious metals categories:

	Average	Predicted	Actual
	Annual SD	10 Yr. SD	10 Yr. SD
Large Growth	8.33%	2.63%	3.51%
Large Value	6.13%	1.94%	2.97%
Small Growth	10.27%	3.25%	2.28%
Small Value	14.63%	4.63%	4.12%
Precious Metals	13.63%	4.31%	4.08%

It turns out that the 10 year SDs are about what one would see with a random walk. But what if there are a few skilled managers hiding in this crowd? How might we find them?

In Sample

Let's say we are looking at 500 money managers, and spot one whose performance seems superlative. Just how extraordinary would this performance have to be to arouse our interest? What about 1 SD above the mean? Hardly. 16% of the managers, or 80 of them, would have done so purely by chance. Nor 2 SDs (2.3% by chance) or even 3 SDs (0.135% by chance, or 1 in 740) would get our attention in a group of 500. Not until we get to 4 SDs (1 in 32,000 by chance) should we suspect skill as the cause.

So, in the large cap category, where 7% of annual SD is expected, we might notice a fund whose performance is 28% above the mean in a given year. And over 15 years, we would want to see the fund beat the average by 28%/sqrt(15) = 7.2% per year. One can derive the general formula:

M = 4*S/sqrt(N)

where M is the margin of superiority required, N is the number of years, and S is the average annual SD of the category. For a small cap fund, where the category SD is, say, 12%, one would have to see a 48% outperformance in one year, or 12% outperformance annualized over 15 years. Lotsa luck.

Or, rearranging the equation for N, the number of years, we get:

N = 16*S²/M²

Let's assume that you've got your eye on a small cap fund which over 15 years has beaten its peers by 3% pa. The above formula tells you that you'd need 256 years of 3% outperformance before you can reasonably conclude that skill is involved.

Out of Sample

If the bar described above seems a bit high, there is another way out. You might accept 3 or even 2 SDs of outperformance if you are willing to place your bet on a given fund and follow it out. Here, you will only have to wait 4s²/m² years to find out if you were right. But you only get to bet on one pony. If your fund/manager outperforms its peers by 5% pa, and the category has 8% annual SD returns scatter, you still have to wait 10 years to be reasonably certain that this could not be due to chance.

The Paradox

The laws of statistics are not kind to money managers. As can be seen above, it is essentially impossible to demonstrate skill for a particular manager, either in or out of sample. The best we can manage are indirect tests of the entire population, such as the examination of the persistence of performance for a large number of funds, or the expected/predicted multiperiod SD method described above.

Perhaps there are a few superior managers out there, but you won't find them before they retire (Lynch), bloat their assets to unmanageable size (Sanborn), or demand an absurd management premium (Buffett).

copyright (c) 1999, William J. Bernstein