Historical Returns—Signal or Noise?

Look at enough financial data and you slowly come to the realization that there is much chaos, and little order. Too many times logically appealing patterns appear, only to vanish as the future too soon becomes the recent past.

How to separate the wheat from the chaff, the signal from the noise? There are no easy answers, but at a bare minimum some guidelines are helpful.

The experience of James O' Shaughnessy provides a cautionary tale, as well as some helpful hints. In 1997 Mr. O'Shaughnessy published What Works on Wall Street, which rapidly became an investment bestseller and classic. Let me say at the outset that this book is a well-written and researched effort which belongs on the shelves of most investors, particularly those unfamiliar with the perils of growth stock investing.

The author ran Standard and Poor's Compustat tapes, containing decades of company specific financial and returns data. He examined many different investing strategies, and found 6 particularly interesting: investing in companies with low price to earnings (PE), price to book (PB), price to cash flow (PC), and price sales (PS) ratios, as well as stocks with high dividends (Div) and price momentum (or relative price strength—RPS).

He further looked at two different size-related strategies—the whole stock market (or "all market") consisting of large- and medium-sized companies, and the biggest companies ("large stocks"). For the 1952-94 period (43 years) here's how they stacked up:

	Large Stocks			All Stocks
	Return (%)	p value		Return (%)	p value
Market	11.41	--		12.81	--
Low PB	14.54	0.025		14.66	0.13
Low PE	12.47	0.07		11.84	0.98
Low PC	14.68	0.02		14.14	0.14
Low PS	13.75	0.05		16.01	0.025
High Div.	13.13	0.12		11.58	0.5
High RS	14.17	0.05		14.44	0.14

First, notice how the value strategies and high relative strength portfolios in general seem to do better than the market. The take home message here is that "value and momentum work well" as investment strategies. (Readers of WWOWS will notice that my return figures are different from the book's. Mr O'Shaughnessy reports average annual returns, whereas the above figures are annualized returns.)

At this point it's worthwhile to interject an abstruse but very important statistical concept. Let's say that we have two series of monthly returns for different market strategies or mutual funds, or even annual batting averages for two different hitters. What are the chances that the difference (if any) between the two strategies, funds, or batters could have occurred by chance? To do this one performs a "t test," which can be found in the statistical package of most spreadsheets.

The t test, somewhat confusingly, yields a "p value," or the probability that the difference between the two means could have been due to chance. The lower the p value, the more likely it is that the difference was due to something else besides chance. Most statisticians draw the line at 0.05. Above this value, suspect chance. Below this value, suspect something else besides chance.

Already we find the author on shaky ground. Yes, the value and RPS strategies produce higher returns, but notice the p values. For example, consider the "large stock" columns. Notice how the high dividend strategy produces almost 2% more annualized return than the market portfolio. Hooray! Unfortunately the p value in the next column tells us that there is a 12% possibility that this could have occurred by chance.

Judged by this standard, only the large-stock/PC, large and all-stock/PS, and large-stock/PB strategies seem to beat the market with any degree of assurance.

Things go rapidly downhill from here, unfortunately. Looking at the above table, one might conclude that the large stock/low PB and all market/low PS strategies work the best. And you'd be wrong, at least in a statistical sense. Consider the p values comparing each pair of large stock strategies:

	PB	PE	PC	PS	Div	RPS
PB	--
PE	0.45	--
PC	0.78	0.28	--
PS	0.51	0.86	0.44	--
Div	0.054	0.36	0.063	0.27	--
RPS	0.97	0.72	0.96	0.78	0.40	--

Going down the first column, we see that even though the PB strategy returned 2.07% more than the PE strategy (first table) there is a 45% possibility that this could have occurred by chance. In fact, there is not one statistically significant pairwise comparison in the above chart. In other words, we cannot say with any degree of assurance that any of the 6 large cap strategies is really any better than any of the others.

Things are not much better for the all stock analysis:

	PB	PE	PC	PS	Div	RPS
PB	--
PE	0.08	--
PC	0.76	0.07	--
PS	0.38	0.016	0.27	--
Div	0.04	0.55	0.07	0.013	--
RPS	0.82	0.23	0.7	0.87	0.10	--

Here, at least, it appears reasonably certain that low PS beats both low PE and high dividends, while low PB also beats high dividends. But is low PS the "king of value strategies," as suggested by the author? No—its returns are not statistically distinguishable from the returns of low PB and low PC.

The above vignette highlights the perils of blindly accepting backtested market strategy results. Say you examine 10 different strategies over a 50-year period (as the author did). Naturally you are going to settle on the one with the highest return. How predictive is that of the strategy's success going forward? In other words, was the historical success of the strategy due to chance or to a real, inherent advantage? There is of course no way of telling for sure, but a slew of p values greater than 0.05 should be a red flag.

This also has great import for the mutual fund investor. Hardly a month passes without someone starting a new investment company based on an historically tested strategy. (Mr. O'Shaughnessy has done so, as have LSV Asset Management, to say nothing of Long Term Capital Management. But that one's a whole 'nother article.) Further, let's say that the strategy has been shown to beat the market by 4% per year over the past 20, 30, or 50 years.

It's a good bet that even in the best of circumstances half, or 2%, of that advantage is due to chance, or "data snooping." You can never know for certain how much a difference in return series is signal (that is, reproducible) and how much is noise, but remember that it's no accident that your eye settles on the most successful strategies only after the fact. Further, the fund's expenses and market costs are very likely to total 2% or more. So in the blink of an eye you're back to the market return, if you're lucky. And if all of the strategy's excess return is due to data snooping (a not uncommon occurrence) you are now 200 bp in the hole each year.

The problems detailed above lie at the heart of the behavioralist/efficient market controversy that rages in financial circles today. It's not that I don't trust the behavioralists to manage my money. Look at it this way. Imagine that the behavioralists are represented by your theater producer cousin, and the efficient marketeers by your accountant cousin. You sure know which one whose table you'll want to share at the next family wedding. And you also know who you'll want to execute grandma's estate.

We are all swimming on an investment beach roiled by waves of noise. Be very careful you don't mistake a line of rogue waves for the turning of the tide.

copyright (c) 1999, William J. Bernstein