I did a lot of work on this very topic when I was considering writing a book back in 2003 warning people to stay out of the market. John mentions Shiller when discussing the trailing PE/10. Take a look, as here is 140 years of data that would track the SPX.
http://www.econ.yale.edu/~shiller/data/ie_data.htm
Here is what I wrote. I think I wrote something simpler and better this morning, but it wouldn't post and I lost it.
Back in 2003, I was going to write a book, but I chickened out. The name of it was going to be, "Is it Safe to Get Back in the Water", meaning, is it safe to get back into stocks. During that period, I did my own personal study on long term data posted by Robert Shiller of the housing index fame, dealing with stocks. The famous return on stocks that is quoted uses 2 years, 1926 and 1950. 1926 was the last year prior to the great run up in stocks prior to the Great Depression and 1950 was the peak in dividends, so either would be a likely year to choose to mark stock returns from. 1926 because it would impress upon people that a depression couldn't stop the market and 1950 because dividends were at all time highs and it would clearly be a year to produce a great return.
In any case, Shiller had everything from earnings to dividends and the CPI number for every year from 1870 on and through this I was able to construct a model of how stocks were valued. I had been taught a formula in college about the value of a stock and the market would be valued in the same fashion. In fact, the data could actually verify what the simple model was. The formula, P=d/(k-g) is a simple discount formula where dividends are discounted the required rate of return minus the rate fo growth. Like all financial formulas, it is actually pretty simple, but finding the components was not. Through this, I made assumptions as to why the market itself was so absurdly high, as dividends were only in the 1.6% range after a sizable adjustment down from a price where dividends on the SPX were as low as 1.08%.
I decided I would use 1926 as a base year as well and what I found was that from 1926 to that time, which I believe was 2003, the long term rate of inflation was 3%, the starting dividend rate was 4.8% and the growth rate was in the 1% range, giving a long term return of about 9%. This would confirm a risk free rate of return plus 2.8% to 3% as being about normal for stocks, as treasuries would generally yield inflation plus 3% over the long haul and BBB long term debt about 5% above inflation. Bonds are safer than stocks because they at least recapture some of their value in what is certain eventual default and bankruptcy of a company.
Contrary to the bullish side of the equation, I didn't focus on the price of stocks, but rather the growth of dividends and the starting rate of dividends. A bull might say that 4.8% is too high of a dividend, but if I had used something like 3%, the rate of return would have been much slower. In fact, I could have moved to the 1929 peak and did just that.
This is where I can draw the parallel to reversion to the mean and what the return on a stock really is. If one is going to compare 1926 and 2003, they need to compare based on the dividend rate and not the price of the market, as they are 2 different things. Maybe 4.8% is too high, but then you have to adjust downward what stocks yield. I do know that 1% was too low and 2% was too low and likely 3% was too low because prior to the 1990's every time the yield hit 3%, the market failed to go higher. The best it could do was move higher with growth. There was 125 years of data lying there and suddenly history didn't apply. Anyone with a brain knows better.
To move forward, I must clarify something. This is something no one has ever mentioned in the public arena of buying stocks or an SPX fund, but there is no averaging in on the market. If you buy a stock and hold it forever, you get what you bought and nothing else. Thus, the mistake of buying a bond with too low of a yield can be made up by the end of the term of the bond, but that can't be done with stock. Thus if the historical yield of the stock market is inflation plus 5.8%, you can't pay inflation plus 4% and ever get back to inflation plus 5.8% without the market making a huge mistake. You can only make inflation plus 4% and that is if you hold the portfolio forever. You could hope for a bigger fool to come along and pay a price sufficient to produce a yield lower than 4% then you could make more than 4% by selling, but not holding. Thus the long term success of buy and hold for a market fund is predicated on whether the return in the future is lower or higher and the lower the acceptable return, the higher sales price that can be achieved. But, the forever return is always going to be where you bought in.
Why can't the growth be higher? There may be times when it is higher, but in truth there is only so much money in the system, so many resources and real growth is quite likely to not involve more money, but lower prices. Another thing is the differential in compound returns. If you take 3% for instance and compound it for 24 years, it doubles. 6% takes 12 years to double, but in 24 years it is 4 times its original size. In 48 years it is 16 times its original size while the 3% is only 4 times. Thus over the 48 years that the typical person would accumulate and live off stocks, the economy would have to support something that could be 4 times its size when it started. The factors don't work and the value of the stock market cannot grow at any appreciable pace greater than the economy as a whole. If you took the more extreme numbers of 9% and we were looking at 6% nominal in the economy, we would have a double in 12 years for the economy and 8 years for the stocks, meaning stocks would have to relatively double the economy over 24 years, as 9% produces an 8X while 6% produces a 4X. The compounds cannot be sustained on any real basis. This is actually true whether you pay dividends or buy back stock.
So, there is only 1 way that you can get an after inflation return on stocks of 6% and that is to price them that way. Stocks haven't been priced in that fashion for 25 years. The question is, why? Growth hasn't been that fantastic, actually pretty normal and below historical for a long period of prosperity. I can only offer 2 reasons, one being the natural, debt bubble that produces excessive money in search of a return and the other being an adoption of a fiction in investing. I believe today that the first reason is the primary reason, but the stock bubble couldn't be so persistent and defended without the fiction.
Here is the fiction. When I went to college, they were teaching something called portfolio theory. What portfolio theory actually meant was you could take several properly correllated risky assets and put them in a portfolio and the portfolio would over time produce the return expected out of the assets. To invest in any one of the assets would produce a return that ranged from either a total loss to a huge bonanza in excess of anything planned. Thus, you could take history and buy it and it would work as long as the assets themselves were priced reasonably going in. Nothing could be done if the assets were bought at yields much lower than necessary.
In seeing this, I wrote a paper years ago called "Who destroyed Portfolio Theory"? My contention was and still is that Wall Street took statistics and came up with this idea that stocks returned X% over time and that all you had to do was buy a portfolio and you would get the return. With the adoption of that attitude along with the credit bubble (money bubble)people started pouring money blindly into S&P funds. Over a period of 10 years or so they drove the yields down on stocks to near 1% before the bubble sprung a leak that was repaired 3 years later. Thus the assumption that all you had to do was pour money into a mutual fund and the fund would make you rich. There is really little chance that once this procedure started that Wall Street set out to take full advantage selling stocks at as high of prices as they could.
Which is the return for stocks? Is it inflation plus 2% or inflation plus 6%? I have seen about everything inbetween adopted in my lifetime. I believe that in 1982 the dividend yield on the SPX topped 6%. Thus from that time on, you could expect a real rate of return of almost 7% plus inflation. IN 2000, you could only expect a return of about 2% plus inflation. This is one reason the market could only make a double top in 2007 against its value 7 years earlier and have only dividends to show for it.
This is what mean reversion is all about. I don't know if the reader has been able to follow what I have written, but figure this one. There is a lot of talk about the 10 year bond being below the dividend rate. They talk about this like it has never happened when in fact the entire period between 1900 and 1950 the dividend rate on the broad market was higher than the yield on treasuries. I would venture that most of the following 15 years were also a period when bond yields were lower than stock dividends. This is in part because inflation wasn't certain and growth was even less certain.
If we return to normal money in a normal growth market, we will again see t-bills priced at inflation plus 3% over any given time frame. We will see long term growth on stocks of between 1/2% and 1% above inflation and we will see risk premiums above 4% again. In fact, the 1966 and 1929 tops were met with 3% dividend rates and the latest dividend I have on the SPX is 28.71 which gives a 957 price on the SPX for a historical top. As dividends fall and stocks fall out of favor and the bubble deflates, we have a long way to go down. We aren't talking about getting 10% on stocks when inflation might be zero and dividend and profit growth negative.