FROM its creation in 1976 to the end of 2003, KKR invested $18.4 billion. The firm calculates that these investments have increased in value to $49.7 billion, of which $39.3 billion has already been banked. Amid much fanfare, in the past 18 months it has returned $9 billion in cash to investors, beating even Carlyle's equally trumpeted cash return of $6.6 billion.

This sort of performance makes it easy to see why investors are so keen on private equity. Yet it has been the exception, not the norm—which has mostly been to lag behind even the public equity markets. And although KKR continues to proclaim that “private equity will outperform public equity and other investments over the long term,” even the best firms may struggle to match their past success. KKR itself is now viewed in the industry as a shadow of its old self, concentrating increasingly on lower-risk, lower-return deals.

From 1980 to 2001, the average private-equity buy-out fund generated slightly lower returns to investors (after subtracting fees to general partners) than they would have obtained by investing in the S&P 500, according to a recent study by Steve Kaplan of the University of Chicago and Antoinette Schoar of the Massachusetts Institute of Technology. The median venture-capital fund also fell just short of matching the S&P, although an average of venture-capital firms weighted by how much capital each had invested beat it by a short head. However, gross returns (ie, before fees) on both buy-outs and venture capital did beat the S&P during that period—probably by a sizeable margin, fees being what they are (see article), concludes the study.

Mr Kaplan and Ms Schoar also found huge differences between individual funds. The top quartile of private-equity funds produced an annual rate of return of 23%, well ahead of the S&P; the bottom quartile earned investors only 4% (see chart 3). Other experts have come to similar conclusions. Greenwich Associates, a pension-fund adviser, says that typically “only the top 25 private-equity funds generate better returns than the S&P.”

An even gloomier picture is painted by one of the most successful investors in private-equity funds, David Swensen, who has been chief investment officer of the now massive Yale University endowment since 1985. In a classic book, “Pioneering Portfolio Management”, published in 2000, Mr Swensen is scathing about the performance of much private equity—particularly as the past two decades may have been uniquely favourable for private equity, with falling interest rates making borrowing cheaper, a relatively small number of private-equity firms competing, lots of badly run companies to improve and a sharp rise in the multiple of share price to profits that investors in public stockmarkets would pay.

And even the performance of the few funds that did beat the S&P needs careful examination. Being much less liquid than public equities, private-equity investments should offer a premium over the S&P to justify the extra risk. This point is lost on some investors, who think that private equity is less risky because its reported value is much less volatile than public equities. But the main reason why private equity is less volatile is that it is only rarely revalued—and then usually in an unsatisfactory way. To say that private equity is less volatile and thus less risky is a bit like saying that the weather does not change much when you stay inside and rarely look out of the window.

But there is a further common flaw in making comparisons with the S&P. Mr Swensen looked at 542 buy-out deals that were started and concluded during 1987-98. At first sight their performance looked impressive, with annual returns of 48%, compared with 17% if the money spent on each deal had been invested in the S&P for the same period of time. But most of these gains, Mr Swensen points out, came from heavy borrowing by buy-out firms seeking to multiply their private-equity bet.

If the same amount of debt had been used to multiply the investments in the S&P, the leveraged portfolio of public equities would have generated an 86% return, beating the buy-outs by nearly 40 percentage points a year—or nearly 50 points after fees. Yale took part in only 118 of the 542 deals, generating gross returns of 63% a year, comfortably beating (even after fees) comparably leveraged S&P investment returns of 41%. Not for nothing is Mr Swensen regarded as a private-equity genius.

Why did the average fund perform so poorly? Size does not seem to have been decisive, according to the study by Mr Kaplan and Ms Schoar. In venture capital, the larger funds of the 1990s tended to perform better than their smaller counterparts in the 1980s. But the opposite was true for buy-out funds, which struggled in the early 1990s even though they had got bigger. One possible explanation is that venture capitalists had much better investment opportunities available to them in the 1990s, as the technology revolution gathered speed, whereas buy-out funds were badly hit by the collapse of the junk-bond market in the late 1980s and the recession of the early 1990s.

Location may also have played a part. Various studies have shown that, during the 1990s at least, European-based private-equity funds (mostly concentrating on buy-outs) delivered better returns than their American counterparts. Possible causes range from less competition to better deals available because of corporate Europe's relative inefficiency.

Not only is there a large gap between the best private-equity funds and the rest, but their respective performances have also been remarkably consistent. Mr Kaplan and Ms Schoar found that the winners in private equity tend to keep on winning, and the bad firms stay bad—if they remain in business. This is in sharp contrast to mutual funds and, to a lesser extent, hedge funds, where star performers tend quickly to return to the average. If a private-equity firm's first fund performs well, its second and third will probably do likewise—even more so in venture capital than in buy-outs.

Why? Some private-equity firms may have better access to “proprietary” deals in which they are the only bidders, so their costs will be lower than if they had to compete for deals. But with the increased use of auctions in sales to private-equity firms, this has become a much less likely explanation for superior performance. More plausibly, venture capitalists with a reputation for successfully nurturing start-ups may be offered better opportunities. Or perhaps there are simply just a few people in private equity who are very much better at it than their rivals. There is, after all, only one Henry Kravis.

Mr Swensen argues that the deals Yale invested in produced better results than the rest of those he studied because his institution took a different approach to private equity. It invested in funds whose deals involved much less debt and much more attention to improving the operating performance of the firms concerned.

Another explanation for the poor performance of many private-equity firms may be the inadequacies of their main investors. The vast majority of capital going into private-equity funds comes from institutional investors of various kinds. Some sorts of institution seem to do a consistently better job than others at picking private-equity winners, according to “Smart Institutions, Foolish Choices?”, a fascinating new study by Ms Schoar with Josh Lerner and Wan Wong, both of Harvard Business School.

Among funds raised in the ten years to 2001, those that attracted investment by the endowments of universities and not-for-profit foundations did 14 percentage points a year better than the average fund. Funds picked by investment advisers (including funds-of-funds, which invest as limited partners in a portfolio of private-equity funds) and banks performed well below average (see chart 4). The more endowments invest in a fund, the better it is likely to perform; the more banks invest, the worse it is likely to do.

According to the study, these striking differences cannot be explained away by the fact that endowments have been involved in private equity for longer, and thus tend to get better access to new funds being raised by the leading private-equity firms. Endowments have been just as good at picking out new private-equity firms that subsequently go on to do well.

The most likely explanation is that the leading endowments have sophisticated, relatively well-paid staff who monitor performance carefully. Harvard's endowment reputedly paid two of its analysts $35m apiece last year. Advisers, banks and others who sell funds-of-funds may have financial incentives to maximise sales, but may not care how the fund itself performs.

Banks' in-house private-equity funds often perform poorly because of conflicts of interest: they may be made to invest in a company because the bank hopes to win other banking business from it later. They also pay their staff less generously than others. Indeed, banks are now quitting the private-equity industry, both because of poor past performance and because of tougher rules on how much capital they must set aside to cover the risk involved.

But even those private-equity funds that have done well in the past may find it harder to do so in the future, as the next article will explain.

Copyright © 2004 The Economist Newspaper and The Economist Group. All rights reserved.