THE contrast between the dream and the often disappointing reality of private equity shows up most clearly in the recent history of venture capital. Investors poured their money into the industry, certain that venture capitalists had discovered a corporate alchemist's stone, a quick and reliable way of turning bright ideas into valuable firms. For a while the industry's success seemed to prove them right.

Paul Gompers and Josh Lerner, in their influential 2001 book, “The Money of Invention”, calculate that over the years, “venture capitalists have created nearly one-third of the total market value of all public companies in the United States.” In 1999-2000, more venture capital was raised than in the entire previous life of the industry, stretching back to the 1940s. But when the resulting bubble burst, it wiped out investments of billions of dollars and killed off thousands of young companies, not all of them daft dotcoms.

Even so, the dream lives on, in America at least. Although investment in venture capital is well down from its peak, many experienced venture capitalists think it is still far too high. This means that many investors will probably be disappointed by the returns on their money. And the task of institutionalising and professionalising the erstwhile craft business of venture capital has become much harder.

Some of the investors who rushed into venture capital in the late 1990s have rushed out again, dismissed by venture veterans as “tourists and day-trippers”. Most of the big industrial firms that set up venture-capital arms have either pruned them or lopped them off altogether. Many of the wealthy individual “angel” investors who provided seed finance for young firms have, in industry parlance, “gone to heaven”: a lot of them were bubble entrepreneurs who thought they were richer than they proved to be.

Likewise, most buy-out firms have ended the flirtation with venture capital that led some to invest their funds directly in start-ups (as Hicks, Muse, Tate & Furst did, badly, in telecoms) and others to set up venture funds, such as KKR's joint venture with Accel Partners, a leading venture-capital firm. Conversely, the long-term trend for firms that started in venture capital (such as Apax, 3i and Warburg Pincus) to move into buy-outs has reasserted itself.

Even so, venture-capital firms may raise as much as $25 billion this year, compared with only $11 billion in 1997. “The amount of capital raised has come down, but it is still high by historic standards,” says John Jaggers of Sevin Rosen, a long-established venture firm that helped to bring on Compaq, among others. “At this level, the industry as a whole will not make money. It needs to be $10 billion or less to get decent returns.” At $25 billion, he reckons, the industry as a whole may do no more than double its money, rather than more than trebling it, as it would usually hope to do. Given that top-quartile firms usually far outperform the rest, for the average firm an industry-wide doubling of the money invested, spread over the typical ten years, would represent a disappointing annual rate of return.

Certainly the influx of capital has brought much more competition to those parts of the economy that innovate through start-ups. For example, around 50 venture-funded firms are now said to be developing products to improve internet security, compared with the pre-bubble norm of 10-15 firms per sector. Clearly this is a recipe for many more corporate failures—yet many investors appear to have concluded that the bubble period is over, and that venture capital is likely to return to its former profitability.

European venture capital got going properly only in the late 1990s and was just showing signs of becoming a serious industry when the bubble burst. That stopped it in its tracks. Biotech has been hit especially hard. “The biotech business model is changing here. A lot of VCs are moving away from early-stage investing to later-stage, near-market opportunities,” says Sir Chris Evans of Merlin Biosciences, a venture firm that specialises in biotech in Europe. With firms like his now concentrating on investments likely to make money within one or two years, he says, “it is really hard for new, young companies to get early-stage funding in Europe now, and this is having a knock-on effect all the way to the universities, from where innovations usually arise.”

Perhaps the bubble in Europe did not last long enough. “It was only really six months, much shorter than in America,” says Anne Glover of Amadeus Capital Partners, a British venture-capital firm. This meant that “venture firms in America were able to get a lot more of their firms into the public markets and returned a lot of money to investors. Europe's didn't.” That difference may help to explain why confidence in America's venture-capital industry has rebounded, whereas Europe's remain depressed. More broadly, says Mr Lerner, Europe lacks an “entrepreneurial infrastructure”—everything from law firms that specialise in venture to seasoned entrepreneurs and a liquid public market for young firms.

Will European governments try to counter the setback by stimulating domestic venture-capital industries? Some of them have tried before. Between 1965 and 1995, Germany launched over 600 government programmes to encourage venture activity, without obvious success. “The experience worldwide where government has tried to encourage venture capital has not been good,” says Colin Blaydon of the Tuck Centre for Private Equity and Entrepreneurship. That is true nowadays even for America's initially useful Small Business Investment Company scheme, introduced nearly half a century ago to provide loans funded by the taxpayer. It became entangled in red tape, but has lingered on.

As new money floods into the industry, a disproportionate amount of it is going to the leading firms. In the buy-out business, well-known firms such as Forstmann Little that went awry have been punished by investors. By contrast, big names in venture capital that performed badly during the bubble period are still able to raise new funds; indeed, some have been turning away would-be investors. The latest funds raised by leading firms have all been smaller than their previous ones, and have been heavily oversubscribed, often two or three times. Kleiner Perkins Caufield & Byers raised $400m, compared with $1 billion last time; Sequoia Capital raised $395m, down from $695m. The only $1 billion venture fund raised lately was by NEA, which does a lot more late-stage investing (and is thus closer to buy-out investing). But that fund, too, was only half the size of the previous one.

After a period when “epiphany replaced rigour”, the leading American venture firms are going back to basics, says Scott Meadows, a former venture capitalist who now teaches at the University of Chicago. They are more disciplined in what they will finance, and try to ensure that they can walk away without a huge bill if the venture is not working, he says.

They are also rediscovering how important it is for venture general partners to mentor the entrepreneurs they finance, including taking seats on the boards of firms in their portfolio. The popular bubble-era idea of setting up “incubators” to bring on lots of companies fast and cheaply has been abandoned in favour of more tried-and-tested venture concepts. The best firms are becoming increasingly careful about what they invest in, and are doing more research before committing themselves. At the same time, some of them have become much more secretive about what they have invested in, in the hope of keeping down the number of imitators.

The top firms are benefiting from a virtuous cycle. Those venture capitalists with the best record of nurturing successful firms tend to attract the best ventures. A lot of entrepreneurs were badly treated after the bubble burst: venture capitalists allocated what money they had left to those firms in their portfolios nearest to achieving an exit, even if others had better long-term prospects. The idea was that if the venture firms returned at least some money to investors, they could quickly get on with raising a new fund. Even the best venture capitalists have upset entrepreneurs with their exit strategy; for example, Google's founders would have preferred to wait longer to do their IPO, but had to rush it because venture capitalists, including Kleiner Perkins, wanted to cash in.

The only serious threat to the growing domination of the leading firms is succession at the top, a problem discussed earlier in this survey. This is arguably even more critical for venture-capital firms than for buy-out firms, because the personality and involvement of the venture capitalist matters far more to a start-up than to a buy-out. Entrepreneurs increasingly want to know which particular partner will work with them before agreeing to a venture firm taking a stake in their business. Yet many leading venture firms, such as Kleiner Perkins and Greylock, have already made a number of successful inter-generational shifts, so the problem may be less pressing than in buy-outs.

Limited partners in venture firms are also becoming increasingly assertive. Kleiner Perkins, for instance, decided to let its limited partners off paying some of the capital they had committed to its worst bubble fund. “Over the next few years, limited partners are likely to be much more rigorous in holding general partners' feet to the fire,” reckons Chicago's Mr Meadows. “They are not any more going to invest in funds where the general partner gets a distribution of cash before 100% of the limited partners' capital has been returned.” During the bubble years, limited partners had to claw back a lot of distributions made to general partners when their funds proved to be less profitable than expected. It is all part of a broader attempt to make private equity more accountable.

Copyright © 2004 The Economist Newspaper and The Economist Group. All rights reserved.