“IF YOU made ‘Private Equity: the Movie', then Michael Douglas would have to play Schwarzman.” The head of one multi-billion-dollar private-equity firm is talking about the head of another, the Blackstone Group's Steve Schwarzman. “I'm joking,” he adds quickly, “Steve and I are good friends.” Perhaps he realises that comparisons with the fictional Wall Street banker famously portrayed by Mr Douglas, Gordon “greed is good” Gekko, are not what his industry needs just now. In fact, Hollywood has already set its sights on the men who run this enormous, relatively unaccountable pool of capital. This year, the Carlyle Group, a huge private-equity firm, has been vilified in Michael Moore's film “Fahrenheit 9/11”, as well as being named as the inspiration for a fictional private-equity firm that tries to install its brainwashed candidate as American president in the remake of “The Manchurian Candidate”.

Yet to study firms such as Blackstone is as good a way as any to find out what is going on at the sharp end of capitalism today. Hedge funds may be sexier, at least for now, but it is surely Mr Schwarzman and his peers in the private-equity industry who control the really smart money and wield the lasting influence. This survey will explain what they do, what challenges they face and what effect they have on the world of business at large.

In 1985, when Blackstone was founded by Mr Schwarzman and Pete Peterson, a former commerce secretary under Richard Nixon, private equity was a cottage industry that few people had heard of. There had always been family-owned private firms, but family owners did not usually aim to sell off the business; they passed it on to the next generation.

Until the late 1970s, the main activity in private equity—buying shares in private companies in the hope of selling them at a higher price later—had been carried out mostly by the investment arms of a few wealthy families, such as the Rockefellers and Whitneys in America, and had generally been confined to venture-capital investment in small, fast-growing businesses. America's venture capitalists have become the envy of the world for developing firms such as Intel and Google from nothing more than a bright idea into big, successful companies. But these days less than one-fifth of the money the industry raises goes on providing venture capital for young firms. Much the larger part of private-equity money is spent on buy-outs of established companies.

The first of today's big private-equity firms, Warburg Pincus, was formed only in the late 1960s, and had to raise money from investors one deal at a time. By the late 1980s private equity had grown big enough to be noticed by the general public, but it made hostile headlines with a wave of debt-financed “leveraged buy-outs” (LBOs) of big, well-known firms. The industry was cast in the role of irresponsible “corporate raider” attacking from the wilder fringes of capitalism. A bestselling book by Bryan Burrough and John Helyar about the $25 billion battle in 1988 for RJR Nabisco branded two private-equity firms, Forstmann Little and Kohlberg Kravis Roberts (KKR), as “Barbarians at the Gate”.

Today, the private-equity industry has moved from the fringe to the centre of the capitalist action. In the process, the leaders of private equity have earned themselves both wealth and respect—if not always respectability. The fabulously rich Mr Schwarzman pops up in the society gossip pages for such things as paying a record $37m for a Manhattan apartment and for demolishing his Florida mansion, allegedly without permission. He is often tipped as treasury secretary in a Republican administration.

In the 1980s private equity was a place for mavericks and outsiders; these days it attracts the most talented members of the business, political and cultural establishment, including many of the world's top managers. Jack Welch, the legendary former boss of GE, is now at Clayton, Dubilier & Rice. Lou Gerstner, who revived IBM, is chairman of Carlyle. Even Bono, the saintly lead singer of rock band U2, is now in the business.

Moreover, as Hollywood has noticed, private-equity firms have become the employer of choice for politicians and government officials returning to the private sector. Blackstone has hired Paul O'Neill, until recently America's treasury secretary. Carlyle has provided lucrative work for numerous luminaries, including George Bush senior, Fidel Ramos, a former president of the Philippines, John Major, a former British prime minister, and Arthur Levitt, a former chairman of America's main financial-markets regulator, the Securities and Exchange Commission (SEC).

Private equity's transformation into a mainstream industry has been greatly helped by a fundamental change in the sort of deals it does. In the late 1980s, funds often borrowed to the hilt to pay for buy-outs, many of which were seen as hostile by the management of the intended targets. Nowadays the buy-out firms' deals involve much less debt. When KKR bought America's Safeway supermarket chain in 1986, it borrowed 97% of the $4.8 billion the deal cost it; now a private-equity firm would typically have to stump up around one-third of the purchase price.

Hostile deals are now extremely rare. Even Britain's Philip Green, one of a small band of powerful individual private-equity financiers, declined to go hostile this year in his bid to buy Marks & Spencer, a British retailer. Indeed, big companies that would once have turned up their noses at an approach from a private-equity firm are now pleased to do business with them. Royal Dutch/Shell, a troubled oil giant, has been negotiating the sale of its liquefied-natural-gas business for $2.45 billion to KKR and Goldman Sachs Capital. Some companies even team up with private-equity firms, as Sony recently did with Texas Pacific Group (TPG) and Providence Equity Partners to buy MGM, a film studio.

Having largely shed the image of corporate wreckers, private-equity firms can now plausibly describe themselves as providing a safe haven in which firms can pursue long-term growth, sheltered from the short-term storms of the public stockmarkets. This role is all the more important because both venture capitalists and buy-out firms work increasingly with firms undergoing big changes. Well-known firms that have recently been “nurtured” by private equity include Burger King, Polaroid, Universal Studios Florida, Houghton Mifflin, Bhs, Ducati Motor and the Savoy Group.

Private-equity firms can also reasonably claim to offer a solution (though an expensive one) to the corporate-governance problems that have blighted so many public companies. “If you examine all the major corporate scandals of the past 25 years, none of them occurred where a private-equity firm was involved,” noted Henry Kravis, one of the founders of KKR, in a recent speech. Private-equity firms, he said, are “vigilant in our role as owners, and we protect shareholder value.” On the other hand, if there were any impropriety in a private company, the public might not get to hear about it.

Clearly, private equity is now a big business. In Britain, for instance, one-fifth of the workforce outside the public sector is employed by firms that are, or have been, invested in by a private-equity firm, according to the British Venture Capital Association. Worldwide, there are more than 2,700 private-equity firms, reckons Goldman Sachs (maybe many more, because in this private world small firms can easily drop below the radar screen). As pension funds, endowments and rich individuals have become increasingly keen investors, the amount of private equity has soared. In 2000 alone, the peak year so far, investors committed about $160 billion to private-equity firms (much of it to venture capital), up from only $10 billion in 1991.

At the same time, there has been a dramatic growth in the size of private-equity funds, and in the size of the top firms that manage them. Most private-equity firms raise funds as limited partnerships. The firm is the general partner that manages the fund and gets paid an annual fee (a percentage of the money in, or promised to, a fund) and later a large slice of any profits; outside investors (who often lock up their money for up to ten years) become limited partners who share only in the profits.

In 1980, the world's biggest fund (KKR's) was $135m. Today there are scores of funds with over $1 billion each. J.P. Morgan's latest one is currently the biggest, at $6.5 billion, ahead of Blackstone's (see chart 2, next page); Permira has Europe's largest, at around $6 billion at today's exchange rate. A $10 billion fund can be only a matter of time, if only for the fabulous annual fees.

Blackstone, which started life as a two-man band working from a single room, has become, in its own words, “a major player in the world of finance”. It employs over 500 people in plush offices in New York's Park Avenue, Boston, Atlanta, London, Paris and Hamburg. The 35-40 firms in which it has a private-equity stake together have over 300,000 employees and annual revenues of over $50 billion—which, were they lumped together as a single conglomerate, would make Blackstone a top-20 Fortune 500 company. Other big private-equity firms can point to similar numbers. TPG's portfolio of firms has 255,000 staff and collective annual revenues of $41 billion; Carlyle's has 150,000 workers and revenues of $31 billion.

Yet the private-equity industry must now grapple with tough new challenges. These fall into three broad and overlapping categories: generating good financial performance; coming up with winning strategies in a rapidly maturing industry; and becoming more accountable to the public, and thus less private.

There are few industries in which the gap between the best and the rest is as large as in private equity. The top firms have delivered far better returns to investors than the stockmarkets have done, but the average private-equity fund has actually produced worse results (after fees) than public equities. That includes buy-out funds as well as the venture-capital funds that destroyed so much capital during the tech bubble a few years ago.

In future, the industry may find it hard to match even this not-too-glittering past performance. Private equity may become a victim of its own success. Techniques such as seeking to maximise cashflow, using debt astutely and paying managers with shares, which were novel when private-equity firms first introduced them in the 1970s, have become standard business practice. As Mr Kravis put it in his recent speech, “Everything we have accomplished in driving corporate excellence makes it harder for us to achieve the returns that our investors expect from us.”

A crucial factor will be whether private-equity firms can genuinely improve the companies they buy. Another will be how easily they can dispose of their investments. Without an “exit”, there can be no profits. Two main exit routes—selling a firm to a big corporate buyer or floating it on a public stockmarket through an initial public offering (IPO) of its shares—have recently been much harder to pursue than in the past; and increasingly popular alternatives, such as selling to another private-equity firm, are becoming controversial.

Much will depend on how investors respond. On one hand, many have been disappointed at private-equity firms' average past performance; on the other, at a time when bonds and public equities are delivering historically low yields, the high returns generated by the best private-equity firms look increasingly enticing. European institutional investors, which have traditionally invested little in private equity, are beginning to show more interest. If investors pump more capital into an industry that arguably already has too much of it—especially now that hedge funds, flush with cash, are also piling into private equity—there is every chance of creating another bubble, hot on the heels of the one in venture capital.

Not only are good opportunities becoming harder to find, but being a maturing industry throws up other tricky issues. Many of the leading private-equity firms are still run by their founders, who are now getting to an age where they have to consider bowing out. As is often the way with charismatic founders, some may linger too long. And even when they go, the handover may prove highly disruptive as some of those passed over for the top job leave the firm. Nor can there be any guarantee that the next generation, clutching their MBAs, will inherit the deal-making magic of the founders.

Will tougher competition and increasingly demanding investors cause the industry to consolidate? Sir Ronald Cohen of Apax Partners thinks that over the next decade the private-equity industry will polarise. At one end, a few big global industry leaders will emerge—“maybe three or four dominant brands with high returns”; at the other, small specialist firms will thrive. In the middle, however, many firms will find it hard to compete. His prediction is plausible, and the losers may include some famous names. Forstmann Little has already said that it will close in 2006. It made some awful telecoms investments during the bubble and has failed to resolve its succession problem.

Some of the biggest private-equity firms are already staking out different territories. KKR and Apax say they will continue to concentrate on private equity. But Blackstone and Carlyle have been adding other financial products to their portfolio. Blackstone, for example, which has long run property funds, is toying with starting a hedge fund as well as beefing up its existing business providing advice on mergers and acquisitions. Diversification, these firms hope, will help them to exploit their expertise and brands—and perhaps to generate a more stable stream of profits that may allow them to float on the stockmarket one day. But critics ask whether there is any real synergy between the different sorts of “alternative assets” they offer.

Will private-equity firms be able to maintain their privacy when transparency is increasingly expected in every walk of life? The answer may depend on politics as much as on economics. Most private-equity firms fiercely oppose greater transparency, arguing that it will rob them of their magic. Many tacitly accept that their performance will soon become subject to much more intense scrutiny, and that they will have to adopt sensible industry standards for valuing their portfolios. But they are desperate to avoid having to disclose details about the performance of individual firms in their portfolios. Such disclosure, they say, would quickly subject those companies to the same sort of damaging short-term pressures that they would face in the public equity markets.

But change is on its way, if only because of the growing amount of money being invested in private equity through public pension funds. In America, freedom of information acts have prompted some public pension funds to provide details of the performance of their investments in different private-equity firms, to the horror of most of the firms concerned. Yet, as Thomas Lee, founder of the eponymous private-equity firm, concedes, “We are using so much public money that we have an obligation if not to be transparent then to be a little less invisible than in the past.” How much less invisible will be explored later in this survey. But first, a quick look at past form.

Copyright © 2004 The Economist Newspaper and The Economist Group. All rights reserved.