Common sense is uncommon in individuals and, at first blush, seems even more so in groups. No one expects crowds to produce useful thought. We fear the tyranny of the majority and mob rule, avoid peer pressure where we can, and immediately see the aptness of Charles Mackay’s 19th-century book title Extraordinary Popular Delusions and the Madness of Crowds.
But the idea of collective intelligence shouldn’t seem so far-fetched. After all, democracy is built on the principle that large groups know how to govern themselves. Commodities markets, which set prices on the basis of group knowledge, play a growing role in everyday business decisions. Even lone geniuses build on the work of others: for every Einstein there are Poincares and Lorentzes and Hilberts lurking in the background.
In fact, evidence of collective intelligence is all around us, and New Yorker writer James Surowiecki collects much of it in The Wisdom of Crowds. [Surowiecki wrote on technology and happiness for the January 2005 issue of Technology Review.] Surowiecki shows how groups can often outthink even the most knowledgeable experts. He offers proof after proof that “the value of expertise is, in many contexts, overrated.” By recounting how the stock market divined that booster rocket manufacturer Morton Thiokol was most to blame for the Challenger shuttle disaster (the official answer came six months later), or how the U.S. Navy found the sunken submarine Scorpion by aggregating the best guesses of a variety of experts, Surowiecki demonstrates that collective intelligence can be harnessed, and that it does not have to be unwieldy. Collections of experts, he concedes, are prone to the ills of groupthink, which can lead to debacles like the Bay of Pigs. But he argues that crowds with certain characteristics – notably, diversity of opinion, independence of opinion, decentralization, and a way to aggregate opinions to arrive at a collective decision – will generally outsmart their most brilliant members. This is true for specific problems and broad ones, Surowiecki says, and for crowds big and small. His premise quickly comes to seem intuitive.
Groups, then, can act as parallel-processing decision engines, pooling disparate knowledge to answer even hard questions in areas like public policy. What we lack, however, is a reliable way to build such decision engines. And Surowiecki’s book, unfortunately, offers no practical solutions.
But technologists, entrepreneurs, and venture capitalists do. The last several years have seen intense interest in developing technology that improves our connectedness (see “Tagging Is It,”). In part, that interest was spurred by the phenomenal success of open-source software, which is built by communities (see “How Linux Could Overthrow Microsoft,”). It also reflects the success of Google and eBay, which have profited by harnessing the collective behaviors of very large groups. Connecting technologies like online social networks and Web logs, or “blogs,” are familiar to many people, and wikis – group Web pages that any member may edit – soon will be (see “Larry Sanger’s Knowledge Free-for-All,” January 2005,). Technologists, then, are already attacking the problem of how to achieve a high group IQ.
Better communications tools are one ingredient. Indeed, Thomas W. Malone of MIT’s Sloan School of Management argues in The Future of Work that ever cheaper and more-useful communications technology will effect a revolution in the way businesses operate. E-mail is the obvious example, but Malone also points to artificial electronic markets, which can aggregate employees’ best guesses about sales, resource allocation, research and development efforts, and even pollution control. That last was done at BP, which used an internal futures market rather than a committee of experts when it wanted to find ways to reduce its emissions.