November 2001

Rethinking the Paradigm Paradigm

The philosophers got it wrong: scientists love new ideas-if they're right.

By Gary Taubes

A few years ago, I wrote a magazine article on the mathematics of the stock market. The assignment required that I spend considerable time interviewing the experts and studying the various theories, which is to say, whether stocks engage in a random walk of unpredictable fluctuations or whether their movements can be predetermined. If the latter is true, then the market is indeed a game that can be beaten by the better players, not just the lucky ones. I concluded, as do most experts and virtually all experimental studies on the subject, that for 99.99 percent or so of traders, buying a stock is a proposition, over the short term, at least, no more predictable than a coin toss, and losing money is as likely a result as making it. The remaining infinitesimal fraction comprises those pros who have spent fortunes on computing systems that will sift through vast amounts of data and find the exceedingly subtle patterns in the ebb and flow of stocks. They are also the ones who have the financial wherewithal to profit from those patterns before they vanish.

The "new economy" arrived, however, shortly after the publication of my article. I watched my friends and relatives, none of whom had shown particular signs of genius, cash in on the explosive growth of high-tech, Internet and dot-com stocks. New rules were in effect, I was told, and money could be made risk-free and hand-over-fist. After 18 months of passive-aggressive skepticism, I decided that perhaps I was wrong and they were right, and I bought a few tech stocks. The market then crashed, as the new economy revealed itself to be the old economy in the emperor's new clothes, and it took the bulk of my investment with it.

The moral of this story, as I see it, is that despite anything the philosopher of science Thomas Kuhn might have said to the contrary, new paradigms are extraordinarily enticing. In science, they are the breakthrough theories or remarkable discoveries, the revolutions-Kuhn's "conceptual world views"-that take a moribund field, mired in a swamp of conflicting data, and move it en masse to a new and fertile intellectual realm. In the business of technology, they are, in effect, the physical or financial manifestations of our prayers being answered: the next ubiquitous operating system, the latest revisionist approach to curing cancer or selling widgets online. They are, in their most trivial manifestations, "the new new thing," to quote the author Michael Lewis. They are also usually too good to be true-but we will get to that later.

As Kuhn saw it, and several generations of scientists, historians and journalists have told it since, new paradigms are accepted slowly, if not over the dead bodies of those who grew up with the old ones. Kuhn documented one great scientist after another, from Copernicus to Darwin to James Clerk Maxwell, who struggled relentlessly against the resistance of mediocre minds and later was vindicated. It was the German physicist Max Planck who set down the definitive words on the subject: "a new scientific truth," Planck wrote, "does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."

The salient question, however, is why the existence of "opponents"? And the answer, noted by Kuhn, although often ignored since, is surprisingly simple: a potential new paradigm or a remarkable breakthrough has opponents primarily because the data supporting it are not persuasive. Albert Einstein, for instance, may have refused to accept that God plays dice with the universe-the essence of quantum mechanics, which holds that the universe at its heart is a probabilistic and uncertain place-simply because the data supporting the existence of God's alleged gambling habit were then ambiguous. After a sufficient hypothesis and test, the data supporting the quantum mechanics revolution became compelling, and even Einstein was convinced (although perhaps not happy about it).