July 2004

Prepared Minds Favor Chance

Continued from page 1

By Michael Schrage

The discontinuity emerges from the vast breadth and scope of data that a Merck, a GE, an Airbus, a Wal-Mart, or a GM can reliably generate. There's an extraordinary clash and convergence of opportunity and intent. On the one hand, innovators are seeking laserlike precision in the focus and specificity of their innovation initiatives. On the other, it's become so cheap and easy to collect data on every aspect of an experiment's progress that the question has become, Why not? Data diversity that would once have been dismissed as chaotic noise is now understood to contain meaningful signals. Correlation becomes the crucible for innovation and insight.

The ongoing explosion in genomic and proteomic sequencing, for example, guarantees that the Mercks, Pfizers, and GlaxoSmithKlines of 2014 will be exploring statistical correlates in dataspheres better measured in exabytes than in gigabytes. A simple mutation in a bacterial colony may prove as medically significant as a nauseous ferret. This is a tale of data-driven scale.

So Merck and Wal-Mart won't merely explore provocative correlations; they'll be exploring the provocative correlations of correlations. These meta-analyses will become how prepared minds cultivate chance as well as exploit it. Innovators will spend less time designing clever experiments to generate data and more time scouring the data to generate hypotheses.

Exploring correlation and causality between multiple monitoring modes should spawn even greater opportunities. We won't be measuring ferret nausea as a function of vomit; we'll be giving those rodents hourly positron emission scans to see how nausea manifests itself inside their brains. Organizations like GE's aircraft engines division already rely upon these kinds of data-driven techniques to optimize their products and processes (see "If It Ain't Broke, Fix It," TR September 2001).

Of course, correlation isn't causality. Remember: there are lies, damned lies, and statistics. Then again, the economics of exploring correlation for innovation are irresistibly tempting.

The future of innovation will increasingly be determined by the future of data-driven statistical techniques. The future of data-driven statistical techniques, however, depends on innovators who grasp that ferret vomit can be a source of inspiration. That's not lucky serendipity; that's good design.