Letter from the Editor

On Risk

How should technologists think about precautions?

  • This letter appeared in the July/August 2010 issue.
  • by Jason Pontin
  • June 22, 2010

I have been thinking about risk.

As I write this column in early June, British Petroleum is still struggling to contain its leaking well in the Gulf of Mexico. After the company’s drilling rig, Deepwater Horizon, exploded on April 20, as many as 19,000 barrels of oil (or as much as 800,000 gallons) spewed into the Gulf every day. A cap is now capturing a little more than 400,000 gallons a day. It is the worst environmental disaster in the history of the United States, but there may be no solution until BP completes two “relief wells” in August.

This story is part of our July/August 2010 Issue
See the rest of the issue
Subscribe

In this issue of Technology Review, David Talbot writes about the increasing incidence of cyber crime and espionage, and the real (if still speculative) risk of outright cyber warfare. In “Moore’s Outlaws,” he quotes Stewart Baker, the former general counsel of the National Security Agency and a former policy chief at the U.S. Department of Homeland Security: “What we’ve been seeing, over the last decade or so, is that Moore’s Law is working more for the bad guys than the good guys. It’s really ‘Moore’s outlaws’ who are winning this fight. Code is more complex, and that means more opportunity to exploit the code. There is more money to be made in exploiting the code, and that means there are more and more sophisticated people looking to exploit vulnerabilities. If you look at things like malware found, or attacks, or the size of the haul people are pulling in, there is an exponential increase.”

Talbot describes experts’ concerns that computer viruses have made millions of machines into “enslaved armies”–botnets–awaiting instruction by malefactors. In the days leading up to April 1, 2009, a worm called Conficker was expected to receive an update from its unknown creator, but no one knew what: “A tweak to Conficker’s code might cause the three million or so machines … to start attacking the servers of some company or government network, vomit out billions of pieces of spam, or just improve the worm’s own ability to propagate.” It’s scary stuff.

In the first case, a complex system of technologies (whose purpose is to extract crude oil five miles under the ocean’s surface) failed; in the second, a more complex system (a global computer network whose purposes are incomprehensibly various, but upon which our technological civilization depends) is failing. These failures are not so much predictable as unsurprising. We expanded our use of vulnerable technologies, because we were dependent upon them. How should we think about the risks inherent in technologies, particularly new technologies?

One possible intellectual tool, popular with environmentalists and policy makers, is the Precautionary Principle, which states that when something is suspected of being harmful, the burden of proof that the thing is not harmful rests with its proponents. This principle has a stronger and a weaker formulation. The stronger calls for regulation of any potentially harmful activity, or refraining from action altogether, until there is consensus that it is safe. The weaker does not demand regulation or bans; it weighs costs against likelihoods. The former says, “Better safe than sorry”; the latter, “Be prepared.”

Although a handful of international agreements have endorsed the strong formulation, and while it possesses a quasi-legal status in European Union law, it is in fact seldom applied. (A notable exception is in the management of global fisheries.) Certainly, governments, corporations, and individuals routinely ignore it when thinking about new technologies. That’s because the idea is “paralyzing,” according to Cass Sunstein, the administrator of the White House Office of Information and Regulatory Affairs (and for many years a professor of law at the University of Chicago, where he wrote about behavioral economics and risk). No one knows how new technologies will be used in the future. There is never any consensus about risks. Crises accompany the development of any new, complex system, but their exact form tends to take us by surprise.

But if the strong formulation of the Precautionary Principle is paralyzing, the weak formulation is almost no help at all. It provides little guidance for thinking about unlikely but potentially catastrophic risks. We need an entirely new principle that will guide our investment in precautionary technology. When a technology fails or is unsustainable, we should be rationally confident that a fix or alternative exists or will exist, because ingenious humans have devised other technologies that will mitigate the crisis or stand in the outmoded technology’s place. Government has a justified role in requiring such precautionary investment in new technologies.

In the absence of a new principle, we have mere optimism.

David Talbot’s feature, which accepts that we’re not likely to build an entirely new, more secure Internet, describes research into new technologies that may make our networks safer. For now, perhaps that’s the best risk management we can expect. Read about them, and write and tell me what you think at jason.pontin@technologyreview.com.

Tech Obsessive?
Become an Insider to get the story behind the story — and before anyone else.

Subscribe today

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning magazine and daily delivery of The Download, our newsletter of what’s important in technology and innovation.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.