Emerging Technology from the arXiv

A View from Emerging Technology from the arXiv

The Dizzying Data Rate Conundrum

The suffocating rate at which data is being produced in many experiments raises the question of how to store it for future generations.

  • June 8, 2009

When it is switched on later this year, the Large Hadron Collider (LHC) at CERN will smash particles together at the rate of 40 million collisions per second. It’s a process that will generate several petabytes of data per year, and one that the LHC has been set up specifically to handle. The data will be kicked, prodded, and crunched before being analyzed and eventually released into the community as a scientific paper for publication.

That’s its short-term fate. The question is what to do with the data in the long term. Should it be archived somewhere and kept for eternity, and if so, how (and why)?

The LHC is emblematic of a broader problem in science, say André Holzner and his buddies at CERN. That problem is a rapidly growing body of data from increasingly sophisticated experiments. Holzner and co say that an increasingly pressing problem is to understand how this data is being kept in disparate facilities around the world, so that future repositories can be designed to do the job in future.

With that in mind, and with generous funding from the European Union, they’ve questioned over 1,000 high-energy physicists linked to CERN about these questions and published the results on the arXiv.

There seems to be general agreement that data preservation is hugely important. But strangely, there is less agreement over what sort of data should be stored–for example, whether to preserve the raw data itself or some higher-level analysis of it. Stranger still is the broad range of opinion over why the data should be kept at all. Only 60 percent of respondents think that the data should be kept so that conclusions can be checked in future.

Clearly, the broad concern over the issue is matched only by the widespread befuddlement over what to do about it.

Which spells bad news for CERN and other data producers. CERN is about to switch on one of the greatest data fire hoses the world has ever seen. If there is to be any multilateral agreement over what to do in the long term with the data it and other projects produce, the discussions need to be settled sooner rather than later.

Ref: arxiv.org/abs/0906.0485: First results from the PARSE.Insight project: HEP survey on data preservation, re-use and (open) access

Tech Obsessive?
Become an Insider to get the story behind the story — and before anyone else.

Subscribe today

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning magazine and daily delivery of The Download, our newsletter of what’s important in technology and innovation.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.