Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Business Report

Bracing for the Data Deluge

Michael Stonebraker helped invent technology that put databases into every business. Now a growing flood of data means he needs to reinvent it.

From Facebook to the Department of Motor Vehicles, the world is catalogued in databases. No one knows it better than MIT adjunct professor and entrepreneur Michael Stonebraker, who has spent the last 25 years developing the technology that made it so. He got his big break by inventing and commercializing technology that underlies most of the databases, known as relational databases, that rule today. But Stonebraker now happily calls his earlier inventions largely obsolete. He’s working on a new generation of database technology that can handle the flood of digital data that is starting to overwhelm established methods.

“Relational databases are omnipresent as the solution for enterprise data. They have been fabulously successful,” Stonebraker says. But he says that the largest database vendors, including Oracle, IBM, and Microsoft, still sell such products as being appropriate for any business. Stonebraker has a different view: that new database technologies are required to handle the exponential increases in the information that businesses must handle. Stonebraker, 67, is already finding success with several of his own new approaches.

One is a database system called C-Store. Unlike most systems in use today, it stores data on disk column by column, not row by row. That simple tweak required a complete rewrite of how databases have worked, but it dovetails neatly with both the way computer memory works and the way databases are accessed. That yields much faster performance and more compressed data.

That tweak and others made by Stonebraker and colleagues at MIT, Brown, Brandeis, Yale, and the University of Massachusetts enabled the launch of Vertica, a company that commercialized C-Store and helped customers to query large databases almost in real time. Vertica was acquired by Hewlett-Packard in February and boasts clients including Comcast, which uses it to monitor the millions of devices that make up its TV and Internet networks, and Groupon, which uses it to analyze the actions of its millions of subscribers.

A related system from Stonebraker and some of the same academic colleagues, H-Store, builds on the same ideas with extra improvements such as running entirely in a computer’s memory, not on disk; this method is particularly useful in online transaction processing. H-Store’s code is open source, but the technology is being commercialized by venture-backed VoltDB, with Stonebraker as CTO. He argues that this kind of use-specific, built-for-speed databasing system is what most enterprises will need to adopt sooner rather than later to deal with the flood of digital data.

Some organizations are already caught in that flood. Consider Facebook. Already host to more digital photos than any other company, Facebook is building new storage and processing infrastructure as fast as it can. Yet it is pushing the database technology it is using to the limit, splitting its famed social graph across 4,000 databases that must all work together as one, Stonebraker says. “They are just dying under the load of the management layer needed to keep this system up,” he says. “They have the hardest database problem on the planet, and there’s no current system that will meet their needs.”

Solutions that Stonebraker is building for a very different sector already drowning in data may eventually help. A few years ago, he heard of the problems facing the Large Synoptic Survey Telescope under construction in Chile. “It is going to assemble 100 petabytes of raw data and derived data,” says Stonebraker, “and they had no clue what to do with that much.” 

Stonebraker and collaborator David DeWitt, affiliated with University of Wisconsin-Madison, built a unique database system named SciDB. The open-source project now has venture backing and a large community of volunteers from within science. But Stonebraker thinks features of SciDB will eventually find favor beyond academia.

“All science data is uncertain and has error bars, unlike the data in a salary database, so SciDB can pay attention to uncertainty. It also cannot overwrite, because science guys never want to throw anything away,” he says. Those features are not so different from the need of the high powered, statistics-heavy analytics or “data science” increasingly at the heart of successful, technology-led businesses. One example is online ad placement: targeting every person individually requires computationally intense analysis to cluster similar people together.

However, Stonebraker doesn’t claim that new database systems like those he is working on can be a panacea for companies suddenly learning the limits of more established technologies. The growing importance of data storage and processing to business of all kinds will require them to make both more of a business priority. “If you’re running a company, you’ve got to engineer in scale from the beginning,”  he says, “because there’s no doubt you will need it later.”

Become an MIT Technology Review Insider for in-depth analysis and unparalleled perspective.

Subscribe today

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Business Impact
Understanding the Customer

How technology advances are changing the economy and providing new opportunities in many industries.

Want more award-winning journalism? Subscribe to Insider Plus.
  • Insider Plus {! insider.prices.plus !}*

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

    See details+

    What's Included

    Unlimited 24/7 access to MIT Technology Review’s website

    The Download: our daily newsletter of what's important in technology and innovation

    Bimonthly print magazine (6 issues per year)

    Bimonthly digital/PDF edition

    Access to the magazine PDF archive—thousands of articles going back to 1899 at your fingertips

    Special interest publications

    Discount to MIT Technology Review events

    Special discounts to select partner offerings

    Ad-free web experience

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.