Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo

 

Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

Some organizations are already caught in that flood. Consider Facebook. Already host to more digital photos than any other company, Facebook is building new storage and processing infrastructure as fast as it can. Yet it is pushing the database technology it is using to the limit, splitting its famed social graph across 4,000 databases that must all work together as one, Stonebraker says. “They are just dying under the load of the management layer needed to keep this system up,” he says. “They have the hardest database problem on the planet, and there’s no current system that will meet their needs.”

Solutions that Stonebraker is building for a very different sector already drowning in data may eventually help. A few years ago, he heard of the problems facing the Large Synoptic Survey Telescope under construction in Chile. “It is going to assemble 100 petabytes of raw data and derived data,” says Stonebraker, “and they had no clue what to do with that much.” 

Stonebraker and collaborator David DeWitt, affiliated with University of Wisconsin-Madison, built a unique database system named SciDB. The open-source project now has venture backing and a large community of volunteers from within science. But Stonebraker thinks features of SciDB will eventually find favor beyond academia.

“All science data is uncertain and has error bars, unlike the data in a salary database, so SciDB can pay attention to uncertainty. It also cannot overwrite, because science guys never want to throw anything away,” he says. Those features are not so different from the need of the high powered, statistics-heavy analytics or “data science” increasingly at the heart of successful, technology-led businesses. One example is online ad placement: targeting every person individually requires computationally intense analysis to cluster similar people together.

However, Stonebraker doesn’t claim that new database systems like those he is working on can be a panacea for companies suddenly learning the limits of more established technologies. The growing importance of data storage and processing to business of all kinds will require them to make both more of a business priority. “If you’re running a company, you’ve got to engineer in scale from the beginning,”  he says, “because there’s no doubt you will need it later.”

1 comment. Share your thoughts »

Credit: CSAIL

Tagged: Business, Business Impact, Understanding the Customer

Reprints and Permissions | Send feedback to the editor

From the Archives

Close

Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me