Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo


Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

On May 6, 2010, stock prices in the United States jerked down and up with incredible speed. Within about five minutes, the Dow Jones Industrial Average plummeted about 600 points, only to regain most of that by a little later in the afternoon. Federal regulators later determined that this “flash crash” was triggered and then exacerbated by automated orders placed by mutual funds and other high-frequency traders. Stock exchanges had to cancel huge numbers of erroneous trades made during the crash.

Even before analysts began to understand what had gone wrong, some firms were able to stay out of the fray through real-time data analysis. For example, the high-frequency trading firm PhaseCapital, based in Boston, made no erroneous trades—an achievement that it credits to its use of complex event-processing software called StreamBase.

Market data comes from a wide variety of sources, such as the New York Stock Exchange, Nasdaq, and Reuters, and can sometimes be unreliable, explains Corwin Yu, director of electronic trading for PhaseCapital. For example, feeds can go down, or quotes can be incorrectly formatted or reflect unrealistically large changes. For companies to trade within seconds on such messy data, the information needs to be processed—in particular, scrutinized for potential errors before used to take any trading action. “It’s extremely fragile,” he says—even when there’s no crisis going on.

To deal with this information, PhaseCapital uses StreamBase, which is designed to accept large amounts of rapidly changing inputs and let organizations rapidly distill it into the insights they need to make decisions.

The typical way to deal with big data is by using databases. However, they aren’t good at processing data in real time; users have to wait until an entire data set has accumulated. StreamBase, however, can process a stream of data as it arrives, analyze it, make decisions about it, and take actions such as trading a stock or flagging a trend. “There’s a whole class of problems that are about real-time data analysis and real-time data processing,” says Richard Tibbetts, founder and CTO of StreamBase Systems.

0 comments about this story. Start the discussion »

Credit: Conrad Warre

Tagged: Business, Business Impact, Understanding the Customer

Reprints and Permissions | Send feedback to the editor

From the Archives


Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me