Skip to Content

Faster than a Flash

One financial firm says real-time data analysis saved it from making erroneous trades during the momentary stock market crash of May 2010.

On May 6, 2010, stock prices in the United States jerked down and up with incredible speed. Within about five minutes, the Dow Jones Industrial Average plummeted about 600 points, only to regain most of that by a little later in the afternoon. Federal regulators later determined that this “flash crash” was triggered and then exacerbated by automated orders placed by mutual funds and other high-frequency traders. Stock exchanges had to cancel huge numbers of erroneous trades made during the crash.

Of data and dollars: Eric Pritchett, CEO of Boston-based high-frequency trading firm PhaseCapital, says that real-time data analysis saved his firm from costly mistakes during last year’s flash crash.

Even before analysts began to understand what had gone wrong, some firms were able to stay out of the fray through real-time data analysis. For example, the high-frequency trading firm PhaseCapital, based in Boston, made no erroneous trades—an achievement that it credits to its use of complex event-processing software called StreamBase.

Market data comes from a wide variety of sources, such as the New York Stock Exchange, Nasdaq, and Reuters, and can sometimes be unreliable, explains Corwin Yu, director of electronic trading for PhaseCapital. For example, feeds can go down, or quotes can be incorrectly formatted or reflect unrealistically large changes. For companies to trade within seconds on such messy data, the information needs to be processed—in particular, scrutinized for potential errors before used to take any trading action. “It’s extremely fragile,” he says—even when there’s no crisis going on.

To deal with this information, PhaseCapital uses StreamBase, which is designed to accept large amounts of rapidly changing inputs and let organizations rapidly distill it into the insights they need to make decisions.

The typical way to deal with big data is by using databases. However, they aren’t good at processing data in real time; users have to wait until an entire data set has accumulated. StreamBase, however, can process a stream of data as it arrives, analyze it, make decisions about it, and take actions such as trading a stock or flagging a trend. “There’s a whole class of problems that are about real-time data analysis and real-time data processing,” says Richard Tibbetts, founder and CTO of StreamBase Systems.

The company offers its platform to customers, but, just as importantly, it gives them application programming interfaces that make it easy for them to develop their own software on top of it. That allows PhaseCapital (which won’t disclose how much it pays for StreamBase’s platform) to apply its own algorithms for scrubbing data and making trades.

Typically, PhaseCapital processes 30,000 to 40,000 “ticks,” or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second—much of it representing stocks swinging wildly. “A lot of scrubs kicked in and realized that this data didn’t make sense,” Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn’t act on that data, it was spared erroneous trades.

Even though many trades were later canceled, Eric Pritchett, PhaseCapital’s CEO, says there are several key reasons that it was important to avoid them anyway. For one thing, he says, it’s never certain what criteria regulators will use to cancel trades, so he wouldn’t want to have to rely on that mechanism. More importantly, he says, erroneous trades throw off a firm’s behavior for the rest of that day. For example, if a trade appears to have been profitable, algorithms may determine that the firm can afford to take riskier actions than it otherwise would. Pritchett says, “Not knowing where you really are with your book and your risk is the number one most dangerous thing that can happen to a trading firm.”

The example illustrates that anomalies in markets present both risks and opportunities, says Adam Honoré, who focuses on financial services technology as research director of the institutional securities practice for Aite Group. And the volume of data involved in trading is only going to increase, he notes.

Any time a company wants to analyze more data, there’s generally a price to pay—the extra processing reduces the speed at which a firm can take action. StreamBase has been a powerful tool in PhaseCapital’s eyes because it is structured so that new data can be filtered without causing significant performance hits, Yu says. “The trick to this game,” he says, “is to be fast and smart.”

Keep Reading

Most Popular

DeepMind’s cofounder: Generative AI is just a phase. What’s next is interactive AI.

“This is a profound moment in the history of technology,” says Mustafa Suleyman.

What to know about this autumn’s covid vaccines

New variants will pose a challenge, but early signs suggest the shots will still boost antibody responses.

Human-plus-AI solutions mitigate security threats

With the right human oversight, emerging technologies like artificial intelligence can help keep business and customer data secure

Next slide, please: A brief history of the corporate presentation

From million-dollar slide shows to Steve Jobs’s introduction of the iPhone, a bit of show business never hurt plain old business.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.