On May 6, 2010, stock prices in the United States jerked down and up with incredible speed. Within about five minutes, the Dow Jones Industrial Average plummeted about 600 points, only to regain most of that by a little later in the afternoon. Federal regulators later determined that this “flash crash” was triggered and then exacerbated by automated orders placed by mutual funds and other high-frequency traders. Stock exchanges had to cancel huge numbers of erroneous trades made during the crash.

Even before analysts began to understand what had gone wrong, some firms were able to stay out of the fray through real-time data analysis. For example, the high-frequency trading firm PhaseCapital, based in Boston, made no erroneous trades—an achievement that it credits to its use of complex event-processing software called StreamBase.

Market data comes from a wide variety of sources, such as the New York Stock Exchange, Nasdaq, and Reuters, and can sometimes be unreliable, explains Corwin Yu, director of electronic trading for PhaseCapital. For example, feeds can go down, or quotes can be incorrectly formatted or reflect unrealistically large changes. For companies to trade within seconds on such messy data, the information needs to be processed—in particular, scrutinized for potential errors before used to take any trading action. “It’s extremely fragile,” he says—even when there’s no crisis going on.

To deal with this information, PhaseCapital uses StreamBase, which is designed to accept large amounts of rapidly changing inputs and let organizations rapidly distill it into the insights they need to make decisions.

The typical way to deal with big data is by using databases. However, they aren’t good at processing data in real time; users have to wait until an entire data set has accumulated. StreamBase, however, can process a stream of data as it arrives, analyze it, make decisions about it, and take actions such as trading a stock or flagging a trend. “There’s a whole class of problems that are about real-time data analysis and real-time data processing,” says Richard Tibbetts, founder and CTO of StreamBase Systems.