We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Business Report

Faster than a Flash

One financial firm says real-time data analysis saved it from making erroneous trades during the momentary stock market crash of May 2010.

On May 6, 2010, stock prices in the United States jerked down and up with incredible speed. Within about five minutes, the Dow Jones Industrial Average plummeted about 600 points, only to regain most of that by a little later in the afternoon. Federal regulators later determined that this “flash crash” was triggered and then exacerbated by automated orders placed by mutual funds and other high-frequency traders. Stock exchanges had to cancel huge numbers of erroneous trades made during the crash.

Of data and dollars: Eric Pritchett, CEO of Boston-based high-frequency trading firm PhaseCapital, says that real-time data analysis saved his firm from costly mistakes during last year’s flash crash.

Even before analysts began to understand what had gone wrong, some firms were able to stay out of the fray through real-time data analysis. For example, the high-frequency trading firm PhaseCapital, based in Boston, made no erroneous trades—an achievement that it credits to its use of complex event-processing software called StreamBase.

Market data comes from a wide variety of sources, such as the New York Stock Exchange, Nasdaq, and Reuters, and can sometimes be unreliable, explains Corwin Yu, director of electronic trading for PhaseCapital. For example, feeds can go down, or quotes can be incorrectly formatted or reflect unrealistically large changes. For companies to trade within seconds on such messy data, the information needs to be processed—in particular, scrutinized for potential errors before used to take any trading action. “It’s extremely fragile,” he says—even when there’s no crisis going on.

To deal with this information, PhaseCapital uses StreamBase, which is designed to accept large amounts of rapidly changing inputs and let organizations rapidly distill it into the insights they need to make decisions.

The typical way to deal with big data is by using databases. However, they aren’t good at processing data in real time; users have to wait until an entire data set has accumulated. StreamBase, however, can process a stream of data as it arrives, analyze it, make decisions about it, and take actions such as trading a stock or flagging a trend. “There’s a whole class of problems that are about real-time data analysis and real-time data processing,” says Richard Tibbetts, founder and CTO of StreamBase Systems.

The company offers its platform to customers, but, just as importantly, it gives them application programming interfaces that make it easy for them to develop their own software on top of it. That allows PhaseCapital (which won’t disclose how much it pays for StreamBase’s platform) to apply its own algorithms for scrubbing data and making trades.

Typically, PhaseCapital processes 30,000 to 40,000 “ticks,” or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second—much of it representing stocks swinging wildly. “A lot of scrubs kicked in and realized that this data didn’t make sense,” Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn’t act on that data, it was spared erroneous trades.

Even though many trades were later canceled, Eric Pritchett, PhaseCapital’s CEO, says there are several key reasons that it was important to avoid them anyway. For one thing, he says, it’s never certain what criteria regulators will use to cancel trades, so he wouldn’t want to have to rely on that mechanism. More importantly, he says, erroneous trades throw off a firm’s behavior for the rest of that day. For example, if a trade appears to have been profitable, algorithms may determine that the firm can afford to take riskier actions than it otherwise would. Pritchett says, “Not knowing where you really are with your book and your risk is the number one most dangerous thing that can happen to a trading firm.”

The example illustrates that anomalies in markets present both risks and opportunities, says Adam Honoré, who focuses on financial services technology as research director of the institutional securities practice for Aite Group. And the volume of data involved in trading is only going to increase, he notes.

Any time a company wants to analyze more data, there’s generally a price to pay—the extra processing reduces the speed at which a firm can take action. StreamBase has been a powerful tool in PhaseCapital’s eyes because it is structured so that new data can be filtered without causing significant performance hits, Yu says. “The trick to this game,” he says, “is to be fast and smart.”

Be the leader your company needs. Implement ethical AI.
Join us at EmTech Digital 2019.

Register now
Next in this Business Report
Understanding the Customer

The increasing power of data analysis technologies is giving companies more opportunities to understand what their customers want and need. Whether they’re scouring transaction records and Web clicks or newer sources of information, such as physical data from sensors and smart phones, companies are trying to improve their customer service and increase sales. The challenge is in choosing which data to crunch and how to act on the results. Throughout May, Business Impact will explore the technologies behind this new wave of data analytics and offer case studies of these ideas in action.

Want more award-winning journalism? Subscribe to Insider Basic.
  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning print magazine, unlimited online access plus The Download with the top tech stories delivered daily to your inbox.

    See details+

    Print Magazine (6 bi-monthly issues)

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.