Skip to Content

Startups Mine the Real-Time Web

There’s more to it than microblog posts and social network updates.
December 9, 2009

The “real-time Web” is a hot concept these days. Both Google and Microsoft are racing to add more real-time information to their search results, and a slew of startups are developing technology to collect and deliver the freshest information from around the Web.

But there’s more to the real-time Web than just microblogging posts, social network updates, and up-to-the-minute news stories. Huge volumes of data are generated, behind the scenes, every time a person watches a video, clicks on an ad, or performs just about any other action online. And if this user-generated data can be processed rapidly, it could provide new ways to tailor the content on a website, in close to real time.

Many Web companies already use analytics to optimize their content throughout the course of a day. Some online news sites will, for example, tweak the layout on their home page by monitoring the popularity of different articles. But traditionally, information has been collected, stored, and then analyzed afterward. Using seconds-old data to tailor content automatically is the next step. In particular, a lot of the information generated in real-time relates to advertising. A few startup companies are developing technologies to process this data rapidly.

Sailesh Krishnamurthy, vice president and cofounder of the data-analysis company Truviso, based in Foster City, CA, points to the hundreds of billions of data points created each day through the delivery of online video. “If you think of each one of those hits and the associated advertisements being served by those hits,” he says, “then it’s this complex ecosystem of companies serving the ads, managing the ads, companies trying to figure out metrics. It’s pretty amazing to think that just that one user interaction leads to this explosion of activity happening under the covers.”

Real-time data analysis has its roots in the financial markets, but Ben Lorica, a senior analyst in the research group at O’Reilly Media, believes that Web companies will want to optimize ads, video, and multimedia campaigns as fast as possible. He adds that services that deliver Web content instantly make the approach relevant to the end users, too. “As people realize that they can push content out and others will start consuming it in real-time, then people will also naturally want the reporting of how that is being consumed in real-time,” he says.

Truviso and another startup, StreamBase, based in Lexington, MA, have created software to process real-time analytics data. Both companies were spun out of university research aimed at processing real-time data from sensor networks, such as those used to monitor environmental conditions. Richard Tibbetts, CTO of StreamBase, explains that financial markets make up about 80 percent of his company’s customers today. Web companies are just starting to adopt the technology.

“You’re going to see real-time Web mashups, where data is integrated from multiple sources,” Tibbetts says. Such a mashup could, for example, monitor second-to-second fluctuations in the price of airline tickets and automatically purchase one when it falls below a certain price.

Truviso recently launched a feature that allows users to calculate unique visitors to a website in real time. This has historically been a difficult problem because several steps must be performed each time to make sure the user is really distinct. Both StreamBase and Truviso rely on accessing conventional, structured databases. Lorica sees potential for real-time analysis of unstructured data–a set of numbers found scattered through a paragraph of text rather than formatted in a chart.

Software frameworks, such as Hadoop and Google’s MapReduce, which process large amounts of Web data using large numbers of computers, are often used to analyze unstructured data. Recent research from Yahoo and the University of California, Berkeley also promises to make these frameworks work in real-time, too.

Joseph Hellerstein, a UC Berkeley professor of computer science who was involved with this work, explains that the key was to find a way to make Hadoop and MapReduce faster and more interactive without compromising their ability to protect data.

Real-time applications, whether using traditional database technology or Hadoop, stand to become much more sophisticated going forward. “When people say real-time Web today, they have a narrow view of it–consumer applications like Twitter, Facebook, and a little bit of search,” says StreamBase’s Tibbetts.

Keep Reading

Most Popular

individual aging affects covid outcomes concept
individual aging affects covid outcomes concept

Anti-aging drugs are being tested as a way to treat covid

Drugs that rejuvenate our immune systems and make us biologically younger could help protect us from the disease’s worst effects.

Europe's AI Act concept
Europe's AI Act concept

A quick guide to the most important AI law you’ve never heard of

The European Union is planning new legislation aimed at curbing the worst harms associated with artificial intelligence.

Uber Autonomous Vehicles parked in a lot
Uber Autonomous Vehicles parked in a lot

It will soon be easy for self-driving cars to hide in plain sight. We shouldn’t let them.

If they ever hit our roads for real, other drivers need to know exactly what they are.

crypto winter concept
crypto winter concept

Crypto is weathering a bitter storm. Some still hold on for dear life.

When a cryptocurrency’s value is theoretical, what happens if people quit believing?

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.