Skip to Content

What’s in a Tweet?

The messages are hard for machines to interpret, but a new approach could help.
April 27, 2010

Researchers at the Palo Alto Research Center (PARC) are developing new ways to deal with the torrent of information flowing from social media sites like Twitter. They have developed a Twitter “topic browser” that extracts meaning from the posts in a user’s timeline. This could help users scan through thousands of tweets quickly, and the underlying technology could also offer novel ways of mining Twitter for information or for creating targeted advertising.

Information flow: Software developed at PARC categorizes and prioritizes the information in a user’s Twitter stream.

The researchers’ idea was to provide a way for users to deal with a large number of Twitter messages quickly. They found that many users wanted to be able to quickly catch up on what’s been going on, without having to go through every single tweet in their timeline.

Ed Chi, area manager and principal scientist for the Augmented Social Cognition Research Group at PARC, says that the information coming through Twitter resembles a stream–users will dip into it from time to time, but they don’t want to consume it all at once. His group’s work is called the “Eddi Project” in reference to the idea of eddies in a stream.

The researchers developed two main ways of filtering Twitter content. The first, presented recently at the ACM Conference on Human Factors in Computing Systems in Atlanta, is a recommendation system that ranks which posts in a Twitter stream a user is likely to find most interesting, based on factors such as the contents of posts as well as his interactions with other Twitter users. The second tool, the Twitter topic browser, summarizes the contents of a user’s timeline so that the user can quickly survey what information has come through Twitter without having to read through every post.


  • See the PARC researchers’ Twitter topic browser in action.

To create this second tool, the researchers focused on identifying the topic of each tweet. Michael Bernstein, a researcher at the Computer Science and Artificial Intelligence Lab at MIT who is involved with the project, says the group found that Twitter users were interested in filtering posts relating to specific topics, and said they found existing methods lacking. “Hashtags”–user-generated annotations that categorize tweets–are perhaps the best current option, but most tweets don’t have these tags. Bernstein notes that Twitter, Google, and other companies are developing ways to identify and categorize the most popular topics of discussion on Twitter–such the Icelandic volcano. But the sheer volume of tweets provides a lot of information for algorithms to use; it’s much harder, he says, to figure out the topic of tweets that are more unique.

A key challenge of extracting meaning from a tweet is its length: no more than 140 characters. Chi says that most natural language processing technology relies on having a larger sample of text to work with. For example, some methods rely on people writing out associations between terms, which requires a lot of work to maintain, and is not the best way to interpret real-time information.

Back in time: The software tools can produce a visualization showing Twitter topics over time. The lower half of this image shows recommended tweets.

The researchers realized, however, that search engines have been dealing with extracting meaning from a small number of words–in the form of search queries–for years.

“The essence of the approach is to coerce a tweet to look more like a search query and then get a search engine to tell us more,” Bernstein says. The researchers first clean up a tweet by pulling common terms, like the Twitter slang “RT,” which means “retweet.” Once their algorithms have focused on likely significant terms, they feed those into the Yahoo’s Build your Own Search Service interface–a Web service that can be used to tap directly into Yahoo’s search result.

The Web is the most up-to-date source of data, Bernstein says, and the pages that come up in search results give enough information for the researchers’ algorithms to produce a list of topics related to the original tweet.

A similar approach could be used with any repository of information, Chi notes, pointing out that companies could use the technology on an intranet to classify bits of information related to more specialized topics.

“Boosting the signal of a tweet by piping it through web search is an application of a well-established information-retrieval technique,” says Daniel Tunkelang, an engineer at Google who is an expert on information retrieval. He compares it to using a thesaurus to set a word in a broader context.

However, Tunkelang says the PARC researchers will have to make sure that the tweet-as-search-query approach doesn’t collide with search engines’ increasing efforts to index tweets. It wouldn’t be good for a tweet to return itself as a result.

Chi says that his team is working on a platform for managing various kinds of information streams. This summer, they plan to increase the scale of the Eddi Project so it can be placed on the live Web for testing. The longer-term goal, Chi says, is to build tools that can be optimized for enterprise customers.

Deep Dive


Our best illustrations of 2022

Our artists’ thought-provoking, playful creations bring our stories to life, often saying more with an image than words ever could.

How CRISPR is making farmed animals bigger, stronger, and healthier

These gene-edited fish, pigs, and other animals could soon be on the menu.

The Download: the Saudi sci-fi megacity, and sleeping babies’ brains

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. These exclusive satellite images show Saudi Arabia’s sci-fi megacity is well underway In early 2021, Crown Prince Mohammed bin Salman of Saudi Arabia announced The Line: a “civilizational revolution” that would house up…

10 Breakthrough Technologies 2023

Every year, we pick the 10 technologies that matter the most right now. We look for advances that will have a big impact on our lives and break down why they matter.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.