companies, the marketplace is still skeptical. A recent survey by Kognito, a business intelligence firm, found that only 14 percent of market research companies surveyed had immediate plans to mine social-networking data at all.
Twitter needs to win more eyeballs, motivate users to tweet more, and make sure the most useful tweets reach people who might benefit from them. The company understands this. “We are coming out of a year, 2009, which was really about scaling up,” Williams says. “We’ve gotten really good about getting [employees] in here and showing that there is interesting scale to work with here. Just like people who are thinking about using Twitter, people coming to work at Twitter thought it was some trivial thing–‘What are you having for lunch?’–not a global real-time way of finding out what is happening right now. As people learn that it fills a role we actually need filled, we will attract more engineering talent and more users.”
And one study of 2,000 U.S. tweets found that much content is of limited use.
The deals with Google and Bing made Twitter profitable. But they are also a means to another end: Twitter skeptics might well be won over if their Web searches start turning up useful tweets. By the same token, a tweet that lands high in Web search results will encourage the person who sent it to keep making timely and useful observations, says Dan Weld, a computer scientist at the University of Washington, Seattle. “When people are able to search tweets more effectively, it will change the content,” he says. “People’s behavior will be affected. But this will require real-time distribution and highly effective search.”
Making real-time search effective is tricky, though. It’s not enough just to give searchers the newest tweets that happen to contain a requested keyword. The reputation of the twitterer means something; if you want fresh information about the Haiti earthquake, for example, you’d like to hear it from responsible sources, not just from anyone who happened to include the word Haiti in a tweet. As a first step, Google evaluates tweets in part with a technological analogue to its PageRank technology, which analyzes the link structure of Web pages to judge their relevance. Generally speaking, the more links to a page–and the more pages linked to the linkers–the more relevant Google’s search engine considers it. Similarly, Google concludes that the more people who “follow” a Twitter user–and the more people who follow those people–the more credible and relevant his or her tweets probably are.
But such efforts are only the beginning. Consider a real-time search for “iPod.” An engineer doing such a search might be seeking insights into its software, a high-school student might be most interested in friends’ opinions or whether the gadget’s retail price has dropped, and a music executive might want to look for trends in what kinds of music people are downloading. Finding out what specific people want might require some analysis of their social networks, their past tweets, and the tweets of the people they follow, says Eugene Agichtein, a computer scientist at Emory University who is doing research on social search.
The location from which a tweet was issued can be an enormous help. Messages from mobile devices with GPS receivers can include location information. Twitter began allowing such information to be attached to tweets last summer, and Google and others are exploring ways to use this data to provide more relevant real-time results. “If you follow me and know that I work in Mountain View or live in Menlo Park, you just assume that when I publish something–’I saw five fire engines’–you know it’s in the general Bay Area,” says Dylan Casey, Google’s product manager for real-time search. “But that comment would become even more powerful if you knew the exact geolocation.”
Twitter itself recently refined its home page’s “Trending Topics” feature–a compendium of the most common phrases appearing in messages–to allow s to see what subjects are being discussed where they live. (Twitter determines the location partly on the basis of twitterers’ IP addresses or their reported home cities.) The new feature, called “Local Trends,” is the next logical step in making real-time search more relevant and interesting. “Search isn’t just a box and a button; it’s about serendipity,” Williams says. “It’s about hopefully, and ideally, surfacing information to you that you, as a twitterer, didn’t ask for, but wanted–and right at that moment. In the world of real-time search, the Holy Grail is that you anticipate what the user wanted. Our opportunity and challenge is solving that problem–thinking of how users behave in the ecosystem. It gets you much quicker to realizing the value of Twitter.”
There is no one magical algorithm that provides serendipity without spam, relevance without rubbish, to all people in all places. But Twitter and other companies see tremendous opportunities in the ocean of tweets–and, more generally, in mining the social Web for information that people want. “What’s really important here is the notion of the social Web and using your social network–people that you trust–for information discovery,” says Feld. “We’ve got a long way to go here, which is really exciting for any entrepreneur playing in this domain.” Even if the future business model of Twitter is not certain, the fast-changing nature of the Web itself provides evidence enough that one will emerge–and might even represent the next seismic shift in the Internet industry.
David Talbot is Technology Review’s chief correspondent.