How Twitter Could Bring Search Up to Speed
Some say that Twitter may be as important to real-time search as YouTube is to video.
When Twitter was introduced in late 2006, asking users to post a 140-character answer to the question “What are you doing?,” many criticized the results as nothing more than a collection of trivial thoughts and inane ramblings. Fast-forward three years, and the number of Twitter users has grown to millions, while the content of the many posts–better known as “tweets”–has shifted from banal to informative.
Twitter users now cover breaking news, posting links to reports, blog posts, and images. Twitter’s search box also reveals what people think of the latest new gadget or movie, letting visitors eavesdrop on often spirited conversations and some insightful opinions.
Earlier this week, on The Charlie Rose Show, Google’s CEO, Eric Schmidt, was asked directly whether Google might be interested in acquiring Twitter. He responded, somewhat coyly, that his company was “unlikely to buy anything right now.”
Nonetheless, as Twitter grows in size and substance, it’s becoming clear that it offers a unique feed of real-time conversation and sentiment. Danny Sullivan, editor of the blog Search Engine Land, compares this to the unique real-time feed of new video content offered by YouTube, which Google acquired in 2006, and says that Twitter could help improve real-time search. Notably, says Sullivan, this is something that Google isn’t particularly good at. Even by scouring news sites, Google simply can’t match the speed and relevancy of social sites like Digg and Twitter, he says.
Twitter’s ability to capture the latest fad is evident from its “trends” feature, which reveals the most talked about topics among Twitterers. At the time this article was written, Twitter users were discussing topics including National Napping Day, DST (daylight savings time), and the new movie Watchmen. A quick search also reveals that five people within the past half hour have posted tweets about last weekend’s Saturday Night Live skit called “The Rock Obama.” The most recent tweet includes a link to the video and was posted just three minutes ago.
Bruce Croft, a professor of computer science at the University of Massachusetts, Amherst, says that Twitter search could perhaps help make news alerts more relevant. “If you could search or track large numbers of conversations, then there would be the possibility of developing alerts when something starts happening,” he says. “And, of course, it’s yet another opportunity to do massive data mining on people’s activities to learn even more about what they are doing and when they are doing it.”
An upswing in the number of marketers and companies that have joined Twitter over the past year also reflects big interest in using Twitter to tap into the online trends. Sullivan admits that right now the average Twitter user isn’t turning to the service for the majority of its searches, but he says that marketers who want to track the sentiment of their brands and products are finding it an indispensable tool. “Is it essential to search Twitter because you’re a brand manager?” asks Sullivan. “Yeah, it’s becoming very important.” He notes that outside search engines, such as Google, cannot easily go through Twitter’s data because it is not releasing it. “If I were over at Google, I’d definitely be concerned about that,” he says.
Meanwhile, Twitter is clearly thinking about ways to better mine its users’ tweets. When you search Twitter, its search engine looks for keywords in the most recent tweets, explains Biz Stone, one of the company’s cofounders. The results are then ranked based on the time when they were posted; for some popular topics, this can mean just seconds ago. The search technology used for this was developed at a startup called Summize, which Twitter acquired last July. Until recently, Twitter users needed to go to the address search.twitter.com to use the technology: a link was buried at the bottom of a user’s homepage. Twitter is now testing a “more integrated” search box at the top of the page for a small number of users, says Stone.
As simple as it sounds, dealing with the volume of messages posted on Twitter in real time poses a big challenge, according to Stone. “Our biggest technical challenge right now is not that exciting: it’s just scaling from an operations and engineering standpoint,” he says. In other words, engineers have to make sure that the system doesn’t crash under the weight of all the messages that Twitter handles.
Summize was given access to a higher volume of tweets than most third-party developers. In the early days of Twitter, it wasn’t a problem to let developers use such a large volume of tweets, but as the service has grown, the application programming interface (API) that allows for such access “requires special design and attention,” Stone says. Engineers at Twitter have also been working on a new, more scalable solution. “We’ll need to be thoughtful about how, when, and who we partner with to test this technology when it’s ready,” he adds.
When asked whether Twitter and Google could work together to build a real-time search engine, Stone is a bit more optimistic. “We’re huge fans of Google,” he says. “And we’d be delighted to partner or work with them in the future.”