Monday, September 08, 2008

Opening Search to Semantic Upstarts

Yahoo's new open-search platform is giving semantic search a helping hand.

By Kate Greene

Credit: Technology Review

Even if you have a great idea for a new search engine, it's far from easy to get it off the ground. For one thing, the best engineering talent resides at big-name companies. Even more significantly, according to some estimates, it costs hundreds of millions of dollars to buy and maintain the servers needed to index the Web in its entirety.

However, Yahoo recently released a resource that may offer hope to search innovators and entrepreneurs. Called Build Your Own Search Service (BOSS), it allows programmers to make use of Yahoo's index of the Web--billions of pages that are continually updated--thereby removing perhaps the biggest barrier to search innovation. By opening its index to thousands of independent programmers and entrepreneurs, Yahoo hopes that BOSS will kick-start projects that it lacks the time, money, and resources to invent itself. Prabhakar Raghavan, head of Yahoo Research and a consulting professor at Stanford University, says this might include better ways of searching videos or images, tools that use social networks to rank search results, or a semantic search engine that tries to understand the contents of Web pages, rather than just a collection of keywords and links.

"We're trying to break down the barriers to innovation," says Raghavan, although he admits that BOSS is far from an altruistic venture. If a new search-engine tool built using Yahoo's index becomes popular and potentially profitable, Yahoo reserves the right to place ads next to its results.

So far, no BOSS-powered site has become that successful. But a number of startups are beginning to build their services on top of BOSS, and Semantic Web companies, in particular, are benefiting from the platform. These companies are developing software to process concepts and meanings in order to better organize information on the Web.

For instance, Hakia, a company based in New York, began building a semantic search engine in 2004. Its algorithms use a database of concepts--people, places, objects, and more--to "understand" concepts in documents. Hakia also creates maps linking together different documents, such as Web pages, based on these concepts in order to understand their relevance to one another. Riza Berkan, CEO of the company, says that focusing on the meaning of pages, instead of simply on the links between them, could serve up more relevant search results and help people find content that they didn't even know they were looking for.