Friday, March 16, 2007

Wikipedians Promise New Search Engine

But Google is likely to be a far harder target than Encyclopaedia Britannica.

By Wade Roush

Search solution? Jimmy Wales's Wikia, which hosts hundreds of special-interest wikis, is getting into the search business with a project that will attempt to incorporate user feedback and Wikipedia-style editorial collaboration into the selection and ranking of search results. Credit: Technology Review

It's been no secret that Jimmy Wales, Wikipedia's founder, has been thinking it's time for a new style of Internet search engine. He has made it plain in public remarks, in postings to electronic mailing lists, and elsewhere over the past six months that he sees drawbacks to fully software-driven search engines such as Google and Yahoo. He has also made plain that he thinks the collaborative, decentralized publishing process behind Wikipedia might just be the answer.

Wales made his intentions official at a March 8 news conference in Tokyo, where he said that his new for-profit company, Wikia, would lead a project to launch a community-driven, open-source search site by the end of 2007. While the technical workings of Wikia Search are still being debated, it's clear that the project will combine fully robotic Web exploration, or "spidering," with Web-based tools that are in the hands of humans--both volunteer editors, who will organize and highlight the best content, and average users, who will vote on the usefulness of each search result, thereby influencing how high these results rank in future searches.

Wikia, based in San Mateo, CA, will host the search site and collect the advertising revenue it generates, but Wales believes that the site can be designed and built mainly by volunteers, through open-source collaboration of the type that gave rise to the Linux operating system. More than 700 volunteer developers are "already hacking away" at the problem using test servers donated to Wikia by supporters, according to Wikia CEO Gil Penchina.

The science of search is still an arcane one. It takes a deep understanding of file systems, index architectures, hard disk performance, networked storage, and fast query-time ranking--not to mention thousands of servers and an extreme amount of bandwidth--to build and run a major search engine. But today, Penchina and Wales argue, there is enough expertise outside the walls of the major search companies to design a competitive open-source search engine--one that they project could attract millions of users daily and capture as much as 5 percent of the $7 billion market for search-related advertising.

All that's left is to build it. Not only does the Wikia search engine not exist yet; the company is still gathering technical suggestions at the most basic level (as can be seen by browsing the project's mailing-list archive).

Wikia Search volunteers--the equivalent of Wikipedia's core community of contributors and editors--might have jobs similar to those of the category editors at the Open Directory Project, a human-edited Web directory hosted by AOL for which each volunteer is in charge of tracking Web resources on a specific subject. Alternatively, all subjects might be open to all editors, who could use a Wikipedia-like system to rank or annotate search results and track other people's revisions (and reverse them in cases of vandalism).

End users, meanwhile, might be asked to give a "thumbs up" or "thumbs down" vote about each individual search result, the way they can at the collaborative news aggregator Digg. Or they might be asked to "tag" or add descriptive words to results, helping others find them later, in the style of the social search sites TagWorld and Prefound and the photo-sharing community Flickr.