Mobile Web Searches Using Pictures

A new Microsoft application lets people search the Internet on their cell phones using a camera instead of a keypad.

Searching for information on your cell phone by typing keywords can be cumbersome. But now researchers at Microsoft have developed a software prototype called Lincoln that they hope will make Web searches easier. According to Larry Zitnick, a Microsoft researcher who works on the project, phones equipped with the software could, for example, access online movie reviews by snapping pictures of movie posters or DVD covers and get product information from pictures of advertisements in magazines or on buses.

Site seeing: Using Microsoft's new image-based Web-search software, a person can take a picture of a magazine, such as Technology Review, with a cell-phone camera and be directed to a website. Credit: Technology Review

Multimedia •  View a chart of available images for Microsoft's new software.

"The main thing we want to do is connect real-world objects with the Web using pictures," says Zitnick. "[Lincoln] is a way of finding information on the Web using images instead of keywords."

The software works by matching pictures taken on phones with pretagged pictures in a database. It provides the best results when the pictures are of two-dimensional objects, such as magazine ads or DVD covers, Zitnick says. (See the accompanying chart to find out how compatible certain pictures are with Lincoln.) Currently, the database contains pictures of DVD covers that link to movie reviews uploaded by Microsoft researchers. However, anyone can contribute his or her pictures and links to the database, and Zitnick hopes that people will fill it with pictures and links to anything from information about graffiti art to scavenger-hunt clues. Right now, Lincoln can only be downloaded for free using Internet Explorer 6 and 7, and it can only run on smart phones equipped with Windows Mobile 5.0 and PocketPCs.

Lincoln is part of a trend to link the physical world with information on the Web, often with the help of cell-phone cameras. Nokia researchers are developing software and hardware that automatically hyperlinks buildings, storefronts, and certain people via a cell-phone camera. (See "Hyperlinking Reality via Phones.") And a handful of companies, including Mobot, based in Lexington, MA, are exploring the marketing capabilities of such technology by connecting pictures of real-world advertisements and company logos to the Web.

According to Zitnick, there are two elements that distinguish his technology from others. First is the fact that anyone can contribute images, links, and comments to the database. The second element is the type of image-recognition system that Microsoft researchers have developed, which Zitnick believes will be able to search through millions of images quickly.

At the heart of the image-recognition engine is an algorithm that analyzes a picture and creates a signature that describes the picture succinctly, using a small amount of data. This signature consists of information that describes the relative position of the pixels and the intensity of a certain feature in a picture, such as the Mona Lisa's smile. In order to make this information easily searchable, data triplets are created from groups of three features. For instance, a triplet might contain information about a close-up of the Mona Lisa's smile, cheek, and nose.

When a picture is taken, the algorithm quickly establishes these data sets and compares them with established sets for the pictures already in the database. Microsoft's approach makes searching through large databases more efficient than other methods that compare a large number of individual features one by one, says Zitnick. Microsoft's engine only has to search for these triplets of data, he says, because the odds of there being many images with the same three data sets are small. "It narrows it down pretty quickly," he says.