Skip to Content

Visual Search for Better Online Shopping

A new website lets people search for hard-to-describe items by using pictures instead of words.
November 8, 2006

Here’s the scenario: You pass a person on the sidewalk wearing a pair of stylish shoes. The leather is light brown, with a rounded toe and a buckle. You’d like to find a similar pair for yourself online. But searching for “shoes, light brown, rounded toe, buckle” probably won’t get you very far.

Launched today, offers a new method of searching–using pictures instead of text–that may provide a better way to shop. The visual search engine uses a picture as a starting point, and it crawls the webpages of more than 200 online stores, including and L.L. Bean, searching for pictures of items similar to the one you’re interested in. Currently, looks at more than two million different items in four categories: shoes, handbags, watches, and jewelry. In the next few months, the company hopes to add shirts, pants, and dresses.

“We realized that the place visual search could add the most value is the place where it’s hard to describe an item with words–where you’d want to submit a photo rather than enter text,” says Munjal Shah, creator of Shah is also the CEO of the photo-sharing website, a site that recognizes faces in submitted photos (see “Face Recognition Software Goes Public”). works by using an image as a springboard for the search. Users can base their search on photos from 200 online retailers, and they can select accessories from celebrity photos in the database. Users can also indicate which characteristics, such as color, shape, or pattern, are most important to them. In addition, they can use traditional text filters to sort by brand, style, and price.

Special software developed by’s team of computer scientists recognizes similar objects by deconstructing pictures of them. Each image is broken down into 10,000 numbers that represent more than 30 features of the item–for example, the full spectrum of colors that appear in a handbag, its lumps and curves, and the glossiness of its exterior. Additionally, a user can highlight a particular feature of the item that he or she likes the most–for instance, the strap of the watch or the shape of its face–and search within that constraint. The 10,000 numbers that describe the original picture are compared with the numbers that describe the pictures on merchants’ websites.

Developing the visual search system was tricky, says Shah. He and his team had to spend a lot of time making sure that their crawler could access the high-resolution version of an image on merchants’ sites (fewer pixels don’t provide as much useful information to compare). And, if a merchant’s website offered multiple views and colors, the Web crawler needed to be able to access those as well. works best with watches and handbags, says Shah, simply because they tend to photograph consistently and there is little glare. Jewelry is more challenging for the search engine to match due to the variation in the way shiny gold and glistening diamonds are lit in photos.

The idea of visual search is certainly not new, says Pawan Sinha, professor of brain and cognitive science at MIT. “Ever since the Web came into being, there has been a large amount of graphical information available,” he says, “and that makes visual search seem like a very attractive idea.” But visual search hasn’t panned out, in part because it’s difficult for a computer to extrapolate context from a photo. For instance, a computer may or may not classify a picture of soldiers raising a flag at Iwo Jima as a World War II event.

Narrowing down the scope of the project to clothing and accessories, Sinha says, helps make the problem more manageable. Still, “it’s a fairly difficult challenge,” he says.

“I think it’s a great idea,” says Sucharita Mulpuru, a senior analyst at Forrester Research. “But I think the big question is how well the algorithm really works–whether or not the product you look for really yields similar results.” She adds that the four categories that features now are “just scratching the surface.” She thinks the concept could have exciting applications beyond clothing and accessories: it could be used to find furniture, rugs, and wallpaper. is a work in progress; it will be tweaked as Shah and his team learn more about how people are using the tool and what they want, he says. And there are still algorithmically challenging aspects of adding shirts to the mix. Shah explains that shirts are usually pictured one of two different ways: either on mannequins or on people, or else lying flat. For computer vision algorithms, it’s difficult to reconcile the two different versions of a shirt. This is a problem that the team is expected to work out in a couple of months, says Shah.

Keep Reading

Most Popular

10 Breakthrough Technologies 2024

Every year, we look for promising technologies poised to have a real impact on the world. Here are the advances that we think matter most right now.

Scientists are finding signals of long covid in blood. They could lead to new treatments.

Faults in a certain part of the immune system might be at the root of some long covid cases, new research suggests.

AI for everything: 10 Breakthrough Technologies 2024

Generative AI tools like ChatGPT reached mass adoption in record time, and reset the course of an entire industry.

What’s next for AI in 2024

Our writers look at the four hot trends to watch out for this year

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.