Skip to Content

AI Advances Make It Possible to Search, Shop with Images

Deep learning software has dramatically improved image recognition tools. Pinterest and are testing it out on shoppers.
November 17, 2015

The Web has changed a lot in the past 20 years, but text search boxes are a constant. Typing text into pixelated oblongs remains a staple of how we interact with websites and mobile apps such as online stores.

Pinterest users can search for items on the service by selecting parts of an image.

Recent advances in image-recognition software are being used to challenge that. The social network Pinterest and the online footwear retailer are testing new ways to search or browse using just images rather than text.

Both companies have turned to a technique known as deep learning, which has recently enabled software to match humans on some benchmarks for image recognition. The technique powers Google’s image search and the photo organization service it launched this summer (see “Google Rolls Out New Automated Helpers”).

Pinterest’s new visual search tool lets you draw a box around something seen in an image on the service to find visually similar items from an index of over a billion. For example, in a quick test of the feature, drawing around a coffee maker seen in a photo of a kitchen turned up others like it, including close-up photos of the exact same model.

Some items summoned by a visual search come with buy buttons attached—a feature Pinterest introduced this summer. The image search function is being rolled out to all users of the company’s website and mobile apps this week. Pinterest’s system learned to understand images by drawing on the text people attached to photos shared on the service.

Companies have tried to use image-search technology to make shopping or discovering products easier before. For example, Amazon bundled an app to look up products snapped in a photo with its flopped Fire smartphone last year. In 2010 Google bought, which had launched a shopping comparison site that could find products that were visually similar to one you selected and even let you highlight important details on an image to guide its selections. Google shows visually similar products on its shopping site today, but doesn’t let you highlight details.

Kevin Jing, head of visual search at Pinterest, says that visual search now has a better chance of becoming indispensable. “Image representation coming from deep learning is much, much more accurate,” he says. “Even this year there has been so much improvement.” The new visual search tools are not perfect, though. Only when a lot of people have got a chance to try them will it become clear whether the underlying technology has improved enough to significantly change how people interact with online services.

Footwear retailer is testing a different approach to visual search also powered by deep learning. The company is the first to make use of image-processing technology aimed at retailers developed by artificial-intelligence startup Sentient, which has raised over $143 million from investors (see “AI Supercomputer Built by Tapping Data Warehouses for Idle Computing Power”). is initially testing Sentient’s technology in the women’s boots section of its Canadian store. Click on the “visual filter” button and you are presented with a grid of 12 images that Sentient’s software thinks represent the most distinct clusters of styles from the catalogue of about 7,000 boots. Select the one closest to what you’re looking for and the software will use the visual characteristics of your pick to refresh the grid to show 11 more that look like it. Repeating that process a few times makes it possible to home in on a selection of boots with very particular characteristics.

Nigel Duffy, Sentient’s chief technology officer, says the new feature demonstrates how software that can understand images makes online shopping more efficient. “This is a category where it’s very hard to describe in words to a search engine what you’re looking for,” he says. “We can get really granular preferences very quickly.”

Roger Hardy, CEO of, says that there is evidence that the visual filter feature increases sales and that he is considering rolling it out to other categories of footwear. Sentient’s Duffy says that he expects to see the underlying technology applied to other products such as jewelry, bags, and other accessories. 

Keep Reading

Most Popular

A Roomba recorded a woman on the toilet. How did screenshots end up on Facebook?

Robot vacuum companies say your images are safe, but a sprawling global supply chain for data from our devices creates risk.

A startup says it’s begun releasing particles into the atmosphere, in an effort to tweak the climate

Make Sunsets is already attempting to earn revenue for geoengineering, a move likely to provoke widespread criticism.

10 Breakthrough Technologies 2023

Every year, we pick the 10 technologies that matter the most right now. We look for advances that will have a big impact on our lives and break down why they matter.

These exclusive satellite images show that Saudi Arabia’s sci-fi megacity is well underway

Weirdly, any recent work on The Line doesn’t show up on Google Maps. But we got the images anyway.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.