Skip to Content

AI Advances Make It Possible to Search, Shop with Images

Deep learning software has dramatically improved image recognition tools. Pinterest and Shoes.com are testing it out on shoppers.
November 17, 2015

The Web has changed a lot in the past 20 years, but text search boxes are a constant. Typing text into pixelated oblongs remains a staple of how we interact with websites and mobile apps such as online stores.

Pinterest users can search for items on the service by selecting parts of an image.

Recent advances in image-recognition software are being used to challenge that. The social network Pinterest and the online footwear retailer Shoes.com are testing new ways to search or browse using just images rather than text.

Both companies have turned to a technique known as deep learning, which has recently enabled software to match humans on some benchmarks for image recognition. The technique powers Google’s image search and the photo organization service it launched this summer (see “Google Rolls Out New Automated Helpers”).

Pinterest’s new visual search tool lets you draw a box around something seen in an image on the service to find visually similar items from an index of over a billion. For example, in a quick test of the feature, drawing around a coffee maker seen in a photo of a kitchen turned up others like it, including close-up photos of the exact same model.

Some items summoned by a visual search come with buy buttons attached—a feature Pinterest introduced this summer. The image search function is being rolled out to all users of the company’s website and mobile apps this week. Pinterest’s system learned to understand images by drawing on the text people attached to photos shared on the service.

Companies have tried to use image-search technology to make shopping or discovering products easier before. For example, Amazon bundled an app to look up products snapped in a photo with its flopped Fire smartphone last year. In 2010 Google bought Like.com, which had launched a shopping comparison site that could find products that were visually similar to one you selected and even let you highlight important details on an image to guide its selections. Google shows visually similar products on its shopping site today, but doesn’t let you highlight details.

Kevin Jing, head of visual search at Pinterest, says that visual search now has a better chance of becoming indispensable. “Image representation coming from deep learning is much, much more accurate,” he says. “Even this year there has been so much improvement.” The new visual search tools are not perfect, though. Only when a lot of people have got a chance to try them will it become clear whether the underlying technology has improved enough to significantly change how people interact with online services.

Footwear retailer Shoes.com is testing a different approach to visual search also powered by deep learning. The company is the first to make use of image-processing technology aimed at retailers developed by artificial-intelligence startup Sentient, which has raised over $143 million from investors (see “AI Supercomputer Built by Tapping Data Warehouses for Idle Computing Power”).

Shoes.com is initially testing Sentient’s technology in the women’s boots section of its Canadian store. Click on the “visual filter” button and you are presented with a grid of 12 images that Sentient’s software thinks represent the most distinct clusters of styles from the catalogue of about 7,000 boots. Select the one closest to what you’re looking for and the software will use the visual characteristics of your pick to refresh the grid to show 11 more that look like it. Repeating that process a few times makes it possible to home in on a selection of boots with very particular characteristics.

Nigel Duffy, Sentient’s chief technology officer, says the new feature demonstrates how software that can understand images makes online shopping more efficient. “This is a category where it’s very hard to describe in words to a search engine what you’re looking for,” he says. “We can get really granular preferences very quickly.”

Roger Hardy, CEO of Shoes.com, says that there is evidence that the visual filter feature increases sales and that he is considering rolling it out to other categories of footwear. Sentient’s Duffy says that he expects to see the underlying technology applied to other products such as jewelry, bags, and other accessories. 

Keep Reading

Most Popular

still from Embodied Intelligence video
still from Embodied Intelligence video

These weird virtual creatures evolve their bodies to solve problems

They show how intelligence and body plans are closely linked—and could unlock AI for robots.

pig kidney transplant surgery
pig kidney transplant surgery

Surgeons have successfully tested a pig’s kidney in a human patient

The test, in a brain-dead patient, was very short but represents a milestone in the long quest to use animal organs in human transplants.

conceptual illustration showing various women's faces being scanned
conceptual illustration showing various women's faces being scanned

A horrifying new AI app swaps women into porn videos with a click

Deepfake researchers have long feared the day this would arrive.

thermal image of young woman wearing mask
thermal image of young woman wearing mask

The covid tech that is intimately tied to China’s surveillance state

Heat-sensing cameras and face recognition systems may help fight covid-19—but they also make us complicit in the high-tech oppression of Uyghurs.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.