AI Advances Make It Possible to Search, Shop with Images

The Web has changed a lot in the past 20 years, but text search boxes are a constant. Typing text into pixelated oblongs remains a staple of how we interact with websites and mobile apps such as online stores.

Recent advances in image-recognition software are being used to challenge that. The social network Pinterest and the online footwear retailer Shoes.com are testing new ways to search or browse using just images rather than text.

Both companies have turned to a technique known as deep learning, which has recently enabled software to match humans on some benchmarks for image recognition. The technique powers Google’s image search and the photo organization service it launched this summer (see “Google Rolls Out New Automated Helpers”).

Pinterest’s new visual search tool lets you draw a box around something seen in an image on the service to find visually similar items from an index of over a billion. For example, in a quick test of the feature, drawing around a coffee maker seen in a photo of a kitchen turned up others like it, including close-up photos of the exact same model.

Some items summoned by a visual search come with buy buttons attached—a feature Pinterest introduced this summer. The image search function is being rolled out to all users of the company’s website and mobile apps this week. Pinterest’s system learned to understand images by drawing on the text people attached to photos shared on the service.

Companies have tried to use image-search technology to make shopping or discovering products easier before. For example, Amazon bundled an app to look up products snapped in a photo with its flopped Fire smartphone last year. In 2010 Google bought Like.com, which had launched a shopping comparison site that could find products that were visually similar to one you selected and even let you highlight important details on an image to guide its selections. Google shows visually similar products on its shopping site today, but doesn’t let you highlight details.

Kevin Jing, head of visual search at Pinterest, says that visual search now has a better chance of becoming indispensable. “Image representation coming from deep learning is much, much more accurate,” he says. “Even this year there has been so much improvement.” The new visual search tools are not perfect, though. Only when a lot of people have got a chance to try them will it become clear whether the underlying technology has improved enough to significantly change how people interact with online services.

Footwear retailer Shoes.com is testing a different approach to visual search also powered by deep learning. The company is the first to make use of image-processing technology aimed at retailers developed by artificial-intelligence startup Sentient, which has raised over $143 million from investors (see “AI Supercomputer Built by Tapping Data Warehouses for Idle Computing Power”).

Shoes.com is initially testing Sentient’s technology in the women’s boots section of its Canadian store. Click on the “visual filter” button and you are presented with a grid of 12 images that Sentient’s software thinks represent the most distinct clusters of styles from the catalogue of about 7,000 boots. Select the one closest to what you’re looking for and the software will use the visual characteristics of your pick to refresh the grid to show 11 more that look like it. Repeating that process a few times makes it possible to home in on a selection of boots with very particular characteristics.

Nigel Duffy, Sentient’s chief technology officer, says the new feature demonstrates how software that can understand images makes online shopping more efficient. “This is a category where it’s very hard to describe in words to a search engine what you’re looking for,” he says. “We can get really granular preferences very quickly.”

Roger Hardy, CEO of Shoes.com, says that there is evidence that the visual filter feature increases sales and that he is considering rolling it out to other categories of footwear. Sentient’s Duffy says that he expects to see the underlying technology applied to other products such as jewelry, bags, and other accessories.