We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not a subscriber? Subscribe now for unlimited access to online articles.

Intelligent Machines

AI Advances Make It Possible to Search, Shop with Images

Deep learning software has dramatically improved image recognition tools. Pinterest and Shoes.com are testing it out on shoppers.

The Web has changed a lot in the past 20 years, but text search boxes are a constant. Typing text into pixelated oblongs remains a staple of how we interact with websites and mobile apps such as online stores.

Pinterest users can search for items on the service by selecting parts of an image.

Recent advances in image-recognition software are being used to challenge that. The social network Pinterest and the online footwear retailer Shoes.com are testing new ways to search or browse using just images rather than text.

Both companies have turned to a technique known as deep learning, which has recently enabled software to match humans on some benchmarks for image recognition. The technique powers Google’s image search and the photo organization service it launched this summer (see “Google Rolls Out New Automated Helpers”).

Pinterest’s new visual search tool lets you draw a box around something seen in an image on the service to find visually similar items from an index of over a billion. For example, in a quick test of the feature, drawing around a coffee maker seen in a photo of a kitchen turned up others like it, including close-up photos of the exact same model.

Some items summoned by a visual search come with buy buttons attached—a feature Pinterest introduced this summer. The image search function is being rolled out to all users of the company’s website and mobile apps this week. Pinterest’s system learned to understand images by drawing on the text people attached to photos shared on the service.

Companies have tried to use image-search technology to make shopping or discovering products easier before. For example, Amazon bundled an app to look up products snapped in a photo with its flopped Fire smartphone last year. In 2010 Google bought Like.com, which had launched a shopping comparison site that could find products that were visually similar to one you selected and even let you highlight important details on an image to guide its selections. Google shows visually similar products on its shopping site today, but doesn’t let you highlight details.

Kevin Jing, head of visual search at Pinterest, says that visual search now has a better chance of becoming indispensable. “Image representation coming from deep learning is much, much more accurate,” he says. “Even this year there has been so much improvement.” The new visual search tools are not perfect, though. Only when a lot of people have got a chance to try them will it become clear whether the underlying technology has improved enough to significantly change how people interact with online services.

Footwear retailer Shoes.com is testing a different approach to visual search also powered by deep learning. The company is the first to make use of image-processing technology aimed at retailers developed by artificial-intelligence startup Sentient, which has raised over $143 million from investors (see “AI Supercomputer Built by Tapping Data Warehouses for Idle Computing Power”).

Shoes.com is initially testing Sentient’s technology in the women’s boots section of its Canadian store. Click on the “visual filter” button and you are presented with a grid of 12 images that Sentient’s software thinks represent the most distinct clusters of styles from the catalogue of about 7,000 boots. Select the one closest to what you’re looking for and the software will use the visual characteristics of your pick to refresh the grid to show 11 more that look like it. Repeating that process a few times makes it possible to home in on a selection of boots with very particular characteristics.

Nigel Duffy, Sentient’s chief technology officer, says the new feature demonstrates how software that can understand images makes online shopping more efficient. “This is a category where it’s very hard to describe in words to a search engine what you’re looking for,” he says. “We can get really granular preferences very quickly.”

Roger Hardy, CEO of Shoes.com, says that there is evidence that the visual filter feature increases sales and that he is considering rolling it out to other categories of footwear. Sentient’s Duffy says that he expects to see the underlying technology applied to other products such as jewelry, bags, and other accessories. 

Learn from the humans leading the way in deep learning at EmTech Next. Register Today!
June 11-12, 2019
Cambridge, MA

Register now
More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe to Print + All Access Digital.
  • Print + All Access Digital {! insider.prices.print_digital !}*

    {! insider.display.menuOptionsLabel !}

    The best of MIT Technology Review in print and online, plus unlimited access to our online archive, an ad-free web experience, discounts to MIT Technology Review events, and The Download delivered to your email in-box each weekday.

    See details+

    12-month subscription

    Unlimited access to all our daily online news and feature stories

    6 bi-monthly issues of print + digital magazine

    10% discount to MIT Technology Review events

    Access to entire PDF magazine archive dating back to 1899

    Ad-free website experience

    The Download: newsletter delivery each weekday to your inbox

    The MIT Technology Review App

You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.