Where’d You Get That Cool Shirt? This Software Knows

In the arms race to deliver ever more relevant and effective online advertising, a startup called Graymatics has a formidable new weapon.

The company has developed software that can automatically identify specific products in visual media—a pair of Ray-Ban sunglasses in a music video, say, or the Banana Republic shirt your friend is wearing in a holiday photo. As a way to drive online purchases, it could be a revenue booster for both content publishers and content platforms like Facebook and YouTube, which are serving up an exploding number of images and video online.

Online videos, especially, haven’t lived up to their potential for driving revenues, even with those annoying ads before a clip. Improving video advertisements is a niche that several young startups, including Graymatics, are looking to fill.

On stage at the Demo startup conference in Santa Clara, California, last week, the company’s executives showed off software that can quickly identify items in videos and photos and then match these with the same or a similar product for sale through various online retailers and marketplaces.

As one example, the software matched the sunglasses worn by Brad Pitt and Angelina Jolie in an image accompanying a news article about the movie stars to similar pairs available on Amazon. A reader online who hovered a cursor over the object would see the tagged link for the product.

At least two other companies offer similar product-recognition services, but Graymatics’ business development director, Michael Scolari, says they require humans to input some data and can handle only still images, not video. His company’s software, which relies on computer-vision and machine-learning techniques developed by researchers in Singapore, is the first to be fully automated, he says.

The software learns by scanning product images available on the Web. It then recognizes objects in images or videos and uses algorithms to break them down into almost two dozen attributes like color, shape, and even texture. Finally, it finds the closest matches in a database of products, such as a feed of Amazon or eBay’s stock, or on specific retailers’ sites.

David Hagan, CEO of Boingo Wireless, who was on a judging panel at the conference, was impressed by the technology’s potential. But he felt its success would hinge on the accuracy of its identifications and matches.

There is reason to be optimistic. J.K. Aggarwal, a computer vision researcher at the University of Texas at Austin, says that the scope of the problem the Graymatics software is tackling is similar to the problem of facial recognition, for which there is already technology in wide use, and that product recognition is, if anything, easier, not harder, to accomplish with current technology.

Founded in 2010, Graymatics is now pitching its software to major online media companies, advertising networks, and platforms for user content, says Scolari, and it is in some preliminary trials. So far, it has signed one deal with the site Metatube, a primarily Spanish-language alternative to YouTube, the results of which will go live soon, he says.

The company isn’t focused solely on advertising. It is also developing options to help companies automatically screen for objectionable content or copyrighted material uploaded by users, among several other applications.

So far, Graymatics has raised seed investments from the office of Singapore’s prime minister and the company Citrix Systems, through its Silicon Valley accelerator program.