Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo


Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

A prototype computer vision system can generate a live text description of what’s happening in a feed from a surveillance camera. Although not yet ready for commercial use, the system demonstrates how software could make it easier to skim or search through video or image collections. It was developed by researchers at the University of California, Los Angeles, in collaboration with ObjectVideo of Reston, VA.

“You can see from the existence of YouTube and all the other growing sources of video around us that being able to search video is a major problem,” says Song-Chun Zhu, lead researcher and professor of statistics and computer science at UCLA.

“Almost all search for images or video is still done using the surrounding text,” he says. Zhu and UCLA colleagues Benjamin Yao and Haifeng Gong developed a new system, called I2T (Image to Text), which is intended to change that.

It puts a series of computer vision algorithms into a system that takes images or video frames as input, and spits out summaries of what they depict. “That can be searched using simple text search, so it’s very human-friendly,” says Zhu.

The team applied the software to surveillance footage in collaboration with Mun Wai Lee of ObjectVideo to demonstrate the strength of I2T. Systems like it might help address the fact that there are more and more surveillance cameras–on the streets and in military equipment, for instance–while the number of people working with them remains about the same, says Zhu.

The first part of I2T is an image parser that decomposes an image–meaning it removes the background, and objects like vehicles, trees, and people. Some objects can be broken down further; for example, the limbs of a person or wheels of a car can be separated from the object they belong to.

Next, the meaning of that collection of shapes is determined. “This knowledge representation step is the most important part of the system,” says Zhu, explaining that this knowledge comes from human smarts. In 2005, Zhu established the nonprofit Lotus Hill Institute in Ezhou, China, and, with some support from the Chinese government, recruited about 20 graduates of local art colleges to work full-time to annotate a library of images to aid computer vision systems. The result is a database of more than two million images containing objects that have been identified and classified into more than 500 categories.

To ensure that workers annotate images in a standard way, software guides them as they work. It uses versions of the algorithms that will eventually benefit from the final data to pick out the key objects for a person to classify, and it suggests how they might be classified based on previous data. The objects inside images are classified into a hierarchy of categories based on Princeton’s WordNet database, which organizes English words into groups according to their meanings. “Once you have the image parsed using that system that also includes the meaning, transcription into the natural language is not too hard,” says Zhu, who makes some of the data available for free to other researchers. “It is high-quality data and we hope that more people are going to use this,” he says.

3 comments. Share your thoughts »

Credit: Song-Chun Zhu/UCLA
Video by University of California, Los Angeles, and ObjectVideo of Reston, Virginia

Tagged: Computing, software, artificial intelligence, video, surveillance, image analysis, image processing

Reprints and Permissions | Send feedback to the editor

From the Archives


Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me