Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo


Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

A single episode of Buffy can have up to 20,000 instances of detected faces, but most of these instances arise from multiple frames of a single character in any given shot. The software tracks key “landmarks” on actor’s faces–nostrils, pupils, and eyes, for instance–and if one of them overlaps with the next frame, the two faces are considered part of a single track. If these landmarks are unclear, though, the software uses a description of clothing to unite two “broken” face tracks. Finally, the software also watches actors’ lips to identify who’s speaking or if the speaker is off screen. Ultimately, the system produces a detailed, play-by-play annotation of the video.

“The general idea is that you want to get more information without having people capture it,” says Alex Berg at the Computer Vision Group at University of California, Berkeley. “If you want to find a particular scene with a character, you have to first find the scenes that contain that character.” He says that Everingham’s research will pave the way for more complex searches of television programming.

Computer scientist Josef Sivic at Oxford’s Visual Geometry Group, who contributed to the Buffy project, says that in the future it will be possible to search for high-level concepts like “Buffy and Spike walking toward the camera hand-in-hand” or all outdoor scenes that contain Buffy.

Timothy Tuttle, vice president of AOL Video, says, “It seems like over the next five to ten years, more and more people will choose what to watch on their own schedule and they will view content on demand.” He also notes that the barrier to adapting technologies like Everingham’s may no longer be technical, but legal.

These legal barriers have been coming down with print media because companies have reaped the financial benefits of searchable content–Google’s Book Scan and Amazon’s search programs have been shown to boost book sales over the last two years.

But it’s unclear whether a searchable video can increase DVD sales in the same way. Currently, Google offers teasers of premium video content, says staff scientist Michele Covell. For certain genres, like sports videos, it’s becoming easier to select a teaser clip that will encourage people to buy the video, she says.

Shumeet Baluja, a staff research scientist at Google, agrees that annotating video over the Web will be a challenge, but over time they’ll be able to add more and more metadata to popular video clips offline, which will improve the speed and accuracy of searches.

0 comments about this story. Start the discussion »

Tagged: Web

Reprints and Permissions | Send feedback to the editor

From the Archives


Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me