Skip to Content

Real-World Video Game

Surveillance was never so much fun.
July 25, 2002

The producers of the 1999 sci-fi martial-arts extravaganza The Matrix used elaborate and costly movie camera technology to circle around characters frozen mid-kung-fu kick. Such vertiginous visual effects can now be generated on the cheap from live images provided by stationary video cameras. But even more pertinent in the post-September 11 era, the package of video-processing and 3-D modeling technologies delivering these tricks promises to set a new standard for surveillance and security systems-and bring new meaning to the term “reality programming.”

The system, developed at Sarnoff in Princeton, NJ, allows users to joystick their way through a live, 3-D scene as if it were the latest video game. Indeed, by stitching together scenes captured by dozens or hundreds of networked cameras, the technology makes it possible to conduct a virtual patrol of an entire urban center, or every hallway in a building, in real time. “It’s difficult for eyeballs to make sense of what a collection of cameras sees. This is an integrated way to use hundreds of cameras to make one display,” says Rakesh Kumar, computer scientist and lead developer of the technology, dubbed “Video Flashlight.”

What makes the new system so unusual is that it melds 3-D models of a background scene-say, a cluster of  buildings-with real-time camera views of the same area. Image-processing chips developed at Sarnoff detect new or moving objects, construct 3-D images of these objects, and integrate them into the model. New software for “tweening”-filling in the gaps between video frames-lets security personnel “fly” around a subject such as a pedestrian, getting a detailed look without jumping between widely separated views. “This is beyond anything else out there” in vision processing, says Mari Maeda, a physicist who shepherded Sarnoff’s project as a program manager at the U.S. Defense Advanced Research Projects Agency.

Uses of the technology could include marking and tracking individuals as they move, or setting off an alarm when an unusual pattern is detected, such as a large group of people entering a building. Sarnoff engineers say the system needs refinement-for example, they aim to make it work with cameras that zoom, pan and tilt, not just fixed ones. But early versions have already been installed at U.S. Army Intelligence headquarters and are under consideration for New York City’s three airports-perhaps bringing us all a step closer to living inside the Matrix.

Keep Reading

Most Popular

The Steiner tree problem:  Connect a set of points with line segments of minimum total length.
The Steiner tree problem:  Connect a set of points with line segments of minimum total length.

The 50-year-old problem that eludes theoretical computer science

A solution to P vs NP could unlock countless computational problems—or keep them forever out of reach.

section of Rima Sharp captured by the LRO
section of Rima Sharp captured by the LRO

The moon didn’t die as early as we thought

Samples from China’s lunar lander could change everything we know about the moon’s volcanic record.

conceptual illustration of a heart with an arrow going in on one side and a cursor coming out on the other
conceptual illustration of a heart with an arrow going in on one side and a cursor coming out on the other

Forget dating apps: Here’s how the net’s newest matchmakers help you find love

Fed up with apps, people looking for romance are finding inspiration on Twitter, TikTok—and even email newsletters.

ASML machine
ASML machine

Inside the machine that saved Moore’s Law

The Dutch firm ASML spent $9 billion and 17 years developing a way to keep making denser computer chips.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.