The producers of the 1999 sci-fi martial-arts extravaganza The Matrix used elaborate and costly movie camera technology to circle around characters frozen mid-kung-fu kick. Such vertiginous visual effects can now be generated on the cheap from live images provided by stationary video cameras. But even more pertinent in the post-September 11 era, the package of video-processing and 3-D modeling technologies delivering these tricks promises to set a new standard for surveillance and security systems-and bring new meaning to the term “reality programming.”
The system, developed at Sarnoff in Princeton, NJ, allows users to joystick their way through a live, 3-D scene as if it were the latest video game. Indeed, by stitching together scenes captured by dozens or hundreds of networked cameras, the technology makes it possible to conduct a virtual patrol of an entire urban center, or every hallway in a building, in real time. “It’s difficult for eyeballs to make sense of what a collection of cameras sees. This is an integrated way to use hundreds of cameras to make one display,” says Rakesh Kumar, computer scientist and lead developer of the technology, dubbed “Video Flashlight.”
What makes the new system so unusual is that it melds 3-D models of a background scene-say, a cluster of buildings-with real-time camera views of the same area. Image-processing chips developed at Sarnoff detect new or moving objects, construct 3-D images of these objects, and integrate them into the model. New software for “tweening”-filling in the gaps between video frames-lets security personnel “fly” around a subject such as a pedestrian, getting a detailed look without jumping between widely separated views. “This is beyond anything else out there” in vision processing, says Mari Maeda, a physicist who shepherded Sarnoff’s project as a program manager at the U.S. Defense Advanced Research Projects Agency.
Uses of the technology could include marking and tracking individuals as they move, or setting off an alarm when an unusual pattern is detected, such as a large group of people entering a building. Sarnoff engineers say the system needs refinement-for example, they aim to make it work with cameras that zoom, pan and tilt, not just fixed ones. But early versions have already been installed at U.S. Army Intelligence headquarters and are under consideration for New York City’s three airports-perhaps bringing us all a step closer to living inside the Matrix.
This startup wants to copy you into an embryo for organ harvesting
With plans to create realistic synthetic embryos, grown in jars, Renewal Bio is on a journey to the horizon of science and ethics.
VR is as good as psychedelics at helping people reach transcendence
On key metrics, a VR experience elicited a response indistinguishable from subjects who took medium doses of LSD or magic mushrooms.
This nanoparticle could be the key to a universal covid vaccine
Ending the covid pandemic might well require a vaccine that protects against any new strains. Researchers may have found a strategy that will work.
This artist is dominating AI-generated art. And he’s not happy about it.
Greg Rutkowski is a more popular prompt than Picasso.
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.