Skip to Content

Gestural Interfaces

Controlling computers with our bodies
Determining depth: PrimeSense’s sensor determines depth by combining a number of techniques, including structured light, where an infrared pattern (red lines) is projected onto objects. How the pattern is distorted gives information about distances. The illustrated example here is an interactive airport information display (gray box), below is the depth sensor (blue box).Determining depth: PrimeSense’s sensor determines depth by combining a number of techniques, including structured light, where an infrared pattern (red lines) is projected onto objects. How the pattern is distorted gives information about distances. The illustrated example here is an interactive airport information display (gray box), below is the depth sensor (blue box).
Determining depth: PrimeSense’s sensor determines depth by combining a number of techniques, including structured light, where an infrared pattern (red lines) is projected onto objects. How the pattern is distorted gives information about distances. The illustrated example here is an interactive airport information display (gray box), below is the depth sensor (blue box).

How do you issue complex commands to a computer without touching it? It’s a crucial issue now that televisions are connected to social networks and cars are fitted with computerized systems for communication, navigation, and entertainment. So Alexander Shpunt has designed a 3-D vision system that lets anyone control a computer just by gesturing in the air.

Shpunt spent five years developing the system at Tel Aviv-based PrimeSense, and Microsoft adopted the technology to power its popular Kinect controller for the Xbox 360 game console. Players can use it to direct characters with their bodies alone—no need for the wands, rings, gloves, or colored tags that previous gestural interfaces relied on to detect the user’s movements.

The key to dispensing with those props was getting the computer to see the world in three dimensions, rather than the two captured by normal cameras. Sensing depth makes it relatively easy to distinguish, say, an arm from a table in the background, and then track the arm’s movement.

Shpunt recalls that when he started developing his system there were a few ways to sense depth—primarily “time of flight” (determining distance from a sensor by measuring how long it takes light or sound to bounce off an object) and “structured light” (projecting patterns of light onto objects and analyzing how the patterns are distorted by the object’s surface). Although there was a lot of academic activity and a few companies built prototypes, there was “nothing really mature” that could be mass-produced, he says. Instead, he built his own system, cobbling together an approach that borrowed from those two techniques as well as stereoscopy—comparing images of the same scene from two different viewpoints.

The Kinect is only the beginning of what Shpunt believes will be a gestural-interface revolution. A small army of hackers, encouraged by PrimeSense, is already retooling the controller to other ends. Researchers at Louisiana State University have rigged a helmetless, gloveless virtual-reality system out of a Kinect unit and an off-the-shelf 3-D TV set. In Australia, a logistics software firm quickly put together a gesture-controlled system for monitoring air traffic. Further real-world applications are easy to imagine, says Shpunt: gaze-tracking heads-up controls for automobiles, touchless interactive displays for shopping malls and airports.

For now, Shpunt is working with computer maker Asus to build gestural controls for today’s increasingly complex and network-connected televisions—essentially turning a TV into a giant iPad that can be operated from the couch without a remote control.

Keep Reading

Most Popular

conceptual illustration showing various women's faces being scanned
conceptual illustration showing various women's faces being scanned

A horrifying new AI app swaps women into porn videos with a click

Deepfake researchers have long feared the day this would arrive.

A view of clouds illuminated by sunlight
A view of clouds illuminated by sunlight

We can’t afford to stop solar geoengineering research

It is the wrong time to take this strategy for combating climate change off the table.

Death and Jeff Bezos
Death and Jeff Bezos

Meet Altos Labs, Silicon Valley’s latest wild bet on living forever

Funders of a deep-pocketed new "rejuvenation" startup are said to include Jeff Bezos and Yuri Milner.

new GPT3 is a good student
new GPT3 is a good student

The new version of GPT-3 is much better behaved (and should be less toxic)

OpenAI has trained its flagship language model to follow instructions, making it spit out less unwanted text—but there's still a way to go.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.