Skip to Content

Get to Grips with Virtual Objects Using This Stripped-Down Glove

A new controller puts sensors on your fingers so you can type without a keyboard and manipulate digital objects.
October 29, 2015

Keyboards, mice, and touch screens all have their uses but have distinct limitations. They’re not particularly good for interacting with 3-D objects, such as those found in virtual reality or design programs. A company called Gest thinks its device, which slides onto the back of your hands and fingers to track their movements, provides a more natural and useful alternative.

A device that attaches sensors to your fingers and hands can track movements in high detail. Use cases include typing on any flat surface like it’s a keyboard.

The Gest controller has four soft, rubbery half-rings that clip onto your fingers, and a pad that straps onto the back of your hand. Each device contains 15 inexpensive sensors that measure motion with gyroscopes and accelerometers, much like those found in mobile phones. Thumb movements are inferred using data from the other sensors.

“The keyboard and mouse don’t make sense for things like augmented reality and virtual reality,” Gest cofounder Mike Pfister says. “It’s not going to succeed unless there is a really intuitive, fundamentally new input method.”

When I met Pfister in a San Francisco coffee shop, he slipped a prototype Gest controller onto each hand and began tapping on the table as if it were a keyboard. As his fingers drummed, potential words popped up on the screen. He tapped and swiped his hand to scroll through and select the correct word. “Hello, welcome to the future of interaction,” he wrote.

This prototype has 15 cheap motion sensors, similar to those inside a smartphone.

Gest doesn’t need you to set up an exact location that corresponds to each key. Instead the device uses the relative motion of your fingers to understand what the user is trying to do. It can also adapt to a person’s individual typing style over time based on the suggested words he or she selects.

The first version of Gest was focused on typing and built at a hackathon, after another of the company’s four cofounders, Sid Srikumar, read about a similar idea in the science fiction novel Accelerando, by British author Charles Stross. They raised some investment funding and participated in the Capital Factory startup incubator, in Austin, Texas.

Gest has since shifted to developing its devices as a way to use natural hand gestures to interact with virtual 3-D objects. Pfister says it could help creatives working with 3-D models or drawings. He believes it could also be a powerful interaction mechanism for the Oculus Rift, heading to market next year, or other virtual reality headsets. During the coffee shop demo, he made a fist and turned his hand to grab and rotate a model of a monkey head on the screen.

Research has shown that being able to interact with a virtual world seen through a headset like the Oculus Rift helps provide a strong sense that the virtual world is real. However, the mechanism to provide that interactivity is unclear.

This non-functional design prototype shows how the final device should look when it is released next year.

Gest will face competition from companies with a range of different controller designs, such as Leap Motion, which uses infrared and LEDs to sense hand movement, and Facebook’s Oculus division, which will next year sell a less naturalistic handheld controller that uses a combination of buttons and hand movements (see “Oculus Hand Controls Are Not Always So Handy”).

Gest plans to ship its first devices, and launch tools for software developers to integrate them into software, before the end of 2016. The company predicts a price of $200.

Keep Reading

Most Popular

still from Embodied Intelligence video
still from Embodied Intelligence video

These weird virtual creatures evolve their bodies to solve problems

They show how intelligence and body plans are closely linked—and could unlock AI for robots.

pig kidney transplant surgery
pig kidney transplant surgery

Surgeons have successfully tested a pig’s kidney in a human patient

The test, in a brain-dead patient, was very short but represents a milestone in the long quest to use animal organs in human transplants.

conceptual illustration showing various women's faces being scanned
conceptual illustration showing various women's faces being scanned

A horrifying new AI app swaps women into porn videos with a click

Deepfake researchers have long feared the day this would arrive.

thermal image of young woman wearing mask
thermal image of young woman wearing mask

The covid tech that is intimately tied to China’s surveillance state

Heat-sensing cameras and face recognition systems may help fight covid-19—but they also make us complicit in the high-tech oppression of Uyghurs.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.