Skip to Content

Get to Grips with Virtual Objects Using This Stripped-Down Glove

A new controller puts sensors on your fingers so you can type without a keyboard and manipulate digital objects.
October 29, 2015

Keyboards, mice, and touch screens all have their uses but have distinct limitations. They’re not particularly good for interacting with 3-D objects, such as those found in virtual reality or design programs. A company called Gest thinks its device, which slides onto the back of your hands and fingers to track their movements, provides a more natural and useful alternative.

A device that attaches sensors to your fingers and hands can track movements in high detail. Use cases include typing on any flat surface like it’s a keyboard.

The Gest controller has four soft, rubbery half-rings that clip onto your fingers, and a pad that straps onto the back of your hand. Each device contains 15 inexpensive sensors that measure motion with gyroscopes and accelerometers, much like those found in mobile phones. Thumb movements are inferred using data from the other sensors.

“The keyboard and mouse don’t make sense for things like augmented reality and virtual reality,” Gest cofounder Mike Pfister says. “It’s not going to succeed unless there is a really intuitive, fundamentally new input method.”

When I met Pfister in a San Francisco coffee shop, he slipped a prototype Gest controller onto each hand and began tapping on the table as if it were a keyboard. As his fingers drummed, potential words popped up on the screen. He tapped and swiped his hand to scroll through and select the correct word. “Hello, welcome to the future of interaction,” he wrote.

This prototype has 15 cheap motion sensors, similar to those inside a smartphone.

Gest doesn’t need you to set up an exact location that corresponds to each key. Instead the device uses the relative motion of your fingers to understand what the user is trying to do. It can also adapt to a person’s individual typing style over time based on the suggested words he or she selects.

The first version of Gest was focused on typing and built at a hackathon, after another of the company’s four cofounders, Sid Srikumar, read about a similar idea in the science fiction novel Accelerando, by British author Charles Stross. They raised some investment funding and participated in the Capital Factory startup incubator, in Austin, Texas.

Gest has since shifted to developing its devices as a way to use natural hand gestures to interact with virtual 3-D objects. Pfister says it could help creatives working with 3-D models or drawings. He believes it could also be a powerful interaction mechanism for the Oculus Rift, heading to market next year, or other virtual reality headsets. During the coffee shop demo, he made a fist and turned his hand to grab and rotate a model of a monkey head on the screen.

Research has shown that being able to interact with a virtual world seen through a headset like the Oculus Rift helps provide a strong sense that the virtual world is real. However, the mechanism to provide that interactivity is unclear.

This non-functional design prototype shows how the final device should look when it is released next year.

Gest will face competition from companies with a range of different controller designs, such as Leap Motion, which uses infrared and LEDs to sense hand movement, and Facebook’s Oculus division, which will next year sell a less naturalistic handheld controller that uses a combination of buttons and hand movements (see “Oculus Hand Controls Are Not Always So Handy”).

Gest plans to ship its first devices, and launch tools for software developers to integrate them into software, before the end of 2016. The company predicts a price of $200.

Keep Reading

Most Popular

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

The problem with plug-in hybrids? Their drivers.

Plug-in hybrids are often sold as a transition to EVs, but new data from Europe shows we’re still underestimating the emissions they produce.

Google DeepMind’s new generative model makes Super Mario–like games from scratch

Genie learns how to control games by watching hours and hours of video. It could help train next-gen robots too.

How scientists traced a mysterious covid case back to six toilets

When wastewater surveillance turns into a hunt for a single infected individual, the ethics get tricky.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.