Intelligent Machines

Camera Lets Blind People Navigate with Gestures

A wearable depth-sensing camera may soon give sightless people a better way to master their environment.

There are 39 million blind people worldwide, according to the World Health Organization.

Eelke Folmer and Vinitha Khambadkar think blind people could do without their white canes and instead navigate with a camera around their necks that gives spoken guidance in response to hand gestures.

Folmer and Khambadkar, researchers at the University of Nevada, presented the technology last week at the ACM Symposium on User Interface Software and Technology. Known as the Gestural Interface for Remote Spatial Perception, which they abbreviate as GIST, the system utilizes a Microsoft Kinect sensor to analyze and identify objects in its field of view. “GIST lets you extract information from your environment,” Folmer says.

The Nevada research draws on the ideas of MIT’s Sixth Sense project, an “augmented reality” effort in which a wearable device projects information onto the physical world and lets the user interact with it by waving, pointing, or making other hand gestures. But in contrast, GIST collects data in response to hand gestures as a way of augmenting blind people’s severely reduced spatial perception.

For example, if someone wearing GIST makes a “V” sign with the index and middle fingers, the device will identify the dominant color in the area encapsulated. If the user holds out a closed fist, the system will identify whether a person is in that direction and how far away he or she is. (See the researchers’ brief demonstration video below.)

Gestures aren’t intended to be the only means of interaction with GIST, however, thanks to the Kinect’s ability to recognize objects, faces, and speech. As Folmer explains: “You say to the sensor something like ‘This is my cup.’ You put it down on the table and say, ‘Hey, where’s my cup?’ It’ll say that it’s right in front of you.” The next trick is figuring out how to continue tracking the object when it moves farther away or behind the user.

The researchers also plan to see whether GIST could effectively tell its wearers who is in front of them, by comparing the faces of people it detects to a small database that the user could set up with voice commands. If that doesn’t work, Folmer says, the researchers might try a system that identifies people based on their body shape.

GIST will soon benefit from the new Kinect 2.0 sensor, which improves upon the original’s tracking and allows precise finger recognition—as opposed to merely hands. The caveat, though, is that the new Kinect is bigger and bulkier, which makes it ill-suited to prolonged wear around the neck.

But Folmer believes it’s just a matter of time until there are Kinect-like sensors small enough to fit into smartphones—in fact, the process is already under way (see “Depth-Sensing Cameras Head to Mobile Devices”). And overall, Folmer believes that as mainstream computing devices get smaller, they will be very useful to vision-impaired people because such devices will rely on interfaces, such as speech, that the blind have been using for decades.

Cut off? Read unlimited articles today.

Become an Insider
Already an Insider? Log in.

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning magazine and daily delivery of The Download, our newsletter of what’s important in technology and innovation.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.