Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo


Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

Criticism of Google Glass has often focused on the way its camera makes surreptitious video recording too easy. Now researchers have shown that footage captured by the face-mounted camera could also pose a security threat.

Software developed by the researchers can automatically recover the passcodes of people recorded on video as they type in their credentials, even when the screen itself is not visible to the camera. The attack works by watching the movement of the fingers to work out what keys they are touching. It also works on footage from camcorders, webcams, and smartphones, but Glass offers perhaps the subtlest way to stage it.

The work suggests that “shoulder surfing”—stealing passwords or other data by watching someone at a computer—could become more of a threat as digital cameras and powerful image processing software become more common.

In tests where people stood three meters away from the camera, the software was around 90 percent accurate at capturing four-character-long strings typed on the iPhone’s QWERTY keyboard. The researchers say that the method could theoretically reconstruct a short e-mail or SMS.

“With Glass it’s very sneaky,” says Qinggang Yue, a grad student at the University of Massachusetts, Lowell, who carried out the research with colleagues Xinwen Fu and Zhen Ling.

When Yue met with MIT Technology Review at the Black Hat security conference, where he had presented his findings on Wednesday, he glanced around the busy press room and instantly identified a handful of people pecking away on touch screens that might be vulnerable to such an attack.

Yue has also shown that video footage can be used to recover passcodes at some distance. In one set of experiments, a camcorder held by someone at a first-floor window was used to successfully capture the passcode of someone using an iPad just over 43 meters away. “With a long-focal-length camera it could be much further,” says Yue.

To capture a passcode, the software must identify the position and orientation of a device’s screen as well as the position of a person’s fingertips tapping on it. Yue and colleagues used machine learning to train software to tackle both those problems. The software runs on a PC, so footage captured with Google Glass must be downloaded to extract any passcodes.

The software automatically finds a device captured in a piece of footage. It then identifies the position of its screen’s four corners, and tracks the velocity of a person’s fingertip.

The researchers are currently testing ways to defend against such software-enhanced shoulder surfing. One countermeasure involves randomly swapping the keys on a standard keypad around, so that software can’t correctly translate each tap. Another involves having buttons drift around instead of staying fixed to a standard grid.

4 comments. Share your thoughts »

Tagged: Computing, Google, EmTech2014, Black Hat, Black Hat security conference, Google Glass, password theft

Reprints and Permissions | Send feedback to the editor

From the Archives


Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me