Skip to Content

An App to Turn Sign Language to Text

Could portable computing transform the ways the deaf communicate?
March 14, 2012

Scientists are working on an app that they say could act as a sort of translator for the deaf. Specifically, the app would leverage the video camera on a portable device to capture sign language and render it as text. The technology, developed by Technabling, a spin-out of the University of Aberdeen, is being called the portable sign language translator, or PSLT.

Said Ernesto Compatangelo, one of the technology’s developers, in a statement: ”The user signs into a standard camera integrated into a laptop, netbook, smart phone or other portable device such as a tablet. Their signs are immediately translated into text, which can be read by the person they are conversing with …The intent is to develop an application—an ‘app’ in smart phone terms—that is easily accessible and could be used on different devices, including smart phones, laptops, and PCs.”

I have to confess that I have minimal personal experience communicating with the deaf, so it’s hard for me to get my head around how exactly this app would be more effective than, say, using pad and paper to communicate. Would the deaf user have to position the device at a distance, sign, and then hand the device over to his interlocutor for reading? It seems to me that the best way for this app to enable a seamless, Babelfish-like experience would be for the hearing person to have his smart phone loaded with the app, have his camera trained on the person who is signing, and furthermore have headphones immediately rendering the text into speech. Technabling, for now, appears to be focusing on the gesture recognition tech, and is less concerned with exactly how the tech would be implemented in app form.

Setting aside the exact manner of implementation, there are actually other, less intuitive reasons to get excited about such an app. One of the coolest things about the app is that it would let users create their own private, or semi-private, languages. Writes Technabling:

“This means that any signer can create her/his own set of signs and … and associate to them their own words and concepts. In this way, signers can bridge the current communication gap with the wider community around them, being able to use whatever jargon they need in whatever situation they may find themselves (e.g., in education, in training, at work, at home, on the go).”

In other words, say you’re hard of hearing, and you’re also extremely interested in computer science. Plenty of new jargon and terminology is constantly emerging in computer science, but American Sign Language is unlikely to keep up with all that jargon. You could use the app to invent signs to express these bits of jargon, saving you the trouble of having to spell out every word letter by letter.

The scientists also see a possible application that doesn’t involve being deaf or hard of hearing at all. They think that people with “reduced mobility and speech difficulties due to ill health or accidents” might also use the technology to issue commands to smart appliances around the house, “using a simple but effective set of hand gestures tailored to their physical capabilities.” In other words, the app might eventually become something like a much smarter version of those clap-on-clap-off lights whose commercials were a parody staple on playgrounds some 30 years ago.

Keep Reading

Most Popular

open sourcing language models concept
open sourcing language models concept

Meta has built a massive new language AI—and it’s giving it away for free

Facebook’s parent company is inviting researchers to pore over and pick apart the flaws in its version of GPT-3

transplant surgery
transplant surgery

The gene-edited pig heart given to a dying patient was infected with a pig virus

The first transplant of a genetically-modified pig heart into a human may have ended prematurely because of a well-known—and avoidable—risk.

Muhammad bin Salman funds anti-aging research
Muhammad bin Salman funds anti-aging research

Saudi Arabia plans to spend $1 billion a year discovering treatments to slow aging

The oil kingdom fears that its population is aging at an accelerated rate and hopes to test drugs to reverse the problem. First up might be the diabetes drug metformin.

Yann LeCun
Yann LeCun

Yann LeCun has a bold new vision for the future of AI

One of the godfathers of deep learning pulls together old ideas to sketch out a fresh path for AI, but raises as many questions as he answers.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.