March 2003

Sign Language

Translation software deciphers billboards and street signs.

By Chip Walter

In our global economy, the ability to understand languages other than one's native tongue grows more important every day. That is why Jie Yang, a computer scientist at Carnegie Mellon University, and his colleagues have created software that reads Chinese signs and quickly translates them into English using nothing more than a palm-size computer equipped with a small camera.

"If you're trying to get somewhere, it helps to be able to understand the signs around you," says Yang, who plans to develop translation modules for Japanese, Arabic, and Korean if funding is available. "There's a growing demand for products like this," he explains.

The seemingly simple task of interpreting street signs actually represents a bundle of problematic processes-from machine vision to character recognition-with which researchers in computer science and artificial intelligence have been struggling for years. Yang chose some of the most promising software solutions and squeezed them into a handheld Pocket PC device. His system starts by scanning an image for hard edges-a sign's borders. Next, it searches for cues such as contrasting, similar-size characters. Optical character recognition software "reads" the words, and other algorithms cluster them into plausible groupings.

Then comes the hard part: the actual translation. "Ambiguity is the biggest problem," says Alex Waibel, director of Carnegie Mellon's Interactive Systems Laboratory, where Yang is based. "Words can have so many different meanings in so many different languages in so many different situations." That's why a device that renders a literal translation can't always do the job. If a Chinese sign meaning "Please do not touch" were translated word for word, for example, it would say, "Please do not move hand."