The U.S. Defense Advanced Research Projects Agency (DARPA) is one of the major funders of statistical machine translation. Last August, DARPA sponsored machine translation tests for Chinese and Arabic documents; a research group from Google scored the highest, nudging out USC’s Information Sciences Institute and IBM’s machine translation arm. Google, which also uses the statistical approach, might have had an edge, Knight notes, because they could use a huge number of computers for the word-crunching, and could draw from the entire Internet for their database of pretranslated documents.

In 2005, DARPA also announced the Global Autonomous Language Exploitation (GALE) program, intended to speed up the computer processing of huge numbers of translated documents acquired by its parent program, the Philadelphia-based Linguistic Data Consortium.** GALE is currently in the first year and will be transcribing speech from broadcast news sources and talk shows in Arabic, Chinese, and English, and also cataloguing text newswire feeds, Web news discussion groups, and blogs in those languages. For now, the project is focused mainly on data collection from these genres, with researchers in the computer and engineering science department at the University of Pennsylvania doing much of the work.

But even with a large collection of translated material, there will still be language issues to sort out. The next step in machine translation research, beyond matching words and phrases, Knight says, is to smooth out the grammatical inconsistencies that arise when words and phrases are strung together. This smoothing can be accomplished by indexing millions of sentences whose structures have been diagrammed at the University of Pennsylvania in the 1990s (the data came from 50,000 sentences in the Wall Street Journal). Similar to the way a database full of words and phrases allows translation software to choose the most statistically probable combination of words, these specific examples of grammar from the diagrammed sentences help the software assign the likelihood of word order, says MIT’s Collins.

This is an advance over the traditional method in which grammar rules were set in an algorithm, he says. Rather than obeying encoded grammar conventions in an algorithm, as with traditional machine translation, the diagrammed sentence database lets the software assign “probabilities and weight on those rules” says Collins. “[The software] is more likely to learn the context,” he says.

In some ways, however, the statistical approach will only be as good as the common instant-messaging translator. Proper names, for instance, still trip up even the most well-read machine translator, and they often just get translated along with the rest of the text. According to his system, Knight admits, the Spanish version of his surname is still “Kevin Caballero.”

** Correction, January 20, 2006: The original version of this story, published January 18, stated that the Linguistic Data Consortium was launched in 2005. In fact, the consortium was launched in 1992, and its Global Autonomous Language Exploitation project was launched in 2005. -- Eds.