Lip-reading is notoriously difficult, depending as much on context and knowledge of language as it does on visual clues. But researchers are showing that machine learning can be used to discern speech from silent video clips more effectively than professional lip-readers can.
In one project, a team from the University of Oxford’s Department of Computer Science has developed a new artificial-intelligence system called LipNet. As Quartz reported, its system was built on a data set known as GRID, which is made up of well-lit, face-forward clips of people reading three-second sentences. Each sentence is based on a string of words that follow the same pattern.
The team used that data set to train a neural network, similar to the kind often used to perform speech recognition. In this case, though, the neural network identifies variations in mouth shape over time, learning to link that information to an explanation of what’s being said. The AI doesn’t analyze the footage in snatches but considers the whole thing, enabling it to gain an understanding of context from the sentence being analyzed. That’s important, because there are fewer mouth shapes than there are sounds produced by the human voice.
When tested, the system was able to identify 93.4 percent of words correctly. Human lip-reading volunteers asked to perform the same tasks identified just 52.3 percent of words correctly.
But as New Scientist reports, another team from Oxford’s Department of Engineering Science, which has been working with Google DeepMind, has bitten off a rather more difficult task. Instead of using a neat and consistent data set like GRID, it’s been using a series of 100,000 video clips taken from BBC television. These videos have a much broader range of language, with far more variation in lighting and head positions.
Using a similar approach, the Oxford and DeepMind team managed to create an AI that was able to identify 46.8 percent of all words correctly. That’s also far better than humans, who recorded just 12.4 percent of words without a mistake. There are clearly lots of reasons why the accuracy is lower, from lighting and orientation to the greater language complexity.
Differences aside, both experiments show AI vastly outperforming humans at lip-reading, and it’s not hard to imagine potential applications for such software. In the future, Skype could fill in the gaps when a caller is in a noisy environment, say, or people with hearing difficulties could hold their smartphone up to “hear” what someone is saying.
A horrifying new AI app swaps women into porn videos with a click
Deepfake researchers have long feared the day this would arrive.
The therapists using AI to make therapy better
Researchers are learning more about how therapy works by examining the language therapists use with clients. It could lead to more people getting better, and staying better.
DeepMind says its new language model can beat others 25 times its size
RETRO uses an external memory to look up passages of text on the fly, avoiding some of the costs of training a vast neural network
2021 was the year of monster AI models
GPT-3, OpenAI’s program to mimic human language, kicked off a new trend in artificial intelligence for bigger and bigger models. How large will they get, and at what cost?
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.