Searching Video Lectures

A tool from MIT finds keywords so that students can efficiently review lectures.

Researchers at MIT have released a video and audio search tool that solves one of the most challenging problems in the field: how to break up a lengthy academic lecture into manageable chunks, pinpoint the location of keywords, and direct the user to them. Announced last month, the MIT Lecture Browser website gives the general public detailed access to more than 200 lectures publicly available though the university's OpenCourseWare initiative. The search engine leverages decades' worth of speech-recognition research at MIT and other institutions to convert audio into text and make it searchable.

Looking at lectures: MIT is offering a video search tool that can pinpoint keywords in audio and video lectures. Here, a search for “exoskeleton and gasoline” results in this video clip. The automated transcript of the lecture appears below the video. Credit: MIT

The Lecture Browser arrives at a time when more and more universities, including Carnegie Mellon University and the University of California, Berkeley, are posting videos and podcasts of lectures online. While this content is useful, locating specific information within lectures can be difficult, frustrating students who are accustomed to finding what they need in less than a second with Google.

"This is a growing issue for universities around the country as it becomes easier to record classroom lectures," says Jim Glass, research scientist at MIT. "It's a real challenge to know how to disseminate them and make it easier for students to get access to parts of the lecture they might be interested in. It's like finding a needle in a haystack."

The fundamental elements of the Lecture Browser have been kicking around research labs at MIT and places such as BBN Technologies in Boston, Carnegie Mellon, SRI International in Palo Alto, CA, and the University of Southern California for more than 30 years. Their efforts have produced software that's finally good enough to find its way to the average person, says Premkumar Natarajan, scientist at BBN. "There's about three decades of work where many fundamental problems were addressed," he says. "The technology is mature enough now that there's a growing sense in the community that it's time [to test applications in the real world]. We've done all we can in the lab."

Story continues below

A handful of companies, such as online audio and video search engines Blinkx and EveryZing (which has licensed technology from BBN) are making use of software that converts audio speech into searchable text. (See "Surfing TV on the Internet" and "More-Accurate Video Search".) But the MIT researchers faced particular challenges with academic lectures. For one, many lecturers are not native English speakers, which makes automatic transcription tricky for systems trained on American English accents. Second, the words favored in science lectures can be rather obscure. Finally, says Regina Barzilay, professor of computer Science at MIT, lectures have very little discernable structure, making them difficult to break up and organize for easy searching. "Topical transitions are very subtle," she says. "Lectures aren't organized like normal text."

To tackle these problems, the researchers first configured the software that converts the audio to text. They trained the software to understand particular accents using accurate transcriptions of short snippets of recorded speech. To help the software identify uncommon words--anything from "drosophila" to "closed-loop integrals"--the researchers provided it with additional data, such as text from books and lecture notes, which assists the software in accurately transcribing as many as four out of five words. If the system is used with a nonnative English speaker whose accent and vocabulary it hasn't been trained to recognize, the accuracy can drop to 50 percent. (Such a low accuracy would not be useful for direct transcription but can still be useful for keyword searches.)