What's Next for the Netflix Algorithms?

Applying the lessons of the $1 million competition to other recommendation systems will be a challenge.

When the Netflix Prize was awarded last month, it ended three years of intense competition aimed at finding a better algorithm for predicting users' movie preferences.

Credit: Technology Review

The winning team, BellKor's Pragmatic Chaos, was the first to forecast Netflix customers' movie ratings with 10 percent better accuracy than the company's in-house system--a feat that many experts believed would be impossible when the million-dollar prize was announced. Netflix plans to offer a second prize, this time for algorithms that predict movie preferences using more user information, such as gender, age, and zip code. But experts say that the real challenge is to find ways to apply the lessons learned through the original Netflix challenge to other recommendation systems.

At the end of October, experts in the field will meet at the ACM Conference on Recommender Systems in New York City to ask, among other things, what has been learned from the Netflix Prize.

Participants in the original Netflix competition trained their algorithms using an enormous collection of data: more than 100 million ratings covering almost 18,000 titles from nearly half a million subscribers. To test their results, their algorithms were tested on a set of data maintained by Netflix and kept secret from the contests to prevent cheating.

Netflix's data presented several formidable obstacles, explains Nicholas Ampazis, an assistant professor in the department of financial and management engineering at the University of the Aegean in Greece, whose team, The Ensemble, ended the contest in second place. The dataset was huge, but it was also sparse, meaning that customers typically rated about 1 percent of the movies they watched. "Cracking the 10 percent barrier thus meant pushing the limits of existing modeling techniques to a significant degree," says Ampazis.

Story continues below

But the challenges presented by the Netflix data also made the competition very valuable, according to Ces Bertino, another member of The Ensemble. Researchers usually have the luxury of choosing datasets, and of having more information about that data. In the Netflix contest, the contestants were forced to apply all algorithms to the same set of frustratingly uneven real-world data. "Because people had to use a fixed dataset, they needed to deal not only with the advantages of a particular method, but also the weaknesses of it," Bertino says. "You could not escape it."

Gavin Potter, who gained recognition for his breaking the top 10 of the Netflix prize in 2008 under the name "Just a guy in a garage," says that a few key realizations allowed the winning algorithms to meet the goal. First, a powerful algorithm for searching for patterns in datasets, a technique known as collaborative filtering, was streamlined so that it could be used on the large Netflix dataset. Second, participants learned to pay attention to certain new types of details, for example the fact that ordering a movie at all indicates some preference for it, even if the customer didn't rate it. Date and time information also proved significant. But the biggest realization, Potter notes, was that blending a variety of approaches yielded the best results.