Skip to Content

Facial Analysis Software Spots Struggling Students

A computer can learn to recognize, and respond intelligently to, users’ emotional state.

Even a good teacher may not always be able to tell, at a glance, which students are quietly struggling and which need more of a challenge. Fortunately, laptops may soon come with enough emotional intelligence built in to do the job for them.

A recent study from North Carolina State University shows how this might work. Researchers there used video cameras to monitor the faces of college students participating in computer tutoring sessions. Using software that had been trained to match facial expressions with different levels of engagement or frustration, the researchers were able to recognize when students were experiencing difficulty and when they were finding the work too easy.

The project suggests a way for technology to help teachers keep track of students’ performance in real time. Perhaps it could even help massively open online courses (or MOOCs), which can involve many thousands of students working remotely, to be more attuned to students’ needs (see “The Crisis in Higher Education”).

It also hints at what could prove to be a broader revolution in the application of emotion-sensing technology. Computers and other devices that identify and respond to emotion—a field of research known as “affective computing”—are starting to emerge from academia. They sense emotion in various ways; some measure skin conductance, while others assess voice tone or facial expressions (see “Wearable Sensor Knows What Overwhelms You” and “Technology that Knows When to Hand You a Hankie”).

The NC State researchers’ experiment involved students who were using software called JavaTutor to learn to write Java code. The researchers analyzed 60 hours of video footage with the Computer Expression Recognition Toolbox, a program that recognizes facial expressions. They compared the software’s conclusions with the students’ own reported state of mind and found that they matched closely. The work will be presented Saturday at the Sixth International Conference on Educational Data Mining.

The ultimate goal is to develop a tutoring system that helps students who are having difficulty and “bolster their confidence and keep them motivated,” says Joseph Grafsgaard, a PhD student at NC State who coauthored a paper on the work.

A number of different researchers are exploring applications for affective computing in education. Jacob Whitehill, a software engineer and research scientist at Emotient, a startup exploring commercial uses of affective computing, recently coauthored an as-yet-unpublished paper showing that facial expressions identified by a computer could predict test performance.

“There is an emerging agreement that facial-expression recognition can play a constructive role in teaching,” Whitehill says. But he notes that feedback about students’ emotional state can impair teachers’ performance if they don’t know how or when to respond. “It’s a hard problem to know how to use these [emotion] sensors effectively,” he says.

Whitehill does, however, believe the technology could prove useful for online learning platforms. “Udacity and Coursera have on the order of a million students, and I imagine some fraction of them could be persuaded to turn their webcams on,” he says. “I think you would learn a lot about what parts of a lecture are working and what parts are not, and where students are getting confused.”

Keep Reading

Most Popular

This new data poisoning tool lets artists fight back against generative AI

The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models. 

Rogue superintelligence and merging with machines: Inside the mind of OpenAI’s chief scientist

An exclusive conversation with Ilya Sutskever on his fears for the future of AI and why they’ve made him change the focus of his life’s work.

Data analytics reveal real business value

Sophisticated analytics tools mine insights from data, optimizing operational processes across the enterprise.

Driving companywide efficiencies with AI

Advanced AI and ML capabilities revolutionize how administrative and operations tasks are done.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.