Skip to Content

Facial Analysis Software Spots Struggling Students

A computer can learn to recognize, and respond intelligently to, users’ emotional state.

Even a good teacher may not always be able to tell, at a glance, which students are quietly struggling and which need more of a challenge. Fortunately, laptops may soon come with enough emotional intelligence built in to do the job for them.

A recent study from North Carolina State University shows how this might work. Researchers there used video cameras to monitor the faces of college students participating in computer tutoring sessions. Using software that had been trained to match facial expressions with different levels of engagement or frustration, the researchers were able to recognize when students were experiencing difficulty and when they were finding the work too easy.

The project suggests a way for technology to help teachers keep track of students’ performance in real time. Perhaps it could even help massively open online courses (or MOOCs), which can involve many thousands of students working remotely, to be more attuned to students’ needs (see “The Crisis in Higher Education”).

It also hints at what could prove to be a broader revolution in the application of emotion-sensing technology. Computers and other devices that identify and respond to emotion—a field of research known as “affective computing”—are starting to emerge from academia. They sense emotion in various ways; some measure skin conductance, while others assess voice tone or facial expressions (see “Wearable Sensor Knows What Overwhelms You” and “Technology that Knows When to Hand You a Hankie”).

The NC State researchers’ experiment involved students who were using software called JavaTutor to learn to write Java code. The researchers analyzed 60 hours of video footage with the Computer Expression Recognition Toolbox, a program that recognizes facial expressions. They compared the software’s conclusions with the students’ own reported state of mind and found that they matched closely. The work will be presented Saturday at the Sixth International Conference on Educational Data Mining.

The ultimate goal is to develop a tutoring system that helps students who are having difficulty and “bolster their confidence and keep them motivated,” says Joseph Grafsgaard, a PhD student at NC State who coauthored a paper on the work.

A number of different researchers are exploring applications for affective computing in education. Jacob Whitehill, a software engineer and research scientist at Emotient, a startup exploring commercial uses of affective computing, recently coauthored an as-yet-unpublished paper showing that facial expressions identified by a computer could predict test performance.

“There is an emerging agreement that facial-expression recognition can play a constructive role in teaching,” Whitehill says. But he notes that feedback about students’ emotional state can impair teachers’ performance if they don’t know how or when to respond. “It’s a hard problem to know how to use these [emotion] sensors effectively,” he says.

Whitehill does, however, believe the technology could prove useful for online learning platforms. “Udacity and Coursera have on the order of a million students, and I imagine some fraction of them could be persuaded to turn their webcams on,” he says. “I think you would learn a lot about what parts of a lecture are working and what parts are not, and where students are getting confused.”

Keep Reading

Most Popular

computation concept
computation concept

How AI is reinventing what computers are

Three key ways artificial intelligence is changing what it means to compute.

still from Embodied Intelligence video
still from Embodied Intelligence video

These weird virtual creatures evolve their bodies to solve problems

They show how intelligence and body plans are closely linked—and could unlock AI for robots.

conceptual illustration showing various women's faces being scanned
conceptual illustration showing various women's faces being scanned

A horrifying new AI app swaps women into porn videos with a click

Deepfake researchers have long feared the day this would arrive.

pig kidney transplant surgery
pig kidney transplant surgery

Surgeons have successfully tested a pig’s kidney in a human patient

The test, in a brain-dead patient, was very short but represents a milestone in the long quest to use animal organs in human transplants.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.