We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Rating Facial Expressions

New software could help mental-health professionals assess patients and ensure that salespeople project a positive attitude.

Software that recognizes and rates smiles was demonstrated recently at an exhibition in Tokyo, where attendees competed to outsmile one another.

Say cheese: Software developed by a Japanese company detects and rates smiles. It’s the latest addition to Omron Corporation’s OKAO Vision facial-recognition software suite.

The smile-checking technology is the latest addition to Omron Corporation’s OKAO Vision software suite, which detects faces in images and can determine the person’s gender and approximate age, or verify his or her identity from a database of faces. The smile ­software is Omron’s first foray into facial-expression detection and analysis, a field that could revolutionize how humans interact with machines, and with each other.

Omron, a Japanese electronics company, won’t say if it plans to commercialize its smile software, which was on show at Japan’s Combined Exhibition of Advanced Technologies. But spokesman James Seddon says that it could be used in digital cameras to help capture people’s broadest smiles, in market research and customer-service training, and even by mental-health professionals to evaluate patients. Sony uses similar technology in some of its newest digital cameras so that they snap pictures when people are smiling their best.

Omron’s smile-measurement software picks up the hallmarks of a smile–such as narrowed eyes, an open mouth, creases around the mouth, and wrinkles turning downward around the eyes–and uses an algorithm to assess the extent of the smile and rate it on a percentage scale. The analysis is performed in real time and only takes about 44 milliseconds using a Pentium 4 3.2-gigahertz PC, Seddon says. The smile software works on images of faces as small as 60 pixels wide.

Omron engineers used about 10,000 images of human faces–some with spontaneous smiles, some with posed smiles, and others sporting different expressions–to train the software to evaluate smiles.

Omron’s smile-recognition technology is part of the company’s OKAO Vision software suite. The face-recognition component of the software analyzes patterns of light and dark in an image to determine if it includes a face. Because it’s based on contrast, Seddon says that the system performs well when dealing with poorly lit images.

Once OKAO Vision has registered a two-dimensional version of a face, it overlays a 3-D mask that allows the software to evaluate the face, even if the subject’s head is turned or she’s looking away from the camera.

Omron envisions the smile software being used in marketing, perhaps to evaluate consumers’ reactions to a new product or to an advertising campaign. A smile checker could also help train customer-service staff to meet Japan’s legendarily high standards, Seddon says.

“Clearly, it’s an interesting thing,” says Joseph Atick of L-1 Identity Solutions, based in Stamford, CT, which supplies identification technology, primarily for security applications. “If you can read people better, you can serve them better.”

A smile in isolation is easy to detect, but the bigger challenge is to develop systems that can recognize the concerto of facial actions that make up complex expressions like confusion, fear, and disgust. “You can also superimpose any other emotion on top of a smile,” says Rana el Kaliouby, a postdoctoral associate at MIT who is developing mind-reading machines. “You can have an angry smile, an interested smile–even a confused smile.”

“We’d like to be able to distinguish those,” says Jeffrey Cohn, director of the Affect Analysis Group, at the University of Pittsburgh.

Sophisticated facial-expression analysis could help mental-health professionals evaluate their patients and monitor their progress. Cohn is studying facial expressions in depressed patients and in people in physical pain.

El Kaliouby says that facial-expression analysis could help autistic people “who need help with real-time analysis of facial expressions and other social cues.”

Facial-expression analysis could also have security applications, although Atick is skeptical that a machine can reliably pick out a terrorist on its own. “That smacks of black magic, voodoo,” he says. “Tell me what my intent is standing in front of passport control. Maybe it’s a long flight and I’m annoyed, but I don’t intend to do harm.”

Become an MIT Technology Review Insider for in-depth analysis and unparalleled perspective.

Subscribe today

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

Want more award-winning journalism? Subscribe to Insider Basic.
  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning print magazine, unlimited online access plus The Download with the top tech stories delivered daily to your inbox.

    See details+

    What's Included

    Unlimited 24/7 access to MIT Technology Review’s website

    The Download: our daily newsletter of what's important in technology and innovation

    Bimonthly print magazine (6 issues per year)

You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.