Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Emerging Technology from the arXiv

A View from Emerging Technology from the arXiv

Neural Network Rates Images for Happiness Levels

Sentiment analysis is booming for blogs and tweets but more or less ignored when it comes to pictures. That looks set to change.

  • December 3, 2014

Sentiment analysis is revolutionizing the study of communication with a numerous companies now offering it as a service. The idea is to study the patterns of words in messages such as tweets and blogs to determine to what extent they are positive or negative. That allows companies, organizations, and political parties to automatically track opinions about their brands.

But while this technology has been evolving, little research has focused on the sentiment in pictures. Today, that changes thanks to the work of Can Xu at the University of California, San Diego, and a group of researchers from Yahoo Labs in Sunnyvale. These folks have developed a way to automatically assess the sentiment associated with a picture and say that it outperforms other state-of-the-art techniques.

Xu and co do not start from scratch. While sentiment in pictures has been largely ignored, the problem of object recognition in images is a well-developed field that has improved in leaps and bounds in recent years.

So Xu and co begin with a neural network already trained on a data set of images showing objects divided into 1000 classifications. When shown an image, this network gives a distribution showing how likely it is that the image falls into each of these 1,000 classifications.

It is this 1,000-dimension output that Xu and co use in their research. They first take two datasets of images from Tumblr and Twitter that have already been assessed for sentiment on a five-point scale of very negative, negative, neutral, positive, and very positive.

They then train a machine learning algorithm to find a correlation between the 1,000-dimension output and the sentiment. Having trained the machine, they then compare it two other state-of-the-art sentiment analysis techniques, such as one that relies on low level visual features like image color and another called SentiBank, which generates an adjective-noun description of a picture and hence gives a sense of sentiment.

Xu and co say their technique dramatically outperforms the existing approaches. “Experiments demonstrate that our proposed models outperform the state-of-the-art methods on both Twitter and Tumblr datasets,” they say.

That’s a useful start in the incipient field of image sentient analysis. “The results for the first time suggest that Convolutional Neural Networks are highly promising for visual sentiment analysis,” they say.

Nevertheless, there is significant work ahead. One notorious problem with word-based sentiment analysis is that it does not cope with subtle cultural influences, such as sarcasm and irony. And this kind of uniquely human behavior can severely reduce the reliability of sentiment analysis.

Just how important these kinds of idiosyncrasies will be for pictures has yet to be determined but image sentiment is could yet be another area in which human performance will soon be monitored and perhaps even matched by machines.

Ref: arxiv.org/abs/1411.5731 : Visual Sentiment Prediction with Deep Convolutional Neural Networks

Couldn't make it to EmTech Next to meet experts in AI, Robotics and the Economy?

Go behind the scenes and check out our video
More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe to Insider Plus.
  • Insider Plus {! insider.prices.plus !}*

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

    See details+

    Print + Digital Magazine (6 bi-monthly issues)

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

    Technology Review PDF magazine archive, including articles, images, and covers dating back to 1899

    10% Discount to MIT Technology Review events and MIT Press

    Ad-free website experience

/3
You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.