Skip to Content

Beauty Now in the Eye of the Algorithm

New image recognition technology judges photographic aesthetics.
November 17, 2011

New technology from Xerox can sort photos not just by their content but also according to their aesthetic qualities, such as which portraits are close-in and well-lit, or which wildlife shots are least cluttered.

Pleasing portrait: The Xerox algorithm deemed the portrait on the left high quality in part because of its lighting and simple black background, while judging the one on the right low quality because of its washed-out lighting.

Still in the prototype stage, the technology could eventually help with tasks like choosing which of hundreds of digital photos taken on a family vacation should appear in a photo album. It could help stock agencies sort photos by their characteristics, and it could be deployed inside a camera to help people delete lower-quality scenes quickly, saving on storage space and hassle.

“What they show is that now you don’t need a human to select images that are going to be judged beautiful,” says Aude Oliva, an associate professor of brain and cognitive sciences at MIT, who also works on image recognition. “You can run the algorithm, and it will give a good estimate.”

The technology—developed at the Xerox Research Center Europe in Grenoble, France—is slated for beta testing with Xerox corporate partners next year, says Craig Saunders, manager of the computer vision research group there. These partners include graphic design firms, online photo-book companies, and stock agencies, all of which might want new ways to sort and find photos.

The Xerox system learns about quality photography by studying photos that had previously been chosen for public display in online photo albums, such as public ones shown on Facebook, or photos tagged as high quality on Flickr. Then it notes common characteristics of these photos.

Not surprisingly, these characteristics often correspond to what experts already understand about good photographs. The best portraits of people, for example, have indirect lighting and blurry or monochromatic backgrounds that help keep the focus on the person. Good beach photos often include silky-looking waves, a trick achieved through slow shutter speeds. And many kinds of photos are appealing because they follow the “rule of threes,” with subjects divided among three zones in the photo. “We try to learn what it is about these features that makes photos ‘good,’” says Saunders. (Examples and demonstrations can be found here.)

Facing the algorithm: Other portraits on which the Xerox system passed judgment include the high-quality ones on the left and the low-quality ones on the right.

The technique builds on a larger body of research, conducted at Xerox and many other labs, that strives to improve image recognition by breaking down photos into what researchers call a visual vocabulary—corners and edges that might define buildings, round shapes that might be wheels, regions of green that might indicate landscapes, and many more such elements (and combinations thereof). The resulting technologies build up knowledge about what pieces correspond to certain types of images by examining Internet-based photos that are already tagged with text identifying what’s in them.

Many research groups, including the one at Xerox, are working on improving not only the accuracy of these methods but also their computational efficiency. For example, Xerox announced recently that it has developed a system capable of finding images that have similar characteristics. It can sort through five million images in less than a second.

Xerox plans to launch this tool next year as a cloud-based service that could be used to refine searches in large image repositories like stock photo agencies. The company also released a related Facebook app, called Catepix, that examines your Facebook photos, categorizes them (portrait, landscape, etc.), and tells you what they say about your personality.

Unfortunately, I have posted only three pictures on Facebook, so the app failed to tell me much of anything. But it did put up a post under my name declaring that I was a portrait kind of guy.

Keep Reading

Most Popular

DeepMind’s cofounder: Generative AI is just a phase. What’s next is interactive AI.

“This is a profound moment in the history of technology,” says Mustafa Suleyman.

What to know about this autumn’s covid vaccines

New variants will pose a challenge, but early signs suggest the shots will still boost antibody responses.

Human-plus-AI solutions mitigate security threats

With the right human oversight, emerging technologies like artificial intelligence can help keep business and customer data secure

Next slide, please: A brief history of the corporate presentation

From million-dollar slide shows to Steve Jobs’s introduction of the iPhone, a bit of show business never hurt plain old business.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.