Skip to Content
Artificial intelligence

The Machine Vision Algorithm Beating Art Historians at Their Own Game

Classifying a painting by artist and style is tricky for humans; spotting the links between different artists and styles is harder still. So it should be impossible for machines, right?

Few areas of academic inquiry have escaped the influence of computer science and machine learning. But one of them is the history of art. The challenge of analyzing paintings, recognizing their artists, and identifying their style and content has always been beyond the capability of even the most advanced algorithms.

That is now changing thanks to recent advances in machine learning based on approaches such as deep convolutional neural networks. In just a few years, computer scientists have created machines capable of matching and sometimes outperforming humans in all kinds of pattern recognition tasks.

Today, we see just how advanced these approaches have become in the hands of Babak Saleh and Ahmed Elgammal at Rutgers University in New Jersey. These guys have used these new machine learning techniques to train algorithms to recognize the artist and style of a fine-art painting with an accuracy that has never been achieved before.

What’s more, the results reveal connections between artists, and between entire painting styles, that art historians have labored for years to understand.

Saleh and Elgammal begin with a database of images of more than 80,000 paintings by more than a 1,000 artists spanning 15 centuries. These paintings cover 27 different styles, each with more than 1,500 examples. The researchers also classify the works by genre, such as interior, cityscape, landscape, and so on.

They then take a subset of the images and use them to train various kinds of state-of-the-art machine-learning algorithms to pick out certain features. These include general, low-level features such as the overall color, as well as more advanced features that describe the objects in the image, such as a horse and a cross. The end result is a vector-like description of each painting that contains 400 different dimensions.

The researchers then test the algorithm on a set of paintings it has not yet seen. And the results are impressive. Their new approach can accurately identify the artist in over 60 percent of the paintings it sees and identify the style in 45 percent of them.

But crucially, the machine-learning approach provides an insight into the nature of fine art that is otherwise hard even for humans to develop. This comes from analyzing the paintings that the algorithm finds difficult to classify.

For example, Saleh and Elgammal say their new approach finds it hard to distinguish between works painted by Camille Pissarro and Claude Monet. But a little research on these artists quickly reveals both were active in France in the late 19th and early 20th centuries and that both attended the Académie Suisse in Paris. An expert might also know that Pissarro and Monet were good friends and shared many experiences that informed their art. So the fact that their work is similar is no surprise.

As another example, the new approach confuses works by Claude Monet and the American impressionist Childe Hassam, who, it turns out, was strongly influenced by the French impressionists and Monet in particular.  These are links that might take a human some time to discover.

The algorithmic approach also finds links between certain artistic styles. For instance, it often confuses examples of abstract expressionism and action paintings, in which artists drip or fling paint and step on the canvas. Saleh and Elgammal again say that this kind of mix-up would be entirely understandable for a human viewer. “’Action painting’ is a type or subgenre of “abstract expressionism,’” they point out.

The algorithm picks out numerous other similarities. It links expressionism and fauvism, which might be expected given that the latter movement is often thought of as a type of expressionism. It links the mannerist and Renaissance styles, which clearly reflects that fact that mannerism is a form of early Renaissance painting.

And it finds other more straightforward links, for example between Renaissance and early Renaissance paintings, between impressionism and post-impressionism, and between cubism and its later manifestation, synthetic cubism.

These connections are well known to art historians, but only because of many decades, or indeed centuries, of scholarship. By contrast, the machine-learning approach spans just a few months.

That has important implications for the study of art history. One application of the new algorithms is to pick out paintings with similar characteristics (see images). That provides a new and powerful tool for historians to look for influences between artists that may never have been aware of.

It also allows a new form of art exploration, jumping from one image to another similar one, in a process that is visually equivalent to finding synonyms.  

Fascinating stuff!

Ref: arxiv.org/abs/1505.00855 : Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature

Deep Dive

Artificial intelligence

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

OpenAI teases an amazing new generative video model called Sora

The firm is sharing Sora with a small group of safety testers but the rest of us will have to wait to learn more.

Google’s Gemini is now in everything. Here’s how you can try it out.

Gmail, Docs, and more will now come with Gemini baked in. But Europeans will have to wait before they can download the app.

Providing the right products at the right time with machine learning

Amid shifting customer needs, CPG enterprises look to machine learning to bolster their data strategy, says global head of MLOps and platforms at Kraft Heinz Company, Jorge Balestra.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.