Skip to Content

The Algorithm Expanding the Science of Color

There is no natural way to place colors in order, and that makes comparing color palettes difficult. One algorithm looks set to change that.
The color palettes extracted from fine art paintings.

The study of color is undergoing a revolution. Until now, color theory has been little more than a collection of ideas based loosely on science.

But the ability to analyze huge data sets of images using machine-vision algorithms is changing that. Suddenly it has become possible to study color and the way it is used in entirely different ways. For example, it is now easy to extract combinations of colors from one picture and paste them onto another.

But this process of color pasting is a blunt tool. It can replace one set of colors with another but it cannot compare these color palettes, decide how specific colors should be matched, or discover whether a palette is missing a color and replace it.

The problem is that there is no way of placing colors in order. And that makes it hard to compare palettes. So image specialists would dearly love to have a natural way to order colors extracted from images and so be able to compare them.  

Enter Huy Phan at the Technical University of Denmark in Copenhagen and a couple of pals, who have found a way to do just that. Their technique is straightforward to implement and immediately leads to a new generation of enriched image filters that are far more flexible than those currently available.

The basic problem is simple to state: given the color palette from two images, what order should the colors appear in to make a meaningful comparison? By meaningful, the team means that colors describing the same objects should be comparable.

Phan and co tackle this as a kind of sorting problem. They first assess an image by plotting the position of the colors it contains—its palette—in a three-dimensional color “space” (every color can be thought of as a three-vector describing how it is composed from a mixture of red, green, and blue, for example).

They then measure the distance between each pair of colors in the palette. Finally, they use an algorithm to work out how to map one palette onto another with minimal distortion. Effectively, this algorithm finds clusters within each palette’s color space.

The data set they use is important. “We consider color palettes extracted from fine art collections, which we believe to be an abundant source of stylistic and unique color themes,” say Phan and co.

That also limits the subject matter, which is crucial. Henri Matisse often used deep reds and greens in his paintings, for example, while Maximilien Luce used strong blues and yellows to shade objects. But because these artists use broadly similar scenes (for example, divided into sky and land), the colors are comparable.

It means that the team can assume that similar clusters of colors describe similar objects. The algorithm then maps these clusters onto each other. In that way, it can take the coloring of sky in one image and apply the same set of colors to the sky in another image.

And because the transformation is applied to the whole space, it becomes straightforward to map any color from palette onto a corresponding color in the other palette.

That leads to a natural way to order colors. Phan and co do it by first specifying the number of colors that appear in each cluster. Phan and co use five as their standard example (higher numbers require significantly more computational resources).

The algorithm then finds the five colors in the first cluster, then in the second, third and so on. Ordering these clusters then allows the palettes in each image to be easily compared.

The method immediately leads to some interesting applications. For example, they create an application called Photo-style Explorer which acts like an Instagram filter but in a continuous color space. “Instead of choosing among a few predefined themes, one can freely surf a continuous space of possible photo colorizations to pick a preferred theme,” they say.

Another option is to recolor photos with different palettes in different parts of the image. So it is possible to use one palette to recolor the sky and another to recolor trees.

It also allows a better analysis of the color signatures of artists and for these signatures to be transferred to other images. So it becomes possible to quickly recolor a photograph using the Renoir’s color palette or Vincent van Gogh’s.

That’s interesting work which expands the way color can be used in modern images.

Ref: http://arxiv.org/abs/1703.06003: Color Orchestra: Ordering Color Palettes for Interpolation and Prediction

Keep Reading

Most Popular

light and shadow on floor
light and shadow on floor

How Facebook and Google fund global misinformation

The tech giants are paying millions of dollars to the operators of clickbait pages, bankrolling the deterioration of information ecosystems around the world.

This new startup has built a record-breaking 256-qubit quantum computer

QuEra Computing, launched by physicists at Harvard and MIT, is trying a different quantum approach to tackle impossibly hard computational tasks.

wet market selling fish
wet market selling fish

This scientist now believes covid started in Wuhan’s wet market. Here’s why.

How a veteran virologist found fresh evidence to back up the theory that covid jumped from animals to humans in a notorious Chinese market—rather than emerged from a lab leak.

protein structures
protein structures

DeepMind says it will release the structure of every protein known to science

The company has already used its protein-folding AI, AlphaFold, to generate structures for the human proteome, as well as yeast, fruit flies, mice, and more.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.