Intelligent Machines

Machine-Vision Algorithm Learns to Transform Hand-Drawn Sketches Into Photorealistic Images

Deep neural networks are beginning to outperform humans in a rapidly increasing variety of vision-related tasks.

Drawing an accurate sketch of a person’s face is an art that is hard for most people to master. But it turns out to be relatively easy for computers. Various programs exist for converting images into line drawings. That often produces a decent start, although these systems can have difficulty with shadows and high contrast.

A more promising approach is to use machine-vision algorithms that rely on neural networks to extract features from an image and use these to produce a sketch. In this area, machines have begun to rival and even outperform humans in producing accurate sketches.

But what of the inverse problem? This starts with a sketch and aims to produce an accurate color photograph of the original face. That’s clearly a much harder task, so much so that humans rarely even try.

Now the machines have cracked this problem. Today, Yagmur Gucluturk, Umut Guclu, and pals at Radboud University in the Netherlands have taught a neural network to turn hand-drawn sketches of faces into photorealistic portraits. The work is yet another demonstration of the way intelligent machines, and neural networks in particular, are beginning to outperform humans in an increasingly wide variety of tasks.

The Radboud team begins with a data set of 200,000 images of faces taken from the Internet. They converted these to line drawings, grayscale sketches and color sketches, using standard image-processing algorithms. This created a substantial training data set to teach a deep convolutional neural network with 11-layers to do the task in reverse: turn a sketch into a photorealistic color photograph of a face.

Having trained the net, the team then put it through its paces using another data set. The task for the neural net was to start with the sketch and produce a photorealistic image.

The results are impressive, particularly when the neural network started with line drawings. ”We found that the line model performed impressively in terms of matching the hair and skin color of the individuals even when the line sketches did not contain any color information,” say the team.

The reason seems clear. “This may indicate that along with taking advantage of the luminance differences in the sketches to infer coloring, the model was able to learn color properties often associated with high-level face features of different ethnicities,” they suggest.

Next the team tested the neural net on an entirely different data set using hand-drawn sketches which are obviously generated in an entirely different way from those the net was trained on. “Once again, the model synthesized photorealistic face images,” they say.

The net wasn’t perfect, of course. In particular, it had trouble when pencil strokes in hand-drawn sketches were not accompanied by shading. This led to in less realistic results. “This can be explained by the lack of such features in the training data of the line sketch model,” they say, adding that this shortcoming can be easily overcome by including examples in the training set that more closely resembling the drawing style of the sketch artists,” say the Radboud researchers.

Finally, the team allowed their neural net to showboat by producing photorealistic images of artists such as Rembrandt and Van Gogh from sketched self-portraits that both artists had made.

That’s impressive work that shows how deep neural networks are rapidly beginning to outperform humans in vision tasks. The obvious immediate application is in forensics, where accurate images of suspects have to be constructed by police artists.

And this is just one way of many ways that these machines are showing their supremacy. Machine-vision algorithms can also do tasks such as copy and paste artistic style from one image to another, add color to grayscale images accurately, and transform low-resolution images into high resolution ones.

All this has happened in just a couple of years of development, and deep neural networks have much further to go. It’s hard to predict what they will be capable of in, say, just a year’s time.

Ref: arxiv.org/abs/1606.03073: Convolutional Sketch Inversion

 

The latest Insider Conversation is live! Listen to the story behind the story.

Subscribe today
Already a Premium subscriber? Log in.

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning magazine and daily delivery of The Download, our newsletter of what’s important in technology and innovation.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.