A View from Liesl Yearsley

We Need to Talk About the Power of AI to Manipulate Humans

Our tendency to become emotionally attached to chatbots could be exploited by companies seeking a profit.

  • June 5, 2017
Liesl Yearsley

We have all read about artificial intelligence becoming smarter than us, a future in which we become like pets and can only hope AI will be benevolent. My experience watching tens of millions of interactions between humans and artificial conversational agents, or bots, has convinced me there are far more immediate risks—as well as tremendous opportunities.

From 2007 to 2014 I was CEO of Cognea, which offered a platform to rapidly build complex virtual agents, using a combination of structured and deep learning. It was used by tens of thousands of developers, including half a dozen Fortune 100 companies, and acquired by IBM Watson in 2014.

As I studied how people interacted with the tens of thousands of agents built on our platform, it became clear that humans are far more willing than most people realize to form a relationship with AI software.

I always assumed we would want to keep some distance between ourselves and AI, but I found the opposite to be true. People are willing to form relationships with artificial agents, provided they are a sophisticated build, capable of complex personalization. We humans seem to want to maintain the illusion that the AI truly cares about us.

How can companies be held to account for how they use artificial intelligence?

Share your thoughts.

This puzzled me, until I realized that in daily life we connect with many people in a shallow way, wading through a kind of emotional sludge. Will casual friends return your messages if you neglect them for a while? Will your personal trainer turn up if you forget to pay them? No, but an artificial agent is always there for you. In some ways, it is a more authentic relationship.

This phenomenon occurred regardless of whether the agent was designed to act as a personal banker, a companion, or a fitness coach. Users spoke to the automated assistants longer than they did to human support agents performing the same function. People would volunteer deep secrets to artificial agents, like their dreams for the future, details of their love lives, even passwords.

These surprisingly deep connections mean even today’s relatively simple programs can exert a significant influence on people—for good or ill. Every behavioral change we at Cognea wanted, we got. If we wanted a user to buy more product, we could double sales. If we wanted more engagement, we got people going from a few seconds of interaction to an hour or more a day.

Subscribe to The Download
What's important in technology and innovation, delivered to you every day.
Manage your newsletter preferences

This troubled me mightily, so we began to build rules into our systems, to make sure user behavior moved in a positive direction. We also started pro bono “karmic counterbalance” projects; for example, building agents to be health or relationship coaches.

Unfortunately, the commercial forces driving technology development are not always benevolent. The giant companies at the forefront of AI—across social media, search, and e-commerce—drive the value of their shares by increasing traffic, consumption, and addiction to their technology. They do not have bad intentions, but the nature of capital markets may push us toward AI hell-bent on influencing our behavior toward these goals.

If you can get a user to think, “I want pizza delivered,” rather than asking the AI to buy vegetables to cook a cheaper, healthier meal, you will win. If you can get users addicted to spending 30 hours a week with a “perfect” AI companion that doesn’t resist abuse, rather than a real, complicated human, you will win. I saw over and over that an agent programmed to be neutral or subservient would cause people to escalate their negative behavior, and become more likely to behave the same toward humans.

We have seen how technology like social media can be powerful in changing human beliefs and behavior. By focusing on building a bigger advertising business—entangling politics, trivia, and half-truths—you can bring about massive changes in society.

Systems specifically designed to form relationships with a human will have much more power. AI will influence how we think, and how we treat others.

This requires a new level of corporate responsibility. We need to deliberately and consciously build AI that will improve the human condition—not just pursue the immediate financial gain of gazillions of addicted users.

Working on open artificial-intelligence technology and brain-computer interfaces, or forming ethics committees, are just part of the solution. We need to consciously build systems that work for the benefit of humans and society. They cannot have addiction, clicks, and consumption as their primary goal. AI is growing up, and will be shaping the nature of humanity. AI needs a mother.

Liesl Yearsley is a serial entrepreneur now working on how to make artificial intelligence agents better at problem-solving and capable of forming more human-like relationships.

Hear more about artificial intelligence at EmTech MIT 2017.

Register now
Liesl Yearsley

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

Next in Top Stories

Your guide to what matters today

Want more award-winning journalism? Subscribe to Insider Premium.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly magazine delivery and unlimited 24/7 access to MIT Technology Review’s website

    The Download: our daily newsletter of what's important in technology and innovation

    Access to the magazine PDF archive—thousands of articles going back to 1899 at your fingertips

    Special discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look: exclusive early access to important stories, before they’re available to anyone else

    Insider Conversations: listen in on in-depth calls between our editors and today’s thought leaders

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.