We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not a subscriber? Subscribe now for unlimited access to online articles.

  • Saiman Chow
  • Intelligent Machines

    An AI Ally to Combat Bullying in Virtual Worlds

    Startup Spirit AI says it has a solution to the growing problem of harassment in online games: a sensitive bot.

    In any fictionalized universe, the distinction between playful antagonism and earnest harassment can be difficult to discern. Name-calling between friends playing a video game together is often a form of camaraderie. Between strangers, however, similar words assume a different, more troublesome quality. Being able to distinguish between the two is crucial for any video-game maker that wants to foster a welcoming community.

    Spirit AI hopes to help developers support players and discourage bullying behavior with an abuse detection and intervention system called Ally. The software monitors interactions between players—what people are saying to each other and how they are behaving—through the available actions within a game or social platform. It’s able to detect verbal harassment and also nonverbal provocation—for example, one player stalking another’s avatar or abusing reporting tools.

    “We’re looking at interaction patterns, combined with natural-language classifiers, rather than relying on a list of individual keywords,” explains Ruxandra Dariescu, one of Ally’s developers. “Harassment is a nuanced problem.”

    When Ally identifies potentially abusive behavior, it checks to see if the potential abuser and the other player have had previous interactions. Where Ally differs from existing moderation software is that rather than simply send an alert to the game’s developers, it is able to send a computer-controlled virtual character to check in with the player—one that, through Spirit AI’s natural-language tools, is able to converse in the game’s tone and style (see “A Video-Game Algorithm to Solve Online Abuse”).

    Ally’s bot might say: “I noticed you and Player 42 seem to be strangers. They may have said something inappropriate to you, so I just wanted to check if you’re okay?” The player is then able to respond to the bot, Dariescu says. If the interaction is dismissed as friendly banter, the system registers this and updates its rules so it won’t intervene over similar behavior in the future. If the user feels abused, the system intervenes with the culprit.

    Ally is compatible with all online games, and Spirit AI is working on adding capabilities to police behavior in virtual-reality games (see “The War on the Disturbingly Real Trolls in Virtual Reality”). The tool can be configured in a variety of ways to interact with players and take different kinds of action. Spirit AI sees Ally as a way to augment a game’s community management team, which can monitor activity in a dashboard.

    Sign up for The Download
    Your daily dose of what's up in emerging technology

    The company, which was officially formed in 2016 and currently has a staff of 15, won’t disclose which game developers it is currently working with, but the software has attracted attention in the gaming world. “On the simplest level, AI-supported community management tools can extend the team’s reach,” says Imre Jele, a veteran designer of vast online multiplayer games such as RuneScape. “But what I’m most excited about is AI systems that go beyond identifying and punishing rule-breaking behavior. Ultimately the goal of these AI team members should be to get out of policing and into shaping positive interactions and building healthier communities.”

    Spirit certainly intends for the software to offer more than a mere digitized police force that can be summoned into online games. “We’ve identified the potential for this technology to not only prevent bad behavior but also encourage good, prosocial behavior,” says Dariescu. The software could also use ambient eavesdropping to identify broader issues that players might have with a game’s design, providing insights to the developer as to how the overall player experience might be improved.

    There are, however, limitations when it comes to AI interventions in cases of harassment, according to Jele. Many interactions between community management and player support teams require deep empathy and sympathy, he says: “The players supported the developers with their money and deserve the best possible care, which only humans can provide at the moment.”

    Learn from the humans leading the way in intelligent machines at EmTech Next. Register Today!
    June 11-12, 2019
    Cambridge, MA

    Register now
    More from Intelligent Machines

    Artificial intelligence and robots are transforming how we work and live.

    Want more award-winning journalism? Subscribe to Print + All Access Digital.
    • Print + All Access Digital {! insider.prices.print_digital !}*

      {! insider.display.menuOptionsLabel !}

      The best of MIT Technology Review in print and online, plus unlimited access to our online archive, an ad-free web experience, discounts to MIT Technology Review events, and The Download delivered to your email in-box each weekday.

      See details+

      12-month subscription

      Unlimited access to all our daily online news and feature stories

      6 bi-monthly issues of print + digital magazine

      10% discount to MIT Technology Review events

      Access to entire PDF magazine archive dating back to 1899

      Ad-free website experience

      The Download: newsletter delivery each weekday to your inbox

      The MIT Technology Review App

    You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.