We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

  • Saiman Chow
  • Intelligent Machines

    An AI Ally to Combat Bullying in Virtual Worlds

    Startup Spirit AI says it has a solution to the growing problem of harassment in online games: a sensitive bot.

    In any fictionalized universe, the distinction between playful antagonism and earnest harassment can be difficult to discern. Name-calling between friends playing a video game together is often a form of camaraderie. Between strangers, however, similar words assume a different, more troublesome quality. Being able to distinguish between the two is crucial for any video-game maker that wants to foster a welcoming community.

    Spirit AI hopes to help developers support players and discourage bullying behavior with an abuse detection and intervention system called Ally. The software monitors interactions between players—what people are saying to each other and how they are behaving—through the available actions within a game or social platform. It’s able to detect verbal harassment and also nonverbal provocation—for example, one player stalking another’s avatar or abusing reporting tools.

    “We’re looking at interaction patterns, combined with natural-language classifiers, rather than relying on a list of individual keywords,” explains Ruxandra Dariescu, one of Ally’s developers. “Harassment is a nuanced problem.”

    When Ally identifies potentially abusive behavior, it checks to see if the potential abuser and the other player have had previous interactions. Where Ally differs from existing moderation software is that rather than simply send an alert to the game’s developers, it is able to send a computer-controlled virtual character to check in with the player—one that, through Spirit AI’s natural-language tools, is able to converse in the game’s tone and style (see “A Video-Game Algorithm to Solve Online Abuse”).

    Ally’s bot might say: “I noticed you and Player 42 seem to be strangers. They may have said something inappropriate to you, so I just wanted to check if you’re okay?” The player is then able to respond to the bot, Dariescu says. If the interaction is dismissed as friendly banter, the system registers this and updates its rules so it won’t intervene over similar behavior in the future. If the user feels abused, the system intervenes with the culprit.

    Ally is compatible with all online games, and Spirit AI is working on adding capabilities to police behavior in virtual-reality games (see “The War on the Disturbingly Real Trolls in Virtual Reality”). The tool can be configured in a variety of ways to interact with players and take different kinds of action. Spirit AI sees Ally as a way to augment a game’s community management team, which can monitor activity in a dashboard.

    Sign up for The Download
    Your daily dose of what's up in emerging technology

    By signing up you agree to receive email newsletters and notifications from MIT Technology Review. You can change your preferences at any time. View our Privacy Policy for more detail.

    The company, which was officially formed in 2016 and currently has a staff of 15, won’t disclose which game developers it is currently working with, but the software has attracted attention in the gaming world. “On the simplest level, AI-supported community management tools can extend the team’s reach,” says Imre Jele, a veteran designer of vast online multiplayer games such as RuneScape. “But what I’m most excited about is AI systems that go beyond identifying and punishing rule-breaking behavior. Ultimately the goal of these AI team members should be to get out of policing and into shaping positive interactions and building healthier communities.”

    Spirit certainly intends for the software to offer more than a mere digitized police force that can be summoned into online games. “We’ve identified the potential for this technology to not only prevent bad behavior but also encourage good, prosocial behavior,” says Dariescu. The software could also use ambient eavesdropping to identify broader issues that players might have with a game’s design, providing insights to the developer as to how the overall player experience might be improved.

    There are, however, limitations when it comes to AI interventions in cases of harassment, according to Jele. Many interactions between community management and player support teams require deep empathy and sympathy, he says: “The players supported the developers with their money and deserve the best possible care, which only humans can provide at the moment.”

    Tech Obsessive?
    Become an Insider to get the story behind the story — and before anyone else.

    Subscribe today
    More from Intelligent Machines

    Artificial intelligence and robots are transforming how we work and live.

    Want more award-winning journalism? Subscribe to Insider Plus.
    • Insider Plus {! insider.prices.plus !}*

      {! insider.display.menuOptionsLabel !}

      Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

      See details+

      Print + Digital Magazine (6 bi-monthly issues)

      Unlimited online access including all articles, multimedia, and more

      The Download newsletter with top tech stories delivered daily to your inbox

      Technology Review PDF magazine archive, including articles, images, and covers dating back to 1899

      10% Discount to MIT Technology Review events and MIT Press

      Ad-free website experience

    You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.