Skip to Content

An AI Ally to Combat Bullying in Virtual Worlds

Startup Spirit AI says it has a solution to the growing problem of harassment in online games: a sensitive bot.
Saiman Chow

In any fictionalized universe, the distinction between playful antagonism and earnest harassment can be difficult to discern. Name-calling between friends playing a video game together is often a form of camaraderie. Between strangers, however, similar words assume a different, more troublesome quality. Being able to distinguish between the two is crucial for any video-game maker that wants to foster a welcoming community.

Spirit AI hopes to help developers support players and discourage bullying behavior with an abuse detection and intervention system called Ally. The software monitors interactions between players—what people are saying to each other and how they are behaving—through the available actions within a game or social platform. It’s able to detect verbal harassment and also nonverbal provocation—for example, one player stalking another’s avatar or abusing reporting tools.

“We’re looking at interaction patterns, combined with natural-language classifiers, rather than relying on a list of individual keywords,” explains Ruxandra Dariescu, one of Ally’s developers. “Harassment is a nuanced problem.”

When Ally identifies potentially abusive behavior, it checks to see if the potential abuser and the other player have had previous interactions. Where Ally differs from existing moderation software is that rather than simply send an alert to the game’s developers, it is able to send a computer-controlled virtual character to check in with the player—one that, through Spirit AI’s natural-language tools, is able to converse in the game’s tone and style (see “A Video-Game Algorithm to Solve Online Abuse”).

Ally’s bot might say: “I noticed you and Player 42 seem to be strangers. They may have said something inappropriate to you, so I just wanted to check if you’re okay?” The player is then able to respond to the bot, Dariescu says. If the interaction is dismissed as friendly banter, the system registers this and updates its rules so it won’t intervene over similar behavior in the future. If the user feels abused, the system intervenes with the culprit.

Ally is compatible with all online games, and Spirit AI is working on adding capabilities to police behavior in virtual-reality games (see “The War on the Disturbingly Real Trolls in Virtual Reality”). The tool can be configured in a variety of ways to interact with players and take different kinds of action. Spirit AI sees Ally as a way to augment a game’s community management team, which can monitor activity in a dashboard.

The company, which was officially formed in 2016 and currently has a staff of 15, won’t disclose which game developers it is currently working with, but the software has attracted attention in the gaming world. “On the simplest level, AI-supported community management tools can extend the team’s reach,” says Imre Jele, a veteran designer of vast online multiplayer games such as RuneScape. “But what I’m most excited about is AI systems that go beyond identifying and punishing rule-breaking behavior. Ultimately the goal of these AI team members should be to get out of policing and into shaping positive interactions and building healthier communities.”

Spirit certainly intends for the software to offer more than a mere digitized police force that can be summoned into online games. “We’ve identified the potential for this technology to not only prevent bad behavior but also encourage good, prosocial behavior,” says Dariescu. The software could also use ambient eavesdropping to identify broader issues that players might have with a game’s design, providing insights to the developer as to how the overall player experience might be improved.

There are, however, limitations when it comes to AI interventions in cases of harassment, according to Jele. Many interactions between community management and player support teams require deep empathy and sympathy, he says: “The players supported the developers with their money and deserve the best possible care, which only humans can provide at the moment.”

Keep Reading

Most Popular

Scientists are finding signals of long covid in blood. They could lead to new treatments.

Faults in a certain part of the immune system might be at the root of some long covid cases, new research suggests.

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

OpenAI teases an amazing new generative video model called Sora

The firm is sharing Sora with a small group of safety testers but the rest of us will have to wait to learn more.

Google’s Gemini is now in everything. Here’s how you can try it out.

Gmail, Docs, and more will now come with Gemini baked in. But Europeans will have to wait before they can download the app.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.