Microsoft is creating an oracle for catching biased AI algorithms

Microsoft is building a tool to automatically identify bias in a range of different AI algorithms. It is the boldest effort yet to automate the detection of unfairness that may creep into machine learning—and it could help businesses make use of AI without inadvertently discriminating against certain people.
Big tech companies are racing to sell off-the-shelf machine-learning technology that can be accessed via the cloud. As more customers make use of these algorithms to automate important judgements and decisions, the issue of bias will become crucial. And since bias can easily creep into machine-learning models, ways to automate the detection of unfairness could become a valuable part of the AI toolkit.
“Things like transparency, intelligibility, and explanation are new enough to the field that few of us have sufficient experience to know everything we should look for and all the ways that bias might lurk in our models,” says Rich Caruna, a senior researcher at Microsoft who is working on the bias-detection dashboard.
Algorithmic bias is a growing concern for many researchers and technology experts (see “Inspecting algorithms for bias”). As algorithms are used to automate important decisions, there is a risk that bias could become automated, deployed at scale, and more difficult for the victims to spot.
Caruna says Microsoft’s bias-catching product will help AI researchers catch more instances of unfairness, although not all. “Of course, we can’t expect perfection—there’s always going to be some bias undetected or that can’t be eliminated—the goal is to do as well as we can,” he says.
“The most important thing companies can do right now is educate their workforce so that they’re aware of the myriad ways in which bias can arise and manifest itself and create tools to make models easier to understand and bias easier to detect,” Caruna adds.
Facebook announced its own tool for detecting bias at its annual developer conference on May 2. Its tool, called Fairness Flow, automatically warns if an algorithm is making an unfair judgement about someone based on his or her race, gender, or age. Facebook says it needed Fairness Flow because more and more people at the company are using AI to make important decisions.
Bin Yu, a professor at UC Berkeley, says the tools from Facebook and Microsoft seem like a step in the right direction, but may not be enough. She suggests that big companies should have outside experts audit their algorithms in order to prove they are not biased. “Someone else has to investigate Facebook's algorithms—they can't be a secret to everyone,” Yu says.
Deep Dive
Artificial intelligence
This new data poisoning tool lets artists fight back against generative AI
The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models.
Rogue superintelligence and merging with machines: Inside the mind of OpenAI’s chief scientist
An exclusive conversation with Ilya Sutskever on his fears for the future of AI and why they’ve made him change the focus of his life’s work.
Driving companywide efficiencies with AI
Advanced AI and ML capabilities revolutionize how administrative and operations tasks are done.
Unpacking the hype around OpenAI’s rumored new Q* model
If OpenAI's new model can solve grade-school math, it could pave the way for more powerful systems.
Stay connected
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.