Skip to Content
MIT News: 77 Mass Ave

Race-detecting AI

Researchers trained an algorithm to identify people’s self-reported race from medical images, but they are mystified as to how it works.

August 24, 2022
MR images of human brain
iStock

Doctors can’t tell a person’s race from medical images such as x-rays and CT scans. But a team including MIT researchers was able to train a deep-learning model to identify patients as white, Black, or Asian (according to their own description) just by analyzing such images—and they still can’t figure out how the computer does it. 

After looking at variables including differences in anatomy, bone density, and image resolution, the research team “could not come anywhere close to identifying a good proxy for this task,” says paper coauthor Marzyeh Ghassemi, PhD ’17, an assistant professor in EECS and the Institute for Medical Engineering and Science (IMES). 

That is concerning, the researchers say, because doctors use algorithms for help with decisions such as whether patients are candidates for chemotherapy or an intensive care unit. Now these findings raise the possibility that the algorithms are “looking at your race, ethnicity, sex, whether you’re incarcerated or not—even if all of that information is hidden,” says coauthor Leo Anthony Celi, SM ’09, a principal research scientist at IMES and an associate professor at Harvard Medical School.

Celi thinks clinicians and computer scientists should turn to social scientists for insight. “We need another group of experts to weigh in and to provide input and feedback on how we design, develop, deploy, and evaluate these algorithms,” he says. “We need to also ask the data scientists, before any exploration of the data: Are there disparities? Which patient groups are marginalized? What are the drivers of those disparities?”

Algorithms often have access to information that humans do not, and this means experts must work to understand the unintended consequences. Otherwise there is no way to prevent the algorithms from perpetuating the existing biases in medical care.

Keep Reading

Most Popular

10 Breakthrough Technologies 2024

Every year, we look for promising technologies poised to have a real impact on the world. Here are the advances that we think matter most right now.

Scientists are finding signals of long covid in blood. They could lead to new treatments.

Faults in a certain part of the immune system might be at the root of some long covid cases, new research suggests.

AI for everything: 10 Breakthrough Technologies 2024

Generative AI tools like ChatGPT reached mass adoption in record time, and reset the course of an entire industry.

OpenAI teases an amazing new generative video model called Sora

The firm is sharing Sora with a small group of safety testers but the rest of us will have to wait to learn more.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.