Skip to Content

The Avatar Will See You Now

Medical centers are testing new, friendly ways to reduce the need for office visits by extending their reach into patients’ homes.

Most patients who enter the gym of the San Mateo Medical Center in California are there to work with physical therapists. But a few who had knee replacements are being coached by a digital avatar instead.

The avatar, Molly, interviews them in Spanish or English about the levels of pain they feel as a video guides them through exercises, while the 3-D cameras of a Kinect device measure their movements. Because it’s a pilot project, Paul Carlisle, the director of rehabilitation services, looks on. But the ultimate goal is for the routine to be done from a patient’s home.

“It would change our whole model,” says Carlisle, who is running the trial as the public hospital looks for creative ways to extend the reach of its overtaxed budget and staff. “We don’t want to replace therapists. But in some ways, it does replace the need to have them there all the time.”

Receiving remote medical care is becoming more common as technologies improve and health records get digitized., the California startup running the trial, is one of more than 500 companies using health-care tools from Nuance, a company that develops speech-recognition and virtual-assistant software. “Our goal is basically to capture the patient’s state of mind and body,” says Ivana Schnur, cofounder of and a clinical psychologist who has spent years developing virtual-reality tools in medicine and mental health.

Using’s platform, patients can communicate their condition to an emotionally reactive avatar through their phone, desktop, or TV. The avatar asks the patient simple questions, and if programmed by a doctor, it can answer questions too—such as what a diabetes patient with high blood-sugar readings should eat that day. The software also collects data from other medical devices that a patient uses, such as a glucose meter, and can capture gestures with a Kinect. The reports sent to the doctor include red-flag notifications to act on right away; charts, graphs, and analytics tracing the patient’s progress over time; and a transcript of the voice interaction.

“A physician’s time is always limited,” says Benjamin Kanter, chief medical information officer at Palomar Health in San Diego. “For a long time, we’ve had the challenge of just getting information into the system. Now the system is starting to actually help me.”

Schnur says one real advance is the avatar itself, which is important in helping both patients and doctors to trust the interactions. Molly, still a work in progress, can modulate her tone of voice and facial expressions. Schnur says that sometimes patients are more willing to share sensitive information with a nonjudgmental avatar than with a doctor.

Patients in San Mateo seem to like the interaction, Carlisle says, and he does too: “I’ve gotten used to the avatar. I look forward to seeing it when it comes online.”

The software, currently in beta, is also being tested at an addiction and detox clinic in California, doing patient intake and assessment in a crowded waiting room. Schnur hopes the system will eventually be used for even more complex tasks. The company, a product of the French telecommunication company Orange’s Silicon Valley incubator program, is working to include additional features, such as the ability to interpret and respond to a patient’s facial expressions.

Of course, doctors see some risks in such approaches, especially if the software makes an error or misinterprets an interaction. Kanter points out that although electronic systems often reduce errors, any errors that occur can propagate more quickly than those made only on paper.

Carlisle, who will enroll 50 to 60 patients by the time the study is done, is looking forward to getting more data. Over time, he hopes, not only will he improve the care of individual patients in their home environments, but what he learns from the data will improve therapy for everyone.

Keep Reading

Most Popular

The inside story of how ChatGPT was built from the people who made it

Exclusive conversations that take us behind the scenes of a cultural phenomenon.

ChatGPT is about to revolutionize the economy. We need to decide what that looks like.

New large language models will transform many jobs. Whether they will lead to widespread prosperity or not is up to us.

Sam Altman invested $180 million into a company trying to delay death

Can anti-aging breakthroughs add 10 healthy years to the human life span? The CEO of OpenAI is paying to find out.

GPT-4 is bigger and better than ChatGPT—but OpenAI won’t say why

We got a first look at the much-anticipated big new language model from OpenAI. But this time how it works is even more deeply under wraps.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.