Microsoft Wants You to Educate Its Virtual Assistant

Microsoft’s new virtual personal assistant Cortana will meet its public in the coming months as the company’s upgraded mobile operating system rolls out to existing and new devices. But according to Larry Heck, a researcher who contributed to the artificial intelligence underpinning Cortana, first impressions will quickly become outdated.

That’s because Microsoft has designed Cortana to get significantly smarter as people use it. Heck claims that Microsoft’s commitment to that approach will allow its virtual assistant, currently described as a beta product, to come closer than competitors like Siri or Google Now to delivering the experience of interacting with a real person. “We can update Cortana for all users on an ongoing basis,” he says. “Cortana should be able to improve really fast.”

Cortana learns about its users—for example, by asking for information such as a favorite sports team, or by deducing the whereabouts of your workplace from your phone’s location data. And Heck’s effort to use data flowing through Cortana to continually improve it will likely be crucial to its future.

Much like Apple’s Siri (see “Social Intelligence”), Cortana has been given a sassy personality to draw people into back-and-forth conversations. Making people feel they can talk back to Cortana raises the chances that Microsoft will learn what they really wanted when the app doesn’t get it right the first time, says Heck.

“It might make a mistake, but two or three turns of the conversation later we find what a person really meant,” he says. “That gets data coming back to us that we can feed back into the machine learning.”

Siri can also handle follow-up questions or commands in some situations, but Cortana does so with greater sophistication. For example, Siri, Google Now, and Cortana all return a list of suggestions when asked, “Find me cheap Japanese restaurants nearby.” But only Cortana responds when then asked, “How long will it take me to get to [restaurant name]?” or “Which ones are open now?”

Cortana can also keep track of the context of a conversation for longer: it understands what is meant if someone goes on to ask, “How long would it take to walk?”

Heck’s team will use tricks from the world of online search to work out when something has gone wrong. A person hitting the back button, or repeating a question in a rephrased form, could indicate that Cortana misunderstood.

Norman Winarsky, vice president of SRI Ventures at SRI International, says Heck’s approach is crucial to getting people to treat Microsoft’s app like a capable human assistant. “The quality of the data they gather and how they learn to use it is going to determine the success of Cortana,” says Winarsky, who cofounded the startup that invented Siri as a spinout from a Department of Defense project carried out by SRI (see “TR10: Intelligent Software Assistant”). “If that works, it will create overwhelming value, but they’re taking on a hard challenge.”

Microsoft’s decision to give Cortana a proto-personality and encourage conversation will help provide the feedback needed to make that strategy work, says Winarsky. But it is also likely to make people less tolerant of any mistakes. “There’s a very strong penalty for mistakes in any kind of dialogue,” he says. “If you’re offered something wrong in that kind of medium, especially by voice, you get very impatient very quickly.”

That may be one reason Google chose to make its own voice-operated mobile assistant, Google Now, personality-free and more like a “dumb” search engine (see “Google’s Answer to Siri Thinks Ahead”). Although hard data is lacking, few iPhone users seem to use Siri very much because they feel its performance falls short of Apple’s hype.

Winarsky says he thinks virtual-assistant technology is best confined to more specialized niches for now. Siri was originally designed only for travel and entertainment before Apple snapped it up and broadened its remit. “In a market vertical, we could understand the intent and language and vocabulary much better,” says Winarsky. “The broader you make it, the more brittle it is.” SRI’s more recent virtual-assistant spinouts—Desti for travel and TempoAI for calendar management—have also been focused on specific areas.

Heck argues that Microsoft’s technology is strong enough to support Cortana’s relatively wide scope of duties without compromising reliability. He also says that the way Cortana is connected to the infrastructure behind Microsoft’s search engine Bing—the kind of resource unavailable to Apple—raises the prospect of major leaps in what the app can understand. Heck is currently using some 300,000 computers to process Web data in Bing’s memory banks in an attempt to find ways to automatically extract knowledge and enable Cortana to respond to new kinds of queries.

“If we can connect Cortana on top of that, we can have just as deep a conversation about fly fishing as about finding restaurants,” says Heck, who hopes to extend Microsoft’s Satori knowledge database—a competitor to Google’s own Knowledge Graph (see “How a Database of the World’s Knowledge Shapes Google’s Future”). “I think Cortana will evolve to be this personal assistant you always go to first, like you go to Web search because of the breadth of things it knows about.”