We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not a subscriber? Subscribe now for unlimited access to online articles.

Intelligent Machines

Microsoft Wants You to Educate Its Virtual Assistant

Microsoft says the AI systems behind its new personal assistant will allow it to quickly become much smarter as more people use it.

An app that really approaches the capabilities of a human personal assistant would change the way people use computing devices of all kinds.

Microsoft’s new virtual personal assistant Cortana will meet its public in the coming months as the company’s upgraded mobile operating system rolls out to existing and new devices. But according to Larry Heck, a researcher who contributed to the artificial intelligence underpinning Cortana, first impressions will quickly become outdated.

cortana logo on device
Teach me: Microsoft says that its virtual assistant app, Cortana, will quickly get smarter thanks to analysis of data generated as people interact with it.

That’s because Microsoft has designed Cortana to get significantly smarter as people use it. Heck claims that Microsoft’s commitment to that approach will allow its virtual assistant, currently described as a beta product, to come closer than competitors like Siri or Google Now to delivering the experience of interacting with a real person. “We can update Cortana for all users on an ongoing basis,” he says. “Cortana should be able to improve really fast.”

Cortana learns about its users—for example, by asking for information such as a favorite sports team, or by deducing the whereabouts of your workplace from your phone’s location data. And Heck’s effort to use data flowing through Cortana to continually improve it will likely be crucial to its future.

Much like Apple’s Siri (see “Social Intelligence”), Cortana has been given a sassy personality to draw people into back-and-forth conversations. Making people feel they can talk back to Cortana raises the chances that Microsoft will learn what they really wanted when the app doesn’t get it right the first time, says Heck.

“It might make a mistake, but two or three turns of the conversation later we find what a person really meant,” he says. “That gets data coming back to us that we can feed back into the machine learning.”

Siri can also handle follow-up questions or commands in some situations, but Cortana does so with greater sophistication. For example, Siri, Google Now, and Cortana all return a list of suggestions when asked, “Find me cheap Japanese restaurants nearby.” But only Cortana responds when then asked, “How long will it take me to get to [restaurant name]?” or “Which ones are open now?”

Cortana can also keep track of the context of a conversation for longer: it understands what is meant if someone goes on to ask, “How long would it take to walk?”

Heck’s team will use tricks from the world of online search to work out when something has gone wrong. A person hitting the back button, or repeating a question in a rephrased form, could indicate that Cortana misunderstood.

Norman Winarsky, vice president of SRI Ventures at SRI International, says Heck’s approach is crucial to getting people to treat Microsoft’s app like a capable human assistant. “The quality of the data they gather and how they learn to use it is going to determine the success of Cortana,” says Winarsky, who cofounded the startup that invented Siri as a spinout from a Department of Defense project carried out by SRI (see “TR10: Intelligent Software Assistant”). “If that works, it will create overwhelming value, but they’re taking on a hard challenge.”

Microsoft’s decision to give Cortana a proto-personality and encourage conversation will help provide the feedback needed to make that strategy work, says Winarsky. But it is also likely to make people less tolerant of any mistakes. “There’s a very strong penalty for mistakes in any kind of dialogue,” he says. “If you’re offered something wrong in that kind of medium, especially by voice, you get very impatient very quickly.”

That may be one reason Google chose to make its own voice-operated mobile assistant, Google Now, personality-free and more like a “dumb” search engine (see “Google’s Answer to Siri Thinks Ahead”). Although hard data is lacking, few iPhone users seem to use Siri very much because they feel its performance falls short of Apple’s hype.

Winarsky says he thinks virtual-assistant technology is best confined to more specialized niches for now. Siri was originally designed only for travel and entertainment before Apple snapped it up and broadened its remit. “In a market vertical, we could understand the intent and language and vocabulary much better,” says Winarsky. “The broader you make it, the more brittle it is.” SRI’s more recent virtual-assistant spinouts—Desti for travel and TempoAI for calendar management—have also been focused on specific areas.

Heck argues that Microsoft’s technology is strong enough to support Cortana’s relatively wide scope of duties without compromising reliability. He also says that the way Cortana is connected to the infrastructure behind Microsoft’s search engine Bing—the kind of resource unavailable to Apple—raises the prospect of major leaps in what the app can understand. Heck is currently using some 300,000 computers to process Web data in Bing’s memory banks in an attempt to find ways to automatically extract knowledge and enable Cortana to respond to new kinds of queries.

“If we can connect Cortana on top of that, we can have just as deep a conversation about fly fishing as about finding restaurants,” says Heck, who hopes to extend Microsoft’s Satori knowledge database—a competitor to Google’s own Knowledge Graph (see “How a Database of the World’s Knowledge Shapes Google’s Future”). “I think Cortana will evolve to be this personal assistant you always go to first, like you go to Web search because of the breadth of things it knows about.”

Learn from the humans leading the way in intelligent machines at EmTech Next. Register Today!
June 11-12, 2019
Cambridge, MA

Register now
More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe to Print Subscription.
  • Print Subscription {! insider.prices.print_only !}*

    {! insider.display.menuOptionsLabel !}

    Six print issues per year plus The Download delivered to your email in-box each weekday.

    See details+

    12-month subscription

    Print magazine (6 bi-monthly issues)

    The Download: newsletter delivery each weekday to your inbox

You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.