In a small, dark, room off a long hallway within a sprawling complex of buildings in Silicon Valley, an array of massive flat-panel displays and video cameras track Grit Denker’s every move. Denker, a senior computer scientist at the nonprofit R&D institute SRI, is showing off Bright, an intelligent assistant that could someday know what information you need before you even ask.
Initially, Bright is meant to cut down on the cognitive overload faced by workers in high-stress, data-intensive jobs like emergency response and network security. Bright may, for instance, aid network administrators in trying to stop the spread of a fast-moving virus by quickly providing crucial infection information, or help 911 operators send the right kind of assistance to the scene of an accident. But like many other technologies developed at SRI, such as the digital personal assistant Siri (now owned by Apple), Bright could eventually trickle down to laptops and smartphones. It might take the form of software that automatically brings up listings for your favorite shows when it thinks you’re about to sit down and watch TV, or searches the Web for information relevant to your latest research project without requiring you to lift a finger.
Already some assistant software, such as Google Now for Android smartphones, tries to predict what information a user may need and serve it up automatically. It does this by, for example, recognizing that the user is waiting at a bus stop and delivering bus timetables. The aim of Bright is to develop something even more sophisticated and capable in an office setting. But the big challenge for Bright and similar projects is: how do you learn from a relatively small amount of information?
Originally created by Stanford University as a research institution in 1946 (it’s been operating independently since 1970), SRI International, based in Menlo Park, California, has developed key technologies including the computer mouse, the LCD, and even the first twinklings of the Internet, called ARPAnet. In recent years, it has had success in the artificial-intelligence field with Siri, which was spun out of a project SRI did for the Department of Defense’s Defense Advanced Research Projects Agency, or DARPA, called CALO (that’s “cognitive agent that learns and organizes”).
Denker describes Bright as a “cognitive desktop” and “a desktop that really understands what you’re doing, and not just for you, but also in a collaborative setting for people.” In its current setup, three cameras stare out at her; a monitor shows where she’s looking and displays a real-time log of every action she takes, as well as a familiar-looking computer desktop of files and folders. When she uses the monitor in front of her to open an e-mail from Wells Fargo bank requesting a meeting, for example, Bright records all her actions on a monitor off to the left, noting that she opened the message, that she spent time looking at it (rather than just gazing elsewhere on the screen), and that she closed it.
As Denker demonstrates Bright’s nascent capabilities, it’s not hard to imagine the technology easing everything from scheduling tasks to searching the Web. She explains that her team is trying to adapt existing computer science techniques that try to increase efficiency by anticipating what information will be needed next and testing different actions in advance to speed up response time. Bright, she says, uses the same ideas to anticipate what the user will want to do, so it requires additional equipment to monitor the user. A touch-sensitive display can track finger touches, and hand motions—such as waving—are tracked too.
While it is being developed for cybersecurity and emergency response, Bright could be tailored for other types of users. In schools, for example, Bright might be able to determine that a student is struggling and adjust itself to better meet his or her needs.
There’s a long way to go, however. The system is currently focused on “cognitive indexing”—the mechanism that ties various clues together and then tries to predict what is important. The team behind Bright also needs to build its abilities to predict interests and automate tasks. And before it can be rolled out anywhere, Bright needs to learn how to study what you’re using your computer for.
Getting to know a user is difficult, says Bill Mark, vice president of information and computing sciences at SRI and one of the principal investigators behind CALO. Mark calls this the “small-data problem”; while “big data” efforts focus on gleaning insights from mountains of information, systems like Bright are looking for patterns in much smaller quantities, and this can be very tricky. The limited data set, combined with users’ tendency to change behavior, is very unfriendly to pattern-finding algorithms, he says: “We’re not putting in that much data. These machine-learning algorithms like to generalize over very large amounts of data.”
There are plenty of other challenges. Krzysztof Gajos, an assistant professor of computer science at Harvard who also spent a year working on CALO, notes that one of the difficulties in building intelligent interactive systems is figuring out how to distinguish mandatory tasks like office work from voluntary tasks like playing games. For office-related tasks, he says, it’s hard to design automation in a way that leaves the user feeling in control and seems worth using even though it will occasionally screw up.
“If you look back to systems like the Microsoft Clippy, you can see an example of a system that failed at that,” Gajos says. “The few times it failed were just so aggravating that it overshadowed any benefits the system might have provided for many users.”
Hear more from Google at EmTech Digital.