If Jian Wang had his way, everything would be digital.
“I hate printers-they turn digital things into analog,” he jokes, wading through a sea of cubicles at Microsoft Research Asia in Beijing, China. Fortyish and lanky, the computer scientist specializes in inventing new computer interfaces to bridge the gap between analog and digital. His own interface, though, is a wide smile, which complements his denim shirt and easygoing manner.
Stopping at a desk, Wang picks up a rectangular, silvery pen about the size of a magic marker and scribbles some corrections on a paper document. But this is no ordinary pen. A few seconds later, his comments appear on a nearby computer screen-superimposed on the electronic version of the document in the exact spot where he wrote on the hard copy. Wang’s pen captures handwriting and lets users make changes to digital files-on paper.
This “universal pen,” as Wang calls it, could transform the way people interact with computers. Unlike gizmos that write on computer displays or special pads of paper, Wang’s invention uses regular ink, works with regular paper, and lets users combine handwritten text and diagrams with digital content from reports, magazines, and Web pages. An executive on a plane trip, for instance, could mark up a paper copy of a report and later transfer the changes to the file on his or her computer automatically.
Wang’s digital pen also reflects an ongoing transformation in the process of invention at some large corporate labs, a hybridization of the lone inventor and traditional corporate R&D. Wang is the pen’s lead inventor, and it is his insight, daring, and creativity that have largely driven the effort to develop it. But at the same time, he could not have made such rapid progress without Microsoft’s collective expertise in pattern recognition algorithms, computer vision, handwriting technologies, and text-editing software. “Personally, I’m really excited about it,” says Rick Rashid, senior vice president of Microsoft Research, whose main facility is in Redmond, WA. “It’s an example of a new kind of product incubation that we do,…one that brings together people with many different skills to solve a unique problem.”
Himalayan stacks of papers fill Wang’s office, despite his proclaimed distaste for printouts. Amidst these reminders of the analog world, the inventor describes the beginnings of the pen project. As a professor of engineering psychology at Zhejiang University in Hangzhou, China, Wang had made his name in human-computer interfaces and virtual-reality systems. Then, in 1998, at the height of his academic career, he received a cryptic e-mail from Kai-Fu Lee, a renowned researcher who was just starting up Microsoft’s Beijing lab. Lee suggested they meet but was too busy to explain why. “I don’t know how he got my name,” laughs Wang, “and he didn’t explain who he was.”
Feeling adventurous, Wang went anyway. He quickly discovered that he and Lee shared the ambition to create a user interface for computers that was based on handwriting. The pen project was inspired in part by Wang’s desire to enable mobile computers to handle handwritten Asian languages. But from his academic research on 3-D interfaces, he had learned that if technology isn’t designed to be practical and appealing to a variety of users, it won’t be widely adopted. “We wanted to make a breakthrough, not improvements,” he says, adding that in Microsoft he had found a perfect partner. “I realized this is the place to be if you want your invention to be used by millions of people instead of just a couple.” With that motivation, Wang left Zhejiang University and joined the Beijing lab in the fall of 1999.
Immediately, Wang, Lee, and a core group of researchers began daily brainstorming sessions to iron out the project’s goals-sessions that eventually included their top boss. “When we talked with Bill [Gates] about this technology,” says Wang, “we realized we are inventing a new kind of document, not just a pen.” The key idea, which fits with Microsoft’s business of making office computers more useful, was that a document could be kept digital even when printed-with the right kind of pen interface and software.
It took the better part of four years to make it work. Not only did Microsoft stick with Wang, but it also allowed him to build a team of about 20 researchers, giving him access to the skills in programming and hardware he needed to design the pen. After a few false starts-including a cumbersome version that inferred what the user was writing from the pen’s movements-Wang’s team took advantage of advances in computer vision algorithms and based the pen’s sensing on a simple digital camera.
The researchers’ first challenge was finding a way to determine the pen’s position on the page. Their solution involves special software that puts a barely visible background pattern, like a watermark, on standard copy paper as a document is printed (see “Putting Pen to…Computer,” below). That enables the computer to figure out not only exactly where the pen is relative to the document but also which document is being modified, because each page has a unique code. A pressure sensor in the pen triggers a tiny embedded camera, which snaps pictures of the user’s writing. The images are stored in the pen on a memory chip like those found in digital cameras; when brought within a few meters of a PC or laptop that has the proper software installed, the pen transmits the images wirelessly using a Bluetooth connection.
Putting pen to…computer
Interpreting those images and incorporating them into digital files turned out to be a tougher problem. The trick, Wang explains, is getting the computer to recognize different types of writing and drawings-to know what’s a box, what’s a sentence, and what’s a doodle-just from a series of photos. First, computer vision algorithms classify sequences of marks as words, diagrams, or shapes, all of which can be manipulated. Then character recognition software-the subject of years of intensive research at Microsoft and elsewhere-makes sense of the handwritten text. On the computer screen, the user’s marks show up as handwriting embedded in the document. Software tools can then convert the writing into typed text and rendered graphics; with these tools, the user can manipulate, say, boxes and text from a hand-drawn flow chart.
In addition to enabling users to import and manipulate handwritten text, tables, and charts, says Wang, the pen will allow multiple collaborators to make comments on separate printouts of a document; the computer could then integrate them all into the same file. The end result, says Wang, will be an interface like a portable scanner-but one smart enough to “understand” the images it captures and to fulfill its inventor’s dream of turning mounds of “analog” paper back into digital files.