May 2000

Speech and Vision

For thousands of years, people have interacted through speech and gesture. Truly easy-to-use machines will do the same.

By Michael Dertouzos

After 40 years of humans serving computers, people are finally beginning to wake up and demand that the relationship be reversed: They want machines to become simpler, address human needs and help increase human productivity. That's as it should be. With computer technology maturing, it's high time, as we have repeatedly said in this column, for makers and users of computers to change their focus from machines to people.

But how do you make a machine simpler? With fewer controls? Not really. As you know from watches and other devices that have one mode-changing button and another to select functions within a mode, you can easily get lost in a confusion of modes and functions. Would machines be simpler if they had fewer capabilities? No! Imagine a car that can only do two things-go and turn right. This car can go anywhere, but you wouldn't prefer it to your own car, which has more capabilities. What is it that we really want when we ask that our computers be easier to use?

True ease of use, as opposed to the perfunctory use of colors and floating animals to create the illusion of "user friendliness," involves means of interaction that are natural to people and therefore require no new learning. Speech and vision are the two principal means we have used to interact with other people and the world around us for thousands of years. That should be enough of a clue to steer our attention to these two modes of interaction. And since vision occupies two-thirds of the human cerebral cortex, we may be tempted to declare it the queen of human-machine communication. That would be an easy-but deceptive-conclusion.

Vision and speech do not serve the same natural roles in communication. Being Greek, I can still hold a "conversation" in Athens, through a car window, using only gestures and grimaces-one clockwise rotation of the wrist means "how are you" while an oscillating motion of the right hand around the index finger with palm extended and sides of mouth drawn downward, means "so-so." A sign language like ASL works even better. But when speech is available, it invariably takes over as the preferred mode of human communication.

That's because among people, speech is used symmetrically for transmitting and receiving concepts. Visual human communication, on the other hand, is highly asymmetric-we can perceive a huge amount of information with our eyes, but can't deliver equally rich visual information with gestures (visual communication would be more symmetric if we all sported display monitors on our chests). The power of vision in humans lies primarily in the one-way digestion of an incredible amount of information-nature's, or God's, way of ensuring survival in a world of friends and enemies, edible and man-eating animals, useful and useless objects, lush valleys and dangerous ravines, where maximum information was essential.