July 2000

Lying With Pixels

Continued from page 3

By Ivan Amato

Deleting people or objects from live video, or inserting prerecorded people or objects into live scenes, is only the beginning of the deceptions becoming possible. Pretty much any piece of video that has ever been recorded is becoming clip art that producers can digitally sculpt into the story they want to tell, according to Eric Haseltine, senior vice president for R&D at Walt Disney Imagineering in Glendale, Calif. With additional video manipulation technologies, previously recorded actors can be made to say and do things they have never actually done or said. "You can have dead actors star again in entirely new movies," says Haseltine.

Contemporary shots featuring footage of dead performers have been around for several years. But the Hollywood illusion-craft that, for example, inserted John Wayne into a TV commercial required painstaking, frame-by-frame post-production work by skilled technicians. There's a big difference now, says Haseltine: "What used to take an hour [per video frame], now can be done in a sixtieth of a second." This dramatic speed-up means that manipulation can be done in real time, on the fly, as a camera records or broadcasts. Not only can John Wayne, Fred Astaire or Saddam Hussein be virtually inserted into pre-produced ads, they could be inserted into, say, a live broadcast of The Drew Carey Show.

The combination of real-time, virtual insertion with existing and emerging post-production techniques opens up a world of manipulative opportunity. Consider Video Rewrite technology, which its developers at the Interval Corp. and the University of California, Berkeley first demonstrated publicly three years ago. With just a few minutes of video of someone talking, their system captures and stores a set of video snapshots of the way that a person's mouth-area looks and moves when saying different sets of sounds. Drawing from the resulting library of "visemes" makes it possible to depict the person seeming to say anything the producers dream up-including utterances that the subject wouldn't be caught dead saying.

In one test application, computer scientist Christoph Bregler, now of Stanford University, and colleagues digitized two minutes of public-domain footage of President John F. Kennedy speaking during the Cuban missile crisis in 1962. Using the resulting viseme library, the researchers created "animations" of Kennedy's mouth saying things he never said, among them, "I never met Forrest Gump." With technology like this, near-future political activists conceivably will be able to orchestrate webcasts of their opponents saying things that might make Howard Stern sound like a mensch.

Haseltine believes video manipulation techniques will quickly be carried to their logical extreme: "I can predict with absolute certainty," he says, "that one person sitting at a computer will be able to write a script, design characters, do the lighting and wardrobe, do all of the acting and dialog, and post production, distribute it on a broadband network, do all of this on a laptop-and viewers won't know the difference."