It's Not What You Say, But How You Say It.(Technology Information)

An automated toolkit puts words in the mouths of virtual characters

For many people, public speaking is a nightmare. Among those who are crippled by such exposure are animated characters. Speech can bring even the best-looking, most realistic-moving digital persona to its knees. This is because human speech involves not only verbal expression, but also a range of intonations, spontaneous hand gestures, and facial movements that are tied not only to the meaning of the words, but also to the manner in which they are spoken.

While procedural-animation techniques have made tremendous strides with respect to re-creating human motion, including facial and hand behavior, "those techniques have not done well when the motion has needed to be synchronized to speech," says Justine Cassell, associate professor at MIT's Media Laboratory and director of the Lab's Gesture and Narrative Language Research Group. "And although lip sync has gotten much better, nobody has worked on synchronizing parts of the body besides the lips with speech."

Procedurally generated animated characters may soon get a chance to be heard, however, thanks to a new technology developed by Cassell and Media Lab colleagues Hannes Vihjalmsson and Timothy Bickmore. Called BEAT for Behavior Expression Animation Toolkit, the procedural system promises an automated method for giving virtual actors a voice by applying rules of speech and expression derived from extensive research into human conversational behavior.

With BEAT, users input typed text to be spoken by an animated human figure. What they get in return are appropriate and synchronized nonverbal behaviors and synthesized speech in a form (XML trees) that can be sent to a number of different animation systems. "BEAT uses linguistic and contextual information contained in the text to control the movements of the hands, arms, and face, as well as the intonation of the voice," says Cassell. "The mapping from text to facial, intonational, and body gestures is contained in a set of rules."

Rules, Rules, and More Rules

Written in JAVA, BEAT consists of various modules, each of which takes tagged text as input and produces tagged text as output. The tags identify the text as being of a certain type so the system knows which rules should be applied to it. The BEAT knowledge base contains information that can be inferred directly from the text, such as the type of object being discussed and the related action.