Speaking with hands: Creating animated conversational characters from recordings of human performance

Matthew Stone, Doug DeCarlo, Insuk Oh, Christian Rodriguez, Adrian Stere, Alyssa Lees, Chris Bregler

Research output: Contribution to conferencePaperpeer-review

84 Scopus citations


We describe a method for using a database of recorded speech and captured motion to create an animated conversational character. People's utterances are composed of short, clearly-delimited phrases; in each phrase, gesture and speech go together meaningfully and synchronize at a common point of maximum emphasis. We develop tools for collecting and managing performance data that exploit this structure. The tools help create scripts for performers, help annotate and segment performance data, and structure specific messages for characters to use within application contexts. Our animations then reproduce this structure. They recombine motion samples with new speech samples to recreate coherent phrases, and blend segments of speech and motion together phraseby-phrase into extended utterances. By framing problems for utterance generation and synthesis so that they can draw closely on a talented performance, our techniques support the rapid construction of animated characters with rich and appropriate expression.

Original languageEnglish (US)
Number of pages8
StatePublished - 2004
EventACM SIGGRAPH 2004, SIGGRAPH 2004 - Los Angeles, CA, United States
Duration: Aug 8 2004Aug 12 2004


Country/TerritoryUnited States
CityLos Angeles, CA

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction


  • Animation
  • Conversational agents
  • Language generation
  • Motion capture
  • Motion synthesis
  • Speech synthesis


Dive into the research topics of 'Speaking with hands: Creating animated conversational characters from recordings of human performance'. Together they form a unique fingerprint.

Cite this