wiki:FaceEngine

FaceEngine

AsapRealizer's default FaceEngine provides flexible support for morph based and/or MPEG4 facial animation.

XML setup

AsapRealizer's default FaceEngine is setup through the FaceEngineLoader xml description.

Typical setup:

<Loader id="faceembodiment">
..
</Loader>
<Loader id="faceengine" 
          loader="asap.faceengine.loader.FaceEngineLoader" requiredloaders="faceembodiment">
    <FaceBinding basedir="" resources="Humanoids/billie/facebinding/" filename="facebinding.xml"/>
</Loader>

The FaceEngine requires a FaceEmbodiment? and a facebinding.

The facebinding

The facebinding is an XML file describing the Binding between BML face behaviors and the motor units that execute them.

Binding a Morph motor plan unit

Example:

<FaceUnitSpec type="faceLexeme">
   <constraints>
     <constraint name="lexeme" value="smilemouth"/>
   </constraints>
   <parametermap>
     <parameter src="amount" dst="intensity"/>
   </parametermap>
   <parameterdefaults>
      <parameterdefault name="intensity" value="1"/>
      <parameterdefault name="targetname" value="KevinHead.mesh-Smile"/>
    </parameterdefaults>
  <FaceUnit type="Morph"/>
</FaceUnitSpec> 

Here, the obligatory

<parametermap>
  <parameter src="amount" dst="intensity"/>
</parametermap>

maps the BML faceLexeme amount parameter directly to the intensity (=blend value) of the bound morphs. The obligatory targetname specifies a comma separated lists of morph targets that are used to achieve this specific BML behavior.

Binding a AU motor plan unit

AU motor plan units have an intensity, au (=au nummer) and side (LEFT, RIGHT, BOTH, NONE(for AUs that don't have a side)) parameter that needs to be set/bound.

Binding all faceFacs behaviors to their corresponding AU motor unit:

<FaceUnitSpec type="faceFacs" namespace="http://www.bml-initiative.org/bml/coreextensions-1.0">
  <constraints/>
  <parametermap>
    <parameter src="amount" dst="intensity"/>
    <parameter src="au" dst="au"/>
    <parameter src="side" dst="side"/>
  </parametermap>
  <FaceUnit type="AU"/>
</FaceUnitSpec>

Binding the faceLexeme "RAISE_LEFT_BROW" to an AU motor unit:

<FaceUnitSpec type="faceLexeme">
  <constraints>
    <constraint name="lexeme" value="RAISE_LEFT_BROW"/>
  </constraints>
  <parametermap>
    <parameter src="amount" dst="intensity"/>
  </parametermap>
  <parameterdefaults>			
    <parameterdefault name="au" value="1"/>
    <parameterdefault name="side" value="LEFT"/>
  </parameterdefaults>
  <FaceUnit type="AU"/>
</FaceUnitSpec>

Binding a configuration of multiple AUs

The FACS motor unit specifies a configuration of AUs, specified by the filename of this configuration and an intensity. The intensity scales the values specified in the FACSConfiguration.

<FaceUnitSpec type="faceLexeme">
  <constraints>			
    <constraint name="lexeme" value="expectingbrows"/>
  </constraints>
  <parametermap>
    <parameter src="amount" dst="intensity"/>
  </parametermap>
  <parameterdefaults>
    <parameterdefault name="intensity" value="1"/>
  </parameterdefaults>
  <FaceUnit type="FACS" filename="Humanoids/shared/faceexpressions/expectingbrows.xml"/>
</FaceUnitSpec>

expectingbrows.xml:

<FACSConfiguration>
   <AU number="1" side="LEFT" value="0.38"/>
   <AU number="1" side="RIGHT" value="0.35"/>
</FACSConfiguration> 

Binding a Plutchik-based emotional expression

requires one to set/bound an intensity, angle and activation parameter (the latter two are coordinates on Plutchik's wheel).

<FaceUnitSpec type="faceLexeme">
  <constraints>			
    <constraint name="lexeme" value="surprise"/>
  </constraints>
  <parametermap>
    <parameter src="amount" dst="intensity"/>
  </parametermap>
  <parameterdefaults>
    <parameterdefault name="angle" value="90"/>
    <parameterdefault name="activation" value="3.6"/>
  </parameterdefaults>
  <FaceUnit type="Plutchik"/>
</FaceUnitSpec>  

Facial Lipsync Provider

The facial animation for lipsync is set up through a lipsync provider:

<Loader id="facelipsync" requiredloaders="faceengine" loader="asap.faceengine.loader.TimedFaceUnitLipSynchProviderLoader">
  <MorphVisemeBinding resources="Humanoids/armandia/facebinding/" filename="ikpvisemebinding.xml"/>
</Loader>

The MorphVisemeBinding binds viseme numbers to morph targets. The blend value of the morph targets is defined by the intensity attribute. The names of the targets are specified as a comma separated list in the target list.

Example: binding IKP visemes numbers (from HmiMaryTTS) to the morphs used in the billy model.

<VisemeToMorphMapping>
  <Mapping viseme="0" intensity="1" target="KevinHead.mesh-Rest,Teeth Lower.mesh-Rest"/>
  <Mapping viseme="1" intensity="0.25" target="KevinHead.mesh-MBP,Teeth Lower.mesh-MBP"/>     <!-- p, b -->
  <Mapping viseme="2" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- t,d,k,g -->
  <Mapping viseme="3" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- n, @n, l, @l -->
  <Mapping viseme="4" intensity="0.25" target="KevinHead.mesh-MBP,Teeth Lower.mesh-MBP"/>     <!-- m -->
  <Mapping viseme="5" intensity="0.25" target="KevinHead.mesh-FV,Teeth Lower.mesh-FV"/>	      <!-- f, v -->
  <Mapping viseme="6" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- s, z -->
  <Mapping viseme="7" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- S, Z, tS, dZ -->
  <Mapping viseme="8" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- h, r, x, N -->
  <Mapping viseme="9" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/> <!-- j, C -->
  <Mapping viseme="10" intensity="0.15" target="KevinHead.mesh-E,Teeth Lower.mesh-E"/>      <!-- i:, I, e:, E:, E -->
  <Mapping viseme="11" intensity="0.25" target="KevinHead.mesh-A,Teeth Lower.mesh-A"/>	<!-- a:, a -->
  <Mapping viseme="12" intensity="0.35" target="KevinHead.mesh-O,Teeth Lower.mesh-O"/>	<!-- o:, O: -->
  <Mapping viseme="13" intensity="0.35" target="KevinHead.mesh-U,Teeth Lower.mesh-U"/>	<!-- u:, U -->
  <Mapping viseme="14" intensity="0.25" target="KevinHead.mesh-E,Teeth Lower.mesh-E"/>	<!-- @, 6 -->
  <Mapping viseme="15" intensity="0.25" target="KevinHead.mesh-DGKNRSY,Teeth Lower.mesh-DGKNRSY"/>	<!-- y:, Y, 2:, 9 -->
</VisemeToMorphMapping>