wiki:UsingAsapRealizerWithIpaaca

Using AsapRealizer with Ipaaca

AsapRealizer can be connected to Billie's rendering environment in Ogre through Ipaaca, can execute Ipaaca BML messages and can provide BML feedback using Ipaaca messages.

Preliminaries

  1. Prepare your system as described in RecommendedSystemSetup
  2. Setup SoaScript
  3. Install AsapRealizer as described in the QuickStart

Connecting Ipaaca to the Ogre rendering environment

  1. Clone billie_asap and soaresource
    soa clone-all billie_asap soaresource
    
  2. Build billie_asap; in the billie_asap dir:
    soa build-all billie_asap
    
  3. Start spread (you can use trailing & to run in the background):
    spread -n localhost &
    
  4. Run billie from the billie_asap/resource dir, it internally unpacks billie from dist
    ./go.billie
    
  5. Run the AsapRealizerDemoExternalRenderer demo in AsapDemo/AsapRealizerDemoExternalRenderer:
    ant main #select asap.realizerdemo.AsapRealizerDemoExternalRenderer
    ant run
    

The externalrenderdemo makes use of the asapbillieipaaca.xml VirtualHumanLoader xml. This hooks the AnimationEngine and the FaceEngine up to a renderbody provided by Ipaaca:

<AsapVirtualHuman>   
...
<Loader id="ipaacaembodiment" loader="asap.ipaacaembodiments.loader.IpaacaEmbodimentLoader"/>
<Loader id="bodyfaceembodiment" requiredloaders="ipaacaembodiment" loader="asap.ipaacaembodiments.loader.IpaacaFaceAndBodyEmbodimentLoader">
  <renaming skeletonRenamingFile="billieskeletonrenaming.xml" morphRenamingFile="billiemorphsrenaming.xml"/>
</Loader>  
<Loader id="mixedskeletonembodiment" loader="hmi.physicsenvironment.MixedSkeletonEmbodimentLoader" requiredloaders="bodyfaceembodiment"/>
<Loader id="animationengine"
          loader="asap.animationengine.loader.MixedAnimationEngineLoader"
          requiredloaders="mixedskeletonembodiment,physicalembodiment">
    <GestureBinding basedir="" resources="Humanoids/billie/gesturebinding/" filename="gesturebinding.xml"/>
    <StartPose resources="Humanoids/billie/restposes/" filename="looselyhangingarms.xml"/>
    <Hns resources="Humanoids/shared/hns" filename="hns.xml"/>
</Loader>

<Loader id="faceengine" 
          loader="asap.faceengine.loader.FaceEngineLoader"
          requiredloaders="bodyfaceembodiment">
    <FaceBinding basedir="" resources="Humanoids/billie/facebinding/" filename="facebinding.xml"/>
</Loader>   
...
</AsapVirtualHuman>   

Using IpaacaWorldEnvironment (e.g. to point to/look at objects)

On the AsapRealizer side:

WorldObjectEnvironment we = new WorldObjectEnvironment();
IpaacaWorldEnvironment ipWEnv = new IpaacaWorldEnvironment(we.getWorldObjectManager());
List<Environment> environments = new ArrayList<Environment>();
environments.add(ipWEnv); 
...
AsapEnvironment ee = new AsapEnvironment();
ee.init(environments, ..

On the OgreRender side:

Create a new project based on the billie_project_asap template:

soa create billieproject_asap myBillie
cd myBillie
soa build-all

Add

Billie	SceneInformation	soaOgreSceneInformationComponent

to resource/plugins.conf

Optionally, add

Billie	RealizerTestScene	soaOgreRealizerTestSceneComponent

to resource/plugins.conf to get a default scene with a few boxes and a moving sphere. Run from the myBillie/resource directory using

./go.billie

Submitting and receiving BML through Ipaaca

With Billie

Start AsapRealizer with

ant run -Drun.main.class="asap.realizerdemo.AsapRealizerDemoExternalRenderer" -Drun.argline="asapbilliersb_ipaacaport.xml"

This moves Billie in the restpose when succesfully started.

In a new console window: send a BML file, wait for it to finish:

ant run -Drun.main.class="asap.realizerdemo.IpaacaBMLSender" -Drun.argline="simpledemo/helloworld2.xml"

In general

  1. Setup a VirtualHumanLoader xml file that uses an IpaacaToBMLRealizerAdapterLoader. An example configuration is provided in AsapDemo/AsapRealizerDemo/java/resource/asaparmandiaipaacaport.xml. To set up/configure your own use:
    <AsapVirtualHuman>
      <Loader id="realizer" loader="asap.realizerembodiments.AsapRealizerEmbodiment">
        ...
        <PipeLoader id="pipe" loader="asap.ipaacaadapters.loader.IpaacaToBMLRealizerAdapterLoader"/>
      </Loader>
      ...
    
  2. Start spread (you can use trailing & to run in the background):
    spread -n localhost &
    
  3. Run the AsapRealizerDemo with the newly defined xml
    ant run -Drun.argline="asaparmandiaipaacaport.xml"
    
  4. The realizer is now ready to receive and submit Ipaaca BML messages. The demo programs in java/src/asap/realizerdemo/AsapRealizerPortToIpaacaDemo.java and python/src/asap_realizerdemo/RealizerPortToIpaacaDemo.py show how to submit BML and receive BML feedback in java and python respectively.
  5. To start the python demo; in the python subdir: [TODO: this probably doesn't work without the python spread library installed.]
    ant resolve
    ant main #select asap_realizerdemo/RealizerPortToIpaacaDemo.py
    ant run
    
  6. To start the java demo; in the java subdir:
    ant main #select asap.realizerdemo.AsapRealizerPortToIpaacaDemo
    ant run
    

Connecting Ipaaca to the Ogre rendering environment AND Submitting and receiving BML through Ipaaca

Setup as above and run the AsapRealizerDemoExternalRenderer with

ant run -Drun.argline="asapbilliersb_ipaacaport.xml"

Connecting to an Ipaaca TTSAgent

Use the following TTS setup in your VirtualHumanLoader:

<Loader id="ttsbinding" loader="asap.ipaacattsbinding.loader.IpaacaTTSBindingLoader">
    <PhonemeToVisemeMapping resources="Humanoids/shared/phoneme2viseme/" filename="sampade2ikp.xml"/>
</Loader>

<Loader id="speechengine" loader="asap.speechengine.loader.SpeechEngineLoader" requiredloaders="facelipsync,ttsbinding">
   <Voice factory="WAV_TTS"/>
</Loader>

Examples config files for this include:

armandia_ipaacatts (armandia character, ipaaca tts, integrated ttsport, hmigraphics renderer)
asapbilliersb_ipaacatts (billie character, ipaaca tts, ogre renderer)
asapbilliersb_ipaacatts_ipaacaport (billie character, ipaaca tts, ipaacaport, ogre renderer)