wiki:MiddleWareConnection

Hooking up AsapRealizer to your desired middleware framework

Communication to and from AsapRealizer is set up using the RealizerPort interface, a lightweight Java interface through which  BML and  BML feedback is communicated as Java Strings. Middleware can connect to the RealizerPort interface using adapters.

Using an existing middleware adapter

Adapters are available to connect an AsapRealizer to Ipaaca, RSB, ActiveMQ or a custom socket connection. You can hook them up to a realizer using the AsapVirtualHuman xml configuration file.

e.g.:

<AsapVirtualHuman>
  <Loader id="realizer" loader="asap.realizerembodiments.AsapRealizerEmbodiment">
    <BMLParser>
      <BMLAttributeExtension class="asap.bml.ext.bmlb.BMLBBMLBehaviorAttributes"/>
      <BMLAttributeExtension class="asap.bml.ext.bmlt.BMLTBMLBehaviorAttributes"/>
    </BMLParser>
	
    <BMLScheduler>
      <SchedulingHandler class="asap.realizer.scheduler.BMLBandTSchedulingHandler" schedulingStrategy="asap.realizer.scheduler.SortedSmartBodySchedulingStrategy"/>
    </BMLScheduler>
    <PipeLoader id="pipe" loader="packagename.YourPreferedMiddleWareToBMLRealizerAdapterLoader"/>
  </Loader>
  ...
</AsapVirtualHuman>

Ipaaca:

<PipeLoader id="pipe" loader="asap.ipaacaadapters.loader.IpaacaToBMLRealizerAdapterLoader"/>

ActiveMQ:

<PipeLoader id="pipe" loader="asap.activemqadapters.loader.ActiveMQToBMLRealizerAdapterLoader"/>

Robotic Service Bus:

<PipeLoader id="pipe" loader="asap.rsbadapters.loader.RsbToBMLRealizerAdapterLoader"/>

Custom socket:

<PipeLoader id="pipe" loader="asap.tcpipadapters.loader.TCPIPToBMLRealizerAdapterLoader">
   <ServerOptions bmlport="6500" feedbackport="6501"/>
</PipeLoader>

Optional, UI for custom socket:

<AsapVirtualHuman>
  <Loader id="realizer" loader="asap.realizerembodiments.AsapRealizerEmbodiment">
  ...
  </Loader>
  <Loader id="guiembodiment" loader="asap.realizerembodiments.JFrameEmbodiment">
  ...
  </Loader>
  
  <Loader id="bridgeui" loader="asap.tcpipadapters.loader.BridgeServerUILoader" requiredloaders="realizer,guiembodiment"/>  
  ...
</AsapVirtualHuman>

Hooking up AsapRealizer by creating a new middleware adapter

If a middleware adapter for your favorite middleware does not exist, it's easy to create your own. In this example I will show how to connect AsapRealizer to a Behavior Planner using messages over the  Robotic Service Bus. To this end, two rsb adapters will be created. The RsbToBMLRealizerAdapter translates Rsb BML messages into BML commands for the BMLRealizerPort and submits feedback from the BMLRealizerPort over rsb. The BMLRealizerToRsbAdapter acts as an RealizerPort. It passes BML commands to the rsb and translates BML feedback from rsb into feedback for listeners registered to it.

RSB setup

RSB requires a running  spread daemon. Start it with

spread -n localhost

Defining the rsb scopes

Rsb messages travel through channels called 'scopes'. For our purposes we define two scopes, one for BML and one for BML feedback:

/**
 * Constants for rsb scopes
 * @author Herwin
 */
public final class RsbAdapterConstants
{
    public static final String BML_SCOPE = "/bml/bml";
    public static final String BML_FEEDBACK_SCOPE = "/bml/bmlfeedback";
    private RsbAdapterConstants(){}
}

RsbToBMLRealizerAdapter

The RsbToBMLRealizerAdapter must transfer BML messages it receives through the BML scope to the connected RealizerPort. This is done using an event handler in an rsb listener. The RsbToBMLRealizerAdapter must submit all feedback messages from the RealizerPort over the Rsb BML_FEEDBACK_SCOPE. This is achieved by connecting the RsbToBMLRealizerAdapter as a feedbacklistener to the RealizerPort and using an rsb informer to send out feedback whenever a feedback callback occurs.

See also  http://docs.cor-lab.de/rsb-manual/0.7/html/examples-basic.html for rsb's information sending/receiving basics.

package asap.rsbadapters;

import lombok.extern.slf4j.Slf4j;
import rsb.AbstractEventHandler;
import rsb.Event;
import rsb.Factory;
import rsb.Informer;
import rsb.InitializeException;
import rsb.Listener;
import rsb.RSBException;
import asap.bml.bridge.RealizerPort;
import asap.bml.feedback.BMLFeedbackListener;

/**
 * Submits rsb BML messages to a RealizerPort; submits RealizerPort feedbacks to the rsb. 
 * Assumes that the connected realizerport is threadsafe (or at least that its performBML function is).
 * @author Herwin
 */
@Slf4j
public class RsbToBMLRealizerAdapter implements BMLFeedbackListener
{
    private final RealizerPort realizerPort;
    private final Informer<String> informer;
    private final Listener listener;
    
    public RsbToBMLRealizerAdapter(RealizerPort port)
    {
        realizerPort = port;
        realizerPort.addListeners(this);
        Factory factory = Factory.getInstance();

        // setup feedback sender
        informer = factory.createInformer(RsbAdapterConstants.BML_FEEDBACK_SCOPE);        

        // setup BML receiver
        listener = factory.createListener(RsbAdapterConstants.BML_SCOPE);
        listener.addHandler(new AbstractEventHandler()
        {
            @Override
            public void handleEvent(Event event)
            {
                realizerPort.performBML(event.getData().toString());
            }
        }, true);
        
        try
        {
            listener.activate();
            informer.activate();
        }
        catch (InitializeException e)
        {
            throw new RuntimeException(e);
        }
    }

    /**
     * Submit feedback over rsb
     */
    @Override
    public void feedback(String feedback)
    {
        try
        {
            informer.send(feedback);
        }
        catch (RSBException e)
        {
            log.warn("Could not submit feedback over RSB.", e);
        }
    }

    public void close()
    {
        informer.deactivate();
        listener.deactivate();
    }
}

BMLRealizerToRsbAdapter

The BMLRealizerToRsbAdapter acts as a RealizerPort. It submits BML messages through the rsb BML scope, using an rsb informer. It receives feedback from the rsb BML_FEEDBACK scope, using an eventhandler of a rsb listener. This feedback is submitted to all registered BMLFeedbackListeners.

package asap.rsbadapters;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import com.google.common.collect.ImmutableList;

import lombok.extern.slf4j.Slf4j;
import rsb.AbstractEventHandler;
import rsb.Event;
import rsb.Factory;
import rsb.Informer;
import rsb.InitializeException;
import rsb.Listener;
import rsb.RSBException;
import asap.bml.bridge.RealizerPort;
import asap.bml.feedback.BMLFeedbackListener;

/**
 * Submits BML through rsb messages; submits received feedback (from rsb) to registered listeners.
 * @author Herwin
 */
@Slf4j
public class BMLRealizerToRsbAdapter implements RealizerPort
{
    private final Informer<String> informer;
    private final Listener listener;
    private List<BMLFeedbackListener> feedbackListeners = Collections.synchronizedList(new ArrayList<BMLFeedbackListener>());
    
    public BMLRealizerToRsbAdapter()
    {
        Factory factory = Factory.getInstance();

        // setup bml sender
        informer = factory.createInformer(RsbAdapterConstants.BML_SCOPE);  
        
        // setup feedback receiver
        listener = factory.createListener(RsbAdapterConstants.BML_FEEDBACK_SCOPE);
        listener.addHandler(new AbstractEventHandler()
        {
            @Override
            public void handleEvent(Event event)
            {
                synchronized(feedbackListeners)
                {
                    for(BMLFeedbackListener fbl:feedbackListeners)
                    {
                        fbl.feedback(event.getData().toString());
                    }
                }
            }
        }, true);
        
        try
        {
            listener.activate();
            informer.activate();
        }
        catch (InitializeException e)
        {
            throw new RuntimeException(e);
        }
    }
    
    @Override
    public void addListeners(BMLFeedbackListener... listeners)
    {
        feedbackListeners.addAll(ImmutableList.copyOf(listeners));        
    }

    @Override
    public void removeAllListeners()
    {
        feedbackListeners.clear();
    }

    @Override
    public void performBML(String bmlString)
    {
        try
        {
            informer.send(bmlString);
        }
        catch (RSBException e)
        {
            log.warn("Cannot send BML ",e);
        }        
    }
    
    public void close()
    {
        listener.deactivate();
        informer.deactivate();
    }
}

Testing the created adapters

A simple way to test the adapters is by mocking up a Realizer (through a RealizerPort) and a Behavior planner (through a BMLFeedbackListener). The realizer is then connected to the RsbToBMLRealizerAdapter; the feedbacklistener to the BMLRealizerToRsbAdapter. A BML message sent from the BMLRealizerToRsbAdapter should arrive at the mocked up realizer. A feedback message sent from the RsbToBMLRealizerAdapter should arrive at the behavior planner:

package asap.rsbadapters;

import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;

import org.junit.After;
import org.junit.Test;

import asap.bml.bridge.RealizerPort;
import asap.bml.feedback.BMLFeedbackListener;

/**
 * Integration tests for the BMLToIpaaca and IpaacaToBML adapters.
 * Requires a running spread daemon.
 * @author Herwin
 */
public class AsaptersIntegrationTest
{
    private RealizerPort mockRealizerPort = mock(RealizerPort.class);
    private BMLFeedbackListener mockFeedbackListener = mock(BMLFeedbackListener.class);
    private BMLRealizerToRsbAdapter bmlToRsb = new BMLRealizerToRsbAdapter();
    private RsbToBMLRealizerAdapter rsbToBML = new RsbToBMLRealizerAdapter(mockRealizerPort);
    
    @After
    public void tearDown()
    {
        bmlToRsb.close();
        rsbToBML.close();
    }
    
    @Test
    public void testPerformBML() throws InterruptedException
    {
        bmlToRsb.performBML("bmltest");
        Thread.sleep(500);
        verify(mockRealizerPort).performBML("bmltest");
    }
    
    @Test
    public void testFeedback() throws InterruptedException
    {
        bmlToRsb.addListeners(mockFeedbackListener);
        rsbToBML.feedback("bmlfeedback");
        Thread.sleep(500);
        verify(mockFeedbackListener).feedback("bmlfeedback");
    }
}

Creating a PipeLoader for RsbToBMLRealizerAdapter

To employ a RsbToBMLRealizerAdapter within the VirtualHumanSpec?, a Loader needs to be defined for it:

package asap.rsbadapters.loader;

import hmi.util.Clock;
import hmi.xml.XMLScanException;
import hmi.xml.XMLTokenizer;

import java.io.IOException;

import asap.bml.bridge.RealizerPort;
import asap.realizerembodiments.PipeLoader;
import asap.rsbadapters.RsbToBMLRealizerAdapter;

/**
 * Loads a RsbToBMLRealizerAdapter from XML
 * @author Herwin
 *
 */
public class RsbToBMLRealizerAdapterLoader implements PipeLoader
{
    private RealizerPort adaptedRealizerPort = null;
    
    @Override
    public void readXML(XMLTokenizer theTokenizer, String id, String vhId, String name, RealizerPort realizerPort, Clock theSchedulingClock)
            throws XMLScanException, IOException
    {
        adaptedRealizerPort = realizerPort;
        new RsbToBMLRealizerAdapter(realizerPort);         
        if (!theTokenizer.atETag("PipeLoader")) throw new XMLScanException("RsbToBMLRealizerAdapter should be an empty element");
    }

    @Override
    public RealizerPort getAdaptedRealizerPort()
    {
        return adaptedRealizerPort;
    }    
}

A testcase can be created to validate the loader parsing and check if teh RsbToBMLRealizerAdapter is connected (as listener) to the provided RealizerPort:

package asap.rsbadapters.loader;

import static org.junit.Assert.assertEquals;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import hmi.util.Clock;
import hmi.xml.XMLTokenizer;

import java.io.IOException;

import org.junit.Test;

import asap.bml.bridge.RealizerPort;
import asap.bml.feedback.BMLFeedbackListener;

/**
 * Unit tests for the RsbToBMLRealizerAdapterLoader
 * @author Herwin
 *
 */
public class RsbToBMLRealizerAdapterLoaderTest
{
    private RealizerPort mockRealizerPort = mock(RealizerPort.class);
    private Clock mockSchedulingClock = mock(Clock.class);
    
    @Test
    public void testReadFromXML() throws IOException
    {
        String pipeLoaderStr = "<PipeLoader id=\"id1\" loader=\"x\"/>";
        XMLTokenizer tok = new XMLTokenizer(pipeLoaderStr);
        tok.takeSTag("PipeLoader");
        RsbToBMLRealizerAdapterLoader loader = new RsbToBMLRealizerAdapterLoader();
        loader.readXML(tok, "id1", "vh1", "name", mockRealizerPort, mockSchedulingClock);
        verify(mockRealizerPort, times(1)).addListeners(any(BMLFeedbackListener[].class));
        assertEquals(mockRealizerPort, loader.getAdaptedRealizerPort());
    }
}

Trying out the rsb adapters

AsapDemo/AsapRealizerDemo contains two programs that can be used to rapidly try out the created adapters and the loader. AsapRealizerDemo can be used to set up an AsapRealizer connected to rsb. To do this, hook up the RsbToBMLRealizerAdapterLoader to a VirtualHumanSpec?:

<AsapVirtualHuman>
  <Loader id="realizer" loader="asap.realizerembodiments.AsapRealizerEmbodiment">
    <BMLParser>
      <BMLAttributeExtension class="asap.bml.ext.bmlb.BMLBBMLBehaviorAttributes"/>
      <BMLAttributeExtension class="asap.bml.ext.bmlt.BMLTBMLBehaviorAttributes"/>
    </BMLParser>
	
    <BMLScheduler>
      <SchedulingHandler class="asap.realizer.scheduler.BMLBandTSchedulingHandler" schedulingStrategy="asap.realizer.scheduler.SortedSmartBodySchedulingStrategy"/>
    </BMLScheduler>
    <PipeLoader id="pipe" loader="asap.rsbadapters.loader.RsbToBMLRealizerAdapterLoader"/>
  </Loader>
  ...
</AsapVirtualHuman>

And run AsapRealizerDemo with this spec:

ant run -Drun.argline="specname.xml" -Drun.main.class="asap.realizerdemo.AsapRealizerDemo"

Run a AsapRealizerPortUIDemo with the BMLRealizerToRsbAdapter as follows:

ant run -Drun.argline="asap.rsbadapters.BMLRealizerToRsbAdapter" -Drun.main.class="asap.realizerdemo.AsapRealizerPortUIDemo"

The AsapRealizerPortUIDemo is now connect to the AsapRealizer through rsb.

If everything works you can send BML to the realizer using the BML textbox; feedback received from the realizer will show up in the feedback panes.

Attachments