wiki:BuildSystem

Version 37 (modified by welberge, 7 years ago) (diff)

--

The ANT based build system

Goal of the build system

The modules in Asap are dependent on many external and internal libraries, sometimes in different versions. Furthermore, each machine (32bit, 64bit, mac, windows, linux) may require its own version of a specific library. The build system was created to:

  • collect all dependent external libraries, and all internal releases, in a structured repository that allows to keep track of versions and dependencies
  • allow one to resolve, from this structured repository, exactly those versions of dependent libraries that are required for your project, on your machine and your operating system
  • make it easy to release new versions of our own modules
  • make it easy to use, in module A, the current latest compile of module B, instead of the released version of module B

Build philosophy

It is often necessary to (re)build projects, even by persons who didn’t develop that project. Since many different tools are being used, we have to agree on a number of basic conventions and procedures.

  • We do not assume that everyone will be using the IDE or any IDE at all. But we do require that a project created and developed, say, by means of Netbeans or Eclipse, can be (re)build without any of these tools available.
  • Since we have to use some common build tool, we have agreed to use  ant as that “common” minimal platform.
  • In principle every project lives in its own directory, containing all relevant source code, test code, project specific data, library files needed by that project, project documentation, etcetera. We have a preferred standard layout for the directory structure inside a project. A project also contains an ant buildfile (build.xml) and usually also a build.properties file. We require that we can (re)build and run a project using ant, without any reliance on development tools that might have been used to develop the project. To be clear: we don’t want to install JBuilder or Netbeans or Eclipse or whatever just to build and run your project. The ant file can be either a simple “standalone” build file, but the preferred way is to use a very small build file that just links to our shared build file (see layout)
  • We have a limited number of shared projects, all of which are available from our GIT repositories:
    • There is a “project” called hmibuild. It contains the shared ant build files. You must have this project in order to use our build system.
    • Shared software is available as source code or as compiled jar files. Most projects use the precompiled library (e.g. jar) files, which are kept in a project’s lib directory. (For tools like Eclipse or Netbeans you must do some configuring in order to use these library files, see below). We use a tool called  ivy, used by our build files, for easy version management of lib files that relies on our web repositories (hmirepo.ewi.utwente.nl, asap-project.org/repo). Basically, when you type “ant resolve”, then ivy will copy the library files needed for your project into the lib directory of your project. What will be copied is derived from a project file called “ivy.xml”.
    • There are a few “projects”, like HmiResource, that contains just “resource” data of all sorts, that is shared between projects. For instance, BML scripts, data for 3D scenes and avatars etcetera lives here. Usually, you can obtain such data also from the web repository, in packaged jar format. Sometimes, you want to actually see and modify that data, and in that case you will need to check out the relevant resource data from the git repositories.
    • Projects import and export class code and data in the form of jar or zip files. Whenever viable, there is no sharing of source code (one exception is C++ code in Linux, which typically has to be recompiled on each system). This ensures that every project can be built stand alone, after importing the necessary library files.

Starting point

Clone the hmibuild repository (see GitRepositories for repository locations). To make the build system work out-of-the box with our other code, this code must reside in the same top level directory as hmibuild.

e.g.

<gitroot>/hmibuild
<gitroot>/AsapDemo
<gitroot>/HmiDemo
<gitroot>/AsapResource
<gitroot>/HmiResource
...etc

And, if you'd like to compile the Asap/Hmi sources:

<gitroot>/HmiCore
<gitroot>/HmiEnvironment
<gitroot>/Asap
....etc

Quickstart: resolving libraries for and running a Java demo

In command prompt the module directory (e.g. AsapDemo/AsapRealizerDemo):

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to select the main class
  • ant run to compile and run the project

Creating your own Java project

The build system assumes the following directory layout:

yourproject/build.xml                 (ant build file)
yourproject/ivy.xml                   (specification of jar dependencies)
yourproject/build.properties          (build properties)
yourproject/src                       (source directory)
yourproject/lib                       (directory for resolved jars (auto filled, do not add custom jars here))
yourproject/lib.local directory       (directory for custom jars for your project)
yourproject/resource                  (directory for data resources for your project)
yourproject/test/ivy.xml              (specification of test specific jar dependencies)
yourproject/test/src                  (test sources)
yourproject/test/lib                  (directory for resolved test jars (auto filled, do not add custom jars here))
yourproject/test/resource             (test specific data)

To use the build system with your own Java project, you'll need to set up a build.xml, build.properties and ivy.xml file. The build.xml file provides the project name and normally simply refers to the default build.xml in hmibuild: e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<project name="HmiNeuroPhysicsDemo"  default="run">
   <import file="../../hmibuild/build.xml" />
</project>

The ivy.xml file is used to specify the library dependencies of the project. For example:

<ivy-module version="2.0">
   <info organisation="HMIDemo" module="HmiNeuroPhysicsDemo"/>
   <configurations>
      <include file="${ivy.settings.dir}/configurations.xml"/>
   </configurations>
   <dependencies>
      <dependency org="HMI"   name="HmiRenderEnvironment"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimationEmbodiments"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimation"     rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiGraphics"      rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiMath"          rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiNeurophysics"  rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiUtil"          rev="latest.${resolve.status}"  />
			
      <dependency org="jcip"      name="jcip-annotations"  rev="latest.release"  />   
      <dependency org="slf4j"     name="slf4j-api"         rev="latest.release"  />   
      <dependency org="logback"   name="logback-classic"   rev="latest.release"  />   
      <dependency org="logback"   name="logback-core"      rev="latest.release"  />   
      <dependency org="Sun"       name="jlfgr"             rev="latest.release"  />               
   </dependencies>
</ivy-module> 

rev="latest.release" means: resolve the latest version of this external library rev="latest.${resolve.status}" means: resolve the latest version of this library of ${resolve.status} resolve.status is set in the build properties, valid values are beta (for daily release) and release (for stable version).

The full list of available external libraries and Hmi libraries can be found here:  http://hmirepo.ewi.utwente.nl/. Asap's libraries names and jars can be found here:  http://asap-project.org/repo/.

The build.properties file specifies several build and run options. For example resource path and runtime arguments for the jvm:

resolve.status=beta
resource.path=${shared.resources}/DefaultShaders/resource;${shared.resources}/LogbackConfigs/resource;${shared.resources}/HmiHumanoidEmbodimentArmandia/resource;
run.jvmargs= -Xms128m -Xmx512m  -Xss5M -Dlogback.configurationFile=LogbackConfigs/warnlogstdout.xml

Use cases

Running and compiling against released versions

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Selecting a main class without running ant main

ant run -Drun.main.class="package.MainClass"

Using runtime parameters

ant run -Drun.argline="runtimeparams"

with runtimeparams a space seperated list of runtime parameters that are to be passed to your Java program.

Testing

  • ant junit to compile and run the test cases
  • ant junitreport to compile and run the test cases and generate an html report in the test/report directory

Resolving dependencies

Dependencies are resolved from

  1. the local.lib directory of a project
  2. the local repository directory of a project (can be set up in its repository directory)
  3. the repositories in  http://asap-project.org/repo/ and  http://hmirepo.ewi.utwente.nl/

`ant resolve' and similar resolve commands empty the lib directory, so use local.lib to store local libraries instead. Local resources are, by convention, stored in a projects resource directory.

Advanced: new/alternative default repositories can be added in !hmibuild/ivy/ivysettings.xml

Advanced

Running and compiling against source code

It is possible (but not recommended) to run and compile your project against the source of other projects. To this end, define the projects source dependencies using rebuild.list:

e.g.

rebuild.list=Hmi/HmiUtil, Hmi/HmiMath, Hmi/HmiAnimation, Hmi/HmiNeurophysics, Hmi/HmiRenderEnvironment, Hmi/HmiAnimationEmbodiments
  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant compilelist to compile dependent modules
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Fixing java.lang.OutOfMemoryError: PermGen space

For large (re)builds, especially of HmiDemo projects with lots of dependencies, the default ant settings do not suffice, and ant might crash because of memory problems; typically

java.lang.OutOfMemoryError: PermGen space

This error can be fixed by explicitly setting the permgensize in the ANT_OPTS environment variable. In linux this is done using:

export ANT_OPTS="-XX:MaxPermSize=1024M"

in e.g. ~/.bashrc. In windows,ant calls the antrc_pre.bat file (if it exists) before running ant, so the environment variable can be set there using:

set ANT_OPTS=-XX:MaxPermSize=1024M

ant_rc.bat should be placed in your HOME directory.

Using a local repository

By default, the Hmi, Asap and external libraries are checked out from the https server and it is strongly advised that you resolve them from there. However, if you (temporarily, e.g. at a conference) don't have an internet connection, you can set up the build system to check out from a local directory (here: d:/hmirepo). This is done as follows:

Create a hmi.build.properties file in your home directory and add:

hmi.repository.filesystem.root=d:/hmirepo
repository.mode=filesystem

Copy the hmirepository to your directory of choice. Rsync can be used to create incremental copies:

rsync -v -a --delete loginname@hmirepo.ewi.utwente.nl:/local2/hmirepo .

Compiling to/as a different Java version

In build.properties:

javac.version=1.6 (both source and target are 1.6)

or

javac.target=1.6  (source may e.g. be Java 1.7, target is 1.6 class code)

Releasing external jar files

hmibuild/publishexternal/publish.py is a python script to release external jar files into e.g. hmirepo. This script will create the ivy xml file, setup up the correct directory structure in the repository and copy over a specified jar. The script assumes that a private key for the server is available in ~/.ssh/id_rsa. Windows users will need to install the winrandom and Crypto packages for Python, in linux (at least on the new techfak netboot machines in Bielefeld) it just works. Run the script as follows:

python ./publish.py -v 1.1 -o myorganization -m mymodule --host welberge@hmirepo.ewi.utwente.nl mymodule-1.1.jar

Releasing your project on a repository

To create a release of your project on the soa, asap or hmi repository:

ant release

This requires that you specify the repository in the build.properties file, e.g.:

publish.resolver=soa.core.repository

Of course you'll also need to have write access to the repository. The following repositories are available by default:

asap.sftp.publish: the asap code ssh repository (web version located at http://asap-project.org/repo/repository)
asap.resource.sftp.publish: the asap resource ssh repository (web version located at http://asap-project.org/repo/resource)
soa.core.repository: the soashared repository, a git repository assumed to be located in your <gitroot>/soashared directory. Send a pull request to Herwin add your local changes into the shared git repository.
hmicore.sftp.publish: the hmi code ssh repository (web version located at http://hmirepo.ewi.utwente.nl/repository)
resource.sftp.publish: the resource hmi repository (web version located at http://hmirepo.ewi.utwente.nl/resource)

Releasing under a different user name

If you need to release under a different name than your username (e.g. at home), use

publish.user.name=yourpublishusername

in your ~/hmi.build.properties file.

More

Detailed explanation of the build system is still being written. For specific questions, ask below.

Discussion

BuildSystem
 unsolved

Reply here to give additional information or ask questions.