wiki:BuildSystem

Version 58 (modified by welberge, 7 years ago) (diff)

--

The ANT based build system

Goal of the build system

The modules in Asap are dependent on many external and internal libraries, sometimes in different versions. Furthermore, each machine (32bit, 64bit, mac, windows, linux) may require its own version of a specific library. The build system was created to:

  • collect all dependent external libraries, and all internal releases, in a structured repository that allows to keep track of versions and dependencies
  • allow one to resolve, from this structured repository, exactly those versions of dependent libraries that are required for your project, on your machine and your operating system
  • make it easy to release new versions of our own modules
  • make it easy to use, in module A, the current latest compile of module B, instead of the released version of module B

Build philosophy

It is often necessary to (re)build projects, even by persons who didn’t develop that project. Since many different tools are being used, we have to agree on a number of basic conventions and procedures.

  • We do not assume that everyone will be using the IDE or any IDE at all. But we do require that a project created and developed, say, by means of Netbeans or Eclipse, can be (re)build without any of these tools available.
  • Since we have to use some common build tool, we have agreed to use  ant as that “common” minimal platform.
  • In principle every project lives in its own directory, containing all relevant source code, test code, project specific data, library files needed by that project, project documentation, etcetera. We have a preferred standard layout for the directory structure inside a project. A project also contains an ant buildfile (build.xml) and usually also a build.properties file. We require that we can (re)build and run a project using ant, without any reliance on development tools that might have been used to develop the project. To be clear: we don’t want to install JBuilder or Netbeans or Eclipse or whatever just to build and run your project. The ant file can be either a simple “standalone” build file, but the preferred way is to use a very small build file that just links to our shared build file (see layout)
  • We have a limited number of shared projects, all of which are available from our GIT repositories:
    • There is a “project” called hmibuild. It contains the shared ant build files. You must have this project in order to use our build system.
    • Shared software is available as source code or as compiled jar files. Most projects use the precompiled library (e.g. jar) files, which are kept in a project’s lib directory. (For tools like Eclipse or Netbeans you must do some configuring in order to use these library files, see below). We use a tool called  ivy, used by our build files, for easy version management of lib files that relies on our web repositories (hmirepo.ewi.utwente.nl, asap-project.org/repo). Basically, when you type “ant resolve”, then ivy will copy the library files needed for your project into the lib directory of your project. What will be copied is derived from a project file called “ivy.xml”.
    • There are a few “projects”, like HmiResource, that contains just “resource” data of all sorts, that is shared between projects. For instance, BML scripts, data for 3D scenes and avatars etcetera lives here. Usually, you can obtain such data also from the web repository, in packaged jar format. Sometimes, you want to actually see and modify that data, and in that case you will need to check out the relevant resource data from the git repositories.
    • Projects import and export class code and data in the form of jar or zip files. Whenever viable, there is no sharing of source code (one exception is C++ code in Linux, which typically has to be recompiled on each system). This ensures that every project can be built stand alone, after importing the necessary library files.

Why dependency management

The contents of lib directories consists of jar les and/or \dll" or \so" les that are necessary for compiling and running the project. The basic strategy is that inter-dependencies between projects are via import/export of library jar files, in preference over direct source code dependencies. Of course we have to face the problem of project versioning. We have stable release versions of projects, but also less stable daily releases and alpha versions. Here we have some conventions and rules. For instance, we do not want a stable release version of project X to be dependent on an unstable alpha version of project Y. The other way around, so an alpha version of X dependent on a release version of Y is OK of course. It will be clear that manual version management is not a good idea: lot's of work, and error prone. Instead, we use a dependency manager,  Ivy.

How does it work?

Every project has an "ivy.xml" file that describes project dependencies. The ant build files, relying on the Ivy system, use these so called "ivy's" for resolving the contents of the project's lib directory. The system is based upon the notion of configurations, versions, and status of versions. For example, our build system uses "alpha", "daily build", and "release" as possible status of some jar version. The ordering of these statuses is relevant. For instance, when you ask for a "latest beta" version of some module Ivy will choose the latest amongst published daily releases and release (but not alpha) versions of that module. Similarly, if you ask for a "latest alpha", anything is acceptable, but if you ask for a "latest release", then only versions classified as "release" are taken into consideration. Ivy allows for several "configurations" of the project, each with its own set of dependencies. Currently, we have configurations for producing alpha, daily build, and release versions of the project itself, and a "test" configuration for (extra) dependencies needed for running tests. (For technical reasons we have two more configurations, called "master" and \default", which are discussed below) The workflow is roughly as follows: first move the project into either the alpha, beta or release configuration, second do your development, including testing etc, third publish an alpha, beta or normal release based upon the current configuration. When you actually publish, we attach a version number to the jar file. Also, the Ivy system records metadata concerning published modules based upon version numbers, to be used later on, when resolving for other projects. It does so by publishing not just a "jar" file (or other "artifacts", as they are called by Ivy), but also an accompanying ivy.xml file, derived from the project's ivy at the moment of publication. In this way, the Ivy system knows not only about direct dependencies, but is also capable of resolving recursive dependencies. This means that, for instance, when my module X declares (just) a dependency on the HmiGraphics package, the resolve process will look into the dependencies of HmiGraphics. The result is that project X will receive jar files for HmiGraphics, but also for HmiAnimation, HmiXml, HmiMath, and HmiUtil, because of (recursive) dependencies. When some jar file is required more than once, say via a direct dependency and also via some indirect dependency, and the versions required are not consistent, then Ivy delivers the "latest" version. So, for instance, in the example above, if project X declares (direct) dependencies on the alpha version of HmiGraphics and the release version of HmiXml, while the alpha version of HmiGraphics declares itself a dependency on the beta version of HmiXml, then project X will receive that beta version of HmiXml. That should be ok: since we are asking for some alpha version (of HmiGraphics, we should allow other alpha and/or beta versions if HmiGraphics actually needs them, even if our own project would be satisfied with the most recent release version.

Version numbering

We use some conventions; the Ivy system does not require some particular numbering scheme, but uses version numbers to determine which version is "more recent". This includes a few subtle cases. For instance, Ivy knows that version 1.4-alpha is before 1.4, and that versions 2.0-rc1 and 2.0-dev384 are both before 2.0. Our conventions:

  • Basically, we use version numbers of the form major.minor where major are minor are numbers. Examples: 0.1, 1.4, 2.0, 2.0.1.
  • We allow sufixes of the form -alpha, to denote alpha release versions, and -devi, where i some number, to denote "daily releases" (or developer versions). So, for instance, 2.0-dev384 is a developer version, working towards full release 2.0.
  • The major.minor form is the preferred form for full releases.
  • The major.minor-alpha form is the preferred form for alpha releases.
  • Sometimes we use version numbers of the form major.minor.maintenance. This form is intended for releases that are bug fixes for normal major.minor releases. Say currently we have a release version 2.3, and we already produced newer beta versions 2.4 and 2.5. Then we detect an annoying bug in 2.3. Now we could correct this and then produce a full release 2.6, but that would also include our new/experimental/buggy beta and alpha code from 2.4, 2.5. The better solution here would be to use git to temporarily revert to the code of version 2.3, do the bug fixing, then publish that as maintenance release version 2.3.1. Note that this is a release version, not an alpha or beta, so it that will become the preferred one when resolving for a "latest release" version.
  • Version numbers for full releases and dev releases are to be unique. Once published, you cannot reuse the same version number for a new release anymore. For alpha versions this would be counterproductive, since alpha versions can be produced in rapid succession, with only very minor differences. For this reason, our strategy is that alpha releases will not be put on the repository, so are not shared, therefore do not need unique version numbers.
  • The major.minor-devi form is used as a "developer version" for release major.minor. Such versions are produced during our "nightly" build process. (Should be running on a daily basis; but when testing during the nightly build produces error, no new versions are published that night). Daily builds are thus more stable than alphas, in that they are guaranteed to pass the test cases. The number i matches to the build number on the daily build server. To guarantee unique daily build numbers, daily builds should therefore only be produced by the build server; if an inbetween daily build is desired, one can create on by starting the daily build job on the build server. TODO: currently only Herwin can do this...

Starting point

Clone the hmibuild repository (see GitRepositories for repository locations). To make the build system work out-of-the box with our other code, this code must reside in the same top level directory as hmibuild.

e.g.

<gitroot>/hmibuild
<gitroot>/AsapDemo
<gitroot>/HmiDemo
<gitroot>/AsapResource
<gitroot>/HmiResource
...etc

And, if you'd like to compile the Asap/Hmi sources:

<gitroot>/HmiCore
<gitroot>/HmiEnvironment
<gitroot>/Asap
....etc

Quickstart: resolving libraries for and running a Java demo

Set up the hmibuild and your project top level directory as described above under "Starting point".

In command prompt the module directory (e.g. AsapDemo/AsapRealizerDemo):

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to select the main class
  • ant run to compile and run the project

Creating your own Java project

The build system assumes the following directory layout:

yourproject/build.xml                 (ant build file)
yourproject/ivy.xml                   (specification of jar dependencies)
yourproject/build.properties          (build properties)
yourproject/src                       (source directory)
yourproject/generatedsrc              (source code generated by a source code generation tool (e.g. [[http://code.google.com/p/protobuf/|protoc]]))
yourproject/lib                       (directory for resolved jars (auto filled, do not add custom jars here))
yourproject/lib.local directory       (directory for custom jars for your project)
yourproject/resource                  (directory for data resources for your project)
yourproject/test/ivy.xml              (specification of test specific jar dependencies)
yourproject/test/src                  (test sources)
yourproject/test/lib                  (directory for resolved test jars (auto filled, do not add custom jars here))
yourproject/test/resource             (test specific data)

To use the build system with your own Java project, you'll need to set up a build.xml, build.properties and ivy.xml file. The build.xml file provides the project name and normally simply refers to the default build.xml in hmibuild: e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<project name="HmiNeuroPhysicsDemo"  default="run">
   <import file="../../hmibuild/build.xml" />
</project>

The ivy.xml file is used to specify the library dependencies of the project. For example:

<ivy-module version="2.0">
   <info organisation="HMIDemo" module="HmiNeuroPhysicsDemo"/>
   <configurations>
      <include file="${ivy.settings.dir}/configurations.xml"/>
   </configurations>
   <dependencies>
      <dependency org="HMI"   name="HmiRenderEnvironment"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimationEmbodiments"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimation"     rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiGraphics"      rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiMath"          rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiNeurophysics"  rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiUtil"          rev="latest.${resolve.status}"  />
			
      <dependency org="jcip"      name="jcip-annotations"  rev="latest.release"  />   
      <dependency org="slf4j"     name="slf4j-api"         rev="latest.release"  />   
      <dependency org="logback"   name="logback-classic"   rev="latest.release"  />   
      <dependency org="logback"   name="logback-core"      rev="latest.release"  />   
      <dependency org="Sun"       name="jlfgr"             rev="latest.release"  />               
   </dependencies>
</ivy-module> 

rev="latest.release" means: resolve the latest version of this external library rev="latest.${resolve.status}" means: resolve the latest version of this library of ${resolve.status} resolve.status is set in the build properties, valid values are beta (for daily release) and release (for stable version).

The full list of available external libraries and Hmi libraries can be found here:  http://hmirepo.ewi.utwente.nl/. Asap's libraries names and jars can be found here:  http://asap-project.org/repo/.

The build.properties file specifies several build and run options. For example resource path and runtime arguments for the jvm:

resolve.status=beta
resource.path=${shared.resources}/DefaultShaders/resource;${shared.resources}/LogbackConfigs/resource;${shared.resources}/HmiHumanoidEmbodimentArmandia/resource;
run.jvmargs= -Xms128m -Xmx512m  -Xss5M -Dlogback.configurationFile=LogbackConfigs/warnlogstdout.xml

Use cases

Running and compiling against released versions

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Selecting a main class without running ant main

ant run -Drun.main.class="package.MainClass"

Using runtime parameters

ant run -Drun.argline="runtimeparams"

with runtimeparams a space seperated list of runtime parameters that are to be passed to your Java program.

Testing

  • ant junit to compile and run the test cases
  • ant junitreport to compile and run the test cases and generate an html report in the test/report directory

Resolving dependencies

Dependencies are resolved from

  1. the local.lib directory of a project
  2. the local repository directory of a project (can be set up in its repository directory)
  3. the repositories in  http://asap-project.org/repo/ and  http://hmirepo.ewi.utwente.nl/

`ant resolve' and similar resolve commands empty the lib directory, so use local.lib to store local libraries instead. Local resources are, by convention, stored in a projects resource directory.

Advanced: new/alternative default repositories can be added in !hmibuild/ivy/ivysettings.xml

Using the build system in eclipse

See UsingAsapRealizerInEclipse

Advanced

Running and compiling against source code

It is possible (but not recommended) to run and compile your project against the source of other projects. To this end, define the projects source dependencies using rebuild.list:

e.g.

rebuild.list=Hmi/HmiUtil, Hmi/HmiMath, Hmi/HmiAnimation, Hmi/HmiNeurophysics, Hmi/HmiRenderEnvironment, Hmi/HmiAnimationEmbodiments
  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant compilelist to compile dependent modules
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Fixing java.lang.OutOfMemoryError: PermGen space

For large (re)builds, especially of HmiDemo projects with lots of dependencies, the default ant settings do not suffice, and ant might crash because of memory problems; typically

java.lang.OutOfMemoryError: PermGen space

This error can be fixed by explicitly setting the permgensize in the ANT_OPTS environment variable. In linux this is done using:

export ANT_OPTS="-XX:MaxPermSize=1024M"

in e.g. ~/.bashrc. In windows,ant calls the antrc_pre.bat file (if it exists) before running ant, so the environment variable can be set there using:

set ANT_OPTS=-XX:MaxPermSize=1024M

ant_rc.bat should be placed in your HOME directory.

Using a local repository

By default, the Hmi, Asap and external libraries are checked out from the https server and it is strongly advised that you resolve them from there. However, if you (temporarily, e.g. at a conference) don't have an internet connection, you can set up the build system to check out from a local directory (here: d:/hmirepo). This is done as follows:

Create a hmi.build.properties file in your home directory and add:

hmi.repository.filesystem.root=d:/hmirepo
repository.mode=filesystem

Copy the hmirepository to your directory of choice. Rsync can be used to create incremental copies:

rsync -v -a --delete loginname@hmirepo.ewi.utwente.nl:/local2/hmirepo .

Compiling to/as a different Java version

In build.properties:

javac.version=1.6 (both source and target are 1.6)

or

javac.target=1.6  (source may e.g. be Java 1.7, target is 1.6 class code)

Releasing external jar files

hmibuild/publishexternal/publish.py is a python script to release external jar files into e.g. hmirepo. This script will create the ivy xml file, setup up the correct directory structure in the repository and copy over a specified jar. The script assumes that a private key for the server is available in ~/.ssh/id_rsa. Windows users will need to install the winrandom and Crypto packages for Python, in linux (at least on the new techfak netboot machines in Bielefeld) it just works. Run the script as follows:

python ./publish.py -v 1.1 -o myorganization -m mymodule --host welberge@hmirepo.ewi.utwente.nl mymodule-1.1.jar

Releasing your project on a repository

To create a release of your project on the soa, asap or hmi repository:

ant release

This requires that you specify the repository in the build.properties file, e.g.:

publish.resolver=soa.core.repository

Of course you'll also need to have write access to the repository. The following repositories are available by default:

asap.sftp.publish: the asap code ssh repository (web version located at http://asap-project.org/repo/repository)
asap.resource.sftp.publish: the asap resource ssh repository (web version located at http://asap-project.org/repo/resource)
soa.core.repository: the soashared repository, a git repository assumed to be located in your <gitroot>/soashared directory. Send a pull request to Herwin add your local changes into the shared git repository.
hmicore.sftp.publish: the hmi code ssh repository (web version located at http://hmirepo.ewi.utwente.nl/repository)
resource.sftp.publish: the resource hmi repository (web version located at http://hmirepo.ewi.utwente.nl/resource)

ant release increments the minor version number, then produces a version with "release" status. When the current version happens to be a developer version, then "incrementing the minor" actually means "stripping off" the -devi suffix.

To increment the major release number use

ant majorrelease

Releasing data as a zip file

To release a project containing only data as a zip file, set the following in ivy.xml:

<publications>
     <artifact type="zip" ext="zip"/>
</publications>

in build.properties

language=zipresource

This will zip up everything in your resource dir and release it when you use one of the release targets (e.g. release, majorrelease). When resolving such a released dataset, it will be unpacked in the lib directory. Such resource project should contain a single top-level directory and no other files in resource to keep the lib dir somewhat clean and readable (this is somewhat enforced by the resolver).

Releasing under a different user name

If you need to release under a different name than your username (e.g. at home), use

publish.user.name=yourpublishusername

in your ~/hmi.build.properties file.

More

Detailed explanation of the build system is still being written. For specific questions, ask below.

Discussion

BuildSystem
 unsolved

Reply here to give additional information or ask questions.