wiki:BuildSystem

Version 63 (modified by welberge, 7 years ago) (diff)

--

The ANT based build system

Goal of the build system

The modules in Asap are dependent on many external and internal libraries, sometimes in different versions. Furthermore, each machine (32bit, 64bit, mac, windows, linux) may require its own version of a specific library. The build system was created to:

  • collect all dependent external libraries, and all internal releases, in a structured repository that allows to keep track of versions and dependencies
  • allow one to resolve, from this structured repository, exactly those versions of dependent libraries that are required for your project, on your machine and your operating system
  • make it easy to release new versions of our own modules
  • make it easy to use, in module A, the current latest compile of module B, instead of the released version of module B

Build philosophy

It is often necessary to (re)build projects, even by persons who didn’t develop that project. Since many different tools are being used, we have to agree on a number of basic conventions and procedures.

  • We do not assume that everyone will be using the IDE or any IDE at all. But we do require that a project created and developed, say, by means of Netbeans or Eclipse, can be (re)build without any of these tools available.
  • Since we have to use some common build tool, we have agreed to use  ant as that “common” minimal platform.
  • In principle every project lives in its own directory, containing all relevant source code, test code, project specific data, library files needed by that project, project documentation, etcetera. We have a preferred standard layout for the directory structure inside a project. A project also contains an ant buildfile (build.xml) and usually also a build.properties file. We require that we can (re)build and run a project using ant, without any reliance on development tools that might have been used to develop the project. To be clear: we don’t want to install JBuilder or Netbeans or Eclipse or whatever just to build and run your project. The ant file can be either a simple “standalone” build file, but the preferred way is to use a very small build file that just links to our shared build file (see layout)
  • We have a limited number of shared projects, all of which are available from our GIT repositories:
    • There is a “project” called hmibuild. It contains the shared ant build files. You must have this project in order to use our build system.
    • Shared software is available as source code or as compiled jar files. Most projects use the precompiled library (e.g. jar) files, which are kept in a project’s lib directory. (For tools like Eclipse or Netbeans you must do some configuring in order to use these library files, see below). We use a tool called  ivy, used by our build files, for easy version management of lib files that relies on our web repositories (hmirepo.ewi.utwente.nl, asap-project.org/repo). Basically, when you type “ant resolve”, then ivy will copy the library files needed for your project into the lib directory of your project. What will be copied is derived from a project file called “ivy.xml”.
    • There are a few “projects”, like HmiResource, that contains just “resource” data of all sorts, that is shared between projects. For instance, BML scripts, data for 3D scenes and avatars etcetera lives here. Usually, you can obtain such data also from the web repository, in packaged jar format. Sometimes, you want to actually see and modify that data, and in that case you will need to check out the relevant resource data from the git repositories.
    • Projects import and export class code and data in the form of jar or zip files. Whenever viable, there is no sharing of source code (one exception is C++ code in Linux, which typically has to be recompiled on each system). This ensures that every project can be built stand alone, after importing the necessary library files.

Why dependency management

The contents of lib directories consists of jar les and/or \dll" or \so" les that are necessary for compiling and running the project. The basic strategy is that inter-dependencies between projects are via import/export of library jar files, in preference over direct source code dependencies. Of course we have to face the problem of project versioning. We have stable release versions of projects, but also less stable daily releases and alpha versions. Here we have some conventions and rules. For instance, we do not want a stable release version of project X to be dependent on an unstable alpha version of project Y. The other way around, so an alpha version of X dependent on a release version of Y is OK of course. It will be clear that manual version management is not a good idea: lot's of work, and error prone. Instead, we use a dependency manager,  Ivy.

How does it work?

Every project has an "ivy.xml" file that describes project dependencies. The ant build files, relying on the Ivy system, use these so called "ivy's" for resolving the contents of the project's lib directory. The system is based upon the notion of configurations, versions, and status of versions. For example, our build system uses "alpha", "daily build", and "release" as possible status of some jar version. The ordering of these statuses is relevant. For instance, when you ask for a "latest beta" version of some module Ivy will choose the latest amongst published daily releases and release (but not alpha) versions of that module. Similarly, if you ask for a "latest alpha", anything is acceptable, but if you ask for a "latest release", then only versions classified as "release" are taken into consideration. Ivy allows for several "configurations" of the project, each with its own set of dependencies. Currently, we have configurations for producing alpha, daily build, and release versions of the project itself, and a "test" configuration for (extra) dependencies needed for running tests. (For technical reasons we have two more configurations, called "master" and \default", which are discussed below) The workflow is roughly as follows: first move the project into either the alpha, beta or release configuration, second do your development, including testing etc, third publish an alpha, beta or normal release based upon the current configuration. When you actually publish, we attach a version number to the jar file. Also, the Ivy system records metadata concerning published modules based upon version numbers, to be used later on, when resolving for other projects. It does so by publishing not just a "jar" file (or other "artifacts", as they are called by Ivy), but also an accompanying ivy.xml file, derived from the project's ivy at the moment of publication. In this way, the Ivy system knows not only about direct dependencies, but is also capable of resolving recursive dependencies. This means that, for instance, when my module X declares (just) a dependency on the HmiGraphics package, the resolve process will look into the dependencies of HmiGraphics. The result is that project X will receive jar files for HmiGraphics, but also for HmiAnimation, HmiXml, HmiMath, and HmiUtil, because of (recursive) dependencies. When some jar file is required more than once, say via a direct dependency and also via some indirect dependency, and the versions required are not consistent, then Ivy delivers the "latest" version. So, for instance, in the example above, if project X declares (direct) dependencies on the alpha version of HmiGraphics and the release version of HmiXml, while the alpha version of HmiGraphics declares itself a dependency on the beta version of HmiXml, then project X will receive that beta version of HmiXml. That should be ok: since we are asking for some alpha version (of HmiGraphics, we should allow other alpha and/or beta versions if HmiGraphics actually needs them, even if our own project would be satisfied with the most recent release version.

Version numbering

We use some conventions; the Ivy system does not require some particular numbering scheme, but uses version numbers to determine which version is "more recent". This includes a few subtle cases. For instance, Ivy knows that version 1.4-alpha is before 1.4, and that versions 2.0-rc1 and 2.0-dev384 are both before 2.0. Our conventions:

  • Basically, we use version numbers of the form major.minor where major are minor are numbers. Examples: 0.1, 1.4, 2.0, 2.0.1.
  • We allow sufixes of the form -alpha, to denote alpha release versions, and -devi, where i some number, to denote "daily releases" (or developer versions). So, for instance, 2.0-dev384 is a developer version, working towards full release 2.0.
  • The major.minor form is the preferred form for full releases.
  • The major.minor-alpha form is the preferred form for alpha releases.
  • Sometimes we use version numbers of the form major.minor.maintenance. This form is intended for releases that are bug fixes for normal major.minor releases. Say currently we have a release version 2.3, and we already produced newer beta versions 2.4 and 2.5. Then we detect an annoying bug in 2.3. Now we could correct this and then produce a full release 2.6, but that would also include our new/experimental/buggy beta and alpha code from 2.4, 2.5. The better solution here would be to use git to temporarily revert to the code of version 2.3, do the bug fixing, then publish that as maintenance release version 2.3.1. Note that this is a release version, not an alpha or beta, so it that will become the preferred one when resolving for a "latest release" version.
  • Version numbers for full releases and dev releases are to be unique. Once published, you cannot reuse the same version number for a new release anymore. For alpha versions this would be counterproductive, since alpha versions can be produced in rapid succession, with only very minor differences. For this reason, our strategy is that alpha releases will not be put on the repository, so are not shared, therefore do not need unique version numbers.
  • The major.minor-devi form is used as a "developer version" for release major.minor. Such versions are produced during our "nightly" build process. (Should be running on a daily basis; but when testing during the nightly build produces error, no new versions are published that night). Daily builds are thus more stable than alphas, in that they are guaranteed to pass the test cases. The number i matches to the build number on the daily build server. To guarantee unique daily build numbers, daily builds should therefore only be produced by the build server; if an inbetween daily build is desired, one can create on by starting the daily build job on the build server. TODO: currently only Herwin can do this...

Starting point

Clone the hmibuild repository (see GitRepositories for repository locations). To make the build system work out-of-the box with our other code, this code must reside in the same top level directory as hmibuild.

e.g.

<gitroot>/hmibuild
<gitroot>/AsapDemo
<gitroot>/HmiDemo
<gitroot>/AsapResource
<gitroot>/HmiResource
...etc

And, if you'd like to compile the Asap/Hmi sources:

<gitroot>/HmiCore
<gitroot>/HmiEnvironment
<gitroot>/Asap
....etc

Quickstart: resolving libraries for and running a Java demo

Set up the hmibuild and your project top level directory as described above under "Starting point".

In command prompt the module directory (e.g. AsapDemo/AsapRealizerDemo):

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to select the main class
  • ant run to compile and run the project

Creating your own project

The build system assumes the following directory layout:

yourproject/build.xml                 (ant build file)
yourproject/ivy.xml                   (specification of dependencies)
yourproject/build.properties          (build properties)
yourproject/src                       (source directory)
yourproject/generatedsrc              (source code generated by a source code generation tool (e.g. protoc))
yourproject/lib                       (directory for resolved jars (auto filled, do not add custom libraries here, nor commit to git))
yourproject/lib.local directory       (directory for custom libraries for your project)
yourproject/resource                  (directory for data resources for your project)
yourproject/privateprops              (private properties for your project, e.g. selected main class, should not be committed to git)
yourproject/build                     (compiled files, e.g. .jar, .pyc, should not be committed to git)
yourproject/dist                      (distribution of your project, e.g. .jar, .zip, should not be committed to git)
yourproject/docs                      (documentation, also for generated javadoc)
yourproject/test/ivy.xml              (specification of test specific jar dependencies)
yourproject/test/src                  (test sources)
yourproject/test/lib                  (directory for resolved test libraries (auto filled, do not add custom jars here))
yourproject/test/resource             (test specific data)

To use the build system with your own project, you'll need to set up a build.xml, build.properties and ivy.xml file. The build.xml file provides the project name and normally simply refers to the default build.xml in hmibuild: e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<project name="HmiNeuroPhysicsDemo"  default="run">
   <import file="../../hmibuild/build.xml" />
</project>

The ivy.xml file is used to specify the library dependencies of the project. For example:

<ivy-module version="2.0">
   <info organisation="HMIDemo" module="HmiNeuroPhysicsDemo"/>
   <configurations>
      <include file="${ivy.settings.dir}/configurations.xml"/>
   </configurations>
   <dependencies>
      <dependency org="HMI"   name="HmiRenderEnvironment"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimationEmbodiments"   rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiAnimation"     rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiGraphics"      rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiMath"          rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiNeurophysics"  rev="latest.${resolve.status}"  />
      <dependency org="HMI"   name="HmiUtil"          rev="latest.${resolve.status}"  />
			
      <dependency org="jcip"      name="jcip-annotations"  rev="latest.release"  />   
      <dependency org="slf4j"     name="slf4j-api"         rev="latest.release"  />   
      <dependency org="logback"   name="logback-classic"   rev="latest.release"  />   
      <dependency org="logback"   name="logback-core"      rev="latest.release"  />   
      <dependency org="Sun"       name="jlfgr"             rev="latest.release"  />               
   </dependencies>
</ivy-module> 

rev="latest.release" means: resolve the latest version of this external library rev="latest.${resolve.status}" means: resolve the latest version of this library of ${resolve.status} resolve.status is set in the build properties, valid values are beta (for daily release) and release (for stable version).

The full list of available external libraries and Hmi libraries can be found here:  http://hmirepo.ewi.utwente.nl/. Asap's libraries names and jars can be found here:  http://asap-project.org/repo/.

The build.properties file specifies several build and run options. For example resource path and runtime arguments for the jvm:

language=java
resolve.status=beta
resource.path=${shared.resources}/DefaultShaders/resource;${shared.resources}/LogbackConfigs/resource;${shared.resources}/HmiHumanoidEmbodimentArmandia/resource;
run.jvmargs= -Xms128m -Xmx512m  -Xss5M -Dlogback.configurationFile=LogbackConfigs/warnlogstdout.xml

Multiple programming languages are supported, including java, python, cpp (C++), scripts (simply releases everything in the scripts dir), none (for top-level projects that manage other projects using multiple programming language) and zipresource (for projects containing only data).

Use cases

Running and compiling against released versions

  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Selecting a main class without running ant main

ant run -Drun.main.class="package.MainClass"

Using runtime parameters

ant run -Drun.argline="runtimeparams"

with runtimeparams a space seperated list of runtime parameters that are to be passed to your Java program.

Testing

  • ant junit to compile and run the test cases
  • ant junitreport to compile and run the test cases and generate an html report in the test/report directory

Resolving dependencies

Dependencies are resolved from

  1. the local.lib directory of a project
  2. the local repository directory of a project (can be set up in its repository directory)
  3. the repositories in  http://asap-project.org/repo/ and  http://hmirepo.ewi.utwente.nl/

`ant resolve' and similar resolve commands empty the lib directory, so use local.lib to store local libraries instead. Local resources are, by convention, stored in a projects resource directory.

Advanced: new/alternative default repositories can be added in !hmibuild/ivy/ivysettings.xml

Using the build system in eclipse

See UsingAsapRealizerInEclipse

Using the build system in NetBeans?

While it is possible to build the code of your project in NetBeans?, eclipse will currently give you a much smoother integration experience. Contributions towards automatically generating NetBeans? files from our build system are very welcome. Currently, NetBeans? projects have to be set up manually.

NetBeans? has, in the background, an ant based build system that is not unlike our own build system. You can combine these, but you need to take some pre- cautions when you create a NetBeans? project. Potential problem: by default, NetBeans? want to create its own build.xml ant fi le in the project directory. The solution is to first create a project, including the appropriate build fi le, and the appropriate directory structure including the lib, the src, test\lib, and test\src directories. Check that it compiles, etcetera, but after that: do an ant clean. (Otherwise, NetBeans? will complain that there is already a build directory in your project.)

Then there are basically two options to proceed: the fi rst is to create a NetBeans? "free form" project. In that case, Netbeans will simply pick up your existing build.xml fi le, and use that for it own builds. Disadvantage: it's ok for basic editing, compiling, etcetera, but you cannot pro t from all Netbeans features. The second option is to create a new NetBeans? "Java project from existing sources". In that case, NetBeans? has its own build fi le, now called nbbuild.xml. Advantage: you can pro fit from all netBeans features, and it will not disturb the shared build files. Disadvantage: you must setup the NetBeans? project carefully and manually update it for library changes, as we will describe now.

  1. We assume that you are using NetBeans? 7.2 (or higher)
  2. create, outside NetBeans?, a basic project in line with our build system. If you have an existing project, that's ok as well. Ensure that you have

\resolved" the project dependencies, so that the contents of the lib and test\lib directories is what you need. (Use ant resolve)

  1. Ensure that the your project has no build directory: for instance use ant clean.
  2. Now create a new project within NetBeans?. Choose "New Java Project with Existing Sources".
  3. Choose the correct project name (by default we pick the project directory name), and select the correct project folder. Enable the "Use dedicated Folder for Storing Libraries" option, but choose a di fferent name than ".\lib". For instance, choose "nbproject\lib". NetBeans? stores some of its "own" info in its "lib" folder. Unfortunately, our version manager Ivy will discard such info from .\lib when you do an "ant resolve" operation, so we must give the NetBeans? lib a special place.
  4. "Next", add the src folder to the list of "Source Package Folders", and add the test\src folder to "Test Package Folders".
  5. "Now you can \Finish", and you have a NetBeans? project. Except that it won't compile, because you must correct the library settings:
  6. The NetBeans? "Projects" panel on the left show two sets of libraries: "Libraries" and \Test Libraries" Right click on "Libraries" and choose "Add JAR/Folder". Navigate to the project's lib folder and add all jars that you find there. Also add the project's resource directory here. (This will

ensure that this directory is on Java's classpath, so the "getResourceAsStream" method for loading resource data will work) Leave the "Reference as Relative Path" option enabled: this prevents NetBeans? from copying into its own nbproject\lib directory. Do the same with "Test Libraries", this time using the jars from test\lib, and the test\resource directory. As you can see we now have a junit-4.8.2.jar, that is present in our lib directory, but also two more NetBeans? "libraries", one for JUnit3.8.2 and one for JUnit4.8.2. The latter two should be removed.

  1. By now, the project should be in shape: no unresolved libraries, it compiles and runs, and JUnits test work.
  2. Note that whenever you change the contents of the lib directories, you will have to go back to the project properties page (from the File menu),

and reestablish the correct libraries settings. For example, say you had some fi le "guava-r06.jar" inside your lib directory, but after the resolve operation it has been replaced by "guava-r07.jar" NetBeans? now automatically removes guava-r06.jar from its libraries, but does not auto-understand that it needs guava-r07.jar instead. Right-click on the Netbeans project name, and choose "Resolve Reference Problems...". Select guava-r06.jar and click "Resolve". Now you can navigate to the project's lib directory, and select the new guava-r07.jar. For the time being, NetBeans? is now satisfied. In reality, it has not really removed the reference to guava-r06.jar, but it has added a redirection inside your own "private" netbeans settings. So next time that you resolve, it cannot fi nd guava-r06.jar again. You can correct this by first removing the guava jar from the NetBeans? library, then adding the correct version. Problem: you cannot remove it while the reference problem is not solved. In the end, the easy solution is to fi rst remove all jar files from lib (and also from test\lib if you expect changes over there) before the resolve, then add them back after the resolve.

building jar files

By default, compiled class fi les are included in the jar fi le produced by a build. But also all fi les inside the src directory, except for .java and .form fi les are included. That is convenient if you want to include "resource" fi les, like data fi les, images , icons etcetera: put them in the appropriate source directory, and use the Java getResourceAsStream() method for reading the data. Although this works, our strong preference is to include \resources" into its own resource directory within your project. (Our ant build system puts your project's own resource directory on the classpath, as well as other directories that you specify in the build.properties file) Netbeans auto-include of src documents is also not so nice if you have some secret-remarks.txt, or some huge remarks.doc word fi le in your source directories. In this case, you must set the files to exclude for packaging. For instance, via: Build | Set Main Project Configuration | Customize..., or via right-clicking the project tab, and selecting \Properties". Choose the \Packaging" tab, and to the "Exclude from jar file", for instance, add, /*.txt, /*.doc.

Advanced

Running and compiling against source code

It is possible (but not recommended) to run and compile your project against the source of other projects. To this end, define the projects source dependencies using rebuild.list:

e.g.

rebuild.list=Hmi/HmiUtil, Hmi/HmiMath, Hmi/HmiAnimation, Hmi/HmiNeurophysics, Hmi/HmiRenderEnvironment, Hmi/HmiAnimationEmbodiments
  • ant resolve to collect all dependent released and external libraries in the lib directory
  • ant compilelist to compile dependent modules
  • ant main to set the class that is to be run
  • ant run to compile and run the project

Fixing java.lang.OutOfMemoryError: PermGen space

For large (re)builds, especially of HmiDemo projects with lots of dependencies, the default ant settings do not suffice, and ant might crash because of memory problems; typically

java.lang.OutOfMemoryError: PermGen space

This error can be fixed by explicitly setting the permgensize in the ANT_OPTS environment variable. In linux this is done using:

export ANT_OPTS="-XX:MaxPermSize=1024M"

in e.g. ~/.bashrc. In windows,ant calls the antrc_pre.bat file (if it exists) before running ant, so the environment variable can be set there using:

set ANT_OPTS=-XX:MaxPermSize=1024M

ant_rc.bat should be placed in your HOME directory.

Using a local repository

By default, the Hmi, Asap and external libraries are checked out from the https server and it is strongly advised that you resolve them from there. However, if you (temporarily, e.g. at a conference) don't have an internet connection, you can set up the build system to check out from a local directory (here: d:/hmirepo). This is done as follows:

Create a hmi.build.properties file in your home directory and add:

hmi.repository.filesystem.root=d:/hmirepo
repository.mode=filesystem

Copy the hmirepository to your directory of choice. Rsync can be used to create incremental copies:

rsync -v -a --delete loginname@hmirepo.ewi.utwente.nl:/local2/hmirepo .

Compiling to/as a different Java version

In build.properties:

javac.version=1.6 (both source and target are 1.6)

or

javac.target=1.6  (source may e.g. be Java 1.7, target is 1.6 class code)

Releasing external jar files

hmibuild/publishexternal/publish.py is a python script to release external jar files into e.g. hmirepo. This script will create the ivy xml file, setup up the correct directory structure in the repository and copy over a specified jar. The script assumes that a private key for the server is available in ~/.ssh/id_rsa. Windows users will need to install the winrandom and Crypto packages for Python, in linux (at least on the new techfak netboot machines in Bielefeld) it just works. Run the script as follows:

python ./publish.py -v 1.1 -o myorganization -m mymodule --host welberge@hmirepo.ewi.utwente.nl mymodule-1.1.jar

Releasing your project on a repository

To create a release of your project on the soa, asap or hmi repository:

ant release

This requires that you specify the repository in the build.properties file, e.g.:

publish.resolver=soa.core.repository

Of course you'll also need to have write access to the repository. The following repositories are available by default:

asap.sftp.publish: the asap code ssh repository (web version located at http://asap-project.org/repo/repository)
asap.resource.sftp.publish: the asap resource ssh repository (web version located at http://asap-project.org/repo/resource)
soa.core.repository: the soashared repository, a git repository assumed to be located in your <gitroot>/soashared directory. Send a pull request to Herwin add your local changes into the shared git repository.
hmicore.sftp.publish: the hmi code ssh repository (web version located at http://hmirepo.ewi.utwente.nl/repository)
resource.sftp.publish: the resource hmi repository (web version located at http://hmirepo.ewi.utwente.nl/resource)

ant release increments the minor version number, then produces a version with "release" status. When the current version happens to be a developer version, then "incrementing the minor" actually means "stripping off" the -devi suffix.

To increment the major release number use

ant majorrelease

Releasing data as a zip file

To release a project containing only data as a zip file, set the following in ivy.xml:

<publications>
     <artifact type="zip" ext="zip"/>
</publications>

in build.properties

language=zipresource

This will zip up everything in your resource dir and release it when you use one of the release targets (e.g. release, majorrelease). When resolving such a released dataset, it will be unpacked in the lib directory. Such resource project should contain a single top-level directory and no other files in resource to keep the lib dir somewhat clean and readable (this is somewhat enforced by the resolver).

Releasing under a different user name

If you need to release under a different name than your username (e.g. at home), use

publish.user.name=yourpublishusername

in your ~/hmi.build.properties file.

More

Detailed explanation of the build system is still being written. For specific questions, ask below.

Discussion

BuildSystem
 unsolved

Reply here to give additional information or ask questions.