wiki:BuildPhilosophy

Build philosophy

It is often necessary to (re)build projects, even by persons who didn’t develop that project. Since many different tools are being used, we have to agree on a number of basic conventions and procedures.

  • We do not assume that everyone will be using the IDE or any IDE at all. But we do require that a project created and developed, say, by means of Netbeans or Eclipse, can be (re)build without any of these tools available.
  • Since we have to use some common build tool, we have agreed to use  ant as that “common” minimal platform.
  • In principle every project lives in its own directory, containing all relevant source code, test code, project specific data, library files needed by that project, project documentation, etcetera. We have a preferred standard layout for the directory structure inside a project. A project also contains an ant buildfile (build.xml) and usually also a build.properties file. We require that we can (re)build and run a project using ant, without any reliance on development tools that might have been used to develop the project. To be clear: we don’t want to install JBuilder or Netbeans or Eclipse or whatever just to build and run your project. The ant file can be either a simple “standalone” build file, but the preferred way is to use a very small build file that just links to our shared build file (see layout)
  • We have a limited number of shared projects, all of which are available from our GIT repositories:
    • There is a “project” called hmibuild. It contains the shared ant build files. You must have this project in order to use our build system.
    • Shared software is available as source code or as compiled jar files. Most projects use the precompiled library (e.g. jar) files, which are kept in a project’s lib directory. (For tools like Eclipse or Netbeans you must do some configuring in order to use these library files, see below). We use a tool called  ivy, used by our build files, for easy version management of lib files that relies on our web repositories (hmirepo.ewi.utwente.nl, asap-project.org/repo). Basically, when you type “ant resolve”, then ivy will copy the library files needed for your project into the lib directory of your project. What will be copied is derived from a project file called “ivy.xml”.
    • There are a few “projects”, like HmiResource, that contains just “resource” data of all sorts, that is shared between projects. For instance, BML scripts, data for 3D scenes and avatars etcetera lives here. Usually, you can obtain such data also from the web repository, in packaged jar format. Sometimes, you want to actually see and modify that data, and in that case you will need to check out the relevant resource data from the git repositories.
    • Projects import and export class code and data in the form of jar or zip files. Whenever viable, there is no sharing of source code (one exception is C++ code in Linux, which typically has to be recompiled on each system). This ensures that every project can be built stand alone, after importing the necessary library files.

Why dependency management

The contents of lib directories consists of jar les and/or \dll" or \so" les that are necessary for compiling and running the project. The basic strategy is that inter-dependencies between projects are via import/export of library jar files, in preference over direct source code dependencies. Of course we have to face the problem of project versioning. We have stable release versions of projects, but also less stable daily releases and alpha versions. Here we have some conventions and rules. For instance, we do not want a stable release version of project X to be dependent on an unstable alpha version of project Y. The other way around, so an alpha version of X dependent on a release version of Y is OK of course. It will be clear that manual version management is not a good idea: lots of work, and error prone. Instead, we use a dependency manager,  Ivy.

How does it work?

Every project has an "ivy.xml" file that describes project dependencies. The ant build files, relying on the Ivy system, use these so called "ivy's" for resolving the contents of the project's lib directory. The system is based upon the notion of configurations, versions, and status of versions. For example, our build system uses "alpha", "daily build", and "release" as possible status of some jar version. The ordering of these statuses is relevant. For instance, when you ask for a "latest beta" version of some module Ivy will choose the latest amongst published daily releases and release (but not alpha) versions of that module. Similarly, if you ask for a "latest alpha", anything is acceptable, but if you ask for a "latest release", then only versions classified as "release" are taken into consideration. Ivy allows for several "configurations" of the project, each with its own set of dependencies. Currently, we have configurations for producing alpha, daily build, and release versions of the project itself, and a "test" configuration for (extra) dependencies needed for running tests. (For technical reasons we have two more configurations, called "master" and "default", which are discussed below) The workflow is roughly as follows: first move the project into either the alpha, beta or release configuration, second do your development, including testing etc, third publish an alpha, beta or normal release based upon the current configuration. When you actually publish, we attach a version number to the jar file. Also, the Ivy system records metadata concerning published modules based upon version numbers, to be used later on, when resolving for other projects. It does so by publishing not just a "jar" file (or other "artifacts", as they are called by Ivy), but also an accompanying ivy.xml file, derived from the project's ivy at the moment of publication. In this way, the Ivy system knows not only about direct dependencies, but is also capable of resolving recursive dependencies. This means that, for instance, when my module X declares (just) a dependency on the HmiGraphics package, the resolve process will look into the dependencies of HmiGraphics. The result is that project X will receive jar files for HmiGraphics, but also for HmiAnimation, HmiXml, HmiMath, and HmiUtil, because of (recursive) dependencies. When some jar file is required more than once, say via a direct dependency and also via some indirect dependency, and the versions required are not consistent, then Ivy delivers the "latest" version. So, for instance, in the example above, if project X declares (direct) dependencies on the alpha version of HmiGraphics and the release version of HmiXml, while the alpha version of HmiGraphics declares itself a dependency on the beta version of HmiXml, then project X will receive that beta version of HmiXml. That should be ok: since we are asking for some alpha version (of HmiGraphics, we should allow other alpha and/or beta versions if HmiGraphics actually needs them, even if our own project would be satisfied with the most recent release version.