Reproducible releases using Ivy, Ant and CVS.

Matthias Kilian
20070827

Note to the reader: for now, this is just a slightly modified copy of an email I sent to the ivy-user mailinglist. However, I'll update it on occasion to make a small article of it. The original mail is always available at MARC.

Oh, and of course here's the link to the Ivy homepage.

If you don't like green on black, just read the plain text version of this page.

You can send any feedback and/or corrections to kili@outback.escape.de. Please use a meaningful subject and avoid HTML emails, else chances are good that your email will end up in my spam folder.

On Fri, Aug 24, 2007 at 03:03:49AM -0700, bhatia wrote:

  > How do we work in the Ivy world ? Checkout a project from CVS, compile it
  > thanks to IvyDE by retrieving dependencies, make changes to it in the local
  > workspace, then what ??? publish it with changes to the ivy repository with
  > rev="CURRENT"... ?? Same logic for the branch ? And when I am ready to
  > commit and tag on a branch, I publish the branch with rev="my-branch-tag" ?

So, since Xavier wrote that a little survey would be welcome, and since this survey may answer your questions, here's what we do. YMMV.

(Assuming development on the HEAD trunk for now, I'll describe when and how we deal with branches in a minute)

Checkout or Update the sources, make the appropriate changes, build, test, commit.
If everything looks well, trigger a release (major, minor or patchlevel) on a dedicated build machine -- we're running cruisecontrol on it, so for HEAD revisions we can use cruisecontrol's JMX interface to trigger a build with a special target. We've a couple of standard targets in some special build.xml that's imported in the build scripts of all projects: release-major, release-minor, release-patch, release-fix (the latter for hot fixes on a branch, see below). The same set exists for milestone builds, but we don't use this yet.
The release-* targets will do the following steps:
- cvs update
- Pull in the current version (major, minor, patch, fix) from a file "version.properties".
- Calculate appropriate new version numbers (increase the requested level, and set all lower levels to 0).
- Write the new version.properties.
- Run the normal build, including ivy resolution and retrieval. We also include release information (status, revision, date, build account) into the manifests of our applications.
- Make a backup copy of ivy.xml (to ivy.xml~). The important thing here is that our project's ivy files typically don't have status, rev etc. set, and also contain "latest.release" or "x.y.+" revisions in their dependencies.
- Deliver the ivy file with appropriate status (currently always "release", since we don't use milestone and integration builds yet, as noted above).
- Publish the delivered ivy file and the artifacts.
- Commit the changed ivy.xml and version.properties with a proper cvs log message (e.g. Release 0.8.15.0), and tag it (with RELEASE_0_8_15_0 in this case).
- Move ivy.xml~ to ivy.xml, to get back the undelivered ivy file.
- Commit again, this time with "Unfreeze" as the cvs commit message.

That's all for releases on the HEAD trunk. When we need a hotfix for an older revision (which happens quite often, since we're not supposed to deploy our latest and greatest features without lots of testing on dedicated QA machines), we proceed as following:

If it's the first hotfix for the revision currently running on our production systems (e.g. RELEASE_0_8_15_0), check out / update to this revision and create a new branch (BRANCH_0_8_15).
Update to the branch.
Modify, build, test, commit (on the branch). Developers must be take special care about the ivy.xml they are using here -- it's the delivered on, containing only fixed dependencies (no latest.* or x.y.+). If that's not desired, the devlopers have to change dependencies manually before committing their changes back to the branch. Actually, we had a case two weeks ago where we *only* changed dependencies for some hot fixes.
On the build machine, checkout that branch, then run ant as above with the release-fix target. This does the same as the other release-* targets.

This gives absolutely reproducible results, which is especially important when you've rather long development cycles with the need of beeing able to reproduce builds from several months ago to apply bug fixes to them. One caveat when using branches for hotfixes like we do: if you bump some dependency to a newer revision, double-check wether all other dependencies are still at the correct revisions; you may have to use force="true" on some dependencies. For example:

A-0.8.15.0 depends on B-1.2.3.0 and on C-7.4.2.8.
B-1.2.3.0 depends on C-7.4.2.8, too.
B-1.3.0.0. depends on C-8.0.0.0.
When you now update A's dependency on B to 1.3.0.0, you'll get C-8.0.0.0 when using the default conflict manager. This may or may not lead to problems.

What's still on my TODO list are the mentioned integration and milestone builds and a better integration of ivy and cruisecontrol (or any other CI-like system).

Milestone builds would be nice, since we could test them on the QA machines and then, if everything is ok, switch them from milestone to release status, using the same mechanism as above. Unfortunately, there're also some problems, since we deploy on WebSphere Application Server, but we don't put EJB and resource bindings into our builds, so this is kind of error-prone, and it's more safe to just copy the tested applications from QA to the production machines.

I'm also planning to support pure integration builds on the CI machine, so tightly coupled projects will be tested without having to make and tag a new release all time.

Oh, and one final remark: NEVER EVER ship code built only in your IDE. It's not reproducible. I've even seen projects that were not buildable on any except a single developer's machine. Even if IvyDE helps a lot, there may still be problems, like a forgotten commits, different locale settings, or just sloppy developers not caring at all about quality. So, ALWAYS use a dedicated build machine that ONLY builds, tags and publishes releases.

Ciao,

Kili