Mark Needham

Thoughts on Software Development

Archive for the ‘Build’ Category

Sbt: Rolling with continuous/incremental compilation and Jetty

without comments

As I mentioned in an earlier post we’re using SBT on our project and one of it’s cool features is that it will listen to the source directory and then automatically recompile the code when it detects file changes.

We’ve also installed the sbt-jetty-embed plugin which allows us to create a war which has Jetty embedded so that we can keep our application containerless.

That plugin adds an action called ‘jetty’ to sbt so we (foolishly in hindsight) thought that we would be able to launch the application in triggered execution mode by making use of a ~ in front of that:

$ ./sbt
> ~jetty

Unfortunately that doesn’t do any continuous compilation which left us quite confused until we realised that RTFM might be a good idea…

What we actually needed to launch the application locally and edit code/see changes as if it’s written in a dynamic language was the following:

$ ./sbt
> jetty-run
> ~ prepare-webapp

Written by Mark Needham

June 10th, 2011 at 12:16 am

Posted in Build,Scala

Tagged with

Sbt: Zipping files without their directory structure

with one comment

We’re using SBT on our project and Pat and I have been trying to work out how to zip together some artifacts so that they’re all available from the top level of the zip file i.e. we don’t want to copy the directory structure where the files come from.

I’ve been playing around with this in the Scala REPL which we can launch with our project’s dependencies loaded with the following command:

./sbt console-project

Our original attempt to zip together the artifacts looked like this:"ops" / "deploy")), "", true, log)

But unfortunately that keeps the directory structure which isn’t what we want!

mneedham@markneedham.home ~/Projects/core$ unzip -l 
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  06-04-11 17:52   ops/
        0  06-04-11 17:52   ops/deploy/
     2534  06-03-11 17:47   ops/deploy/
 --------                   -------
     2534                   3 files

Pat figured out that what we needed to do was make use of the ## function after our path so our code would read like this:"ops" / "deploy") ##), "", true, log)

Et voila:

mneedham@markneedham.home ~/Projects/core$ unzip -l 
  Length     Date   Time    Name
 --------    ----   ----    ----
     2534  06-03-11 17:47
 --------                   -------
     2534                   1 file

The ## function is defined like so and converts a path object into a BaseDirectory:

override def ## : Path = new BaseDirectory(this)

The code in FileUtilities that generates an entry for each file in the zip file looks like this:

               def makeFileEntry(path: Path) =
                        val relativePath = path.relativePathString("/")
                        log.debug("\tAdding " + path + " as " + relativePath + " ...")
                        val e = createEntry(relativePath)
                        e setTime path.lastModified
                def addFileEntry(path: Path)
                        val file = path.asFile
                                output putNextEntry makeFileEntry(path)
                                transferAndClose(new FileInputStream(file), output, log)
                                log.warn("\tFile " + file + " does not exist.")

Line 179 is where the meta data is defined for the archive and it makes use of “relativePathString” which has been overriden by BaseDirectory to return “”:

private final class BaseDirectory(private[sbt] val path: Path) extends Path
        override def ## : Path = this
        override def toString = path.toString
        def asFile = path.asFile
        def relativePathString(separator: String) = ""
        def projectRelativePathString(separator: String) = path.projectRelativePathString(separator)
        private[sbt] def prependTo(s: String) = "." + sep + s 

Line 176 returns the file in its original location so it can still be copied into the archive.

The problem with using an identifier like ## is that it’s very difficult to Google so you end up trawling the source code for its uses or hoping that you can find the explanation for its use in the documentation!

Written by Mark Needham

June 4th, 2011 at 5:24 pm

Posted in Build,Scala

Tagged with ,

Nant: Populating templates

without comments

One of the common tasks that we need to do on every project I’ve worked on is ensure that we can create a web.config file for the different environments that we need to deploy our application to.

Nant has quite a neat task called ‘expandproperties‘ which allows us to do this quite easily.

In our build file we would have the following:

  <property name ="configFile" value="${environment}.properties" readonly="true"/>
  <if test="${not file::exists(configFile)}">
    <fail message="Configuration file '${configFile}' could not be found." />
  <include buildfile ="${configFile}" />
  <target name="GenerateConfigFiles">
    <foreach item="File" property="TemplateFile">
          <include name="ProjectDirectory/**.template"/>
        <copy file="${TemplateFile}" tofile="${string::replace(TemplateFile,'.template','')}" overwrite="true">
            <expandproperties />

There would be corresponding ‘.template’ and ‘.properties’ files containing the various variables we want to set, like so:

	 <property name="SomeHost" value=""/>


<?xml version="1.0"?>
      <add key="SomeHost="${SomeHost}"/>

We would call the build file like so:

nant GenerateConfigFiles -D:environment=dev

We can then change that ‘environment’ variable depending which one we need to generate the configuration files for.

Written by Mark Needham

January 16th, 2010 at 12:13 am

Posted in Build

Tagged with ,

Cruise Agents: Reducing ‘random’ build failures

with 3 comments

As I mentioned previously we’re making use of multiple cruise agents in our build to allow us to run our acceptance tests in parallel, therefore allowing a build which would be nearly 2 hours if run in sequence to be completed in around 10 minutes.

Early on with this approach we were getting a lot of failures in our builds which weren’t directly related to the code being changed and were more to do with the various dependencies we were making use of.

We needed these dependencies but we were only finding out if they were actually setup correctly much later on instead of doing an environment check first.

One example of this was the Shibboleth daemon which needed to be started in order for our tests to be able to login to the single sign on system.

We have it running as a Windows service and despite the fact we had it setup to ‘auto start’ whenever the operating system was running we often had failures where it had just failed to start with no trace in any of the error logs as to why that was the case.

The way we have currently got around this problem is by writing a custom nant task which checks if the service is running and if not then starts it up by making use of the ‘ServiceController‘ class.

[TaskName ("startShibboleth")]
public class StartShibbolethDaemon : Task
	protected override void ExecuteTask() 
		var shibbolethDaemon = new ServiceController("shibbolethServiceName");
		if(shibbolethDaemon.Status == ServiceControllerStatus.Running) 
			Project.Log(Level.Info, "Shibboleth already running");
			Project.Log(Level.Info, "Shibboleth started");

We can then reference that task in our build before we run any tests which rely on it. We decided to write a custom task instead of using the built in servicecontroller task so that we could record whether or not it was already running and I couldn’t see a way to do that with the built in task.

Another quite common trend of build failures came about when our tests tried to connect to selenium server and were unable to do so because it wasn’t currently running – we have a batch file to manually start it up on our agents as we are running the build from a Windows service which means we can’t directly start processes from the file system the way we can when we run the build locally.

The original way we got around this problem was to add the selenium server startup script to the ‘StartUp’ folder of the ‘Administrator’ user on each of the cruise agents.

While this works fine when a user is actually logged into the virtual machine that hosts the agent we noticed that quite often an agent would register itself with the cruise server but we had forgotten to login on that virtual machine and any builds assigned to it would fail.

My colleague came up with quite a neat way of getting around this problem by automatically logging in the cruise agents which can be done by adding the following registry entry by putting the following in a file with the extension ‘reg’ and then double clicking it.

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon]

I think the key for us here was ensuring that the build was pretty much self contained and able to get itself into a working state if its dependencies aren’t setup correctly.

We spent a bit of time tracking the ‘random’ failures and working out which ones were addressable and which ones we needed to gather more information on. We certainly have some work to do in this area but we’ve managed to sort out some of the persistent offenders!

It’s really easy to just ignore the supposedly random failures and just re-run the build but it doesn’t really solve the underlying problem. We’ve found it’s quite rare that there are ever actually random failures in the build, just failures that we don’t know enough about yet.

Written by Mark Needham

July 25th, 2009 at 11:28 am

Posted in Build

Tagged with

Build: Using virtual machines to run it in parallel

with 3 comments

One of the things that we’ve been working on lately to improve the overall time that our full build takes to run is to split the acceptance tests into several small groups of tests so that we can run them in parallel.

We are using Cruise as our build server so the ability to have multiple agents running against different parts of the build at the same time comes built it.

We then had to work out what the best environment to run more agents would be.

I guess the traditional approach to the problem would be to get some extra machines to use to run the build but this wasn’t possible so our initial solution was to use our developer virtual machines as build agents.

This worked in principle but the environment became way too slow to work on when the build was running and the selenium windows that popped up when each test was running became amazingly annoying. We could only really use this approach to set up agents on developer machines that weren’t being used that day.

We therefore came across the idea of having 2 virtual machines running on each developer machine, splitting the resources of the machine across the two virtual machines.

From experimenting it seems like the agent used to run a build with selenium tests in it needs to have around 1.2 GB of RAM assigned to it to run at a reasonable rate. Our developer machines have a total of 3 GB of RAM so the majority of the rest is being used by the development virtual machine.

Our development virtual machine environments run marginally slower when there is a build running on that machine but we’ve reduced that problem by ensuring we have the build running across as many developer machines as possible so that it runs less frequently on each machine

I think it serves as a worthwhile trade off for helping to reduce our total build time significantly – running sequentially it would take over an hour to run all our tests but in parallel we’ve managed to get a 10-15 minute turn around time at worst.

While trying to see if we could make that any faster we toyed with the idea of having two build virtual machines on the same box but that didn’t seem to provide the improvement that we expected due to I/O restraints – our build has quite a bit of writing and reading from disc and having two builds doing this at the same time seemed to slow down both of them.

Overall though it’s working out pretty well for us in terms of providing us a quick feedback mechanism of whether or not our system is working correctly end to end.

Written by Mark Needham

May 21st, 2009 at 6:02 pm

Posted in Build

Tagged with , ,

Nant include task – namespace matters

with 4 comments

We’ve been trying to include some properties into our build file from a properties file today but no matter what we tried the properties were not being set.

We eventually realised that the build file has an XML Namespace set on the project element.

<project name="..." xmlns="">

It turns out that if you want to include a properties file in your build file, like so:

<include buildfile="properties.xml" />

…you need to put the namespace on the project attribute of that file as well, otherwise its properties don’t get picked up.

Our properties file therefore needs to look like this, assuming that we have a namespace set on the build file.

<project name="properties" xmlns="">
	<property name="foo" value="bar" />

What’s a bit confusing is that on the NAnt documentation page for the include task it says that project element attributes are ignored! That’s not what we found!

Written by Mark Needham

February 3rd, 2009 at 10:43 am

Posted in Build

Tagged with ,

Cruise: Pipelining for fast visual feedback

without comments

One of the cool features in build servers like Cruise and Team City is the ability to create build pipelines.

I have done a bit of work using this feature in previous projects but the key driver for doing so there was to create a chain of producers/consumers (producing and consuming artifacts) eventually resulting in a manual step to put the application into a testing environment.

While this is certainly a good reason to create a build pipeline, a colleague pointed out an equally useful way of using this feature to split the build into separate steps pipelined together.

By doing this we get a nice graphical display from the cruise dashboard which allows us to see where the build is failing, therefore pointing out where we need to direct our focus.


One way to use the pipelines is to work out the distinct potential areas where you would want to signal that something needs to be investigated and then make each of these targets a separate build target.

For example we could set it up like so:

No dependency build =>
Services build =>
End to End smoke test build =>
Full build

Benefits of this approach

The benefit of this approach is that it helps to create more confidence in the build process.

When we have a long running build it is easy to get into a state where it is failing after 3/4 of the build has run and all we get is the red failed build to indicate something has gone wrong. We can drill down to find out where the failure is but it’s not as obvious.

The approach we have taken to checking in is that it is fine to do as long as the first stage of the build is green. This has worked reasonably well so far and failure further down stream has been fixed relatively quickly.

Things to watch for

We have setup the final step of the build to be a manual step due to the fact that it takes quite a long time to run and we’ve been unable to get a dedicated machine to run an agent on. Ideally we would have it running constantly on its own agent.

This isn’t run as frequently as when we had it running automatically and I guess the danger is that we are pushing problems further down stream rather than catching them early. Hopefully this issue will be solved if we can get a dedicated agent running this build.

We’re still looking for ways to improve our build process but this is what’s currently working for reasonably well for us at the moment. It would be interesting to hear what others are doing.

Written by Mark Needham

January 19th, 2009 at 9:38 pm

Posted in Build

Tagged with ,

Build: Red/Green for local build

without comments

One thing I’m learning from reading The Toyota Way is that visual indicators are a very important part of the Toyota Production System, and certainly my experience working in agile software development is that the same is true there.

We have certainly learnt this lesson with regards to continuous integration – the build is either red or green and it’s a very obvious visual indicator of the code base at any moment in time.

On my last two projects we have setup build lights which project the build status even more visually without us even having to go to the reporting page.

I have never thought about applying this principle when running the build locally but my colleague James Crisp has setup our build to light up the whole background of the command prompt window red or green depending on the status code we get back from running Nant.

James has promised to post the code we used from the command prompt to do this on his blog – I’ve tried to work out how to achieve the same effect in the Unix shell but I haven’t figured out how to do it yet. I’ve worked out how to set the background colour for the prompt but not for the whole window – if anyone knows how to do it, let me know!

The idea is fairly simple though:

  • Run the build
  • If the exit status is 1, make the background red
  • If the exit status is 0, make the background green

Written by Mark Needham

November 15th, 2008 at 8:26 am

Posted in Build

Tagged with

buildr – using another project’s dependencies

with one comment

Through my continued use of buildr on my current project one thing we wanted to do last week was to run our production code tests using some code from the test-utilities project along with its dependencies.

I thought this would be the default behaviour but it wasn’t. Looking at the documentation suggested we could achieve this by calling ‘compile.dependencies’ on the project, but from what I can tell you still need to explicitly state that you want to use the main test utilities code as well.

The following code in our buildfile does the job:

DEPENDENCY_JAR='depedency:dependency:jar:1.0' # change this to whatever the path to the dependency is
define "test.utilities" do
  compile.with DEPENDENCY_JAR
define "main.code" do
  # Some other code
  test.with project("test.utilities"), project("test.utilities").compile.dependencies

It seems a bit verbose but it achieves our objective in a cleaner way than having to repeat test-utilities dependencies when we run main.code’s tests.

I’m sure there must be an even cleaner way than this but I’m not yet aware of it!

Written by Mark Needham

October 26th, 2008 at 8:54 pm

Posted in Build

Tagged with

Build: Checkout and Go

with 5 comments

On the previous project I was working on one of the pain points we were having was around setting up developer environments such that you could get the code up and running on a machine as quickly as possible.

I would go to a newly formatted machine ready to set it up for development and run into a cascading list of dependencies I hadn’t considered.

SVN wasn’t installed, then Ruby, then we had the wrong version of Java and all the while we were wasting time when this process could have been automated.

One of the things we were working on on my previous project was how to get our project into such a state that we could just checkout the code on a machine and get it ready to run on there relatively quickly.

I haven’t yet come up with a good solution for automating all dependency checking but an approach that my colleague has been advocating on my current project is that of aiming for a ‘Checkout and Go‘ build.

The idea is fairly simple – when a developer joins the team they should be able to checkout all the code from source control, browse to the root directory and run a script called ‘go’ which will download and install everything needed to run the build on their machine.

We are using buildr, so in our case this mainly involves populating our JRuby and Ruby gem repositories. buildr takes care of downloading our Java dependencies.

Potential issues

The original Checkout and Go build script was only written for use from the Unix shell which obviously restricted its use somewhat! A couple of people on the team were running on Windows so we now have two versions of a very similar script.

There are some parts of our setup process which only need to be run the first time the developer runs the build. There is a trade off to be made with regards to whether it is better to incorporate these into the main script and write the associated logic around ensuring they don’t get run every time as opposed to leaving developers to run them once when they first checkout the code.

One interesting thing we have noticed is that although the theory is these scripts only need to be run once, there are actually new dependencies being introduced now and then which requires them to be re-run. If there is frequent change then the benefit of the Checkout and Go approach clearly increases.

Alternative approach

The alternative to the Checkout and Go approach is to have a setup where we only provide a build file which, working on the assumption that the developer will take care of the dependencies themself – these dependencies would then typically be written up on a Wiki page.

The thing I like about this approach is that it is much simpler to setup initially – there is no code to write around automating the download and setup of dependencies.

The disadvantage is that it is non deterministic – as we are relying on a human to execute a series of instructions the chance of someone doing something just slightly different is higher, therefore leading to the problem of developer machines not being setup in an identical manner.

In Conclusion

We don’t have this process perfect yet and there is a bit of a trade off with regards to making the go script really complicated to allow a completely automated process as opposed to covering maybe 90% of the setup in an automated script and then having the remaining 10% done manually or in other test scripts which a developer can run when required.

This idea has been the best one that I have seen with regards to getting developer environments setup and running quickly but it would be good to know if anyone has any better ideas around this.

Written by Mark Needham

October 19th, 2008 at 10:49 pm

Posted in Build

Tagged with