Mark Needham

Thoughts on Software Development

Archive for the ‘Build’ tag

Nant: Populating templates

without comments

One of the common tasks that we need to do on every project I’ve worked on is ensure that we can create a web.config file for the different environments that we need to deploy our application to.

Nant has quite a neat task called ‘expandproperties‘ which allows us to do this quite easily.

In our build file we would have the following:

build-file.build

  <property name ="configFile" value="${environment}.properties" readonly="true"/>
  <if test="${not file::exists(configFile)}">
    <fail message="Configuration file '${configFile}' could not be found." />
  </if>
  <include buildfile ="${configFile}" />
 
  <target name="GenerateConfigFiles">
    <foreach item="File" property="TemplateFile">
      <in>
        <items>
          <include name="ProjectDirectory/**.template"/>
        </items>
      </in>
      <do>
        <copy file="${TemplateFile}" tofile="${string::replace(TemplateFile,'.template','')}" overwrite="true">
          <filterchain>
            <expandproperties />
          </filterchain>
        </copy>
      </do>
    </foreach>
  </target>

There would be corresponding ‘.template’ and ‘.properties’ files containing the various variables we want to set, like so:

dev.properties

 <project>
	 <property name="SomeHost" value="http://my-host.com:8080"/>
 </project>

web.template.config

<?xml version="1.0"?>
...
<configuration>
   <appSettings>
      <add key="SomeHost="${SomeHost}"/>
   </appSettings>
</configuration>
...

We would call the build file like so:

nant -buildfile:build-file.build GenerateConfigFiles -D:environment=dev

We can then change that ‘environment’ variable depending which one we need to generate the configuration files for.

Written by Mark Needham

January 16th, 2010 at 12:13 am

Posted in Build

Tagged with ,

Build: Using virtual machines to run it in parallel

with 3 comments

One of the things that we’ve been working on lately to improve the overall time that our full build takes to run is to split the acceptance tests into several small groups of tests so that we can run them in parallel.

We are using Cruise as our build server so the ability to have multiple agents running against different parts of the build at the same time comes built it.

We then had to work out what the best environment to run more agents would be.

I guess the traditional approach to the problem would be to get some extra machines to use to run the build but this wasn’t possible so our initial solution was to use our developer virtual machines as build agents.

This worked in principle but the environment became way too slow to work on when the build was running and the selenium windows that popped up when each test was running became amazingly annoying. We could only really use this approach to set up agents on developer machines that weren’t being used that day.

We therefore came across the idea of having 2 virtual machines running on each developer machine, splitting the resources of the machine across the two virtual machines.

From experimenting it seems like the agent used to run a build with selenium tests in it needs to have around 1.2 GB of RAM assigned to it to run at a reasonable rate. Our developer machines have a total of 3 GB of RAM so the majority of the rest is being used by the development virtual machine.

Our development virtual machine environments run marginally slower when there is a build running on that machine but we’ve reduced that problem by ensuring we have the build running across as many developer machines as possible so that it runs less frequently on each machine

I think it serves as a worthwhile trade off for helping to reduce our total build time significantly – running sequentially it would take over an hour to run all our tests but in parallel we’ve managed to get a 10-15 minute turn around time at worst.

While trying to see if we could make that any faster we toyed with the idea of having two build virtual machines on the same box but that didn’t seem to provide the improvement that we expected due to I/O restraints – our build has quite a bit of writing and reading from disc and having two builds doing this at the same time seemed to slow down both of them.

Overall though it’s working out pretty well for us in terms of providing us a quick feedback mechanism of whether or not our system is working correctly end to end.

Written by Mark Needham

May 21st, 2009 at 6:02 pm

Posted in Build

Tagged with , ,

Nant include task – namespace matters

with 4 comments

We’ve been trying to include some properties into our build file from a properties file today but no matter what we tried the properties were not being set.

We eventually realised that the build file has an XML Namespace set on the project element.

<project name="..." xmlns="http://nant.sf.net/schemas/nant.xsd">

It turns out that if you want to include a properties file in your build file, like so:

<include buildfile="properties.xml" />

…you need to put the namespace on the project attribute of that file as well, otherwise its properties don’t get picked up.

Our properties file therefore needs to look like this, assuming that we have a namespace set on the build file.

<project name="properties" xmlns="http://nant.sf.net/schemas/nant.xsd">
	<property name="foo" value="bar" />
</project>

What’s a bit confusing is that on the NAnt documentation page for the include task it says that project element attributes are ignored! That’s not what we found!

Written by Mark Needham

February 3rd, 2009 at 10:43 am

Posted in Build

Tagged with ,

Cruise: Pipelining for fast visual feedback

without comments

One of the cool features in build servers like Cruise and Team City is the ability to create build pipelines.

I have done a bit of work using this feature in previous projects but the key driver for doing so there was to create a chain of producers/consumers (producing and consuming artifacts) eventually resulting in a manual step to put the application into a testing environment.

While this is certainly a good reason to create a build pipeline, a colleague pointed out an equally useful way of using this feature to split the build into separate steps pipelined together.

By doing this we get a nice graphical display from the cruise dashboard which allows us to see where the build is failing, therefore pointing out where we need to direct our focus.

pipeline_activity_small.png

One way to use the pipelines is to work out the distinct potential areas where you would want to signal that something needs to be investigated and then make each of these targets a separate build target.

For example we could set it up like so:

No dependency build =>
Services build =>
End to End smoke test build =>
Full build

Benefits of this approach

The benefit of this approach is that it helps to create more confidence in the build process.

When we have a long running build it is easy to get into a state where it is failing after 3/4 of the build has run and all we get is the red failed build to indicate something has gone wrong. We can drill down to find out where the failure is but it’s not as obvious.

The approach we have taken to checking in is that it is fine to do as long as the first stage of the build is green. This has worked reasonably well so far and failure further down stream has been fixed relatively quickly.

Things to watch for

We have setup the final step of the build to be a manual step due to the fact that it takes quite a long time to run and we’ve been unable to get a dedicated machine to run an agent on. Ideally we would have it running constantly on its own agent.

This isn’t run as frequently as when we had it running automatically and I guess the danger is that we are pushing problems further down stream rather than catching them early. Hopefully this issue will be solved if we can get a dedicated agent running this build.

We’re still looking for ways to improve our build process but this is what’s currently working for reasonably well for us at the moment. It would be interesting to hear what others are doing.

Written by Mark Needham

January 19th, 2009 at 9:38 pm

Posted in Build

Tagged with ,

Build: Red/Green for local build

without comments

One thing I’m learning from reading The Toyota Way is that visual indicators are a very important part of the Toyota Production System, and certainly my experience working in agile software development is that the same is true there.

We have certainly learnt this lesson with regards to continuous integration – the build is either red or green and it’s a very obvious visual indicator of the code base at any moment in time.

On my last two projects we have setup build lights which project the build status even more visually without us even having to go to the reporting page.

I have never thought about applying this principle when running the build locally but my colleague James Crisp has setup our build to light up the whole background of the command prompt window red or green depending on the status code we get back from running Nant.

James has promised to post the code we used from the command prompt to do this on his blog – I’ve tried to work out how to achieve the same effect in the Unix shell but I haven’t figured out how to do it yet. I’ve worked out how to set the background colour for the prompt but not for the whole window – if anyone knows how to do it, let me know!

The idea is fairly simple though:

  • Run the build
  • If the exit status is 1, make the background red
  • If the exit status is 0, make the background green

Written by Mark Needham

November 15th, 2008 at 8:26 am

Posted in Build

Tagged with

Build: Checkout and Go

with 5 comments

On the previous project I was working on one of the pain points we were having was around setting up developer environments such that you could get the code up and running on a machine as quickly as possible.

I would go to a newly formatted machine ready to set it up for development and run into a cascading list of dependencies I hadn’t considered.

SVN wasn’t installed, then Ruby, then we had the wrong version of Java and all the while we were wasting time when this process could have been automated.

One of the things we were working on on my previous project was how to get our project into such a state that we could just checkout the code on a machine and get it ready to run on there relatively quickly.

I haven’t yet come up with a good solution for automating all dependency checking but an approach that my colleague has been advocating on my current project is that of aiming for a ‘Checkout and Go‘ build.

The idea is fairly simple – when a developer joins the team they should be able to checkout all the code from source control, browse to the root directory and run a script called ‘go’ which will download and install everything needed to run the build on their machine.

We are using buildr, so in our case this mainly involves populating our JRuby and Ruby gem repositories. buildr takes care of downloading our Java dependencies.

Potential issues

The original Checkout and Go build script was only written for use from the Unix shell which obviously restricted its use somewhat! A couple of people on the team were running on Windows so we now have two versions of a very similar script.

There are some parts of our setup process which only need to be run the first time the developer runs the build. There is a trade off to be made with regards to whether it is better to incorporate these into the main script and write the associated logic around ensuring they don’t get run every time as opposed to leaving developers to run them once when they first checkout the code.

One interesting thing we have noticed is that although the theory is these scripts only need to be run once, there are actually new dependencies being introduced now and then which requires them to be re-run. If there is frequent change then the benefit of the Checkout and Go approach clearly increases.

Alternative approach

The alternative to the Checkout and Go approach is to have a setup where we only provide a build file which, working on the assumption that the developer will take care of the dependencies themself – these dependencies would then typically be written up on a Wiki page.

The thing I like about this approach is that it is much simpler to setup initially – there is no code to write around automating the download and setup of dependencies.

The disadvantage is that it is non deterministic – as we are relying on a human to execute a series of instructions the chance of someone doing something just slightly different is higher, therefore leading to the problem of developer machines not being setup in an identical manner.

In Conclusion

We don’t have this process perfect yet and there is a bit of a trade off with regards to making the go script really complicated to allow a completely automated process as opposed to covering maybe 90% of the setup in an automated script and then having the remaining 10% done manually or in other test scripts which a developer can run when required.

This idea has been the best one that I have seen with regards to getting developer environments setup and running quickly but it would be good to know if anyone has any better ideas around this.

Written by Mark Needham

October 19th, 2008 at 10:49 pm

Posted in Build

Tagged with

Where are we now? Where do we want to be?

without comments

Listening to Dan North speaking last week I was reminded of one of my favourite NLP[*] techniques for making improvements on projects.

The technique is the TOTE (Test, Operate, Test, Exit) and it is a technique designed to help us get from where we are now to where we want to be via short feedback loops.

On my previous project we had a situation where we needed to build and deploy our application in order to show it to the client in a show case.

The first time we did this we did most of the process manually – it took three hours and even then still didn’t work properly. It was clear that we needed to do something about this. We needed the process to be automated.

Before we did this I mentally worked out what the difference was between where we were now and what the process would look like when we were at our desired state.

A full description of the technique can be found in the NLP Workbook but in summary, these are the steps for using the TOTE technique.

  1. What is our present state?
  2. What is our desired state?
  3. What specific steps and stages do we need to go through to get there?

We then execute an action and then re-compare the current state to the desired state until they are the same. Then we exit.

In our situation:

  1. To deploy our application we need to manually build it then copy the files to the show case machine.
  2. We want this process to happen automatically each time a change is made to the application

For step 3 to make sense more context is needed.

For our application to be ready to use we needed to build it and deploy it to the user’s desktop and build and deploy several other services to an application repository so that our application could stream them onto the desktop.

The small steps for achieving step 3 were:

  • Write a build for building the application and making the artifacts available
  • Write a script to deploy the application to the user’s desktop
  • Edit the build for building the services to make artifacts available
  • Write a script to deploy these services to the repository

This evolved slightly so that we could get Team City to simulate the process for our functional testing and then run a script to deploy it on the show case machine but the idea is the same.

There’s nothing mind blowing about this approach. It’s just a way of helping us to clarify what exactly it is we want to do and providing an easy way of getting there as quickly as possible.


* Sometimes when I mention NLP people get a bit defensive as it has been fed to them previously as a tool kit for solving all problems.

We need to remember that NLP is a set of communication techniques gathered from observing effective communicators and then recorded so that others could learn from this.

When used properly ideas from NLP can help us to clarify what we want to do and improve our ability to communicate with each other.

Written by Mark Needham

September 20th, 2008 at 5:32 pm

Configurable Builds: One configuration file per machine

without comments

I’ve covered some of the ways that I’ve seen for making builds configurable in previous posts:

One which I haven’t covered which my colleagues Gil Peeters and Jim Barritt have pointed out is having a build with one configuration file for each machine.

Again the setup is fairly similar to one configuration per user or environment. Using Nant we would have the following near the top of the build file:

<property name="machine.name" value="${environment::get-machine-name()}" />
<include buildfile="${trunk.dir}\config\${machine.name}.properties.xml" />

We could then have one configuration for each developer machine:

machine1.properties.xml

<?xml version="1.0" ?>
<properties>
	<property name="property1" value="onevalue" />
</properties>

machine2.properties.xml

<?xml version="1.0" ?>
<properties>
	<property name="property1" value="anothervalue" />
</properties>

The build file can be run using the following command:

nant -buildfile:build-file.build target-name

The benefit of this approach can be seen (as Gil points out) in pair programming where the settings on any one machine will always be the same regardless of who is logged in. We also still get the advantage of being able to use remote resources on developer machines.

Having machine specific configuration also allows more flexibility for configurations on continuous integration for example. To quote Gil:

Each CI build (multiple builds per build server) get’s it’s own [configuration] based on the build host and build name.

The disadvantage again is we have to add a new configuration file every time we want to run the build on a different machine.

Written by Mark Needham

September 13th, 2008 at 3:54 am

Posted in Build

Tagged with , ,

Configurable Builds: Overriding properties

without comments

Sometimes when configuring our build for flexibility we don’t need to spend the time required to create one build configuration per user or one build configuration per environment.

In these cases we can just override properties when we call Nant from the command line.

One recent example where I made use of this was where we had one configuration file with properties in but wanted to override a couple of them when we ran the continuous integration build.

Since build properties are immutable (i.e. once they are set they can’t be changed) if we set them from the command line the build script makes use of these values.

For example we might have the following in our build file:

<property name="repository.url" value="http://localhost:3000" />

But when we’re running it on cruise control we have the repository on a different machine. We can override it like so:

nant -buildfile:build-file.build target-name -D:repository.url=http://some-remote-url

If we have more than one property we want to override it might be a bit annoying to have to pass them all via the command line. We can define them in a file to overcome this problem:

nant -buildfile:build-file.build target-name @ci.properties.xml

where ci.properties.xml contains the following:

-D:repository.url=http://remote-url:3000
-D:some.other.property=newvalue

If you start seeing a lot of properties in this file then it is probably an indicator that you need to have a more robust solution but this works for providing simple flexibility.

Written by Mark Needham

September 2nd, 2008 at 2:49 pm

Posted in Build

Tagged with , ,

Configurable Builds: One configuration file per user

with 2 comments

Following on from my first post about making builds configurable, the second way of doing this that I have seen is to have one configuration build file per user.

This approach is more useful where there are different configurations needed on each developer machine. For example, if the databases being used for development are on a remote server then each developer machine would be assigned a database with a different name.

The setup is fairly similar to configuring by environment – the main difference is that we don’t have to pass the user in as a parameter. The following would go near the top of the build file:

<property name="user" value="${environment::get-user-name()}" />
<include buildfile="${trunk.dir}\config\${user}.properties.xml" />

We can then have different configurations for two developer machines like so:

developer1.properties.xml

<?xml version="1.0" ?>
<properties>
	<property name="property1" value="onevalue" />
</properties>

developer2.properties.xml

<?xml version="1.0" ?>
<properties>
	<property name="property1" value="anothervalue" />
</properties>

We can then run the build file like this:

nant -buildfile:build-file.build target-name

The disadvantage of this approach is that every time a new developer joins the team they need to create a new configuration file with their settings in. We also need to ensure that the continuous integration build is running using an independent user account. It provides more flexibility and is easier to setup on the plus side.

My colleague Jim Barritt points out a similar technique his team is using here.

Written by Mark Needham

September 2nd, 2008 at 1:53 pm

Posted in Build

Tagged with , ,