Mark Needham

Thoughts on Software Development

Where are we now? Where do we want to be?

Listening to Dan North speaking last week I was reminded of one of my favourite NLP[*] techniques for making improvements on projects.

The technique is the TOTE (Test, Operate, Test, Exit) and it is a technique designed to help us get from where we are now to where we want to be via short feedback loops.

On my previous project we had a situation where we needed to build and deploy our application in order to show it to the client in a show case.

The first time we did this we did most of the process manually – it took three hours and even then still didn’t work properly. It was clear that we needed to do something about this. We needed the process to be automated.

Before we did this I mentally worked out what the difference was between where we were now and what the process would look like when we were at our desired state.

A full description of the technique can be found in the NLP Workbook but in summary, these are the steps for using the TOTE technique.

1. What is our present state?
2. What is our desired state?
3. What specific steps and stages do we need to go through to get there?

We then execute an action and then re-compare the current state to the desired state until they are the same. Then we exit.

In our situation:

1. To deploy our application we need to manually build it then copy the files to the show case machine.
2. We want this process to happen automatically each time a change is made to the application

For step 3 to make sense more context is needed.

For our application to be ready to use we needed to build it and deploy it to the user’s desktop and build and deploy several other services to an application repository so that our application could stream them onto the desktop.

The small steps for achieving step 3 were:

• Write a build for building the application and making the artifacts available
• Write a script to deploy the application to the user’s desktop
• Edit the build for building the services to make artifacts available
• Write a script to deploy these services to the repository

This evolved slightly so that we could get Team City to simulate the process for our functional testing and then run a script to deploy it on the show case machine but the idea is the same.

There’s nothing mind blowing about this approach. It’s just a way of helping us to clarify what exactly it is we want to do and providing an easy way of getting there as quickly as possible.

* Sometimes when I mention NLP people get a bit defensive as it has been fed to them previously as a tool kit for solving all problems.

We need to remember that NLP is a set of communication techniques gathered from observing effective communicators and then recorded so that others could learn from this.

When used properly ideas from NLP can help us to clarify what we want to do and improve our ability to communicate with each other.

Written by Mark Needham

September 20th, 2008 at 5:32 pm

Configurable Builds: One configuration file per machine

I’ve covered some of the ways that I’ve seen for making builds configurable in previous posts:

One which I haven’t covered which my colleagues Gil Peeters and Jim Barritt have pointed out is having a build with one configuration file for each machine.

Again the setup is fairly similar to one configuration per user or environment. Using Nant we would have the following near the top of the build file:

<property name="machine.name" value="${environment::get-machine-name()}" /> <include buildfile="${trunk.dir}\config\${machine.name}.properties.xml" /> We could then have one configuration for each developer machine: machine1.properties.xml <?xml version="1.0" ?> <properties> <property name="property1" value="onevalue" /> </properties> machine2.properties.xml <?xml version="1.0" ?> <properties> <property name="property1" value="anothervalue" /> </properties> The build file can be run using the following command: nant -buildfile:build-file.build target-name The benefit of this approach can be seen (as Gil points out) in pair programming where the settings on any one machine will always be the same regardless of who is logged in. We also still get the advantage of being able to use remote resources on developer machines. Having machine specific configuration also allows more flexibility for configurations on continuous integration for example. To quote Gil: Each CI build (multiple builds per build server) get’s it’s own [configuration] based on the build host and build name. The disadvantage again is we have to add a new configuration file every time we want to run the build on a different machine. Written by Mark Needham September 13th, 2008 at 3:54 am Posted in Build Tagged with , , Configurable Builds: Overriding properties without comments Sometimes when configuring our build for flexibility we don’t need to spend the time required to create one build configuration per user or one build configuration per environment. In these cases we can just override properties when we call Nant from the command line. One recent example where I made use of this was where we had one configuration file with properties in but wanted to override a couple of them when we ran the continuous integration build. Since build properties are immutable (i.e. once they are set they can’t be changed) if we set them from the command line the build script makes use of these values. For example we might have the following in our build file: <property name="repository.url" value="http://localhost:3000" /> But when we’re running it on cruise control we have the repository on a different machine. We can override it like so: nant -buildfile:build-file.build target-name -D:repository.url=http://some-remote-url If we have more than one property we want to override it might be a bit annoying to have to pass them all via the command line. We can define them in a file to overcome this problem: nant -buildfile:build-file.build target-name @ci.properties.xml where ci.properties.xml contains the following: -D:repository.url=http://remote-url:3000 -D:some.other.property=newvalue If you start seeing a lot of properties in this file then it is probably an indicator that you need to have a more robust solution but this works for providing simple flexibility. Written by Mark Needham September 2nd, 2008 at 2:49 pm Posted in Build Tagged with , , Configurable Builds: One configuration file per user with 2 comments Following on from my first post about making builds configurable, the second way of doing this that I have seen is to have one configuration build file per user. This approach is more useful where there are different configurations needed on each developer machine. For example, if the databases being used for development are on a remote server then each developer machine would be assigned a database with a different name. The setup is fairly similar to configuring by environment – the main difference is that we don’t have to pass the user in as a parameter. The following would go near the top of the build file: <property name="user" value="${environment::get-user-name()}" /> <include buildfile="${trunk.dir}\config\${user}.properties.xml" />

We can then have different configurations for two developer machines like so:

developer1.properties.xml

<?xml version="1.0" ?> <properties> <property name="property1" value="onevalue" /> </properties>

developer2.properties.xml

<?xml version="1.0" ?> <properties> <property name="property1" value="anothervalue" /> </properties>

We can then run the build file like this:

nant -buildfile:build-file.build target-name

The disadvantage of this approach is that every time a new developer joins the team they need to create a new configuration file with their settings in. We also need to ensure that the continuous integration build is running using an independent user account. It provides more flexibility and is easier to setup on the plus side.

My colleague Jim Barritt points out a similar technique his team is using here.

Written by Mark Needham

September 2nd, 2008 at 1:53 pm

Posted in Build

Tagged with , ,

Configurable Builds: One configuration file per environment

One of the most important things when coding build files is to try and make them as configurable as possible.

At the very least on an agile project there will be a need for two different configurations – one for developer machines and one for continuous integration.

On my last two .NET projects we have setup our Nant build to take in a parameter which indicates which build configuration should be used. We then have a configuration file by that name which contains the environment specific data.

The build file would contain the following code before anything else:

<fail unless="${property::exists('environment')}" message="You must provide the environment property to the build script using -D:environment=[dev|ci]" /> <include buildfile="${trunk.dir}\config\${environment}.properties.xml" /> dev.properties.xml would look like this: <?xml version="1.0" ?> <properties> <property name="property1" value="value1" /> <property name="property2" value="value2" /> </properties> We would call the build file for the dev environment like so: nant -buildfile:build-file.build target-name -D:environment=dev Configuring the build this way assumes that the dev builds all have the same properties. On the projects where I used this approach this was the case. The disadvantage of it is that you need to remember to pass in the environment variable each time you call the build. This can be countered by wrapping the full nant call in a batch script if it becomes too much of a hassle. Written by Mark Needham September 2nd, 2008 at 1:50 am Posted in Build Tagged with , , scp Nant Task – ‘scp’ failed to start. The system cannot find the file specified without comments I was trying to make use of the Nant Contrib scp task earlier and was getting an error message which at the time seemed a bit strange (now of course having solve the problem it is obvious!) This was the task I was running: <scp file="someFile.txt" server="some.secure-server.com" /> This was the error: 1 2 'scp' failed to start. The system cannot find the file specified I ran it in debug mode to try and see what was going on and got this stack trace: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 NAnt.Core.BuildException: C:\projects\project1\default.build(18,4): 'scp' failed to start. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) at System.Diagnostics.Process.Start() at NAnt.Core.Tasks.ExternalProgramBase.StartProcess() --- End of inner exception stack trace --- at NAnt.Core.Tasks.ExternalProgramBase.StartProcess() at NAnt.Core.Tasks.ExternalProgramBase.ExecuteTask() at NAnt.Contrib.Tasks.ScpTask.ExecuteTask() at NAnt.Core.Task.Execute() at NAnt.Core.Target.Execute() at NAnt.Core.Project.Execute(String targetName, Boolean forceDependencies) at NAnt.Core.Project.Execute() at NAnt.Core.Project.Run() Eventually a colleague and I realised that the scp task was assuming that there was an executable called ‘scp’ on the path. This can be overriden by setting the ‘program’ attribute to whatever you want. In this case since we were running from Windows we downloaded Putty’s pscp executable and put that on the Windows path. The code to call the scp task now looks like this: <scp file="someFile.txt" server="some.secure-server.com" program="pscp" /> If you don’t want to put it on the path then the following works just as well: <scp file="someFile.txt" server="some.secure-server.com" program="c:\path\to\pscp.exe" /> Written by Mark Needham August 30th, 2008 at 4:30 pm Posted in Build Tagged with , , NCover Nant Team City Integration with 2 comments I’ve been spending quite a bit of time setting up NCover and then integrating it into Team City. I’ve read some posts which cover parts of this process but nothing which covers the end to end process so hopefully my experience can help to fill that void. Step 1 Download NCover 1.5.8, NCover Explorer 1.4.0.7, NCover Explorer Extras 1.4.0.5 from Kiwidude’s website and the NCover website . Step 2 Put the following into your Nant build file: 1 2 <loadtasks assembly="..\lib\NCoverExplorer.Extras\NCoverExplorer.NAntTasks.dll"/> <exec program="regsvr32" workingdir="..\lib\NCover-1.5.8" commandline="/s coverlib.dll"/> I put this right at the top of the build but I expect it doesn’t matter where it goes as long as it’s called at some stage before NCover and NCover Explorer are called. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 <macrodef name="cover.tests"> <attributes> <attribute name="in.assemblies" /> </attributes> <sequential> <ncover program="..\lib\NCover-1.5.8\NCover.Console.exe" commandLineExe="..\lib\nunit-2.4\nunit-console.exe" commandLineArgs="${build.dir}\UnitTests\UnitTests.dll" coverageFile="${report.dir}\Unit.Test.Coverage.xml" assemblyList="${in.assemblies}" />   <ncoverexplorer program="..\lib\NCoverExplorer\NCoverExplorer.Console.exe" projectName="Project" reportType="ModuleClassSummary" outputDir="${report.dir}" xmlReportName="TestCoverage.xml" htmlReportName="TestCoverage.html" showExcluded="True" satisfactoryCoverage="80" > <fileset> <include name="${report.dir}\Unit.Test.Coverage.xml" /> </fileset> <exclusions> <exclusion type="Assembly" pattern="*.Tests" /> <exclusion type="Namespace" pattern="*.Tests*" /> </exclusions> </ncoverexplorer> </sequential> </macrodef>

This macro can then be called as follows:

1 2 3 <target name="cover.unit.tests" <cover.tests in.assemblies="Project1;Project1" /> </target>

N.B. The projects passed in as the ‘in.assemblies’ argument should be semi colon separated.

Step 3

The next step is to setup the artifacts for your project. From the Team City admin panel navigate to the project configuration settings and select artifacts.

Add the following to the ‘Artifact paths’:

1 TestCoverage.html

It should now show up as a viewable artifact from the project listing page.

Step 4

To get the coverage report to show up on a tab on the build summary page we need to edit the main-config.xml file

The location of this file can be found by browsing to ‘Administration > Server Configuration’ from the Team City admin panel

Add the following line after the other ‘report-tab’ entries in this file:




Potential Problems

I encountered some problems in getting this up and running. They were as follows:

NCover: Profiled process terminated. Profiler connection not established

After some Googling I found this post which explains how to solve the problem.

To summarise this problem occurs when trying to run NCover without Administrative privileges. The coverlib.dll shipped with NCover needs to be registered. This can be done two ways:

1) Put the following code into your build file right at the top

1 <exec program="regsvr32" workingdir="\path\to\ncover" commandline="/s coverlib.dll"/>

2) Run the same command from the command line

1 C:\path\to\NCover-1.5.8>regsvr32 CoverLib.dll

This error occurred when I was using version 1.0.1 of NCover and to cut a long story short, you need to upgrade to get rid of the problem.

More details are on this post.

The information here has been accumulated from my experiences, this post on NCover integration and the official documentation.

Written by Mark Needham

August 25th, 2008 at 9:29 pm

Posted in Build

Tagged with , , ,

Encapsulation in build scripts using nant

When writing build scripts it’s very easy for it to descend into complete Xml hell when you’re using a tool like nant.

I wondered previously whether it was possible to TDD build files and while this is difficult given the dependency model most build tools follow. That doesn’t mean we can’t apply other good design principles from the coding world however.

Encapsulation is one of the key principles of OOP and it can be applied in build files too. Stephen Chu talks about this in his post on Pragmatic Nant Scripting where he recommends having 3 different levels of targets to help create this encapsulation.

I’ve been trying to follow this advice with our build scripts and today Bernardo made the suggestion of using macros in an English readable way. He calls it OO Scripting – it’s effectively a DSL inside a DSL if you like.

I was having problems with the ncover nant task – the following error message was being thrown every time I called it:

I managed to find the source code for that class and had a look at it but I couldn’t figure out what was going wrong without debugging through it. The strange thing was that it worked fine from the command line which suggested to me that I was getting something simple wrong.

I created a cover.tests macro to encapsulate the details of how I was executing the coverage.

The plan was to get it working using an exec call to the ncover executable and then phase the ncover nant task back in when I’d figured out what I was doing wrong.

This is what I started out with:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 <macrodef name="cover.tests"> <attributes> <attribute name="in.assemblies" /> </attributes> <sequential> <copy file="\path\to\Coverage.xsl" tofile="${report.dir}\Coverage.xsl" /> <exec program="..\lib\NCover-1.5.8\NCover.Console.exe"> <arg value="..\lib\nunit-2.4\nunit-console.exe" /> <arg value="${build.dir}\UnitTests\UnitTests.dll" /> <arg value="//a" /> <arg value="${in.assemblies}" /> <arg value="//x" /> <arg value="${report.dir}\Unit.Test.Coverage.xml" /> </exec> </sequential> </macrodef>

//a is the assemblies to include in the report

//x is the name of the report xml file which will be created

The full list is here.

The macro was called like this:

 1 2 3   

I substituted the ncover task back in with the same parameters as above and low and behold it worked!

 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16   

I’m not sure exactly what the problem parameter was but encapsulating this part of the build gave me the option of working that out in a way that impacted very little of the rest of the build file.

*Update*
Fixed the first example to include the opening as pointed out by Vikram in the comments. Thanks again Vikram for pointing that out!

Written by Mark Needham

August 21st, 2008 at 12:40 am

Posted in Build

Tagged with , , , ,

Building in release mode with no pdbs with msbuild

I’ve been having trouble trying to work out how to build our projects in msbuild in release mode without creating the customary pdb files that seem to be created by default.

I tried calling msbuild.exe with the ‘Release’ configuration:

'C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.Exe ( Proj.csproj /p:OutputPath=\output\path\ /p:Configuration=Release)'

To no avail. It still created the pdb file. Next I tried setting the ‘DebugSymbols’ property to false:

'C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.Exe ( Proj.csproj /p:OutputPath=\output\path\ /p:Configuration=Release /p:DebugSymbols=false)'

Still it created the file. Finally I found this post which suggested that you actually needed to make the change in the Proj.csproj file itself.

I changed this part of the file so that DebugType is now ‘none’. It had a value of ‘pdbonly’ when I opened the file.

 none true bin\Release\ TRACE prompt 4

The pdb is no longer created.

*Update*
This can also be done by passing /p:DebugType=none as a command line argument as Tim points out in the comments.

Written by Mark Needham

August 20th, 2008 at 6:50 pm

Posted in .NET,Build

Tagged with , , , ,

with one comment

I’ve been trying to integrate NCover into our build and probably making life harder for myself than it needs to be.

The title refers to the error message that I was getting when trying to run the ncover nant task on version 1.0.1 of NCover earlier today.

 1 2 3 4 5 6 7 8  [ncover] Starting 'C:\Program Files\NCover\ncover-console.exe (//r "\long\path\to\tmp392.tmp.ncoversettings" )' in 'C:\my-project\trunk\src' [ncover] Unhandled Exception: System.ArgumentException: Requested value '/r' was not found. [ncover] at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase) [ncover] at NCover.Utilities.Arguments.ParseArgument(String arg, CommandLineArgument& key, String& value) in C:\to ols\eclipse3M6\workspace\ncover\src\NCover\Utilities\Arguments.cs:line 192 ...

After some inspired Googling my colleague managed to work out that the problem was that you can’t pass a settings file path which has spaces in to the ncover executable, hence the error message. It’s the same problem in handling spaces that I mentioned in an earlier post on msbuild.

The advice on the forum was to upgrade to one of the more recent versions where the bug has been fixed. Downloads of the free version of NCover (it becomes paid for at version 2.0) are available here.

Written by Mark Needham

August 19th, 2008 at 9:18 pm

Posted in Build

Tagged with , , ,