Mark Needham

I had my first opportunity to participate in release plan estimation over the last couple of weeks. I’ve done estimation before but never at such a high level.

When doing this it appeared clear that there were two situations that we were trying to avoid:

Under estimating

Under estimating is where we predict that the amount of time taken to complete a piece of work will be less than it actually is.

This can happen for a variety of reasons, including:

• We make some invalid assumptions – e.g. we assume integration with a 3rd party will be easy and then encounter unforeseen problems when we try to do it.
• Not considering all the edge cases – at the time of estimation we may not have all the acceptance criteria which could mean that some scenarios are not considered when estimating.

If we do end up underestimating the size of work problems will eventually manifest. Although the business may be happy with you to start with, eventually either they are going to be very unhappy when these aren’t met. The most likely outcome then is that the team is going to end up on a death march to meet the original estimates, probably resulting in resignations after the project reaches completion.

The key to handling the situation where we realise we have under estimated a piece of work is to inform the business about the situation as quickly as possible. Waiting until the delivery date before saying anything is pointless.

Over estimating

This is where we predict that the amount of time taken to complete a piece of work with be greater than it actually is.

Unlike under estimating the effects of over estimating are likely to be felt immediately.

For example, you may end up with the business being unable to afford to do the project if the estimates are too far removed from what they are expecting. As I mentioned earlier this up front transparency may in fact be in the best interests of the client even if it doesn’t seem it at the time.

Alternatively estimates may be challenged and pressure applied for them to be reduced. If they were actually accurate in the first place and the over estimation is only a perception then we may now have the problem that the new estimates are in fact under estimated.

If we reach the end of the piece of work and realise that we have finished earlier than expected due to over estimating then it may actually prove to be a good thing. We can then go and add more testing around our code, refactor code to make it more maintainable or whatever else the business sees as a valuable use of this unexpected free time.

So what should we do?

Clearly it is important to try and avoid under or over estimating, but from my experience it is often better to err on the side of caution when making estimates. The natural tendency for software developers is to underestimate how long it takes to do something. It nearly always takes longer to do something than you expect it to.

In terms of concrete things that we can do to try and avoid either of the above situations from happening:

• Estimate original velocity by taking cards without their point/ideal day estimates on them and get the developers to predict what they can complete in an iteration – average the point/ideal day totals to come up with an expected velocity
• Review this velocity after a couple of iterations to see if it’s actually accurate – communicate a changed velocity with the business
• Make sure the requirements are clear and that the developers understand exactly what is required to complete each story

Obviously even after doing all these things it is possible to get an original estimate wrong. That’s how it needs to be communicated though; it is an estimate, not a guarantee and it could end up being too high or too low.

I was trying to make use of the Nant Contrib scp task earlier and was getting an error message which at the time seemed a bit strange (now of course having solve the problem it is obvious!)

This was the task I was running:

<scp file="someFile.txt" server="some.secure-server.com" />

This was the error:

1
2

'scp' failed to start. 
     The system cannot find the file specified

I ran it in debug mode to try and see what was going on and got this stack trace:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

NAnt.Core.BuildException: C:\projects\project1\default.build(18,4):
'scp' failed to start. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at NAnt.Core.Tasks.ExternalProgramBase.StartProcess()
   --- End of inner exception stack trace ---
   at NAnt.Core.Tasks.ExternalProgramBase.StartProcess()
   at NAnt.Core.Tasks.ExternalProgramBase.ExecuteTask()
   at NAnt.Contrib.Tasks.ScpTask.ExecuteTask()
   at NAnt.Core.Task.Execute()
   at NAnt.Core.Target.Execute()
   at NAnt.Core.Project.Execute(String targetName, Boolean forceDependencies)
   at NAnt.Core.Project.Execute()
   at NAnt.Core.Project.Run()

Eventually a colleague and I realised that the scp task was assuming that there was an executable called ‘scp’ on the path.

This can be overriden by setting the ‘program’ attribute to whatever you want. In this case since we were running from Windows we downloaded Putty’s pscp executable and put that on the Windows path. The code to call the scp task now looks like this:

<scp file="someFile.txt" server="some.secure-server.com" program="pscp" />

If you don’t want to put it on the path then the following works just as well:

<scp file="someFile.txt" server="some.secure-server.com" program="c:\path\to\pscp.exe" />

I mentioned earlier that I have been playing around with LINQ to Xml for parsing a Visual Studio csproj file.

While having namespace issues I decided to try and parse a simpler Xml file to try and work out what I was doing wrong.

Given this fragment of Xml:

<Node>
  <InnerNode>mark</InnerNode>
  <InnerNode>needham</InnerNode>
</Node>

I wanted to get a collection(IEnumerable) of InnerNode values.

Unfortunately my over enthusiasm to use anonymous types meant that I caused myself more problems than I needed to. This was my original (failed) effort at doing so:

1
2
3
4
5
6
7
8

var innerNodes = from node in projectFile.Descendants("Node").Elements()
                 select new {InnerNode = node.Value};
 
IList<string> innerNodesAsCollection = new List<string>();
foreach (var innerNode in innerNodes)
{
    innerNodesAsCollection.Add(innerNode.InnerNode);
}

A very round about way of solving the problem I’m sure you’ll agree. I was sure this should be easy to do but I was making it very complicated indeed. A bit more googling revealed that I could put items straight into the collection from the LINQ query by not using anonymous types:

1
2

IEnumerable<string> innerNodes = from node in projectFile.Descendants("Node").Elements()
                                 select node.Value;

Much less code, much simpler, and a lesson in the art of not over complicating things.

As I mentioned in my earlier post I have been working with Facebook’s Thrift messaging project.

Unfortunately there are not currently any C# examples of how to use the Data Transfer Objects the Thrift compiler generates for us on the official wiki.

We managed to figure out how to do it by following the Java instructions and converting them into C# code. Before writing any code we need to import Thrift.dll into our Visual Studio project.

Assuming that we have the following Thrift definition file:

namespace csharp Test.Thrift
 
struct FooBarMessageThrift {
1: string Foo
2: string Bar
}

When we run the Thrift compiler we will end up with the FooBarMessageThrift class. I won’t post this class here as it’s all codegen.

The easiest way to transport this class around is by converting it to a byte array and transporting that:

1
2
3
4
5
6
7
8

var fooBarMessage = new FooBarMessageThrift {Foo = "foo", Bar = "bar"};
var stream = new MemoryStream();
 
TProtocol tProtocol = new TBinaryProtocol(new TStreamTransport(stream, stream));
 
fooBarMessage.Write(tProtocol);
 
byte[] content = stream.ToArray();

To read the byte array back into FooBarMessageThrift we do this:

1
2
3
4
5

var stream = new MemoryStream(content);
TProtocol tProtocol = new TBinaryProtocol(new TStreamTransport(stream, stream));
 
var barFooMessageThrift = new BarFooMessageThrift();
barFooMessageThrift.Read(tProtocol);

‘content’ in this example is the byte[] created in the first example, and that’s all there is to it!

Thrift is a Facebook released open source project for cross language serialisation and RPC communication.

We made use of it for our message definition layer – when it comes to messaging I’m a fan of the event based approach so we left the RPC stuff well alone.

Why Thrift?

The reason we used Thrift in the first place was because we had a requirement to get interoperability between a Java and .NET application across a message bus and it provided an easy way to do this.

Google’s protocol buffers offers an alternative solution in this area. I don’t think there is much difference between the two approaches – we just happened to come across Thrift first and it seemed to solve our problems so we went with it.

How it works

The idea behind it is that you can create Thrift definition files with the elements that you want to include in your message. You then run a compiler which generates code for Data Transfer Objects (DTO) in Java, C#, Ruby and various other languages.

Issues we came across

There were a few issues that we came across when using Thrift. They are not really Thrift specific but would always be the case when taking this approach to message definition:

Data Mapping

We wanted to have rich message objects in our code. This meant that we had to write a mapping layer which translated the Thrift DTO messages into our richer objects. This eventually ended up taking up quite a bit of our time – it once took 90 minutes to create the Thrift message, compile the Java/C# DTOs and write the mappers.

Versioning

The perennial problem of versioning was still something of an issue although several areas had been addressed.

Each property on a message had a corresponding boolean flag which determined whether or not that property has been set (i.e. whether it had a value). This meant that in theory it was possible to remove a property from the message definition cleanly, although it was still up to the client to check these flags each time they handled a message.

Using this mechanism would place the responsibility on the client to deal with potential problems – if the flags were ignored then things would break.

Re-ordering of properties was also interesting. In theory as long as the types of the properties are the same then it is possible. For example, if you had:

struct FooBarMessageThrift {
1: string Foo
2: string Bar
}

And you changed it to:

struct FooBarMessageThrift {
1: string Bar
2: string Foo
}

It would still work although you would end up with one client writing in to Foo and the other picking up the value of Bar for Foo. The reading and writing of the messages across the wire completely depends on the order in which they are specified in the thrift file.

If we try and reorder elements with different types even the above is not possible and we end up with corrupt data:

struct FooBarMessageThrift {
1: i32 Foo
2: string Bar
}

struct FooBarMessageThrift {
1: string Bar
2: i32 Foo
}

As you might imagine this can quickly get very confusing so we decided that the way forward was to not re-order and rather than remove a property, simply not assign a value to it if it is no longer in use.

After analysing the potential problems in these areas, a colleague came up with an interesting idea that when versioning messages we should treat it like an API – add but never remove.

There are certainly uses for Thrift and Google Protocol Buffers in the world of messaging but as with everything there are trade offs that we need to be aware of.

Thanks to Mark Thomas for working with me to write this.

I’ve been working with a colleague on parsing a Visual Studio project file using LINQ to effectively create a DOM of the file.

The first thing we tried to do was get a list of all the references from the file. It seemed like a fairly easy problem to solve but for some reason nothing was getting returned:

1
2
3
4

XDocument projectFile = XDocument.Load(projectFilePath.Path);
 
var references = from itemGroupElement in projectFile.Descendants("ItemGroup").First().Elements()
                 select itemGroupElement.Attribute("Include").Value;

We are selecting all the occurrences of ‘ItemGroup’, taking the first occurrence, getting all the elements inside it (i.e. all the Reference elements) and then selecting the value of the ‘Include’ attribute. A fragment of the csproj file is as follows:

1
2
3
4
5
6
7

<ItemGroup>
	<Reference Include="System" />
	<Reference Include="System.Core">
		<RequiredTargetFramework>3.5</RequiredTargetFramework>
	 </Reference>
...
</ItemGroup>

After several hours of trial and error it turned out that we just needed to include the namespace of the file when querying. The new and now working code looks like this:

1
2
3
4
5

XNamespace projectFileNamespace = "http://schemas.microsoft.com/developer/msbuild/2003";
XDocument projectFile = XDocument.Load(projectFilePath.Path);
 
var references = from itemGroupElement in projectFile.Descendants(projectFileNamespace + "ItemGroup").First().Elements()
                 select itemGroupElement.Attribute("Include").Value;

There are two quite clever things going on with the way this is done

1) There is an implicit type conversion defined on XNamespace which allows us instatiate it using a string.
2) The addition(+) operator has been overloaded on XNamespace so that it can combine the namespace with the local name (‘ItemGroup’). This is described in more detail here.

On one of my previous projects one of the problems that we had to solve was how to handle balances – we were working on a cash service for a financial services company.

The main discussion often centres around how often the balance should be updated. From my experience there are two main ways that we can go about this:

Real time update after every transaction

This is perhaps the most obvious approach and the implementation is fairly simple. After every transaction is executed the appropriate accounts should be debited/credited as part of that unit of work.

The benefit of doing it this way is that when we want to get the latest balance for an account we can easily do it without the need for any calculations. The balance stored in the database is always accurate.

The problems begin when the number of transactions being executed starts to increase. The likelihood is that most transactions will involve one of the bank’s house accounts which means that you very quickly end up with severe database locking issues.

These can theoretically be solved by using row level locking but it’s only a matter of time before the performance of the system is completely non existent.

Batch updates at varying intervals

The alternative is to not store the balance in real time and instead have a batch job running each night (for example) that updates it until the end of that day.

This approach was first shown to me by a colleague who also pointed out this article by Pat Helland on the benefits of bringing the ideas of accountancy into software design.

The trade off that we incur as a result of taking this approach is that we now need to create a batch job to calculate how the balances have changed each day and update these in the database.

Finding the current balance for an account also becomes more difficult as we now need to take the previous day’s balance and apply the impact of today’s transactions to come up with a real time balance.

The problems with database locking are removed, however, and this approach scales much more easily.

There are other interesting challenges around handling data like this in financial systems but balances strike me as particularly interesting since the most obvious solution is not necessarily the best one.

One of the first things that I do when I go onto a project is setup a ReSharper template for writing tests.

I generally set this up so that when I type ‘should’ I can press tab and it will automatically create an outline of a test method for me.

Creating a template is as simple as going to ‘ReSharper > Live Templates’ from Visual Studio.

I have attached several templates that I seem to end up writing over and over again.

To import these go to ‘ReSharper > Live Templates’ and click on the Import button on the menu.

• should Resharper template
• NUnit Test Fixture template
• Implement interface template

To make class creation templates available on the ‘Add New Item From Template’ menu right click on the project, ‘Add > New From Template > More’ then select the appropriate template and tick the box which says ‘Add to quicklist’

I touched earlier on the transparency of agile and I’ve been thinking about some of the ways that we track/report information in agile projects.

In all the projects I’ve been involved in the data being tracked almost exclusively referred to development time. One of the advantages of having continuous analysis/development/testing is that we are able to reduce the time spent on the System Integration and User Acceptance Testing phases of the project.

In a typical waterfall project most bugs would be found in these two stages of a project, and they would typically be fairly lengthy affairs.

The idea behind several of the agile practices is that the number of bugs/defects in the code base will be significantly lower than you would experience with a more traditional approach. Therefore the amount of time reserved SIT and UAT can be significantly reduced.

From what I’ve seen this is not always pointed out. It is almost like it is considered implicit in the fact that we are using an agile approach to software delivery and does not need to be explicitly stated.

I am now starting to believe that it does, and more than that – it should be actively pointed out as a benefit of using an agile approach.

We should be pointing out that ‘Yes, it may take longer to produce the code than would be possible if we just coded it straight out with no testing but the total time from the first line of code being written until the time it’s production ready will be less’. We should also point out that the benefits of our approach will only increase as more code is written and the practices practiced come into their own.

I’m not sure what the best way of doing this would be – perhaps it could be something as simple as including estimates of how long SIT/UAT will take as part of a release plan and then including these on the burn up chart.

Or is there another tool available to do this?

One of the key ideas behind agile software development is providing information as early as possible to allow the business to best make decisions.

There are a variety of ways that this is done including the use of burn up charts, estimates of scope and velocity for example. This data is compiled to try and give an accurate idea of how long a project is likely to take so that the business can work out early on whether the value it adds is worth the expected cost.

I strongly believe in this approach, certainly favouring it over other approaches that I used before working at ThoughtWorks. However, over the last couple of months I have started to wonder whether providing the business with this information so straight up actually works against you.

I believe the way that the information is viewed is different depending on when it is provided.

For example, if you have a piece of work which is due to take 12 weeks and you work out after 3 weeks that based on the current velocity it is likely to take 14 weeks you are in a weaker position than if you keep this information to yourself and then ask for another two weeks when you get to the 12 week mark.

From the business’ perspective I suppose it makes sense – if after 3 weeks you’re already 2 weeks behind then who knows how far you will be behind after 14 weeks.

I am not suggesting that it’s better to hold back this information but surely there has to be a better way to present it because from what I’ve seen the baseline figure is what’s looked at and the details are ignored.

Nor am I advocating that we under estimate on purpose to get the baseline figure lower. Clearly you will end up ruining your delivery reputation if you consistently over promise and under deliver.

The driver behind the agile approach is to always add business value, and while as a developer I would prefer to try and provide something to the client, as Mark points out, it has become clear to me that maybe sometimes cancellation is the best way of doing this.