Mark Needham

I've spent most of my time working on agile software development teams over the last few years so for the most part each pair is only working on one story, keeping the work in progress low and allowing them to focus on that piece of work until it's completed.

My pair and I ended up in a therefore somewhat unusual situation last week where we were attempting to work on three things at the same time and weren't doing a particularly great job on any of them.

It wasn't immediately obvious to me that we were doing this since the two extra tasks that we were working on were related to deployment issues on different environments.

However we eventually started making mistakes and in rushing to rectify those made even more mistakes since we were still trying to concentrate on three different things.

It became much more obvious at this point that we needed to just pick one of the items and focus on that until we were done which also served as a reminder that it's good to use the story board as an indicator of what everyone is working on.

In this case two of the tasks we were working on weren't on the story board otherwise it would have been more obvious to the rest of the team that we shouldn't have been working on two of them.

I've never really noticed the problems of context switching before so it was interesting to get such a stark example to remind me of its dangers.

Something which an agile approach on a project typically gives us is the ability to change requirements rapidly based on the different types of feedback we typically get over the course of the project.

One way that we can lose this advantage is by getting caught up by the number of story points being completed and using this as the measure of success.

The flexibility to change has an impact on the number of story points that may be completed in a given iteration – if we start doing some work on a story and then get feedback from the business while it is still in progress it's possible that we will end up with more work to do than we had previously.

Once a story is estimated we don't typically go back and change that estimate unless there was a change in the assumptions being made before we play it. This means that any changes we make based on feedback would not result in re-estimation of the story card.

We now have more work to do without our measurement mechanism recognising that.

It gets even worse if we start working on a feature and then get feedback from the business that the feature is no longer useful to them and we should just stop working on it.

At this point a fairly common response is to 'lock down' the requirements so that we'll only work on the currently defined requirements and no new ones or changes will be permitted.

While this might make some logical sense it misses the point that we're supposed to try and deliver something that is actually useful rather than something which just meets the initial requirements.

In reality one of two things, or perhaps a combination of both, ends up happening:

• There will still be changes and all we've done is create the illusion that there won't be any.
• The business will now push to get exactly the features that were specified originally even if the development team ends up spending a disproportionate amount of time creating those features for the value they actually provide.

Story points are really useful for allowing us to get an idea of the relative size of stories and can be helpful for getting a rough understanding of the amount of work we can complete in a given time period but we need to be careful how we use them.

Our goal at the end of the day is to deliver working software.

Story points are just a mechanism for giving us some feedback on how well we are doing that. They're not the goal in itself.

I recently finished reading Lean Thinking and one of the things that was repeatedly emphasised is the need to look at the process as a whole rather than trying to optimise each part individually.

If we phrased this in a similar way to the Agile Manifesto it would probably read 'Big Picture over Local Optimisations'.

The examples in Lean Thinking tend to be more manufacturing focused but I think this idea can certainly be applied in thinking about software projects too.

Individual vs Team Focus

For me on a software development team what we care about is the quality of the team as a whole not the individuals on it.

It's all good being very strong technically but you have to be able to work effectively with other people to solve problems otherwise your value to a development team is much reduced.

I wrote a bit about this previously in my lean look at pair programming but I think it goes beyond just ensuring that we are sharing knowledge around the team by having people collaborating closely with each other.

When I first joined ThoughtWorks my former colleague Fred George was always talking about the value of being poly skilled – having the ability to carry out more than one role – and I think this is lost when we refer to people as being their role i.e. 'you ARE a developer' rather than 'you CAN PLAY the developer role'.

If we have people on a team who are polyskilled or generalising specialist (although I think generalising specialist refers more to poly skilled across a role rather than being able to carry out multiple roles) then we can get away with building teams which are smaller in number and which are more resilient and able to respond when team members are ill or absent.

Horizonal vs Vertical teams

Creating teams of people by the layer or horizontal which they work on rather than the piece of functionality or vertical which they are working on seems to be one of the most obvious ways of locally optimising instead of looking at the big picture i.e. the delivery of a piece of functionality to the business.

I guess the theory is that if we put everyone working on similar things on the same layer in the same team then it will result in greater productivity than having people who are working on different areas of the system but have the same underlying goals.

The problem is that we create a lot of handover points – opportunities for context and information to be lost – and the ineffectiveness of this approach as a communication mechanism means that it takes much longer to integrate the components from each horizontal than it would have done with a team with people from each layer in it.

In addition any techniques used to try and increase the productivity of an individual team will be ineffective since they will only satisfy a local optima and will have a knock on effect to the rest of the system.

For example, let's say we have three teams A, B and C and team B aren't as productive as we were expecting. Applying pressure to team B may have an impact on their immediate productivity but it's likely to affect the other two teams since team B will probably be less communicative towards those teams since they are trying to optimise their own performance.

Having a vertical team which is responsible for the delivery of a feature or features end to end works much better from my experience.

Colocated vs Distributed teams

One of my favourite agile ideas is having a co-located team, with the business people, analysts and developers all working in the same physical location.

Sometimes though the decision is taken to distribute teams across multiple locations – be it because that's more convenient or cheaper to do.

The big picture i.e. project delivery is likely to become more difficult to achieve due to the increased cost of communication that has been created.

For me there is nothing better than face to face communication – even with phone calls/instant messaging/emailing it is still an extremely valuable form of communication. With any of the other three it's much more difficult to read what someone is really thinking and you can come across as being much more aggressive than you intended to in any of those mediums.

My colleagues in India/China will certainly have some tools/techniques to bridge these gaps that I'm not aware of but I still remain convinced that if we have the choice then a co-located team is the best choice.

In Summary

These are just some of the ideas that came to mind when trying to apply one of the principles of lean to software development teams. Although many of these may seem obvious sometimes it takes another angle of looking at the problem to make that visible.

If you can think of any others feel free to mention them in the comments.

Chris Johnston has another interesting post in which he writes about the practice of re-estimating cards after they have been completed.

I think this somewhat misses the point that the estimate is indeed supposed to be an estimate. It might turn out to be too optimistic or too pessimistic, the theory being that overall we will end up with a reasonable balance that will allow us to make a prediction on how much work we believe we can complete in a certain time period.

I've always seen estimates of story cards to be a relative thing – on all the teams I've worked on after initial cards have been estimated we've looked at the difficulty of upcoming cards and tried to score them relative to the ones already estimated.

Re-estimating cards (presumably) individually means that you lose the benefits of this approach and end up with a fairly meaningless value.

We are also ignoring the inherent uncertainty involved in estimating if we do so afterwards and I don't really see it helping us that much for future predictions since there is still going to be uncertainty with regards to the implementation of future cards no matter what we try to do.

If data needs to be collected after a card has been played then I would say the time it took to complete a card might be a more useful metric although there are more than likely going to be some cards that take longer than expected and some that take less than expected, so I'm not sure what extra value this data would provide.

As Chris points out if there are times, when we come to play a card or just after we start work on it, when we realise that some of our assumptions are incorrect therefore meaning that the estimate is inaccurate.

In this case I think it's reasonable to talk through the new assumptions and re-estimate the card so that we have a more accurate estimation of its difficulty.

When trying to work out how much work we can do in a certain time box, averaging the data we have from the original estimates is likely to provide the most accurate prediction in my opinion.

Apart from times when we have story cards overflowing from one iteration to the next, meaning that we end up with one high scoring and then one low scoring iteration, I have found on my projects that the amount of work points wise doesn't tend to vary very much which lends credence to the theory that the estimates eventually balance out.

My colleague Chris Johnston wrote recently about his experiences in agile software development, posing some questions that he has.

Specifically:

• Why comments are evil?
• Why design is evil?
• Why must you pair all the time?
• Why Agile principles become Agile rules?

Now I'm assuming that most (if not all) of Chris' experiences with agile have been at ThoughtWorks, in which case the mix of agile we use on our projects tends to be a combination of Scrum and Extreme Programming.

The last question seems the most logical to start with:

Why is it that for many people and many shops, Agile principles become rules and the only way to do things? I have met people and worked in shops were they have Agile all figured out and any deviation is heresy. If you are doing Agile this one way, then you are unprofessional and a very bad programming.

I'm with Chris on this one – as far as I understand agile is all about adapting a set of principles to the environment that we find ourself in. It's unlikely that the exact same approach will work in two different projects.

I've worked on 6 or 7 projects at ThoughtWorks and each has taken a slightly different approach to agile, varying the amount of process we use, the amount and type of tracking done, the approach to gathering requirements, and so on.

Our client's wishes, the availability of the business, the number of integration points, the size of the team, are all things that will influence the approach that we end up taking.

Why must you pair all the time?

Extreme Programming (XP) defines a set of practices that we can use on an agile project although there are certainly other approaches that are equally valid.

Pair Programming is one of the XP practices rather than something you need to do in order to be 'agile'.

I've worked on teams where we haven't paired, teams where some people on the team pair and teams where everyone does pair all the time but I wouldn't say as a rule people have to pair all the time.

When a team is responsible for the delivery of some software pair programming is the best approach that I have worked with for ensuring that people on the team have a good understanding of all areas of the code base.

While someone working alone in 'flow' might be able to write some brilliant code, noone else on the team knows anything about that code meaning that the knowledge of the implementation takes longer to work its way to the rest of the team.

Why design is evil?

I don't think design is evil and again all the teams I've worked on have done some design work up front, it would be fairly foolish not to.

Finding the balance between over architecting up front and doing just enough design to allow us to keep moving forward is the key here.

Why comments are evil?

I've previously written about the dangers of commenting code so I don't want to revisit that too much, suffice to say that it it usually possible to describe what you want to in a comment using an appropriate method name instead.

My experience with comments is that they go out of date more quickly than code does and therefore can end up being misleading to someone reading the code.

The only real time I have seen a valid use for them is when used to describe a problem the code is getting around where the explanation is fairly verbose and can't be expressed easily in code – for example, when using one of the jQuery validators there was some strange behaviour with regards to the behaviour of the plugin when different options were set.

For a while we didn't put a comment and others in the team would change the code not realising that it would cause a problem.

With the comment in place people have a better context before they make a change.

One of the inevitabilities of most projects is that at some stage there is going to need be some sort of integration.

The likes of Alistair Cockburn in Crystal Clear and Andy Hunt/Dave Thomas in The Pragmatic Programmer talk of the need to do integration early rather than letting it wait until later, but why?

Get the pain out the way

To some degree every time we try to integrate there is going to be some level of pain – for me it therefore makes sense that we take this pain early on when we have the chance to do something about it rather than leaving it until later and being surprised at the problems it causes.

We never really know that the system actually works as expected until it is fully integrated, and integration inevitably leads to situations which we didn't know existed previously.

The potential problem we need to be careful about here is that we still deliver features with business value while doing our early integration otherwise we end up with the problem that we seem to be delivering absolutely nothing.

Code can be more example driven

When we integrate later on one problem we face is that we have to try and assume what the integration will be like rather than actually knowing.

While it is possible to somewhat isolate ourselves from other systems, at some stage we need to integrate and this can typically lead to different data flowing through, in a different format than expected and we need to handle these differences.

If we integrate late we are putting ourselves into a guessing game of what we think we are going to integrate against and this leads us down the path of assumption driven development.

This inevitably leads to us coding for some situations which won't actually happen when we are integrated and missing out some situations which we didn't actually know could happen until we integrated.

Less inventory

In lean terms inventory is a product or part of a product which is basically in a state of waiting – either waiting for something to be done to it or waiting for a customer to buy it.

I think we can find a similar analogy in terms of story cards where a part of that story is to integrate with another system for example.

When we do the integration work needed later on the additional problem we face is that we naturally lose some knowledge in the time that the story is sitting waiting.

In addition there is also wasted time as people regain the context of how the integration fits into the overall functionality of the story.

I've worked on a few different agile projects and one of the things that hasn't been completely consistent is when we consider a story to be 'done'.

There seem to a few different approaches, each of which has its benefits and drawbacks.

Why do we care?

We care about 'done' for tracking the points we have achieved in an iteration and for knowing when we have added the value the story provides. For this post I'm interested in 'done' in terms of when we count the story towards our points total.

Business proxy signoff

With this approach we count a story as done when it has been signed of by a business proxy, typically a client business analyst or maybe a combination of a BA/QA.

The benefit of this approach is that we have normally have better access to a BA meaning that we don't end up having stories piled up waiting to be signed off as being done.

I have seen 2 mini approaches, both of which require a story walk through with the BA/QA:

• Acceptance tests pass => Done
• Acceptance tests pass + exploratory testing satisfactory => Done

With the first approach any bugs beyond the acceptance criteria will be tracked as bugs whereas with the latter the story will be moved back and just considered incomplete.

The disadvantage of either of these approaches is that we are counting something as done before it's actually been signed off by the business, which means that we might have a false sense of the value we're delivering. If the business proxies are communicating regularly with the business though I don't think it's necessarily a problem.

Business signoff

With this approach we only count a story as done when it has been signed off by our business stakeholder.

The benefit is that when a story is classified as done we have actually delivered something that is acceptable to the business.

On the other hand, when using this approach it's fairly important to have access to the business stakeholder on a regular basis if the velocity of the team is to be measured accurately. When we don't have this access we can end up with a backlog of stories waiting for sign off, all the while lengthening the time from when it was last worked on to when it is eventually considered complete.

In production

This is an approach which I haven't seen used on a project but which I read across in a post written by David Laribee.

I can see how this makes sense as we haven't actually delivered any value to our customer until the software is actually being used.

I think the difficulty of using this approach would be if we aren't releasing code into production regularly – we wouldn't get feedback on how we're progressing and therefore would have more difficulty noticing when things in our process aren't working as well as we'd hope.

In Summary

I think the further down the process we consider a story as done the more accurately our points total represents the value added to the customer.

Although we can get quicker 'points feedback' by classifying done earlier in our process, the accuracy of what that points total actually means is perhaps lowered.

I'm not sure which approach is the best one because each has its benefits and drawbacks. It will be interesting to see whether there are any further variations on what is considered 'done' on future projects I work on.

I am aware that I may be asking completely the wrong question and what we should be asking is 'Does it matter when it's done other than for reasons of tracking velocity?'

I came across an interesting article written for Visual Studio Magazine about agile methodologies where the author makes what I consider to be some misconceptions.

The first is around the level of experience of people working on an agile team:

For example, agile teams have a tendency to require a high level of experience and professionalism just to join the team.

I wouldn't say I have a high level of experience and I've been working on agile teams for the past two years, just one data point suggesting that this statement is not actually accurate.

I think it is important to have experienced team members, at the Journeyman or Master level to use the lingo of Software Craftsmanship, but I don't think you'd want your team to be completely filled with these types of people.

This is one of the things that is pointed out in Pragmatic Learning and Thinking referring to the Dreyfus Model definition of expert in this case. We need to have people of differing levels of experience if we are to create a successful team.

Ideally you want a mix of skills on a team: having an all-expert team is fraught with its own difficulties; you need some people to worry about the trees while everyone is pondering the forest.

This is referring to the fact that as people become more competent they tend to start focussing more on the big picture than they did when they just started learning. As the authors point out, it's useful to have some people on the team who just want to focus on the details and get things done.

Of course I'm sure there are examples of 'dream teams' of software developers who have worked successfully together, but maybe this is more the exception than the rule.

The second point the author raises whether agile is just about making developers comfortable by allowing them to write more code:

Finally, I wonder how much of agile development exists simply to make developers more comfortable. Most developers love to write code, and agile does a pretty good job of rationalizing why developers don't need to do anything else. Design? Code it with test-driven development. Requirements gathering? I've heard proponents of agile development say that the user doesn't know what they want until they see something, so you might as well get straight to coding.

It is fair to say that when working in an agile way we get to the code more quickly than when following some other approaches but surely that's a good thing?

Although certainly some part of design is best done at the whiteboard, we can't actually know whether our designs are going to work unless we put them in the code so I don't see design by TDD being a problem.

There is a need to do some requirements gathering before we start working on features/stories but quite often we don't find out about constraints and trade offs until we begin implementing a feature, at which time the customer can decide what approach they want us to take. The goal here is to try and provide them quick feedback so that what we develop is actually what they want.

I would encourage the author to go and see an agile team at work (maybe using the Patterns and Practices team at Microsoft as an example) and see whether he believes this points still hold true.

While reading The Toyota Way one of the ideas which stood out for me was the constant mentioning of organisations which picked bits of The Toyota Way, implemented them, achieved some short term gains but then eventually these improvements and went back to the way they were before.

I noticed a similar theme coming out in the series of posts in the last week or so about the decline of agile.

I have worked on several projects over the last couple of years with varying levels of agile being applied. I am aware that agile is considered to be something that you are rather than that you do but for this post agile refers to agile principles and practices.

I have noticed that it is much easier to get working software out the door when we follow these principles and practices but becomes way more difficult as soon as we are unable to follow some of them.

This is a similar idea to one I noticed in The Toyota Way:

Early on in the book the TPS House was mentioned – the idea being that everything must be strong from the use of the tools to the belief in the philosophy. It seemed to me from reading this that applying Toyota's ideas successfully is an all or nothing game – i.e. it would be very difficult to be successful in the long term by just picking and choosing certain ideas without having the other ones to support them.

One of the common misunderstandings about agile is that writing tests makes us move more slowly and that removing them will allow us to go more quickly. While this may be true in the short term it eventually catches up with us and we can look forward to long sessions with the debugger trying to work out why the code isn't working as we expected.

A couple of months ago I was considering whether there are any agile practices which are absolutely key to ensuring success. I ran this idea by several colleagues who all pointed out that it was the principles that were absolutely key and not necessarily the practices which are derived from them.

I believe we can therefore make a link between the practices and principles of agile, such that if we don't believe in thoroughly testing our code, for example, then we are not adhering to the principle of delivering working software or paying continuous attention to technical excellence.

Having said that I have also worked on projects where delivering frequently was not something which the business considered beneficial and we only delivered software at the end of the project. We did take the time to carry out practice deployments more frequently than that to ensure we knew the deployment process but there was no actual deliverable until right at the end, and despite not completely adhering to this principle we were still able to successfully deliver.

As Jeremy Miller points out in his response to the original post:

In the early days, XP was roundly criticized because every practice only seemed to be safe due to the existence of the other practices. Take one out, and the others weren't that great by themselves. I think that's more or less a fair criticism, but my response is simply to adopt, and effectively apply, everything you need to make Agile development successful.

Life is made much easier if everyone involved into the project buys into the agile or lean approach but even if they don't it is still possible to find ways to succeed, it just might be a bit more difficult.

Sometimes it's the seemingly small details of the agile/XP approach to software development that make it so much more effective than the traditional approach.

I was reminded of this last week with regards to having co-located teams with the developers, BAs, QAs and the business people all sitting in close proximity.

I was working on the auto completion function for one of our screens and the QA on the team, who was sitting next to me, asked me if I could look through the acceptance criteria that he was working on.

This conversation was almost perfectly on cue because I was just about at the point where I had covered all the happy path cases and needed to understand the other paths. While looking through the acceptance criteria we needed some clarification with regards to exactly what the business wanted.

We went over to where the BAs were sitting and posed the question. Luckily they had just had the conversation with the business people and were able to explain the intent behind the criteria.

Interestingly writing this has made me realise that there are a couple of points in our process where more communication would make things flow more smoothly but seeing the speed with which we were able to go from confusion over a requirement to complete understanding was a really great reminder for me.