Mark Needham

In Continuous Delivery the authors talk about the danger of ultimate configurability…

Configurable software is not always the cheaper solution it appears to be. It’s almost always better to focus on delivering the high-value functionality with little configuration and then add configuration options later when necessary

…and from my experience when you take this over configurability to its logical conclusion you end up developing a framework that can hopefully just be 'configured' for any number of 'front ends'.

This seems to be quite a common thing to do across organisations and typically the decision about what needs to go into the framework is made before there's been much/any development on the applications which will make use of said framework.

In addition the framework is typically built by a different team to the ones who are going to be working on the applications which make use of it.

As a result it's very difficult for that team to know exactly what they should be building so we'll typically end up with something that is overcomplicated for the situations it's actually required for.

In my mind the problem that we create for ourselves is the same as when we try to write a massive piece of code all in one go instead of driving it out through examples.

We try to imagine how the code might be used rather than knowing how it will actually be used.

Udi Dahan talks about favouring use over reuse and I think this is the ultimate example of not doing that.

A more effective approach would be to develop those websites/front ends individually and then let the shared 'framework' evolve from there.

That way we know that we've extracted some shared ideas because we needed to rather than because we thought we might need to.

Even then we need to be careful about what we share between applications because often it might be best to just accept a bit of duplication to avoid the implicit dependency created between teams.

Seth Godin recently wrote a post about 'the fear tax' which he describes as a 'tax' that we pay when we do something in order to try and calm our fear about something else but don't necessarily end up calming those fears.

We pay the fear tax every time we spend time or money seeking reassurance. We pay it twice when the act of seeking that reassurance actually makes us more anxious, not less.

I think one common example of a time we fall into this trap when developing software is around the security of financial systems.

Due to legal requirements that firms operating in that domain operate under we can often end up with a very complicated security solution which is unnecessary for allowing us to achieve what we want to.

On one system we worked on had a requirement to write some Javascript code to encrypt the data sent from the browser which was then sent securely through SSL to the web server.

Once it reached the web server we then had to send it in a SOAP message to a backend server where it could be unencrypted through the use of a private key.

Since one end point was in Java and one .NET there were slight incompatibilities in the way they handled security so it ended up taking a lot of time to actually get it working properly.

We had this extra security around the messages on the backend to protect against an unauthorised server trying to send messages to that server.

This despite the fact that the backend server was behind a firewall which would not accept any requests that came from servers outside the specified IP addresses of servers known to be in the DMZ.

In other words in order to remain compliant we were paying a significant fear tax in terms of increased complexity of our code base and the time taken to get all the code working in the first place.

Another simpler example of this that is often found in code is checking whether an object being passed around the code is null.

I've worked on code bases where we've ended up checking whether a particular object is null 2 or 3 times.

Alternatively we can use a pattern to take this problem away and then we don't have to worry about it again.

Seth's closing advice is as follows:

Instead of forgetting about the wasted anxiety after the fact, perhaps we ought to keep a log of how often we needlessly pay the fear tax.

Instead of over-staffing, over-planning, over-meeting and over-analyzing, perhaps organizations should take lower-cost steps and actually ship.

Think about how much you could get done if you didn't have to pay a tax to amplify or mollify your fear…

Jonas Boner recently linked to a really cool (and short) presentation by John Cleese about creativity which I think is very applicable to software development.

Cleese describes some observations he's made about creativity from his experiences working in comedy. These were some of the key ideas:

Plan to throw one away?

Cleese describes a situation where he wrote a script for Fawlty Towers and then lost it. He decided to rewrite it from memory and after he'd done that he found the original.

He was surprised to see that the rewritten version was actually an improvement over the original even though he'd written it much more quickly the second time and concludes that his unconscious mind must have still been working on the script even after he'd stopped writing it.

This seems to be similar to the ground that Fred Brooks was covering when he suggested that we should plan to throw one away because we will anyway.

It would be really interesting to see how much more quickly we'd be able to write a software system assuming all other things stay the same and we're able to build on the mistakes and things we learnt from the first attempt.

Sleeping on a problem

While discussing sketch writing Cleese points out that when he got stuck while writing at night and couldn't think what to write next he would just go to bed.

He found that when he woke up the next day and went back to the problem the solution was immediately obvious and he couldn't remember why he'd got stuck in the first place.

This is something that Andy Hunt talks about in Pragmatic Thinking and Learning as a useful technique for ensuring that we get our right brain involved in the problem solving process and I've written previously about the advantages of stepping away from a problem when we get stuck.

Creating a tortoise enclosure

Cleese suggests that we need to restrict both time and space in order to be creative.

We need to ensure that we set a restricted period of time during which we won't be disrupted and use that time to think.

The closest thing I can think of in the agile world is the idea of not context switching but it seems to go beyond that.

I like the underlying idea that we need to create some constraints in order for creativity to happen. It often seems like really good ideas come from someone being put in a situation where they can't do what they'd normally do and therefore need to innovate in order to 'survive'

I spent a bit of time working through the first Micro Testing album of the Industrial Logic eLearning suite a few weeks ago and there's an interesting piece of advice towards the end of the album:

Microtesting is not a formula. It's a technique. When microtesting rigorously, you will be called constantly to make judgments like these, between one set of names and another, and their corresponding approaches.
Remember the judgment premise.

Don't let any technique or recommendation or rule of thumb become an excuse for not exercising your judgment.

While I think this is good advice I thought it was interesting that it seems to goes against the rules of thumb approach that I described in a blog post towards the end of last year.

It seems like Kerievksy and co are encouraging us to 'get out of the shu box', as Alistair Cockburn would put it, and into the reflective 'Ha' stage of the 'Shu Ha Ri' learning model.

• Shu, the student copies techniques without adapting them.
• Ha, the student reflects on what has been learned and breaks free from traditions.
• Ri, the student is now a practitioner extending the art.

I've previously written about my own learning style which seems to favour a strong adherence to practices without too much questioning while I get used to them although I appreciate that others seem to reach the reflection stage quicker than I do.

Talking with a colleague about the value of pair programming he described one team he'd seen that had decided that it was an unnecessary practice to follow even though they hadn't spent very long using it.

As a result the team had become very siloed with each team member becoming an expert in one area of the code base, eventually leading to a situation where it was impossible to rotate anyone on the team.

From my experience we end up with the same situation if we try to make a judgement call on when and when not to follow a practice without having sufficient experience to work out how we might need to vary its use in different circumstances.

The last paragraph of Rachel Davies' post also points this out:

Learning new ways of working takes time. As Ron Jeffries once said "They're called practices for a reason".."You have to have done them. Practice makes perfect."

It's certainly useful to get into the habit of reflecting on what we're doing but I think we also need to be aware that we may not yet have the experience to make every judgement.

Nikhil and I were discussing some of the ideas around Test Driven Development earlier in the week and at one stage I pointed out that I quite liked Bryan Liles' idea of 'make it pass or change the message'.

Bryan suggests that when we have a failing test our next step should be to make that test pass or at least write some code which results in us getting a different error message and hopefully one step closer to making the test pass.

As I described this to Nikhil he pointed out that this is probably what most people are doing anyway and now the technique just has a name.

I think this is probably a fair assessment but then again I find it very useful when people give a name to common techniques/patterns as it makes them much easier to talk about without having to fill in a whole load of context.

For example Joshua Kerievsky has come up with some names to describe incremental refactoring techniques such as parallel change and narrowed change.

Parallel change describes a technique where we want to change a method signature but don't want to break all the clients of that method by directly changing it.

Instead we create the new method alongside the current one and gradually move the clients to call the new one instead. When they've all moved we can delete the original method.

Narrowed change describes a technique where we try to reduce the number of places where we have to make the change we want to make.

It's much easier to use the vocabulary that Kerievsky has come up with than to have to describe the techniques each time!

However I do think it is possible to go too far with naming things to the point that there are actually so many names that it's incredibly difficult to remember them all.

xUnit Patterns seems like an example of this to me.

There are an incredible number of patterns described in that book and while they do all describe slightly different scenarios I'm not necessarily convinced that having this many different patterns makes our discussions easier.

Overall though I think having names for common patterns in software development is a good thing and it does add value even though it seems like we've just 'given something a name'.

One of the more interesting presentations doing the rounds on twitter and on our internal mailing lists is the following one by Dan Pink titled 'Drive – The surprising truth about what motivates us'.

This topic generally interests me anyway but it's quite intriguing that the research Dan has gathered support for what I imagine many people intrinsically knew.

Incentives

The presentation dispels the myth that money always works as a motivator for getting people to do what we want them to do. Rather its value is more noticeable as an anti de-motivator – Pink suggests that organisations need to pay people enough so that money is no longer an issue.

He goes on to detail research which shows that money doesn't work as a motivator when a task requires even rudimentary cognitive skill which suggests it wouldn't be all that effective in a software development context.

Following on with this logic Pink suggests that before he did his research he would have thought that if you wanted to get innovation in an organisation then that would best be achieved by giving out an innovation bonus. He now suggests this wouldn't be effective.

We had an interesting example of this recently where the ThoughtWorks UK and Australia offices tried to run an iPad App competition with the prize of an iPad for the winning entry.

This worked fine in Australia but didn't at all in the UK. Since the 'incentive' was the same in both cases I'm inclined to believe that the reason it worked better in Australia was due to the fact that the guys there managed to create a spirit of competition between the Melbourne & Sydney offices and between the participants.

How do we get better performance?

Pink suggests that there are three main factors which lead to better performance & personal satisfaction – autonomy, mastery and purpose – and I think that to an extent we can get all of these with the agile approach to software development.

Autonomy

Pink suggests that people like to be self directed and direct their own lives and he gives the example of Atlassian's Fedex days as an example of developers being given the freedom to do whatever they want for 24 hours and coming up with some really useful products at the end of it.

I think we have autonomy to an extent on software projects. We might not have the ability to choose exactly what problem we're solving for our customer but we do mostly have the ability to choose how we're going to solve it.

We can choose how we're going to design the solution, discussing various trade offs with our customer if there's more than one way to solve it…

Mastery

Pink points out that many people play musical instruments at the weekend even though they have little chance of ever earning money from doing so and he suggests that the reason for this is its fun and we have an urge to get better.

Software development is almost a perfect fit for mastery because there are so many different areas for improvement no matter how good you are.

Corey Haines has been pushing the idea of 'Learn To Type Week' which is just one example of this.

I thought I was pretty good at touch typing but after trying out the different exercises he's been linking to I realise that there are still areas in which I can improve upon and it's quite fun trying out the exercises and trying to get better.

Apart from that there are code katas, code koans, new languages and many other ways in which we can improve ourselves.

Purpose

Pink describes purpose as "what gets you up in the morning racing to go to work" and suggests that we need something more than just making profit.

I mostly don't feel that I'm actually "going to work" on the projects I've been on because the majority of the time the other two factors are being met and I have the opportunity to work with passionate people who want to deliver high quality software together.

I'm not sure if that quite gets to the essence of purpose but it certainly doesn't feel that what I'm working on is pointless – we're building something which helps solve a problem for our customer and that seems like a valuable thing to do to me.

Overall

I found this presentation really interesting and the way it's presented with the cartoons is really cool as well.

I haven't yet read Dan Pink's book 'Drive' which presumably expands upon this talk but I've read 'A Whole New Mind' which was interesting reading so I'm sure I'll read this one at some stage.

A couple of weeks ago Dermot and I showcased a piece of functionality that we'd been working on – notably hiding some options in a drop down list.

We showcased this piece of functionality to the rest of the team in Firefox and it all worked correctly.

Our business analyst, who was also acting as QA, then had a look at the story in Internet Explorer 6 and we promptly realised that the way we'd solved the problem didn't actually work in IE6.

In retrospect we should have showcased the story in IE6 in order to shorten the feedback cycle but if we take that logic even further than it's clear that we should be running our application in IE6 frequently as we're developing functionality.

It's a dilemma that we've faced on nearly every project I've worked on recently.

We know in the back of our minds that we need to make it work on Internet Explorer 6 but because of Firebug the speed of development is siginifcantly quicker if we use Firefox.

It's almost as if we're trading off the longer term safety we would have if we use IE6 all the time for the quick feedback cycles we get from the Firebug console/CSS editor when we're fiddling around with Javascript and CSS.

The way that we're working at the moment is to continue using Firefox for local development but trying to make sure that we test and showcase in IE6.

It's not a foolproof approach, as can be seen by the example I gave at the beginning of this post, so I'd be interested if anyone has any clever ideas for dealing with the situation where we have a requirement to make our application IE6 compatible.

On the project that I've been working on over the past few months one of the key benefits of the application was its ability to perform various calculations based on user input.

In order to check that these calculators are producing the correct outputs we created a series of acceptance tests that ran directly against one of the objects in the system.

We did this by defining the inputs and expected outputs for each scenario in an excel spreadsheet which we converted into a CSV file before reading that into an NUnit test.

It looked roughly like this:

We found that testing the calculations like this gave us a quicker feedback cycle than testing them from UI tests both in terms of the time taken to run the tests and the fact that we were able to narrow in on problematic areas of the code more quickly.

As I mentioned on a previous post we've been trying to move the creation of the calculators away from the 'CalculatorProvider' and 'CalculatorFactory' so that they're all created in one place based on a DSL which describes the data required to initialise a calculator.

In order to introduce this DSL into the code base these acceptance tests acted as our safety net as we pulled out the existing code and replaced it with the new DSL.

We had to completely rewrite the 'CalculationService' unit tests so those unit tests didn't provide us much protection while we made the changes I described above.

The acceptance tests, on the other hand, were invaluable and saved us from incorrectly changing the code even when we were certain we'd taken such small steps along the way that we couldn't possibly have made a mistake.

This is certainly an approach I'd use again in a similar situation although it could probably be improved my removing the step where we convert the data from the spreadsheet to CSV file.

I recently watched a presentation given by Joshua Kerievsky from the Lean Software & Systems conference titled 'The Limited Red Society' in which describes an approach to refactoring where we try to minimise the amount of time that the code is in a 'red' state.

This means that the code should be compiling and the tests green for as much of this time as possible .

I think it's very important to follow these principles in order to successfully refactor code on a project team and it's an approach that my colleague Dave Cameron first introduced me to when we worked together last year.

These are some of my observations and thoughts on the talk:

• Kerievsky talks about parallel change which is where we want to make some changes to a bit of code and instead of changing it directly we create a parallel implementation and then gradually change the clients of that code to use the new approach.

  An example of this which I wrote about last year was when we wanted to move the creation of objects from using object initializer to the builder pattern. Instead of doing it all in one go we had both approaches in the code base at the same time and gradually moved all the existing code to use the new approach. We also tried to use this approach to change the API of one of the main objects in the CC.NET code in a dojo last year.

  Kent Beck talks about a similar approach in his QCon talk titled 'Responsive Design' from November 2008.

• One interesting point that Josh made in the Q&A session is that there may be times when we don't necessarily want to go straight for parallel change – it might be easier to use a narrowed change approach first.

  With narrowed change we would first look to reduce the number of places where we have to make the change we want to make.

  For example if we want to change an object to internally use a list instead of an array we could first isolate the places where we add to or retrieve from the data structure into methods. We would then call those methods instead of accessing the data structure directly.

  This way we can reduce the number of places we need to change when we eventually change the array to a list.

  I haven't used this approach before but will look to do so in future.

• There was a discussion about small steps and big leaps at the end where one of the attendees pointed out that it often takes much longer to take a small steps approach rather than just taking one big leap.

  Josh pointed out that the appropriate choice depends on the risk involved in the refactoring situation – if it's low risk then perhaps it does make sense to just change the code in one go. However, an additional advantage of the small steps approach is that it makes it much easier to do a graceful retreat if the refactoring gets out of hand and we end up yak shaving.

Although I already knew some of the approaches shown in this video it's always interesting to see how experienced practitioners put them to use and there were a couple of new ideas that I hadn't come across before.

I particularly liked the fact that there was a 20 minute Q&A section at the end. The discussion is quite interesting to listen to.

I've written previously about the Mikado method and how I've made use of it for identifying ways in which I could refactor code but I think this approach is more generally applicable for any kind of code investigation.

Our application has a lot of calculations in it and we've been trying to refactor the code which wires all the calculators up to make use of a DSL which reveals the intention of the code more as well as making it easier to test.

Unfortunately after changing the code to use this approach one of the calculations was about by about £15.00 in one of our acceptance tests.

We didn't have the best logging around all the calculators so it wasn't immediately obvious where to start looking for the problem.

I decided to sketch out the ways the calculators interacted with each other and then follow the deepest path – which was Calculator D – Calculator G – Calculator H – to see if there were any differences in the values being calculated when running the old and new versions of the code.

Interestingly there was no problem in that bit of the code which then allowed me to rule out that whole part of the tree and then start looking at the other calculators to try and find the problem.

I'd previously been trying to work out what was going on just by reading through the code but found it incredibly difficult to remember where I'd already investigated so drawing the diagram really helped with that too.