Mark Needham

In Continuous Delivery the authors talk about the danger of ultimate configurability…

Configurable software is not always the cheaper solution it appears to be. It’s almost always better to focus on delivering the high-value functionality with little configuration and then add configuration options later when necessary

…and from my experience when you take this over configurability to its logical conclusion you end up developing a framework that can hopefully just be 'configured' for any number of 'front ends'.

This seems to be quite a common thing to do across organisations and typically the decision about what needs to go into the framework is made before there's been much/any development on the applications which will make use of said framework.

In addition the framework is typically built by a different team to the ones who are going to be working on the applications which make use of it.

As a result it's very difficult for that team to know exactly what they should be building so we'll typically end up with something that is overcomplicated for the situations it's actually required for.

In my mind the problem that we create for ourselves is the same as when we try to write a massive piece of code all in one go instead of driving it out through examples.

We try to imagine how the code might be used rather than knowing how it will actually be used.

Udi Dahan talks about favouring use over reuse and I think this is the ultimate example of not doing that.

A more effective approach would be to develop those websites/front ends individually and then let the shared 'framework' evolve from there.

That way we know that we've extracted some shared ideas because we needed to rather than because we thought we might need to.

Even then we need to be careful about what we share between applications because often it might be best to just accept a bit of duplication to avoid the implicit dependency created between teams.

Seth Godin recently wrote a post about 'the fear tax' which he describes as a 'tax' that we pay when we do something in order to try and calm our fear about something else but don't necessarily end up calming those fears.

We pay the fear tax every time we spend time or money seeking reassurance. We pay it twice when the act of seeking that reassurance actually makes us more anxious, not less.

I think one common example of a time we fall into this trap when developing software is around the security of financial systems.

Due to legal requirements that firms operating in that domain operate under we can often end up with a very complicated security solution which is unnecessary for allowing us to achieve what we want to.

On one system we worked on had a requirement to write some Javascript code to encrypt the data sent from the browser which was then sent securely through SSL to the web server.

Once it reached the web server we then had to send it in a SOAP message to a backend server where it could be unencrypted through the use of a private key.

Since one end point was in Java and one .NET there were slight incompatibilities in the way they handled security so it ended up taking a lot of time to actually get it working properly.

We had this extra security around the messages on the backend to protect against an unauthorised server trying to send messages to that server.

This despite the fact that the backend server was behind a firewall which would not accept any requests that came from servers outside the specified IP addresses of servers known to be in the DMZ.

In other words in order to remain compliant we were paying a significant fear tax in terms of increased complexity of our code base and the time taken to get all the code working in the first place.

Another simpler example of this that is often found in code is checking whether an object being passed around the code is null.

I've worked on code bases where we've ended up checking whether a particular object is null 2 or 3 times.

Alternatively we can use a pattern to take this problem away and then we don't have to worry about it again.

Seth's closing advice is as follows:

Instead of forgetting about the wasted anxiety after the fact, perhaps we ought to keep a log of how often we needlessly pay the fear tax.

Instead of over-staffing, over-planning, over-meeting and over-analyzing, perhaps organizations should take lower-cost steps and actually ship.

Think about how much you could get done if you didn't have to pay a tax to amplify or mollify your fear…

Jonas Boner recently linked to a really cool (and short) presentation by John Cleese about creativity which I think is very applicable to software development.

Cleese describes some observations he's made about creativity from his experiences working in comedy. These were some of the key ideas:

Plan to throw one away?

Cleese describes a situation where he wrote a script for Fawlty Towers and then lost it. He decided to rewrite it from memory and after he'd done that he found the original.

He was surprised to see that the rewritten version was actually an improvement over the original even though he'd written it much more quickly the second time and concludes that his unconscious mind must have still been working on the script even after he'd stopped writing it.

This seems to be similar to the ground that Fred Brooks was covering when he suggested that we should plan to throw one away because we will anyway.

It would be really interesting to see how much more quickly we'd be able to write a software system assuming all other things stay the same and we're able to build on the mistakes and things we learnt from the first attempt.

Sleeping on a problem

While discussing sketch writing Cleese points out that when he got stuck while writing at night and couldn't think what to write next he would just go to bed.

He found that when he woke up the next day and went back to the problem the solution was immediately obvious and he couldn't remember why he'd got stuck in the first place.

This is something that Andy Hunt talks about in Pragmatic Thinking and Learning as a useful technique for ensuring that we get our right brain involved in the problem solving process and I've written previously about the advantages of stepping away from a problem when we get stuck.

Creating a tortoise enclosure

Cleese suggests that we need to restrict both time and space in order to be creative.

We need to ensure that we set a restricted period of time during which we won't be disrupted and use that time to think.

The closest thing I can think of in the agile world is the idea of not context switching but it seems to go beyond that.

I like the underlying idea that we need to create some constraints in order for creativity to happen. It often seems like really good ideas come from someone being put in a situation where they can't do what they'd normally do and therefore need to innovate in order to 'survive'

I spent a bit of time working through the first Micro Testing album of the Industrial Logic eLearning suite a few weeks ago and there's an interesting piece of advice towards the end of the album:

Microtesting is not a formula. It's a technique. When microtesting rigorously, you will be called constantly to make judgments like these, between one set of names and another, and their corresponding approaches.
Remember the judgment premise.

Don't let any technique or recommendation or rule of thumb become an excuse for not exercising your judgment.

While I think this is good advice I thought it was interesting that it seems to goes against the rules of thumb approach that I described in a blog post towards the end of last year.

It seems like Kerievksy and co are encouraging us to 'get out of the shu box', as Alistair Cockburn would put it, and into the reflective 'Ha' stage of the 'Shu Ha Ri' learning model.

• Shu, the student copies techniques without adapting them.
• Ha, the student reflects on what has been learned and breaks free from traditions.
• Ri, the student is now a practitioner extending the art.

I've previously written about my own learning style which seems to favour a strong adherence to practices without too much questioning while I get used to them although I appreciate that others seem to reach the reflection stage quicker than I do.

Talking with a colleague about the value of pair programming he described one team he'd seen that had decided that it was an unnecessary practice to follow even though they hadn't spent very long using it.

As a result the team had become very siloed with each team member becoming an expert in one area of the code base, eventually leading to a situation where it was impossible to rotate anyone on the team.

From my experience we end up with the same situation if we try to make a judgement call on when and when not to follow a practice without having sufficient experience to work out how we might need to vary its use in different circumstances.

The last paragraph of Rachel Davies' post also points this out:

Learning new ways of working takes time. As Ron Jeffries once said "They're called practices for a reason".."You have to have done them. Practice makes perfect."

It's certainly useful to get into the habit of reflecting on what we're doing but I think we also need to be aware that we may not yet have the experience to make every judgement.

Nikhil and I were discussing some of the ideas around Test Driven Development earlier in the week and at one stage I pointed out that I quite liked Bryan Liles' idea of 'make it pass or change the message'.

Bryan suggests that when we have a failing test our next step should be to make that test pass or at least write some code which results in us getting a different error message and hopefully one step closer to making the test pass.

As I described this to Nikhil he pointed out that this is probably what most people are doing anyway and now the technique just has a name.

I think this is probably a fair assessment but then again I find it very useful when people give a name to common techniques/patterns as it makes them much easier to talk about without having to fill in a whole load of context.

For example Joshua Kerievsky has come up with some names to describe incremental refactoring techniques such as parallel change and narrowed change.

Parallel change describes a technique where we want to change a method signature but don't want to break all the clients of that method by directly changing it.

Instead we create the new method alongside the current one and gradually move the clients to call the new one instead. When they've all moved we can delete the original method.

Narrowed change describes a technique where we try to reduce the number of places where we have to make the change we want to make.

It's much easier to use the vocabulary that Kerievsky has come up with than to have to describe the techniques each time!

However I do think it is possible to go too far with naming things to the point that there are actually so many names that it's incredibly difficult to remember them all.

xUnit Patterns seems like an example of this to me.

There are an incredible number of patterns described in that book and while they do all describe slightly different scenarios I'm not necessarily convinced that having this many different patterns makes our discussions easier.

Overall though I think having names for common patterns in software development is a good thing and it does add value even though it seems like we've just 'given something a name'.

One of the more interesting presentations doing the rounds on twitter and on our internal mailing lists is the following one by Dan Pink titled 'Drive – The surprising truth about what motivates us'.

This topic generally interests me anyway but it's quite intriguing that the research Dan has gathered support for what I imagine many people intrinsically knew.

Incentives

The presentation dispels the myth that money always works as a motivator for getting people to do what we want them to do. Rather its value is more noticeable as an anti de-motivator – Pink suggests that organisations need to pay people enough so that money is no longer an issue.

He goes on to detail research which shows that money doesn't work as a motivator when a task requires even rudimentary cognitive skill which suggests it wouldn't be all that effective in a software development context.

Following on with this logic Pink suggests that before he did his research he would have thought that if you wanted to get innovation in an organisation then that would best be achieved by giving out an innovation bonus. He now suggests this wouldn't be effective.

We had an interesting example of this recently where the ThoughtWorks UK and Australia offices tried to run an iPad App competition with the prize of an iPad for the winning entry.

This worked fine in Australia but didn't at all in the UK. Since the 'incentive' was the same in both cases I'm inclined to believe that the reason it worked better in Australia was due to the fact that the guys there managed to create a spirit of competition between the Melbourne & Sydney offices and between the participants.

How do we get better performance?

Pink suggests that there are three main factors which lead to better performance & personal satisfaction – autonomy, mastery and purpose – and I think that to an extent we can get all of these with the agile approach to software development.

Autonomy

Pink suggests that people like to be self directed and direct their own lives and he gives the example of Atlassian's Fedex days as an example of developers being given the freedom to do whatever they want for 24 hours and coming up with some really useful products at the end of it.

I think we have autonomy to an extent on software projects. We might not have the ability to choose exactly what problem we're solving for our customer but we do mostly have the ability to choose how we're going to solve it.

We can choose how we're going to design the solution, discussing various trade offs with our customer if there's more than one way to solve it…

Mastery

Pink points out that many people play musical instruments at the weekend even though they have little chance of ever earning money from doing so and he suggests that the reason for this is its fun and we have an urge to get better.

Software development is almost a perfect fit for mastery because there are so many different areas for improvement no matter how good you are.

Corey Haines has been pushing the idea of 'Learn To Type Week' which is just one example of this.

I thought I was pretty good at touch typing but after trying out the different exercises he's been linking to I realise that there are still areas in which I can improve upon and it's quite fun trying out the exercises and trying to get better.

Apart from that there are code katas, code koans, new languages and many other ways in which we can improve ourselves.

Purpose

Pink describes purpose as "what gets you up in the morning racing to go to work" and suggests that we need something more than just making profit.

I mostly don't feel that I'm actually "going to work" on the projects I've been on because the majority of the time the other two factors are being met and I have the opportunity to work with passionate people who want to deliver high quality software together.

I'm not sure if that quite gets to the essence of purpose but it certainly doesn't feel that what I'm working on is pointless – we're building something which helps solve a problem for our customer and that seems like a valuable thing to do to me.

Overall

I found this presentation really interesting and the way it's presented with the cartoons is really cool as well.

I haven't yet read Dan Pink's book 'Drive' which presumably expands upon this talk but I've read 'A Whole New Mind' which was interesting reading so I'm sure I'll read this one at some stage.

A couple of weeks ago Dermot and I showcased a piece of functionality that we'd been working on – notably hiding some options in a drop down list.

We showcased this piece of functionality to the rest of the team in Firefox and it all worked correctly.

Our business analyst, who was also acting as QA, then had a look at the story in Internet Explorer 6 and we promptly realised that the way we'd solved the problem didn't actually work in IE6.

In retrospect we should have showcased the story in IE6 in order to shorten the feedback cycle but if we take that logic even further than it's clear that we should be running our application in IE6 frequently as we're developing functionality.

It's a dilemma that we've faced on nearly every project I've worked on recently.

We know in the back of our minds that we need to make it work on Internet Explorer 6 but because of Firebug the speed of development is siginifcantly quicker if we use Firefox.

It's almost as if we're trading off the longer term safety we would have if we use IE6 all the time for the quick feedback cycles we get from the Firebug console/CSS editor when we're fiddling around with Javascript and CSS.

The way that we're working at the moment is to continue using Firefox for local development but trying to make sure that we test and showcase in IE6.

It's not a foolproof approach, as can be seen by the example I gave at the beginning of this post, so I'd be interested if anyone has any clever ideas for dealing with the situation where we have a requirement to make our application IE6 compatible.

One thing that we tried to avoid on the project that I've been working on is making use of C# expressions trees in production code.

We found that the areas of the code where we compiled these expressions trees frequently showed up as being the least performant areas of the code base when run through a performance profiler.

In a discussion about the ways to improve the performance of an application Christian pointed out that once we've identified the area for improvement there are two ways to do this:

• Do it less
• Find another way to do it

We were able to find another way to achieve what we wanted without using them and we favoured this approach because it was much easier to do,

An alternative would have been to cache the compilation of the expression so that it wouldn't happen every single time a request passed through that code path.

A lot of the time it seems like it's actually possible to just do something less frequently rather than changing our approach…

For example:

• Caching a HTTP response so that not every request has to go all the way to the server.
• Grouping together database inserts into a bulk query rather than executing each one individually.

… and most of the time this would probably be a simpler way to solve our problem.

I quite like this heuristic for looking at performance problems but I haven't done a lot of work in this area so I'd be interested in hearing other approaches as well.

I recently watched a presentation given by Joshua Kerievsky from the Lean Software & Systems conference titled 'The Limited Red Society' in which describes an approach to refactoring where we try to minimise the amount of time that the code is in a 'red' state.

This means that the code should be compiling and the tests green for as much of this time as possible .

I think it's very important to follow these principles in order to successfully refactor code on a project team and it's an approach that my colleague Dave Cameron first introduced me to when we worked together last year.

These are some of my observations and thoughts on the talk:

• Kerievsky talks about parallel change which is where we want to make some changes to a bit of code and instead of changing it directly we create a parallel implementation and then gradually change the clients of that code to use the new approach.

  An example of this which I wrote about last year was when we wanted to move the creation of objects from using object initializer to the builder pattern. Instead of doing it all in one go we had both approaches in the code base at the same time and gradually moved all the existing code to use the new approach. We also tried to use this approach to change the API of one of the main objects in the CC.NET code in a dojo last year.

  Kent Beck talks about a similar approach in his QCon talk titled 'Responsive Design' from November 2008.

• One interesting point that Josh made in the Q&A session is that there may be times when we don't necessarily want to go straight for parallel change – it might be easier to use a narrowed change approach first.

  With narrowed change we would first look to reduce the number of places where we have to make the change we want to make.

  For example if we want to change an object to internally use a list instead of an array we could first isolate the places where we add to or retrieve from the data structure into methods. We would then call those methods instead of accessing the data structure directly.

  This way we can reduce the number of places we need to change when we eventually change the array to a list.

  I haven't used this approach before but will look to do so in future.

• There was a discussion about small steps and big leaps at the end where one of the attendees pointed out that it often takes much longer to take a small steps approach rather than just taking one big leap.

  Josh pointed out that the appropriate choice depends on the risk involved in the refactoring situation – if it's low risk then perhaps it does make sense to just change the code in one go. However, an additional advantage of the small steps approach is that it makes it much easier to do a graceful retreat if the refactoring gets out of hand and we end up yak shaving.

Although I already knew some of the approaches shown in this video it's always interesting to see how experienced practitioners put them to use and there were a couple of new ideas that I hadn't come across before.

I particularly liked the fact that there was a 20 minute Q&A section at the end. The discussion is quite interesting to listen to.

Something which Dermot, Christian and I noticed last week is that on our project we've reached the stage where we intuitively know what the underlying problem is for any given error message in the application we're working on.

We're pretty much at the stage where we're effectively pattern matching what's going on without needing to think that much anymore.

This is a good thing because it saves a lot of time analysing every single message to try and work out what's going on – I think this means that we've reached a higher level of the Dreyfus model when it comes to this particular situation.

The problem with getting too used to this approach is that eventually we'll come across a problem that we haven't come across before and if we don't recognise that this is the case we'll end up getting very frustrated.

I had the chance to work with Ade Oshineye a few years ago and he always encouraged me to "quit thinking and look" when it came to problem solving.

This idea is derived from a book titled '9 indispensable rules of debugging' and the thinking behind it is that we often go straight to the solution for a problem without spending the time to understand what the problem actually is.

We actually came across a situation like this recently while investigating a problem in production.

We were getting an exception which looked pretty similar to other problems that we'd seen previously so we immediately tried that solution without any further investigation.

It had no impact at all so we had to go back and actually look at the error message we were receiving before trying something else.

As it turned out the time we wasted picking the wrong solution was less than 30 seconds but I think we had got to the point where a bit of complacency had crept in and we believed that there weren't any ways the application could go wrong that we hadn't seen before.

Talking further about this with Dermot he pointed out that this situation was akin to discovering an unknown unknown i.e. we came across a problem that we didn't know we didn't know about since we thought we knew about all of them!

Errol Morris wrote an interesting article about this recently in the New York Times in which referenced Dunning and Kruger's paper 'Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments'.

The learning for me from this is that while intuition is very useful it's also important to at least be aware that we probably don't know everything and that we may well come across new situations and will have to approach them as a novice again.