Mark Needham

Thoughts on Software Development

Archive for January, 2011

Increasing team sizes: Boredom

with 2 comments

Although the majority of the teams that I’ve worked on over the past few years have been relatively small in size I have worked on a few where the team size has been pretty big and perhaps inevitably the productivity has felt much lower.

I think this is somewhat inevitable since although the overall throughput of these teams may be higher than on smaller teams, due to problems such as having difficulty parallelising work, not every pair is working at maximum productivity.

This tends to mean that you’ll end up with some pairs who have development work to do for only part of the iteration and will then be unable to pick up a new story because there are dependencies on other stories which are currently in progress.

As a result the people in this position will get extremely bored and eventually they’ll start to distract the other people on the team which means that their productivity goes down as well.

Another side effect of knowing that there isn’t anything else for you to pick up is that you subconsciously won’t finish the story you’re working on as quickly as you otherwise might be able to.

One suggestion I’ve heard is that people should pick up technical debt when they find themselves in this situation but from what I’ve noticed unless you were the one who noticed the technical debt there is both context and motivation lacking.

Another possibility in this type of situation is for the developers to try and help out elsewhere perhaps by doing some technical analysis on upcoming stories or helping out testers with their work.

The former seems to work reasonably well but the problem when doing this offshore is that you can often only get up to a certain point at which you need some help from guys onshore.

As a result technical analysis often isn’t enough to keep a pair occupied for that long.

I’ve previously seen the latter work reasonably well whereby a tester would help the developer to work out which unhappy path scenarios needed to be automated and the developer could then write the automation code.

I haven’t seen that work particularly well on my current team and I think the reason is probably down to the fact that we keep separate sets of ‘dev’ and ‘QA’ functional tests.

I can’t currently think of a good solution to this problem other than matching the amount of work and the number of pairs more carefully so that everyone is occupied for the majority of the time.

Written by Mark Needham

January 27th, 2011 at 10:59 pm

Posted in Agile

Tagged with

The Five Orders of Ignorance – Phillip G. Armour

with 3 comments

While trawling the comments of Dan North’s ‘Deliberate Discovery‘ post I came across an interesting article written by Phillip G. Armour titled ‘The Five Orders of Ignorance‘.

The main thing I took from the article is that the author uses the metaphor of software as a ‘knowledge acquisition activity’ for which he then defines five orders of ignorance that we can have in our attempts to acquire that knowledge.

These are as follows:

  1. 0th Order Ignorance – Lack of Ignorance i.e. I know how to do something and can display by lack of ignorance with some sort of output
  2. 1st Order Ignorance – Lack of Knowledge i.e. I don’t know something but I know that I don’t know how to do it and I know what I need to learn in order to be able to do it.
  3. 2nd Order Ignorance – Lack of Awareness i.e. I don’t know that I don’t know something.
  4. 3rd Order Ignorance – Lack of Process i.e. I don’t know a suitability efficient way to find out I don’t know that I don’t know something
  5. 4th Order Ignorance – Meta Ignorance i.e. I don’t know about the five orders of igorance

Armour points out that the biggest problems when building any system are 2nd and 3rd order ignorance

We can reduce these types of ignorance by having people on the team who have experienced similar situations before and therefore don’t have as much ‘lack of awareness’ as other people who haven’t experienced the same situations.

On the majority of projects that I’ve worked on the main source of ignorance for us has often been around the politics within the organisation.

Most of the developers I work with tend to be quite straight forward in their communication which often isn’t the best style of communication when working with people from another ‘vendor’ or from the client.

I think you do become better at dealing with these situations after you’ve come across them a few times although from my observations each situation has played out a little differently even if the underlying reasons for the human behaviour were the same.

A few months ago Toni wrote a blog post about the way that he works at Forward in which he describes how they don’t have a need for several of the ‘agile practices’ often used on teams.

I’ve also been questioning the point of several of them as it feels like we sometimes just dogmatically follow a practice because that’s what we’ve always done.

Thinking about it from the angle of ignorance I’m now more inclined to believe that the intention of the practices is to help us reduce our ignorance in some way:

  • Stand-up: Helps to reduce the ignorance that people have about what others are working on and whether someone else can help us reduce our own ignorance in some way
  • Retrospective: Helps to reduce our ignorance of how to work more effectively in a team in the context we’re in
  • Writing tests: Helps to reduce someone else’s ignorance about what a piece of code is supposed to do when they come across it. Also provides protection around our ignorance of what we might break by changing code

From what I know about Forward it seems like people would have less ignorance than you might have if you were working as a consultant in a big bureaucratic organisation.

Of course the fact that they only hire very strong developers presumably also means that their technological ignorance is also reduced and it now makes more sense to me how they’re able to work effectively with such a stream lined process.

I do think it still makes sense to question the practices we’re using and constantly assess whether they are actually helping us to reduce our ignorance.

If not then we might as well use the time to do something else.

These are the main two things that came to mind for me while reading the article but I’m sure there are other things to be gained from it.

Armour has also written another article titled ‘The Nature of Software and the Laws of Software Process‘ in which he expands on the orders of ignorance further.

Written by Mark Needham

January 26th, 2011 at 6:08 pm

Deliberate Discovery: The stuff I don’t know list

with 3 comments

Towards the end of Dan North’s post on Deliberate Discovery he makes the following suggestion:

There is much more to say about deliberate discovery. Think about applying the principle to learning a new language, or picking up a new technology, or a new domain. What could you do to identify and reduce your ignorance most rapidly?

This reminded me a lot of what I used to do when I came across things that I didn’t know how to do a few years ago.

I would write down the thing that I didn’t know and then make sure that I looked up the things on my list at the end of the day and then discuss what I’d read/learnt with a colleague the next day.

More recently I’ve fallen into what I’d call a complacent pattern where if I find something that I don’t know 100% how to do I don’t go out of my way to check that I actually know how to do it now.

I have more confidence that I’ll be able to work out how to do something if the need arises but I don’t gain that knowledge right away.

Effectively I’ve continued identifying my ignorance but haven’t taken the steps to reduce it.

Now that I’ve realised I’m doing this I’ve started writing down things that I don’t know on a “Stuff I don’t know list” and I’m back to spending some time each evening to run through the list and at least gain some knowledge on each item.

One problem I used to find when I did this before was that I’d start reading about one thing and then I’d notice something else that I didn’t know, go off and read about that and so on until I’d forgotten what I was originally trying to learn.

To try and combat that I’ve been time-boxing myself with a pomodoro while I’m working through the list and then adding any new concepts I come across to the end of the list.

Another mistake I’ve made/sometimes do make is reading very passively i.e. I read the information but don’t really understand it.

To get around that problem I’ve been talking to colleagues about what I’ve been reading, as suggested by Andy Hunt in Pragmatic Thinking and Learning.

In addition I tend to remember things better if I actually get to use them in practice rather than just learning the theory.

Since the majority of the reading I’ve been doing lately has been around networking I’ve been using tcpdump and wireshark so I can see the theory in action.

Unfortunately those are another two things I don’t know so I’m learning about them as I go.

Written by Mark Needham

January 26th, 2011 at 6:07 pm

Distributed Agile: Stories – Negotiable

with 3 comments

I was recently reading an article about how to write meaningful user stories and towards the end of it the author mentioned the INVEST acronym which suggests that stories should be:

  1. Independent
  2. Negotiable
  3. Valuable
  4. Estimable
  5. Small
  6. Testable

From what I’ve seen the most difficult one to achieve in a distributed context is that stories should be ‘negotiable’, in particular when it comes to negotiating the way that the UX of a bit of functionality should work.

On most of the projects that I’ve worked on the people designing the UX tend to work slightly detached from the development team and then send their designs over as wire frames.

It’s not the most ideal setup even if you’re working onshore but it becomes even more challenging when you’re working offshore.

Typically onshore if a particular user flow was very difficult to implement then one of the developers might go and talk with the UX guys and then explain the problem and give another potential solution.

The trade off between the cost of implementation and the user experience that the UX person has suggested is clearly outlined in these conversations.

When that feedback comes from offshore it has much less impact and I think it comes across much more as a criticism of someone’s work rather than an attempt to be pragmatic in helping the client to deliver a product within a time frame.

One possible solution to this problem if you have some onshore colleagues is to have them go and talk about the problem but we’ve found it difficult to do that because we try to split onshore/offshore stories so that we’re working on different parts of the code base.

Therefore it is pretty difficult for an onshore developer to go and discuss it for you.

To add to the problem we tend to realise the difficult of implementing a certain UX during the middle of our day which means we need to wait half a day to see if we can get it changed.

Given that time lag we often just end up designing it the way that’s been specified so that we don’t ‘waste’ time.

It’s clearly not an idea situation so I’d be keen to hear if anyone has come up with ideas to get around this.

Written by Mark Needham

January 24th, 2011 at 3:34 am

Posted in Agile,Distributed Agile

Tagged with ,

While in India: Osmotic communication

with 2 comments

One of the things that has been puzzling me during my time in India is the amount of time that is spent in meetings pushing information to people rather than them pulling it.

In previous projects that I’ve worked on a lot of the knowledge was moved between around as a result of osmotic communication

Osmotic communication means that information flows into the background hearing of members of the team, so that they pick up relevant information as though by osmosis.

This is normally accomplished by seating them in the same room.

Then, when one person asks a question, others in the room can either tune in or tune out, contributing to the discussion or continuing with their work.

After thinking about it a bit more I’ve identified a few reasons why I think this approach doesn’t work so well on my current project.

The ability to ignore background noise

The ThoughtWorks Pune office is pretty much one big room with over 100 people in it so the amount of background noise is fairly constant.

When we’ve tried to run our standup near where we sit it’s been impossible to hear people standing 5 or 6 metres away from you.

You therefore end up getting into the habit of blocking out anything going on around you as was evident when there was drilling being done in the office.

The side effect of doing this, of course, is that there could be relevant discussions going on around you and you’d be much less likely to notice.

Team size/distribution

The second problem is that the team is distributed which means that we’re never going to overhear any conversations in Chicago and vice versa.

To add to this our team is now 25+ in size so it’s spread out over 3 tables one of which is only within shouting distance.

Martin Fowler wrote an article about team rooms quite recently and while the ideas make sense I’m not sure how they scale as team size increases.

Overlap of roles

Although this by no means applies to everyone that I’ve worked with I’ve noticed a tendency for people to stick to their role much more than I have in other countries.

Therefore a developer will only do ‘developer tasks’ and a QA will only do ‘QA tasks’ for example.

When you have overlap such that a developer may be ‘pairing’ with a QA or BA on something then the types of conversations that are head seem to lead to information moving around the team more effectively.

Written by Mark Needham

January 24th, 2011 at 3:33 am

Posted in Communication

Tagged with

Listening to feedback mechanisms

without comments

In Growing Object Oriented Software the authors talk about the value of listening to our tests to understand potential problems with our code and I’ve started to notice recently that there are implicit feedback mechanisms dotted around at a higher level which we can also listen to.

A couple of examples come to mind:

Nothing to show in the showcase

I’ve worked on a couple of projects where we’ve got to the end of the iteration and realised that we don’t actually have anything tangible to show the product owner.

This tends to be because we’ve spent the majority of our time working on ‘technical stories’ which typically have no visible output.

This may be as a result of horizontal slicing of work rather than vertical slicing but what it essentially means is that the work you’ve done has no immediate value to anyone.

Sometimes there are technical aspects to a solution which have a lot of unknowns and combining those with a vertically sliced story would mean that the story becomes too big to fit in an iteration.

From my experience a way around this which has worked well before is to run a spike/experiment card where we work out all the technical details/gotchas and setup any necessary ‘framework’ and then have the other vertically sliced chunks of work plug into this.

Having said that I’m no expert in breaking down work into manageable chunks so if anyone has any other ideas I’d be interested in hearing about those.

Boring meetings

The typical response if someone complains of being bored in a meeting or walks out of a meeting is that they should deal with it but that response doesn’t help us that much.

I think if people are bored then it’s an indication that the meeting isn’t particularly interactive/inclusive or that the person shouldn’t be in the meeting.

We can try to resolve that first point by having meetings which are designed more as a discussion/learning forum rather than a brain dump session by one person to the group.

This encourages more of a pull approach over a push one and seems to be more effective at keeping people’s attention as long as the topic of the discussion is actually relevant to them.

Frequently it seems that a more effective approach would be to have fewer people involved in a meeting and have one of the participants quickly summarise what happened for the rest of the team.

Written by Mark Needham

January 21st, 2011 at 3:46 am

Posted in Feedback

Tagged with

Coding: Spike Driven Development

with 5 comments

While reading Dan North’s second post about software craftsmanship I was able to resonate quite a lot with a point he made in the ‘On value’ section:

I’m not going to mandate test-driving anything (which is a huge about-face from what I was saying a year ago), unless it will help. Copy-and-paste is fine too. (Before you go all shouty at me again, hold off until I blog about the benefits of copy-and-paste, as it appears in a couple of patterns I’m calling Spike and Stabilize and Ginger Cake. You might be surprised.)

I’ve been finding that quite frequently with some of the problems I’ve worked on recently that we haven’t known exactly how to solve it when we started and ended up hacking/spiking the code quite a bit at that stage until we figured out what we needed to do.

Dan replied with the following on twitter when I mentioned this to him:

@markhneedham kind of deliberately discovering the best way forward by actively reducing your ignorance? hmm, might catch on…

He wrote about this a few months ago in a post titled ‘Introducing Deliberate Discovery‘.

I nearly always feel pretty bad when I take this approach because I’m not TDDing code which easily be written using that style.

The reason I don’t is that it slows me down when I’m in discovery mode.

I could TDD what I think the code needs to look like and end up with nice looking code which doesn’t actually solve the problem but that seems to miss the point.

Another approach could be to drive from a higher level by using cucumber or a similar tool but that always ends up being quite fiddly from my experience and again it slows down the discovery that I want to make.

We had a recent example of this while trying to work out how to display the next part of a page we were working on.

Our initial approach was to make an AJAX call to the server and get back a JSON representation of our resource which we could render on the page.

While implementing this approach we realised that there were already a some post backs being made to the server from various other parts of the page which resulted in the page being refreshed.

We initially thought that this would mean that we could use the data being passed to the view on those post backs to do what we wanted.

We kept a copy of our original code and then started trying to implement what we thought would be a simpler solution.

Unfortunately after playing around with that approach for a few hours we realised that it wasn’t going to work because loading the data that way led to another part of the page getting completely screwed up.

We therefore ended up back at the first approach again.

One side effect of taking that second approach was that eventually the logic got too complicated for us to verify that what we had written was correct just by eye balling it.

We therefore slowed down a bit and TDD’d the code so that we could document our understanding of what was supposed to happen.

Uncle Bob wrote a reply to Dan a couple of days ago in which he ends with the following:

So when you see someone wearing a green wrist-band that says “Clean Code” or “Test First” or “Test Obsessed”

It’s a promise made to one’s self: “I will do a good job. I will not rush. I will write tests. I will go fast by going well. I will not write crap. And I will practice, practice practice so that I can be a professional.”

The code I’m writing during the time when I’m discovering what I need to do certainly isn’t clean or test first but it is part of an attempt to allow me to figure out what to do in order to go faster.

I’m more used to taking the approach I’ve described when explicitly working on ‘spikes’ or ‘experiments‘ but in a way what we’ve done is perhaps closer to concurrent set based engineering.

I think the skill I still quite haven’t got right is realising when I’ve taken the spiking/hacking too far and would benefit from going and TDDing a bit of code but I’m sure I’ll figure it out with practice.

Written by Mark Needham

January 19th, 2011 at 5:46 pm

Posted in Coding

Tagged with

MySQL: The used command is not allowed with this MySQL version

with 12 comments

For my own reference more than anything else, on my version of MySQL on Mac OS X, which is:

mysql5 Ver 14.14 Distrib 5.1.48, for apple-darwin10.4.0 (i386) using readline 6.1

When I try to use the ‘LOAD DATA LOCAL’ option to load data into tables I get the following error message:

ERROR 1148 (42000) at line 4: The used command is not allowed with this MySQL version

Which we can get around by using the following flag as described in the comments of the documentation:

mysql --local-infile -uroot -pandsoon

Or by putting the following entry in a config file (in my case ~/.my.cnf):


I tried a bit of googling to see if I could work out why it happens but I’m still none the wiser.

Written by Mark Needham

January 18th, 2011 at 6:58 pm

Posted in Software Development

Tagged with

Installing git-svn on Mac OS X

with 4 comments

I somehow managed to uninstall git-svn on my machine and Emmanuel Bernard’s blog post suggested it could be installed using ports:

sudo port install git-core +svn

I tried that and was ending up with the following error:

--->  Computing dependencies for git-core
--->  Dependencies to be installed: p5-svn-simple subversion-perlbindings apr-util db46 cyrus-sasl2 neon serf subversion p5-term-readkey
--->  Verifying checksum(s) for db46
Error: Checksum (md5) mismatch for patch.
Error: Checksum (md5) mismatch for patch.
Error: Checksum (md5) mismatch for patch.
Error: Checksum (md5) mismatch for patch.
Error: Target org.macports.checksum returned: Unable to verify file checksums

A bit of googling led me to the macports website where it was pointed out that this problem was happening in that user’s case because the link to the patch now redirects to a HTML page:

It’s actually the fact that the links pointing to those patches have become invalid.

redirects to the Berkley-DB webpage

and you get a HTML instead of the real patch.

I’m not sure if that’s the case for me because only the Checksum is failing, nothing is mentioned about downloading from Oracle’s website.

By adapting one of the suggestions for downloading the patches manually I arrived at the following combination of commands to fix the db46 dependency:

cd /opt/local/var/macports/distfiles/db4/4.6.21_6/
sudo rm -rf patch*
for i in 1 2 3 4; do echo $i && sudo curl$i -O; done

Followed by:

sudo port install git-core +svn

And I’m back in the game!

Written by Mark Needham

January 15th, 2011 at 7:05 pm

mount_smbfs: mount error..File exists

with 8 comments

I’ve been playing around with mounting a Windows file share onto my machine via the terminal because I’m getting bored of constantly having to go to Finder and manually mounting it each time!

After a couple of times of mounting and unmounting the drive I ended up with this error:

> mount_smbfs //mneedham@punedc02/shared punedc02_shared/
mount_smbfs: mount error: /Volumes/punedc02_shared: File exists

I originally thought the ‘file exists’ part of the message was suggesting that I’d already mounted a share on ‘punedc02_shared’ but calling the ‘umount’ command led to the following error:

> umount punedc02_shared
umount: punedc02_shared: not currently mounted

I had actually absent mindedly gone and mounted the drive elsewhere through Finder which I only realised after reading Victor’s comments on this post.

Make sure that you already do not have the same share mounted on your Mac.

I had //host/share already mounted in /Volumes/share, so when I tried to mount //host/share to /Volumes/newshare it gave me the “file exists” error.

I learnt, thanks to the forums, that you can see which devices are mounted by using ‘df’.

This is where Finder had mounted the drive for me:

> df
Filesystem                 512-blocks      Used  Available Capacity  Mounted on
//mneedham@punedc02/shared  209696376 199773696    9922680    96%    /Volumes/shared

Since the shared drive gets unmounted when I disconnect from the network I decided to write a shell script that would set it up for me again.

function mount_drive {
  mkdir -p $2
  mount_smbfs $1 $2 
drives_to_unmount=`df | awk '/mneedham@punedc02/ { print $6 }'`
if [ "$drives_to_unmount" != "" ]; then
  echo "Unmounting existing drives on punedc02: \n$drives_to_unmount"
  umount $drives_to_unmount
mount_drive //mneedham@punedc02/media /Volumes/punedc02_media 
mount_drive //mneedham@punedc02/shared /Volumes/punedc02_shared

At the moment I’ve just put that in ‘/usr/bin’ so that it’s available on the path.

If there’s a better way to do this or a way to simplify the code please let me know.

I did come across a few ways to do the mounting using Apple Script in this post but that’s not half as fun as using the shell!

Written by Mark Needham

January 15th, 2011 at 6:31 pm

Posted in Shell Scripting

Tagged with