Mark Needham

Thoughts on Software Development

Gaming the system: Some project examples

with 2 comments

Earlier this year Liz Keogh gave a talk at QCon London titled ‘Learning and Perverse Incentives: The Evil Hat‘ where she eventually encouraged people to try and game the systems that they take part in.

Over the last month or so we’ve had two different metrics visibly on show and are therefore prime targets for being gamed.

The first metric is one we included on our build radiator which shows how many commits to the git repository each person has for that day.

We originally created the metric to try and see which people were embracing git and committing locally and which were still treating it like Subversion and only committing when they had something to push to the central repository.

The other advantage we wanted to try and encourage is that by creating lots of small commits it’s easier for someone browsing ‘git log’ to see what’s happened over time just from glancing at the commit messages.

Bigger commits tend to mean that changes have been made in multiple places and perhaps not all those changes are related to each other.

Since we made that metric visible the number of commits have visibly increased and it’s mostly been positive because people tend to push to the central repository quite frequently.

There have, however, been a couple of occasions where people have made 10/15 commits locally over the day and then pushed them all at the end of the day and gone straight to the top of the leader board.

IMG 20111026 175248 1

The disadvantage of this approach is that it means other people on the team aren’t integrating with your changes until right at the end of the day which can lead to merge hell for them.

There have also been some times when people’s count has artificially increased because they’ve checked in, broke the build and then checked in again to fix it.

We’re going to try and find a way to combine local commits with remote pushes in a combined metric as our next trick.

Another metric which we’ve recently made visible is the number of points that we’ve completed so far in the iteration.

Previously we’ve had this data available in our Project Manager’s head and in Mingle but since a big part of how the team is judged is based on the number of points ‘achieved’ the team asked for the score to be made visible.

Since that happened from my observation we’ve ‘achieved’ or got very close to the planned velocity every week whereas before that it was a bit hit and miss.

I think sub consciously the estimates made on stories have started to veer towards the cautious side whereas previously they were probably more optimistic.

Another change in behaviour I’ve noticed is that people tend to postpone any technical tasks they have to do when we’re near the end of an iteration and instead keep focus on the story to ensure it gets completed in time.

We’ve also seen a couple of occasions where people stayed 2/3 hours longer on the last day of the iteration to ensure that stories got signed off so the points could be counted.

It’s been quite interesting to observe how behaviour can change based on increasing the visibility of metrics even when in the first case it’s actually irrelevant to the perception of the team.

Be Sociable, Share!

Written by Mark Needham

October 26th, 2011 at 11:55 pm

Posted in Systems Thinking

Tagged with

  • Regarding commits: Don’t forget about developers using squashed merges and interactive rebasing and decreasing the number of commits which eventually make it to the remote repository. I commit *very frequently* locally, but routinely squash them into fewer more logical commits before pushing. Depending on what the metrics end up being used for, it’d work out better for me to just push everything un-squashed, but that negatively affects the codebase in favor of gaming the results (unless people like “did the thing” commits!).

  • Yeh that’s a good point but I don’t think we’re at the level of comfort with git that people are actually doing that at the moment!

    My pair for the last few days (who’s very good with git) and I have been squashing commits to the extent that we don’t push commits that are going to break the build. 

    We’ve been trying to find small enough steps when changing code that we do have a logical commit after each change. I’m not sure if that ends up with too much verbosity about how we’re going about changing the code because every single step we took is visible rather than just the end result.