Mark Needham

We recently had a requirement to dynamically update a drop down list based on how the user had filled in other parts of the page.

Our initial approach was to populate the drop down with all potential options on page load and then add CSS selectors to the options that we wanted to hide. That worked fine in Chrome and Firefox but Internet Explorer seems to ignore CSS selectors inside a drop down list so none of the options were being hidden.

We therefore had to try and dynamically add and remove options from the drop down list instead.

The list that we initially loaded onto the page was like this:

<select id="foo" name="foo">
<option value="A" selected="selected">A</option>
<option value="B">B</option>
<option value="C">C</option>
<option value="Not applicable">Not applicable</option>
</select>

Christian eventually came up with the following solution to hide/show those options and select the appropriate one after several hours of trial and error:

// captured on page load
var originalFooOptions = $("#foo > option");

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

var foo = $("#foo");
var dependentField = $("#fieldFooIsDependentOn");
 
if(notApplicable()) {
	foo.children("option[value!='Not applicable']").remove();
	foo.val("Not applicable");
} else {
	foo.empty();
	$(originalFooOptions).each(function() {
		var newOption = $(this).clone();
		if ($(this).val() === dependentField.val()) {
			newOption.attr("selected", "selected");
		}
		newOption.appendTo(foo);
	});	
}

Another approach we tried was to try and dynamically update ‘foo’ rather than removing all the items and then adding them back.

Unfortunately we kept getting an ‘unspecified error’ on Internet Explorer 6 when we tried to set the selected value with that approach.

I wrote a couple of weeks ago about how we’d been trying to make use of the NHibernate 2nd level cache and we were able to cache our data by following the various posts that I listed.

Unfortunately when we ran some performance tests we found that the performance of the application was significantly worse than when we just wrote our own ‘cache’ – an object which had a dictionary containing the reference data items we’d previously tried to lookup and the appropriate values.

We don’t need to handle cache invalidation. The client’s policy is to restart production servers every night so if we want to update any of the reference data then we just need to make sure a database script is run before the servers get restarted.

Explicit transactions when reading

There is a post on the NHibernate Profiler website which describes why we should not use implicit transactions when using NHibernate. Instead we should create explicit ones:

When we don’t define our own transactions, it falls back into implicit transaction mode, where every statement to the database runs in its own transaction, resulting in a large performance cost (database time to build and tear down transactions), and reduced consistency.

Even if we are only reading data, we should use a transaction, because using transactions ensures that we get consistent results from the database.

We ended up with something like this:

public class OurRepository
{
	public ReferenceDataObject Find(ReferenceDataObjectKey key)
	{
		using(var session = SessionFactory.OpenSession())
		{
			using(var tx = session.BeginTransaction())
			{
				var query = session.Linq<ReferenceDataObject>().Where(r => r.Key == key);
				query.QueryOptions.SetCachable(true).SetCacheMode(CacheMode.Normal);
				// and so	 on
			}
		}
 
		// return the object
	}
}

As well as performance tests, we found that our integration tests became much slower than when we used our own ‘cache’.

We have some tests which look up 100s of different bits of reference data and the total time taken to run those tests went from around 4 seconds up to 30 seconds.

As I understand it, putting a transaction around a query means that we create a transaction with the database on every request even if the query is cached and we’re going to retrieve the results from the 2nd level cache rather than the database.

We took out the transaction which reduced the time taken to 7 seconds but it was still slower than when we used our hand rolled cache.

It seems like we must be doing something wrong or not understand something with respect to the NHibernate 2nd level cache because it seems ridiculous that the performance could be this different?

Something which Dermot, Christian and I noticed last week is that on our project we’ve reached the stage where we intuitively know what the underlying problem is for any given error message in the application we’re working on.

We’re pretty much at the stage where we’re effectively pattern matching what’s going on without needing to think that much anymore.

This is a good thing because it saves a lot of time analysing every single message to try and work out what’s going on – I think this means that we’ve reached a higher level of the Dreyfus model when it comes to this particular situation.

The problem with getting too used to this approach is that eventually we’ll come across a problem that we haven’t come across before and if we don’t recognise that this is the case we’ll end up getting very frustrated.

I had the chance to work with Ade Oshineye a few years ago and he always encouraged me to “quit thinking and look” when it came to problem solving.

This idea is derived from a book titled ‘9 indispensable rules of debugging‘ and the thinking behind it is that we often go straight to the solution for a problem without spending the time to understand what the problem actually is.

We actually came across a situation like this recently while investigating a problem in production.

We were getting an exception which looked pretty similar to other problems that we’d seen previously so we immediately tried that solution without any further investigation.

It had no impact at all so we had to go back and actually look at the error message we were receiving before trying something else.

As it turned out the time we wasted picking the wrong solution was less than 30 seconds but I think we had got to the point where a bit of complacency had crept in and we believed that there weren’t any ways the application could go wrong that we hadn’t seen before.

Talking further about this with Dermot he pointed out that this situation was akin to discovering an unknown unknown i.e. we came across a problem that we didn’t know we didn’t know about since we thought we knew about all of them!

Errol Morris wrote an interesting article about this recently in the New York Times in which referenced Dunning and Kruger’s paper ‘Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments‘.

The learning for me from this is that while intuition is very useful it’s also important to at least be aware that we probably don’t know everything and that we may well come across new situations and will have to approach them as a novice again.

One of my favourite patterns from Ade Oshineye and Dave Hoover’s ‘Apprenticeship Patterns‘ is ‘Be the worst‘ which is described as follows:

Surround yourself with developers who are better than you. Find a stronger team where you are the weakest member and have room to grow.

Be the Worst was the seminal pattern of this pattern language. It was lifted from some advice that Pat Metheny offered to young musicians: “Be the worst guy in every band you’re in.” Pat’s advice struck a chord with Dave, and was one of the reasons he started writing this book.

Since I started working at ThoughtWorks it hasn’t been that difficult to follow this pattern and I’ve been the least experienced developer on the majority of teams that I’ve worked on.

I always thought this was a pretty good thing as since I’m always surrounded by people who know much more than I do about various aspects of software development the opportunity to learn is very high.

Recently conversations with several different colleagues leave me questioning whether at some stage we need to move away from this pattern, perhaps temporarily to improve skills in other areas.

The authors do cover this a bit in the book:

Being in a strong team can make you feel as if you are performing better. The other members of that team will often prevent you from making mistakes, and help you recover from mistakes so smoothly that you won’t realize that you may not be learning as much as you think. It’s only when you work on your own that you will see how much your team increases your productivity and realize how much you have learned.

Making technical decisions

One area in which I’ve noticed that this is true is when it comes to making technical decisions on a project. Quite often I find that even though I have an idea of what the solution to a problem should be I end up deferring the decision to someone more senior in the team.

I don’t think there’s a problem with discussing solutions with others in the team but it certainly seems that it would be a better learning experience to be in a situation where I was forced to make the call and then see how things went as a result of that decision.

It’s certainly possible to engineer a situation where you have to make that type of decision by working on open source projects but it’s still useful to get experience on real projects as well.

I guess the easiest way is to be on a team where you aren’t the worst so that by default you’ll end up in a position where you have to make those calls.

An alternative is for more senior people to encourage others to make decisions with the knowledge that at least they will be there to help recover the situation if it goes wrong. I’ve seen some of my colleagues use this approach and it seems to work reasonably well.

It’s still not a perfect approach though because often someone more experienced may know intuitively that an approach isn’t going to work but will struggle to explain why they know that.

Solving problems

The other situation where ‘be the worst’ is perhaps limiting is when it comes to solving problems on a team.

At the first sign of being ‘stuck’ there is a real temptation to ask someone for help even though you might be able to solve the problem alone given enough time.

A colleague in my first job suggested that whenever I got stuck it was worth struggling with it for an hour before asking for help. I don’t know if that’s too prescriptive but there certainly seems to be some merit in the idea.

I’d be interested in hearing others’ thoughts on this and whether there is in fact a point at which you can grow more by focusing less on learning all the time from others and more on stretching yourself to find situations in which to take the responsibility for making decisions based on that knowledge.

One problem we’ve come across a few times over the last couple of months while using Mercurial is the situation where we want to quickly commit a local change without committing other local changes that we’ve made.

The example we came across today was where we wanted to make a change to the build file as we’d made a mistake in the target that runs on our continuous integration server and hadn’t noticed for a while during which time we’d accumulated other local changes.

The following is a rough diagram of the situation we had:

We had multiple file changes in our working directory which hadn’t yet been checked in to the local repository or the central repository.

We wanted to push just the change in blue.

My initial thought was that I could check in just that one file into our local repository and then push it to the central one.

hg ci -m "mark: updating build file to fix build" -A /path/to/build.file

I then wanted to push that change but when I went to do so I realised that they were other incoming changes which we hadn’t yet integrated with.

In order to integrate with those changes we need to make sure that we don’t have any locally uncommitted changes which of course in this scenario we do since we deliberately chose not to check in some of our local changes.

One way around this would be to just force the push and ignore the need to integrate with the remote changes but that doesn’t seem the right approach to me but I’m not sure what is.

We ended up just checking in everything we had locally, commenting out the bits that we were currently working on, merging with the remote changes and then pushing everything to the remote repository.

That’s obviously a really poor way of solving the problem so I’d be interested in what a good way to solve this problem would be!

Roy Osherove wrote a post about a month ago describing the different maturity levels of software teams and the strategies that he uses when leading each of these which I found quite interesting.

He describes the following states of maturity for a team:

• Chaotic Stage – the state where a team does not possess the skills, motives or ambition to become a mature self managing team.
• Mid-Life stage – where a team possesses some skills for self management and decision making , and can make some of its own decisions without needing a team lead.
• Mature stage – where a team is practically fully self managing and a team leader is mostly a coach rather than a decision maker.

Later on in the article he goes on to describe the best approach as a leader of each of these types of team:

• The chaotic stage needs a strong, dictator like leadership
• The Mid-Life stage needs a coach-dictator
• The Mature stage needs a coach

My experience over the last few years is that a lot of tech leads seem to start off with the assumption that their team is a chaotic one and that they need to make every decision if a project is going to be successful.

Perhaps this is a good place to start but I think it’s important to keep checking/iterating your approach to see whether it’s still appropriate. Teams tend to improve as time goes on to the point where you don’t need to decide as much for the team as you originally did.

The dictator approach seems to work more effectively in smaller teams in non political environments where it is possible for the tech lead to be around frequently and have the time available to make the majority of decisions.

When we don’t have that i.e. it’s a bigger team or political situation then it doesn’t seem to scale since that person becomes the bottleneck as they don’t have the time available to get involved in every decision that gets made.

Even if it does work the team leader can end up discouraging people from taking responsibility by always taking it themselves instead of coaching people so that they become more effective at decision making.

Ironically this isn’t what they were trying to achieve but not allowing people to be responsible for decisions means that they’ll be less likely to take responsibility in the future i.e. it’s a self fulfilling prophecy.

I appreciate that there is a fine balance to strike between allowing people to self manage and potentially make mistakes and wanting to keep things progressing by making a decision which you intuitively know is right but somehow we need to find it!

An interesting quote from a post by Leanne Chase about leadership sums up my current opinion on where we should aim to drive software teams towards:

Finally, I heard the ESPN announcer say at one point last night after a Kobe Bryant 3-pointer that Kobe was essentially telling the team – no worries, get on my back, I’ll carry you. That sort of mentality must be exhausting in basketball, at work and in life. I don’t know about you but I would much rather have a team mentality.

I’ve never yet had to lead a software team so all the above is from observation of how others have carried out this role. It’s interesting picking up ideas from each of them and hopefully I’ll be able to make use of these if I’m in that role in the future.

Dermot and I were doing a bit of work on a mini testing DSL that we’ve been writing to try and make some of our interaction tests a bit more explicit and one of the things that we wanted to do was find out which method was being called on one of our collaborators.

We have a stub collaborator which gets injected into our system under test. It looks roughly like this:

public class StubCollaborator : IGotForcedToCollaborate
{
	public double Method1()
	{
		return CannedValue();
	}
 
	public double Method2()
	{
		return CannedValue();
	}
 
	private double CannedValue()
	{
		return 10;
	}
 
}

We wanted to try and capture which of the methods on that object had been called by our system under test and then assert on that value from our test.

While trying to work out how to do this we came across the ‘StackTrace’ object. We use it like so to work out which public method on that object has been called:

public class StubCollaborator : IGotForcedToCollaborate
{
	private MethodBase methodCalled;
 
	..
 
	private double CannedValue()
	{
		methodCalled =  new StackTrace().GetFrames()
                			.Select(f =>f.GetMethod())
                			.Where(m => m.DeclaringType.Name == GetType().Name)
                			.Where(m => m.IsPublic)
                			.First()
		return 10;
	}
 
	public string GetMethodCalled()
	{
		return methodBase.Name;
	}	
 
}

We needed to only find the public methods because ‘CannedValue’ was showing up on the stack trace and since it’s private this was an easy way to exclude it.

I’m sure there are other ways to get this type of information but we were able to solve our problem really quickly with this solution.

I’ve had the iPad for about a month now and since my colleagues Martin Fowler, Neal Ford and Chris Stevenson have already previously written about their experiences with it I thought I’d share the way I’m using it as well.

Twitter

I follow a lot of people involved in software development on twitter and come across a lot of interesting articles/blogs that people link to or write. A lot of the time I don’t really want to read those posts when I come across them – it would be much better if I could just save them to read later on.

Before I had the iPad the way I did this was just by having loads of tabs open on Chrome which tended to result in my computer grinding to a halt because of the memory being consumed by having all those articles open.

On the iPad I’ve been making use of the Osfoora HD twitter client which is integrated with Instapaper – an application which allows you to save web pages and then read them offline.

I’m online most of the time so I was initially unsure how much value I’d get out of it. I now read almost every technical article I come across via Instapaper.

I’ve been posting to twitter way more because it’s now so easy to post links there which is the main way that I use it.

Reading technical material

Before I bought the iPad I wanted to make sure that I’d be able to read pdfs on it. I have a lot of technical books as pdfs and I’ve wanted to buy a device that would allow me to read those more easily since I don’t like reading pdfs on my laptop unless they’re purely coding books.

Luckily the Good Reader application allows me to do exactly that and the ability to crop the pages to be exactly how you want is fantastic and something that I’d never even thought of until I saw it.

I use Google Reader for my RSS feeds so none of the RSS reader applications really appealed to me until I came across reeder which basically sits on top of Google Reader and downloads all the unread items for offline viewing.

It’s integrated with twitter as well so I can easily post any interesting things that I read.

Blogging

I’m using the Blog Press application which I’ve found to be better than the Word Press one because it allows me to cut and paste text around the post – I found that I couldn’t do this with the Word Press application.

Ben Parr has a more extensive post on his experiences blogging from the iPad but overall for me it’s still not as good as blogging from Mars Edit on the Mac.

It’s a bit of a hassle copying links into posts because you need to manually write the Html to do so which is pretty painful in itself but also because you have to keep switching to other applications to get the links in the first place.

I have been using BlogPress to quickly copy/paste any interesting quotes from the blogs that I read but more often than not I end up posting it as a draft and then tidying it up with Mars Edit later on.

Videos

Watching videos on the iPad is amazing – the picture is obviously much bigger than it would be on an iPhone and it’s much more convenient than it would be on a laptop.

I watch technical videos now and then for which the Air Video application comes in really handy. It can convert any video files you have on your computer to a format that can be watched on the iPad although there are several formats which work without conversion as well.

InfoQ and Skills Matter both frequently post technical presentations and this week NDC 2010 has made a lot of the presentations from that conference available to download.

A couple of months ago I wrote about an approach to stories that Christian has been encouraging on our project whereby we slim stories down to allow us to deliver the core functionality of the application as quickly as possible.

In our case we had a requirement to setup a range of different parameters used to lookup reference data used in the different calculations that we have in our application.

At the time we created an interface that the rest of the application would interact with so that we could easily substitute the real version in when we needed to:

We released the first version of the application about a month ago and finally last week implemented the story where the data would move from being in memory to being in a database table.

One of the requirements which we had delayed by only having these parameters in memory was the ability to easily modify them.

Any changes that needed to be made required an update of the code and then redeployment whereas with the database approach we would only have needed to deploy a delta script.

In the event there has only been one occasion so far where those parameters needed to be updated so it hasn’t proved to be a problem.

Discussing this with a colleague on Friday he pointed out that what we’d done originally was to accept technical debt in our solution knowing that at some stage in the future we would need to address that.

The interesting thing about this case was that we knew exactly when we were going to repay that debt whereas it’s often the case that we create technical debt in a code base and vaguely know that at some stage we’ll address it.

As Martin Fowler points out in his entry on technical debt:

Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline.

We probably saved at least a day’s worth of effort by delaying this decision and were able to work on functionality that the business needed by the deadline instead. We then paid back that debt last week when we had more slack in the system.

The benefits of getting something into the market place quickly are much greater than I previously imagined and I think we can look at our assumptions of how a solution ‘needs’ to be designed much more closely to see if we can make these trade offs more frequently.

I was reading a recent blog post by Gabriel Schenker where he discusses how his team is making use of Git and about half way through he says the following:

When using Git as your SCM it is normal to work for quite a while – maybe for a couple of days – in a local branch and without ever pushing the changes to the origin. Usually we only push when a feature is done or a defect is completely resolved.

We’ve been using Mercurial on the project I’m currently working on over the past few months and although it’s a similar tool we’ve been following a different approach.

We’ve got it setup the same way we would setup Subversion:

We’ve been trying to push to the central repository as frequently as possible, just as we would if we were using Subversion.

I don’t know the Git workflow that well because I haven’t used it on a project yet but we’ve always found that it’s beneficial to integrate with code being written by others on the team as frequently as possible.

Not doing this can lead to the problems which Martin Fowler outlines in his post about feature branches.

We’ve tried to ensure that after every commit the build still passes although we do sometimes have broken versions in the code committed locally because we don’t run our full test suite before every local check in.

Even if a feature isn’t completed I still think it’s valuable to have what we’ve done so far checked in and it also helps remove the problem with needing to backup local repositories:

Since we are going to work locally potentially for days without pushing to the origin (our central repository) we might well loose our work if we have a hard disk crash or our office is flooded. Thus we need some backup strategy.

We just need to make sure the central repository is being backed up and then the danger of losing our work is significantly reduced.