Mark Needham

One of the things that we've been working on lately to improve the overall time that our full build takes to run is to split the acceptance tests into several small groups of tests so that we can run them in parallel.

We are using Cruise as our build server so the ability to have multiple agents running against different parts of the build at the same time comes built it.

We then had to work out what the best environment to run more agents would be.

I guess the traditional approach to the problem would be to get some extra machines to use to run the build but this wasn't possible so our initial solution was to use our developer virtual machines as build agents.

This worked in principle but the environment became way too slow to work on when the build was running and the selenium windows that popped up when each test was running became amazingly annoying. We could only really use this approach to set up agents on developer machines that weren't being used that day.

We therefore came across the idea of having 2 virtual machines running on each developer machine, splitting the resources of the machine across the two virtual machines.

From experimenting it seems like the agent used to run a build with selenium tests in it needs to have around 1.2 GB of RAM assigned to it to run at a reasonable rate. Our developer machines have a total of 3 GB of RAM so the majority of the rest is being used by the development virtual machine.

Our development virtual machine environments run marginally slower when there is a build running on that machine but we've reduced that problem by ensuring we have the build running across as many developer machines as possible so that it runs less frequently on each machine

I think it serves as a worthwhile trade off for helping to reduce our total build time significantly – running sequentially it would take over an hour to run all our tests but in parallel we've managed to get a 10-15 minute turn around time at worst.

While trying to see if we could make that any faster we toyed with the idea of having two build virtual machines on the same box but that didn't seem to provide the improvement that we expected due to I/O restraints – our build has quite a bit of writing and reading from disc and having two builds doing this at the same time seemed to slow down both of them.

Overall though it's working out pretty well for us in terms of providing us a quick feedback mechanism of whether or not our system is working correctly end to end.

Lu Ning and I have been working on trying to improving the consistency of our selenium driven acceptance test build, the main problem being the number of false negatives we've been getting.

The reason for these false negatives has been due to the difficulty in working out the necessary condition to wait for when making ajax calls using the jQuery library.

The conditions can be a combination of elements on the page appearing, disappearing, being enabled or disabled and sometimes we don't actually have any visible change on the page which we can hook into to know whether or not an ajax call has returned.

We tried several different solutions to this problem, each one resulting in less and less code until we came across some jQuery ajax hooks which we weren't previously aware of.

The events we looked at were ajaxComplete, ajaxError, ajaxSend, ajaxStart, ajaxStop and ajaxSuccess which don't seem to fire exactly when you might expect.

We had thought that we might just be able to listen to ajaxStop and then we would know when all the ajax calls on the page had completed but this didn't seem to fire all the time. A couple of the other events were equally troublesome but we've ended up with the following code to allow us to tell when ajax requests have returned so we can continue with the next selenium command in our test.

// extend jQuery ajax with the capability of remembering the count of active ajax requests
$.activeAjaxRequestCount = 0;
 
$().ajaxSend(function() {
  $.activeAjaxRequestCount++;
  // register lazily to make sure it's the last one and get executed after all other handlers
  if (!$._ajaxErrorHandlerAdded) {
    $().ajaxError(function() {
      $.activeAjaxRequestCount--;
    });
    $._ajaxErrorHandlerAdded = true;
  }
})
.ajaxSuccess(function() {
  $.activeAjaxRequestCount--;
});

We have some customised error handling in our code which already hooks into the ajaxError event so we wanted to make sure that we could add a handler which was fired after all the production code handlers had completed.

We originally tried just directly hooking up a handler to the ajaxError event inside the document ready but we found that this handler was getting fired before the customised error handling one.

Lu therefore came up with the idea of creating this handler when we send our first ajax request which means that it will be the last handler fired when an error is raised.

We created a decorator around the selenium client in order to allow us to use this code from our C# code:

public class SeleniumDecorator : ISelenium 
{
	public void Click(string locator, int timeout)
	{
      	    WaitComplete(() => selenium.Click(locator), timeout);
	}
 
	private void WaitComplete(Action fn, int timeout)
	{
  		selenium.RunScript("selenium.browserbot.getCurrentWindow();window.jQuery.activeAjaxRequestCount=0;");
  		fn();
  		WaitForCondition("selenium.browserbot.getCurrentWindow();window.jQuery.activeAjaxRequestCount===0;", timeout.ToString());
	}
}

We are also wrapping other selenium calls which might trigger ajax calls which include 'FireEvent', 'Type' and 'Select'.

The other clever thing which Josh discovered here is that to access jQuery libraries from selenium we need to refer to 'window.jQuery' – we then seem to have access to any jQuery related code that we have loaded on the page.

I've discussed with a couple of colleagues the merits of doing this and it was pointed out to me that it's not generally a good idea to put code in just to make something more testable at the acceptance level.

In this case we are using the activeAjaxRequestCount in production code for being able to monitor the state of our ajax requests so it does have another use as well.

I've not used Selenium much in my time – all of my previous projects have been client side applications or service layers – but I've spent a bit of time getting acquainted with it this week.

While activating some acceptance tests this week I noticed quite a strange error happening if the tests ran in a certain order:

com.thoughtworks.selenium.SeleniumException: ERROR: Current window or frame is closed! 
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:73)
at com.thoughtworks.selenium.HttpCommandProcessor.getString(HttpCommandProcessor.java:154)
at com.thoughtworks.selenium.HttpCommandProcessor.getBoolean(HttpCommandProcessor.java:227)
at com.thoughtworks.selenium.DefaultSelenium.isElementPresent(DefaultSelenium.java:394)

I tried commenting out some of the tests and then running the other ones and everything worked fine but when I ran all of them the problem returned.

Eventually after some eagle eyed debugging by a colleague of mine we realised that the only difference was the lack of the following line in the failing test:

selenium.selectWindow(null);

A quick look at the documentation explains precisely why this works:

if windowID is null, (or the string "null") then it is assumed the user is referring to the original window instantiated by the browser).

What had in fact happened was that the previous test had launched a pop up window and when it closed that window the focus wouldn't return to the original window launched when Selenium first started unless we executed the above method call.