Mark Needham

Thoughts on Software Development

Archive for December, 2011

WebDriver: Getting it to play nicely with Xvfb

with 2 comments

Another thing we’ve been doing with WebDriver is having it run with the FirefoxDriver while redirecting the display output into the Xvfb framebuffer so that we can run it on our continuous integration agents which don’t have a display attached.

The first thing we needed to do was set the environment property ‘webdriver.firefox.bin’ to our own script which would point the display to Xvfb before starting Firefox:

import java.lang.System._
lazy val firefoxDriver: FirefoxDriver = {
  setProperty("webdriver.firefox.bin", "/our/awesome/starting-firefox.sh")
  new FirefoxDriver()
}

Our first version of the script looked like this:

/our/awesome/starting-firefox.sh

#!/bin/bash
 
rm -f ~/.mozilla/firefox/*/.parentlock
rm -rf /var/go/.mozilla
 
 
XVFB=`which xVfb`
if [ "$?" -eq 1 ];
then
    echo "Xvfb not found."
    exit 1
fi
 
$XVFB :99 -ac &
 
 
BROWSER=`which firefox`
if [ "$?" -eq 1 ];
then
    echo "Firefox not found."
    exit 1
fi
 
export DISPLAY=:99
$BROWSER &

The mistake we made here was that we started Xvfb in the background which meant that sometimes it hadn’t actually started by the time Firefox tried to connect to the display and we ended up with this error message:

No Protocol specified
Error cannot open display :99

We really wanted to keep Xvfb running regardless of whether the Firefox instances being used by WebDriver were alive or not so we moved the starting of Xvfb out into a separate script which we run as one of the earlier steps in the build.

We also struggled to get the FirefoxDriver to kill itself after each test as calling ‘close’ or ‘quit’ on the driver didn’t seem to kill off the process.

We eventually resorted to putting a ‘pkill firefox’ statement at the start of our firefox starting script:

/our/awesome/starting-firefox.sh

#!/bin/bash
 
rm -f ~/.mozilla/firefox/*/.parentlock
rm -rf /var/go/.mozilla
 
pkill firefox
 
BROWSER=`which firefox`
if [ "$?" -eq 1 ];
then
    echo "Firefox not found."
    exit 1
fi
 
export DISPLAY=:99
$BROWSER &

It’s a bit hacky but it does the job more deterministically than anything else we’ve tried previously.

Written by Mark Needham

December 15th, 2011 at 11:19 pm

Posted in Software Development

Tagged with

WebDriver: Getting it to play nicely with jQuery ColorBox

without comments

As I mentioned in an earlier post about removing manual test scenarios we’ve been trying to automate some parts of our application where a user action leads to a jQuery ColorBox powered overlay appearing.

With this type of feature there tends to be some sort of animation which accompanies the overlay so we have to wait for an element inside the overlay to become visible on the screen before trying to do any assertions on the overlay.

We have a simple method to do that:

def iWaitUntil(waitingFor: => Boolean) {
  for (i <- 1 to 5) {
    if(waitingFor) {
      return
    }
   Thread.sleep(200)
  }
}

It can then be called like this in our tests:

def driver: WebDriver = new FirefoxDriver()
 
iWaitUntil(driver.findElements(By.cssSelector(".i-am .inside-the-colorbox h3")).nonEmpty)
driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText should equal("Awesome Title")

Annoyingly what we noticed was that this wasn’t enough and although the h3 element was coming back as being visible it was then failing the following assertion because ‘getText’ was returning nothing despite the fact that it clearly had text inside it!

Uday came up with the neat idea of adding an additional wait clause which would wait until the text was non empty so we now have something like this…

def driver: WebDriver = new FirefoxDriver()
 
iWaitUntil(driver.findElements(By.cssSelector(".i-am .inside-the-colorbox h3")).nonEmpty)
driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText should equal("Awesome Title")
iWaitUntil(driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText != "")

…which seems to do the job nicely.

An alternative approach would have been to disable the animation of jQuery ColorBox just for our tests but the approach we took was a much quicker win at the time.

We did realise later on that we didn’t need to write our own wait method since WebDriver has one built into the API but I guess they both do similar things so it’s not such a big problem.

Written by Mark Needham

December 13th, 2011 at 11:31 pm

Posted in Software Development

Tagged with

The 5 Whys/Root cause analysis – Douglas Squirrel

with 4 comments

At XP Day I was chatting to Benjamin Mitchell about the 5 whys exercises that we’d tried on my team and I suggested that beyond Eric Ries’ post on the subject I hadn’t come across an article/video which explained how to do it.

Benjamin mentioned that Douglas Squirrel had recently done a talk on this very subject at Skillsmatter and as with most Skillsmatter talks there’s a video of the presentation online.

Gojko wrote a post summarising the talk at the time but I was interested in seeing how a 5 whys facilitated by Douglas would compare to the ones that we’d done.

These were some of my observations/learnings:

  • Douglas started off with a similar approach to the one we tried in our last attempt whereby he listed all the initial problems across the board and then worked through them.

    One thing he did much better was ensuring that the 5 whys were covered for each problem before moving onto the next one. He described this as ‘move down, then across‘ and made the interesting observation that when you get to the real root cause (in theory the 5th why) there will be a pause and it will hurt.

    I don’t remember noticing that in any of our 5 whys which means, Douglas suggests, that ‘you[/we] are not doing it right’. In terms of actually getting to the root cause he’s probably right but you can still learn some useful things even if you don’t dig down that far.

  • He also made the suggestions that we shouldn’t follow whys which we can immediately see are not going to go anywhere – we’d be better off going down one of the other nodes which might lead us to some useful learning.

    I think we made the mistake of following some nodes which we could tell were going to go nowhere the first time that we did the exercise and ended up reaching a 5th why which was so general that we couldn’t do anything with it.

    On the other hand I think it probably takes a couple of goes at the 5 whys before you can say with certainty that following a why is going to go nowhere.

  • Another suggestion was to ensure that everyone linked with the problem being discussed is in the room, partly so that they don’t end up being made the scape goat in absentia.

    In the two exercises we’ve run we only included the people on our immediate team and we did reach a point where it was difficult to work out what the answer to some of the whys should be because the person who could answer that question wasn’t in the room.

    It does obviously make it more logistically difficult to organise the meeting, especially if you have people working in different countries.

  • Squirrel suggested then any actions that come out of the meeting should be completable in a week which helps to ensure that they’re realistic and proportionate to the problem.

    If something goes wrong once then we don’t necessarily need to make massive changes to avoid it in future, it might be sufficient to just make some small changes and then observe if things have improved.

Overall I found the talk quite useful and it was especially helpful to be able to see how a more experienced facilitator, like Douglas, was able to guide the discussion back into the framework so that it didn’t drift off.

I’m not yet convinced that we would want to run a 5 whys exercise every week which is what I’ve heard suggested before – I think the format could quickly become dull to people as with any other meeting format when used repeatedly.

Written by Mark Needham

December 10th, 2011 at 2:11 pm

Continuous Delivery: Removing manual scenarios

without comments

On the project that I’m currently working on we’re trying to move to the stage where we’d be able to deploy multiple times a week while still having a reasonable degree of confidence that the application still works.

One of the (perhaps obvious) things that we’ve had to do as a result of wanting to do this is reduce the number of manual scenarios that our QAs need to run through.

At the moment it takes a few hours of manual testing time on top of all our automated scenarios before we can put the application in production which is fine if you release infrequently but doesn’t really scale.

Following the ideas of pain driven development we delayed automating any bits of the application if we couldn’t work out how to automate them easily.

For example we have quite a few Javascript driven light boxes which pop up on various users actions and we weren’t able to test using the HTML Unit Driver in Web Driver so we created automated tests only for the non Javascript version of those features.

We’ve started work on automating these scenarios recently and although we’re having to invest some time in fiddling around with the Firefox Driver to do so, it will save us time on each deployment so it should be worth it.

My colleague Harinee showed me an interesting post by Anand Bagmar in which he describes a cost/value matrix which can help us decide whether or not to automate something.

In this case the lightbox feature was initially only used in a couple of places and we realised it would be quite difficult to test it compared to our current approach so it fitted in the ‘Naaaah’ section (Low value, High cost).

Over time it has been introduced in more and more places so it’s become more important that we find a way to automatically test it since the application can fail in many more places if it doesn’t work.

It’s therefore moved into section #2 (High value, High cost) and become a candidate for automation.

Although not directly related to manual scenarios, one thing we haven’t quite worked out yet is what the balance should be between getting something into production and ensuring that it isn’t full of bugs.

This is also something which James Birchler addresses in his post about IMVU’s approach to continuous deployment:

We’re definitely not perfect: with constrained QA resources and a persistent drive by the team to deliver value to customers quickly, we do often ship bugs into production or deliver features that are imperfect. The great thing is that we know right away what our customers want us to do about it, and we keep on iterating.

At the moment the application is only being beta tested by a small number of users so we’re veering towards pushing something even if it’s not perfect but as we open it up to more people I imagine it’ll become a bit more stringent.

At times it’s quite painful working out what you need to change in your approach to make it possible to release frequently but it’s cool to see the progress we’re making as we get better at doing so.

Written by Mark Needham

December 5th, 2011 at 11:13 pm