Mark Needham

Thoughts on Software Development

A new found respect for acceptance tests

with 8 comments

On the project that I’ve been working on over the past few months one of the key benefits of the application was its ability to perform various calculations based on user input.

In order to check that these calculators are producing the correct outputs we created a series of acceptance tests that ran directly against one of the objects in the system.

We did this by defining the inputs and expected outputs for each scenario in an excel spreadsheet which we converted into a CSV file before reading that into an NUnit test.

It looked roughly like this:

acceptance-tests.jpg

We found that testing the calculations like this gave us a quicker feedback cycle than testing them from UI tests both in terms of the time taken to run the tests and the fact that we were able to narrow in on problematic areas of the code more quickly.

As I mentioned on a previous post we’ve been trying to move the creation of the calculators away from the ‘CalculatorProvider’ and ‘CalculatorFactory’ so that they’re all created in one place based on a DSL which describes the data required to initialise a calculator.

In order to introduce this DSL into the code base these acceptance tests acted as our safety net as we pulled out the existing code and replaced it with the new DSL.

acceptance-tests-part2.jpg

We had to completely rewrite the ‘CalculationService’ unit tests so those unit tests didn’t provide us much protection while we made the changes I described above.

The acceptance tests, on the other hand, were invaluable and saved us from incorrectly changing the code even when we were certain we’d taken such small steps along the way that we couldn’t possibly have made a mistake.

This is certainly an approach I’d use again in a similar situation although it could probably be improved my removing the step where we convert the data from the spreadsheet to CSV file.

Be Sociable, Share!

Written by Mark Needham

July 11th, 2010 at 5:08 pm

Posted in Testing

Tagged with ,

  • Pingback: Tweets that mention A new found respect for acceptance tests at Mark Needham -- Topsy.com()

  • Pingback: Twitted by philip_schwarz()

  • Pingback: The Morning Brew - Chris Alcock()

  • Hello Mark,

    To me, the natural approach to software development is to submit code to implacable ‘wide-and-wild’ refactorings until it matures to the point in which the ‘senses are pleased’. No progress without the protection of a reassuring safety net.

    I understand acceptance checks as sanity checks concerned with the detection of large-grained deviations from the system’s intended goals (ultimately, its raison d’etre), allowing quick recalibration while things can still be undone/fixed in reasonable terms.

    You say that acceptance tests are invaluable even in situations in which we are fooled to believe that ‘no harm can result from such a small change’. This is a perfect statement! I think this is one of those ‘obvious’ truths that nonetheless requires people some time and experience until it is fully appreciated.

    Just wanted to add that in my view unit tests differ from acceptance tests in the way that they are indeed ‘more’ disposable in nature. They usually do not age well and, along time, tend to be discarded or commented out as a source of documentation. Their use is frequently more ad-hoc and temporary, although of course no less important.

  • Hi Mark,

    Thanks for sharing your practices. I’ve used Excel spreadsheets to drive tests of some complex calculations in the past, and even though it could have been considered almost as unit tests (testing one single service), they were still acceptance tests since the business analyst was able to “validate” them (still with help from developers anyway).

    I was using Apache POI to read straight from .xls files. Now I’d have a look at the various BDD frameworks as well for that kind of data-driven testing.

  • @cyrille – yeh we were able to validate the spreadsheet with the business owner who had the most knowledge around that area of the application so having it in a spreadsheet was useful to us for the same reason that you describe.

    We did originally look at using Ruby via IronRuby but noone knew it well enough on the team so we didn’t go with that.

    The other benefit of the spreadsheet is that for some of the calculations we had the formulas in the spreadsheet so it was effectively a double booking system for ensuring that we had worked them out correctly!

  • Pingback: TDD: I hate deleting unit tests at Mark Needham()

  • Pingback: Technical Debt around release time at Mark Needham()