Mark Needham

Thoughts on Software Development

Testing End Points: Integration tests vs Contract tests

with 3 comments

We recently changed the way that we test against our main integration point on the project I’ve been working on so that in our tests we retrieve the service object from our dependency injection container instead of ‘newing’ one up.

Our tests therefore went from looking like this:

[Test]
public void ShouldTestSomeService()
{
	var someService = new SomeService();
	// and so on
}

To something more like this:

[Test]
public void ShouldTestSomeService()
{
	var someService = UnityFactory.Container.Resolve<ISomeService>();
	// and so on
}

This actually happened as a side effect of another change we made to inject users into our system via our dependency injection container.

We have some ‘authenticated services’ which require the request to contain a SAML token for a valid user so it seemed to make sense to use the container in the tests instead of having to duplicate this piece of behaviour for every test.

We needed to add our fake authorised user into the container for our tests but apart from this the container being used is the same as the one being used in the production code.

Our tests are therefore now calling the services in a way which is much closer to the way that they are called in the code than was previously the case.

I think this is good as it was previously possible to have the tests working but then have a problem calling the services in production because something in the container wasn’t configured properly.

The downside is that these tests now have more room for failure than they did previously and they are not just testing the end point which was their original purpose.

In a way what we have done is convert these tests from being contract tests to integration tests.

I like the new way but I’m not completely convinced that it’s a better approach.

Written by Mark Needham

October 25th, 2009 at 12:04 am

Posted in Testing

Tagged with

  • http://www.orfjackal.net Esko Luontola

    How long does it take to run the tests?

    For me, because of using TDD, it’s important to be able to run the tests many times per minute, after every minor change. Otherwise it would reduce my productivity.

    In one project where I’m using Guice as the DI container, some of the tests are wired together using Guice, but the majority of the tests use manual DI and only initialize the portion under test. The tests which use the DI container take maybe 5-10ms longer to run, because of the container initialization (in total the project has right now 437 tests which run in 2.2 seconds). I’m trying to avoid using a DI container in the tests, whenever possible, because it would slow down the test suite more than very focused tests with manual DI.

  • http://www.markhneedham.com Mark Needham

    That’s an interesting observation about the run time, hadn’t thought about that actually.

    The tests I’m describing take around 10 minutes to run – they are all making calls across the network to a service layer and then checking that the responses of those calls are as expected.

    We actually do some other tests with the DI container as well just to check that our controllers are correctly wired up. Haven’t particularly noticed that they run more slowly but might be something to look at.

  • http://www.orfjackal.net Esko Luontola

    J.B. Rainsberger talks about integration tests in http://www.infoq.com/presentations/integration-tests-scam and the time to run them and their fragility are the main arguments against them. He also points out, that it’s best to test the system’s “basic correctness” using focused/unit tests. Integration tests should be used for things like scalability, reliability, security etc. and not for basic correctness.

    I also have some integration tests to make sure that the DI container is wired up correctly. Some such tests are necessary.

    I just measured it with System.nanoTime(), and initializing Guice with a couple of modules takes 5 ms. It’s not very bad, compared to how slow tests using a database would be, and since it’s CPU bound it can be speeded up by parallelizing the test execution on multiple cores. But in any case it’s many times slower than tests which execute in less than 1 ms. If there are lots of such tests then it will add up.