Mark Needham

Thoughts on Software Development

TDD: I hate deleting unit tests

with 10 comments

Following on from my post about the value we found in acceptance tests on our project when doing a large scale refactoring I had an interesting discussion with Jak Charlton and Ben Hall about deleting unit tests when they’re no longer needed.

The following is part of our discussion:

Ben:

@JakCharlton @markhneedham a lot (not all) of the unit tests created can be deleted once the acceptance tests are passing…

Jak:

@Ben_Hall @markhneedham yep I agree, but that isn’t what TDD really advocates – its a balance, unit tests work well in some places

Me:

@Ben_Hall @JakCharlton gotta be courageous to do that.Its like you’re ripping away the safety net. Even if it might be an illusion of safety

Jak:

@markhneedham one of the XP principles … Courage 🙂

dragons2.jpg

While Jak and Ben are probably right I do find myself feeling way more anxious about deleting test code than I would deleting production code.

I think that this is mostly because when I delete production code we usually have some tests around that code so there is a degree of safety in doing so.

Deleting tests seems a bit more risky because there’s much more judgement involved in working out whether we’re removing the safety net that we created by writing those tests in the first place.

The diagram on the right hand side shows the way I see the various safety nets that we create to protect us from making breaking changes in production code.

In this case it might seem that a unit test is providing safety but it’s now an illusion of safety and in actual fact it’s barely protecting us at all.

I find it much easier to delete a unit test if it’s an obvious duplicate or if we’ve completely changed the way a piece of code works such that the test will never pass again anyway…

..and I find it more difficult to judge when we end up with tests which overlap while testing similar bits of functionality.

Do others feel like this as well or am I just being overly paranoid?

Either way does anyone have any approaches that give you ore confidence that you’re not deleting something that will come back to haunt you later?

Be Sociable, Share!

Written by Mark Needham

July 15th, 2010 at 11:15 pm

Posted in Testing

Tagged with

  • I personally like the rapid feedback that unit tests offer me over integration or acceptance tests. Acceptance tests are for the customer, unit tests are for me and my team, the developers.

    I do have a tendency to delete tests more than others on my team, but it is a matter of measuring the risk vs. the maintenance cost of the test. Code is code and it whether it is test code or implementation code, there is a cost to keeping it around. However, I generally favor keeping the unit tests around, because they’re fast and that’s what feedback is all about.

  • Whoa…

    Don’t keep the acceptance tests and delete the unit tests. Keep the unit tests and delete the acceptance tests!

    At least when I do this, my acceptance tests are not as thorough and much more expensive to run than my super fast and localized unit tests. Acceptance tests, in my mind, should be totally redundant by the time there are tests and code actually written. (in practice they aren’t, but that should be the goal)

    Either way, you still have the problem of wanting to delete unit tests are aren’t carrying their weight anymore… Dunno the many great strategies for that. Would love to hear what people suggest!

  • Hello Mark,

    The butterfly effect, or the ripple that becomes a tsunami, derives not only from tests that are removed to satisfy the minimalist god but also from collective blindness, those damned unseen paths that go directly into the dragon’s mouth.

    The test uncertainty principle: before asking if one specific test can be really removed, can we confidently state that we have all the nets we need?

    Uncertainty suggests that the more tests we have the better we are. Redundancy is better than free fall. Why take chances? Besides, unit tests and acceptance tests resemble micro and macro economic indicators, enlightening us in different ways.

    Regarding the comments of the last post, I was wondering the effort necessary to maintain the Excel spreadsheet. Making a connection to the point of this article: how much time and sweat do we want to spend with test maintenance? Unit tests are disposable, true, but if they are still working well, let them be.

    Note: retired tests can be commented them out instead of deleted, as a form of documenting past whens and whys.

  • I don’t think I actually made it clear enough that I’m not for the idea of deleting unit tests just because we have acceptance tests covering the same area of functionality – rather that was just a discussion which reminded me that in general I hate deleting unit tests!

    @Claudio – I like the thinking around test redundancy and free fall. That seems a bit like a trade off so it’d be interesting to brainstorm the factors which would lead you to favouring either approach in a given situation.

    Test maintenance vs safety is definitely an interesting topic – maybe that’s another discussion altogether! We’ve fallen at different points along the trade off ladder for this one on every project that I’ve worked on.

    @Jeremy – that’s an interesting angle! I hadn’t really thought about deleting acceptance tests. Not sure whether i hate deleting them even more or not!

    @Michael – it would be interesting to hear how you judge maintenance vs risk – that seems to be the underlying thing here.

  • Pingback: The Morning Brew - Chris Alcock » The Morning Brew #644()

  • Pingback: Tweets that mention TDD: I hate deleting unit tests at Mark Needham -- Topsy.com()

  • I suppose that some depends on what you mean by the terms “unit test” and “acceptance test”. Some use the word “acceptance tests” for end-to-end tests and others for business-logic-facing tests (would there be some better word for it?). And the same for unit tests – people mean by it isolated object tests (which execute quickly) or implementation-facing tests.

    Most of the tests I write are unit tests in the sense that I can execute all my tests quickly. (If it takes longer than 5-10 seconds to recompile and run all tests after a one-liner change, I will be seriously annoyed.) At the same time they are also acceptance tests in the sense that they have been written from the component’s user’s point of view – what features and external behaviour it provides [1].

    I very seldom throw away my tests, because I can do quite big refactorings without having to change my tests, because the behaviour provided by the system stays the same (otherwise it would not be refactoring). If I need to make architectural changes, then it usually means that some components become obsolete, in which case I may throw away the whole test class together with the implementation, because the system will not anymore need the behaviour which was defined by that test class. But if the behaviour is still needed (the implementation just must be rewritten), then I may be able to reuse at least the test names, and just rewrite the test body together with the new implementation [1].

    Nowadays, after reading “Growing Object-Oriented Software, Guided by Tests”, I also write some end-to-end tests which I use to make sure that all the pieces, especially external systems, fit together. Running the end-to-end tests is slow (each test is launched in its own process), so running them while refactoring is not very feasible, and I don’t rely on them for the basic correctness [2] of my system. Also they can’t test all the corner cases, which are better covered at the object level tests. I tend to write my end-to-end tests using example-style instead of specification-style [1], so that the end-to-end tests cover some scenarios through the system, but they don’t attempt to document in detail how the system works.

    [1] http://blog.orfjackal.net/2010/02/three-styles-of-naming-tests.html
    [2] http://www.jbrains.ca/permalink/253

  • Thomas Eyde

    I hate deleting unit tests, too.

    I delete unit tests which fails or don’t compile when I know they test the wrong behaviour, or the feature no longer exists.

    When I stumble over a passing test which I know is no longer valid, I create a folder “Obsolete” and move the test class over there. The nest time something in this class breaks, I can save myself some energy from investigating the issue and just delete it.

  • Hi Mark,

    Just returned to this topic because I have remembered of an old time debate about physical versus logical deletes in databases. As you know, people advocates that data should never be removed. “Flag it, don’t discard it”. Paranoia? Good sense?

    Answering your question above, I would always be OK with redundancy and never worry too much with achieving optimal Occarian reduced sets. Only when tests fail, either because refactorings ‘rippled’ or because the truth of tests are no longer valid, I would return my attention to the safety net. And then I would have no issues with deletion, after all, as you say, we must have courage. But I would be inclined to preserve those tests that work as memory of design decisions and which document design evolution.

    Still touching the spreadsheet topic, I would keep refactoring energies aimed at production code, not at the test apparatus. “Laziness and simplicity”.

    Again I am sliding into the maintenance vs safety conversation, so at the end I think this is my key driving factor: the amount of effort involved. Something like “it it does not smell, leave it”.

    Otherwise, taking it to the limit, I am afraid of ending up building safety nets for safety nets, in an endless Escherian loop.

    Note: Escherian and Occarian reflect a programmer’s mood when there is no caffeine available. Yet.

  • @Claudio – that’s interesting, I hadn’t considered that the tests would prove useful for describing design evolution.

    So data wise yeh I’ve heard of that idea and the trade off is that you’re using up more space by storing that data – probably not a big deal in most cases.

    Unit test wise you’re potentially taking up programmer time reading tests which may not be that useful anymore. I wonder if it’s value to flag those tests in some way to indicate that they were from an earlier design but maybe aren’t 100% necessary now.