JUnit Theories: First Thoughts
One of my favourite additions to JUnit 4.4 was the @Theory annotation which allows us to write parameterised tests rather than having to recreate the same test multiple times with different data values or creating one test and iterating through our own collection of data values.
Previously, as far as I'm aware, it was only possible to parameterise tests by using the TestNG library which has some nice ideas around grouping tests but had horrible reporting the last time I used it.
To create parameterisable tests using Theories we need to write some code like the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import org.junit.Test; import org.junit.experimental.theories.DataPoints; import org.junit.experimental.theories.Theories; import org.junit.experimental.theories.Theory; import org.junit.runner.RunWith; @RunWith(Theories.class) public class SomeTest { @Theory public void testTheNewTheoriesStuff(int value) { // test which involves int value } public static @DataPoints int[] values = {1,2,3,4,5}; } |
The 'testTheNewTheoriesStuff' Theory is then executed with each of the values defined in the values array decorated with the @DataPoints annotation.
The error message reported for a failure is reasonably good and makes it quite easy to figure out which one of the data points causes the problem.
An example error message for an assertion which failed inside a theory might look like this:
org.junit.experimental.theories.internal.ParameterizedAssertionError: testTheNewTheoriesStuff(values[1])
It's 0 indexed so this error message tells us that there was an error when running the theory with the 2nd data point, therefore allowing us to go and work out why that's the case and fix it.
This approach is actually particularly useful for testing the scope in which classes we pull from a dependency injection container are available from in our application.
Another potential use for this would be to test the edge cases of our classes – perhaps this would work best if we can randomise the data it uses.
This seems to be more the approach Microsoft are taking with the the Pex framework, a similar idea in the .NET space.
Thanks.
I wrote a test program using @RunWith(Theories.class) and had few static public variables that act has @Datapoint. I had some parameterized methods and used @Test (instead of @Theory) annotation.
@RunWith(Theories.class)
public class TestCalculatorUsingDataPoints {
Calculator calculator = null;
@DataPoint
public static int first_number = 10;
@DataPoint
public static int second_number = 20;
@Before
public void setUp() {
calculator = new Calculator();
}
@After
public void tearDown() {
calculator = null;
}
@Test
public void add(int number1, int number2) {
int actualResult = calculator.add(number1, number2);
//assertEquals(expectedResult, actualResult);
int expectedResult = number1 + number2;
System.out.println("add " + number1 + " " + number2);
assertThat(actualResult, is(expectedResult));
}
}
JUnit did not throw any error that @Theory should be used. JUnit recognized the @Test methods and call the same method with each datapoint.
Result:
add 10 10
add 10 20
add 20 10
add 20 20
Isn't weird?
Sai
13 Jan 09 at 10:23 pm
Theories are my favourite feature too. I like it because it makes it possible to define tests for common properties of classes.
http://blog.bader-jene.de/?p=30
lexi
11 Feb 09 at 4:26 pm
Hi lexi ,
Can you please let us know how your test method are running without error.
I have used @RunWith(Theories.class) but my test methods are expecting @Theory , if i will annotate with @Test then i am getting errors that test methods should not have any arguments
sudhanshu
13 Oct 09 at 1:11 pm
Sorry this is not meant for lexi , this is for Sai
sudhanshu
13 Oct 09 at 1:12 pm
Hi,
About the tests with the Theories, I can't use it to do what you are saying:
The 'testTheNewTheoriesStuff' Theory is then executed with each of the values defined in the values array decorated with the @DataPoints annotation.
When I use an array of Objects, the @Test/@Theory will expect an array as parameter in the method….So if I have a array of int as DataPoints, I must use an array of int as parameter in the test….. :/
Di
7 Nov 09 at 2:54 am
Just a note on randomizing the data. This is never a good idea. If your testing boundaries, they are easy to work out but should be fixed. There are constants available for most in java. If you have your own provide boundary constants.
Random data = random failure which is very hard to track down.
Martin Harris
28 Apr 10 at 12:48 pm
@Martin: Regarding randomization of test data: Your own comment disproves your point. If you have random data and you get a failure, then that suggests that you have a flaw. Clearly, your static tests didn't isolate the flaw. Had you relied on them, you would be blissfully ignorant of the flaw.
If you have identified your test cases with adequate thoroughness, you would not need randomness. Every variant that could produce a different result is tested. Code coverage analysis helps in this regard.
But with a sufficiently complex process that you are testing, having 100% code coverage does not mean that you are bug free. Using random input data, and knowing what the correct output should be, is a form of model-based testing, and it is valid.
Another approach that is more methodical is to generate every possible permutation of inputs (assuming that there are not an infinite number of them) and doing so programmatically, and using your model to know the expected output. Then, you can generate your test cases.
Sometimes, though, you can filter this down significantly by using equivalence classes.
Still other times, knowing what kind of data may get thrown at your code is hard to predict. Analysis of production data helps there. Unfortunately, you can seldom test against ALL of production code. And testing a subset means a lot of redundant testing, and the possibility of missing important test cases. In that situation, apply equivalence classes to individual records, and divide the production data into the smallest set of records that gives you representation of every known class of data. I once did this to a nearly half-billion record data set, and the result was around 0.25%. Yes, that's 1/4th of one percent. There is a lot of redundancy in production data.
John Arrowwood
2 May 10 at 8:10 pm
I agree with much of what you say but random is still a poor way to generate the data. With the random approach you may never expose the flaw. I would prefer intelligent selected slices with bias based on production data.
I quote from http://xunitpatterns.com/Generated%20Value.html
Variation: Random Generated Value
One way to get good test coverage without spending a lot of time analyzing the behavior and generating test conditions is to use different values each time we run the tests. Using a Random Generated Value is one way to do this. While this may seem like a good idea, it makes the tests non-deterministic and can make debugging failed tests very difficult. Ideally, when a test fails, we want to be able to repeat that test failure on demand. This can be made possible by logging the Random Generated Value as the test is run and showing it as part of the test failure. We then need to find a way to force the test to use that value when we are troubleshooting the failed test. In most cases all this is too much effort for the potential benefit but when we need this technique, we really need it.
@JohnArrowwood
10 May 10 at 1:28 pm