I was testing the BlockProximityFusion class which is used to merge together adjacent text blocks.

I started off calling that class like this:

import static java.util.Arrays.asList;
 
@Test
public void willCallBlockProximityFustion() throws Exception {    
    TextDocument document = new TextDocument(asList(contentBlock("some words"), contentBlock("followed by more words")));
    BlockProximityFusion.MAX_DISTANCE_1.process(document);
}
 
private TextBlock contentBlock(String words) {
    TextBlock textBlock = new TextBlock(words, new BitSet(), wordCount(words), 0, 0, 0, 0);
    textBlock.setIsContent(true);
    return textBlock;
}

Which blows up like this:

java.lang.UnsupportedOperationException
	at java.util.AbstractList.remove(AbstractList.java:144)
	at java.util.AbstractList$Itr.remove(AbstractList.java:360)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusion.process(BlockProximityFusion.java:115)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusionTest.willCallBlockProximityFustion(BlockProximityFusionTest.java:63)

The code around that area is trying to remove an element from an iterator…

113
114
115
116
117

                if (ok) {
                    prevBlock.mergeNext(block);
                    it.remove();
                    changes = true;
                } else {

…which was created from the list that we passed into the constructor of TextDocument:

98

        for (Iterator<TextBlock> it = textBlocks.listIterator(offset); it

The remove method is not implemented on the list created by ‘Arrays.asList’ which is weird since I thought it created an ArrayList which does implement remove!

I’ve now learnt that the ArrayList created by ‘Arrays.asList’ is actually a private inner class of Arrays and doesn’t implement the remove method!

Who knew…

As a result we delegate from Scala code to the system unzip command like so:

def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We ran into a problem where the unzipping process was hanging and executing ‘ps’ showed us that the ‘unzip’ process was stuck in the ‘pipe_w’ (pipe waiting) state which suggested that it was waiting for some sort of input.

After a bit of googling Duncan found this blog which explained that we needed to process the output stream from our process otherwise it might end up hanging

a.k.a. RTFM:

The Runtime.exec methods may not work well for special processes on certain native platforms, such as native windowing processes, daemon processes, Win16/DOS processes on Microsoft Windows, or shell scripts.

The created subprocess does not have its own terminal or console. All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (Process.getOutputStream(), Process.getInputStream(), Process.getErrorStream()).

The parent process uses these streams to feed input to and get output from the subprocess.

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

For most of the zip files we presumably hadn’t been reaching the limit of the buffer because the list of files being sent to STDOUT by ‘unzip’ wasn’t that high.

In order to get around the problem we needed to gobble up the output stream from unzip like so:

import org.apache.commons.io.IOUtils
def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val thisVariableIsNeededToSuckDataFromUnzipDoNotRemove = "Output: " + IOUtils.readLines(process.getInputStream)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We need to do the same thing with the error stream as well in case ‘unzip’ ends up overflowing that buffer as well.

On a couple of blog posts that we came across it was suggested that we should ‘gobble up’ the output and error streams on separate threads but we weren’t sure why exactly that was considered necessary…

If anyone knows then please let me know in the comments.

It’s a command line application which is called through the main method of ‘Program’ and since there’s no dependency injection we need to be able to set System.in and System.out in order to do any testing.

My initial thinking was that it should be possible to fake System.in with the following code:

String input = "1\n9\n";
System.setIn(new ByteArrayInputStream(input.getBytes()));

This works fine when I just want to simulate one value being passed to System.in but it doesn’t work so well if I want to simulate passing more than one value because we had a BufferedReader being created each time we loop.

...
while(true) {
	...
	InputStreamReader inputStream = new InputStreamReader(System.in);
	BufferedReader reader = new BufferedReader(inputStream);
	...
}

This means that the second time System.in gets read it is empty.

Jim and I paired on the problem for a bit and came to the conclusion that we’d need to ‘stub’ the ‘read’ method of ‘InputStream’ if we wanted to be able to control exactly what was being returned by System.in.

We eventually ended up with the following StubbedInputStream:

class StubbedInputStream extends InputStream {
    private Queue<String> input;
 
    public StubbedInputStream(Queue<String> input) {
        this.input = input;
    }
 
    @Override
    public int read(byte[] bytes, int i, int i1) throws IOException {
        if(input.isEmpty()) {
            return -1;
        }
 
        int byteLocation = 0;
        for(byte b : input.remove().getBytes()) {
            bytes[byteLocation] = b;
            byteLocation++;
        }
        bytes[byteLocation] = "\n".getBytes()[0];
        return byteLocation + 1;
    }
 
	public static InputStreamBuilder stubInputStream() {
        return new InputStreamBuilder();
    }
	...
}

Which can be constructed using the following DSL:

System.setIn(stubInputStream().toReturn("1").then("9").atSomePoint());

The code we wrote is on github – I’m not sure that it covers every possible scenario that you might come up with but it does pass the tests that I’ve managed to come up with!

If there were gaps at the beginning or end of the sequence then that was fine but gaps in the middle of the sequence were not.

null, 1, 2, 3 => no gaps
1, 2, 3, null => no gaps
1, null, 2, 3 => gaps

The Java version looked a bit like this:

public boolean hasGaps(List<BigInteger> values) {
    Iterator<BigInteger> fromHead = values.iterator();
    while (fromHead.hasNext() && fromHead.next() == null) {
        fromHead.remove();
    }
 
    Collections.reverse(values);
 
    Iterator<BigInteger> fromTail = values.iterator();
    while (fromTail.hasNext() && fromTail.next() == null) {
        fromTail.remove();
    }
 
    return values.contains(null);
}

We take the initial list and then remove all the null values from the beginning of it, then reverse the list and remove all the values from the end.

We then check if there’s a null value and if there is then it would indicate there is indeed a gap in the list.

To write this function in Clojure we can start off by using the ‘drop-while‘ function to get rid of the trailing nil values.

I started off with this attempt:

(defn has-gaps? [list]
    let [no-nils] [drop-while #(= % nil) list]
  no-nils)

Unfortunately that gives us the following error!

Can't take value of a macro: #'clojure.core/let (NO_SOURCE_FILE:16)

It thinks we’re trying to pass around the ‘let’ macro instead of evaluating it – I forgot to put in the brackets around the ‘let’!

I fixed that with this next version:

(defn has-gaps? [list]
    (let [no-nils] [drop-while nil? list]
  no-nils))

But again, no love:

java.lang.IllegalArgumentException: let requires an even number of forms in binding vector (NO_SOURCE_FILE:23)

The way I understand it the ‘let’ macro takes in a vector of bindings as its first argument and what I’ve done here is pass in two vectors instead of one.

In the bindings vector we need to ensure that there are an even number of forms so that each symbol can be bound to an expression.

I fixed this by putting the two vectors defined above into another vector:

(defn has-gaps? [list]
    (let [[no-nils] [(drop-while nil? list)]]
  no-nils))

We can simplify that further so that we don’t have nested vectors:

(defn has-gaps? [list]
    (let [no-nils (drop-while nil? list)]
  no-nils))

The next step was to make ‘no-nils’ a function so that I could make use of that function when the list was reversed as well:

(defn has-gaps? [list]
    (let [no-nils (fn [x] (drop-while nil? x))]
  (no-nils list)))

I then wrote the rest of the function to reverse the list and then check the remaining list for nil:

(defn has-gaps? [list]
    (let [[no-nils] [(fn [x] (drop-while nil? x))]
          [nils-removed] [(fn [x] ((comp no-nils reverse no-nils) x))]]
  (some nil? (nils-removed list))))

The ‘comp‘ function can be used to compose a set of functions which is what I needed.

It seemed like the ‘nils-removed’ function wasn’t really necessary so I inlined that:

(defn has-gaps? [list]
    (let [no-nils (fn [x] (drop-while nil? x))]
  (some nil? ((comp no-nils reverse no-nils) list))))

The function can now be used like this:

user=> (has-gaps? '(1 2 3))
nil
user=> (has-gaps? '(nil 1 2 3))
nil
user=> (has-gaps? '(1 2 3 nil))
nil
user=> (has-gaps? '(1 2 nil 3))
true

I’d be intrigued to know if there’s a better way to do this.

Map

Map evaluates a high order function on all the elements in a collection and then returns a new collection containing the results of the function evaluation.

Given the numbers 1-5, return the square of each number

Java

int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    System.out.println(f(number));
}
 
private int f(int value) {
    return value*value;
}

C#

new List<int> (new[] {1, 2, 3, 4, 5}.Select(x => x*x)).ForEach(Console.WriteLine);

F#

[1..5] |> List.map (fun x -> x*x) |> List.iter (printfn "%d");;

Filter

Filter applies a predicate against all of the elements in a collection and then returns a collection of elements which matched the predicate.

Given the numbers 1-5, print out only the numbers greater than 3:

Java

int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    f(number);
}
 
private void f(int value) {
    if(value > 3) {
        System.out.println(value);
    }
}

C#

new List<int> { 1,2,3,4,5}.FindAll(x => x > 3).ForEach(Console.WriteLine);

F#

[1..5] |> List.filter (fun x -> x > 3) |> List.iter (printfn "%d");;

Reduce

Reduce applies a high order function against all the elements in a collection and then returns a single result.

Given a list of numbers 1-5, add them all together and print out the answer

Java

int sum = 0;
int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    sum += number;
}
System.out.println(sum);

C#

Console.WriteLine(new[] {1, 2, 3, 4, 5}.Aggregate(0, (accumulator, x) => accumulator + x));

F#

[1..5] |> List.fold_left (+) 0 |> printfn "%d";;

In Summary

I was surprised that we could achieve these results in relatively few lines of Java. The C# and F# versions are still more concise but the Java version isn’t too bad. The Apache Commons Library has a class which allows you to write these in a functional way but the need to use anonymous methods means it’s not as clean as what you can achieve in C# and F#.

I think there is still a bit of a mindset switch to make from thinking procedurally about these things to thinking in a way that allows you to make the most of functional programming concepts.

Keeping the code as declarative as possible and reducing the amount of state in our code are the most obvious things I have learned so far from playing with F#.

The Format

We used the Randori approach again with 6 people participating for the majority of the session. Everyone paired with everyone else at least once and sometimes a couple of times.

We had the pair driving at the front of the room and everyone else further back to stop the tendency of observers to whiteboard stuff.

What We Learnt

• Modeling games is really good for practicing design skills. Most people had played the game so we had domain experts who could use their knowledge to help drive out the API of the various classes. We didn’t get to the scoring part of the game in the time available but it was quite cool to see our code with all the terms detailed in Wikipedia’s entry on the term.
• We managed to drive the design much more effectively than we have done on previous sessions. The flexibility to move between classes depending on where it made most sense to test from next was finally there and we didn’t end up with the problem we’ve had on previous sessions where we ended up with coarsely grained tests and then tried to code the whole application in one go.
• It was quite painful for me personally having to manually perform operations on collections in Java rather than having the selection of functional operators that are available in C# 3.0.
• It wasn’t a new learning but I’ve noticed in my project work that I’ve become a lot more keen to keep the steps really small – there is a bit of pressure on you to do this in a dojo situation and I think it’s just continued over from there. Every time I try to be too clever and take a big step something inevitably doesn’t work and I end up doing the small steps anyway. It’s also a lot of fun coding in this type of environment and watching how others approach problems and how they pair with each other. If you get a chance to attend a dojo I think it’d definitely be worthwhile.

Other Dojo Thoughts

• Some ideas for future coding dojos that we discussed were:
  • Concurrency – using the Retlang/Jetlang libraries
  • Do some stuff with Web Driver
  • Modeling games
  • Taking an open source project and refactoring it
• I notice there are a couple of sessions of coding/coding dojos planned for Jason Gorman’s Software Craftsmanship conference. It will be interesting to see how those work out, especially if there are high numbers of participants. We’ve always had a fairly small number of people involved which I think has helped to keep everyone involved. I’m not convinced it would be effective with many more participants.

Previously, as far as I’m aware, it was only possible to parameterise tests by using the TestNG library which has some nice ideas around grouping tests but had horrible reporting the last time I used it.

To create parameterisable tests using Theories we need to write some code like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import org.junit.Test;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import org.junit.runner.RunWith;
 
@RunWith(Theories.class)
public class SomeTest {
	@Theory
	public void testTheNewTheoriesStuff(int value) {
		// test which involves int value	
	}
 
	public static @DataPoints int[] values = {1,2,3,4,5};
}

The ‘testTheNewTheoriesStuff’ Theory is then executed with each of the values defined in the values array decorated with the @DataPoints annotation.

The error message reported for a failure is reasonably good and makes it quite easy to figure out which one of the data points causes the problem.

An example error message for an assertion which failed inside a theory might look like this:

org.junit.experimental.theories.internal.ParameterizedAssertionError: testTheNewTheoriesStuff(values[1])

It’s 0 indexed so this error message tells us that there was an error when running the theory with the 2nd data point, therefore allowing us to go and work out why that’s the case and fix it.

This approach is actually particularly useful for testing the scope in which classes we pull from a dependency injection container are available from in our application.

Another potential use for this would be to test the edge cases of our classes – perhaps this would work best if we can randomise the data it uses.

This seems to be more the approach Microsoft are taking with the the Pex framework, a similar idea in the .NET space.

Nearly all the objects in our system are being created by Pico Container so we decided that writing an interceptor that hooked into Pico Container would be the easiest way to intercept and log any exceptions throw from our code.

Our initial Googling led us to the AOP Style Interception page on the Pico website which detailed how we could create a static proxy for a class that we put in the container.

The code to do this was as follows:

1
2
3
4
5
6
7
8

        DefaultPicoContainer pico = new DefaultPicoContainer(new Intercepting());
        pico.addComponent(Interceptable.class, ConcreteInterceptable.class);
        Intercepted intercepted = pico.getComponentAdapter(Interceptable.class).findAdapterOfType(Intercepted.class);
        intercepted.addPreInvocation(Interceptable.class, new InterceptableReporter(intercepted.getController()));
        intercepted.addPostInvocation(Interceptable.class, new InterceptableReporter(intercepted.getController()));
 
        Interceptable a1 = pico.getComponent(Interceptable.class);
        a1.methodThatThrowsException();

1
2
3

public interface Interceptable {
    void methodThatThrowsException();
}

1
2
3
4
5
6
7
8
9
10
11
12
13

    private static class InterceptableReporter implements Interceptable {
        private Intercepted.Controller controller;
 
        public InterceptableReporter(Intercepted.Controller controller) {
            this.controller = controller;
        }
 
 
        public void methodThatThrowsException() {
            System.out.println("error happened");
 
        }
    }

While this approach works, the problem is that we need to define an individual proxy for every class that we want to intercept. It works as a strategy if we just need to intercept a few classes but not on a larger scale.

Luckily it is possible to create a dynamic proxy on the container so that we can intercept all the objects without having to create a static proxy for each one.

The code to do this was as follows:

1
2
3
4
5

        DefaultPicoContainer pico = new DefaultPicoContainer(new LoggingAwareByDefault());
        pico.addComponent(Interceptable.class, ConcreteInterceptable.class);
 
        Interceptable interceptable = pico.getComponent(Interceptable.class);
        interceptable.methodThatThrowsException();

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import org.apache.commons.logging.LogFactory;
import org.picocontainer.Characteristics;
import org.picocontainer.ComponentAdapter;
import org.picocontainer.ComponentMonitor;
import org.picocontainer.LifecycleStrategy;
import org.picocontainer.Parameter;
import org.picocontainer.behaviors.AbstractBehaviorFactory;
 
import java.util.Properties;
 
public class LoggingAwareByDefault extends AbstractBehaviorFactory {
    private static final String DO_NOT_LOG_NAME = "support-team-opt-out";
    public static final Properties DO_NOT_LOG = Characteristics
            .immutable(DO_NOT_LOG_NAME, Characteristics.TRUE);
 
 
    public <T> ComponentAdapter<T> createComponentAdapter(ComponentMonitor componentMonitor,
                                                          LifecycleStrategy lifecycleStrategy,
                                                          Properties componentProperties,
                                                          Object componentKey, Class<T> componentImplementation,
                                                          Parameter... parameters) {
        if (removePropertiesIfPresent(componentProperties, DO_NOT_LOG)) {
            return super.createComponentAdapter(componentMonitor, lifecycleStrategy, componentProperties, componentKey,
                    componentImplementation, parameters);
        } else {
            return new LoggingAware<T>(super.createComponentAdapter(componentMonitor,
                    lifecycleStrategy, componentProperties, componentKey,
                    componentImplementation, parameters));
        }
 
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import org.apache.commons.logging.Log;
import org.picocontainer.ComponentAdapter;
import org.picocontainer.ComponentMonitor;
import org.picocontainer.PicoContainer;
import org.picocontainer.behaviors.HiddenImplementation;
 
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
 
public class LoggingAware<T> extends HiddenImplementation {
    public LoggingAware(ComponentAdapter delegate) {
        super(delegate);
    }
 
    protected Object invokeMethod(Object componentInstance, Method method, Object[] args, PicoContainer container)
            throws Throwable {
        ComponentMonitor componentMonitor = currentMonitor();
        try {
            componentMonitor.invoking(container, this, method, componentInstance);
            long startTime = System.currentTimeMillis();
            Object object = method.invoke(componentInstance, args);
            componentMonitor.invoked(container,
                                     this,
                                     method, componentInstance, System.currentTimeMillis() - startTime);
            return object;
        } catch (final InvocationTargetException ite) {
            componentMonitor.invocationFailed(method, componentInstance, ite);
 
            // log the error
 
            throw ite.getTargetException();
        }
 
    }
}

From what I recall from looking at the source code I think in order to create a proxy around an object it needs to implement an interface otherwise the proxy will not be created.

There are several reasons why I find the Hamcrest matcher approach to be more productive – it’s more flexible, more expressive and when an assertion fails we have a much better idea about why it has failed than if we use a JUnit assertion for example.

This applies especially when we get a test failing as part of the build as compared to running a test from the IDE where the source code is close at hand and non descriptive error messages may not be such a problem.

It therefore makes sense when writing custom Hamcrest matchers to ensure that we do indeed provide a clear error message so that it is obvious how to fix the test.

The convention seems to be that we should first state the static method name of the matcher and then in brackets list the expected arguments.

To give an example from a matcher we wrote yesterday:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import org.hamcrest.Description;
import org.hamcrest.Factory;
import org.hamcrest.TypeSafeMatcher;
 
public class ContainsAllOf<T> extends TypeSafeMatcher<T> {
    private String[] messages;
 
    public ContainsAllOf(String... messages) {
        this.messages = messages;
    }
 
    public void describeTo(Description description) {
        description.appendText("containsAllOf(");
        for (String message : messages) {
            description.appendText(",");
            description.appendValue(message);
        }
        description.appendText(")");
    }
 
    @Factory
    public static <T> ContainsAllOf containsAllOf(String... messages) {
        return new ContainsAllOf(messages);
    }
 
    public boolean matchesSafely(T t) {
        return contains(t, messages);
    }
 
    private boolean contains(T t, String[] messages) {
        boolean containsAllMessages = true;
        for (String message : messages) {
            if (!t.toString().contains(message)) {
                return false;
            }
        }
        return containsAllMessages;
    }
 
}

If we call this in our test with a value that doesn’t exist:

assertThat("mark's cool message", containsAllOf("mark", "cool", "message", "notThere"));

Running the test results in the following error:

java.lang.AssertionError: 
Expected: containsAllOf(,"mark","cool","message","notThere")
     got: "mark's cool message"

We can easily see what the problem is and how to go about fixing it, which I feel is the most important thing when it comes to test assertions.

For me an exception is about a situation which is exceptional, and if we know that there is a possibility of it happening and even have that possibility defined in our code then it doesn’t seem all that exceptional to me.

Having said that they do at least provide information which you can’t help but notice about what can go wrong when you make a call to a particular method.

The problem is that often these checked exceptions just get passed on – i.e. not handled – until we end up with an exception on the page the user sees which is completely irrelevant to the action they are trying to undertake.

To give an example, we have been using the OGNL library to hydrate some objects for testing using the builder pattern.

We have something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public class FooBuilder {
    private String bar;
 
    public FooBuilder setBar(String bar) {
        this.bar = bar;
        return this;
    }
 
    public Foo toFoo() {
        Foo foo = new Foo();
        setValue(foo, "bar", bar);
        return foo;
    }
 
    protected void setValue(Object object, String propertyName, Object propertyValue) {
        try {
            OgnlWrapper.setValue(object, propertyName, propertyValue);
        } catch (OgnlException e) {
            throw new RuntimeException(e);
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import ognl.DefaultMemberAccess;
import ognl.MemberAccess;
import ognl.Ognl;
import ognl.OgnlContext;
import ognl.OgnlException;
 
public class OgnlWrapper {
 
    public static void setValue(Object object, String propertyName, Object propertyValue) throws OgnlException {
        Ognl.setValue(propertyName, createOgnlContext(), object, propertyValue);
    }
 
    private static OgnlContext createOgnlContext() {
        MemberAccess memberAccess = new DefaultMemberAccess(true);
        OgnlContext ognlContext = new OgnlContext();
        ognlContext.setMemberAccess(memberAccess);
        return ognlContext;
    }
}

We can then build an instance of ‘Foo’ like so:

Foo foo = new FooBuilder().setBar("barValue").toFoo();

What is interesting here is not the OGNL library in itself but the checked ‘OgnlException’ which the ‘Ognl.setValue(…)’ method defines.

If I am using the FooBuilder I don’t care how the Foo object is created, all I care is that I get it. Therefore we don’t want to bubble the implementation details of how we are creating the object upwards.

I only care about the OgnlException if I am calling the OgnlWrapper and therefore that is where the exception should be caught and then rethrown as a Runtime exception.

I like to refer to this area of OgnlWrapper callees as being a bounded context – that exception should only be applicable in that particular area and beyond that it should not exist.

Doing this allows us more flexibility around the way we implement things. If I decide in the future to use a different library instead of OGNL to do the same job I don’t need to worry that the callees of FooBuilder will all need to be updated. I can just make the change inside FooBuilder and that’s it!