Mark Needham

I’ve been playing around with the boilerpipe code base by writing some tests around it to check my understanding but ran into an interesting problem using java.util.Arrays.asList to pass a list into one of the functions.

I was testing the BlockProximityFusion class which is used to merge together adjacent text blocks.

I started off calling that class like this:

import static java.util.Arrays.asList;
 
@Test
public void willCallBlockProximityFustion() throws Exception {    
    TextDocument document = new TextDocument(asList(contentBlock("some words"), contentBlock("followed by more words")));
    BlockProximityFusion.MAX_DISTANCE_1.process(document);
}
 
private TextBlock contentBlock(String words) {
    TextBlock textBlock = new TextBlock(words, new BitSet(), wordCount(words), 0, 0, 0, 0);
    textBlock.setIsContent(true);
    return textBlock;
}

Which blows up like this:

java.lang.UnsupportedOperationException
	at java.util.AbstractList.remove(AbstractList.java:144)
	at java.util.AbstractList$Itr.remove(AbstractList.java:360)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusion.process(BlockProximityFusion.java:115)
	at de.l3s.boilerpipe.filters.heuristics.BlockProximityFusionTest.willCallBlockProximityFustion(BlockProximityFusionTest.java:63)

The code around that area is trying to remove an element from an iterator…

113
114
115
116
117

                if (ok) {
                    prevBlock.mergeNext(block);
                    it.remove();
                    changes = true;
                } else {

…which was created from the list that we passed into the constructor of TextDocument:

98

        for (Iterator<TextBlock> it = textBlocks.listIterator(offset); it

The remove method is not implemented on the list created by ‘Arrays.asList’ which is weird since I thought it created an ArrayList which does implement remove!

I’ve now learnt that the ArrayList created by ‘Arrays.asList’ is actually a private inner class of Arrays and doesn’t implement the remove method!

Who knew…

On the system that I’m currently working on we have a data ingestion process which needs to take zip files, unzip them and then import their contents into the database.

As a result we delegate from Scala code to the system unzip command like so:

def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We ran into a problem where the unzipping process was hanging and executing ‘ps’ showed us that the ‘unzip’ process was stuck in the ‘pipe_w’ (pipe waiting) state which suggested that it was waiting for some sort of input.

After a bit of googling Duncan found this blog which explained that we needed to process the output stream from our process otherwise it might end up hanging

a.k.a. RTFM:

The Runtime.exec methods may not work well for special processes on certain native platforms, such as native windowing processes, daemon processes, Win16/DOS processes on Microsoft Windows, or shell scripts.

The created subprocess does not have its own terminal or console. All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (Process.getOutputStream(), Process.getInputStream(), Process.getErrorStream()).

The parent process uses these streams to feed input to and get output from the subprocess.

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

For most of the zip files we presumably hadn’t been reaching the limit of the buffer because the list of files being sent to STDOUT by ‘unzip’ wasn’t that high.

In order to get around the problem we needed to gobble up the output stream from unzip like so:

import org.apache.commons.io.IOUtils
def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val thisVariableIsNeededToSuckDataFromUnzipDoNotRemove = "Output: " + IOUtils.readLines(process.getInputStream)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We need to do the same thing with the error stream as well in case ‘unzip’ ends up overflowing that buffer as well.

On a couple of blog posts that we came across it was suggested that we should ‘gobble up’ the output and error streams on separate threads but we weren’t sure why exactly that was considered necessary…

If anyone knows then please let me know in the comments.

We ran a refactoring dojo a couple of days ago at ThoughtWorks University and in preparation I wrote some system level tests around the coding problem that we were going to use during the session.

It’s a command line application which is called through the main method of ‘Program’ and since there’s no dependency injection we need to be able to set System.in and System.out in order to do any testing.

My initial thinking was that it should be possible to fake System.in with the following code:

String input = "1\n9\n";
System.setIn(new ByteArrayInputStream(input.getBytes()));

This works fine when I just want to simulate one value being passed to System.in but it doesn’t work so well if I want to simulate passing more than one value because we had a BufferedReader being created each time we loop.

...
while(true) {
	...
	InputStreamReader inputStream = new InputStreamReader(System.in);
	BufferedReader reader = new BufferedReader(inputStream);
	...
}

This means that the second time System.in gets read it is empty.

Jim and I paired on the problem for a bit and came to the conclusion that we’d need to ‘stub’ the ‘read’ method of ‘InputStream’ if we wanted to be able to control exactly what was being returned by System.in.

We eventually ended up with the following StubbedInputStream:

class StubbedInputStream extends InputStream {
    private Queue<String> input;
 
    public StubbedInputStream(Queue<String> input) {
        this.input = input;
    }
 
    @Override
    public int read(byte[] bytes, int i, int i1) throws IOException {
        if(input.isEmpty()) {
            return -1;
        }
 
        int byteLocation = 0;
        for(byte b : input.remove().getBytes()) {
            bytes[byteLocation] = b;
            byteLocation++;
        }
        bytes[byteLocation] = "\n".getBytes()[0];
        return byteLocation + 1;
    }
 
	public static InputStreamBuilder stubInputStream() {
        return new InputStreamBuilder();
    }
	...
}

Which can be constructed using the following DSL:

System.setIn(stubInputStream().toReturn("1").then("9").atSomePoint());

The code we wrote is on github – I’m not sure that it covers every possible scenario that you might come up with but it does pass the tests that I’ve managed to come up with!

A function that we had to write in Java on a project that I worked on recently needed to indicate whether there was a gap in a series of data points or not.

If there were gaps at the beginning or end of the sequence then that was fine but gaps in the middle of the sequence were not.

null, 1, 2, 3 => no gaps
1, 2, 3, null => no gaps
1, null, 2, 3 => gaps

The Java version looked a bit like this:

public boolean hasGaps(List<BigInteger> values) {
    Iterator<BigInteger> fromHead = values.iterator();
    while (fromHead.hasNext() && fromHead.next() == null) {
        fromHead.remove();
    }
 
    Collections.reverse(values);
 
    Iterator<BigInteger> fromTail = values.iterator();
    while (fromTail.hasNext() && fromTail.next() == null) {
        fromTail.remove();
    }
 
    return values.contains(null);
}

We take the initial list and then remove all the null values from the beginning of it, then reverse the list and remove all the values from the end.

We then check if there’s a null value and if there is then it would indicate there is indeed a gap in the list.

To write this function in Clojure we can start off by using the ‘drop-while‘ function to get rid of the trailing nil values.

I started off with this attempt:

(defn has-gaps? [list]
    let [no-nils] [drop-while #(= % nil) list]
  no-nils)

Unfortunately that gives us the following error!

Can't take value of a macro: #'clojure.core/let (NO_SOURCE_FILE:16)

It thinks we’re trying to pass around the ‘let’ macro instead of evaluating it – I forgot to put in the brackets around the ‘let’!

I fixed that with this next version:

(defn has-gaps? [list]
    (let [no-nils] [drop-while nil? list]
  no-nils))

But again, no love:

java.lang.IllegalArgumentException: let requires an even number of forms in binding vector (NO_SOURCE_FILE:23)

The way I understand it the ‘let’ macro takes in a vector of bindings as its first argument and what I’ve done here is pass in two vectors instead of one.

In the bindings vector we need to ensure that there are an even number of forms so that each symbol can be bound to an expression.

I fixed this by putting the two vectors defined above into another vector:

(defn has-gaps? [list]
    (let [[no-nils] [(drop-while nil? list)]]
  no-nils))

We can simplify that further so that we don’t have nested vectors:

(defn has-gaps? [list]
    (let [no-nils (drop-while nil? list)]
  no-nils))

The next step was to make ‘no-nils’ a function so that I could make use of that function when the list was reversed as well:

(defn has-gaps? [list]
    (let [no-nils (fn [x] (drop-while nil? x))]
  (no-nils list)))

I then wrote the rest of the function to reverse the list and then check the remaining list for nil:

(defn has-gaps? [list]
    (let [[no-nils] [(fn [x] (drop-while nil? x))]
          [nils-removed] [(fn [x] ((comp no-nils reverse no-nils) x))]]
  (some nil? (nils-removed list))))

The ‘comp‘ function can be used to compose a set of functions which is what I needed.

It seemed like the ‘nils-removed’ function wasn’t really necessary so I inlined that:

(defn has-gaps? [list]
    (let [no-nils (fn [x] (drop-while nil? x))]
  (some nil? ((comp no-nils reverse no-nils) list))))

The function can now be used like this:

user=> (has-gaps? '(1 2 3))
nil
user=> (has-gaps? '(nil 1 2 3))
nil
user=> (has-gaps? '(1 2 3 nil))
nil
user=> (has-gaps? '(1 2 nil 3))
true

I’d be intrigued to know if there’s a better way to do this.

I wrote a post about a month ago on using functional collection parameters in C# and over the weekend Fabio and I decided to try and contrast the way you would do this in Java, C# and then F# just for fun.

Map

Map evaluates a high order function on all the elements in a collection and then returns a new collection containing the results of the function evaluation.

Given the numbers 1-5, return the square of each number

Java

int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    System.out.println(f(number));
}
 
private int f(int value) {
    return value*value;
}

C#

new List<int> (new[] {1, 2, 3, 4, 5}.Select(x => x*x)).ForEach(Console.WriteLine);

F#

[1..5] |> List.map (fun x -> x*x) |> List.iter (printfn "%d");;

Filter

Filter applies a predicate against all of the elements in a collection and then returns a collection of elements which matched the predicate.

Given the numbers 1-5, print out only the numbers greater than 3:

Java

int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    f(number);
}
 
private void f(int value) {
    if(value > 3) {
        System.out.println(value);
    }
}

C#

new List<int> { 1,2,3,4,5}.FindAll(x => x > 3).ForEach(Console.WriteLine);

F#

[1..5] |> List.filter (fun x -> x > 3) |> List.iter (printfn "%d");;

Reduce

Reduce applies a high order function against all the elements in a collection and then returns a single result.

Given a list of numbers 1-5, add them all together and print out the answer

Java

int sum = 0;
int[] numbers = { 1,2,3,4,5};
for (int number : numbers) {
    sum += number;
}
System.out.println(sum);

C#

Console.WriteLine(new[] {1, 2, 3, 4, 5}.Aggregate(0, (accumulator, x) => accumulator + x));

F#

[1..5] |> List.fold_left (+) 0 |> printfn "%d";;

In Summary

I was surprised that we could achieve these results in relatively few lines of Java. The C# and F# versions are still more concise but the Java version isn’t too bad. The Apache Commons Library has a class which allows you to write these in a functional way but the need to use anonymous methods means it’s not as clean as what you can achieve in C# and F#.

I think there is still a bit of a mindset switch to make from thinking procedurally about these things to thinking in a way that allows you to make the most of functional programming concepts.

Keeping the code as declarative as possible and reducing the amount of state in our code are the most obvious things I have learned so far from playing with F#.

We have been making good use of Hamcrest matchers on my current project for making assertions, and have moved almost entirely away from the more traditional JUnit assertEquals approach.

There are several reasons why I find the Hamcrest matcher approach to be more productive – it’s more flexible, more expressive and when an assertion fails we have a much better idea about why it has failed than if we use a JUnit assertion for example.

This applies especially when we get a test failing as part of the build as compared to running a test from the IDE where the source code is close at hand and non descriptive error messages may not be such a problem.

It therefore makes sense when writing custom Hamcrest matchers to ensure that we do indeed provide a clear error message so that it is obvious how to fix the test.

The convention seems to be that we should first state the static method name of the matcher and then in brackets list the expected arguments.

To give an example from a matcher we wrote yesterday:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import org.hamcrest.Description;
import org.hamcrest.Factory;
import org.hamcrest.TypeSafeMatcher;
 
public class ContainsAllOf<T> extends TypeSafeMatcher<T> {
    private String[] messages;
 
    public ContainsAllOf(String... messages) {
        this.messages = messages;
    }
 
    public void describeTo(Description description) {
        description.appendText("containsAllOf(");
        for (String message : messages) {
            description.appendText(",");
            description.appendValue(message);
        }
        description.appendText(")");
    }
 
    @Factory
    public static <T> ContainsAllOf containsAllOf(String... messages) {
        return new ContainsAllOf(messages);
    }
 
    public boolean matchesSafely(T t) {
        return contains(t, messages);
    }
 
    private boolean contains(T t, String[] messages) {
        boolean containsAllMessages = true;
        for (String message : messages) {
            if (!t.toString().contains(message)) {
                return false;
            }
        }
        return containsAllMessages;
    }
 
}

If we call this in our test with a value that doesn’t exist:

assertThat("mark's cool message", containsAllOf("mark", "cool", "message", "notThere"));

Running the test results in the following error:

java.lang.AssertionError: 
Expected: containsAllOf(,"mark","cool","message","notThere")
     got: "mark's cool message"

We can easily see what the problem is and how to go about fixing it, which I feel is the most important thing when it comes to test assertions.

One of the features that I dislike in Java compared to C# is checked exceptions.

For me an exception is about a situation which is exceptional, and if we know that there is a possibility of it happening and even have that possibility defined in our code then it doesn’t seem all that exceptional to me.

Having said that they do at least provide information which you can’t help but notice about what can go wrong when you make a call to a particular method.

The problem is that often these checked exceptions just get passed on – i.e. not handled – until we end up with an exception on the page the user sees which is completely irrelevant to the action they are trying to undertake.

To give an example, we have been using the OGNL library to hydrate some objects for testing using the builder pattern.

We have something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public class FooBuilder {
    private String bar;
 
    public FooBuilder setBar(String bar) {
        this.bar = bar;
        return this;
    }
 
    public Foo toFoo() {
        Foo foo = new Foo();
        setValue(foo, "bar", bar);
        return foo;
    }
 
    protected void setValue(Object object, String propertyName, Object propertyValue) {
        try {
            OgnlWrapper.setValue(object, propertyName, propertyValue);
        } catch (OgnlException e) {
            throw new RuntimeException(e);
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import ognl.DefaultMemberAccess;
import ognl.MemberAccess;
import ognl.Ognl;
import ognl.OgnlContext;
import ognl.OgnlException;
 
public class OgnlWrapper {
 
    public static void setValue(Object object, String propertyName, Object propertyValue) throws OgnlException {
        Ognl.setValue(propertyName, createOgnlContext(), object, propertyValue);
    }
 
    private static OgnlContext createOgnlContext() {
        MemberAccess memberAccess = new DefaultMemberAccess(true);
        OgnlContext ognlContext = new OgnlContext();
        ognlContext.setMemberAccess(memberAccess);
        return ognlContext;
    }
}

We can then build an instance of ‘Foo’ like so:

Foo foo = new FooBuilder().setBar("barValue").toFoo();

What is interesting here is not the OGNL library in itself but the checked ‘OgnlException’ which the ‘Ognl.setValue(…)’ method defines.

If I am using the FooBuilder I don’t care how the Foo object is created, all I care is that I get it. Therefore we don’t want to bubble the implementation details of how we are creating the object upwards.

I only care about the OgnlException if I am calling the OgnlWrapper and therefore that is where the exception should be caught and then rethrown as a Runtime exception.

I like to refer to this area of OgnlWrapper callees as being a bounded context – that exception should only be applicable in that particular area and beyond that it should not exist.

Doing this allows us more flexibility around the way we implement things. If I decide in the future to use a different library instead of OGNL to do the same job I don’t need to worry that the callees of FooBuilder will all need to be updated. I can just make the change inside FooBuilder and that’s it!

A couple of months ago my colleague Mark Thomas posted about working on a C# project after 10 years working in Java, and being someone who has worked on projects in both languages fairly consistently (3 Java projects, 2 .NET projects) over the last two years I thought it would be interesting to do a comparison between the two.

The standard ThoughtWorks joke is that you just need to remember to capitalise the first letter of method names in C# and then you’re good to go but I think there’s more to it than that.

The Language & Framework

There is really not much difference between the syntax of Java and C# and I’m not that interested in going into it it massive detail here. There are other websites which cover it in more detail.

Language features wise C# seems to be marginally ahead – the introduction of lambda expressions, implicitly typed local variables and extension methods in C# 3.0 is something not yet matched in Java.

From my experience C#/.NET has much better support for front end rich GUI applications (WinForms, WPF) while Java is probably better for back end work. When it comes to web applications Java probably holds a marginal edge although the soon to be production released ASP.NET MVC framework is a very nice piece of kit.

I have no data to justify saying that, merely thoughts based on experience, but from conversations with friends who work in investment banking I have learned that this is the way the two languages are used there as well.

Other language support

If you are looking for language support on the respective platforms beyond Java/C#, Java probably has a slight edge.

Groovy is a dynamic lanuage with a Java style syntax and should therefore be easier for Java developers to pick up. I’m not aware of a dynamic language with C# style syntax for .NET although Boo is an alternative which compiles to run on the Common Language Infrastructure.

If you need Ruby support Java has JRuby while .NET has IronRuby. JRuby is the more mature of the two options here. If Python is what you need then both contenders compete here too with Java’s offering of Jython and .NET’s IronPython.

Functional language wise .NET has a CTP release of F#, while Java has support for Scala.

Use of 3rd party APIs/Open Source Software

I’ve found that in the Java projects that I’ve worked on use significantly more open source software than the .NET ones. I’m yet to be convinced that this is necessarily a good thing although my Java colleagues are confident that it is.

To give an example, there are multiple different Java libraries for Xml parsing whereas in C# everyone just uses the default one that’s provided.

This provides the opportunity to learn new and better ways of doing things on the one hand, but the potential to spend serious amounts of time evaluating which tool to use instead of just getting on with it on the other.

From a Java perspective it certainly provides extra challenges in trying to get your applications integrated with the range of different application and web servers on the market. In .NET it would simply be a case of getting it to work on IIS – of course easier said than done!

IDEs

I think Java clearly leads in this area with IntelliJ out ahead of anything else I’ve ever worked with. Eclipse is a popular open source alternative but for me it is far less intuitive to use than IntelliJ.

Visual Studio only becomes usable once Resharper is installed but when that’s done it becomes better than eclipse if not quite as usable as IntelliJ. My colleague Pat Kua also listed some ideas to make it run even better. SharpDevelop is a free IDE for .NET development although I haven’t used it so I’m not sure how good it is.

Build and Deployment

Partly due to its better support of Ruby, Java has a much wider range of tools for working with the build.

In .NET NAnt is the only serious contender, and although msbuild is often used to handle the compiling of the code its verbosity of non intuitive approach means I can’t imagine recommending it for a whole build file.

Java wise we have Ant, Maven, a Groovy based wrapper around Ant called Gant, the Ruby based buildr and the dependency management tool Ivy.

Communities

From my experience the community around .NET is more accessible to your average developer than what I’ve noticed in the Java world.

The Alt.NET group is an initiative started last year by several of the leading lights in the .NET world and aims to make the world of .NET development a better and more productive place.

Java has the Java Community Process driving it forward from a community perspective and perhaps due to the lower reliance on the drag and drop tools which are encouraged by Microsoft tools, the standard of your average Java developer may in fact be higher.

When it comes to finding the answers to questions both are mainstream enough that this is fairly easy.

Overall

I’ve tried to cover some of the areas which I considered important when using these two approaches. I’m sure there are some comparisons I have missed out so it would be interesting to hear from others who have worked on both platforms.

This is all written from my knowledge (and a bit of research) so if I’ve missed anything please mention it in the comments.

*Updated*
The paragraph about ‘Other Language Support’ was updated to reflect Robin Clowers’ comments.

A colleague and I have spent the past couple of days spiking solutions for connecting to LDAP servers from Ruby.

We decided that the easiest way to do this is by using OpenDS, an open source directory service based on LDAP.

One option we came up with for doing this was to make use of the Java libraries for connecting to the LDAP server and then calling through to these from our Ruby code using the Ruby Java Bridge.

This post is not about Ruby, but about how we did it in Java to check that the idea was actually feasible.

The interfaces and classes we need to use to do this are not very obvious so it was a little bit fiddly getting it to work. The following code seems to do the trick though:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import org.opends.server.admin.client.ldap.JNDIDirContextAdaptor;
 
import javax.naming.directory.DirContext;
import javax.naming.NamingException;
import javax.naming.Context;
import javax.naming.ldap.LdapName;
import javax.naming.ldap.InitialLdapContext;
 
import com.sun.jndi.ldap.LdapCtx;
 
import java.util.Hashtable;
 
public class OpenDs {
 
    public static void main(String[] args) throws NamingException {
        DirContext dirContext = createLdapContext();
        JNDIDirContextAdaptor adaptor =  JNDIDirContextAdaptor.adapt(dirContext);
 
        // do other stuff with the adaptor
    }
 
    private static DirContext createLdapContext() throws NamingException {
        Hashtable env = new Hashtable();
        env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
        env.put(Context.PROVIDER_URL, "ldap://localhost:389");
        env.put(Context.SECURITY_AUTHENTICATION, "simple");
        env.put(Context.SECURITY_PRINCIPAL, "cn=Directory Manager");
        env.put(Context.SECURITY_CREDENTIALS, "password");
 
        return new InitialLdapContext(env, null);
    }
}

Some points about the code:

• Port 389 is the default port for the LDAP server so unless it’s in use this is probably the port you need to connect to.
• ‘Directory Manager’ is the default ‘Root User DN’ that was setup when we installed OpenDS although there is more information on what this value may need to be on the official documentation.
• We originally tried to connect using JNDIDirContextAdaptor.simpleBind(…) but it didn’t seem to work for us so we went with the JNDIDirContextAdaptor.adapt(…) approach.

While playing around with JBehave I was reminded of my dislike of the import static feature which was introduced in Java 1.5.

Using import static allows us to access static members defined in another class without referencing the class name. For example suppose we want to use the following method in our code:

Math.max(1,2);

Normally we would need to include the class name (Math) that the static function (max) belongs to. By using the import static we can reference max like so:

import static java.lang.Math.max;
...
max(1,2);

The benefit of this approach is that it makes the code read more fluently but the disadvantage is that you can’t immediately tell where a method lives. I want to be able to tell what is going on in the code from looking at it and anything which prevents this is a hindrance.

The official documentation even suggests using this functionality sparingly:

So when should you use static import? Very sparingly! Only use it when you’d otherwise be tempted to declare local copies of constants, or to abuse inheritance (the Constant Interface Antipattern). In other words, use it when you require frequent access to static members from one or two classes.

On my last project we ended up saying that import static was allowed in test code because there were relatively few places the static methods could be imported from, but when it came to production code the fully qualified path was required.