Mark Needham

Thoughts on Software Development

Archive for the ‘Java’ Category

Java: Incrementally read/stream a CSV file

with 3 comments

I’ve been doing some work which involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents and save it into a list of maps.

This works when the contents of the file fit into memory but is problematic for larger files where I needed to stream the file and process each line individually rather than all of them after the file was loaded.

I initially wrote a variation on totallylazy’s Strings#lines to do this and while I was able to stream the file I made a mistake somewhere which meant the number of maps on the heap was always increasing.

After spending a few hours trying to fix this Michael suggested that it’d be easier to use an iterator instead and I ended up with the following code:

public class ParseCSVFile {
    public static void main(String[] args) throws IOException
    {
        final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' );
        final String[] fields = csvReader.readNext();
 
        Iterator<Map<String, Object>>() lazilyLoadedFile = return new Iterator<Map<String, Object>>()
        {
            String[] data = csvReader.readNext();
 
            @Override
            public boolean hasNext()
            {
                return data != null;
            }
 
            @Override
            public Map<String, Object> next()
            {
                final Map<String, Object> properties = new HashMap<String, Object>();
                for ( int i = 0; i < data.length; i++ )
                {
                    properties.put(fields[i], data[i]);
                }
 
                try
                {
                    data = csvReader.readNext();
                }
                catch ( IOException e )
                {
                    data = null;
                }
 
                return properties;
            }
 
            @Override
            public void remove()
            {
                throw new UnsupportedOperationException();
            }
        };
    }	
}

Although this code works it’s not the most readable function I’ve ever written so any suggestions on how to do this in a cleaner way are welcome.

Written by Mark Needham

October 14th, 2013 at 7:27 am

Posted in Java

Tagged with

jackson-core-asl – java.lang.AbstractMethodError: org.codehaus.jackson.JsonNode.getValueAsText()Ljava/lang/String;

without comments

Ian and I were doing a bit of work on an internal application which processes JSON messages and interacts with AWS and we started seeing the following exception after doing an upgrade of jackson-mapper-asl from 1.8.9 to 1.9.13:

2013-09-13 11:01:50 +0000: Exception while handling {MessageId: 7e695fb3-549a-4b
40-b1cf-9dbc5e97a8df, ... }
java.lang.AbstractMethodError: org.codehaus.jackson.JsonNode.getValueAsText()Lja
va/lang/String;                                                                 
...                                                                
        at com.amazonaws.services.sqs.AmazonSQSAsyncClient$20.call(AmazonSQSAsyn
cClient.java:1200)                                                              
        at com.amazonaws.services.sqs.AmazonSQSAsyncClient$20.call(AmazonSQSAsyn
cClient.java:1191)                                                              
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)   
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)             
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)                                                                      
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)                                                                      
        at java.lang.Thread.run(Thread.java:724)

I’d not seen this exception before but we figured that it was probably happening because we had conflicting versions of some Jackson related JAR on the class path.

We could see in the external libraries section of IntelliJ that we had both the 1.8.9 and 1.9.13 versions of jackson-mapper-asl but we weren’t sure which dependency was pulling in the earlier version.

Alistair pointed out quite a neat command you can pass to Maven which shows transitive dependencies so we gave that a try:

$ mvn dependency:tree
...
[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ load-generator ---
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.5.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.2.2:compile (version managed from 4.2)
[INFO] |  +- commons-codec:commons-codec:jar:1.6:compile
[INFO] |  \- org.codehaus.jackson:jackson-core-asl:jar:1.8.9:compile
...
[INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |   \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
...

As you can see, we have two different versions of the json-core-asl JAR and the earlier version was being pulled in by the aws-java-sdk. We excluded its transitive dependency by changing our pom file to read like this:

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.5.6</version>
 
      <exclusions>
        <exclusion>
          <groupId>org.codehaus.jackson</groupId>
          <artifactId>jackson-core-asl</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

After that everything worked swimmingly.

Written by Mark Needham

September 14th, 2013 at 12:06 am

Posted in Java

Tagged with ,

Jersey Client: java.net.ProtocolException: Server redirected too many times/Setting cookies on request

without comments

A couple of weeks ago I was trying to write a test around some OAuth code that we have on an internal application and I was using Jersey Client to send the various requests.

I initially started with the following code:

Client = Client.create();
ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class );

but when I ran the test I was getting the following exception:

com.sun.jersey.api.client.ClientHandlerException: java.net.ProtocolException: Server redirected too many  times (20)
	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151)
	at com.sun.jersey.api.client.Client.handle(Client.java:648)
	at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
	at com.sun.jersey.api.client.WebResource.get(WebResource.java:191)
	at com.neotechnology.testlab.manager.webapp.AuthenticationIntegrationTest.shouldRedirectToGitHubForAuthentication(AuthenticationIntegrationTest.java:81)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at com.neotechnology.kirkaldy.testing.Resources$1.evaluate(Resources.java:84)
	at com.neotechnology.kirkaldy.testing.FailureOutput$2.evaluate(FailureOutput.java:37)
	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
Caused by: java.net.ProtocolException: Server redirected too many  times (20)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1446)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:249)
	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
	... 28 more

If we check the traffic going across port 59680 we can see what’s going wrong:

$ sudo ngrep -d lo0 port 59680
interface: lo0 (127.0.0.0/255.0.0.0)
filter: (ip) and ( port 59680 )
#####
T 127.0.0.1:59704 -> 127.0.0.1:59680 [AP]
  GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive....
##
T 127.0.0.1:59680 -> 127.0.0.1:59704 [AP]
  HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=mdyw3a4fmqc1b6p53birm4dd;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59679/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length
  : 0..Server: Jetty(8.1.8.v20121106)....
###########
T 127.0.0.1:59707 -> 127.0.0.1:59680 [AP]
  GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive....
##
T 127.0.0.1:59680 -> 127.0.0.1:59707 [AP]
  HTTP/1.1 302 Found..Cache-Control: no-cache..Set-Cookie: JSESSIONID=8gggez0ns9ftiex4314mbgz9;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106)....
###########
T 127.0.0.1:59713 -> 127.0.0.1:59680 [AP]
  GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive....
##

The response we receive includes a direction to the client to store a cookie but we can see on the next request that the cookie hasn’t been included.

I came across this post which had a few suggestions on how to get around the problem but the only approach that worked for me was to use jersey-apache-client for which I added the following dependency:

<dependency>
  <groupId>com.sun.jersey.contribs</groupId>
  <artifactId>jersey-apache-client</artifactId>
  <version>1.13</version>
  <type>jar</type>
</dependency>

I then change my client code to read like this:

ApacheHttpClientConfig config = new DefaultApacheHttpClientConfig();
config.getProperties().put(ApacheHttpClientConfig.PROPERTY_HANDLE_COOKIES, true);
ApacheHttpClient client = ApacheHttpClient.create( config );
client.setFollowRedirects(true);
client.getClientHandler().getHttpClient().getParams().setBooleanParameter( HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, true );
ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class );

If we run that and watch the output using ngrep we can see that it now handles cookies correctly:

$ sudo ngrep -d lo0 port 59680
Password:
interface: lo0 (127.0.0.0/255.0.0.0)
filter: (ip) and ( port 59680 )
	#####
T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP]
  GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680....
##
T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP]
  HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:60322/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length
  : 0..Server: Jetty(8.1.8.v20121106)....
##
T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP]
  GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/....
##
T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP]
  HTTP/1.1 302 Found..Cache-Control: no-cache..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106)....
##
T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP]
  GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/....
##
T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP]
  HTTP/1.1 200 OK..Vary: Accept-Encoding..Accept-Ranges: bytes..Content-Type: text/html..Content-Length: 2439..Last-Modified: Tue, 23 Jul 2013 10:48:15 GMT..Server: Jetty(8.1.8.v20121106)....<!DOCTYPE html>.<html>.<head>.  <title>Tes
  t Lab</title>.  <meta name="viewport" content="width=device-width initial-scale=1.0 maximum-scale=1.0 user-scalable=0"/>.  <link rel="stylesheet" type="text/css" href="css/bootstrap.css">.  <link rel="stylesheet" type="text/css" hr
  ef="css/bootstrap-responsive.css">.  <link rel="stylesheet" type="text/css" href="css/test-lab.css">.  <link rel="icon" type="image/png" href="images/testlab-16.png">.  <script src="js/require.js"></script>.  <script src="js/d3.v3.
  js" charset="UTF-8"></script>.  <script src="js/jquery-1.8.3.min.js"></script>.  <script src="js/bootstrap.js"></script>.  <script src="js/test-lab.js"></script>.  <script type="text/javascript" src="js/google-analytics.js"></scrip
  t>.</head>.<body>..<a href="navbar.html" class="inline-replacement"></a>..<div class="container">..  <h1 class="title">Neo Technology Test Lab</h1>..  <div class="dev-mode">Viewing dev-mode</div>..  <h2>Benchmark Series Results</h2
  >..  <table id="benchmarkseries_list" class="table">.    <thead></thead>.    <tbody></tbody>.  </table>..  <a href="new-benchmark-series.html" class="btn load-modal"><i class="icon-plus"></i> Create Benchmark Series...</a>..  <h2>T
  ests</h2>..  <table id="test_list" class="table">.    <thead></thead>.    <tbody></tbody>.  </table>..  <a href="run-test.html" class="btn load-modal"><i class="icon-plus"></i> Run Test...</a>.  <a href="run-benchmark.html" class="
  btn load-modal"><i class="icon-plus"></i> Run Benchmark...</a>..  <h2>Clusters</h2>..  <table id="cluster_list" class="table"></table>..  <a href="new-database-cluster.html" class="btn load-modal"><i class="icon-plus"></i> New Data
  base Cluster...</a>.  <a href="new-load-generation-cluster.html" class="btn load-modal"><i class="icon-plus"></i> New Load Generation Cluster...</a>..  <h2>Datasets</h2>..  <table id="dataset_list" class="table">.    <thead></thead
  >.    <tbody></tbody>.  </table>..  <a href="new-dataset.html" class="btn load-modal"><i class="icon-plus"></i> Create New Dataset...</a>..  <a href="confirm.html" class="inline-replacement"></a>..</div>..<script type="text/javascr
  ipt">.  (function ().  {.    window.setInterval( function ().    {.      d3.json( "/public/authenticated", function ( status ).      {.        if ( !status.authenticated ).        {.          window.location.reload();.        }.
     } );.    }, 5000 );.    kirkaldy.initializeUI();.  })();.</script>..</body>.</html>.

Written by Mark Needham

August 17th, 2013 at 8:25 pm

Posted in Java

Tagged with ,

Jersey Client: com.sun.jersey.api.client.UniformInterfaceException

without comments

As I mentioned in a post a couple of weeks ago we’ve been doing some which involved calling the neo4j server’s HA URI to determine whether a machine was slave or master.

We started off with the following code using jersey-client:

public class HaSpike {
    public static void main(String[] args) {
        String response = client()
                .resource("http://localhost:7474/db/manage/server/ha/slave")
                .accept(MediaType.TEXT_PLAIN)
                .get(String.class);
 
        System.out.println("response = " + response);
    }
 
    private static Client client() {
        DefaultClientConfig defaultClientConfig = new DefaultClientConfig();
        defaultClientConfig.getClasses().add(JacksonJsonProvider.class);
        return Client.create(defaultClientConfig);
    }
}

which works fine when the server is actually a slave:

response = true

but blows up in style if the server is the master:

Exception in thread "main" com.sun.jersey.api.client.UniformInterfaceException: GET http://localhost:7474/db/manage/server/ha/slave returned a response status of 404 Not Found
	at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686)
	at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
	at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:507)
	at HaSpike.main(HaSpike.java:10)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

We return a 404 status code from that URI if you’re not a slave because it simplifies things for upstream load balancers but I thought Jersey would just return the body of the response rather than throwing an exception.

A quick browse of the Jersey code showed a way around this:

    private <T> T handle(Class<T> c, ClientRequest ro) throws UniformInterfaceException, ClientHandlerException {
        setProperties(ro);
        ClientResponse r = getHeadHandler().handle(ro);
 
        if (c == ClientResponse.class) return c.cast(r);
 
        if (r.getStatus() < 300) return r.getEntity(c);
 
        throw new UniformInterfaceException(r,
                ro.getPropertyAsFeature(ClientConfig.PROPERTY_BUFFER_RESPONSE_ENTITY_ON_EXCEPTION, true));
    }

WebResource#handle gets called by WebResource#get and if we pass ClientResponse.class to it instead of String.class we can get around this because the code returns without checking the status of the response.

Our code needs to read like this:

public class HaSpike {
    public static void main(String[] args) {
        ClientResponse response = client()
                .resource("http://localhost:7474/db/manage/server/ha/slave")
                .accept(MediaType.TEXT_PLAIN)
                .get(ClientResponse.class);
 
        System.out.println("response = " + response.getEntity(String.class));
    }
 
    ...
}

And if we run it, this time we get the expected result:

response = false

Written by Mark Needham

August 11th, 2013 at 8:07 am

Posted in Java

Tagged with

Jersey Client: Testing external calls

with one comment

Jim and I have been doing a bit of work over the last week which involved calling neo4j’s HA status URI to check whether or not an instance was a master/slave and we’ve been using jersey-client.

The code looked roughly like this:

class Neo4jInstance {
    private Client httpClient;
    private URI hostname;
 
    public Neo4jInstance(Client httpClient, URI hostname) {
        this.httpClient = httpClient;
        this.hostname = hostname;
    }
 
    public Boolean isSlave() {
        String slaveURI = hostname.toString() + ":7474/db/manage/server/ha/slave";
        ClientResponse response = httpClient.resource(slaveURI).accept(TEXT_PLAIN).get(ClientResponse.class);
        return Boolean.parseBoolean(response.getEntity(String.class));
    }
}

While writing some tests against this code we wanted to stub out the actual calls to the HA slave URI so we could simulate both conditions and a brief search suggested that mockito was the way to go.

We ended up with a test that looked like this:

@Test
public void shouldIndicateInstanceIsSlave() {
    Client client = mock( Client.class );
    WebResource webResource = mock( WebResource.class );
    WebResource.Builder builder = mock( WebResource.Builder.class );
 
    ClientResponse clientResponse = mock( ClientResponse.class );
    when( builder.get( ClientResponse.class ) ).thenReturn( clientResponse );
    when( clientResponse.getEntity( String.class ) ).thenReturn( "true" );
    when( webResource.accept( anyString() ) ).thenReturn( builder );
    when( client.resource( anyString() ) ).thenReturn( webResource );
 
    Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave();
 
    assertTrue(isSlave);
}

which is pretty gnarly but does the job.

I thought there must be a better way so I continued searching and eventually came across this post on the mailing list which suggested creating a custom ClientHandler and stubbing out requests/responses there.

I had a go at doing that and wrapped it with a little DSL that only covers our very specific use case:

private static ClientBuilder client() {
    return new ClientBuilder();
}
 
static class ClientBuilder {
    private String uri;
    private int statusCode;
    private String content;
 
    public ClientBuilder requestFor(String uri) {
        this.uri = uri;
        return this;
    }
 
    public ClientBuilder returns(int statusCode) {
        this.statusCode = statusCode;
        return this;
    }
 
    public Client create() {
        return new Client() {
            public ClientResponse handle(ClientRequest request) throws ClientHandlerException {
                if (request.getURI().toString().equals(uri)) {
                    InBoundHeaders headers = new InBoundHeaders();
                    headers.put("Content-Type", asList("text/plain"));
                    return createDummyResponse(headers);
                }
                throw new RuntimeException("No stub defined for " + request.getURI());
            }
        };
    }
 
    private ClientResponse createDummyResponse(InBoundHeaders headers) {
        return new ClientResponse(statusCode, headers, new ByteArrayInputStream(content.getBytes()), messageBodyWorkers());
    }
 
    private MessageBodyWorkers messageBodyWorkers() {
        return new MessageBodyWorkers() {
            public Map<MediaType, List<MessageBodyReader>> getReaders(MediaType mediaType) {
                return null;
            }
 
            public Map<MediaType, List<MessageBodyWriter>> getWriters(MediaType mediaType) {
                return null;
            }
 
            public String readersToString(Map<MediaType, List<MessageBodyReader>> mediaTypeListMap) {
                return null;
            }
 
            public String writersToString(Map<MediaType, List<MessageBodyWriter>> mediaTypeListMap) {
                return null;
            }
 
            public <T> MessageBodyReader<T> getMessageBodyReader(Class<T> tClass, Type type, Annotation[] annotations, MediaType mediaType) {
                return (MessageBodyReader<T>) new StringProvider();
            }
 
            public <T> MessageBodyWriter<T> getMessageBodyWriter(Class<T> tClass, Type type, Annotation[] annotations, MediaType mediaType) {
                return null;
            }
 
            public <T> List<MediaType> getMessageBodyWriterMediaTypes(Class<T> tClass, Type type, Annotation[] annotations) {
                return null;
            }
 
            public <T> MediaType getMessageBodyWriterMediaType(Class<T> tClass, Type type, Annotation[] annotations, List<MediaType> mediaTypes) {
                return null;
            }
        };
    }
 
    public ClientBuilder content(String content) {
        this.content = content;
        return this;
    }
}

If we change our test to use this code it now looks like this:

@Test
public void shouldIndicateInstanceIsSlave() {
    Client client = client().requestFor("http://localhost:7474/db/manage/server/ha/slave").
                             returns(200).
                             content("true").
                             create();
    Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave();
    assertTrue(isSlave);
}

Is there a better way?

In Ruby I’ve used WebMock to achieve this and Ashok pointed me towards WebStub which looks nice except I’d need to pass in the hostname + port rather than constructing that in the code.

Written by Mark Needham

July 28th, 2013 at 8:43 pm

Posted in Java

Tagged with ,

Jersey: Listing all resources, paths, verbs to build an entry point/index for an API

without comments

I’ve been playing around with Jersey over the past couple of days and one thing I wanted to do was create an entry point or index which listed all my resources, the available paths and the verbs they accepted.

Guido Simone explained a neat way of finding the paths and verbs for a specific resource using Jersey’s IntrospectionModeller:

AbstractResource resource = IntrospectionModeller.createResource(JacksonResource.class);
System.out.println("Path is " + resource.getPath().getValue());
 
String uriPrefix = resource.getPath().getValue();
for (AbstractSubResourceMethod srm :resource.getSubResourceMethods())
{
    String uri = uriPrefix + "/" + srm.getPath().getValue();
    System.out.println(srm.getHttpMethod() + " at the path " + uri + " return " + srm.getReturnType().getName());
}

If we run that against j4-minimal‘s JacksonResource class we get the following output:

Path is /jackson
GET at the path /jackson/{who} return com.g414.j4.minimal.JacksonResource$Greeting
GET at the path /jackson/awesome/{who} return javax.ws.rs.core.Response

That’s pretty neat but I didn’t want to have to manually list all my resources since I’ve already done that using Guice .

I needed a way to programatically get hold of them and I partially found the way to do this from this post which suggests using Application.getSingletons().

I actually ended up using Application.getClasses() and I ended up with ResourceListingResource:

@Path("/")
public class ResourceListingResource
{
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response showAll( @Context Application application,
                             @Context HttpServletRequest request)
    {
        String basePath = request.getRequestURL().toString();
 
        ObjectNode root = JsonNodeFactory.instance.objectNode();
        ArrayNode resources = JsonNodeFactory.instance.arrayNode();
 
        root.put( "resources", resources );
 
        for ( Class<?> aClass : application.getClasses() )
        {
            if ( isAnnotatedResourceClass( aClass ) )
            {
                AbstractResource resource = IntrospectionModeller.createResource( aClass );
                ObjectNode resourceNode = JsonNodeFactory.instance.objectNode();
                String uriPrefix = resource.getPath().getValue();
 
                for ( AbstractSubResourceMethod srm : resource.getSubResourceMethods() )
                {
                    String uri = uriPrefix + "/" + srm.getPath().getValue();
                    addTo( resourceNode, uri, srm, joinUri(basePath, uri) );
                }
 
                for ( AbstractResourceMethod srm : resource.getResourceMethods() )
                {
                    addTo( resourceNode, uriPrefix, srm, joinUri( basePath, uriPrefix ) );
                }
 
                resources.add( resourceNode );
            }
 
        }
 
 
        return Response.ok().entity( root ).build();
    }
 
    private void addTo( ObjectNode resourceNode, String uriPrefix, AbstractResourceMethod srm, String path )
    {
        if ( resourceNode.get( uriPrefix ) == null )
        {
            ObjectNode inner = JsonNodeFactory.instance.objectNode();
            inner.put("path", path);
            inner.put("verbs", JsonNodeFactory.instance.arrayNode());
            resourceNode.put( uriPrefix, inner );
        }
 
        ((ArrayNode) resourceNode.get( uriPrefix ).get("verbs")).add( srm.getHttpMethod() );
    }
 
 
    private boolean isAnnotatedResourceClass( Class rc )
    {
        if ( rc.isAnnotationPresent( Path.class ) )
        {
            return true;
        }
 
        for ( Class i : rc.getInterfaces() )
        {
            if ( i.isAnnotationPresent( Path.class ) )
            {
                return true;
            }
        }
 
        return false;
    }
 
}

The only change I’ve made from Guido Simone’s solution is that I also call resource.getResourceMethods() because resource.getSubResourceMethods() only returns methods which have a @Path annotation.

Since we’ll sometimes define our path at the class level and then define different verbs that operate on that resource it misses some methods out.

If we run a cURL command (piped through python to make it look nice) against the root we get the following output:

$ curl http://localhost:8080/ -w "\n" 2>/dev/null | python -mjson.tool
 
{
    "resources": [
        {
            "/bench": {
                "path": "http://localhost:8080/bench",
                "verbs": [
                    "GET",
                    "POST",
                    "PUT",
                    "DELETE"
                ]
            }
        },
        {
            "/sample/{who}": {
                "path": "http://localhost:8080/sample/{who}",
                "verbs": [
                    "GET"
                ]
            }
        },
        {
            "/jackson/awesome/{who}": {
                "path": "http://localhost:8080/jackson/awesome/{who}",
                "verbs": [
                    "GET"
                ]
            },
            "/jackson/{who}": {
                "path": "http://localhost:8080/jackson/{who}",
                "verbs": [
                    "GET"
                ]
            }
        },
        {
            "/": {
                "path": "http://localhost:8080/",
                "verbs": [
                    "GET"
                ]
            }
        }
    ]
}

Written by Mark Needham

July 21st, 2013 at 11:07 am

Posted in Java

Tagged with

Jersey Server: com.sun.jersey.api.MessageException: A message body writer for Java class org.codehaus.jackson.node.ObjectNode and MIME media type application/json was not found

with one comment

I’ve been reacquainted with my good friend Jersey over the last couple of days and in getting up and running was reminded that things which seemed easy at the time aren’t as easy when starting from scratch.

I eventually settled on using Sunny Gleason‘s j4-minimal repository which wires up Jersey with Jackson, Guice and Jetty which seemed like a good place to start.

I prefer building up JSON objects explicitly rather than setting up automatic mapping so the first thing I did was change the JacksonResource to have an end point that returned a manually built up JSON object:

@Path("/jackson")
public class JacksonResource {
    @GET
    @Produces( { MediaType.APPLICATION_JSON })
    @Path("/awesome/{who}")
    public Response sayOtherGreeting(@PathParam("who") String name) {
        ObjectNode result = JsonNodeFactory.instance.objectNode();
        result.put("name", name);
        return Response.ok().entity(result).build();
    }
}

When I tried to hit that in the browser I ended up with the following exception:

SEVERE: The registered message body writers compatible with the MIME media type are:
application/json ->
  com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$App
*/* ->
  com.sun.jersey.core.impl.provider.entity.FormProvider
  com.sun.jersey.server.impl.template.ViewableMessageBodyWriter
  com.sun.jersey.core.impl.provider.entity.StringProvider
  com.sun.jersey.core.impl.provider.entity.ByteArrayProvider
  com.sun.jersey.core.impl.provider.entity.FileProvider
  com.sun.jersey.core.impl.provider.entity.InputStreamProvider
  com.sun.jersey.core.impl.provider.entity.DataSourceProvider
  com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General
  com.sun.jersey.core.impl.provider.entity.ReaderProvider
  com.sun.jersey.core.impl.provider.entity.DocumentProvider
  com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider
  com.sun.jersey.core.impl.provider.entity.SourceProvider$SourceWriter
  com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General
  com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General
  com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General
  com.sun.jersey.json.impl.provider.entity.JSONWithPaddingProvider
  com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General
  com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General
  com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$General
  com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$General
  com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy
 
Jul 21, 2013 11:11:17 AM com.sun.jersey.spi.container.ContainerResponse logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: com.sun.jersey.api.MessageException: A message body writer for Java class org.codehaus.jackson.node.ObjectNode, and Java type class org.codehaus.jackson.node.ObjectNode, and MIME media type application/json was not found
	at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:285)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1451)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1338)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
	at org.eclipse.jetty.server.Server.handle(Server.java:350)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.sun.jersey.api.MessageException: A message body writer for Java class org.codehaus.jackson.node.ObjectNode, and Java type class org.codehaus.jackson.node.ObjectNode, and MIME media type application/json was not found
	... 35 more

This is exactly the same problem that I wrote about last year with respect to Jersey Client.

It took me a little while but I eventually found a solution to the problem hidden about half way down a StackOverflow post for a similar exception.

We need to add the following property to our Jersey config:

Map<String, Object> config = new HashMap<String, Object>();
config.put("com.sun.jersey.api.json.POJOMappingFeature", true);

If we edit the SampleConfig from the template it will now look like this:

public class SampleConfig extends GuiceServletContextListener {
    @Override
    protected Injector getInjector() {
        return Guice.createInjector(new ServletModule() {
            @Override
            protected void configureServlets() {
                /* bind the REST resources */
                bind(BenchResource.class);
                bind(SampleResource.class);
                bind(JacksonResource.class);
 
                /* bind jackson converters for JAXB/JSON serialization */
                bind(MessageBodyReader.class).to(JacksonJsonProvider.class);
                bind(MessageBodyWriter.class).to(JacksonJsonProvider.class);
                Map<String, String> initParams = new HashMap<String, String>();
                initParams.put("com.sun.jersey.config.feature.Trace",
                        "true");
                initParams.put("com.sun.jersey.api.json.POJOMappingFeature", "true");
                serve("*").with(
                        GuiceContainer.class,
                        initParams);
            }
        });
    }
 
}

If we hit ‘/jackson/awesome/mark’ it now returns JSON as expected:

$ curl http://localhost:8080/jackson/awesome/mark -w "\n"
{"name":"mark"}

Written by Mark Needham

July 21st, 2013 at 10:37 am

Posted in Java

Tagged with

Java: Finding/Setting JDK/$JAVA_HOME on Mac OS X

without comments

As long as I’ve been using a Mac I always understood that if you needed to set $JAVA_HOME for any program, it should be set to /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK.

On my machine this points to the 1.6 JDK:

$ ls -alh /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK
/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK -> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents

This was a bit surprising to me since I’ve actually got Java 7 installed on the machine as well so I’d assumed the symlink would have been changed:

$ java -version
java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

Andres and I were looking at something around this yesterday and wanted to set $JAVA_HOME to the location of the 1.7 JDK on the system if it had been installed.

We eventually came across the following article which explains that you can use the /usr/libexec/java_home command line tool to do this.

For example, if we want to find where the 1.7 JDK is we could run the following:

$ /usr/libexec/java_home -v 1.7
/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home

And if we want 1.6 the following does the trick:

$ /usr/libexec/java_home -v 1.6
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

We can also list all the JVMs installed on the machine:

$ /usr/libexec/java_home  -V
Matching Java Virtual Machines (3):
    1.7.0_09, x86_64:	"Java SE 7"	/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home
    1.6.0_45-b06-451, x86_64:	"Java SE 6"	/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
    1.6.0_45-b06-451, i386:	"Java SE 6"	/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
 
/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home

I’m not sure how I’ve never come across this command before but it seems pretty neat.

Written by Mark Needham

June 15th, 2013 at 10:28 am

Posted in Java

Tagged with

Jersey: com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class […] and MIME media type application/json was not found

with one comment

We’ve used the Jersey library on the last couple of Java based applications that I’ve worked on and one thing we’ve done on both of them is write services that communicate with each other using JSON.

On both occasions we didn’t quite setup the Jersey client correctly and ended up with an error along these lines when making a call to an end point:

com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class java.util.ArrayList, and Java type java.util.ArrayList<com.blah.Message>, and MIME media type application/json was not found
! at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:561)
! at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:535)
! at com.sun.jersey.api.client.WebResource.handle(WebResource.java:696)
! at com.sun.jersey.api.client.WebResource.access$300(WebResource.java:74)
! at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:512)
...
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
! at java.lang.reflect.Method.invoke(Method.java:601)
! at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
! at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
! at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
! at com.yammer.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:34)
! at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
! at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
! at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)
! at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
! at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesR

To get around this problem we need to make sure we’ve added the ‘JacksonJsonProvider’ to the Jersey client config like so:

DefaultClientConfig defaultClientConfig = new DefaultClientConfig();
defaultClientConfig.getClasses().add(JacksonJsonProvider.class);
Client client = Client.create(defaultClientConfig);

I’m pretty sure that’s documented somewhere in the depths of the Jersey wiki but since we’ve now ended up debugging this problem twice I thought it was worth writing down!

Written by Mark Needham

November 28th, 2012 at 6:03 am

Posted in Java

Tagged with ,

Java: java.lang.UnsupportedClassVersionError – Unsupported major.minor version 51.0

without comments

On my current project we’ve spent the last day or so setting up an environment where we can deploy a couple of micro services to.

Although the machines are Windows based we’re deploying the application onto a vagrant managed VM since the production environment will be a flavour of Linux.

Initially I was getting quite confused about whether or not we were in the VM or not and ended up with this error when trying to run the compiled JAR:

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/whatever/SomeService : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(Unknown Source)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$000(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: com/whatever/SomeService. Program will exit.

These error means that you compiled the code with a higher JDK than the one you’re now trying to run it against.

Since I was accidentally trying to run the JAR against our Windows environment’s 1.6 JDK rather than the VM’s 1.7 JDK this is exactly what I was doing!

Muppet!

Written by Mark Needham

November 24th, 2012 at 8:49 am

Posted in Java

Tagged with