Mark Needham

Thoughts on Software Development

Archive for the ‘Hibernate’ Category

NHibernate 2nd level cache: Doing it wrong?

with 2 comments

I wrote a couple of weeks ago about how we’d been trying to make use of the NHibernate 2nd level cache and we were able to cache our data by following the various posts that I listed.

Unfortunately when we ran some performance tests we found that the performance of the application was significantly worse than when we just wrote our own ‘cache’ – an object which had a dictionary containing the reference data items we’d previously tried to lookup and the appropriate values.

We don’t need to handle cache invalidation. The client’s policy is to restart production servers every night so if we want to update any of the reference data then we just need to make sure a database script is run before the servers get restarted.

Explicit transactions when reading

There is a post on the NHibernate Profiler website which describes why we should not use implicit transactions when using NHibernate. Instead we should create explicit ones:

When we don’t define our own transactions, it falls back into implicit transaction mode, where every statement to the database runs in its own transaction, resulting in a large performance cost (database time to build and tear down transactions), and reduced consistency.

Even if we are only reading data, we should use a transaction, because using transactions ensures that we get consistent results from the database.

We ended up with something like this:

public class OurRepository
{
	public ReferenceDataObject Find(ReferenceDataObjectKey key)
	{
		using(var session = SessionFactory.OpenSession())
		{
			using(var tx = session.BeginTransaction())
			{
				var query = session.Linq<ReferenceDataObject>().Where(r => r.Key == key);
				query.QueryOptions.SetCachable(true).SetCacheMode(CacheMode.Normal);
				// and so	 on
			}
		}
 
		// return the object
	}
}

As well as performance tests, we found that our integration tests became much slower than when we used our own ‘cache’.

We have some tests which look up 100s of different bits of reference data and the total time taken to run those tests went from around 4 seconds up to 30 seconds.

As I understand it, putting a transaction around a query means that we create a transaction with the database on every request even if the query is cached and we’re going to retrieve the results from the 2nd level cache rather than the database.

We took out the transaction which reduced the time taken to 7 seconds but it was still slower than when we used our hand rolled cache.

It seems like we must be doing something wrong or not understand something with respect to the NHibernate 2nd level cache because it seems ridiculous that the performance could be this different?

Written by Mark Needham

June 29th, 2010 at 6:45 am

Posted in Hibernate

Tagged with

Fluent NHibernate and the 2nd level cache

with 6 comments

We’ve been trying to cache some objects using NHibernate’s second level cache which always proves to be a trickier task than I remember it being the previous time!

We’re storing some reference data in the database and then using LINQ to NHibernate to query for the specific row that we want based on some user entered criteria.

We can cache that query by calling ‘SetCacheable’ on the ‘QueryOptions’ property of our query:

1
2
3
4
5
6
7
8
9
public class OurRepository
{
	public ReferenceDataObject Find(string criteria1, string criteria2)
	{
		var query = session.Linq<ReferenceDataObject>();
		query.QueryOptions.SetCachable(true).SetCacheMode(CacheMode.Normal);
		// and so on
	}
}

That helps to ensure that if we do the exact same query again it won’t go to the database again. Instead it will just retrieve the id of the appropriate object from the 2nd level cache and then go and retrieve that.

In order to ensure that the ‘ReferenceDataObject’ does not get retrieved from the database each time we need to ensure that it is cached as well.

We can do that by adding the following to its mapping file:

public class ReferenceDataObjectMappings : ClassMap<ReferenceDataObject>
{
	...
 
	Cache.ReadOnly();
 
	// or
 
	Cache.ReadWrite();
 
	// depending on whether we want to update the value in the cache based on it changing in the database or not
 
	..
 
}

The next step is to ensure that we have the query cache and second level cache turned on in the place where we configure our session factory:

Fluently
.Configure()
.Database(MsSqlConfiguration.MsSql2000.ConnectionString("connection string")
	.Cache(c => c.UseQueryCache().ProviderClass(typeof(NHibernate.Caches.SysCache.SysCacheProvider).AssemblyQualifiedName)))
.Mappings(m => m.FluentMappings.AddFromAssemblyOf<ICurrentSessionFactory>())
.BuildSessionFactory();

The ‘SysCacheProvider’ is one we can use for web applications although there are all sorts of others available too.

Having got all this setup we wrote an integration test which added an item to the reference data table and then tried to retrieve it several times with the theory being that we would only see one call to the database.

[Test]
public void ShouldCacheTheReferenceData()
{
	var myReferenceObject = new ReferenceDataObject(...);
 
	using(var session = SessionFactory.OpenSession())
	{
		session.Save(myReferenceDataObject);
		session.Flush();
	}
 
	// try and retrieve the object a few times here
}

Unfortunately we were seeing the database being hit every single time which confused us greatly until we came across the following explanation on Gabriel Schenker’s blog:

A common error (It happened to me as well!) is to forget to commit or omit a transaction when adding or changing an entity/aggregate to the database. If we now access the entity/aggregate from another session then the 2nd level cache will not be prepared to provide us the cached instances and NHibernate makes an (unexpected round trip to the database).

We need to change our test to commit the object in a transaction if we want the object to be propagated down to the 2nd level cache:

[Test]
public void ShouldCacheTheReferenceData()
{
	var myReferenceObject = new ReferenceDataObject(...);
 
	using(var session = SessionFactory.OpenSession())
	{
		using(var tx = session.BeginTransaction())
		{
    			session.Save(myReferenceObject);
    			tx.Commit();        // important otherwise caching does NOT work!
		}
	}
 
	// try and retrieve the object a few times here
}

Written by Mark Needham

June 16th, 2010 at 12:07 am

Posted in Hibernate

Tagged with ,

Fluent NHibernate: Seeing the mapping files generated

without comments

We’ve been fiddling around with Fluent NHibernate a bit over the last couple of days and one of the things that we wanted to do was output the NHibernate mapping files being generated so we could see if they were as expected.

I couldn’t figure out how to do it but thanks to the help of James Gregory, Andrew Bullock and Matthew Erbs on twitter this is the code that you need in order to do that:

1
2
3
4
5
Fluently
.Configure()
.Database(MsSqlConfiguration.MsSql2000.ConnectionString("connection string"))
.Mappings(m => m.FluentMappings.ExportTo("c:\\directory-to-output-files-to"))
.BuildSessionFactory();

The 4th line is the important one here and we can choose which directory the mappings files get outputted to and then check that everything is getting setup as we’d expect.

Written by Mark Needham

June 15th, 2010 at 11:15 pm

Posted in Hibernate

Tagged with

Functional collectional parameters: Some thoughts

with 6 comments

I’ve been reading through a bit of Steve Freeman and Nat Pryce’s ‘Growing Object Oriented Software guided by tests‘ book and I found the following observation in chapter 7 quite interesting:

When starting a new area of code, we might temporarily suspend our design judgment and just write code without attempting to impose much structure.

It’s interesting that they don’t try and write perfect code the first time around which is actually something I thought experienced developers did until I came across Uncle Bob’s Clean Code book where he suggested something similar.

One thing I’ve noticed when working with collections is that if we want to do something more complicated than just doing a simple map or filter then I find myself initially trying to work through the problem in an imperative hacky way.

When pairing it sometimes also seems easier to talk through the code in an imperative way and then after we’ve got that figured out then we can work out a way to solve the problem in a more declarative way by making use of functional collection parameters.

An example of this which we came across recently was while looking to parse a file which had data like this:

some,random,data,that,i,made,up

The file was being processed later on and the values inserted into the database in field order. The problem was that we had removed two database fields so we needed to get rid of the 2nd and 3rd values from each line.

var stringBuilder = new StringBuilder();
using (var sr = new StreamReader("c:\\test.txt"))
{
    string line;
 
    while ((line = sr.ReadLine()) != null)
    {
        var values = line.Split(',');
 
        var localBuilder = new StringBuilder();
        var count = 0;
        foreach (var value in values)
        {
            if (!(count == 1 || count == 2))
            {
                localBuilder.Append(value);
                localBuilder.Append(",");
            }
            count++;
        }
 
        stringBuilder.AppendLine(localBuilder.ToString().Remove(localBuilder.ToString().Length - 1));
    }
}
 
using(var writer = new StreamWriter("c:\\newfile.txt"))
{
    writer.Write(stringBuilder.ToString());
    writer.Flush();
}

If we wanted to refactor that to use a more declarative style then the first thing we’d look to change is the for loop populating the localBuilder.

We have a temporary ‘count’ variable which is keeping track of which column we’re up to and suggests that we should be able to use one of the higher order functions over collection which allows us to refer to the index of the item.

In this case we can use the ‘Where’ function to achieve this:

...
while ((line = sr.ReadLine()) != null)
{
    var localBuilder = line.Split(',').
                        Where((_, index) => !(index == 1 || index == 2)).
                        Aggregate(new StringBuilder(), (builder, v) => builder.Append(v).Append(","));
 
    stringBuilder.AppendLine(localBuilder.ToString().Remove(localBuilder.ToString().Length - 1));
}

I’ve been playing around with ‘Aggregate’ a little bit and it seems like it’s quite easy to overcomplicate code using that. It also seems that when using ‘Aggregate’ it makes sense if the method that we call on our seed returns itself rather than void.

I didn’t realise that ‘Append’ did that so my original code was like this:

    var localBuilder = line.Split(',').
                        Where((_, index) => !(index == 1 || index == 2)).
                        Aggregate(new StringBuilder(), (builder, v) => {
                           builder.Append(v);
                           builder.Append(",");
                           return builder;
                        });

I think if we end up having to call functions which return void or some other type then it would probably make sense to add on an extension method which allows us to use the object in a fluent interface style.

Of course this isn’t the best solution since we would ideally avoid the need to remove the last character to get rid of the trailling comma which could be done by creating an array of values and then using ‘String.Join’ on that.

Given that I still think the solution written using functional collection parameters is easier to follow since we’ve managed to get rid of two variable assignments which weren’t interesting as part of what we wanted to do but were details about that specific implementation.

Written by Mark Needham

January 20th, 2010 at 10:45 pm

Posted in Coding,Hibernate

Tagged with ,

Testing Hibernate mappings: Setting up test data

with 3 comments

Continuing with my mini Hibernate mappings series, this post talks about the different ways of setting up the test data for our Hibernate tests.

  1. Where to test the mappings from?
  2. How to test for equality?
  3. How to setup the test data?

There are a couple of ways that we can setup data for Hibernate tests.

Insert Hibernate Object

This approach involves creating a new object and saving it to the database using the save method on the Hibernate session.

The benefit of this approach is that our test is pretty clean. It looks like any old unit test where we create an object and then check that what we get back is the same object.

The problem is that we are effectively testing two things – the ability to save and then load our Hibernate object. We will be able to tell whether or not our Hibernate mappings are correct using this approach, but the failures we get when they do fail may not be that obvious – it could just be a database exception which makes the test fail.

SQL Insertion

The other approach I have seen is to write manual JDBC calls to insert data into the various tables in our database and then check that we can load our object from the database using Hibernate.

The advantage of this is that our test is now only testing the mappings when we load the data from the database which helps reduce our test’s invariants or potential points of failure.

The disadvantage is that the tests can seem very brittle – if we make a small change to the column names in our tables then the test setup code may now fail to work anymore.

I’m not really completely happy with either of these approaches – neither seems optimal to me but both can help us achieve our objective. It’s just a matter of choosing which trade off we’ve prepared to accept.

Written by Mark Needham

October 30th, 2008 at 11:24 pm

Posted in Hibernate,Testing

Tagged with ,

Testing Hibernate mappings: Testing Equality

with 2 comments

I started a mini Hibernate series with my last post where I spoke of there being three main areas to think about when it comes to testing:

  1. Where to test the mappings from?
  2. How to test for equality?
  3. How to setup the test data?

Once we have worked out where to test the mappings from, if we have decided to test them through either our repository tests or directly from the Hibernate session then we have some choices to make around how to test for equality.

I’ve seen this done in several ways:

Override equals

This was the first approach I saw and in a way it does make some sort of sense to test like this.

We don’t have to expose any of the internals of the class and we can get feedback on whether our objects have the same fields values or not. In addition we can normally get the IDE to generate the code for the equals method so it doesn’t require much extra effort on our behalf.

Typically an equality test along these lines would look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@Entity
public class Foo {
	@Column(name="BAR")
	private String bar;
 
	public Foo(String bar) {
		this.bar = bar;
	}
 
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
 
        Foo foo = (Foo) o;
 
        return !(bar != null ? !bar.equals(foo.bar) : foo.bar != null);
 
    }
}
1
2
3
4
5
6
import static org.hamcrest.MatcherAssert.assertThat;
...
Foo expectedFoo = new Foo("barValue");
Foo foo = getFooFromHibernate();
 
assertThat(foo, equalTo(expectedFoo));

The problem with this approach is that objects which are in Hibernate are likely to be entities and therefore their equality really depends on whether or not they have the same identity, not whether they have the same values. Therefore our equals method on the object should only compare the id value of the object to determine equality.

Implementing the equals method just for testing purposes may also be considered a code smell.

Getters

This approach involves adding getters to our objects to check that the values of each field have been set correctly.

While this approach is marginally better than not testing the mappings at all, the temptation to then use these getters in other pieces of the code can lead to us having objects with no behaviour at all with our logic spread all over the application.

1
2
3
4
5
6
7
8
9
10
11
12
13
@Entity
public class Foo {
	@Column(name="BAR")
	private String bar;
 
	public Foo(String bar) {
		this.bar = bar;
	}
 
	public getBar() {
		return this.bar;
	}
}
1
2
3
4
5
6
import static org.hamcrest.MatcherAssert.assertThat;
...
String bar = "barValue";
Foo foo = getFooFromHibernate();
 
assertThat(foo.getBar(), equalTo(bar));

Reflection

An approach I was introduced to recently involves using reflection to check that Hibernate has hydrated our objects correctly.

We initially rolled our own ‘Encapsulation Breaker’ to achieve this before realising that the OGNL library did exactly what we wanted to do.

By adding a custom Hamcrest matcher into the mix we end up with quite a nice test for verifying whether our mappings are working correctly.

1
2
3
4
5
@Entity
public class Foo {
	@Column(name="BAR")
	private String bar;
}
1
2
3
4
import static org.hamcrest.MatcherAssert.assertThat;
...
Foo foo = getFooFromHibernate();
assertThat(foo, hasMapping("bar", equalTo("someValue")));
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
public class HasMapping<T> extends BaseMatcher<T> {
    private String mapping;
    private Matcher<T> mappingValueMatcher;
 
    public HasMapping(String mapping, Matcher<T> mappingValueMatcher) {
        this.mapping = mapping;
        this.mappingValueMatcher = mappingValueMatcher;
    }
 
    public void describeTo(Description description) {
        description.appendText("A mapping from ");
        description.appendText(mapping);
        description.appendText(" that matches ");
        valueMatcher.describeTo( description );
    }
 
    @Factory
    public static <T> HasMapping hasMapping(String mapping, Matcher<T> mappingValueMatcher) {
        return new HasMapping(mapping, mappingValueMatcher);
    }
 
    public boolean matches(Object o) {
        try {
            Object value = OgnlWrapper.getValue(mapping, o);
            return mappingValueMatcher.matches(value);
        } catch (OgnlException e) {
            return false;
        }
    }
 
}

The drawback of this approach is that if we change the names of the fields on our objects we need to make a change to our test to reflect the new names.

I ran into the example dilemma a bit while writing this but hopefully the ideas have been expressed in the code presented. I didn’t want to put too much code in this post but if you’re interested in what the OgnlWrapper does I posted more about this on my post about Java checked exceptions.

Written by Mark Needham

October 29th, 2008 at 6:03 pm

Posted in Hibernate,Testing

Tagged with ,

Testing Hibernate mappings: Where to test from?

with 3 comments

I’ve had the opportunity to work with Hibernate and it’s .NET twin NHibernate on several of my projects and one of the more interesting decisions around its use is working out the best way to test the hibernate mappings that hook together our domain model and the database.

There are three decisions to make around how best to do this:

  1. Where to test the mappings from?
  2. How to test for equality?
  3. How to setup the test data?

This post will focus on the ways I have seen with regards to choosing where to test the mappings from.

Functional Tests

This approach advocates only testing whether we have setup the mappings correctly when we run our acceptance or functional tests – we do not write tests specifically for testing hibernate mappings.

The benefit of this approach is that we are more likely to have acceptance tests in place, so this is just another thing that they can be used to catch.

While this approach is better than not testing at all, from my experiences the test feedback cycle is too slow – it takes too long to change one of the hibernate mappings and then run the test to check if it worked or not.

Repository Tests

With this approach we test whether our hibernate mappings are working as part of our repository tests.

The tricky thing with testing our hibernate mappings this way is that typically we only want to set up one object in the database and then test that Hibernate hydrates it correctly, but our repository doesn’t necessarily need a method for finding a single object.

We either end up adding on a method just for testing or we have to try and find our object from a list of other objects and then test it.

On the other hand, this approach does seem to work quite well when we have quite chatty repositories which provide a degree of flexibility around how we can retrieve our objects.

Direct Tests

This approach is my current favourite and involves loading the object directly from the Hibernate session and then testing it.

I was introduced to this idea by a colleague of mine and it seems to fit the idea of testing just one thing more closely than the other two approaches.

The strange thing about this approach is that we are testing directly with an API that is hidden from our system beyond the Repository.

In terms of simplicity with regards to testing hibernate mappings, however, this is the best approach I have seen.

I did a quick survey of some people last week and the most popular way of testing the mappings expressed was using Repository tests.

This post covers the other ways I have seen – are there any others people have come across or are using?

Written by Mark Needham

October 27th, 2008 at 10:55 pm

Posted in Hibernate,Testing

Tagged with , ,