Mark Needham

We have a lot of Xml in our application and one of the things that we need to do reasonably frequently in our test code is create elements which have optional attributes on them.

Our simple first approach looked like this:

def createElement(attribute: Option[String]) = if(attribute.isDefined) <p bar={attribute.get} /> else <p />

That works but it always seemed like we should be able to do it in a simpler way.

Our first attempt was this:

def createElement(attribute: Option[String]) = <p bar={attribute} />

But that ends up in a compilation error:

error: overloaded method constructor UnprefixedAttribute with alternatives:
  (key: String,value: Option[Seq[scala.xml.Node]],next: scala.xml.MetaData)scala.xml.UnprefixedAttribute <and>
  (key: String,value: String,next: scala.xml.MetaData)scala.xml.UnprefixedAttribute <and>
  (key: String,value: Seq[scala.xml.Node],next1: scala.xml.MetaData)scala.xml.UnprefixedAttribute
 cannot be applied to (java.lang.String, Option[String], scala.xml.MetaData)
       def createElement1(attribute: Option[String]) = <p bar={attribute} />

We really need to extract the string value from the option if there is one and not do anything if there isn’t one but with the above approach we try to shove an option in as the attribute value. Unfortunately there isn’t an overload of the constructor which lets us do that.

Eventually one of my colleagues suggested we try passing null in as the attribute value if we had a None option:

def createElement(attribute: Option[String]) = <p bar={attribute.getOrElse(null)} />

Which works pretty well:

scala> createElement(Some("mark"))
res0: scala.xml.Elem = <p bar="mark"></p>
 
scala> createElement(None)
res1: scala.xml.Elem = <p ></p>

We recently wanted to replace a trait mixed into one of our classes with a fake version to make it easier to test but forgot how exactly to do that!

The class is roughly like this:

trait Foo { def foo : String = "real foo" } 
class Mark extends Foo {}

We originally tried to replace it like this:

trait BrokenFakeFoo { def foo : String = "broken fake foo" }
val m = new Mark with BrokenFakeFoo

error: overriding method foo in trait Foo of type => String;
 method foo in trait BrokenFakeFoo of type => String needs `override' modifier
       val m = new Mark with BrokenFakeFoo

If m compiled it would have two versions of foo but it wouldn’t know which one to use, hence the error message.

Attempt two was this:

trait BrokenFakeFoo { override def foo : String = "broken fake foo" }

error: method foo overrides nothing
       trait BrokenFakeFoo { override def foo : String = "broken fake foo" }

As Uday pointed out, what we actually need to do is make our fake trait extend the original one and then override the method.

trait FakeFoo extends Foo { override def foo : String = "fake foo" }
val m = new Mark with FakeFoo

m.foo
> res5: String = fake foo

Since FakeFoo is the right most of the traits mixed into Mark its foo method will be used over the Foo one mixed into Mark on its class definition.

I’ve generally avoided using for expressions in Scala because the keyword reminds me of for loops in Java/C# and I want to learn to program in a less imperative way.

After working with my colleague Mushtaq I realised that in some cases using for comprehensions can lead to much more readable code.

An interesting use case where this is the case is when we want to create an object from a bunch of parameters that may or may not be set. i.e. a bunch of options.

For example we might take some input from the user where they have to enter their name but could choose to leave the field blank:

val maybeFirstName : Option[String] = Some("Mark")
val maybeSurname : Option[String] = None

We only want to create a Person if they have provided both names.

The for comprehension works quite well in allowing us to do this:

case class Person(firstName:String, surname:String)

scala> for { firstName <- maybeFirstName; surname <- maybeSurname } yield Person(firstName, surname)
res27: Option[Person] = None

If we set the surname to have a value:

val maybeSurname : Option[String] = Some("Needham")

Running the same for comprehension will yield a Person

scala> for { firstName <- maybeFirstName; surname <- maybeSurname } yield Person(firstName, surname)
res29: Option[Person] = Some(Person(Mark,Needham))

From what I understand when we have multiple values assigned using ‘<-' inside a for comprehension, each value will have flatMap called on it except for the last one which will have map called instead.

The equivalent code if we didn’t use a for comprehension would therefore look like this:

scala> maybeFirstName.flatMap { firstName => maybeSurname.map { surname => Person(firstName, surname) } } 
res43: Option[Person] = Some(Person(Mark,Needham))

For me the for comprehension expresses intent much better and it seems to excel even more as we add more values to the comprehension.

A couple of colleagues and I were having an interesting discussion this afternoon about the visibility of functions which are mixed into an object from a trait.

The trait in question looks like this:

trait Formatting {
  def formatBytes(bytes: Long): Long = {
    math.round(bytes.toDouble / 1024)
  }
}

And is mixed into various objects which need to display the size of a file in kB like this:

class SomeObject extends Formatting {
 
}

By mixing that function into SomeObject any of the clients of SomeObject would now to be able to call that function and transform a bytes value of their own!

The public API of SomeObject is now cluttered with this extra method although it can’t actually do any damage to the state of SomeObject because it’s a pure function whose output depends only on the input given to it.

There are a couple of ways I can think of to solve the modifier ‘problem’:

• Make formatBytes a private method on SomeObject
• Put formatBytes on a singleton object and call it from SomeObject

The problem with the first approach is that it means we have to test the formatBytes function within the context of SomeObject which makes our test much more difficult than if we can test it on its own.

It also makes the discoverability of that function more difficult for someone else who has the same problem to solve elsewhere.

With the second approach we’ll have a dependency on that singleton object in our object which we wouldn’t be able to replace in a test context even if we wanted to.

While thinking about this afterwards I realised that it was quite similar to something that I used to notice when i was learning F# – the modifiers on functions don’t seem to matter if the data they operate on is immutable.

I often used to go back over bits of code I’d written and make all the helper functions private before realising that it made more sense to keep them public but group them with similar functions in a module.

I’m moving towards the opinion that if the data is immutable then it doesn’t actually matter that much who it’s accessible to because they can’t change the original version of that data.

private only seems to make sense if it’s a function mutating a specific bit of data in an object but I’d be interesting in hearing where else my opinion doesn’t make sense.

We’re using WebDriver on my project to automate our functional tests and as a result are using the Page Object pattern to encapsulate each page of the application in our tests.

We’ve been trying to work out how to effectively reuse code since some of the pages have parts of them which work exactly the same as another page.

For example we had a test similar to this…

class FooPageTests extends Spec with ShouldMatchers with FooPageSteps {
  it("is my dummy test") {
    ...
    then(iShouldNotSeeAnyCommonLinks())
  }
}

…where FooPageSteps extends CommonSteps which contains the common assertions:

trait FooPageSteps extends CommonSteps {
  override val page = new FooPage(driver)
}

trait CommonSteps {
  val page : FooPage
  val driver: HtmlUnitDriver
 
  def iShouldNotSeeAnyCommonLinks() {
    page.allCommonLinks.isEmpty should equal(true)
  }
}

FooPage looks like this:

class FooPage(override val driver:WebDriver) extends Page(driver) with CommonSection {
 
}
 
abstract class Page(val driver: WebDriver) {
  def title(): String = driver.getTitle;
}
 
trait CommonSection {
  val driver:WebDriver
  def allCommonLinks:Seq[String] = driver.findElements(By.cssSelector(".common-links li")).map(_.getText)
}

We wanted to reuse CommonSteps for another page like so:

trait BarPageSteps extends CommonSteps {
  override val page = new BarPage(driver)
}
 
class BarPage(override val driver:WebDriver) extends Page(driver) with CommonSection {
 
}

But that means that we need to change the type of page in CommonSteps to make it a bit more generic so it will work for BarPageSteps too.

Making it of type Page is not enough since we still need to be able to call the allCommonLinks which is mixed into FooPage by CommonSection.

We therefore end up with the following:

trait CommonSteps {
  val page : Page with CommonSection
  val driver: HtmlUnitDriver
 
  def iShouldNotSeeAnyCommonLinks() {
    page.allCommonLinks.isEmpty should equal(true)
  }
}

We’re able to mix in CommonSection just for this instance of Page which works pretty well for allowing us to achieve code reuse in this case!

We have several different test builders in our code base which look roughly like this:

case class FooBuilder(bar : String, baz : String) {
	def build = new Foo(bar, baz)
}

In our tests we originally used them like this:

class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(FooBuilder(bar = "Bar", baz = "Bazz")))
		// and so on...
	}
}

This works well but we wanted our tests to only contain domain language and no implementation details.

We therefore started pulling out methods like so:

class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(aFooWithBarAndBaz("Bar", "Bazza")))
		// and so on...
	}
 
	def aFooWithBarAndBaz(bar:String, baz;String) = FooBuilder(bar = bar, baz = baz)
}

This was fine to start with but we eventually ended up with 10-12 different variations oh how Foo could be constructed, negating the value that the builder pattern provides.

Instead what we can do is use of an alias of FooBuilder to achieve something equally readable:

package object TestSugar {
	val aFooWith = FooBuilder
}

We can then use aFooWith like so:

import TestSugar._
 
class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(aFooWith(bar = "Bar", baz = "Bazza")))
		// and so on...
	}
}

We could also achieve that by renaming FooBuilder to aFooWith but that makes it much less discoverable whereas this solution lets us achieve both goals.

The package object approach isn’t really needed – we could easily put those vals onto an Object or Class but they don’t really seem to belong to any which is why we’ve gone for this approach.

One of the language features available to us in Scala which I think is having a big impact in helping us to make our code base easier to follow is the companion object.

We’ve been using companion objects quite liberally in our code base to define factory methods for our classes.

As I mentioned in a previous post a lot of our objects are acting as wrappers around XML documents and we’ve been pushing some of the data extraction from the XML into companion objects so that our classes can take in non XML values.

This means we can test the data extraction against the companion object and then create simpler tests against any other logic in the object because we don’t have to create XML documents in each of our tests.

The following is an example of a Foo object being constructed with data from an XML document:

object Foo {
   def apply(element: Node) = {
    val bar = element.attribute("bar").get.head.text
    val baz = (element \\ "baz").text
    new Foo(bar, baz)
  }
}

There is also some other logic around how a collection of Foos should be ordered and by using the companion object to parse the XML we can create a test with appropriate bar and baz values to test that.

case class Foo(bar: String, baz:String) extends Ordered[Foo] {
   def compare(that: Foo) = {
     // logic to compare Foos
   }
}

Before we had the companion object we were putting the logic to create Foo inside the object where it is created from which increased the complexity of that object and made it more difficult for people to read.

We’ve also been using this approach to build up page objects representing sub sections of a page in our Web Driver tests and it seems to work quite nicely there as well.

We’ve been coding in Scala on my project for around 6 weeks now and are getting to the stage where we’re probably becoming a big dangerous with our desire to try out some of the language features.

One that we’re trying out at the moment is the implicit key word which allows you to pass arguments to objects and methods without explicitly defining them in the parameter list.

The website we’re working on needs to be accessible in multiple languages and therefore we need to be able to translate some words before they get displayed on the page.

Most of the time it’s just static labels which need to be internationalised but there are a few words which are retrieved from the database and aren’t as easy to deal with.

We introduced the idea of the LanguageAwareString which acts as a wrapper around a String and has its own toString method which delegates to a Language class which contains a dictionary

It’s defined like this:

case class LanguageAwareString(ignorantValue:String)(implicit val language : Language) {
  def toString = language.translate(ignorantValue)
}

We didn’t want to have to pass Language to the LanguageAwareString factory method every time we’re going to be calling it in quite a few places.

We therefore create an implicit val at the beginning of our application in the Scalatra entry code

class Controllers extends ScalatraFilter with ScalateSupport {
	...
	implicit def currentLanguage : Language = // work out the current language
}

As I understand it, whenever the Scala compiler encounters an implicit it looks in its execution scope for any value defined as implicit with the expected type.

As long as there’s only one such value in the scope it make use of that value but if there’s more than one we’d see a compilation error since it wouldn’t know which one to use.

We therefore needed to define Language as an implicit on all the classes/methods which the code follows on its way down to LanguageAwareStrong.

The problem we’ve had is that it’s not immediately obvious what’s going on to someone who hasn’t come across implicit before and we therefore end up having to go the above each time!

We’ve decided that to ease that transition we’d explicitly pass Language down through the first few classes so that it’s more obvious what’s going on.

We therefore have code like this in a few places:

new ObjectThatTakesLanguageImplicitly(someArg)(currentLanguage)

Maybe we can phase that out as people get used to implicit or maybe we’ll just get rid of implicit and decide it’s not worth the hassle!

In a previous post I included a code sample where we were formatting a page range differently depending on whether the start page and end pages were the same.

The code looked like this:

trait PageAware {
  def startPage:String
  def endPage:String
  def pageRange = if(firstPage == lastPage) "page %s".format(firstPage) else "pages %s-%s".format(firstPage, lastPage)
}

Looking at the if statement on the last line we were curious whether it would be possible to get rid of it and replace it with something else.

In Java we could use the ternary operator:

public class PageAware {
  public String pageRange() {
    return (firstPage == lastPage) ? String.format("page %s", firstPage) : String.format("pages %s-%s", firstPage, lastPage)
  }
}

The if/else statement in Scala is supposed to replace that as far as I understand but I think the ternary operator looks neater.

Beyond defining that we played around with some potential alternatives.

We could use a Map to store the true and false values as keys:

trait PageAware {
  def pageRange = Map(true -> "page %s", false -> "pages %s-%s")(firstPage == lastPage).format(firstPage, lastPage)
}

Uday came up with a pattern matching solution which looks like this:

trait PageAware {
  def pageRange = ((firstPage, lastPage) match { case (`firstPage`, `firstPage`) => "page %s" case _ => "pages %s-%s"}).format(firstPage, lastPage)
}

Unfortunately both of these solutions are significantly less readable than the if/else one so it seems like this is one of the situations where it doesn’t actually make sense to get rid of it.

More than a few times recently we’ve wanted to use pattern matching on a collection of pairs/tuples and have run into trouble doing so.

It’s easy enough if you don’t try and pattern match:

> List(("Mark", 4), ("Charles", 5)).filter(pair => pair._2 == 4)
res6: List[(java.lang.String, Int)] = List((Mark,4))

But if we try to use pattern matching:

List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4)

We end up with this error:

<console>:1: error: illegal start of simple expression
       List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4)

It turns out that we can only use this if we pass the function to filter using {} instead of ():

> List(("Mark", 4), ("Charles", 5)).filter { case(name, number) => number == 4 }
res7: List[(java.lang.String, Int)] = List((Mark,4))

It was pointed out to me on the Scala IRC channel that the reason for the compilation failure has nothing to do with trying to do a pattern match inside a higher order function but that it’s not actually possible to use a case token without the {}.

[23:16] mneedham: hey – trying to understand how pattern matching works inside higher order functions. Don’t quite get this code -> https://gist.github.com/1079110 any ideas?

[23:17] dwins: mneedham: scala requires that “case” statements be inside curly braces. nothing to do with higher-order functions

[23:17] mneedham: is there anywhere that’s documented or is that just a known thing?

[23:18] mneedham: I expected it to work in normal parentheses

[23:21] amacleod: mneedham, it’s documented. Whether it’s documented simply as “case statements need to be in curly braces” is another question

The first line of Section 8.5 ‘Pattern Matching Anonymous Functions’ of the Scala language spec proves what I was told:

Syntax:
BlockExpr ::= ‘{’ CaseClauses ‘}

It then goes into further detail about how the anonymous function gets converted into a pattern matching statement which is quite interesting reading.