Mark Needham

Another presentation that I gave at XP Day was one covering some visualisations Liz, Uday and I have created from various data we have about our project, gathered from Git, Go and Mingle.

These were some of the things that I learned from doing the presentation:

• The various graphs I presented in the talk have a resolution of 1680 x 1050 which is a much higher resolution than what was available on the projector.

  As a result it was necessary to scroll up and down/side to side when demonstrating each visualisation so that people could actually see them.

  Either I need to work out how to get the resolution of the projector higher or be able to shrink the images to the right size so they’d fit more naturally. I imagine the later would be easier to achieve.

• My machine refused to switch to Powerpoint when I was presenting so I had to wing it a bit from my memory of how the talk was meant to go.

  As a result of not having the slides to show I ended up just showing the code that we’d written to create the graphs.

  I didn’t think this would work very well but the feedback I got suggested that people enjoyed seeing the code behind the visualisations.

I had a discussion with people during the talk and with others at XP Day about how I could change the visualisations so that they were more useful. These were some of the ideas that other people had:

• Matt Jackson suggested that it would be interesting to graph how often the last ten builds were broken so you could see how it was trending.
• Actionable metrics – we had a discussion about what somebody is supposed to do as a result of seeing a visualisation of something i.e. what action do we want them to take.

  We achieve this in some cases e.g. with the pair stair it’s clear who you haven’t paired with recently and the impetus is therefore on you to address that if you want to.

• Phil Parker suggested that metrics that you have an emotional response to are the most effective ones in his experience.

  I think this links to the idea of them being actionable in that if you have an emotional response to something then it often makes you want to go and do something about it.

• I had an interesting discussion with Benjamin Mitchell in which he suggested that an interesting question to ask is ‘what would better look like?‘ and another one would be ‘what rule would we have to have in place for people to behave like this?‘.

  We realised that in some cases it’s not really clear what better would look like since you can end up with two potentially competing ‘good’ practices e.g. checking in frequently is good but if everyone does it together then it can lead to the build breaking which isn’t as good.

We haven’t tried to tidy up any of the code that we used but it is available on the following github accounts if anyone’s interested:

• node.js dashboard
• Clojure backed visualisations
• Javascript Pair Stair

As the closing part of a Scala Experience Report Liz and I gave at XP Day we detailed a retrospective that we’d carried out on the project after 3 months where the team outlined the positives/negatives of working with Scala.

The team members who were there right at the beginning of the project 3 months earlier had come up with what they thought the proposed benefits/drawbacks would be so it was quite interesting to look at our thoughts at both times.

Some of this is available in our slides from the talk but Nat Pryce suggested it’d be interesting to post it up in more detail.

We weren’t aware that we’d be doing this exercise until the session where we did it and noone looked at the original answers so hopefully some of the potential biases have been removed!

JUNE

• +++ Increased developer productivity
  • Higher-level language constructs (functional programming, actors, pattern matching, mixins, etc.)
  • Less code -> less time spent reading code / less defects
  • Syntax is better suited for writing DSLs (e.g. SBT, Scalatra, ScalaTest, etc.)
• +++ Bigger potential to attract talented developers (not using the same old ‘boring’ stack)
• ++ Gentle learning curve for Java devs
• + Built-in support at language-level for handling XML
• + Comes with SBT, a powerful build tool
• + Seamlessly integrates with Java and it’s ecosystem
• + Runs on the JVM (i.e. no operational concerns)

• — Bigger potential to screw things up (think: “with great power comes…”)
• – Tool support is less mature and polished (e.g. IDEs, profilers, metrics, etc.)
• - Community is younger and smaller
• - Scala compiler seems to be slower than Java counterparts

SEPTEMBER

Liked:

• +8 Easy to learn
• +8 Functional Language (Immutable, closures, etc)
• +6 Concise code
• +5 SBT power
• +4 Case classes
• +4 XML support
• +4 Java integration
• +3 List processing
• +3 DSL support
• +2 Helpful community (IRC, StackOverflow)
• +2 Performance

Disliked:

• -8 IDE support (refactoring, plugin quality)
• -5 Slow compiler
• -3 Code can become complex to read
• -2 Lack of XPath support in XML
• -2 SBT complexity
• -2 Immature frameworks

Quite a few of the expected benefits from June were observed in June, such as having to write less code, functional programming constructs, XML support and the ability to write DSLs.

The community was one benefit which wasn’t expected – we’ve found that every time we get stuck on something we can go on Stack Overflow and find the answer and if that doesn’t work then someone on IRC will be able to help us almost immediately.

Complexity

Our experience with Scala’s complexity partly matches with that of Stephen Coulbourne who suggests the following:

Scala appears to have attracted developers who are very comfortable with type theory, hard-core functional programming and the mathematical end of programming.

…

There is also a sense that many in the Scala community struggle to understand how other developers cannot grasp Scala/Type/FP concepts which seem simple to them. This sometimes leads Scala aficionados to castigate those that don’t understand as lazy or poor quality developers.

We’ve tried to be reasonably sensible with the language and only used bits of it that the whole team are likely to understand rather than learning some obscure way of solving a problem and checking that in.

On the other hand reading the code of Scala libraries such as scalaz or SBT is something that I, at least, find extremely difficult.

Changing the SBT build files can be quite a scary experience while you try and remember what all the different symbols mean and how they integrate together.

Learning curve

The learning curve for Java developers has been a bit of a mixed experience.

When we started working on the project we were effectively writing Java in Scala and we’ve slowly learnt/introduced more Scala features into our code as time has passed.

I think everyone who has come on that journey has found the transition reasonably okay but we’ve had other team members who joined later on and went straight into code that they weren’t familiar with and for them it’s been more difficult.

Again, again!

It will be interesting to see the team’s thoughts if we do the exercise again 3 more months on.

I would imagine there would be more ‘dislikes’ around code complexity now that the code has grown even more in size.

It probably also mean the lack of IDE support becomes more annoying as people want to refactor code and can’t get the seamless experience that you get when editing Java code.

At XP Day my colleague Liz Douglass and I presented the following experience report on our last 6 months working together on our project.

We wanted to focus on answering the following questions with our talk:

• Should the project have been done in Java?
• Does it really speed up development as was hoped?
• What features of the language and patterns of usage have been successes?
• Is it easier to maintain and extend than an equivalent Java code base?

We covered the testing approach we’ve taken, our transition from using Mustache as our templating language to using Jade and the different features of the language and how we’ve been using/abusing them.

The approach we used while presenting was to cover each topic in chronological order such that we showed how the code had evolved from June until November and the things we’d learned over that time.

It was actually an interesting exercise to go through while we were preparing the talk and I think it works reasonably well as it makes it possible to take people on the same journey that you’ve been on.

These were a few of the points that we focused on in the talk:

• In our code base at the moment we have 449 unit tests, 280 integration tests and 353 functional tests which is a much different ratio than I’ve seen on other code bases that I’ve worked on.

  Normally we’d have way more unit tests and very few functional tests but a lot of the early functionality was transformations from XML to HTML and it was really easy to make a functional test pass so all the tests ended up there.

  Unfortunately the build time has grown in line with the approach as you might expect!

• We originally started off using Mustache as our templating language but eventually switched to Jade because we were unable to call functions from Mustache templates. This meant that we ended up pushing view logic into our models.

  The disadvantage of switching to Jade is that it becomes possible to put whatever logic you want into the Jade files so we have to remain more disciplined so we don’t create an untestable nightmare.

• On our team the most controversial language feature that we used was an implicit value which we created to pass the user’s language through the code base so we could display things in English or German.

  Half the team liked it and the other half found it very confusing so we’ve been trying to refactor to a solution where we don’t have it anymore.

  Our general approach to writing code in Scala has been to write it as Java-like as possible so that we don’t shoot ourselves in the foot before we know what we’re doing and it’s arguable that this is one time when we tried to be too clever.

• In the closing part of our talk we went through a code retrospective which we did with our whole team a couple of months ago.

  In that retrospective we wrote down the things we liked about working with Scala and the things that we didn’t and then compared them with a similar list which had been created during the project inception.

  Those are covered on the last few slides of the deck but it was interesting to note that most of the expected gains were being achieved and some of the doubts hadn’t necessarily materialised.

Our conclusion was that we probably would use Scala if we were to redo the project again, mainly because the data we’re working with is all XML and the support for that is much better in Scala then in Java.

There is much less code than there would be in an equivalent Java code base but I think the maintenance of it probably requires a bit of time working with Scala, it wouldn’t necessarily be something a Java developer would pick up immediately.

We’re still learning how to use traits and options but they’ve worked out reasonably well for us. We haven’t moved onto any of the complicated stuff such as what’s in scalaz and I’m not sure we really need to for a line of business application.

In terms of writing less code using Scala has sped up development but I’m not sure whether the whole finds Scala code as easy to read as they would the equivalent Java so it’s debatable whether we’ve succeeded on that front.

Another session that I attended at XP Day was one facilitated by Steve Freeman and Joseph Pelrine where we discussed the Cynefin model, something that I first came across earlier in the year at XP 2011.

We spent the first part of the session drawing out the model and coming up with some software examples which might fit into each domain.

• Simple – when you’re going to checkin run the build
• Complicated – certain types of architectural decisions
• Complex – task estimation
• Chaos – startup explosion

Steve pointed out that with simple/complicated the important thing to remember is that things on the right hand side are repeatable whereas on the other side we could do the same thing again and get a completely different result.

The most interesting part of the discussion for me was when Chris Matts joined in and suggested that in his experience people generally preferred to be in one of the quadrants more than the others.

He used Dan North as his example, suggesting that Dan prefers to be in chaotic situations.

I think I like being in the complex domain when you don’t really know what’s going to happen. I find it quite boring when things are predictable.

Traditional project managers would probably prefer to be in the simple/complicated domains because things are a bit more certain over that size.

Liz and I were discussing afterwards whether that tendency is what tends to lead to people becoming generalists rather than specialists.

If you were to become a specialist in a subject then it would suggest to me that a lot of your time would be spent in the complicated domain honing your skills.

Another discussion was around the desire when building systems to try and move the building of that system, which originally starts off being complex, into the complicated and finally into the simple domain.

Nat Pryce pointed out that we can often end up pushing a system back into chaos if we try and force it into the simple domain.

Pushing something into simple would suggest that anyone would be able to make changes to it without having any specialist/expert skills.

Someone else in the group pointed out that it’s often been thought that we can make the programming of systems something so simple that anyone can do it but that so far that theory has been proved false.

Overall this was an interesting session for me and it makes it a bit easier to understand some of the things that I see in the projects that I work on.

Recommended reading from the session

• Sense and Respond by Stephen Parry
• Joseph Perline on Social Complexity & Coaching Self Organising Teams

I’m attending XP Day this year and the first talk I attended was one by Julian Kelsey and Andrew Parker titled ‘Refactoring to functional style’.

I’ve worked on a Scala project for the last 6 months and previously given a couple of talks about adopting a functional style of programming in C# so this is a subject area that I find quite interesting.

The talk focused on 5 refactorings that the presenters have identified to help move imperative code to a more functional style:

• Isolate mutation – keeping mutation in one place rather than leaking it everywhere
• Isolate predicate – making it possible to filter collections
• Separate loops – iterating over collections more than once if we’re doing more than one thing with the collection
• Decide on branches once – putting conditional logic into a map as functions
• Separate sequence of operations from execution of operations – composing functions and executing them at the end

Since they were coding in Java they made use of the Google Guava collections library to make it easier to work with collections in a functional way.

As you might imagine some of the code ends up being quite verbose due to the inability to pass functions around in Java.

I was reminded of a coding dojo we did a couple of years ago where we compared how code written using lambdaj would compare to Scala code.

Despite the verbosity it was interesting to see that it’s actually possible to achieve a similar style of programming to what you would expect in languages like Scala, F# and Clojure.

My former colleague Dan Bodart has an alternative library for working with collections in Java called totallylazy which based on some of the latest commits looks quite neat.

One interesting thing the speakers suggested is that they are better able to see data dependencies in their code when chaining functions together which they wanted to apply to that data.

I hadn’t really thought about the data dependencies before but I generally find code written using function composition to be easier to read than any other approach I’ve seen so far.

The main reason I picked up for why the authors thought we would want to adopt a functional approach to start with is the fact that it limits the number of things that we have to reason about.

Interestingly Jon Tirsen recently tweeted the following:

In my experience large purely functional codebases are very painful. Shared immutable, local mutable is the way to go.

We’ve mostly kept our Scala code base immutable but it’s not large by any measure (5,000 lines of production code so far) and probably not as complex as the domains Jon has worked with.

It’s an interesting observation though…immutability is no silver bullet!

On the system that I’m currently working on we have a data ingestion process which needs to take zip files, unzip them and then import their contents into the database.

As a result we delegate from Scala code to the system unzip command like so:

def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We ran into a problem where the unzipping process was hanging and executing ‘ps’ showed us that the ‘unzip’ process was stuck in the ‘pipe_w’ (pipe waiting) state which suggested that it was waiting for some sort of input.

After a bit of googling Duncan found this blog which explained that we needed to process the output stream from our process otherwise it might end up hanging

a.k.a. RTFM:

The Runtime.exec methods may not work well for special processes on certain native platforms, such as native windowing processes, daemon processes, Win16/DOS processes on Microsoft Windows, or shell scripts.

The created subprocess does not have its own terminal or console. All its standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (Process.getOutputStream(), Process.getInputStream(), Process.getErrorStream()).

The parent process uses these streams to feed input to and get output from the subprocess.

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

For most of the zip files we presumably hadn’t been reaching the limit of the buffer because the list of files being sent to STDOUT by ‘unzip’ wasn’t that high.

In order to get around the problem we needed to gobble up the output stream from unzip like so:

import org.apache.commons.io.IOUtils
def extract {
  var command = "unzip %s -d %s" format("/file/to/unzip.zip", "/place/to/unzip/to")
  var process: Process = null
 
  try {
    process = Runtime.getRuntime.exec(command)
    val thisVariableIsNeededToSuckDataFromUnzipDoNotRemove = "Output: " + IOUtils.readLines(process.getInputStream)
    val exitCode = process.waitFor
  } catch {
    case e : Exception => // do some stuff
  } finally {
    // close the stream here
  }
}

We need to do the same thing with the error stream as well in case ‘unzip’ ends up overflowing that buffer as well.

On a couple of blog posts that we came across it was suggested that we should ‘gobble up’ the output and error streams on separate threads but we weren’t sure why exactly that was considered necessary…

If anyone knows then please let me know in the comments.

I recently came across a really cool blog post by Dr Nic titled ‘How to stop killing people with your public speeches‘ where he talks about the importance of practicing our presentations so that they actually make an impact on our audience.

Towards the end of the post he suggests joining Toastmasters as a useful first step for getting used to speaking to a group of people and as an added bonus you get feedback after each speech you give.

When I finished university in 2005 the one thing that I feared above all else was speaking in front of a group of people.

At the time I was an avid reader of Steve Pavlina’s blog and in one of his posts he recommended Toastmasters as a way of overcoming the fear.

I attended Toastmasters sessions every fortnight for about 18 months in 2006/2007, getting through 7 of the 10 speeches, before I decided to go off and learn open mic standup comedy, but that’s another story.

It’s a really safe environment to practice in because it doesn’t matter if you make mistakes and the other people are there to help you improve.

Apart from getting comfortable speaking to a group, the main thing I learnt from Toastmasters is that it’s much more fun doing a speech if you can see that the audience is engaged.

In order for that to happen that meant that I needed to write a speech which I found fun to deliver, an approach which can be applied to any presentation that we give.

In my case this means that there has to be some sort of humour in the talk but I imagine this may be different for other people.

A couple of years ago while I was preparing for a presentation I was going to give to the Sydney ALT .NET user group, Erik Doernenburg suggested to me that the talk would be more interesting if I put a personal spin on it and told it as a story.

I found that it was actually much more interesting to prepare the talk once I took this approach and it was easier to present since I was framing the talk as being about my experience rather than (implicitly) claiming to be an expert of some sort.

When I started at Toastmasters I used to write out my whole talk word for word but now I tend to have a rough outline in my head which leaves some room to ad lib depending on how it’s going.

I now feel really comfortable in front of a group of people so I’d certainly second Dr Nic in recommending Toastmasters as an excellent way of overcoming any fears of public speaking and honing the skill.

Useful resources

When I was a trainer at ThoughtWorks University presentations Sumeet recommended the following books which I think are worth reading:

• The Presentation Secrets of Steve Jobs – How to be Insanely Great in Front of Any Audience
• The Naked Presenter – Delivering Powerful Presentations With or Without Slides
• Brain Rules – 12 Principles for Surviving and Thriving at Work, Home, and School

We have some code in our application where we parse reasonably complex XML structures and then sometimes choose to get rid of certain elements from the structure.

When we wanted to get rid of an element we replaced that element with a SpecialNode:

val emptyNode = new scala.xml.SpecialNode() {
  def buildString(sb:StringBuilder) = new StringBuilder()
  def label = null
}

Unfortunately when you call #text on the node it results in the following exception which we only found out today:

> emptyNode.text
 
java.lang.StackOverflowError
	at scala.xml.NodeSeq$$anonfun$text$1.apply(NodeSeq.scala:152)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
	at scala.collection.Iterator$class.foreach(Iterator.scala:652)
	at scala.collection.LinearSeqLike$$anon$1.foreach(LinearSeqLike.scala:50)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
	at scala.xml.NodeSeq.foreach(NodeSeq.scala:43)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:194)
	at scala.xml.NodeSeq.map(NodeSeq.scala:43)
	at scala.xml.NodeSeq.text(NodeSeq.scala:152)
	at scala.xml.Node.text(Node.scala:200)

The way to get around that problem is to override the text method so it returns empty:

val emptyNode = new scala.xml.SpecialNode() {
  def buildString(sb:StringBuilder) = new StringBuilder()
  def label = null
  override def text = ""
}

> emptyNode.text
res1: String = ""

It took a seriously long time for us to track down what was going on and that bit of code wasn’t unit tested.

#fail

Towards the end of the week before last and the beginning of last week we’d been having quite a few problems with our QA environment to the point where we were unable to deploy anything to it for 3 days.

A few weeks ago I wrote about a 5 whys exercise that we did in a retrospective and in our weekly code review we decided to give it a go and see what we could learn.

We started with the question ‘Why was there a mess?‘ and then branched out the first level whys since it was fairly clear that there wasn’t only one thing which had contributed to our problems.

We ended up with 4 answers to the first why:

• There was a DNS change
• Volume was deleted from our QA server
• System tests failing
• Change in one project hanging QA deployment
• Main build broken for a while

We then worked across the whiteboard taking each of these in turn.

I think our approach allowed us to avoid part of ‘the cult of the root cause‘ which Don Reinertsen wrote about.

It still wasn’t quite spot on due to some mistakes I made while facilitating but these were my observations:

• Once we got to answering the whys for the 4th and 5th first level whys the whiteboard was way too cluttered and it had become quite difficult to see exactly where we’d got up to.

  As a result we lost the discipline around answering the question why and drifted off into general discussion around the original question but stopped drilling down further looking for a potential root cause.

  The next time I think it would probably work better to look for the first why and collect any potential other whys on the same level in a ‘parking lot’ type area which we could then go to later on.

• Having said that, a neat thing about having the whys alongside each other was that we were able to see that the first two whys were linked to each other.

  Both changes had been done by someone in the operations team based on conversations they had with people on our team.

  We realised that our communication with the operations team hadn’t been entirely clear and had left room for doubt which had led to unexpected changes being made to the servers.

  This was an example of us stopping before we’d drilled down to 5 levels having realised that we could influence the situation positively even if we hadn’t found the root cause of the problem.

• Drilling down into the ‘System tests failing’ led to the most interesting insights:
  • System tests failing
    • Noone cares about them
    • We can push to QA even if they’re broken
    • Used to them failing
      • Perception amongst devs that they’re flaky
        • There had previously been a time when data changed frequently and broke them.
    • Seen as being owned by the QAs
      • The tests were defined by QAs
    • The time from checkin to system tests failing is quite long

  Looking back at this now we probably should have drilled a bit further down on some of the whys.

  We actually ended up discussing the perception amongst the developers that the tests were flaky and it was pointed out that most of the failures were actually real.

  We don’t currently have a ‘stop the line’ mentality if the systems tests fail but have agreed to adopt that approach for the next iteration and check at the end of this week to see if we’ve improved.

• Even though I didn’t facilitate the exercise perfectly I think there was still a far greater level of analysis done by the team in this exercise than in others that I’ve seen.

  I’ve noticed that a lot of retrospective type exercises tend to only encourage surface level analysis so we never really go deeper into a subject and see if we can actually make some useful changes to the way that we work.

We had a problem to solve earlier in the week where we wanted to try and find out which files we had ingested into our database based on a unique identifier.

We had a few hundred thousand files to search through to try and find the ones where around 50,000 identifiers were mentioned so that we could re-ingest them.

Running a normal grep for each identifier individually took a ridiculously long time so we needed to find a way to search for all of the identifiers at the same time to speed up the process.

Luckily my colleague knew about fgrep which allowed us to do this.

fgrep is essentially grep (or egrep) with no special characters. If you want to search for a simple string without wild cards, use fgrep. The fgrep version of grep is optimized to search for strings as they appear on the command line, so it doesn’t treat any characters as special.

We created a file containing all the identifiers:

identifiers.txt

identifier1
identifier2
identifier3

And then created the following command to identify which files those identifiers existed in:

fgrep -Rl -f identifiers.txt .

We passed the ‘-l’ flag because we don’t care where in the file the identifier matches, we just care that it exists in the file.

If we only have a few different things to search for then we could supply those directly to ‘fgrep’ without the file:

fgrep -Rl -e "identifier1" -e "identifier2" -e "identifier3" .

I haven’t used ‘fgrep’ before but it came in quite useful for us here. I also came across this article which explains the different variants of grep in more details.