Mark Needham

We’ve had an xsbt branch on our gitolite powered repository for the last couple of weeks while we worked out how to move our build from sbt 0.7 to sbt 0.10 but having finally done that we needed to delete it.

I originally tried running the following command from one of our developer workstations:

git push origin :xsbt

But ended up with the following error:

remote: error: denying ref deletion for regs/head/xsbt

! [remote rejected] xsbt (deletion prohibited)

A bit of googling led me to this stackoverflow thread which suggested that you needed to be an administrator in order to delete a remote branch.

Once we’ve done that we can run the following command on each machine to delete the remote tracking reference to the repository:

git branch -d -r origin/xsbt

A few days ago I tweeted that I didn’t really see the point in structured types in Scala…

Not sure I understand where you would use structural types in #scala instead of defining a method on a trait http://bit.ly/jgiW7b

…but today my colleague Uday came up with a cool way of combining self type annotations with structured types inside a trait we defined.

We had some code duplicated across two classes which looked roughly like this:

class OnePageType {
  lazy val peopleNodes = root \\ "SomeNode" \ "SomeSubNode" \ "People" \ "Person"
  private def fullName(personName: Node): String = // code to build person's name
 
  lazy val people: String = peopleNodes.map(fullName).mkString(", ")
}

class AnotherPageType {
  lazy val peopleNodes = root \\ "OtherNode" \ "OtherSubNode" \ "People" \ "Person"
  private def fullName(personName: Node): String = // code to build person's name
 
  lazy val people: String = peopleNodes.map(fullName).mkString(", ")
}

The first line is different but the other two are identical because the data is stored in exactly the same format once we get down to that level.

Since We want to keep the XPathish queries as descriptive as possible so that we don’t accidentally end up pulling the wrong elements onto the page, making those a bit looser wasn’t an option in this case.

Instead we pulled out a trait like so:

1
2
3
4
5
6
7

trait People {
  self: {val peopleNodes: NodeSeq} =>
 
  private def fullName(personName: Node): String = // code to build person's name
 
  lazy val people: String = peopleNodes.map(fullName).mkString(", ")
}

Which we include in the classes like this:

class OnePageType extends People {}
class AnotherPageType extends People {}

What we’re done on line 2 of the People trait is to define a self annotation which says that we need a val of peopleNodes to be present on the classes in which the trait is mixed.

If a val of peopleNodes doesn’t exist then the class won’t compile!

In this case the structure type works quite well because we wouldn’t really want to pull out peopleNodes into a trait just to reference it as a self type annotation.

In ‘Thinking In Systems: A Primer‘ one of the most interesting ideas that Donella Meadows describes is what Herbert Simon coined ‘bounded rationality‘:

Bounded rationality means that people make quite reasonable decisions based on the information they have. But they don’t have perfect information, especially about more distant parts of the system

Later on in the chapter the following idea is suggested:

If you become a manager, you probably will stop seeing labour as a deserving partner in production, and start seeing it as a cost to be minimised.

This helps explains something that I’ve noticed happen quite frequently.

Someone who was previously non management gets pulled into a management position and ‘mysteriously’ starts acting exactly like all the others in that type of role rather than having a holistic view.

The strange thing is that we don’t expect this to happen. The person was on ‘our’ side very recently so surely they should be able to see both perspectives!

Esther Derby referred to this problem in her keynote at XP2011 where she talked about two different types of information that occur in a system:

• Day to day information – this is possessed by people ‘on the ground’
• System information – this is possessed by people ‘in management’

When the people who recently moved into a management position are challenged on this they will often point out that “you can’t see the bigger picture” which is true but still doesn’t account for the fact that they probably aren’t seeing it either!

We’re both just seeing different parts of the system.

Meadows goes on to point out that the design of the system tends to encourage this type of behaviour:

Seeing how individual decisions are rational within the bounds of the information available does not provide an excuse for narrow-minded behaviour. It provides an understanding of why that behaviour arises.

Taking out one individual from a position of bounded rationality and putting in another person is not likely to make much difference. Blaming the individual rarely helps create a more desirable outcome.

Meadows finishes this section of the book with the following suggestion which I think is especially useful in a consulting environment where both consultants and management quite obviously tend to suffer from bounded rationality.

It’s amazing how quickly and easily behaviour changes can come, with even slight enlargement of bounded rationality, by providing better, more complete, timelier information.

I’ve seen various attempts at trying to help people enlarge their bounded rationality at ThoughtWorks, such as:

• Presentations by the finance director showing where the revenue of the company gets spent
• Management team members taking the time to have one on one discussions with consultants
• Discussions about the sales pipeline and the types of work available in the market

I think if this type of thing happened more frequently then you’d probably see an enlargement of everyone’s bounded rationality which would be useful for all involved!

As I’ve mentioned before, we’re making use of a MarkLogic database on the project I’m working on which means that we’re getting quite big XML data structures coming into our application whenever we execute a query.

The normal way that I’ve seen for dealing with external systems would be to create an anti corruption layer where we initialise objects in our system with the required data from the external system.

In this case we’ve decided that approach doesn’t seem to make as much sense because we don’t need to do that much with the data that we get back.

We effectively map straight into a read model where the only logic is some formatting for how the data will be displayed on the page.

The read model objects look a bit like this:

class Content(root : xml.Node) {
    def numberOfResults: Int = (root \ "@count").text.toInt
}

They are just lightweight wrapper objects and we make use of Scala’s XML support to retrieve the various bits of content onto the page.

The advantage of doing things this way is that it means we have less code to write than we would with the serialisation/deserialisation approach although it does mean that we’re strongly coupled to the data format that our storage mechanism uses.

However, since this is one bit of the architecture which is not going to change it seems to makes sense to accept the leakage of that layer.

So far the approach seems to be working out fine but it’ll be interesting to see how well it holds up if those lightweight wrappers do end up needing to have more logic in them.

As I mentioned a couple of months ago, while I was in India I was acting as the Tech Lead on the project the TWU grads were working on and one thing I learnt from doing that is the importance of trying to keep the passion of the developers on the team.

When we started off I was more focused on trying to encourage the team to try and develop as many of the stories as possible.

This meant that on one occasion when one of the developers was really keen to try something out which he thought would make our life easier I overruled him in favour of working on a story that more directly contributed to what we were trying to do.

Luckily he didn’t get dissuaded by me doing that and a few days later he again asked if he could explore that avenue as he was convinced it would be useful.

We decided that it would be worth trying for the rest of the day and it was noticeable that he was significantly more pumped up/enthusiastic on most things that he worked on after that.

I appreciate that I was in an unusual situation where the output of the project came second to the learning of the participants but I still feel like there is something to be said for helping encourage and keep the passion of people on our teams.

It will mean sacrificing a bit of short term productivity, but if it means that the person has satisfied their curiosity about the viability of a bit of technology or even better has solved a problem for the team by doing so then it might be worth it.

As I found out it’s much easier to knock someone back and you have justification to do so because what they want to do isn’t directly on the value path of what’s being paid for but that might not be the best decision if you look at the whole!

I’d be curious to hear the opinions of people who have played the tech lead role frequently because my observations are only based on a short stint and from being on the other side of the equation!

We have a variety of val, lazy val and def definitions across our code base but have been led to believe that idiomatic Scala would have us using lazy val as frequently as possible.

As far as I understand so far this is what the different things do:

• val evaluates as soon as you initialise the object and stores the result.
• lazy val evaluates the first time that it’s accessed and stores the result.
• def executes the piece of code every time – pretty much like a Java method would.

In Java, C# or Ruby I would definitely favour the 3rd option because it reduces the amount of state that an object has to hold.

I’m not sure that having that state matters so much in Scala because all the default data structures we use are immutable so you can’t do any harm by having access to them.

I recently read an interesting quote from Rich Hickey which seems applicable here:

To the extent the data is immutable, there is little harm that can come of providing access, other than that someone could come to depend upon something that might change. Well, okay, people do that all the time in real life, and when things change, they adapt.

If the data was mutable then it would be possible to change it from any other place in the class which would make it difficult to reason about the object because the data might be in an unexpected state.

If we define something as a val in Scala then it’s not even possible to change the reference to that value so it doesn’t seem problematic.

Perhaps I just require a bit of a mind shift to not worry so much about state if it’s immutable.

It’s only been a few weeks so I’d be interested to hear the opinions of more seasoned Scala users.

—

I’ve read that there are various performance gains to be had from making use of lazy val or def depending on the usage of the properties but that would seem to be a premature optimisation so we haven’t been considering it so far.

We’re using the Mustache templating engine on my project at the moment and one thing that we wanted to do was build a comma separated list.

Mustache is designed so that you pretty much can’t do any logic in the template which made it really difficult to do what we wanted.

It’s easy enough to get a comma after each item in a list with something like the following code:

{{#people}}<a href="/link/to/{{toString}}">{{toString}}</a>{{/people}}

where people is passed to the template as a collection of strings.

To get rid of the trailing comma we ended up building a collection of Pairs containing the person’s name and a boolean value indicating whether or not to show the comma.

We need to show the comma before every element except for the first one so we can pass the following collection to the template:

val values = names.zipWithIndex.map { case(item, index) => if(index == 0) (item, false) else (item, true) }

zipWithIndex returns a collection of pairs containing the original strings and their position in the collection.

We can then map to a different result for just the first element and then use those pairs in the template like so:

{{#people}} {{#_2}}, {{/_2}}<a href="/link/to/{{_1}}">{{_1}}</a>{{/people}}

It’s truly horrendous so if anyone knows a better way then please let me know!

One of the stories we worked on last week had us needing to be able to customise the output of a MarkLogic search query to include some elements which aren’t included in the default view.

We started off with this:

search.xqy

xquery version "1.0-ml";
import module namespace search = "http://marklogic.com/appservices/search" at "/MarkLogic/appservices/search/search.xqy";
 
declare variable $term as xs:string := xdmp:get-request-field("query", "");
 
search:search($term)

Which gives us back a list of results showing where in the documents the search term appeared.

We wanted to be able to get the title of the document and some other meta data about it though so we needed to make use of the transform-results option to do this.

The original query changes to look like this:

search.xqy

xquery version "1.0-ml";
import module namespace search = "http://marklogic.com/appservices/search" at "/MarkLogic/appservices/search/search.xqy";
 
declare variable $term as xs:string := xdmp:get-request-field("query", "");
 
search:search($term,
   <options xmlns="http://marklogic.com/appservices/search">
     <transform-results apply="transformed-result" ns="http://www.markhneedham.com/search" at="transform.xqy" />
   </options>)

And then we have another xquery file which we use to do the transformation:

transfom.xqy

xquery version "1.0-ml";
module namespace custom-search = "http://markhneedham.com/search";
import module namespace search = "http://marklogic.com/appservices/search" at "/MarkLogic/appservices/search/search.xqy";
 
declare function custom-search:transformed-result(
         $result as node(),
         $ctsquery as schema-element(cts:query),
         $options as element(search:transform-results)?
) as element(search:snippet)
{
<search:snippet>
{
(
         search:snippet($result, $ctsquery, $options)/*,
         <extra>{$result//*:Id}{$result//*:Title}</extra>
)
}
</search:snippet>
};

We put the extra elements that we wanted to include in our search result in the tag but we can name that anything we want.

The code snippets assume that both search.xqy and transform.xqy are both at the top level of the documents database.

I’ve been playing around with Chef Solo on Fedora and executing the following:

sudo chef-solo -c config/solo.rb -j config/node.json

(where node.json just contains the example code from the resolver example on the Chef documentation page and the cookbooks folder contains all the opscode cookbooks.)

leads to the following error:

...
ERROR: Running exception handlers
ERROR: Exception handlers complete
FATAL: Stacktrace dumped to /home/mark/chef-solo/chef-stacktrace.out
FATAL: ArgumentError: Attribute domain is not defined!

A bit of googling led me to believe that this error is happening because the machine doesn’t have a fully qualified domain name (fqdn) defined which can be seen by calling the following command:

$ hostname -f
> hostname: Name of service not known

One way to fix it is to add the following entry to /etc/hosts

127.0.0.1	mark-fedora

Which results in the script running fine with no errors.

...
INFO: Running report handlers
INFO: Report handlers complete

A suggestion I read while googling about fqdn was to add the hostname of the machine into a file called /etc/HOSTNAME but that didn’t seem to have any impact for me.

On the Mac hostname -f works fine even without an entry like the above in /etc/hosts so I’m not entirely sure how it all works!

If anyone could explain it to me that’d be awesome.

We’re using the MarkLogic database on my current project and something that we wanted to do recently was delete all the documents as part of a deployment script.

Getting all of the documents is reasonably easy – we just need to make a call to the doc() function.

We can then iterate through the documents like so:

for $doc in doc() return $doc

We wanted to make use of the xdmp:document-delete function to tear down all of the modules but that needs a uri representing the location of the document in the database which isn’t available in $doc:

xdmp:document-delete

xdmp:document-delete(
$uri as xs:string
) as empty-sequence()

Summary:

Deletes a document from the database.

A colleague pointed out that what we needed to do was pass the document to xdmp:node-uri and then our troubles would be over!

The final solution therefore looks like this:

for $doc in doc() return xdmp:document-delete(xdmp:node-uri($doc))

I was expecting that we would delete the documents by some sort of identifier but I guess this approach makes more sense given the way the data is stored.

It all seems a bit esoteric at the moment!