Mark Needham

Thoughts on Software Development

Archive for the ‘Javascript’ Category

Leaflet JS: Resizing a map to keep a circle diameter inside it

with 2 comments

I’ve been working on creating a UI to make searching for the football stadiums that I wrote about last week a bit easier and I thought I’d give Leaflet JS a try.

Leaflet is a Javascript library which was recommended to me by Jason Neylon) and can be used as a wrapper around Open Street Map.

I started by creating a simple form where you could fill in a lat/long and distance and it would centre the map on that lat/long and show you a list of the stadiums within that diameter next to the map.

Having done that I wanted to draw the diameter onto the map and then show the location of the stadiums which fitted inside the circle.

I had the following code to centre the map and draw a circle:

  var distance = 10;
  $("#inputDistance").val(distance);
  var latLong=[51.505, -0.11398315429687499];
  $("#inputLatLong").val(latLong)
 
  var map = L.map('map').setView(latLong,11);
  var layer = L.tileLayer('http://{s}.tile.cloudmade.com/e7b61e61295a44a5b319ca0bd3150890/997/256/{z}/{x}/{y}.png', { maxZoom: 18 });
  layer.addTo(map);
 
  var currentDiameter = L.circle(latLong, distance * 1000);
  currentDiameter.addTo(map);
 
  var currentPositionMarker = L.marker([latLong[0], latLong[1]]);
  currentPositionMarker.addTo(map);

which creates this map:

Map

I wanted to be able to change the diameter of the circle from the form and have it pick up more stadiums which I did with the following code:

  $("#inputDistance").change(function() {	
    map.removeLayer(currentDiameter);
    currentDiameter = L.circle(currentPositionMarker.getLatLng(), $("#inputDistance").val() * 1000);
    currentDiameter.addTo(map);
  });

I updated the diameter to be 16km and the map looked like this:

Map diameter

It just about fits inside the map but setting it to anything higher means that the area of the diameter falls outside of the visible map which is annoying.

I wanted to be able to resize the map when the circle changed in size and after a bit of browsing of the Leaflet code I came across a function called ‘fitBounds’ which lets us achieve this. I changed the code like so:

  $("#inputDistance").change(function() {	
    map.removeLayer(currentDiameter);
    currentDiameter = L.circle(currentPositionMarker.getLatLng(), $("#inputDistance").val() * 1000);
    currentDiameter.addTo(map);
    map.fitBounds(currentDiameter.getBounds());
  });

Now if I change the distance the map resizes too:

Map diameter fixed

Much better!

The full code to do this reads like so:

$(document).ready(function() {		
  var distance = 10;
  $("#inputDistance").val(distance);
  var latLong=[51.505, -0.11398315429687499];
  $("#inputLatLong").val(latLong)
 
  var map = L.map('map').setView(latLong,11);
  var layer = L.tileLayer('http://{s}.tile.cloudmade.com/e7b61e61295a44a5b319ca0bd3150890/997/256/{z}/{x}/{y}.png', { maxZoom: 18 });
  layer.addTo(map);
 
  var currentDiameter = L.circle(latLong, distance * 1000);
  currentDiameter.addTo(map);
 
  var currentPositionMarker = L.marker([latLong[0], latLong[1]]);
  currentPositionMarker.addTo(map);
 
  $("#inputDistance").change(function() {	
    map.removeLayer(currentDiameter);
    currentDiameter = L.circle(currentPositionMarker.getLatLng(), $("#inputDistance").val() * 1000);
    currentDiameter.addTo(map);
    map.fitBounds(currentDiameter.getBounds());
  }); 
});

The code for this is all on github although I’ve refactored it a bit now so it doesn’t look exactly like this. I tried to put it on jsfiddle as well but it didn’t seem to work very well so screenshots it is!

Written by Mark Needham

June 30th, 2013 at 10:23 pm

Posted in Javascript

Tagged with ,

jQuery: Collecting the results from a collection of asynchronous requests

with 5 comments

Liz and I recently spent some time building a pair stair to show how long ago people had paired with each other and one of the things we had to do was make AJAX requests to get the pairing data for each person and then collate it all to build the stair.

Pair stair

The original attempt to do this looked a bit like this:

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];
 
var grid = [];
$.each(people, function(index, person) {
  $.getJSON('/git/pairs/' + person, function(data) {
    // parse data and create somethingCool
    grid.push(somethingCool);
  });  
});
 
// do something with grid

When we try to do something with grid it is of course empty because we’ve attempted to access it before all of the callbacks (which populate it) have returned.

Pedro Teixeira has a nice blog post which explains how to solve this problem in node.js and we can use the same pattern here.

We need to write our own looping mechanism which is able to determine when the last callback has returned.

This is done by creating a copy of the people array and then manually iterating through it using shift.

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];
var peopleCopy = people.slice(0), grid = [];
(function getPairs() {
  var person = peopleCopy.shift();
 
  if(peopleCopy.length == 0) {
    // do something with grid
  } else {
    $.getJSON("/git/pairs" + person, function(data) {
      // parse data and create somethingCool
      grid.push(somethingCool);
      getPairs();		
    })						
  }
})();

I tried to extract the asynchronous looping and ended up with the following function:

function asyncLoop(collection, seedResult, loopFn, completionFn) {
  var copy = collection.slice(0);
  (function loop() {
    var item = copy.shift();
 
    if(copy.length == 0) {
      completionFn(seedResult);
    } else {
      loopFn(item, seedResult, loop);
    }	
  })();	
}

Which could be called like this:

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];
asyncLoop(people, [], function(name, grid, callBackFn) {
  // parse data and create something cool
  grid.push(somethingCool);
  callBackFn();
}, function(grid) {
  // do something with grid
});

I’m not sure that it reads that much clearer but it does push some of the boiler plate code away.

Written by Mark Needham

September 25th, 2011 at 9:26 am

Posted in Javascript,jQuery

Tagged with ,

node.js: child_process.exec not returning all results

with one comment

I’ve been playing around with some node.js code to get each of the commits from our git repository but noticed that it didn’t seem to be returning me all the results.

I had the following code:

var exec = require('child_process').exec;
 
var gitRepository = '/some/local/path';
 
exec('cd ' + gitRepository + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw ', function(error, stdout, stderror) {
  var commits = stdout.split("\n");
 
  // do some stuff with commits
});

We have around 2000 commits in the repository but I was only getting back 1600 of them when I checked the length of commits.

Eventually I decided to print out what was in error and got the following message:

error: Error: maxBuffer exceeded.

Going back to the documentation revealed my mistake:

maxBuffer specifies the largest amount of data allowed on stdout or stderr – if this value is exceeded then the child process is killed.

The default options are

{ encoding: ‘utf8′,
timeout: 0,
maxBuffer: 200*1024,
killSignal: ‘SIGTERM’,
cwd: null,
env: null }

The limit is 2048000 which is around about the number of bytes being returned when I get to 1600 commits.

Changing the code to increase the buffer sorts it out:

var exec = require('child_process').exec;
 
var gitRepository = '/some/local/path';
 
exec('cd ' + gitRepository + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw ', {maxBuffer: 500*1024}, function(error, stdout, stderror) {
  var commits = stdout.split("\n");
 
  // do some stuff with commits
});

Written by Mark Needham

September 22nd, 2011 at 7:55 pm

Posted in Javascript

Tagged with ,

Javascript: Internet Explorer 8 – trim() leads to ‘Object doesn’t support this property or method’ error

with 3 comments

We make use of the Javascript trim() function in our application but didn’t realise that it isn’t implemented by Internet Explorer until version 9.

This led to the following error on IE8 when we used it:

Message: Object doesn’t support this property or method
Line: 18
Char: 13
Code: 0
URI: http://our.app/file.js

There’s a stackoverflow thread suggesting some different ways of implementing your own ‘trim()’ method but since we’re using jQuery already we decided to just use the ‘$.trim()’ function from there.

Therefore:

var cleaned = ourString.trim();

becomes:

var cleaned = $.trim(ourString);

I’m sure I must have come across this before but I don’t remember when!

Written by Mark Needham

September 13th, 2011 at 1:33 pm

Posted in Javascript

Tagged with

Learning node.js: Step

with 5 comments

I’ve been playing around with node.js to generate some graphs from our git repository which effectively meant chaining together a bunch of shell commands to give me the repository data in the format I wanted.

I was able to do this by making use of child_process which comes with the core library.

The first version looked like this:

var exec = require('child_process').exec, _ = require("underscore");
...
function parseCommitsFromRepository(fn) {
  var gitRepository = "/tmp/core";
  var gitPlayArea = "/tmp/" + new Date().getTime();
 
  exec('cd ' + gitRepository + ' && git reset HEAD', function() {
    exec('git clone ' + gitRepository + ' ' + gitPlayArea, function() {
      exec('cd ' + gitPlayArea + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw', function(blank, gitEntries) {
        var commits = _(gitEntries.split("\n")).chain()
                        .filter(function(item) { return item != ""; })
                        .map(function(item) { return item.split("|") })
                        .filter(function(theSplit) { return theSplit !== undefined && theSplit[1] !== undefined && theSplit[2] !== undefined; })
                        .map(function(theSplit) {  
                          var date = new Date(theSplit[1].trim().split(" ")[0]*1000);
                          return {message: theSplit[2].trim(), date: date.toDateString(), time : date.toTimeString()}; })
                        .value();			
        fn(commits);
      });		
    });
  });
}

node.js has an asynchronous programming model so the majority of the time we have to pass callbacks to other functions which get called when the asynchronous computation has completed.

In this case there’s an order dependency in the parseCommitsFromRepository function such that we need to nest the second call to exec inside the callback from the first call.

i.e. we don’t want to get the log of the repository before we’ve cloned the repository to the location that we’re trying to get that log from.

As you create more and more order dependencies between asynchronous functions the nesting becomes greater and the code moves more and more to the right hand side of the screen.

I came across the Step library which allows you to stack up functions and have the results from each one get passed on to the next.

I decided to try it in my code and it ended up looking like this:

function parseCommitsFromRepository(fn) {	
  var gitRepository = "/tmp/core";
  var gitPlayArea = "/tmp/" + new Date().getTime();	
  Step(
    function getRepositoryUpToDate() { exec('cd ' + gitRepository + ' && git reset HEAD', this); },
    function cloneRepository()       { exec('git clone ' + gitRepository + ' ' + gitPlayArea, this); },
    function getGitEntries()         { exec('cd ' + gitPlayArea + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw', this); },
    function handleResponse(blank, gitEntries) {
      var commits = _(gitEntries.split("\n")).chain()
                      .filter(function(item) { return item != ""; })
                      .map(function(item) { return item.split("|") })
                      .filter(function(theSplit) { return theSplit !== undefined && theSplit[1] !== undefined && theSplit[2] !== undefined; })
                      .map(function(theSplit) {  
                        var date = new Date(theSplit[1].trim().split(" ")[0]*1000);
                        return {message: theSplit[2].trim(), date: date.toDateString(), time : date.toTimeString()}; })
                      .value();			
      fn(commits);
    }
  );	
}

An interesting side effect of using this approach is that we can describe what each exec call is doing in the name of the function that executes it.

Another neat thing about this library is that I can easily wrap those functions inside a logging function if I want to see on the console where the process has got up to:

function log(message, fn) {
  return function logMe() {
    console.log(new Date().toString() + ": " + message);
     fn.apply(this, arguments);
  }
}
function parseCommitsFromRepository(fn) {	
  var gitRepository = "/tmp/core";
  var gitPlayArea = "/tmp/" + new Date().getTime();	
  Step(
    log("Resetting repository", function getRepositoryUpToDate() { exec('cd ' + gitRepository + ' && git reset HEAD', this); }),
    log("Cloning repository", function cloneRepository()         { exec('git clone ' + gitRepository + ' ' + gitPlayArea, this); }),
    log("Getting log", function getGitEntries()                  { exec('cd ' + gitPlayArea + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw', this); }),
    log("Processing log", function handleResponse(blank, gitEntries) {
      var commits = _(gitEntries.split("\n")).chain()
                      .filter(function(item) { return item != ""; })
                      .map(function(item) { return item.split("|") })
                      .filter(function(theSplit) { return theSplit !== undefined && theSplit[1] !== undefined && theSplit[2] !== undefined; })
                      .map(function(theSplit) {  
                        var date = new Date(theSplit[1].trim().split(" ")[0]*1000);
                        return {message: theSplit[2].trim(), date: date.toDateString(), time : date.toTimeString()}; })
                      .value();			
      fn(commits);
    })
  );	
}

I then get this output when executing the function:

Sun Sep 11 2011 23:33:09 GMT+0100 (BST): Resetting repository
Sun Sep 11 2011 23:33:11 GMT+0100 (BST): Cloning repository
Sun Sep 11 2011 23:33:24 GMT+0100 (BST): Getting log
Sun Sep 11 2011 23:33:24 GMT+0100 (BST): Processing log

There are more cool ways to use the Step library on the github page – what I’ve described here is only a very simple use case.

Written by Mark Needham

September 11th, 2011 at 10:37 pm

Posted in Javascript

Tagged with ,

node.js: Building a graph of build times using the Go API

with 3 comments

I’ve been playing around with node.js again and one thing that I wanted to do was take a CSV file generated by the Go API and extract the build times so that we could display it on a graph.

Since I don’t have a Go instance on my machine I created a URL in my node application which would mimic the API and return a CSV file.

I’m using the express web framework to take care of some of the plumbing:

dashboard.js

var express = require('express')
var app = express.createServer();
 
app.get('/fake-go', function(req, res) {
  fs.readFile('go.txt', function(err, data) {
    res.attachment("data.csv");
    res.end(data, 'UTF-8');		
  });
});

go.txt is just in my home directory and looks like this:

cruise_agent,cruise_job_duration,cruise_job_id,cruise_job_result,cruise_pipeline_counter,cruise_pipeline_label,cruise_stage_counter,cruise_timestamp_01_scheduled,cruise_timestamp_02_assigned,cruise_timestamp_03_preparing,cruise_timestamp_04_building,cruise_timestamp_05_completing,cruise_timestamp_06_completed,tests_failed_count,tests_ignored_count,tests_total_count,tests_total_duration
TheOriginalAndTheBest,275,1812,Passed,647,0.647,1,2011-08-02T14:48:33+01:00,2011-08-02T14:48:45+01:00,2011-08-02T14:48:56+01:00,2011-08-02T14:48:57+01:00,2011-08-02T14:53:11+01:00,2011-08-02T14:53:32+01:00,0,0,375,0.076
TheOriginalAndTheBest,20,1815,Cancelled,648,0.648,1,2011-08-02T15:09:32+01:00,2011-08-02T15:09:46+01:00,2011-08-02T15:09:56+01:00,2011-08-02T15:09:56+01:00,,2011-08-02T15:10:17+01:00,,,,
TheOriginalAndTheBest,268,1817,Passed,649,0.649,1,2011-08-02T15:14:20+01:00,2011-08-02T15:14:30+01:00,2011-08-02T15:14:40+01:00,2011-08-02T15:14:41+01:00,2011-08-02T15:18:49+01:00,2011-08-02T15:19:09+01:00,0,0,368,0.074
TheOriginalAndTheBest,272,1822,Passed,650,0.650,2,2011-08-02T15:30:31+01:00,2011-08-02T15:30:41+01:00,2011-08-02T15:30:51+01:00,2011-08-02T15:30:52+01:00,2011-08-02T15:35:05+01:00,2011-08-02T15:35:24+01:00,0,0,368,0.083
TheOriginalAndTheBest,271,1825,Passed,651,0.651,1,2011-08-02T15:38:33+01:00,2011-08-02T15:38:44+01:00,2011-08-02T15:38:54+01:00,2011-08-02T15:38:54+01:00,2011-08-02T15:43:06+01:00,2011-08-02T15:43:26+01:00,0,0,368,0.093

I wanted to create an end point which I could call and get back a JSON representation of all the different builds.

app.get('/go/show', function(req, res) {
  var site = http.createClient(3000, "localhost"); 
  var request = site.request("GET", "/fake-go", {'host' : "localhost"})
  request.end();
  request.on('response', function(response) {
    var data = "";
    response.setEncoding('utf8');
 
    response.on('data', function(chunk) {
      data += chunk;
    });
 
    response.on('end', function() {
      var lines = data.split("\n"), buildTimes = [];
      lines.forEach(function(line, index) {
        var columns = line.split(",");
        if(index != 0 && nonEmpty(columns[9]) && nonEmpty(columns[11]) && columns[3] == "Passed") {
          buildTimes.push({ start :  columns[9], end : columns[11]});
        }
      });
 
      res.contentType('application/json');
      res.send(JSON.stringify(buildTimes));			
    });
  });	
});
 
function isEmpty(column) {
  return column !== "" && column !== undefined
}

I should probably use underscore.js for some of that code but I didn’t want to shave that yak just yet!

I have a default route setup so that I can just go to localhost:3000 and see the graphs:

app.get('/', function(req, res){
  res.render('index.jade', { title: 'Dashboard' });
});

On the client side we can then create a graph using the RGraph API:

index.jade

h2(align="center") Project Dashboard
script
  function drawGoGraph(buildTimes) {		
    var go = new RGraph.Line('go', _(buildTimes).map(function(buildTime) { return (new Date(buildTime.end) - new Date(buildTime.start)) / 1000 }).filter(function(diff) { return diff > 0; }));
    go.Set('chart.title', 'Build Times');		
    go.Set('chart.gutter.top', 45);
    go.Set('chart.gutter.bottom', 125);
    go.Set('chart.gutter.left', 50);
    go.Set('chart.text.angle', 90);
    go.Set('chart.shadow', true);
    go.Set('chart.linewidth', 1);
 
    go.Draw();		
  }
 
  $(document).ready(function() {
    $.getJSON('/go/show', function(data) {
	  drawGoGraph(data);
    });
  });
 
div(align="center")
  canvas(id="go", width="500", height="400")
    [Please wait...]

We just do some simple subtraction between the start and end build times and then filter out any results which have an end time before the start time. I’m not entirely sure why we end up with entries like that but having those in the graph totally ruins it!

We include all the .js files in the layout.jade file.

layout.jade

!!! 5
html(lang="en")
  head
    title Project Dashboard
    script(src='RGraph/libraries/RGraph.common.core.js')
    script(src="RGraph/libraries/RGraph.common.context.js")
    script(src="RGraph/libraries/RGraph.common.annotate.js")
    script(src="RGraph/libraries/RGraph.common.tooltips.js")
    script(src="RGraph/libraries/RGraph.common.zoom.js")
    script(src="RGraph/libraries/RGraph.common.resizing.js")
    script(src="RGraph/libraries/RGraph.line.js")
    script(src="jquery-1.6.2.min.js ")
    script(src="underscore-min.js")

Et voila:

Build graph

Written by Mark Needham

August 13th, 2011 at 2:52 pm

Posted in Javascript

Tagged with

node.js: A little application with Twitter & CouchDB

with 5 comments

I’ve been continuing to play around with node.js and I thought it would be interesting to write a little application to poll Twitter every minute and save any new Tweets into a CouchDB database.

I first played around with CouchDB in May last year and initially spent a lot of time trying to work out how to install it before coming across CouchDBX which gives you one click installation for Mac OS X.

I’m using sixtus’ node-couch library to communicate with CouchDB and I’ve written a little bit of code that allows me to call the Twitter API.

What did I learn?

  • I’ve been reading through Brian Guthrie’s slides from his ‘Advanced Ruby Idioms So Clean You Can Eat Off Of Them‘ talk from RubyConfIndia and one of the suggestions he makes is that in Ruby there are only 6 acceptable types of signatures for functions:
    • 0 parameters
    • 1 parameter
    • 2 parameters
    • A hash
    • 1 parameter and a hash
    • A variable number of arguments passed in as an array

    It seems to me that the same guidelines would be applicable in JavaScript as well except instead of a Hash we can pass in an object with properties and values to serve the same purpose. A lot of the jQuery libraries I’ve used actually do this anyway so it’s an idiom that’s in both languages.

    I originally wrote my twitter function it so that it would take in several of the arguments individually:

    export.query = function(username, password, method) { ... }

    After reading Brian’s slides I realised that this was quickly going to become a mess so I’ve changed the signature to only take in the most important parameter (‘method’) on its own with the other parameters passed in an ‘options’ object:

    export.query = function(method, options) { ... }

    I’ve not written functions that take in parameters like this before but I really like it so far. It really helps simplify signatures while allowing you to pass in extra values if necessary.

  • I find myself porting higher order functions from C#/F# into JavaScript whenever I can’t find a function to do what I want – it’s fun writing them but I’m not sure how idiomatic the code I’m writing is.

    For example I wanted to write a function to take query parameters out of an options object and create a query string out of them. I adapted the code from node-couch and ended up with the following:

    Object.prototype.filter = function(fn) {
        var result = {};
        for (var name in this) {
            if (this.hasOwnProperty(name)) {
                if (fn.call(this[name], name, this[name])) {
                    result[name] = this[name];
                }
            }
        }
        return result;
    };
     
    Object.prototype.into = function(theArray, fn) {
        for (var name in this) {
            if (this.hasOwnProperty(name)) {
                theArray.push(fn.call(this[name], name, this[name]));
            }
        }
        return theArray;
    };
     
    function encodeOptions(options) {
        var parameters = [];
        if (typeof(options) === "object" && options !== null) {
            parameters = options
                            .filter(function(name) {
                                return !(name === "username" || name === "password" || name === "callback");})
                            .into([], function(name, value) {
                                return encodeURIComponent(name) + "=" + encodeURIComponent(value); });
        }
        return parameters.length ? ("?" + parameters.join("&")) : "";
    }

    I’m not sure how wise it is adding these functions to the object prototype – I haven’t had any problems so far but I guess if other libraries I’m using changed the prototype of these built in types in the same way as I am then I might get unexpected behaviour.

    Would the typical way to defend against this be to check if a function is defined before trying to define one and throwing an exception if so? Or is adding to the prototype just a dangerous thing to do altogether?

    Either way I’m not altogether convinced that the code with these higher order functions actually reads better than it would without them.

  • I’m finding it quite interesting that a lot of the code I write around node.js depends on callbacks which means that if you have 3 operations that depend on each other then you end up with nested callbacks which almost reads like code written in a continuation passing style.

    For example I have some code which needs to do the following:

    • Query CouchDB to get the ID of the most recently saved tweet
    • Query Twitter with that ID to get any tweets since that one
    • Save those tweets to CouchDB
    var server = http.createServer(function (req, res) {
        couchDB.view("application/sort-by-id", {
            descending : true,
            success : function(response) {
                twitter.query("friends_timeline", {
                    ...
                    since_id : response.rows[0].value.id,
                    callback : function(tweets) {
                        tweets.each(function(tweet) {
                            couchDB.saveDoc(tweet, {
                                success : function(doc) {
                                    sys.log("Document " + doc._id + " successfully saved")
                                },
                                error : function() {
                                    sys.log("Document failed to save");
                                }
                            });
                        }); 
     
                    }
                });
            },
            error : function() {
                sys.log("Failed to retrieve documents");
            }
        });
     
        ...
    });

    There’s a ‘success’ callback for calling ‘couchDB.view’ and then a ‘callback’ callback for calling ‘twitter.query’ and finally a ‘success’ callback from calling ‘couchDB.saveDoc’.

    To me it’s not that obvious what the code is doing at first glance – perhaps because I’m not that used to this style of programming – but I’m intrigued if there’s a way to write the code to make it more readable.

  • I haven’t yet worked out a good way to test drive code in a node.js module. As I understand it all the functions we define except for ones added to the ‘exports’ object are private to the module so there’s no way to test against that code directly unless you pull it out into another module.

    At the moment I’m just changing the code and then restarting the server and checking if it’s working or not. It’s probably not the most effective feedback cycle but it’s working reasonably well so far.

I’ve put the code that I’ve written so far as gists on github:

That can be run with the following command from the terminal:

node twitter-server.js

Written by Mark Needham

March 21st, 2010 at 10:13 pm

Posted in Javascript

Tagged with ,

node.js: First thoughts

with 2 comments

I recently came across node.js via a blog post by Paul Gross and I’ve been playing around with it a bit over the weekend trying to hook up some code to call through to the Twitter API and then return the tweets on my friend timeline.

node.js gives us event driven I/O using JavaScript running server side on top of Google’s V8 JavaScript engine.

Simon Willison has part of a presentation on slideshare where he describes the difference between the typical thread per request approach and the event based approach to dealing with web requests using the metaphor of bunnies. He also has a blog post where he describes this is more detail.

Another resource I found useful is a video from jsconf.eu where the creator of node.js, Ryan Dahl, explains the philosophy behind event driven I/O and gives several examples using node.js.

These are some of my thoughts so far:

  • I’m not used to have so many callbacks spread all around the code and I’m still getting used to the idea that they aren’t executed until the event actually happens!

    I often find myself looking at a piece of code and not understanding how it can possibly work because I’m assuming that the function passed in is executed immediately when in fact it isn’t.

  • If you make a web request the response comes back in chunks so the callback we setup to capture the response will be called multiple times with different parts of the response message.

    For example I have this code to call Twitter and return all my friends’ status updates:

    var sys = require("sys"),
        http = require('http')
     
    exports.getTweets = function(callBack) {
        var twitter = http.createClient(80, "www.twitter.com");
        var request = twitter.request("GET", "/statuses/friends_timeline.json",
                                      {"host": "www.twitter.com",
                                       "Authorization" : "Basic " + "xxx"});
     
        request.addListener('response', function (response) {
            var tweets = "";
     
            response.addListener("data", function (chunk) {
                tweets += chunk;
            });
     
            response.addListener("end", function() {
                callBack.call(this, tweets);
            });
        });
     
        request.close();
    };

    I originally thought that the listener for ‘data’ would only be called once but it gets called 8 times sometimes so that I’ve created the ‘tweets’ variable which allows us to wait until we have the full response before firing the callback when the ‘end’ event is fired.

    I’m not sure whether I’m missing the point a bit by doing this and I think I possibly need to get more used to designing functions which can deal with streams rather than expecting to have all of the data.

  • It seems like node.js would be perfect for a version of my colleagues Julio Maia and Fabio Lessa’s http-impersonator which is a Java application used to record and replay requests/responses made across http-based protocols.

    I haven’t quite worked out the best way to test the above code – ideally I want to stub out the HTTP request so that the test doesn’t have to go across the wire. Micheil Smith pointed me towards fakeweb which allows the faking of HTTP requests/responses in Ruby so I’ll probably have a go at creating something similar.

So far node.js seems really cool and writing code using it is really fun. I’m still not sure exactly where it will fit in some of the architectures that I’ve worked on but the model it encourages feels really natural to work with.

Written by Mark Needham

March 15th, 2010 at 12:09 am

Posted in Javascript

Tagged with

Javascript: Function scoping

with 6 comments

My colleague John Hume wrote an interesting post about his experience with the ‘const’ keyword in ActionScript where he describes the problems with trying to capture a loop variable in a closure and then evaluating it later on in the code.

Since ActionScript and JavaScript are both dialects of ECMAscript, this is a problem in JavaScript as well, and is due to the fact that variables in JavaScript have function scope rather than block scope which is the case in many other languages.

This problem would tend to reveal itself in code where we try to capture a loop variable in an anonymous function and use it later on, like so:

function getValues() {
    var x = new Array();
    for(var i=0; i < 10; i++) {
       x[i] = function() { return i; }
    }
    return x;
};
 
var values = getValues();
for(var j=0; j < values.length; j++) {
    console.log(values[j]());
}

We might expect that to print the sequence of numbers 0-9 on the screen but what we actually get is ’10’ printed 10 times.

There are a couple of things that I initially found strange about this:

  1. Why doesn’t it print out the numbers 0-9?
  2. Given that it doesn’t do that why does it print out ’10’ 10 times instead of ‘9’ 10 times?

The answer to the first question is that ‘i’ gets assigned a new value on each iteration of the loop and we don’t evaluate ‘i’ until we evaluate the anonymous function on line 11.

The value when we do evaluate it would be the last value that it was set to by the loop which in this case that would be ’10’ because that’s the value that ‘i’ has to be in order for the loop to terminate.

This is actually a problem in C# as well – the following code will output ’10’ 10 times as well:

[Test]
public void ClosureOnTheSameValue()
{
    var values = new List<Func<int>>();
    for(int i=0; i < 10; i++)
    {
        values.Add(() => i);
    }
 
    foreach (var value in values)
    {
        Console.WriteLine(value());
    }
}

Again we capture ‘i’ inside a closure and since we only evaluate that value when it’s actually used it will always refer to the last value that ‘i’ was set to which in this case means that it will always output a value of 10.

To fix this in C# we could just create a temporary variable – something which Resharper will actually suggest to us:

[Test]
public void ClosureOnDifferentValue()
{
    var values = new List<Func<int>>();
    for(int i=0; i < 10; i++)
    {
        var idash = i;
        values.Add(() => idash);
    }
 
    foreach (var value in values)
    {
        Console.WriteLine(value());
    }
}

This works in C# because variables have block scope which means that we have a new version of ‘idash’ for each of the functions that we add to the ‘values’ collection.

Sadly the same trick doesn’t work in JavaScript because variables have function scope in Javascript:

function getValues() {
    var x = new Array();
    for(var i=0; i < 10; i++) {
       var idash = i;
       x[i] = function() { return idash; }
    }
    return x;
};
 
var values = getValues();
for(var j=0; j < values.length; j++) {
    console.log(values[j]());
}

The ‘idash’ temporary variable that we created to try and solve the problem gets assigned a new value in each iteration of the loop because that variable is only declared once for the whole function.

The code above could be written like this to make that clearer:

function getValues() {
    var x = new Array();
    var idash;
 
    for(var i=0; i < 10; i++) {
       idash = i;
       x[i] = function() { return idash; }
    }
    return x;
};
 
var values = getValues();
for(var j=0; j < values.length; j++) {
    console.log(values[j]());
}

As John points out:

Here’s something I either never knew or at some point forgot about JavaScript: variables are lexically scoped, but only function bodies introduce new lexical scopes.

In this case we actually end up printing ‘9’ 10 times because that’s the maximum value that gets assigned to ‘idash’.

One solution is to create a temporary variable inside an anonymous function that we execute immediately, like this:

function getValues() {
    var x = new Array();
    for(var i=0; i < 10; i++) {
        (function() {
            var idash = i;
            x[i] = function() { return idash; } })();
    }
    return x;
};
 
var values = getValues();
for(var j=0; j < values.length; j++) {
    console.log(values[j]());
}

Now ‘idash’ is scoped inside the anonymous function and we therefore end up with a new value each time like we want.

Raph pointed out that we could achieve the same thing in a simpler way with the following code:

function getValues() {
    var x = new Array();
    for(var i=0; i < 10; i++) (function(i) {
        x[i] = function() { return i; };
    })(i);
    return x;
};
 
var values = getValues();
for(var j=0; j < values.length; j++) {
    console.log(values[j]());
}

Here we define a for loop with just a single statement so we can lose the ‘{}’ and just call an anonymous function passing in ‘i’.

Of course this example is truly contrived but I wanted to pick something simple enough that I could try and follow exactly how it worked.

I’m not entirely sure of the terminology around closures and scoping so if I’ve described anything incorrectly then please correct me!

Written by Mark Needham

March 10th, 2010 at 11:06 pm

Posted in Javascript

Tagged with

Javascript: The ‘new’ keyword

with one comment

I came across an interesting post by John Resig where he describes a ‘makeClass’ function that he uses in his code to create functions which can instantiate objects regardless of whether the user calls that function with or without the new keyword.

The main reason that the new keyword seems to be considered harmful is because we might make assumptions in our function that it will be called with the new keyword which changes the meaning of ‘this’ inside that function.

For example in my Bowling Game example I assume that the ‘BowlingGame’ function will be called with the new keyword.

I wanted to see if I could refactor that code to use the module pattern instead so as a first step I changed the instantiation of the ‘bowlingGame’ variable in the tests to not call the function with ‘new’ to see if it would make any noticeable difference:

Screw.Unit(function() {
	describe("bowling game scorecard", function() {
		var bowlingGame;
 
		before(function() {
	    	     bowlingGame = BowlingGame();
	  	});

There is no noticeable difference in the way any of the tests work but in fact all of the functions I defined have been added to the global object (in this case window) instead of onto ‘BowlingGame’.

I changed one of the tests to check that this was the case…

...
			it("should score a single throw", function() {
				console.log(window.roll);
 
				bowlingGame.roll(5);
				(19).times(function() { gutterBall(); });
 
				expect(bowlingGame.score()).to(equal, 5);
			});
...

…which logs ‘undefined’ to Firebug if the new keyword is used to instantiate ‘bowlingGame’ and ‘function()’ if it wasn’t.

The danger here is that you could change the meaning of the ‘roll’ function outside of the ‘BowlingGame’ if you wanted to.

To give a contrived example perhaps we could change ‘roll’ so that it actually called the original function twice instead of once:

...
			it("should score a single throw", function() {
				var originalRoll = window.roll;
				window.roll = function() {
					originalRoll.apply(this, arguments);
					originalRoll.apply(this, arguments);
					console.log("roll isn't what you'd expect anymore")				
				};
 
				bowlingGame.roll(5);
				(19).times(function() { gutterBall(); });
 
				expect(bowlingGame.score()).to(equal, 5);
			});	
...

In this case you would probably never do that because it’s just a small bit of code but you wouldn’t want to add random functions to the global object in any reasonably sized javascript application.

Shog9 points to a bit of code which allows us to stop users from calling constructor functions without the new keyword:

BowlingGame  = function() {
	if ( !(this instanceof arguments.callee) ) 
	   throw Error("Constructor called as a function");
...

When ‘BowlingGame’ is called without the new keyword then ‘this’ will refer to ‘window’ which means that it won’t be an instance of ‘arguments.callee’ which in this case is the ‘BowlingGame’ function.

Written by Mark Needham

March 6th, 2010 at 3:16 pm

Posted in Javascript

Tagged with