Mark Needham

I’ve written a couple of posts over the last few months about my experiences with A/B testing and one conversation we often used to have was around user experience vs conversion rate.

Once you start running an A/B test it encourages you to focus more on the conversion rate of users in different parts of the flow and your inclination is to make changes that increase that conversion rate.

Another one of our drivers is to provide the best user experience that we can to our customers and since sometimes this means that the best thing for them is not to switch it seems that these two must be in conflict.

I found it particularly interesting seeing how the conversion rate could be impacted by the way that information was displayed to a user.

This was an idea that I first came across when reading about how the Obama campaign used A/B testing where they noticed big changes in conversion rates by making small tweaks to sentences and imagery.

Our goal from a user experience perspective was to put all the information in front of the user so that they could make an informed choice about what to do.

Initially we made the negative features of the plans very prominent and had them in a large font which led to a drop in conversion.

We assumed that people were now giving more importance to the negative features than was warranted e.g. some plans had a cancellation fee but it typically only accounted for 5% of the saving they’d make by switching to the plan.

When the product is a bit more complicated we could argue that we improve the user experience by helping the user to make an appropriate choice.

On a website the way that we do this is by how we display information by changing the font size, font weight, positioning and a variety of other things.

It’s an interesting balance to find between the two drivers but if we veer towards conversion at all costs then although we’ll get a higher conversion rate in the long term we’ll have some frustrated customers who won’t use our website again.

If we look at it that way then the two drivers don’t seem so opposed to each other.

I’ve been using Sublime a bit recently and one thing I wanted to do was put neo4j cypher queries into files with arbitrary extensions and have them recognised as cypher files every time I open them.

I’m using the cypher Sublime plugin to get the syntax highlighting but since I’ve got my cypher in a .haml file it only remembers that it should have cypher highlighting as long as the file is open.

As soon as I close and then re-open the file it goes back to being highlighted as HAML.

I initially thought that the way around this would be to write a plugin which kept track of files that you’d manually assigned a syntax to but then I came across the ApplySyntax plugin which seems even better.

ApplySyntax allows you to assign syntaxes to files based on regular expression matching on the file name or on the first line of the file.

At the moment, the easiest way to detect that a file is a cypher query is that the first line will begin with ‘START’ so I wrote the following in my user settings file:

~/Library/Application Support/Sublime Text 2/Packages/User/ApplySyntax.sublime-settings

{
	"reraise_exceptions": false,
	"new_file_syntax": false,
	"syntaxes": [
		{			
			"name": "Cypher",
			"rules": [
				{"first_line": "^START"}
			]
		}	
	]
}

ApplySyntax is a pretty neat plugin, worth having a look if you have this problem to solve!

When I first started working at uSwitch Sid installed a couple of ‘productivity applications’ on my Mac which I’ve found pretty useful but from talking to others I realised they aren’t known/being used by everyone.

Alfred

Alfred is a Quick Silver replacement which allows you to quickly open applications, find files, search Google and more. Even though we’re not using half of its features it’s still proved to be useful.

I quite like the calculator feature which we’ve been using for adhoc calculation like working out how much free memory there was on a server or the conversion rate on part of an A/B test.

Moom

The other application is Moom which allows you to move/resize windows.

I didn’t see the point when I first saw it but it’s actually really useful when you’re working on a big monitor and want to put say the terminal alongside the browser.

We have the following shortcuts set up:

That allows us to type ‘Ctrl + Space’ to make the window fill the left hand side of the screen, ‘Alt + Space’ to make it fill the right hand side of the screen and ‘Alt + Ctrl + Space’ to fill the whole screen.

You can also set up shortcuts to allow you to move a window between displays or to rearrange the windows based on certain events.

Highly recommended!

If anyone knows any other cool tools like this I’d love to hear about them.

A few months ago I wrote about my initial experiences with A/B testing and since then we’ve been working on another one and learnt some things around reporting on these types of tests that I thought was interesting.

Reporting as a first class concern

One thing we changed from our previous test after a suggestion by Mike was to start treating the reporting of data related to the test as a first class citizen.

To do this we created an end point which the main application could send POST requests to in order to record page views and various other information about users.

On our previous test we’d derived the various conversion rates from our main transactional data store but it was really slow and painful because the way we structure data in there is optimised for a completely different use case.

Having just the data we want to report on in a separate data store has massively reduced the time spent generating reports.

However, one thing that we learnt about this approach is that you need to spend some time thinking about what data is going to be needed up front.

If you don’t then it will have to be added later on and the reporting on that metric won’t cover the whole test duration.

Drilling down to get insight

In the first test we ran we only really looked at conversion at quite a high level which is good for getting an overview but doesn’t give much insight into what’s going on.

For this test we started off with higher level metrics but a few days in became curious about what was going on between two of the pages and so created a report that segmented users based on an action they’d taken on the first page.

This allowed us to rule out a theory about a change in conversion which we had initially thought was down to a change we’d made but actually proved to be because of a change in an external factor.

The frustrating part of drilling down into the data is that you don’t really know what is it you’re going to want to zoom in on so you have to write code for the specific scenario each time!

Detecting bugs

We generate browser specific metrics on each test that we run and while the conversion rate is generally similar between them there have been some times when there’s a big drop in one browser.

More often than not when we’ve drilled into this we’ve found that there was actually a Javascript bug that we hadn’t detected and we can then go back and sort that out.

An alternative approach would be to have an automated Javascript/Web Driver test suite which ran against each browser. We’ve effectively traded off the maintenance cost of that for what is usually a small period of inconvenience for some users.

As I mentioned a couple of weeks ago I’ve been working on a tutorial about thinking through problems in graphs and since it’s a Sinatra application I thought thin would be a decent choice for web server.

In my initial setup I had the following nginx config file which was used to proxy requests on to thin:

/etc/nginx/sites-available/thinkingingraphs.conf

upstream thin {
  server 127.0.0.1:3000;
}
 
server {
  listen       80 default;
  server_name _;
  charset utf-8;
 
  rewrite  ^\/status(.*)$  $1 last;
 
  gzip  on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_types       text/plain application/xml text/xml text/css application/x-javascript application/xml+rss text/javascript application/json;
 
  gzip_vary on;
 
  access_log  /var/www/thinkingingraphs/shared/log/nginx_access.log;
  error_log  /var/www/thinkingingraphs/shared/log/nginx_error.log;
 
  root   /var/www/thinkingingraphs/current/public;
 
  location / {
    proxy_pass http://thin;
  }
 
  error_page  404              /404.html;
  error_page   500 502 503 504  /500.html;
}

I had an upstart script which started the thin server…

/etc/init/thinkingingraphs.conf

script
  export RACK_ENV=production
  export RUBY=ruby
 
  cd /var/www/thinkingingraphs/current
  exec su -s /bin/sh vagrant -c '$RUBY -S bundle exec thin -p 3000 start >> /var/www/thinkingingraphs/current/log/production.log 2>&1'
end script

… and then I used the following capistrano script to stop and start the server whenever I was deploying a new version of the application:

config/deploy.rb

namespace :deploy do
  task(:start) {}
  task(:stop) {}
 
  desc "Restart Application"
  task :restart do
    sudo "stop thinkingingraphs || echo 0"
    sudo "start thinkingingraphs"
  end
end

The problem with this approach is that some requests receive a 502 response code while its restarting:

$ bundle exec cap deploy

$ while true; do curl -w %{http_code}:%{time_total} http://localhost/ -o /dev/null -s; printf "\n"; sleep 0.5; done
 
200:0.076
200:0.074
200:0.095
502:0.003
200:0.696

I wanted to try and make a no downtime deploy script and I came across a couple of posts which helped me work out how to do it.

The first step was to make sure that I had more than one thin instance running so that requests could be sent to one of the other ones while a restart was in progress.

I created the following config file:

/etc/thin/thinkingingraphs.yml

chdir: /var/www/thinkingingraphs/current
environment: production
address: 0.0.0.0
port: 3000
timeout: 30
log: log/thin.log
pid: tmp/pids/thin.pid
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
servers: 3
daemonize: true
onebyone: true

One of the other properties that we need to set is ‘onebyone’ which means that when you restart thin it will take down the thin instances one at a time. This means one of the other two can handle incoming requests.

We’ve set the number of servers to 3 which will spin up 3 instances on ports 3000, 3001 and 3002.

I changed my upstart script to look like this:

/etc/init/thinkingingraphs.conf

script
  export RACK_ENV=production
  export RUBY=ruby
 
  cd /var/www/thinkingingraphs/current
  exec su -s /bin/sh vagrant -c '$RUBY -S bundle exec thin -C /etc/thin/thinkingingraphs.yml start >> /var/www/thinkingingraphs/current/log/production.log 2>&1'
end script

I also had to change the capistrano script to call ‘thin restart’ instead of stopping and starting the upstart script:

config/deploy.rb

namespace :deploy do
  task(:start) {}
  task(:stop) {}
 
  desc "Restart Application"
  task :restart do
    run "cd #{current_path} && bundle exec thin restart -C /etc/thin/thinkingingraphs.yml"
  end
end

Finally I had to make some changes to the nginx config file to send on requests to other thin instances if the first attempt failed (due to it being restarted) using the proxy_next_upstream method:

/etc/nginx/sites-available/thinkingingraphs.conf

upstream thin {
  server 127.0.0.1:3000 max_fails=1 fail_timeout=15s;
  server 127.0.0.1:3001 max_fails=1 fail_timeout=15s;
  server 127.0.0.1:3002 max_fails=1 fail_timeout=15s;
}
 
server {
  listen       80 default;
  server_name _;
  charset utf-8;
 
  rewrite  ^\/status(.*)$  $1 last;
 
  gzip  on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_types       text/plain application/xml text/xml text/css application/x-javascript application/xml+rss text/javascript application/json;
 
  gzip_vary on;
 
  access_log  /var/www/thinkingingraphs/shared/log/nginx_access.log;
  error_log  /var/www/thinkingingraphs/shared/log/nginx_error.log;
 
  root   /var/www/thinkingingraphs/current/public;
 
  location / {
    proxy_pass http://thin;
    proxy_next_upstream error timeout http_502 http_503;
  }
 
  error_page  404              /404.html;
  error_page   500 502 503 504  /500.html;
}

We’ve also made a change to our upstream definition to proxy requests to one of the thin instances which will be running.

When I deploy the application now there is no downtime:

$ bundle exec cap deploy

$ while true; do curl -w %{http_code}:%{time_total} http://localhost/ -o /dev/null -s; printf "\n"; sleep 0.5; done
 
200:0.094
200:0.095
200:0.082
200:0.102
200:0.080
200:0.081

The only problem is that upstart now seems to have lost a handle on the thin processes and from what I can tell there isn’t a master process which upstart could get a handle on so I’m not sure how to wire this up.

Any ideas welcome!

After coming across this post about why you should use Sublime Text I decided to try using it a bit more and one of the things that I missed from Textmate was the way you can select the current file on the sidebar.

In Textmate the shortcut to do that is ‘Cmd + Ctrl + R’ so I wanted to be able to do something similar or configure Sublime so it responded to the same shortcut.

The option to reveal a file in the side bar is accessible from the context menu by right clicking on the contents of a file after it’s opening and selecting ‘Reveal in Side Bar’ which is a good start.

To map that to a key binding we need to go ‘Preferences > Key Bindings (User)’ and put the following into that file:

[
	{ "keys": ["ctrl+super+r"], "command": "reveal_in_side_bar" }
]

Of course if we already have other custom key bindings then we can just add it after those instead.

We can work out what the name of commands are by turning on command logging in the Sublime console.

We need to first open the console with ‘Ctrl + `” and then paste the following:

sublime.log_commands(True)

Any commands that we run will now have their name printed in the console window. e.g.

>>> sublime.log_commands(True)
command: context_menu {"event": {"button": 1, "x": 390.21484375, "y": 329.66796875}}
command: reveal_in_side_bar
command: rename_path {"paths": ["/Users/markhneedham/code/thinkingingraphs/public/js/bootstrap.js"]}
no command for selector: noop:
command: show_panel {"panel": "console", "toggle": true}

We can then setup appropriate key bindings for whichever commands we like.

I part time administrate a football forum that I used to run when I was at university and one problem we had recently was that some of the tables/indices had got corrupted when MySQL crashed due to a lack of disc space.

We weren’t seeing any visible sign of a problem in any of the logs but whenever you tried to query one of the topics it wasn’t returning any posts.

I eventually came across a useful article which explained how to check whether some of the tables in a MySQL database had been corrupted and how to fix them.

I first shutdown the database using the following command:

mysqladmin shutdown

And then I ran this command to check on the status of each of the tables:

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk $path; done

This gave an output like the following for each table:

Checking MyISAM file: /var/lib/mysql/forum/forum.MYI
Data records:     217   Deleted blocks:       4
myisamchk: warning: 1 client is using or hasn't closed the table properly
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check record links
MyISAM-table '/var/lib/mysql/forum/forum.MYI' is usable but should be fixed

If you pass the ‘–recover’ flag to myisamchk it will attempt to fix any problems it finds. I therefore ran the following command:

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk --recover $path; done

After I’d run that it seemed to fix most of the problems we’d been experiencing. There are still a couple of edge cases left but at least the majority of the forum is now in a usable state.

I think we could just as easily run myisamchk by passing a wildcard selection of files for it to run against but I didn’t realise that until afterwards!

The following would therefore work just as well:

myisamchk --recover /var/lib/mysql/forum/*.MYI

Despite the fact that I’ve been working full time in software for almost 8 years now every now and then I still need a reminder of how useful reading logs can be in helping solve problems.

I had a couple of such instances recently which I thought I’d document.

The first was a couple of weeks ago when Tim and I were pairing on moving some applications from Passenger to Unicorn and were testing whether or not we’d done so successfully.

We were doing this by creating an /etc/hosts entry from our top level domain to an nginx proxy node which was to forward on the request to the application server.

Request -> nginx on proxy node -> nginx on app server node -> unicorn on app server node

This didn’t work and we got a 404 response code so I logged onto the server hosting the application server and started writing our a cURL command to simulate what the proxy should be doing to see if the problem was there.

After watching me do this a couple of times Tim suggested that we might be more successful if we opened a bunch of tabs on the shell tailing the various log files that the request should pass through.

We set up tail commands against the following files:

• nginx access log on proxy node
• nginx error log on proxy node
• nginx access log on the app server node
• unicorn log on the app server node

Immediately it became clear that we actually had a problem on the proxy node because we’d configured one of the nginx directives incorrectly.

Once we’d fixed this the request flowed through smoothly.

We extended this tailing of files idea when testing multiple nodes through a load balancer except this time we made use of Fabric to make things a bit easier.

The second was earlier this week when Baris and I were trying to configure puppet so that we could install different Ruby versions on different machines.

We were having trouble figuring out why the wrong version was getting installed so eventually we chatted to Andy who amongst other things had a look at the apt history log @ /var/log/apt/history.log and was able to figure out how this was happening.

Lesson of the last two weeks: embrace the logs!

A couple of weeks ago Tim and I were spinning up a new service on a machine which wasn’t quite working so we were manually making changes to the /etc/nginx/nginx.conf file and restarting nginx to try and sort it out.

This process is generally not that interesting – you open the file in vi, make some changes, close it, then restart nginx and see if it works. If not then you open the file again and repeat.

Except Tim had a slight variation on this workflow which is an improvement that I don’t want to forget!

Once we’d finished making the changes to the file in vi Tim hit ‘Ctrl + Z‘ which suspended the vi process and put us back at the shell prompt.

We could then restart nginx or do whatever else we needed to do and then type ‘fg‘ to go back into vi again.

Not only is this workflow quicker, it also keeps the history of the changes that we’ve made to the file so if one of our changes really screws things up we can easily undo it. Previously we’d have to remember what changes we’d made and do that manually.

In summary this workflow is a pretty simple idea but nevertheless one I had never thought about or seen anyone else do and I’ll be using it in future.

As I mentioned in my previous post I wanted to add televised games to my football graph and the Premier League website seemed like the best case to find out which games those were.

I initially tried to use Nokogiri to grab the data that I wanted…

> require 'nokogiri'
> require 'open-air'
> tv_times = Nokogiri::HTML(open('http://www.premierleague.com/en-gb/matchday/broadcast-schedules.tv.html?rangeType=.dateSeason&country=GB&clubId=ALL&season=2012-2013&isLive=true'))

…but when I tried to query by CSS selector for all the matches nothing came back:

> tv_times.css(".broadcastschedule table.contentTable tbody tr")
=> []

I was a bit surprised but read somewhere that I should check if there were any errors while parsing the document. In fact there were quite a few!

> tv_times.errors
=> [#<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, #<Nokogiri::XML::SyntaxError: Element script embeds close tag>, ...]

I ran the document through the W3C markup validation service and it didn’t seem to find any problem with it.

Next I tried stripping out all the script tags using loofah before manually removing them but neither of those approaches helped.

I’ve previously used Web Driver to scrape web pages but I’d found that Nokogiri was much faster so I stopped using it.

Since my new library wasn’t playing ball I thought I’d quickly see if Web Driver was up to the challenge and indeed it was:

require "selenium-webdriver"
 
driver = Selenium::WebDriver.for :chrome
driver.navigate.to "http://www.premierleague.com/en-gb/matchday/broadcast-schedules.tv.html?rangeType=.dateSeason&country=GB&clubId=ALL&season=2012-2013&isLive=true"
 
matches = driver.find_elements(:css, '.broadcastschedule table.contentTable tbody tr')
matches.each do|tr| 	
  match = tr.find_element(:css, "td.show a").text
  broadcaster = tr.find_element(:css, "td.broadcaster img").attribute("src")
  tv_channel = broadcaster.include?("sky-sports") ? "Sky" : "ESPN"
 
  puts "#{match},#{tv_channel}"
end
 
driver.quit

$ ruby tv_games.rb 
Newcastle United vs Tottenham Hotspur,ESPN
Wigan Athletic vs Chelsea,Sky
Manchester City vs Southampton,Sky
Everton vs Manchester United,Sky
Swansea City vs West Ham United,Sky
Chelsea vs Newcastle United,ESPN
...

Ideally I’d like to use Nokogiri to do this job but it’s decided that the document is invalid and it can’t parse it properly so Web Driver is a pretty decent replacement I reckon!