Mark Needham

Thoughts on Software Development

Archive for the ‘Ruby’ Category

Ruby: invalid multibyte char (US-ASCII)

with one comment

I’ve used Ruby on and off for the last few years but somehow had never come across the following error which we got last week while attempting to print out a currency value:

blah.ruby

amount = "£10.00"
puts amount
$ ruby blah.ruby 
blah.ruby:1: invalid multibyte char (US-ASCII)
blah.ruby:1: invalid multibyte char (US-ASCII)

Luckily my pair Jae had come across this before and showed me a blog post which explains what’s going on and how to sort it out.

It turns out that since Ruby 1.9 we need to explicitly specify which encoding we want to use for strings otherwise it will default to US-ASCII in my case at least.

We can get around this by explicitly telling the interpreter that we want to use UTF-8 encoding by including the following comment at the top of our file:

# encoding: UTF-8
amount = "£10.00"
puts amount

Then when we interpret the file this time:

$ ruby blah.ruby 
£10.00

Since we were only editing one file it made sense to just set the comment manually but if we need to do that in a more widespread way then we’d want to use the magic_encoding gem.

Yehuda Katz also wrote a post a few years ago explaining all things encoding in more detail.

Written by Mark Needham

January 27th, 2013 at 3:14 pm

Posted in Ruby

Tagged with

Bitwise operations in Ruby and Haskell

with 4 comments

Part of one of the most recent problems in the Algorithms 2 course required us to find the ‘neighbours’ of binary values.

In this case a neighbour is described as being any other binary value which has an equivalent value or differs in 1 or 2 bits.

e.g. the neighbours of ‘10000’ would be ‘00000’, ‘00001’, ‘00010’, ‘00100’, ”01000′, ‘10001’, ‘10010’, ‘10011’, ‘10100’, ‘10101’, ‘10110’, ‘11000’, ‘11001’, ‘11010’ and ‘11100’

I initially treated ‘10000’ as an array of 1s and 0s and wrote a function to recursively come up with the above combinations before it was pointed out to me that it’d be much easier to use bit wise logic instead.

I don’t remember every having to write any code using bit wise operations since university so I thought I’d document what I learnt.

The first thing to do is convert those binary values into a decimal equivalent which is pretty easy in Ruby:

> "10000".to_i(2)
=> 16

In Haskell I stole a function from PLEAC to do the job:

> import Data.Char
> (foldr (\c s -> s * 2 + c) 0 . reverse . map digitToInt) "10000"
16

I initially worked out the neighbours by hand but I need to work out how to convert that into code so I ran through an example.

We want to get from ‘10000’ (16) to ‘00000’ (0) which we can do by using an XOR operation:

Binary XOR Operator copies the bit if it is set in one operand but not both.

'10000' XOR '10000'

In Ruby it would read like this:

> 16 ^ 16
=> 0

If we do the same XOR operation but changing the other bits instead we end up with all the neighbours of distance 1:

> [0,1,2,4,16].map { |x| 16 ^ x }
=> [16, 17, 18, 20, 0]

We can generalise the function that creates that array of values like so:

> bits = 5
> offsets = (0..(bits - 1)).map { |x| 2 ** x }
=> [1, 2, 4, 8, 16]

or if we want to use bit shifting:

> offsets = (0..(bits - 1)).map { |x| 1 << x }
=> [1, 2, 4, 8, 16]

With that approach we’re moving the “left operands value is moved left by the number of bits specified by the right operand.” i.e. we move ‘1’ left by 0 bits (from ‘1’to ‘1’), then by 1 bit (from ‘1’ to ’10’), then by 2 bits (from ‘1’ to ‘100’) and so on.

We still need to get all the neighbours which differ by 2 bits which we can do by getting all the ways that we can combine those offsets together. The combination function and Bitwise or do the trick here:

> offsets.combination(2).to_a
=> [[1, 2], [1, 4], [1, 8], [1, 16], [2, 4], [2, 8], [2, 16], [4, 8], [4, 16], [8, 16]]
> offsets.combination(2).to_a.map { |a,b| a|b }
=> [3, 5, 9, 17, 6, 10, 18, 12, 20, 24]

Now if we put it all together:

> initial_value = "10000"
=> "10000"
> bits = initial_value.length
=> 5
> value = "10000".to_i(2)
=> 16
> single_bit_offsets = (0..(bits - 1)).map { |x| 1 << x }
=> [1, 2, 4, 8, 16]
> all_the_offsets = offsets + offsets.combination(2).to_a.map { |a,b| a|b }
=> [1, 2, 4, 8, 16, 3, 5, 9, 17, 6, 10, 18, 12, 20, 24]
> all_the_offsets.map { |offset| value ^ offset }.sort 
=> [0, 1, 2, 4, 8, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28]

The final Haskell version looks like this:

> import Data.Char
> import Data.Bits
> import Data.List
 
> let initialValue = "10000"
> let bits = length initialValue
> let value = (foldr (\c s -> s * 2 + c) 0 . reverse . map digitToInt) initialValue
> let singleBitOffsets = map (shiftL 1) [0..(bits - 1)] :: [Int]
> let allTheOffsets = singleBitOffsets ++ map (\(x:y:_) -> (x .|. y)) (combinationsOf 2 singleBitOffsets) :: [Int]
> Data.List.sort $ map (xor value) allTheOffsets 
[0,1,2,4,8,17,18,19,20,21,22,24,25,26,28]

I took the combinationsOf function from David Amos’ Combinatorics Generation library and it reads like so:

-- subsets of size k
combinationsOf 0 _ = [[]]
combinationsOf _ [] = []
combinationsOf k (x:xs) = map (x:) (combinationsOf (k-1) xs) ++ combinationsOf k xs

A real life version of this problem occurs in the world of genome analysis where scientists need to work out whether phylogenetic profiles are functionally linked.

Written by Mark Needham

December 31st, 2012 at 1:14 pm

Posted in Haskell,Ruby

Tagged with , ,

Ruby: Finding where gems are

with 2 comments

In my infrequent travels into Ruby land I always seem to forget where the gems that I’ve installed actually live on the file system but my colleague Nick recently showed me a neat way of figuring it out.

If I’m in the folder that contains all my ThoughtWorks graph code I’d just need to run the following command:

$ gem which rubygems
/Users/mneedham/.rbenv/versions/jruby-1.6.7/lib/ruby/site_ruby/1.8/rubygems.rb

I then loaded up irb and wrote a simple cypher query executed using neography:

> require 'rubygems'
=> true
 
> require 'neography'
=> true
 
> neo = Neography::Rest.new(:port => 7476)
=> #<Neography::Rest:0x40d3ab8b @protocol="http://", @server="localhost", @cypher_path="/cypher", @log_file="neography.log", @authentication={}, @directory="", @log_enabled=false, @gremlin_path="/ext/GremlinPlugin/graphdb/execute_script", @parser={:parser=>CrackParser}, @max_threads=20, @port=7476>
 
> neo.execute_query("START n = node(1) RETURN n")
=> {"data"=>[[{"outgoing_relationships"=>"http://localhost:7476/db/data/node/1/relationships/out", "data"=>{"thoughtquitter"=>true, "name"=>"Marjorie Pries", "type"=>"person"}, "traverse"=>"http://localhost:7476/db/data/node/1/traverse/{returnType}", "all_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/all/{-list|&|types}", "property"=>"http://localhost:7476/db/data/node/1/properties/{key}", "self"=>"http://localhost:7476/db/data/node/1", "properties"=>"http://localhost:7476/db/data/node/1/properties", "outgoing_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/out/{-list|&|types}", "incoming_relationships"=>"http://localhost:7476/db/data/node/1/relationships/in", "extensions"=>{}, "create_relationship"=>"http://localhost:7476/db/data/node/1/relationships", "paged_traverse"=>"http://localhost:7476/db/data/node/1/paged/traverse/{returnType}{?pageSize,leaseTime}", "all_relationships"=>"http://localhost:7476/db/data/node/1/relationships/all", "incoming_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/in/{-list|&|types}"}]], "columns"=>["n"]}

If I want to debug the execute_query function of neography then I’d need to make a change to /Users/mneedham/.rbenv/versions/jruby-1.6.7/lib/ruby/gems/1.8/gems/neography-0.0.26/lib/neography/rest.rb which is just a case of going up a few levels from where rubygems.rb lives and then finding the appropriate gem.

If I change that function to print a stupid message…

       def execute_query(query, params = {})
          puts "just testing you come up"
           options = { :body => {:query => query, :params => params}.to_json, :headers => {'Content-Type' => 'application/json'} }
           result = post(@cypher_path, options)
       end

…when we run it through irb again we should see it:

> neo.execute_query("START n = node(1) RETURN n")
just testing you come up
=> {"data"=>[[{"outgoing_relationships"=>"http://localhost:7476/db/data/node/1/relationships/out", "data"=>{"thoughtquitter"=>true, "name"=>"Marjorie Pries", "type"=>"person"}, "traverse"=>"http://localhost:7476/db/data/node/1/traverse/{returnType}", "all_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/all/{-list|&|types}", "property"=>"http://localhost:7476/db/data/node/1/properties/{key}", "self"=>"http://localhost:7476/db/data/node/1", "properties"=>"http://localhost:7476/db/data/node/1/properties", "outgoing_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/out/{-list|&|types}", "incoming_relationships"=>"http://localhost:7476/db/data/node/1/relationships/in", "extensions"=>{}, "create_relationship"=>"http://localhost:7476/db/data/node/1/relationships", "paged_traverse"=>"http://localhost:7476/db/data/node/1/paged/traverse/{returnType}{?pageSize,leaseTime}", "all_relationships"=>"http://localhost:7476/db/data/node/1/relationships/all", "incoming_typed_relationships"=>"http://localhost:7476/db/data/node/1/relationships/in/{-list|&|types}"}]], "columns"=>["n"]}

And now hopefully I won’t forget where to find the gems!

Written by Mark Needham

August 25th, 2012 at 10:00 am

Posted in Ruby

Tagged with

Ruby: neo4j gem – LoadError: no such file to load — active_support/core_ext/class/inheritable_attributes

with one comment

I’ve been playing around with neo4j again over the past couple of days using the neo4j.rb gem to build up a graph.

I installed the gem but then ended up with the following error when I tried to ‘require neo4j’ in ‘irb’:

LoadError: no such file to load -- active_support/core_ext/class/inheritable_attributes
  require at org/jruby/RubyKernel.java:1033
  require at /Users/mneedham/.rbenv/versions/jruby-1.6.7/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
   (root) at /Users/mneedham/.rbenv/versions/jruby-1.6.7/lib/ruby/gems/1.8/gems/neo4j-1.3.1-java/lib/neo4j.rb:9
  require at org/jruby/RubyKernel.java:1033
  require at /Users/mneedham/.rbenv/versions/jruby-1.6.7/lib/ruby/gems/1.8/gems/neo4j-1.3.1-java/lib/neo4j.rb:59
   (root) at src/main/ruby/neo_test.rb:2

It seems a few others have come across this problem as well and the problem seems to be that ActiveSupport 3.2 isn’t yet supported by the gem so we need to use an earlier version of that.

There were a few suggestions in the comments but it was the following that worked for me:

sudo gem uninstall railties 
sudo gem install railties --version 3.1

Written by Mark Needham

April 14th, 2012 at 10:21 am

Posted in neo4j,Ruby

Tagged with

Ruby: Refactoring from hash to object

without comments

Something I’ve noticed when I play around with Ruby in my own time is that I nearly always end up with the situation where I’m passing hashes all over my code and to start with it’s not a big deal.

Unfortunately I eventually get to the stage where I’m effectively modelling an object inside a hash and it all gets very difficult to understand.

I’ve written a few times before about incrementally refactoring code so this seemed like a pretty good chance for me to try that out.

The code in the view looked something like this:

<% @tweets.each do |tweet| %>
  <%= tweet[:key] %>  <%= tweet[:value][:something_else] %>
<% end %>

@tweets was being populated directly from a call to CouchDB so to start with I needed to change it from being a collection of hashes to a collection of objects:

I changed the Sinatra calling code from:

get '/' do
  @tweets = get_the_couchdb_tweets_hash
end

to:

get '/' do
  tweets_hash = get_the_couchdb_tweets_hash
  @tweets = tweets_hash.map { |tweet| TweetViewModel.new(tweet) }
end

where TweetViewModel is defined like so:

class TweetViewModel
  attr_accessor :key, :value
 
  def initialize(tweet_hash)
    @key = tweet_hash[:key]
    @value = tweet_hash[:value]
  end
 
  def get(lookup)
    if lookup == :key
      key
    else
      value
    end
  end
 
  alias_method :[], :get
end

The next step was to get rid of the get method and rename those attr_accessor methods to something more intention revealing.

class TweetViewModel
  attr_accessor :url, :messages
 
  def initialize(tweet_hash)
    @url = tweet_hash[:key]
    @messages = tweet_hash[:value]
  end
end
<% @tweets.each do |tweet| %>
  <%= tweet.url %>  <%= tweet.messages[:something_else] %>
<% end %>

I originally didn’t realise how easy it would be to make the TweetViewModel pretend to temporarily be a Hash but it actually made it really easy for me to change the code and know that it was working the whole way.

For someone with more Ruby experience perhaps it wouldn’t be necessary to break out the refactoring like this because they could fairly confidently do it in one go.

Written by Mark Needham

February 27th, 2011 at 8:10 pm

Posted in Incremental Refactoring,Ruby

Tagged with

Ruby: Where to define the method?

with 5 comments

In our application we deal with items which can be put into a shopping cart.

An item is defined like so:

class Item < ActiveRecord::Base
 
end

One problem that we had to solve recently was working out how to display a message to the user if the item they wanted to buy was out of stock.

We can find out if items are out of stock by making a call to an external service:

1
2
3
4
5
6
7
def out_of_stock_items
  items_from_external_service_call.map do |i|
    item = look_up_item_by(i[:id])
    item.number_available = i[:number_available]
    ...
  end
end

Line 5 is the interesting one because we needed to work out where to define the ‘number_available’ method.

The easiest way to do it is this:

class Item < ActiveRecord::Base
  attr_accessor :number_available
end

But it seems a bit misleading because the concept of availability doesn’t exist in every context that we use the ‘Item’ object.

Another approach could be to add an instance method to each of the cart items in that context which is a bit more complicated:

def out_of_stock_items
  items_from_external_service_call.map do |i|
    item = look_up_item_by(i[:id])
 
    def item.number_available=(value)
      @number_available = value
    end
 
    def item.number_available
      @number_available
    end
 
    item.number_available = i[:number_available]
    ...
  end
end

Right now we’ve gone for the easier option and put the method onto the ‘Item’ class but I’d be curious how others would solve this problem…

Written by Mark Needham

February 3rd, 2011 at 7:37 pm

Posted in Ruby

Tagged with

Rails: Using helpers inside a controller

with one comment

For about an hour or so this afternoon we were following the somewhat evil practice of using a method defined in a helper inside a controller.

The method was defined in the ApplicationHelper module:

module ApplicationHelper
	def foo
		# do something
	end
end

So we initially assumed that we’d just be able to reference that method inside any of our controllers since they all derive from ApplicationController.

That wasn’t the case so our next attempt was to try and add it as a helper:

class FooController < ApplicationController
	helper :application
end

Which makes it accessible from the view but not from the controller…

Eventually we called Ashwin to help us out and he came across this thread on dzone.

About half way down the page ovhaag points out that we can use ‘@template’ to get access to helper methods:

In any controller, there is a “@template”-instance and you can call helper methods on this.

I found this trick in http://media.railscasts.com/videos/132_helpers_outside_views.mov

Ryan is not sure if this use is intended but it is very short and today it works.

We can use that instance variable like so:

class FooController < ApplicationController
	def our_method
		# We can call foo like this
		@template.foo
	end
end

We eventually found out another way to do what we wanted but it seems like a neat little trick

Written by Mark Needham

January 11th, 2011 at 5:09 pm

Posted in Ruby

Tagged with

Ruby: Sorting by boolean fields

with one comment

We were doing a bit of work on RapidFTR in the ThoughtWorks Pune office today and one problem my pair and I were trying to solve was how to sort a collection of objects by a boolean field.

Therefore given the following array of values:

form_sections = [FormSection.new(:enabled => false, :name => "a", :order => 1), 
                 FormSection.new(:enabled => true, :name => "b", :order => 2)]

We wanted to display those form sections which were disabled at the bottom of the page.

We originally tried the following:

form_sections.sort_by { |row| [row.enabled, row.order] }

But got the following error:

undefined method `<=>' for true:TrueClass

We figured we’d need to convert the true and false values to equivalent values which is demonstrated on this Stack Overflow post:

form_sections.sort_by { |row| [row.enabled ? 0 : 1, row.order] }

I didn’t realise it would be that simple to do – it’s pretty neat.

Written by Mark Needham

January 8th, 2011 at 1:15 pm

Posted in Ruby

Tagged with

Ruby: One method, two parameter types

with 2 comments

One interesting thing that I’ve noticed while coding in Ruby is that due to the dynamicness of the language it’s possible to pass values of different types into a given method as parameters.

For example, I’ve recently come across a few examples of methods designed like this:

1
2
3
4
5
6
def calculate_foo_prices(foos)
   ...
   [foos].flatten.each do |foo|
      # do something
   end
end

This allows us to use the method like this:

# foos would come in as an array from the UI
foos = [Foo.new, Foo.new, Foo.new]
calculate_foo_prices(foos)

Or like this:

calculate_foo_prices(Foo.new)

It becomes quite confusing to understand why what is supposedly already a collection is being put inside an array on line 3 of the first example when you first read it.

An alternative would be to pull out a different method for calculating the price of the single Foo:

def calculate_foo_price(foo)
   calculate_foo_prices([foo])
end

And then simplify the original method:

1
2
3
4
5
6
def calculate_foo_prices(foos)
   ...
   foos.each do |foo|
      # do something
   end
end

While writing this I was thinking that another way could be to change the original method to look like this by using the splat operator:

1
2
3
4
5
6
def calculate_foo_prices(*foos)
   ...
   foos.each do |foo|
      # do something
   end
end

Which means that we can use the same method for both situations:

calculate_foo_prices(Foo.new)
# foos would come in as an array from the UI
foos = [Foo.new, Foo.new, Foo.new]
calculate_foo_prices(*foos)

I’m guessing the latter is more idiomatic Ruby or perhaps there’s another way I’m not aware of yet?

Written by Mark Needham

December 7th, 2010 at 5:01 am

Posted in Ruby

Tagged with

Ruby: Exiting a ‘loop’ early

with 6 comments

We recently had a problem to solve which at its core required us to iterate through a collection, look up a value for each key and then exit as soon as we’d found a value.

The original solution looped through the collection and then explicitly returned once a value had been found:

def iterative_version
  v = nil
  [1,2,3,4,5].each do |i|
    v = long_running_method i
    return v unless v.nil?
  end
  v
end
def long_running_method(value)
  puts "inside the long running method with #{value}"
  return nil if value > 3
  value
end

Which we run like so:

p "iterative value is #{iterative_version.to_s}"

This prints the following when we run it:

inside the long running method with 1
"iterative value is 1"

I figured there must be a more functional way to solve the problem and I eventually came up with this:

def functional_version
  [1,2,3,4,5].map {|i| long_running_method i }.find { |i| !i.nil? }
end

Which prints the following when we run it:

inside the long running method with 1
inside the long running method with 2
inside the long running method with 3
inside the long running method with 4
inside the long running method with 5
"functional value is 1"

The problem is that collections in Ruby are eager evaluated so we evaluate every single item in the collection before we get the first non nil value.

Luckily the lazylist gem comes to our rescue and allows us to solve the problem in a functional way:

require 'lazylist'
def lazy_version
  lazy_list([1,2,3,4,5]).find { |i| !i.nil? }
end
 
def lazy_list(values)
  list(long_running_method(values.first)) { lazy_list(values - [values.first]) } 
end

Running that gives us this:

inside the long running method with 1
"lazy value is 1"

I’ve never come across a problem where I needed to use a lazy list but finally I have and I think the version which uses it is pretty neat.

Written by Mark Needham

December 1st, 2010 at 5:56 pm

Posted in Ruby

Tagged with