Mark Needham

Thoughts on Software Development

Archive for the ‘capistrano’ tag

No downtime deploy with capistrano, Thin and nginx

with 4 comments

As I mentioned a couple of weeks ago I’ve been working on a tutorial about thinking through problems in graphs and since it’s a Sinatra application I thought thin would be a decent choice for web server.

In my initial setup I had the following nginx config file which was used to proxy requests on to thin:

/etc/nginx/sites-available/thinkingingraphs.conf

upstream thin {
  server 127.0.0.1:3000;
}
 
server {
  listen       80 default;
  server_name _;
  charset utf-8;
 
  rewrite  ^\/status(.*)$  $1 last;
 
  gzip  on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_types       text/plain application/xml text/xml text/css application/x-javascript application/xml+rss text/javascript application/json;
 
  gzip_vary on;
 
  access_log  /var/www/thinkingingraphs/shared/log/nginx_access.log;
  error_log  /var/www/thinkingingraphs/shared/log/nginx_error.log;
 
  root   /var/www/thinkingingraphs/current/public;
 
  location / {
    proxy_pass http://thin;
  }
 
  error_page  404              /404.html;
  error_page   500 502 503 504  /500.html;
}

I had an upstart script which started the thin server…

/etc/init/thinkingingraphs.conf

script
  export RACK_ENV=production
  export RUBY=ruby
 
  cd /var/www/thinkingingraphs/current
  exec su -s /bin/sh vagrant -c '$RUBY -S bundle exec thin -p 3000 start >> /var/www/thinkingingraphs/current/log/production.log 2>&1'
end script

… and then I used the following capistrano script to stop and start the server whenever I was deploying a new version of the application:

config/deploy.rb

namespace :deploy do
  task(:start) {}
  task(:stop) {}
 
  desc "Restart Application"
  task :restart do
    sudo "stop thinkingingraphs || echo 0"
    sudo "start thinkingingraphs"
  end
end

The problem with this approach is that some requests receive a 502 response code while its restarting:

$ bundle exec cap deploy
$ while true; do curl -w %{http_code}:%{time_total} http://localhost/ -o /dev/null -s; printf "\n"; sleep 0.5; done
 
200:0.076
200:0.074
200:0.095
502:0.003
200:0.696

I wanted to try and make a no downtime deploy script and I came across a couple of posts which helped me work out how to do it.

The first step was to make sure that I had more than one thin instance running so that requests could be sent to one of the other ones while a restart was in progress.

I created the following config file:

/etc/thin/thinkingingraphs.yml

chdir: /var/www/thinkingingraphs/current
environment: production
address: 0.0.0.0
port: 3000
timeout: 30
log: log/thin.log
pid: tmp/pids/thin.pid
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
servers: 3
daemonize: true
onebyone: true

One of the other properties that we need to set is ‘onebyone’ which means that when you restart thin it will take down the thin instances one at a time. This means one of the other two can handle incoming requests.

We’ve set the number of servers to 3 which will spin up 3 instances on ports 3000, 3001 and 3002.

I changed my upstart script to look like this:

/etc/init/thinkingingraphs.conf

script
  export RACK_ENV=production
  export RUBY=ruby
 
  cd /var/www/thinkingingraphs/current
  exec su -s /bin/sh vagrant -c '$RUBY -S bundle exec thin -C /etc/thin/thinkingingraphs.yml start >> /var/www/thinkingingraphs/current/log/production.log 2>&1'
end script

I also had to change the capistrano script to call ‘thin restart’ instead of stopping and starting the upstart script:

config/deploy.rb

namespace :deploy do
  task(:start) {}
  task(:stop) {}
 
  desc "Restart Application"
  task :restart do
    run "cd #{current_path} && bundle exec thin restart -C /etc/thin/thinkingingraphs.yml"
  end
end

Finally I had to make some changes to the nginx config file to send on requests to other thin instances if the first attempt failed (due to it being restarted) using the proxy_next_upstream method:

/etc/nginx/sites-available/thinkingingraphs.conf

upstream thin {
  server 127.0.0.1:3000 max_fails=1 fail_timeout=15s;
  server 127.0.0.1:3001 max_fails=1 fail_timeout=15s;
  server 127.0.0.1:3002 max_fails=1 fail_timeout=15s;
}
 
server {
  listen       80 default;
  server_name _;
  charset utf-8;
 
  rewrite  ^\/status(.*)$  $1 last;
 
  gzip  on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_types       text/plain application/xml text/xml text/css application/x-javascript application/xml+rss text/javascript application/json;
 
  gzip_vary on;
 
  access_log  /var/www/thinkingingraphs/shared/log/nginx_access.log;
  error_log  /var/www/thinkingingraphs/shared/log/nginx_error.log;
 
  root   /var/www/thinkingingraphs/current/public;
 
  location / {
    proxy_pass http://thin;
    proxy_next_upstream error timeout http_502 http_503;
  }
 
  error_page  404              /404.html;
  error_page   500 502 503 504  /500.html;
}

We’ve also made a change to our upstream definition to proxy requests to one of the thin instances which will be running.

When I deploy the application now there is no downtime:

$ bundle exec cap deploy
$ while true; do curl -w %{http_code}:%{time_total} http://localhost/ -o /dev/null -s; printf "\n"; sleep 0.5; done
 
200:0.094
200:0.095
200:0.082
200:0.102
200:0.080
200:0.081

The only problem is that upstart now seems to have lost a handle on the thin processes and from what I can tell there isn’t a master process which upstart could get a handle on so I’m not sure how to wire this up.

Any ideas welcome!

Written by Mark Needham

April 23rd, 2013 at 11:25 pm

Posted in Software Development

Tagged with , ,

Capistrano: Host key verification failed. ** [err] fatal: The remote end hung up unexpectedly

with 3 comments

As I mentioned in my previous post I’ve been deploying a web application to a vagrant VM using Capistrano and my initial configuration was like so:

require 'capistrano/ext/multistage'
 
set :application, "thinkingingraphs"
set :scm, :git
set :repository,  "git@bitbucket.org:markhneedham/thinkingingraphs.git"
set :scm_passphrase, ""
 
set :ssh_options, {:forward_agent => true, :paranoid => false, keys: ['~/.vagrant.d/insecure_private_key']}
set :stages, ["vagrant"]
set :default_stage, "vagrant"
 
set :user, "vagrant"
server "192.168.33.101", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/thinkingingraphs"

When I ran ‘cap deploy’ I ended up with the following error:

  * executing "git clone -q git@bitbucket.org:markhneedham/thinkingingraphs.git /var/www/thinkingingraphs/releases/20130414171523 && cd /var/www/thinkingingraphs/releases/20130414171523 && git checkout -q -b deploy 6dcbf945ef5b8a5d5d39784800f4a6b7731c7d8a && (echo 6dcbf945ef5b8a5d5d39784800f4a6b7731c7d8a > /var/www/thinkingingraphs/releases/20130414171523/REVISION)"
    servers: ["192.168.33.101"]
    [192.168.33.101] executing command
 ** [192.168.33.101 :: err] Host key verification failed.
 ** [192.168.33.101 :: err] fatal: The remote end hung up unexpectedly

As far as I can tell the reason for this is that bitbucket hasn’t been verified as a host by the VM and therefore the equivalent of the following happens when it tries to clone the repository:

$ ssh git@bitbucket.org
The authenticity of host 'bitbucket.org (207.223.240.182)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?

Since we aren’t answering ‘yes’ to that question and bitbucket isn’t in our ~/.ssh/known_hosts file it’s not able to continue.

One solution to this problem is to run the ssh command above and then answer ‘yes’ to the question which will add bitbucket to our known_hosts file and we can then run ‘cap deploy’ again.

It’s a bit annoying to have that manual step though so another way is to set cap to use pty by putting the following line in our config file:

set :default_run_options, {:pty => true}

Now when we run ‘cap deploy’ we can see that bitbucket automatically gets added to the known_hosts file:

    servers: ["192.168.33.101"]
    [192.168.33.101] executing command
 ** [192.168.33.101 :: out] The authenticity of host 'bitbucket.org (207.223.240.181)' can't be established.
 ** RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
 ** Are you sure you want to continue connecting (yes/no)?
 ** [192.168.33.101 :: out] yes
 ** [192.168.33.101 :: out] Warning: Permanently added 'bitbucket.org,207.223.240.181' (RSA) to the list of known hosts.

As far as I can tell this runs the command using a pseudo terminal and then automatically adds bitbucket into the known_hosts file but I’m not entirely sure how that works. My google skillz have also failed me so if anyone can explain it to me that’d be cool

Written by Mark Needham

April 14th, 2013 at 6:18 pm

Posted in DevOps

Tagged with

Capistrano, sed, escaping forward slashes and ‘p’ is not ‘puts’!

with 2 comments

Priyank and I have been working on automating part of our deployment process and one task we needed to do as part of this is replace some variables used in one of our shell scripts.

All the variables in the script refer to production specific locations but we needed to change a couple of them in order to run the script in our QA environment.

We’re therefore written a sed command, which we call from Capistrano, to allow us to do this.

The Capistrano script looks a little like this:

task :replace_in_shell do
	directory = "/my/directory/path"
	sed_command = "sed 's/^some_key.*$/#{directory}/' shell_script.sh > shell_script_with_qa_variables.sh"
	run sed_command
end

Unfortunately this creates the following sed command which isn’t actually valid syntactically:

sed 's/^some_key.*$//my/directory/path/' shell_script.sh > shell_script_with_qa_variables.sh

We decided to use ‘gsub’ to escape all the forward slashes in the directory path and to work out which parameters we needed to pass to ‘gsub’ we started using irb.

Executing gsub with the appropriate parameters leads us to believe that 2 backslashes will be added:

ruby-1.8.7-p299 > "/my/directory/path".gsub("/", "\\/")
 => "\\/my\\/directory\\/path"

This is because there IRB is implicitly called ‘inspect’ on the result which shows a different string than what we would actually get.

While writing this blog post I’ve also learnt (thanks to Ashwin) that ‘p’ is not the same as ‘puts’ which is what I originally thought and has been driving me crazy as I try to understand why everything I print includes an extra backslash!

The following code:

p "/mark/dir/".gsub("/", "\\/")

is the same as typing:

puts "/mark/dir/".gsub("/", "\\/").inspect

We were able to change our Capistrano script to escape forward slashes like so:

task :replace_in_shell do
	directory = "/my/directory/path"
	sed_command = "sed 's/^some_key.*$/#{directory.gsub("/", "\\/"}/' shell_script.sh > shell_script_with_qa_variables.sh"
	run sed_command
end

Written by Mark Needham

November 18th, 2010 at 6:40 pm

Posted in Ruby

Tagged with , ,