Posted by Peter Vanbroekhoven on Aug 04, 2010

The past two days I've been on a veritable odyssey trying to deploy a Rails 3 application on Ruby 1.9.2, complete with sirens leading me astray with their seductive song and six-headed monsters nearly killing me. It's an adventure alright.

We have been using a home-brewed time management application based on Google Calendar for a while now at XAOP and we are pretty happy with how it works. We decided to rewrite it from scratch in Rails 3. Determined to live on the edge on this one, we built it on top of Ruby 1.9.2 and threw ourselves at its new features such as the new hash syntax.

Two days ago the time had come to deploy a first version. The original application was deployed on Heroku, but Heroku supports neither Ruby 1.9.2 nor Rails 3.0.0 RC. We need the latter for a feature introduced after the fourth beta. The same goes for Engine Yard's cloud platforms. And thus, in a heroic move, we decided to deploy the application ourselves on Amazon EC2. Having done this before, this seemed like a good idea in anticipation of support from Heroku/EY.

Simian express

First off, building the application was not hard compared to Rails 2.3 on Ruby 1.8.6. Most of the changes in Rails and ActiveRecord make a lot of sense and are incredibly easy to get used to. Ruby 1.9 support in gems and plugins is pretty good already. We had to wait for the mysql2 gem to get the encodings right, and some plugins used methods that were renamed or had disappeared from Rails 3, but these issues have been reported and should have been fixed. Monkey patching is great as a temporary solution in these cases.

Wrestling with Amazons

Setting up a database with Amazon RDS was a breeze. Starting an EC2 instance running Ubuntu was a piece of cake. Been there done that. No complaints in this department.

The keeper of secrets of the forest

Next up was the installation of the necessary packages: libreadline, libssl, libxml2, libxslt, apache2, lib-mysqlclient, and dev packages for all the libs in there. We need libreadline for getting history and input editing support in IRB. We need libssl for SSL support. We need libxml2 znd libxslt for Nokogiri. Surprisingly, the dev packages for zlib were already installed on the EC2 image I was using (we need zlib support for Rubygems to work). Apparently, the image also had a decent build environment installed so I didn't need to install it myself.

Painting the town Ruby

I decided to use RVM to install Ruby on the EC2 instance. This was painless, though I had to redo the installation after I realized that I would need access to the Ruby installation as root to start the Rails processes and so I needed to install it system-wide. On Ubuntu, loading RVM in the shell profile is as easy as executing this:

sudo "ln -s /usr/local/lib/rvm /etc/profile.d/rvm.sh"

For reference, we are using ruby-1.9.2-preview3. So far, so good.

Crossroads

Once the machine was set up, it was time to decide how to deploy and how to manage our Rails processed. For the latter, Phusion Passenger is a popular choice but it doesn't appear to have support for Ruby 1.9.2 yet -- it is planned for the 3.0 version. I considered using Moonshine for the full deployment stack, but it doesn't support Ruby 1.9.2 either. After a couple of false starts, I decided to set it all up myself. I was going to use good old Capistrano and the familiar Mongrel cluster with an Apache in front of it.

Digging for gems

Given our prior experience with Capistrano, getting the code deployed to the EC2 instance was easy enough. Capistrano has some nice hooks that allow you to customize certain things, such as creating a shared database.yml pointing to the RDS instance. I also added a hook to install the gems based on the Gemfile. This is where I hit the first hitch: calls to run in Capistrano can't find the gem command or any executables for installed gems. RVM has Capistrano support that installs rvm-shell as the default shell in Capistrano which has RVM preloaded. This does not solve the problem when you run sudo because it resets the PATH variable and the executables still cannot be found. You can adapt the sudoers file such that PATH isn't reset, and then the executables are found, but then Rubygems can't find any of the installed gems.

The solution to this is very simple: use sudo -i. The simplest solutions are really the hardest to find. You can set up Capistrano to always use sudo -i by adding this to your config/deploy.rb:

set :sudo, "sudo -i"

This means that root's profile is loaded anew on each request which could be slower, but this is hardly noticeable. I'm not entirely happy with this solution because it supposes that Capistrano performs no extra magic on the command string. It works, though, and there may be a better solution in Capistrano but I have a hard time finding some things in the Capistrano documentation. This is a bit surprising given that Jamis Buck, Capistrano's creator, was praised for his awesome documentation in the days when Ruby documentation was very scarce or simply non-existing.

Deploying the Calabrian cavalry

Given our prior experience with Capistrano, it was easy enough to get our code onto the EC2 instance. Our code is in a private github project and we don't want to put our keys on the EC2 instance, but this option in Capistrano does the trick:

set :deploy_via, :copy

It tells Capistrano to check out the source on our development machine with our GitHub keys, zip the source up, copy it to the server and unpack it there.

Blustering through clustering

Then I turned to setting up a Mongrel cluster. There's a Ruby 1.9 compatible fork of mongrel_cluster on GitHub that we can use. Setting it up is all standard stuff, but our Rails processes crash with the following error:

no such file to load -- dispatcher

When starting the Rails server as a daemon with the -d option, the process also crashes during start-up. A quick Google search on that error message reveals that they removed the dispatcher file from Rails at some point. The same Google search tells me that Thin had this problem too but they already fixed it. So I resorted to using a Thin cluster.

As it turns out, Thin is written in C++ and requires g++. This renewed my gripes with Ubuntu's, and in general Debian's package management. The g++ executable now needs to be installed from its separate package. This is pretty silly because as far as I know g++ is just a shell script that calls gcc with some extra parameters to make it behave as a C++ compiler. Joy!

Regardless, setting up a Thin cluster is easier than setting up a Mongrel cluster. Installing the start-up script is a matter of doing:

  sudo "thin install"
  sudo "/usr/sbin/update-rc.d -f thin defaults"

All you need to do now is put a Thin config file for your application in /etc/thin and it will be run at boot. You can generate a skeleton file by running this command:

thin config -C /etc/thin/myapp.yml

I couldn't find any Capistrano recipes for Thin, but these do the trick just fine:

namespace :deploy do
  desc "Start the Thin processes"
  task :start do
    run "sudo -i thin start -C /etc/thin/timetag.yml"
  end
  desc "Stop the Thin processes"
  task :stop do
    run "sudo -i thin stop -C /etc/thin/timetag.yml"
  end
  desc "Restart the Thin processes"
  task :restart do
    run "sudo -i thin restart -C /etc/thin/timetag.yml"
  end
end

Indians on the war path

Now we are ready to set up Apache for load balancing. This blog post still does the trick. You just have to remember to enable the required modules:

sudo "ln -sf ../mods-available/proxy_balancer.load ../mods-available/rewrite.load ../mods-available/proxy.load ../mods-available/proxy.conf ../mods-available/proxy_http.load ../mods-available/proxy_connect.load /etc/apache2/mods-enabled/"

The grey havens

The whole process was more of a hassle than I hoped it would be. It gives me a renewed appreciation of companies such as Heroku and Engine Yard. It has a happy end, but it's not all bliss yet. We use PDFKit to generate PDF exports, and PDFKit doesn't work with Thin for some reason. Requests for PDF versions of a page seem to hang indefinitely. On our development machines, running Mac OS X, it seems to work if we start Thin in threaded mode, but on Ubuntu this makes things unpredictable and often extremely slow to the extent where it is beyond unusable.

I haven't tested any other web servers; we will probably wait until Mongrel catches up or until Heroku/EY tackle all the deployment issues for us. They are probably waiting for the final release of Ruby 1.9.2, but that should be this weekend if Yugui can manage to get it done.