Simple background tasks with Rails on Windows

Posted by Stijn Pint on May 20, 2009

For one of our Rails projects, I needed a simple solution to run some background tasks on demand of the user (various calculations or data imports which can take several minutes to complete).
There are several solutions out there, like Starling, Workling, backgrounDRB ,… but I really wanted something without too much complexity. Another catch: our project is running on a Windows production server… yeah, I know…

So this is what we have:

On the application side, users can request several “server tasks”.


These are then stored in the database and are managed through an ActiveRecord model “ServerTask”.


The model stores info on the user, the type of request and has a simple state machine on it (waiting – running – finished – crashed)

p. This system basically acts as a queue, where some worker can pick up a task and execute it.


The worker can just be a simple rake task which checks the queue for new tasks.

For this application I needed only one worker. There are several reasons:

• Tasks can be dependent on each other (can’t execute 2 at the same time or they need to be executed in a specific order)
• CPU power is limited (Windows VMWare)
• Intranet application: number of users will not be that large, so the waiting times should not be too excessive.
  
  p. The solution I came up with at first (and which I dumped some time ago…) used Windows Scheduled Tasks :

I created a scheduled task which starts the worker (a simple rake task) every 24 hours. The worker in turn runs for 24 hours, checking the queue every 10 seconds or so.


The 24 hour restart is necessary to free up memory; the memory usage of the worker tends to get bigger and bigger with time…

Why I wanted to replace this:

• Extra deployment effort necessary to manage the possibly running worker and scheduled task
• In case the worker crashed for some reason (application crash or windows error) it was impossible to recover (manually restart or wait until next scheduled task run)
• Risk of having multiple worker instances running
• Manually terminating running worker processes through the Windows Task Manager is not easy: just another ruby.exe process in the list.

New solution

To solve these problems, I looked for a solution with a background rake task which is started on demand and that stops when the queue is empty. (With thanks to Ryan Bates’ railscast )


I accomplished this by using a Rails observer; the code follows.

The observer server_task_observer.rb :

<code class='ruby'>
class ServerTaskObserver < ActiveRecord::Observer

  def after_create(server_task)
    if File.exists?("#{RAILS_ROOT}/log/running.txt")
      # We're setting a timeout on the servertasks, so running.txt should never be older than 60 minutes
      # if so, process has probably crashed (in a dirty windows way) and we'll try to recover (in a my own dirty way :-)
      if Time.now > File.mtime("#{RAILS_ROOT}/log/running.txt") + (TASK_TIMEOUT + 1.minute)
        File.delete("#{RAILS_ROOT}/log/running.txt")
          
        # clear state of the running task if necessary to unblock queue
        # normally done in 'ensure', but might happen after windows crash
        task = ServerTask.find_in_state(:first, :running)
        task.crash! if task
        
        run_server_task #restart the worker
      end
    else
      run_server_task # start the worker
    end
  end

  def run_server_task
    File.open("#{RAILS_ROOT}/log/running.txt", 'w') do |f|
      f << Time.now.to_s
    end
    call_rake('server_task')
  end
end

In environment.rb (or wherever you want to put it) :

<code class='ruby'>
TASK_TIMEOUT = 60.minutes 

def call_rake(task, options = {})
  options[:rails_env] ||= RAILS_ENV
  args = options.map { |n, v| "#{n.to_s.upcase}='#{v}'" }

  if ENV["OS"].blank? # linux
    system "/usr/bin/rake #{task} #{args.join(' ')} --trace 2>&1 >> #{RAILS_ROOT}/log/rake.log &" #--trace 2>&1 >> #{RAILS_ROOT}/log/rake.log
  else # Windows
    # quoting is important !!!!!!!!!!!!!
    system %Q(start cmd /c "title #{task} & cd #{APP_CONFIG[:application_path]} & rake --trace #{task} #{args.join(' ')}")
  end
end 

The worker server_task.rake :

<code class='ruby'>
desc 'Execute the tasks in the queue'
task :server_task => :environment do

  restart = true
  while restart
    restart = false
    todo = ServerTask.find_in_state(:first, :waiting, :order => "created_at ASC, id asc")

    begin
      # execute if servertask found and not already one running (sometimes double rake task running).
      if (todo && ServerTask.find_in_state(:all, :running).empty?)   # run if no other task running
        # reset timestamp
        File.open("#{RAILS_ROOT}/log/running.txt", 'w') do |f|
          f << Time.now.to_s
        end
        Timeout.timeout(TASK_TIMEOUT) do
          todo.execute # execute specific server task, implement in server_task.rb
        end
        restart = true
      end
    rescue
      p 'error rescued and reraised'
      raise
    ensure
      if todo && todo.current_state == :running
        todo.crash!
        p 'crashed event'
      end
      File.delete("#{RAILS_ROOT}/log/running.txt") unless restart
    end
  end
end 

When a server task is requested by the user, the observer will try to start the worker.

To avoid multiple instances of the worker, I use a file ‘running.txt’ which is created when the worker starts and gets destroyed when it’s finished.

There is an extra difficulty here:


Although the ‘running.txt’ file is destroyed in an ensure block, the block isn’t guaranteed to be executed, for instance when the worker windows process gets killed for some reason.


This would leave the system in a dirty state and no new worker can be started.


To allow the system to recover I had to include an extra check:


A single background task cannot run more than 60 minutes (raises a Timeout error), so in the observer I check the timestamp of the file ‘running.txt’. If it’s older than 60 minutes, the file is deleted and I force a restart of the worker.

We have this setup running now for the past few months, and it’s causing significantly fewer headaches than before. Better than aspirin!

Feel free to leave some feedback !


What are you using to execute long running tasks with Rails (on Windows or not) ?

blog comments powered by Disqus