Setup Ruby on Rails with Sidekiq on Heroku

Sidekiq with is the ultimate async background job processing tool for Rails applications. This guide walks through how to setup Sidekiq with Rails on Heroku.

Sidekiq Intro

Sidekiq is a asynchronous background queuing and job processing gem. Sidekiq processes are threaded, allowing workers (jobs) to run in parallel. This allows us to pass off time consuming or I/O heavy tasks to a background process, instead forcing your end users hang for a response.

Sidekiq depends on Redis, a ultra-fast memory key/value data-store.

Although we’re using Sidekiq with Rails in this tutorial, Sidekiq will work independently or alongside most other Ruby web frameworks.

Installation

Before we setup Sidekiq, you’ll need a running Rails application, a database server (I personally use PostgreSQL) and a Redis server. Technically the PG DB (Postgres database) is not actually required to run Sidekiq, however, you’ll want to operate Rails with a DB.

Our Heroku production environment will run all 4 mechanisms on separate instances;

1. Rails running a web server as a process
2. Sidekiq as a worker process
3. PostgreSQL as a database service
4. Redis as a data store service

For Heroku, you’ll require at least 2 Dynos, 1 for Rails and 1 for the Sidekiq process. Redis and PostgreSQL run as add-on services.

During this tutorial I’m going to skip setting up Rails and PostgreSQL, I’ll assume this part is already setup.

Local Redis Setup

Using a Mac you can use this commands to install Redis:

brew install redis

This will run through the installation process

Production Redis

For Heroku I suggest you use the heroku-redis add-on, it’s free and good for 30MB space with 30 connections. I’ve never had any issues using this add-on with Sidekiq.

Important! - If your using Redis already as a cache store, be sure to use a separate Redis add-on instance just for Sidekiq. Redis caching and background jobs queuing processes don’t work well together…

Setup Sidekiq on Rails

Add Sidekiq to your Gemfile

gem 'sidekiq'

Run bundler

bundle install

Create 2 new files in your Rails app ./ root directory

Create a file Procfile

web:    bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml

Create a file Procfile.dev

redis:  redis-server
worker: bundle exec sidekiq
rails:  PORT=3000 bundle exec rails server

Create a file config/sidekiq.yml

development:
  :concurrency: 5
production:
  :concurrency: 10
:queues:
  - high
  - default
  - mailers

Note - You can you have as many queues as you like, but I suggest sticking to 3 max.

Create a file config/initializers/sidekiq.rb

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('REDIS_URL'), size: 1, network_timeout: 5 }
end

Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL'), size: 12, network_timeout: 5 }
end

Sidekiq::Extensions.enable_delay!

We set the redis server size to 12, 10 for each of the Sidekiq worker threads and 2 additional for Sidekiq’s internal use. 10 matches the value we set for production concurrency in the Sidekiq config file config/sidekiq.yml. Whenever you change the production concurrency value, make sure to keep a +2 buffer value for Sidekiq redis config server value above.

Sidekiq::Extensions.enable_delay! will allow you to call delay on certain objects and parse them into Sidekiq. I would suggest using this only for mailers or similar. Example UserMailer.delay.welcome(user.id).

If your already using Redis for caching, change REDIS_URL to SIDEKIQ_REDIS_URL and make sure to update this environment variable with the url of the 2nd Redis instance on Heroku.

(Optional) Edit config/application.rb with a single line

class Application < Rails::Application
  config.active_job.queue_adapter = :sidekiq
end

This allows you to use ActiveJob with the Sidekiq backend (if desired).

Setup Sidekiq Web UI

Sidekiq comes with a super handy web UI to allow CRUD management of Sidekiq in real-time. It only takes a simple step to setup the web UI.

Create a new route in config/routes.rb

require 'sidekiq/web'

Rails.application.routes.draw do
  authenticate :user, lambda { |u| u.admin? } do
    mount Sidekiq::Web => '/sidekiq'
  end
  get '/sidekiq' => redirect('/')
end

Change .admin? to whatever role you have in your system. It’s important to restrict access to the Sidekiq web UI.

If you do not have a admin or super_admin user roles you can use this route instead;

require 'sidekiq/web'

Rails.application.routes.draw do
  Sidekiq::Web.use Rack::Auth::Basic do |username, password|
    ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(username), ::Digest::SHA256.hexdigest(ENV['SIDEKIQ_USERNAME'])) &
      ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(password), ::Digest::SHA256.hexdigest(ENV['SIDEKIQ_PASSWORD']))
  end
  mount Sidekiq::Web, at: '/sidekiq'
end

Remember to add SIDEKIQ_USERNAME and SIDEKIQ_PASSWORD to your Heroku environment variables. You can also replace ENV['SIDEKIQ_USERNAME'] and ENV['SIDEKIQ_PASSWORD'] with Rails.application.credentials.sidekiq_username and Rails.application.credentials.sidekiq_password if your using Rails credentials.

Process Manager Setup - Local Development

For running Rails and Sidekiq locally, it’s easier to manage multiple services with a process manager. You can use any process manager, I’m using Hivemind.

brew install hivemind

Starting Sidekiq - Local Development

Great news! Your done setting up Sidekiq. Now we need to start creating some awesome workers to process those heavy tasks.

Before we start creating workers let’s boot up Sidekiq locally alongside your Rails application.

Be sure to exit any Rails server processes before executing this command.

hivemind Procfile.dev

This will start Redis, Sidekiq and Rails from 1 terminal window. You can quit anytime using CMD+C, stopping all 3 processes.

Head over to http://localhost:3000/sidekiq and you’ll see Sidekiq’s web UI.

Creating Workers

Now that we have Sidekiq up and running alongside Rails, we need create some workers to starting using it.

Create new folder directory app/workers

You can also use ActiveJob directly, since we changed the default queue adapter to Sidekiq. I’ll explain the API usage for both options shortly.

Create your first worker

rails g sidekiq:worker RunDailyReports

This will create a new file app/workers/run_daily_report_worker.rb

class RunDailyReportWorker
  include Sidekiq::Worker

  def perform(*args)
    # do something
  end
end

Let’s go ahead and add some Sidekiq options to this worker;

class RunDailyReportWorker
  include Sidekiq::Worker
  sidekiq_options queue: :high, retry: 3

  def perform(*args)
    # do something
  end
end

These options tell Sidekiq to add this worker to the high queue and only retry the worker 3 times if any exceptions are raised. Sidekiq will stagger the retry frequency for each failure. The default retry attempts is 25. Workers reaching the max retry threshold are moved to the dead queue, seen on the Sidekiq web UI. Full details on how Sidekiq manages retries can be found here.

ActiveJob

We’ve seen how to create a Sidekiq workers (my preferred setup), how about ActiveJob? Let’s create a new a similar job.

rails g job RunDailyReport

This will create a new file called app/jobs/run_daily_report_job.rb

class RunDailyReportJob < ApplicationJob
  queue_as :default

  def perform(*args)
    # Do something later
  end
end

Note - As of Sidekiq 6.0.1 you can sidekiq_options in your ActiveJobs. In previous versions this is not possible. Another drawback with using ActiveJob is the retry mechanism, currently controlled by ActiveJob.

Checkout all the details on the Sidekiq wiki for using ActiveJob. I highly suggest sticking to Sidekiq workers.

Sidekiq Usage (Queue API)

Now we can run any of the following commands from anywhere in your application;

Sidekiq Worker

RunDailyReportWorker.perform_async
RunDailyReportWorker.perform_async(arg)
RunDailyReportWorker.perform_async(1,2)
RunDailyReportWorker.perform_in(5.minutes)
RunDailyReportWorker.perform_in(5.minutes, args)

ActiveJob

RunDailyReportJob.perform_now
RunDailyReportJob.perform_later
RunDailyReportJob.perform_now(arg)
RunDailyReportJob.perform_later(1,2)

Running Sidekiq in Production (Heroku)

Steps

1. Deploy your app to Heroku
2. (Optional) Setup any ENV variables for sidekiq (as mentioned above) using the Heroku dashboard. Note - This is optional as we setup default values for all the sidekiq ENV variables above. For most apps these default will be fine.
3. Navigate to the Resources tab on the Heroku dashboard. There is a section at the for managing Dynos and their process types. One of those options will appears as worker. You have to turn this on to boot sidekiq.

Example - Heroku Dyno process manager dashboard

The second list item named worker has to be activated to boot Sidekiq.

Final thoughts

Before throwing any work at Sidekiq, I highly suggest reading the Sidekiq wiki for best practices. Sidekiq embraces concurrency, a new concept to learn for some.

The setup above is just a basic configuration to get you up and running for small to moderate apps running on Heroku.

Happy coding 😉