sidekiq_publisher

This gem provides support to enqueue jobs for Sidekiq by first staging the job in Postgres and relying on a separate process to communicate with Sidekiq/Redis.

The publisher process is alerted that a job is available to be published using Postgres NOTIFY/LISTEN.

This approach has the benefit that jobs can be published as part of a transaction that modifies the system of record for the application. It also allows jobs to be created even when Sidekiq/Redis is temporarily unavailable. The separate publisher process handles retries and ensure that each job is delivered to Sidekiq.

⚠️ Not all jobs are staged in Postgres. This is determined dynamically: if the job is enqueued from within an ActiveRecord transaction, then it is staged in Postgres. If not, then it bypasses Postgres and is enqueued directly to Redis via Sidekiq.

Installation

Add this line to your application's Gemfile:

gem "sidekiq_publisher"

And then execute:

$ bundle

Or install it yourself as:

$ gem install sidekiq_publisher

Run the generator to create migrations for the jobs table and notifications:

$ rails generate sidekiq_publisher:install

Configuration

This gem uses the following configuration:

• logger: the logger for this gem to use.
• exception_reporter: a Proc that will be called with an exception
• metrics_reporter: an optional object to record metrics. See below.
• batch_size: the maximum number of jobs that will be enqueued together to Sidekiq
• job_retention_period: the duration that published jobs will be kept in Postgres after they have been enqueued to Sidekiq

Metrics Reporter

The metrics reporter that can be configured with an object that is expected to respond to the following API:

count(metric_name, count)
gauge(metric_name, count)

Metrics will be reported for:

• the number of jobs published in each batch
• the number of jobs purged

Unpublished Jobs

There is also a module that can be used to record a metric for the number of unpublished jobs:

SidekiqPublisher::ReportUnpublishedCount.call

It is recommended to call this method periodically using something like cron or clockwork.

Instrumentation

Instrumentation of this library is implemented using ActiveSupport::Notifications.

The support for the configurable metrics_reporter and exception_reporter options is implemented using ActiveSupport::Subscriber.

If an alternate integration is required for metrics or error reporting then it can be implemented using outside this library based on these examples.

Tracing

The instrumentation in the library also supports integration with application tracing products, such as Datadog APM.

There is an optional integration with Datadog APM that can be required:

require "sidekiq_publisher/datadog_apm"

This file must be required in addition including the sidekiq_publisher gem or requiring sidekiq_publisher.

This integration covers all of the sections of the library that are instrumented and serves an example for implementing trace reporting for other products outside this library.

Usage

ActiveJob Adapter

This gem includes an adapter to use SidekiqPublisher with ActiveJob. This adapter must be explicitly required:

require "active_job/queue_adapters/sidekiq_publisher_adapter"

The adapter can also be required via your Gemfile:

gem "sidekiq_publisher", require: ["sidekiq_publisher", "active_job/queue_adapters/sidekiq_publisher_adapter"]

The adapter to use with ActiveJob must be specified in Rails configuration

# application.rb
config.active_job.queue_adapter = :sidekiq_publisher

# or directly in configuration
Rails.application.config.active_job.queue_adapter = :sidekiq_publisher

To selectively roll out the SidekiqPublisher, the adapter can be overridden for a specific job class:

class MyJob < ApplicationJob
  self.queue_adapter = :sidekiq_publisher
end

ActiveJob Exception Reporting

Many exception monitoring service (e.g. Sentry, Airbrake, Honeybadger, etc) already provide basic integration support for Sidekiq. These integration should also work with SidekiqPublisher. However, you may need to explicitly include ActiveJob::QueueAdapters::SidekiqPublisherAdapter as a compatible adapter for this to work properly.

Alternatively, you can manually report the exception:

retry_on SomeError, attempts: 10 do |_job, exception|
 Raven.capture_exception(exception, extra: { custom: :foo }) # Reporting using the Sentry gem
end

SidekiqPublisher::Worker

Sidekiq workers are usually defined by including Sidekiq::Job or Sidekiq::Worker in a class.

To use the SidekiqPublisher, this can be replaced by including SidekiqPublisher::Worker. The usual perform_async, etc methods will be available on the class but jobs will be staged in the Postgres table.

Tying to a transaction

To guarantee that your job is enqueued when there's a change to the system of record, simply publish it during the transaction representing that change. Usually, that can be accomplished by publishing in one of the ActiveRecord callbacks that are called within-transaction (e.g. after_save, but not after_commit and its derivatives):

class Frob < ApplicationRecord
  after_save do
    MyJob.perform_later id
  end
end

For considering more complicated situations (e.g. jobs that should be guaranteed during specific changes across models), the rails guides on querying and callbacks, and the documentation on transactions in ActiveRecord are good resources to consult.

Running

The publisher process that pulls the job data from Postgres and puts them into Redis can be run with a rake task that is added via Railtie for Rails applications:

bundle exec rake sidekiq_publisher:publish

Testing

Sidekiq

When using sidekiq_publisher directly with Sidekiq workers, the testing modes provided by Sidekiq are supported.

Require the sidekiq_publisher/testing file. (This should only be done in test!)

require "sidekiq_publisher/testing"

This file requires "sidekiq/testing" so there is no need to explictly require both. Note that by default, Sidekiq sets the test mode to fake and stores jobs in a jobs array for each worker class.

To have SidekiqPublisher continue to insert jobs into a table within tests call Sidekiq::Testing.disable!.

ActiveJob

When using the sidekiq_publisher adapter for ActiveJob, use the ActiveJob test adapter if you want to run jobs inline during tests.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ezcater/sidekiq_publisher.

License

The gem is available as open source under the terms of the MIT License.