Ent Rolling Restarts

Do you have long-running jobs which might take minutes or hours to finish? Do you like to deploy frequently? Sidekiq's traditional TSTP+TERM shutdown process can cause jobs to rerun, especially if they are long-running.

Sidekiq Enterprise 1.7.0 adds support for rolling restarts. In this mode, a Sidekiq process will stop fetching jobs immediately but will not exit until all jobs are complete. There is no limit to the time it can continue running. Upon signalling a rolling restart, a new process will be started to pick up new jobs.

How

einhorn is a special process manager which manages the rolling restart. Use it to run sidekiq or sidekiqswarm:

einhorn -m10 -- bundle exec sidekiqswarm [...cli args...]

To trigger a rolling restart, you tell einhorn to upgrade the binary after deploying the new code:

einhornsh --execute upgrade

If the new process is still running after 10 seconds, the old process will get USR2 and gracefully shutdown.

Integration

Modern init systems implement restart as explicit start and stop operations. You can use reload to trigger a rolling restart.

Ideally you'll have two types of deploys: the default, "minor" deploy which reloads Sidekiq and a "major" deploy which stops Sidekiq completely, runs any migrations and then starts it back up.

# For Ubuntu systemd users
# /lib/systemd/system/sidekiq.service
ExecReload=/usr/local/bin/bundle exec einhornsh --execute upgrade

Then a rolling restart is as simple as:

# systemd
systemctl reload sidekiq

Capistrano

If you are using Capistrano to deploy, rolling restarts will not pick up the change to the current symlink. You need to use Einhorn's --reexec-as option:

Sidekiq service file:
ExecStart=/usr/local/bin/einhorn --reexec-as /usr/local/bin/einhorn-myapp -- bundle exec sidekiqswarm

einhorn-myapp file:
#!/bin/bash

cd /var/www/myapp/current # change this to your app's current symlink
exec /usr/local/bin/einhorn "$@"

Notes

• Old and new processes will be running at the same time so memory usage will increase.
• Consider compatibility between old and new code versions; take care around database migrations.
• Rolling restarts work with both the sidekiq and sidekiqswarm binaries.
• If you manually quiet a process (via Web UI or API), you must manually terminate it too. Rolling restarts only affect active processes.
• Heroku's preboot feature implements rolling restarts but only for web dynos.