Ent Encryption

Certain industries have strong regulation around the storage of personal data. Private medical data, financial data, social security numbers, credit card numbers, all of these things are sensitive. Sidekiq Enterprise v1.3.0+ supports transparent encryption of the last job argument (aka the secret bag) so sensitive data at rest in Redis cannot be seen.

Enable Encryption

In your initializer, activate the feature. I recommend putting this as the last item in your initializer:

Sidekiq::Enterprise::Crypto.enable(active_version: 1) do |version|
  # this block should return the key for the version N
  #
  # every time you need to rotate the crypto key, you should bump
  # the active_version. this block should always return the key for 
  # the given version.
  #
  # You can store the key for each version in a file, in an ENV variable or on a remote
  # keystore - it's up to you how to do key management.
end

You can create a new random key in irb like so:

require 'openssl'
File.open("/var/crypto/secret.1.key", "w:ASCII-8BIT") { |file| file.write(OpenSSL::Cipher.new("aes-256-gcm").random_key) }

Configure your Private Jobs

Sidekiq uses a special pattern to keep sensitive data private while also allowing some arguments to be cleartext to help with debugging job failures in the Web UI. The last argument for an encrypted job should be a Hash; it is the secret bag and will be encrypted - all other arguments are cleartext to aid with debugging.

This has one special side effect: all encrypted jobs must take >= 2 arguments. If you don't wish to have any cleartext, you can use perform(nil, secret_bag).

Tell Sidekiq to keep a job's arguments private:

class PrivateJob
  include Sidekiq::Job
  sidekiq_options encrypt: true

  def perform(x, y, secret_bag)
  end
end

Now create a new job with the secret bag as the last parameter:

SecretJob.perform_async(1, 2, {"ssn" => "123-45-6789"})

Within Redis, the job data will look something like this:

{"class"=>"SecretJob", "args"=>[1, 2, "BAhTOhFTaWRla2lxOjpFbmMIOgdpdiIVo1mbHmnVxiOITRFamysuBzoGdmkGOglibG9iIjUsYKydz73NLUB952KmES-K2LURDF6I_dWTBtzOX6493OaqNl6KZta19vWTKRwX9KQ="], "retry"=>true, "queue"=>"default", "encrypt"=>true, "jid"=>"b5f7df591b368910db3fc2d2", "created_at"=>1468597911.397162, "enqueued_at"=>1468597911.3972108}

Notes

• The implementation uses OpenSSL's aes-256-gcm Cipher, which is the strongest widely available cipher as of 2016. This cipher requires OpenSSL 1.0.1; the code will fallback to using aes-256-cbc if running on an older OpenSSL version. The code can decrypt jobs using either cipher - you may upgrade without worry.
• The secret bag will be encrypted in the Web UI, when returned from the API, etc. Only when your job is actually executed will it be decrypted.
• The encryption header adds about 150 bytes to the size of arguments plus 30% Base64 encoding overhead. Arguments which are 1000 bytes in plaintext are about 1500 bytes when encrypted. The Base64 encoding is necessary as JSON content must be valid UTF8 and the encrypted data is binary.
• ONLY the last argument is encrypted. Any error message and backtrace will still be plaintext within a job. Be careful not to expose any sensitive data when raising errors.
• The unique jobs feature will not work on encrypted jobs, since the encrypted argument is always unique. If a Worker has both encrypt and unique_for options, Sidekiq will raise an error.