Remove cron dependency for worker health checker

[zachmargolis]

Why: cron (leader host) is not compatible with autoscaling groups

[monfresh]

Does this mean we can't use cron jobs for anything? If so, will we need to rework everything in config/schedule.rb? Also, what will be the new frequency of checking the worker health?

[brodygov]

Looks good to me. It would also be good to put some comments in the commit or in the code to explain that we now have the health check kick off the dummy job.

[brodygov]

Right, in an auto scaling group all the instances are identical, so we don't have any way to ensure that cron jobs are only run on a single instance.

I've seen a few different strategies to deal with this most commonly:

• Have a leader election to elect one app server leader, and have it run all the jobs
• Run jobs from cron on all servers, but use a distributed lock system (could be backed by redis or the database) so that a server takes out a lock for the job before running it to ensure that it only gets run once per interval
• Run jobs from a queueing system that is guaranteed to enqueue jobs at the rate you expect, either by triggering from an external system like new relic, or having a distributed queue system (which probably does a leader election or locks internally)

[zachmargolis]

We're currently working on getting ahold of Sidekiq Enterprise to run our cronjobs for us for things like SFTPing