Cailey/docs/cronjobs #548

caggles · 2020-07-21T19:38:16Z

No description provided.

StevenBarre · 2020-07-21T23:01:56Z

However, we are not going to provide specific limits on what counts as "too often" or "too long" and which require action on your part.

I would like to outright ban crons that run every minute, but "every 5 minutes" and less frequent is up to the discretion of the app team.

jefkel · 2020-07-22T16:51:21Z

I would suggest that a "bad" cronjob is one where the overhead is greater than the job workload (and you can make that a "per hour" calculation.)
Assuming that a pod takes 90 seconds to provision and be ready to work. (can also be measured based on the pod you're using for your job for a more accurate calculation)

eg: your job is running and it takes 1 second, and runs every 5 minutes.
cluster work = 90s * 20 = 30 minutes of work / hour
App work = 1 x 20 = 20 seconds of work / hour

This is a "Bad" cron - 20 seconds of application workload for 30 minutes of cluster effort. Recommend you stick this in a long running pod and scale the resources super low.

backup eg: backup cron runs 1 every 4 hours and takes 10 minutes to complete once running.
cluster work = 90s / 4 = 23s of work / hour
App work = 600 / 4 = 150s of work / hour

woot, this is a "Good" cron - 25 seconds of cluster work for 6x the app work...

jefkel · 2020-07-22T16:59:04Z

Regarding long running "Cron" pods:

The current https://github.com/BCDevOps/backup-container has built in support for running as a long running pod or a kube cronjob. It's using go-crond as it's cron timer. I'd check it out that reference (since it's a BCGov community implementation) before looking at spending time evaluating another option like Jobber. (not that Jobber isn't interesting, but is there a reason we need to invent a new method instead of referencing Wade's work and calling it done?)

jefkel

More specific ways for teams to calculate if their Cron is wasting resources (as per comment above)

Switch jobber mention to point to backup-container for a sample of a long running pod that implements a cron.

StevenBarre · 2020-07-27T20:04:19Z

👍

jefkel · 2020-07-28T19:21:57Z

docs/CronjobManagement.md

+
+### Cronjobs Scheduled Too Often
+
+We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.


Suggested change

We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.

We have a handy rule of thumb for deciding how often is too often - when your application workload duration is more than the cluster overhead duration for starting the job(s). (See below for an example ratio calculation)

jefkel · 2020-07-28T19:25:56Z

docs/CronjobManagement.md

+
+We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.
+
+If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object.


Suggested change

If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object.

If you find that you need to run a cronjob every minute, then you are far away from the "application workload > cluster overhead to start the workload" rule of thumb, and your job could be having a negative impact on cluster health. In these cases, your cronjob may be turned off my the Platform team and informed that you need to adjust your scheduled job design before re-enabling.

jefkel · 2020-07-28T19:26:47Z

docs/CronjobManagement.md

+We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.
+
+If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object.
+If every minute is necessary for your app, you *must* use a persistent pod.


Suggested change

If every minute is necessary for your app, you *must* use a persistent pod.

If every minute is necessary for your app, this sounds like a great opportunity for a persistent cron deployment.

jefkel · 2020-07-28T19:28:00Z

docs/CronjobManagement.md

+If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object.
+If every minute is necessary for your app, you *must* use a persistent pod.
+
+Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part.


Suggested change

Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part.

Outside of this rule of thumb, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part.

jefkel · 2020-07-28T19:29:58Z

docs/CronjobManagement.md

+
+This version of the command will only pull up those cronjobs running once per minute - if you want to find cronjobs with other schedules, just adjust the grep filter accordingly.
+
+### Cronjobs with a Bad Ratio of Work to Overhead


Suggested change

### Cronjobs with a Bad Ratio of Work to Overhead

### Example calcuating Cronjob Ratio of Work to Overhead

jefkel · 2020-07-28T19:32:17Z

docs/CronjobManagement.md

+
+There are a number of ways to approach creating this kind of persistent pod.
+
+A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.


Suggested change

A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.

An existing example (with great documentation) of a long-running pod that implements a cronjob is the [Backup Container](https://github.com/BCDevops/backup-container)

Another option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.

jefkel · 2020-07-28T19:32:54Z

docs/CronjobManagement.md

+A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.
+If your team finds a good solution that works well, we invite you to include a link to your deployment config on this document!
+
+There is also an example of a long-running pod that implements a cronjob in the [backup container](https://github.com/BCDevOps/backup-container).


If above change to jobber line is accepted, then simply delete this line.

Suggested change

There is also an example of a long-running pod that implements a cronjob in the [backup container](https://github.com/BCDevOps/backup-container).

caggles added 3 commits July 21, 2020 12:30

cronjob doc init

a25a9f0

cronjob doc init

9c9c1d5

add jobber reference

8d95754

caggles requested review from StevenBarre and jefkel July 21, 2020 22:02

caggles mentioned this pull request Jul 21, 2020

CronJob's every minute - ask teams to reduce these and possibly put them into a persistent pod. #291

Closed

jefkel suggested changes Jul 22, 2020

View reviewed changes

requested updates

f59843b

caggles requested a review from jefkel July 27, 2020 19:21

add backup container mention

1971876

mitovskaol approved these changes Aug 6, 2020

View reviewed changes

jefkel suggested changes Aug 6, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cailey/docs/cronjobs #548

Cailey/docs/cronjobs #548

caggles commented Jul 21, 2020

StevenBarre commented Jul 21, 2020

jefkel commented Jul 22, 2020 •

edited

Loading

jefkel commented Jul 22, 2020

jefkel left a comment

StevenBarre commented Jul 27, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020

jefkel Jul 28, 2020


		### Cronjobs Scheduled Too Often

		We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.

	We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.
	We have a handy rule of thumb for deciding how often is too often - when your application workload duration is more than the cluster overhead duration for starting the job(s). (See below for an example ratio calculation)


		We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job.

		If you find that you need to run a cronjob every minute, the Platform Team has decided that you cannot do this using the Openshift cronjob object.

	If every minute is necessary for your app, you must use a persistent pod.
	If every minute is necessary for your app, this sounds like a great opportunity for a persistent cron deployment.

	Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which require action on your part.
	Outside of this rule of thumb, we are not going to provide specific limits on what counts as "too often" or "too long" and which require action on your part.


		This version of the command will only pull up those cronjobs running once per minute - if you want to find cronjobs with other schedules, just adjust the grep filter accordingly.

		### Cronjobs with a Bad Ratio of Work to Overhead

	### Cronjobs with a Bad Ratio of Work to Overhead
	### Example calcuating Cronjob Ratio of Work to Overhead


		There are a number of ways to approach creating this kind of persistent pod.

		A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.

-A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.
+An existing example (with great documentation) of a long-running pod that implements a cronjob is the [Backup Container](https://github.com/BCDevops/backup-container)
+Another option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem.

Cailey/docs/cronjobs #548

Are you sure you want to change the base?

Cailey/docs/cronjobs #548

Conversation

caggles commented Jul 21, 2020

StevenBarre commented Jul 21, 2020

jefkel commented Jul 22, 2020 • edited Loading

jefkel commented Jul 22, 2020

jefkel left a comment

Choose a reason for hiding this comment

StevenBarre commented Jul 27, 2020

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel Jul 28, 2020

Choose a reason for hiding this comment

jefkel commented Jul 22, 2020 •

edited

Loading