feat: Support for scheduled Workflows with CronWorkflow CRD

[simster7]

Closes: #1229

Design

Implements a CronWorkflow CRD that defines a scheduled Workflow. The design is such that any Workflow can be simply converted to a CronWorkflow by replacing kind: Workflow with kind: CronWorkflow and adding some CronWorkflowOptions (in short: CronWorkflow = Workflow + CronWorkflowOptions).

Example:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron-wf
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: whalesay
    templates:
    - name: whalesay
      container:
        image: alpine:3.6
        command: [sh, -c]
        args: ["date; sleep 90"]

Implementation

Implemented by creating a new controller that will be embedded within workflow-controller (i.e. the user will not need to install any separate controllers). The cron controller will monitor all CronWorkflow resources and will execute and terminate Workflows based on their respective CronWorkflowOptions.

Features/Vision:

• Virtually no conversion to set up a new CronWorkflow
• Works will all K8s CronJob options
• Works with WorkflowTemplates

This is still a WIP. TODO:

• Finish implementing K8s CronJob options:
  • schedule
  • startingDeadlineSeconds
  • concurrencyPolicy
  • suspend
  • successfulJobsHistoryLimit
  • failedJobsHistoryLimit
• Support exhaustive logging and event recording under the CronWorkflow CRD (i.e. make kubectl describe cronwf [...] useful)
• Write argo CLI integration
• Add CronWorkflow Validation
• Testing

[simster7]

@alexec Added tests and enhanced the e2e suite, please review

[alexec]

Test failure:

workflow/cron/operator.go:84:10: Errorf format %w has unknown verb w

[alexec]

Conditional approval - can you fix the broken test?

[alexec]

anyway to reduce this time to <30s?

[simster7]

Unfortunately, I don't think so. I need at least two cron cycles for this test, the least significant of which is a minute. I could mock the cron framework to run -- say -- every 15 seconds, but since we are dealing with k8s objects I worry that the short time might cause these tests to be flakey. To mediate the effects of this I decided to run these particular tests in parallel, so the entire cron test suite should take at most 3 minutes.

Let me know what you think.

[alexec]

would it be possible for this to wait until a condition is met and the exit (unless it timesout) - that way test might be faster

[simster7]

I intended this to be a general wait function, I think it could potentially be useful as is in the future.

W.r.t. the cron tests, see #1758 (comment)

[simster7]

Test failure:
workflow/cron/operator.go:84:10: Errorf format %w has unknown verb w

Will fix.

Seems like E2E tests are not excluded when running make test, will fix this

[alexec]

I don't actually see cron test running on CI. I think you need a block similar to this:

https://github.com/argoproj/argo/blob/83ae2df4130468a95b720ce33c9b9e27e7005b17/test/e2e/smoke_test.go#L42

[rdigiorgio]

So, I guess timezone configuration is, like for CronJob spec, not going to be supported ? :(

[evaneaston]

@evaneaston There is currently no plan to provide time zone support. Do you have a use case where you think it would be useful?

I though I had replied, but clearly I never submitted it. And since there was another comment about timezone support added today by @rdigiorgio, I'll include what I wanted to say earlier:

Use Case: Being able to schedule something periodically during, say business hours M-F using simple cron ranges when your business hours stradle UTC midnight. The patterns would be much simpler if you could evaluate them in the context of a specific timezone

That said, it's pretty straightforward schedule a workflow to run every period, but start with a step workflow whose sole job can checks if the time is in a local timezone window and whose result is a conditional for continuing the workflow.

It's not as efficient as being able to evaluate cron expressions in the context of a timezone, and results in a lot of noop workflows, but it works, as a workaround.

[rdigiorgio]

Indeed, we can find plenty of workarounds, but it seems to be cumbersome and moreover, playing with infrastructure that can scale from 0 to X, it is sad to say "Ok let's run the workflow, wait for a node to span (GKE autoscale feature), the first step says that we should not go forward and abort the workflow, wait for the cluster to scale down" = pay extra bucks because CRON does not support timezones.

By the way, I am nowhere near to know how "hard" it would be to implement on CronWorkflows, but I really thought this new feature would come bundled with timezones support and I would be able to drop all those K8S CronJobs

[evaneaston]

@rdigiorgio Agreed.

I was hoping that since argo-events supported timezone in the calendar event source, argo CronWorkflows might too.

It looks like robfig/cron v3 now supports timezones. So perhaps implementing support for timezones isn't too hard to add...not that I have plans to take up go and implement the change.

[simster7]

No reason timezone support can't be added in the future :) Opened to #1931 track this

[LucaSoato]

Sorry if the question seems stupid, but how does the scheduler work? Is there a single point of failure, or is it stable enough to keep running workflows even if a node goes down?

[simster7]

The scheduler will be running as long as the workflow-controller is running. I.e., if you can run workflows, then the scheduler should be running. If the workflow-controller goes down at some point, once it restarts the scheduler will run any missed CronWorkflows as specified per activeDeadlineSeconds