Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for swarm jobs #40307

Merged
merged 1 commit into from
Jan 20, 2020
Merged

Add support for swarm jobs #40307

merged 1 commit into from
Jan 20, 2020

Conversation

dperny
Copy link
Contributor

@dperny dperny commented Dec 12, 2019 •
edited
Loading

proposal (in swarmkit): moby/swarmkit#2852

- What I did

Adds support for Jobs in Swarm Mode

Jobs are Services which run some number of Tasks to completion (the container exiting with code 0) and then stop executing new Tasks.

- How I did it

  • Bumps swarm vendoring to the latest master, which includes all of the swarm-side jobs changes.
  • Adds two new service modes: ReplicatedJob and GlobalJob.
    • ReplicatedJob has two fields: MaxConcurrent, which sets the maximum number of tasks to run at the same time for the job, and TotalCompletions, which is the number of Completed tasks desired for the job. If MaxConcurrent is not set, it will default to 1. If TotalCompletions is not set, it will default to the value of MaxConcurrent
    • GlobalJob has no fields. Like a global service, global jobs are scheduled on every node in the cluster matching placement constraints.
  • Adds a new field to the Service object, present when the Service is a job: JobStatus, which includes a counter for running the same job multiple times.
  • Adds a new field to the Task object, present when it belongs to a job: JobIteration, which is identical to the value of JobStatus.JobIteration on the Service that spawned this Task
  • Updates swagger.yml with documentation for all new fields.
  • Updates version-history.md to include information about new jobs fields.

- How to verify it

Includes new integration tests for jobs functionality.

- Description for the changelog

Swarm Mode now supports running Jobs.

@dperny
Copy link
Contributor Author

dperny commented Dec 12, 2019

Mostly running this PR as WIP because my local dev box is giving me trouble, and I want to run against CI a bit to see if the problem is more general.

@dperny dperny force-pushed the swarm-jobs branch 3 times, most recently from fff938e to bb94bda Compare December 13, 2019 18:25
@dperny dperny mentioned this pull request Dec 13, 2019
Closed
@thaJeztah thaJeztah added area/swarm impact/api impact/changelog kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny status/2-code-review labels Dec 27, 2019
@dperny dperny force-pushed the swarm-jobs branch 5 times, most recently from c930baa to dbd0ef6 Compare January 8, 2020 16:36
@dperny
Copy link
Contributor Author

dperny commented Jan 9, 2020

It seems that in CI, the Jobs test takes too long to complete 20 iterations. I've cut down the total to 7, which is less than the minimum completion I saw, which was 8.

@thaJeztah
Copy link
Member

@derny this needs a rebase now

@dperny
Copy link
Contributor Author

dperny commented Jan 9, 2020

@thaJeztah Yes, it does, but in any case, the development on Jobs has happened all in a feature branch in Swarmkit, and that feature branch has not quite yet been merged into swarmkit master.

@dperny dperny mentioned this pull request Jan 10, 2020
Merged
@dperny dperny changed the title WIP: Add support for swarm jobs Add support for swarm jobs Jan 13, 2020
@dperny
Copy link
Contributor Author

dperny commented Jan 13, 2020

No longer WIP.

cpuguy83
cpuguy83 reviewed Jan 13, 2020
View reviewed changes
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when requested number of completions is hit and there are still others running?

daemon/cluster/convert/service.go Outdated Show resolved Hide resolved
api/types/swarm/service.go Show resolved Hide resolved
@dperny
Copy link
Contributor Author

dperny commented Jan 13, 2020

Swarmkit will only schedule the number of Tasks needed to reach TotalCompletions, so there won't be any Tasks still running when TotalCompletions is reached. MaxConcurrent establishes the upper bound on how many are scheduled.

@dperny
Add swarm jobs
30d9fe3 
Adds support for ReplicatedJob and GlobalJob service modes. These modes
allow running service which execute tasks that exit upon success,
instead of daemon-type tasks.

Signed-off-by: Drew Erny <[email protected]>
cpuguy83
cpuguy83 approved these changes Jan 14, 2020
View reviewed changes
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Nicely done!

@collabnix
Copy link

Would love to blog around this. Which Docker release is it coming with? Beta bits?

@dperny dperny mentioned this pull request Jan 16, 2020
Merged
@dperny
Copy link
Contributor Author

dperny commented Jan 16, 2020

CLI PR is open, pending the merge of this PR.

@cpuguy83 cpuguy83 mentioned this pull request Jan 17, 2020
Closed
@AkihiroSuda AkihiroSuda merged commit f9d136b into moby:master Jan 20, 2020
@arseniybanayev
Copy link

Amazing. Thank you for the work! When is this planned to be released?

@BretFisher
Copy link

👏 👏 💯

@crazy-max
Copy link
Member

Very nice, thanks for this! I'll soon be able to archive swarm-cronjob 😅

@tsbx
Copy link

tsbx commented Jan 20, 2020

@dperny Thanks for this new feature 👏
@crazy-max When reading the documentation, it seems different from your (crazy 😄 ) swarm-cronjob project: here there is a limited number of executions (TotalCompletions) and no cron equivalent functionality, so please don't archive it. It's still very useful ❤️

@AkihiroSuda AkihiroSuda mentioned this pull request Jan 21, 2020
Closed
@ivorscott
Copy link

@dperny you're a blessing to open source. thank you.

@djmaze djmaze mentioned this pull request Jan 31, 2020
Open
@alexanderadam alexanderadam mentioned this pull request Apr 2, 2020
Closed
@thaJeztah thaJeztah added this to the 20.03.0 milestone Apr 2, 2020
@djmaze djmaze mentioned this pull request Apr 11, 2020
Closed
@abellion abellion mentioned this pull request Jun 9, 2020
Closed
@Ohtar10
Copy link

Ohtar10 commented Oct 21, 2020 •
edited
Loading

This is really good stuff!

Question: Would it be possible to specify jobs via compose/stack file too? or this will only work via CLI?

Edit: Moving the question to docker/cli#2262 which is where it was being discussed.

@alexellis
Copy link
Contributor

Nice work! I would have loved this three years ago 😄 I've actually seen a rise in interest in alexellis/jaas so I think this will be well received by the community of users still on Swarm.

@Akasurde Akasurde mentioned this pull request Nov 1, 2020
Open
@ollypom ollypom mentioned this pull request Jan 4, 2021
Closed
@ghost ghost mentioned this pull request Jan 19, 2021
Open
@chris34 chris34 mentioned this pull request Feb 20, 2022
Closed
@miberecz miberecz mentioned this pull request Feb 20, 2023
Closed
@pinkfrog9
Copy link

Is it possible to create cron job with swarm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AkihiroSuda AkihiroSuda AkihiroSuda approved these changes

@cpuguy83 cpuguy83 cpuguy83 approved these changes

Assignees
No one assigned
Labels
area/swarm impact/api impact/changelog kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny status/2-code-review
Projects
None yet
Milestone
20.10.0
Development

Successfully merging this pull request may close these issues.