Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flag to automatically attempt to distribute parallel stages across nodes #5002

Closed
abayer opened this issue Aug 6, 2019 · 0 comments · Fixed by #5003
Closed

Add flag to automatically attempt to distribute parallel stages across nodes #5002

abayer opened this issue Aug 6, 2019 · 0 comments · Fixed by #5003
Assignees
Labels
area/tekton area/tekton indicates that the issue was reported against the Servereless Jenkins X Pipelines kind/enhancement An enhancement of an existing feature priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@abayer
Copy link
Contributor

abayer commented Aug 6, 2019

A problem we hit in Jenkins X's own production builds for the jenkins-x-builders repo is that Kaniko, particularly with very large images and layers, can cause problems with the Kubernetes node's container GC, particularly if there are multiple Kaniko pods running on the node at the same time. This is annoying. =)

Something that might help us with this - and that will hopefully be useful in other contexts - would be to put an affinity policy on the PipelineRun saying "if possible, put the pods for stages in this run on different nodes from each other". It wouldn't be perfect, but it would definitely help.

Now, this is not saying I'm going to add the ability to configure full affinity policies explicitly on the pipeline - first, the only way I can make this work requires using the PipelineRun name (to use in an anti-affinity policy that says "avoid nodes with a pod already on them with the label tekton.dev/pipelineRun=THIS_PIPELINERUN_NAME, and since the name is unique per run, you can't exactly hardcode the name into the jenkins-x.yml. Second, I'm not sure there's enough value in allowing full affinity policy configuration to be worth the added complexity. So I'm just going to add an option (say, distributeParallelAcrossNodes: true) that can be set on a pipeline's options (but not a stage, since we only have control over the affinity policy on the PipelineRun).

@abayer abayer added kind/enhancement An enhancement of an existing feature priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/tekton area/tekton indicates that the issue was reported against the Servereless Jenkins X Pipelines labels Aug 6, 2019
@abayer abayer self-assigned this Aug 6, 2019
abayer added a commit to abayer/jx that referenced this issue Aug 6, 2019
jenkins-x-bot pushed a commit that referenced this issue Aug 7, 2019
daveconde pushed a commit to daveconde/jx that referenced this issue Apr 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tekton area/tekton indicates that the issue was reported against the Servereless Jenkins X Pipelines kind/enhancement An enhancement of an existing feature priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant