Skip to content

Make sure SIGTERM shutdown the controller manager and scheduler API server#22543

Merged
deads2k merged 5 commits intoopenshift:masterfrom
mfojtik:stop-gracefully
Apr 12, 2019
Merged

Make sure SIGTERM shutdown the controller manager and scheduler API server#22543
deads2k merged 5 commits intoopenshift:masterfrom
mfojtik:stop-gracefully

Conversation

@mfojtik
Copy link
Copy Markdown
Contributor

@mfojtik mfojtik commented Apr 11, 2019

Right know the upstream does not call the server.Shutdown() when the controller manager receives SIGTERM. The SIGTERM stopCh is not even wired into controller manager and scheduler.

In that case, when the controller manager or scheduler are terminated, their ports will keep open until kernel timeout is hit (~60s++) and the port is freed. Since for these two components we use host network, this is causing the operator observing container restart in pod and ultimately setting failure condition to true.

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Apr 11, 2019
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 11, 2019
@mfojtik mfojtik force-pushed the stop-gracefully branch 2 times, most recently from 7eaf8ca to add0bf5 Compare April 11, 2019 11:54
@mfojtik mfojtik changed the title UPSTREAM: 0000: close stop channel for kube controller manager Make sure SIGTERM shutdown the controller manager and scheduler API server Apr 11, 2019
@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 11, 2019
@mfojtik mfojtik force-pushed the stop-gracefully branch 2 times, most recently from c97e9bc to 8fd48ef Compare April 11, 2019 12:58
@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 11, 2019

/retest

2 similar comments
@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 11, 2019

/retest

@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 11, 2019

/retest

@mfojtik mfojtik changed the title Make sure SIGTERM shutdown the controller manager and scheduler API server WIP Make sure SIGTERM shutdown the controller manager and scheduler API server Apr 11, 2019
@mfojtik mfojtik changed the title WIP Make sure SIGTERM shutdown the controller manager and scheduler API server WIP: Make sure SIGTERM shutdown the controller manager and scheduler API server Apr 11, 2019
@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 11, 2019
@mfojtik mfojtik changed the title WIP: Make sure SIGTERM shutdown the controller manager and scheduler API server Make sure SIGTERM shutdown the controller manager and scheduler API server Apr 11, 2019
Comment thread test/util/server/server.go Outdated
@deads2k
Copy link
Copy Markdown
Contributor

deads2k commented Apr 11, 2019

Wow, this is a dense pull. I've had to go back to review other blockers. That's interesting

@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 11, 2019

/refresh

@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 12, 2019

/retest

@mfojtik mfojtik removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 12, 2019
@mfojtik
Copy link
Copy Markdown
Contributor Author

mfojtik commented Apr 12, 2019

/retest

@sttts
Copy link
Copy Markdown
Contributor

sttts commented Apr 12, 2019

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 12, 2019
@openshift-ci-robot
Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mfojtik, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@deads2k
Copy link
Copy Markdown
Contributor

deads2k commented Apr 12, 2019

flake fix, green merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants