fix(gateway): recover from stale PID file after SIGKILL/OOM - #29349
Closed
Tranquil-Flow wants to merge 1 commit into
Closed
fix(gateway): recover from stale PID file after SIGKILL/OOM#29349Tranquil-Flow wants to merge 1 commit into
Tranquil-Flow wants to merge 1 commit into
Conversation
…ousResearch#13655) After a SIGKILL or OOM kill, atexit handlers never fire and gateway.pid is left behind with a dead PID. The next startup hits FileExistsError from the O_CREAT|O_EXCL write and exits with 'PID file race lost', even though no other gateway is running. The FileExistsError handler now reads the existing PID file and checks whether the recorded process is still alive. If the PID is stale, the file is force-unlinked and the write is retried. Only if the PID is confirmed alive (or a second race is lost) does the gateway exit. Tests: 12 new tests covering record parsing, stale detection, recovery, and double-race scenarios.
alt-glitch
added
type/bug
Collaborator
Contributor
Author
|
Closing in favor of #25569 (by @zccyman) which fixes the stale PID issue at the source inside itself, rather than in the caller's FileExistsError handler. Comparison:
Both approaches preserve atomic O_CREAT|O_EXCL semantics for genuine concurrent races. #25569's approach is preferable because it makes write_pid_file() self-healing for all callers, not just start_gateway(). Thanks to alt-glitch for flagging the competing PRs (#25569, #9703, #14609). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closing in favor of #25569 (by @zccyman) which fixes the stale PID issue at the source inside itself, rather than in the caller's FileExistsError handler.
Comparison:
Both approaches preserve atomic O_CREAT|O_EXCL semantics for genuine concurrent races. #25569's approach is preferable because it makes self-healing for all callers, not just .
Thanks to alt-glitch for flagging the competing PRs (#25569, #9703, #14609).