Repository navigation
fix(deploy): recover webhook rollouts from checkout drift #250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
aac28f9
fix(deploy): enforce archive-reset checkout hygiene
LucasSantana-Dev 2357f5f
fix(deploy): address CodeRabbit review — guard repoRoot spread and ro…
LucasSantana-Dev 755ce63
fix(deploy): enable webhook output capture in hooks.json
LucasSantana-Dev a5be450
chore: sync CHANGELOG and README from main to resolve merge conflict
LucasSantana-Dev bb047fd
chore: sync AGENTS.md from main to resolve conflict
LucasSantana-Dev 58b5f16
chore: sync CHANGELOG and README from main to resolve merge conflict
LucasSantana-Dev 5aa2dd4
chore: sync CHANGELOG and README from main to resolve merge conflict
LucasSantana-Dev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| name: lucky-deploy-recovery | ||
| description: Use when Lucky Deploy to Homelab fails due webhook lock contention, dirty checkout drift, or runtime precheck failures. | ||
| --- | ||
|
|
||
| # Lucky Deploy Recovery | ||
|
|
||
| ## When to use | ||
|
|
||
| - `Deploy to Homelab` is red on `main` | ||
| - Webhook response returns `another deploy is already running` | ||
| - Webhook response mentions local changes, overwritten merge files, or unmerged files | ||
| - Auth-config smoke never starts because webhook trigger failed | ||
|
|
||
| ## Failure buckets | ||
|
|
||
| - `LOCK_CONTENTION`: concurrent webhook deploy already running | ||
| - `CHECKOUT_RECOVERY_FAILED`: target checkout drift or merge-conflict state | ||
| - `RUNTIME_PRECHECK_FAILED`: compose/migration/health precheck failed after webhook accepted | ||
|
|
||
| ## Triage sequence | ||
|
|
||
| 1. Confirm latest failing run and failed step log: | ||
|
|
||
| ```bash | ||
| gh run list --branch main --workflow "Deploy to Homelab" --limit 5 | ||
| gh run view <RUN_ID> --log-failed | ||
| ``` | ||
|
|
||
| 2. Validate target host checkout state: | ||
|
|
||
| ```bash | ||
| ssh server-do-luk 'cd /home/luk-server/Lucky && git status --short --branch' | ||
| ``` | ||
|
|
||
| 3. If dirty checkout is present, archive drift and clean checkout: | ||
|
|
||
| ```bash | ||
| ssh server-do-luk 'cd /home/luk-server/Lucky && git stash push -u -m "manual-deploy-unblock-$(date -u +%Y%m%dT%H%M%SZ)"' | ||
| ssh server-do-luk 'cd /home/luk-server/Lucky && git fetch origin main && git reset --hard origin/main && git clean -fd' | ||
| ``` | ||
|
|
||
| 4. Rerun deploy workflow: | ||
|
|
||
| ```bash | ||
| gh run rerun <RUN_ID> | ||
| gh run watch <RUN_ID> --exit-status | ||
| ``` | ||
|
|
||
| 5. Post-deploy smoke: | ||
|
|
||
| ```bash | ||
| curl -i https://lucky-api.lucassantana.tech/api/health | ||
| curl -i https://lucky-api.lucassantana.tech/api/health/auth-config | ||
| curl -i https://lucky-api.lucassantana.tech/api/auth/discord | ||
| ``` | ||
|
|
||
| ## Rerun policy | ||
|
|
||
| - One immediate rerun is allowed for `LOCK_CONTENTION`. | ||
| - If rerun still fails with `CHECKOUT_RECOVERY_FAILED`, perform host checkout cleanup before another rerun. | ||
| - Do not keep blind rerunning on repeated `CHECKOUT_RECOVERY_FAILED`; require host cleanup evidence first. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.