Skip to content

Recover bounded golden route migrations - #160

Merged
lawrencecchen merged 2 commits into
mainfrom
fix/golden-propagation-recovery
Aug 5, 2026
Merged

lawrencecchen merged 2 commits into
mainfrom
fix/golden-propagation-recovery

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

The staging golden run reached its two-minute route-proof limit while existing Codex streams remained healthy. Google Cloud documents URL-map propagation as taking several minutes. Cleanup then received duplicate SIGTERM delivery and was interrupted before restoring the URL map and releasing the deploy lock.

This keeps migration proof bounded at five minutes, raises the correlated proof-attempt ceiling to use that window, and makes direct process signaling a fallback when process-group signaling fails.

Regression sequence:

  • eaa8967 changes boundary and cleanup tests; the old implementation rejects the valid 4:59 proof.
  • fddbe82 implements the bounded recovery.

Verified:

  • go test ./...
  • focused observer tests, 10 repetitions
  • focused observer race test
  • bash -n deploy/gcp/switch-front-migration.sh

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Extend the golden-route migration propagation window to 5 minutes and harden cleanup, preventing stuck deploys when URL-map propagation is slow or SIGTERM delivery is flaky.

  • Bug Fixes
    • Bump migration propagation limit from 2 to 5 minutes in the observer, deploy/gcp/deployment-contract.py, deploy/gcp/validate-deploy-evidence.py, and deploy/gcp/switch-front-migration.sh.
    • Raise destination proof max attempts from 16 to 64 to fit the longer window.
    • Improve termination: try process-group SIGTERM first, then fall back to direct process SIGTERM; add a test to verify graceful cleanup.
    • Update acceptance and deploy script tests to cover the new 5-minute boundary and cleanup behavior.

Written for commit fddbe82. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Improvements

    • Extended migration propagation and destination-proof acceptance windows from two to five minutes.
    • Increased destination-proof retry capacity from 16 to 64 attempts.
    • Improved process-group termination so graceful shutdown handlers can complete cleanup reliably.
  • Tests

    • Updated migration timing coverage for the five-minute boundaries.
    • Added validation for graceful process termination and delayed cleanup.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The migration acceptance window increased from two to five minutes, and destination-proof retries increased from 16 to 64. Unix process termination now falls back to individual signaling only when group signaling fails, with graceful cleanup coverage.

Changes

Migration timing and proof limits

Layer / File(s) Summary
Migration runtime limits
cmd/subrouter-transport-observer/golden.go, cmd/subrouter-transport-observer/golden_migration_transition.go, deploy/gcp/switch-front-migration.sh
Migration propagation limits now allow five minutes. Destination-proof retries now allow 64 attempts.
Migration validation boundaries
cmd/subrouter-transport-observer/golden_acceptance_regression_test.go, cmd/subrouter/deploy_scripts_test.go, deploy/gcp/deployment-contract.py, deploy/gcp/validate-deploy-evidence.py
Validation and regression tests accept timings below five minutes and reject the five-minute boundary.

Unix process termination

Layer / File(s) Summary
Graceful process termination
cmd/subrouter-transport-observer/golden_process_unix.go, cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go
terminateProcessGroup signals the individual process only when process-group signaling fails. A Unix test verifies graceful SIGTERM cleanup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: recovery of bounded golden route migrations.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/golden-propagation-recovery

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go`:
- Around line 102-104: Bound the cleanup wait after
terminateProcessGroup(command) so command.Wait() cannot block indefinitely. Wait
for the command to exit with a timeout, invoke killProcessGroup(command) when
the timeout expires, then fail the test; preserve the existing graceful-exit
failure handling for commands that return an error.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ada62be6-dae1-4bcf-8261-92b7f57d0997

📥 Commits

Reviewing files that changed from the base of the PR and between c428e78 and fddbe82.

📒 Files selected for processing (9)
  • cmd/subrouter-transport-observer/golden.go
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go
  • cmd/subrouter-transport-observer/golden_migration_transition.go
  • cmd/subrouter-transport-observer/golden_process_unix.go
  • cmd/subrouter/deploy_scripts_test.go
  • deploy/gcp/deployment-contract.py
  • deploy/gcp/switch-front-migration.sh
  • deploy/gcp/validate-deploy-evidence.py

Comment on lines +102 to +104
terminateProcessGroup(command)
if err := command.Wait(); err != nil {
t.Fatalf("cleanup helper did not exit gracefully: %v", err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the referenced test and helper.
fd -a 'golden_deployment_action_lifecycle_unix_test.go' . || true
file="$(fd 'golden_deployment_action_lifecycle_unix_test.go' . | head -n 1)"
if [ -z "${file:-}" ]; then
  echo "Referenced file not found"
  exit 0
fi

echo "== file =="
echo "$file"
echo "== outline =="
ast-grep outline "$file" || true
echo "== relevant lines =="
nl -ba "$file" | sed -n '1,150p'

echo "== terminateProcessGroup usages =="
rg -n "terminateProcessGroup|Wait\\(|signal\\(|kill\\(" "$file" .

Repository: manaflow-ai/subrouter

Length of output: 938


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="./cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go"
echo "== relevant lines 1-220 =="
awk '{printf "%6d\t%s\n", NR, $0}' "$file" | sed -n '1,220p'

echo "== helper function definitions and usages =="
awk '{printf "%6d\t%s\n", NR, $0}' "$file" | grep -nE 'func terminateProcessGroup|terminateProcessGroup\\(|\\.Wait\\(|\\.Start\\(|\\.StartProcess|process\\.|signal\\(|syscall\\.|Cleanup\\('

echo "== go files in package =="
find cmd/subrouter-transport-observer -maxdepth 2 -type f -name '*.go' -print | sort

Repository: manaflow-ai/subrouter

Length of output: 7284


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="./cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go"

awk '
  /^func (configure|kill|terminate)ProcessGroup/ {p=1}
  p {print sprintf("%6d\t%s", NR, $0)}
  p && /^[}]$/{p=0}
' "$file"

echo "== process helper calls =="
awk '{printf "%6d\t%s\n", NR, $0}' "$file" | grep -nE 'configureProcessGroup|killProcessGroup|terminateProcessGroup|command\\.Wait\\(|\\.Start\\('

echo "== files with process helpers =="
rg -n "func (configure|kill|terminate)ProcessGroup|configureProcessGroup|killProcessGroup|terminateProcessGroup|Wait\\(" cmd/subrouter-transport-observer -g '*.go'

Repository: manaflow-ai/subrouter

Length of output: 210


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="./cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go"

echo "== process helper slices =="
python3 - <<'PY'
from pathlib import Path
import re
p=Path("cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go")
text=p.read_text()
for name in ["configureProcessGroup","killProcessGroup","terminateProcessGroup"]:
    print(f"\n== func {name} ==")
    m=re.search(rf"(?ms)^func {name}\b.*?\n\}}", text)
    print(m.group(0) if m else "not found")
PY

echo "== relevant calls =="
python3 - <<'PY'
import re
from pathlib import Path
p=Path("cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go")
lines=p.read_text().splitlines()
for i,l in enumerate(lines,1):
    if any(s in l for s in ["configureProcessGroup","killProcessGroup","terminateProcessGroup","command.Wait",".Wait"]):
        print(f"{i}: {l}")
PY

Repository: manaflow-ai/subrouter

Length of output: 510


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== all process group symbols =="
rg -n "processGroup|ProcessGroup|TerminateProcessGroup|killProcessGroup|configureProcessGroup" . -g '*.go'

echo "== git status/stat =="
git diff --stat || true
git status --short || true

Repository: manaflow-ai/subrouter

Length of output: 3481


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== process helper definitions =="
awk '
  /^func (configureProcessGroup|interruptProcessGroup|terminateProcessgroup|killProcessGroup|terminateProcessGroup)\b/ {p=1}
  p {print sprintf("%6d\t%s", NR, $0)}
  p && /^[^{]*\}/ && NR>begin {p=0}
  {if (/^func /) begin=NR}
' cmd/subrouter-transport-observer/golden_process_unix.go cmd/subrouter-transport-observer/golden.go | sed -n '1,120p'

echo "== golden process_unix.go =="
cat -n cmd/subrouter-transport-observer/golden_process_unix.go

echo "== related waits in golden_process_unix.go candidates =="
rg -n "\bWait\(|killProcessGroup|terminateProcessGroup" cmd/subrouter-transport-observer/golden_process_unix.go cmd/subrouter-transport-observer/golden.go

Repository: manaflow-ai/subrouter

Length of output: 2213


Bound the process wait.

terminateProcessGroup(command) tries SIGTERM once, then falls back once. If both attempts fail, command.Wait() at Cmd.Wait() can block indefinitely and prevent the t.Cleanup at killProcessGroup(command) + command.Wait() from running. Wait with a timeout; kill the helper with killProcessGroup(command) if it does not exit; then fail the test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go`
around lines 102 - 104, Bound the cleanup wait after
terminateProcessGroup(command) so command.Wait() cannot block indefinitely. Wait
for the command to exit with a timeout, invoke killProcessGroup(command) when
the timeout expires, then fail the test; preserve the existing graceful-exit
failure handling for commands that return an error.

@lawrencecchen
lawrencecchen merged commit b0610ff into main Aug 5, 2026
7 checks passed
@lawrencecchen
lawrencecchen deleted the fix/golden-propagation-recovery branch August 5, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant