Skip to content

Make hosted cutovers zero-gap - #186

Merged
lawrencecchen merged 35 commits into
mainfrom
fix/gcp-zero-gap-cutover
Aug 6, 2026
Merged

lawrencecchen merged 35 commits into
mainfrom
fix/gcp-zero-gap-cutover

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Aug 6, 2026 •

Copy link
Copy Markdown
Contributor

Transfers the live TCP listener from the legacy supervisor to the stable front with pidfd_getfd, preserving existing accepted connections while new connections move without a GCP route change.

The golden gate now proves the same kernel socket is active in the front, retires only the old listener descriptor, and requires held HTTP/WebSocket Codex sessions plus a fresh public Codex session to remain healthy. It retains the slot rehearsal, rollback, and final activation cycle.

Verification:

  • go test ./... -count=1
  • Linux amd64 static build
  • production-systemd listener takeover smoke on subrouter-staging
  • unpinned Codex 0.146.0 WebSocket turn through hosted staging
  • staging and production public health checks

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Enables zero-gap cutover by duplicating the live TCP listener from the legacy supervisor into the stable front, keeping all connections and avoiding any GCP route change. Adds durable ownership across restarts and safely retires ambiguous successor activations.

  • New Features

    • Resumable handoff via Unix --listener-transfer-socket; listener-transfer sends the exact listener FD, validates numeric addresses, and rejects shared or non-absolute sockets.
    • Persist and confirm the public listener in systemd FDSTORE keyed by its canonical numeric address; await MAINPID barrier; Linux-only SIGHUP reload; roll back if a confirmed successor exits; retire ambiguous successor activations gracefully.
    • Front hot reload with descriptor handoff and gating: gate ownership before controls, commit the control socket after promotion, and safely recover across restarts; supervisor supports --takeover-listener-pid and --takeover-listener-fd; added takeover/transfer tests and a production systemd smoke test.
  • Migration

    • Evidence now includes listener details, ActiveBackendURL, and SourceListenerRetiredAt; validators enforce a single final-cutover with a bounded window and require retirement before destination proof. Candidate bumped to v0.1.69; added golang.org/x/sys; golden runner supports --account-id.
    • Install front slots, set SUBROUTER_FRONT_ADDR and SUBROUTER_FRONT_TRANSFER_SOCKET. Run migrate-to-front-slots, then switch-front-migration --operation final-cutover for descriptor handoff. Deployment lock preserves only the committed legacy sampler when a front handoff checkpoint exists; finalize-legacy-retirement verifies front listener ownership and keeps the disabled legacy process alive until sessions drain.

Written for commit ac8e838. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features

    • Added seamless front-listener handoff, enabling service replacement without interrupting active connections.
    • Added listener takeover, restoration, status, and transfer commands.
    • Added configurable listener addresses and deployment support for ownership transitions.
    • Added OAuth account ID support with updated Spark model and stream-output defaults.
  • Improvements

    • Simplified migrations to a single, evidence-validated final cutover.
    • Improved listener continuity, connection draining, and legacy-service retirement validation.
    • Added resumable handoff checkpoints and stronger deployment safeguards.
    • Updated continuity tooling for release candidate v0.1.69.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds listener transfer and takeover support, updates deployment scripts for resumable listener handoff, and replaces rehearsal/rollback migration phases with one final cutover. Golden continuity now validates listener ownership evidence and supports OAuth account propagation with v0.1.69.

Changes

Listener handoff migration

Layer / File(s) Summary
Listener transfer runtime
cmd/subrouter/front.go, cmd/subrouter/listener_*.go, cmd/subrouter/front_fdstore_*.go, cmd/subrouter/front_handoff_*.go, cmd/subrouter/main.go, cmd/subrouter/supervisor.go, cmd/subrouter/*takeover*_test.go
The front service supports listener replacement, descriptor transfer, inherited listeners, systemd descriptor storage, hot reload, draining, and listener status reporting. Linux takeover uses pidfd descriptor duplication.
Deployment listener operations
deploy/gcp/install-front-slots.sh, deploy/gcp/smoke-listener-takeover.sh, deploy/gcp/switch-front-migration.sh, deploy/gcp/finalize-legacy-retirement.sh, deploy/gcp/deployment-contract.py, deploy/gcp/deploy-lock.sh, cmd/subrouter/deploy_scripts_test.go
Deployment helpers configure listener addresses, activate and restore takeover, persist resumable handoff checkpoints, validate ownership, preserve sampler state, and test cleanup behavior.
Final cutover evidence
cmd/subrouter-transport-observer/golden_migration_*.go, cmd/subrouter-transport-observer/golden.go, cmd/subrouter-transport-observer/testdata/golden_fake_client/main.go, cmd/subrouter-transport-observer/golden_*test.go, deploy/gcp/validate-deploy-evidence.py
Migration now performs one final cutover from preparation. Evidence records listener descriptors, inode identity, shared kernel sockets, source retirement timing, backend linkage, and post-handoff liveness.
Golden runtime configuration
cmd/subrouter-transport-observer/golden.go, cmd/subrouter-transport-observer/golden_options_test.go, deploy/gcp/golden-local-mac-production-continuity.sh, go.mod, cmd/subrouter/sr_test.go
Golden runs pin v0.1.69, use Spark defaults, validate and propagate OAuth account IDs, and make golang.org/x/sys a direct dependency.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant switch-front-migration.sh
  participant legacyService
  participant stableFront
  participant validate-deploy-evidence.py
  participant finalize-legacy-retirement.sh
  switch-front-migration.sh->>legacyService: capture listener PID, FD, and inode
  switch-front-migration.sh->>stableFront: activate listener-FD takeover
  stableFront->>switch-front-migration.sh: report inherited socket and preserved connections
  switch-front-migration.sh->>legacyService: retire source listener
  switch-front-migration.sh->>validate-deploy-evidence.py: emit final-cutover evidence
  validate-deploy-evidence.py->>finalize-legacy-retirement.sh: validate handoff and retirement evidence
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 1.16% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: preventing gaps during hosted listener cutovers.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gcp-zero-gap-cutover

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

🧹 Nitpick comments (7)
cmd/subrouter-transport-observer/golden_options_test.go (1)

39-47: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a negative case for v0.1.68.

This test accepts v0.1.69 and rejects v0.1.63. It does not prove that the previous v0.1.68 candidate is rejected. Add that case so the test enforces an exact release pin instead of only rejecting the bootstrap candidate.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter-transport-observer/golden_options_test.go` around lines 39 -
47, Add a negative test case in the existing golden-argument test around
parseGoldenArgs that replaces the accepted v0.1.69 candidate with v0.1.68 and
asserts parsing fails, matching the existing v0.1.63 rejection assertion so the
exact release pin is enforced.
cmd/subrouter/listener_transfer_unix.go (1)

53-58: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the no-op continue and log the failed transfer.

The continue at Line 56 is the last statement in the loop body, so it changes nothing. More important, requestErr is discarded without a log line. A rejected or failed descriptor transfer then leaves no trace on the front, which makes a failed cutover hard to diagnose. Log the error at warning level. Do not log the request body.

♻️ Proposed change
 		requestErr := f.handleListenerTransfer(connection)
 		_ = connection.Close()
 		if requestErr != nil {
-			continue
+			slog.Warn("front listener transfer failed", "error", requestErr)
 		}
 	}

Add "log/slog" to the imports.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/listener_transfer_unix.go` around lines 53 - 58, Update the
listener transfer loop around handleListenerTransfer to remove the no-op
continue and log any non-nil requestErr at warning level using slog, without
logging the request body. Add the required log/slog import and preserve
connection.Close behavior.
cmd/subrouter/listener_address.go (1)

19-29: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Simplify the port comparison and confirm the wildcard rule is intentional.

Two points in this function:

  1. Line 15 already rejects a port mismatch by string comparison. The integer re-parse at lines 26-28 can therefore never change the port verdict. Its only remaining effect is to reject non-numeric ports. Drop the re-parse and compare hosts directly.
  2. A wildcard expectedHost of "", 0.0.0.0, or :: matches any actual host. A listener bound to 127.0.0.1:31416 therefore satisfies an expected address of 0.0.0.0:31416. The takeover in cmd/subrouter/listener_takeover_linux.go Line 50 then reports success for a socket that is not publicly reachable. Confirm that this permissive direction is intended for the handoff validation.
♻️ Proposed simplification
 	actualIP, expectedIP := net.ParseIP(actualHost), net.ParseIP(expectedHost)
 	if actualIP != nil && expectedIP != nil {
 		return actualIP.Equal(expectedIP)
 	}
-	actualPortNumber, actualErr := strconv.Atoi(actualPort)
-	expectedPortNumber, expectedErr := strconv.Atoi(expectedPort)
-	return actualErr == nil && expectedErr == nil && actualPortNumber == expectedPortNumber &&
-		strings.EqualFold(actualHost, expectedHost)
+	return strings.EqualFold(actualHost, expectedHost)
 }

Remove the strconv import if you apply this change.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/listener_address.go` around lines 19 - 29, Update the listener
address comparison to remove the redundant strconv-based port parsing and
compare the already-validated port values directly, removing the unused strconv
import. Also review the wildcard handling in the listener address function and
confirm whether expectedHost values "", "0.0.0.0", and "::" should intentionally
match every actual host; if not, restrict the wildcard behavior so takeover
validation cannot accept a non-publicly reachable listener.
cmd/subrouter/main.go (1)

1364-1364: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider documenting the listener-transfer subcommand in the usage text.

runForProgram now dispatches listener-transfer, and deploy/gcp/install-front-slots.sh and deploy/gcp/smoke-listener-takeover.sh invoke it directly. The usage text documents only the front --listener-transfer-socket flag. Add one line so operators can discover the helper and its required flags.

📝 Proposed usage addition
   %[1]s front --backend-id ID --backend-address ADDRESS [--backend-network tcp|unix] [--addr 127.0.0.1:31415] [--control-socket /var/run/subrouter-front.sock] [--listener-transfer-socket /var/run/subrouter-front-listener.sock]
+  %[1]s listener-transfer --socket PATH --address ADDRESS --source-pid PID --source-fd FD
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/main.go` at line 1364, Update the usage text in runForProgram’s
help output to add a dedicated listener-transfer subcommand entry, documenting
its required flags, alongside the existing front usage line. Keep the existing
front documentation unchanged.
deploy/gcp/switch-front-migration.sh (1)

330-332: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The persistent sampler leaks if the run fails after the commit point.

cleanup stops ${persistent_legacy_sampler_unit} only when transition_committed is 0. This is intentional so finalize-legacy-retirement.sh can adopt it. If a later step of this script fails after Line 403, the unit and its sentinel stay on the instance with no adopter, and a subsequent run creates another unit. Consider stopping the sampler on a non-zero exit status even after the commit, or documenting the manual cleanup path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/gcp/switch-front-migration.sh` around lines 330 - 332, Update cleanup
in switch-front-migration.sh so a non-zero exit status also stops the persistent
legacy sampler after transition_committed is set, preventing orphaned units and
sentinels when post-commit steps fail. Preserve the existing adoption behavior
for successful committed runs and ensure the cleanup condition uses the script’s
status variable.
deploy/gcp/validate-deploy-evidence.py (1)

875-875: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the useless conditional.

Both branches return legacy_url, so the condition has no effect. Ruff reports RUF034. The value is correct for both cases: after listener-fd takeover the front destination still serves the legacy backend URL, and a rollback destination is the legacy backend. Simplify the expression and keep the intent in a comment.

♻️ Proposed refactor
-    expected_destination_url = legacy_url if destination_kind == "front" else legacy_url
+    # Listener-fd takeover keeps the legacy backend URL for both destinations.
+    expected_destination_url = legacy_url
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/gcp/validate-deploy-evidence.py` at line 875, Replace the conditional
assignment to expected_destination_url with a direct legacy_url assignment, and
add a concise comment preserving why both destination kinds use the legacy
backend URL after listener-fd takeover or rollback.

Source: Linters/SAST tools

cmd/subrouter-transport-observer/golden_acceptance_regression_test.go (1)

55-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test the listener-retirement-before-proof rule in isolation.

SourceListenerRetiredAt = "2026-08-02T00:00:01.1Z" is after ActivatedAt = "2026-08-02T00:00:01Z", so the generic timestamp ordering rejects this input before the listener_retired <= proof_received check runs. Set retirement after activation but before ReceivedAt, or add a second case that passes ordering and violates only that rule.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter-transport-observer/golden_acceptance_regression_test.go` around
lines 55 - 63, Update
TestGoldenMigrationRequiresListenerRetirementBeforeFreshPublicProof so
SourceListenerRetiredAt remains after ActivatedAt but occurs before ReceivedAt,
ensuring generic timestamp validation passes and only the
listener-retirement-before-proof rule causes rejection; alternatively add a
separate case with that isolated ordering.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/subrouter-transport-observer/golden_migration_evidence.go`:
- Around line 518-520: Add a lower-bound validation for listenerRetired in the
ordering condition around parseGoldenEvidenceTime: require
listenerRetired.Before(requested) alongside the existing upper-bound checks,
ensuring SourceListenerRetiredAt is not missing or earlier than
TransitionRequestedAt while preserving the other timestamp validations.

In `@cmd/subrouter-transport-observer/golden_options_test.go`:
- Line 27: Replace the personal-looking account identifier in the golden options
test fixture with the synthetic value test-account@example.com everywhere it
appears, including all arguments and assertions associated with the test.
- Around line 82-83: The failure in the golden child environment assertion must
not print the full environment. Update the check around
containsGoldenEnvironment to report only redacted evidence, such as whether the
expected SUBROUTER_CODEX_ACCOUNT_ID key is present, while preserving the
existing failure behavior.

In `@cmd/subrouter/front.go`:
- Around line 199-201: Replace context-free networking calls to satisfy noctx:
update openFreshPublicListener in cmd/subrouter/front.go:199-201 to use
net.ListenConfig.Listen with a context; update httptest.NewRequest at
cmd/subrouter/front_takeover_test.go:37-37 to its context-aware variant; and
replace net.DialTimeout at cmd/subrouter/front_takeover_test.go:90-90,
cmd/subrouter/front_takeover_test.go:182-182,
cmd/subrouter/listener_transfer_unix_test.go:78-78, and
cmd/subrouter/listener_transfer_unix_test.go:82-82 with DialContext using a
configured timeout. Share one dial helper between both test files, placing it in
a file compatible with the Unix-only build constraint, then run go test ./....
- Around line 209-216: Prevent shutdown from deadlocking in the listener
lifecycle around startServingLocked, runOnListeners, and listenerWG.Wait: ensure
retired listener goroutines cannot block when reporting to the bounded
f.listenerResults channel. Either make their result sends non-blocking or drain
f.listenerResults concurrently during shutdown while waiting for listenerWG,
preserving normal result processing.

In `@deploy/gcp/finalize-legacy-retirement.sh`:
- Around line 100-103: Validate accepting_new_public_false_at immediately after
extracting it, rejecting empty or literal "null" values with a clear error via
die before emitting retirement evidence. Keep the existing cutover_run_label
validation unchanged.
- Around line 158-176: Update adopt_legacy_sampler to assert that sampler_unit
is active before accepting sampler_result and sampler_oom_result. Record a
baseline sample count or result-file mtime at adoption, then update
stop_legacy_sampler to verify the corresponding value changed during the drain
window before trusting the metrics; retain the existing numeric validation and
sampler_adopted state handling.

In `@deploy/gcp/install-front-slots.sh`:
- Around line 385-393: Update the listener_status probes in
restore_front_bootstrap so each call runs in a subshell, containing the
die-induced exit and allowing the elif recovery branch to execute. Preserve the
existing checks for ports 31416 and 31415 and the replace-listener behavior.
- Around line 297-302: Declare python3 as a required dependency in both listener
scripts: in deploy/gcp/install-front-slots.sh lines 297-302, add a command check
beside the existing systemd-run check with the specified listener acceptance
probe error; in deploy/gcp/smoke-listener-takeover.sh lines 24-26, include
python3 in the command dependency list used by the script’s guard.
- Around line 360-371: Update the listener acceptance verification around the
existing acceptance loop to capture the pre-transfer
listener.accepted_connections value before systemd-run replaces/transfers the
listener, then require the post-transfer accepted count to be greater than that
baseline. Preserve the existing numeric validation, retry behavior, and failure
message while changing both the loop termination and final check from accepted >
0 to accepted > accepted_before.

In `@deploy/gcp/switch-front-migration.sh`:
- Around line 368-374: Strengthen the jq validation in the listener-status
parsing block near legacy_listener so .inode must match the safe expected inode
format, not merely have string type, before assigning source_listener_inode.
Keep the existing PID and FD numeric checks and service/port validation
unchanged, and ensure the validated inode can be safely interpolated into the
remote command used later.

---

Nitpick comments:
In `@cmd/subrouter-transport-observer/golden_acceptance_regression_test.go`:
- Around line 55-63: Update
TestGoldenMigrationRequiresListenerRetirementBeforeFreshPublicProof so
SourceListenerRetiredAt remains after ActivatedAt but occurs before ReceivedAt,
ensuring generic timestamp validation passes and only the
listener-retirement-before-proof rule causes rejection; alternatively add a
separate case with that isolated ordering.

In `@cmd/subrouter-transport-observer/golden_options_test.go`:
- Around line 39-47: Add a negative test case in the existing golden-argument
test around parseGoldenArgs that replaces the accepted v0.1.69 candidate with
v0.1.68 and asserts parsing fails, matching the existing v0.1.63 rejection
assertion so the exact release pin is enforced.

In `@cmd/subrouter/listener_address.go`:
- Around line 19-29: Update the listener address comparison to remove the
redundant strconv-based port parsing and compare the already-validated port
values directly, removing the unused strconv import. Also review the wildcard
handling in the listener address function and confirm whether expectedHost
values "", "0.0.0.0", and "::" should intentionally match every actual host; if
not, restrict the wildcard behavior so takeover validation cannot accept a
non-publicly reachable listener.

In `@cmd/subrouter/listener_transfer_unix.go`:
- Around line 53-58: Update the listener transfer loop around
handleListenerTransfer to remove the no-op continue and log any non-nil
requestErr at warning level using slog, without logging the request body. Add
the required log/slog import and preserve connection.Close behavior.

In `@cmd/subrouter/main.go`:
- Line 1364: Update the usage text in runForProgram’s help output to add a
dedicated listener-transfer subcommand entry, documenting its required flags,
alongside the existing front usage line. Keep the existing front documentation
unchanged.

In `@deploy/gcp/switch-front-migration.sh`:
- Around line 330-332: Update cleanup in switch-front-migration.sh so a non-zero
exit status also stops the persistent legacy sampler after transition_committed
is set, preventing orphaned units and sentinels when post-commit steps fail.
Preserve the existing adoption behavior for successful committed runs and ensure
the cleanup condition uses the script’s status variable.

In `@deploy/gcp/validate-deploy-evidence.py`:
- Line 875: Replace the conditional assignment to expected_destination_url with
a direct legacy_url assignment, and add a concise comment preserving why both
destination kinds use the legacy backend URL after listener-fd takeover or
rollback.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ef3632c9-a650-48e8-a6f8-7405e8357894

📥 Commits

Reviewing files that changed from the base of the PR and between 965a183 and 2ec7ee2.

📒 Files selected for processing (31)
  • cmd/subrouter-transport-observer/golden.go
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • cmd/subrouter-transport-observer/golden_counter_continuity.go
  • cmd/subrouter-transport-observer/golden_deployment_action_lifecycle_unix_test.go
  • cmd/subrouter-transport-observer/golden_local_mac_harness_test.go
  • cmd/subrouter-transport-observer/golden_migration_cycle.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter-transport-observer/golden_migration_transition.go
  • cmd/subrouter-transport-observer/golden_options_test.go
  • cmd/subrouter-transport-observer/golden_runtime_continuity_test.go
  • cmd/subrouter-transport-observer/testdata/golden_fake_client/main.go
  • cmd/subrouter/deploy_scripts_test.go
  • cmd/subrouter/front.go
  • cmd/subrouter/front_takeover_test.go
  • cmd/subrouter/listener_address.go
  • cmd/subrouter/listener_takeover_linux.go
  • cmd/subrouter/listener_takeover_other.go
  • cmd/subrouter/listener_takeover_test.go
  • cmd/subrouter/listener_transfer_other.go
  • cmd/subrouter/listener_transfer_unix.go
  • cmd/subrouter/listener_transfer_unix_test.go
  • cmd/subrouter/main.go
  • cmd/subrouter/supervisor.go
  • deploy/gcp/finalize-legacy-retirement.sh
  • deploy/gcp/golden-local-mac-production-continuity.sh
  • deploy/gcp/install-front-slots.sh
  • deploy/gcp/migrate-to-front-slots.sh
  • deploy/gcp/smoke-listener-takeover.sh
  • deploy/gcp/switch-front-migration.sh
  • deploy/gcp/validate-deploy-evidence.py
  • go.mod
💤 Files with no reviewable changes (1)
  • cmd/subrouter-transport-observer/golden_counter_continuity.go

Comment thread cmd/subrouter-transport-observer/golden_migration_evidence.go Outdated
Comment thread cmd/subrouter-transport-observer/golden_options_test.go Outdated
Comment thread cmd/subrouter-transport-observer/golden_options_test.go Outdated
Comment thread cmd/subrouter/front.go
Comment thread cmd/subrouter/front.go
Comment thread deploy/gcp/finalize-legacy-retirement.sh
Comment thread deploy/gcp/install-front-slots.sh
Comment thread deploy/gcp/install-front-slots.sh
Comment thread deploy/gcp/install-front-slots.sh Outdated
Comment thread deploy/gcp/switch-front-migration.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
deploy/gcp/deploy-lock.sh (1)

124-128: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider deriving the preserved sentinel set instead of listing targets.

The preserve branch removes only the front, slot-a, and slot-b markers. Any future sampler target gains an unremoved /tmp/subrouter-rss-${owner_cleanup_label}-*.running marker on lock release. Invert the rule and remove every marker except the legacy one, so new targets are cleaned by default.

♻️ Proposed refactor
-      owner_cleanup_command="trap \"if test -f ${owner_cleanup_preserve_path}; then rm -f -- /tmp/subrouter-rss-${owner_cleanup_label}-front.running /tmp/subrouter-rss-${owner_cleanup_label}-slot-a.running /tmp/subrouter-rss-${owner_cleanup_label}-slot-b.running; else rm -f -- /tmp/subrouter-rss-${owner_cleanup_label}-*.running; fi; rm -f -- /tmp/install-front-slots-${owner_cleanup_label}.sh /tmp/deployment-contract-${owner_cleanup_label}.py\" EXIT;"
+      owner_cleanup_command="trap \"for marker in /tmp/subrouter-rss-${owner_cleanup_label}-*.running; do test -e \\\"\\\${marker}\\\" || continue; if test -f ${owner_cleanup_preserve_path} && test \\\"\\\${marker}\\\" = /tmp/subrouter-rss-${owner_cleanup_label}-legacy.running; then continue; fi; rm -f -- \\\"\\\${marker}\\\"; done; rm -f -- /tmp/install-front-slots-${owner_cleanup_label}.sh /tmp/deployment-contract-${owner_cleanup_label}.py\" EXIT;"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/gcp/deploy-lock.sh` around lines 124 - 128, Update the preserve branch
of the owner_cleanup_command trap to remove all
/tmp/subrouter-rss-${owner_cleanup_label}-*.running markers except the legacy
preserved sentinel, rather than hard-coding front, slot-a, and slot-b. Keep
removal of the install and deployment contract files unchanged, and leave the
non-preserve branch behavior intact.
cmd/subrouter/deploy_scripts_test.go (1)

1146-1160: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The fake gcloud reimplements the preserve rule, so the real trap body is not covered.

Lines 1149-1156 decide which sentinels to remove inside the test double. The test therefore proves that subrouter_acquire_deploy_lock forwards the preserve path, not that the trap emitted by deploy-lock.sh removes the correct markers. If the trap body drops slot-a or preserves the wrong marker, this test still passes.

Assert on the emitted remote command, or execute the trap body from remote_command directly.

💚 Proposed change to run the real trap body
 remote_command="$*"
 cleanup() {
-  if [[ "${remote_command}" == *"test -f ${REMOTE_PRESERVE_MARKER}"* && -f "${REMOTE_PRESERVE_MARKER}" ]]; then
-    unlink "${REMOTE_FRONT_SENTINEL}" 2>/dev/null || true
-  else
-    unlink "${REMOTE_FRONT_SENTINEL}" 2>/dev/null || true
-    unlink "${REMOTE_LEGACY_SENTINEL}" 2>/dev/null || true
-  fi
+  # Execute the trap body that deploy-lock.sh generated instead of restating it.
+  trap_body="${remote_command#*bash -c \'trap \"}"
+  trap_body="${trap_body%%\" EXIT;*}"
+  eval "${trap_body//sudo /}"
 }

The rewritten double still needs the sentinel paths to point at the temporary directory. Adjust the generated marker prefix in the harness if you take this direction.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/deploy_scripts_test.go` around lines 1146 - 1160, Update the
fake gcloud harness in writeExecutableTestFile so it no longer reimplements the
preserve-rule cleanup logic. Instead, execute or otherwise assert against the
trap body contained in remote_command, ensuring the real deploy-lock.sh trap
removes and preserves the correct sentinels; when executing it, rewrite the
generated marker prefix so sentinel paths resolve within the temporary test
directory.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/gcp/finalize-legacy-retirement.sh`:
- Line 241: Before constructing the handoff_checkpoint command, extract
`.front.slot` from CUTOVER_EVIDENCE into a local variable and validate that it
is present, non-null, and matches the expected slot format. Use only the
validated variable when invoking gcloud_ssh, preserving the existing
validate-front-handoff-checkpoint arguments and preventing untrusted text from
being interpolated into the remote shell command.

In `@deploy/gcp/switch-front-migration.sh`:
- Around line 495-500: Update the listener validation in the legacy listener
ownership check before the source_listener_inode assignment to require the
inode’s expected safe format, not merely a JSON string type. Keep the existing
PID and FD validations and ensure only values safe for interpolation into the
single-quoted remote command reach source_listener_inode.

---

Nitpick comments:
In `@cmd/subrouter/deploy_scripts_test.go`:
- Around line 1146-1160: Update the fake gcloud harness in
writeExecutableTestFile so it no longer reimplements the preserve-rule cleanup
logic. Instead, execute or otherwise assert against the trap body contained in
remote_command, ensuring the real deploy-lock.sh trap removes and preserves the
correct sentinels; when executing it, rewrite the generated marker prefix so
sentinel paths resolve within the temporary test directory.

In `@deploy/gcp/deploy-lock.sh`:
- Around line 124-128: Update the preserve branch of the owner_cleanup_command
trap to remove all /tmp/subrouter-rss-${owner_cleanup_label}-*.running markers
except the legacy preserved sentinel, rather than hard-coding front, slot-a, and
slot-b. Keep removal of the install and deployment contract files unchanged, and
leave the non-preserve branch behavior intact.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: de277976-f623-4c27-b6d7-7533dbf49ba8

📥 Commits

Reviewing files that changed from the base of the PR and between 2ec7ee2 and f120780.

📒 Files selected for processing (13)
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter-transport-observer/testdata/golden_fake_client/main.go
  • cmd/subrouter/deploy_scripts_test.go
  • cmd/subrouter/front.go
  • cmd/subrouter/front_takeover_test.go
  • cmd/subrouter/listener_address.go
  • deploy/gcp/deploy-lock.sh
  • deploy/gcp/deployment-contract.py
  • deploy/gcp/finalize-legacy-retirement.sh
  • deploy/gcp/install-front-slots.sh
  • deploy/gcp/switch-front-migration.sh
  • deploy/gcp/validate-deploy-evidence.py
🚧 Files skipped from review as they are similar to previous changes (7)
  • cmd/subrouter/listener_address.go
  • cmd/subrouter-transport-observer/testdata/golden_fake_client/main.go
  • deploy/gcp/validate-deploy-evidence.py
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter/front.go
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • deploy/gcp/install-front-slots.sh

Comment thread deploy/gcp/finalize-legacy-retirement.sh Outdated
Comment thread deploy/gcp/switch-front-migration.sh

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (3)
cmd/subrouter/front_fdstore_linux.go (2)

49-55: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Rename port to slot.

frontListenerDescriptorStoreSlot returns a full slot key that encodes family, address, and port. The name port describes only part of the value.

♻️ Proposed rename
 func frontListenerStoreName(address net.Addr) (string, error) {
-	port, err := frontListenerDescriptorStoreSlot(address)
+	slot, err := frontListenerDescriptorStoreSlot(address)
 	if err != nil {
 		return "", fmt.Errorf("front listener address for descriptor store is invalid: %w", err)
 	}
-	return "subrouter-front-listener-" + port, nil
+	return "subrouter-front-listener-" + slot, nil
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/front_fdstore_linux.go` around lines 49 - 55, Rename the local
variable port to slot in frontListenerStoreName, including its declaration,
error-path context if applicable, and concatenation when constructing the
returned store name; preserve the existing behavior and use of
frontListenerDescriptorStoreSlot.

30-37: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Collapse the dead notified branch.

Both branches return nil, so the notified check has no effect. storeFrontListener ignores the bool result. Return the error directly.

♻️ Proposed simplification
-	notified, err := notifySystemdDescriptorStore("FDSTORE=1\nFDNAME="+name, file)
-	if err != nil {
-		return err
-	}
-	if !notified {
-		return nil
-	}
-	return nil
+	_, err = notifySystemdDescriptorStore("FDSTORE=1\nFDNAME="+name, file)
+	return err
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/front_fdstore_linux.go` around lines 30 - 37, Update
storeFrontListener to return the result of notifySystemdDescriptorStore
directly, removing the unused notified variable and redundant conditional while
preserving the existing error propagation.
cmd/subrouter/front_fdstore_linux_test.go (1)

81-84: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the FDNAME value, not only the verb.

The test checks FDSTORE=1 and FDSTOREREMOVE=1. It does not check FDNAME. FDSTOREREMOVE removes a stored descriptor by name, so a wrong or mismatched FDNAME would silently break retention while this test still passes. Compare both messages against frontListenerStoreName(listener.Addr()).

💚 Proposed assertion
+	wantName, err := frontListenerStoreName(listener.Addr())
+	if err != nil {
+		t.Fatal(err)
+	}
 	storeMessage, storeDescriptors := readNotification()
-	if !strings.Contains(storeMessage, "FDSTORE=1") || len(storeDescriptors) != 1 {
+	if !strings.Contains(storeMessage, "FDSTORE=1") ||
+		!strings.Contains(storeMessage, "FDNAME="+wantName) || len(storeDescriptors) != 1 {
 		t.Fatalf("store notification = %q with %d descriptors", storeMessage, len(storeDescriptors))
 	}
 	removeMessage, removeDescriptors := readNotification()
-	if !strings.Contains(removeMessage, "FDSTOREREMOVE=1") || len(removeDescriptors) != 0 {
+	if !strings.Contains(removeMessage, "FDSTOREREMOVE=1") ||
+		!strings.Contains(removeMessage, "FDNAME="+wantName) || len(removeDescriptors) != 0 {
 		t.Fatalf("remove notification = %q with descriptors %v", removeMessage, removeDescriptors)
 	}

As per coding guidelines: "Use go test ./... before handing off changes."

Also applies to: 106-109

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/subrouter/front_fdstore_linux_test.go` around lines 81 - 84, Update the
assertions in the FDSTORE and FDSTOREREMOVE notification checks to validate that
each message contains the expected FDNAME value from
frontListenerStoreName(listener.Addr()), in addition to the existing verb
checks. Ensure the test verifies both notifications use the same
listener-derived name, then run go test ./... before handoff.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/subrouter/front_handoff_unix.go`:
- Around line 355-361: Update promoteFrontSuccessor to inspect the boolean
result from notifyFrontMainPID and return an error when notification was not
sent, including when NOTIFY_SOCKET is unset. Preserve no-socket success only if
an explicitly supported non-systemd hot-reload path requires it; otherwise
ensure handoffToSuccessor cannot mark reload complete and exit unsupervised.

In `@cmd/subrouter/front_notify_other.go`:
- Around line 1-7: Update the build constraint above notifyFrontMainPID in
cmd/subrouter/front_notify_other.go (lines 1-7) to !linux so the no-op
implementation covers every non-Linux platform. In
cmd/subrouter/front_handoff_other.go (lines 1-20), confirm the fallback compiles
and preserve promoteFrontSuccessor’s explicit unsupported error without
delegating to the no-op notifier.

In `@deploy/gcp/smoke-listener-takeover.sh`:
- Around line 26-28: Update the port validation block in
smoke-listener-takeover.sh so backend_port, test_port, and bootstrap_port are
each validated as numeric and within the valid 0–65535 range before use. Replace
the incorrect bootstrap_port-only arithmetic check with per-variable range
checks, and make each error message identify the port variable being rejected.

---

Nitpick comments:
In `@cmd/subrouter/front_fdstore_linux_test.go`:
- Around line 81-84: Update the assertions in the FDSTORE and FDSTOREREMOVE
notification checks to validate that each message contains the expected FDNAME
value from frontListenerStoreName(listener.Addr()), in addition to the existing
verb checks. Ensure the test verifies both notifications use the same
listener-derived name, then run go test ./... before handoff.

In `@cmd/subrouter/front_fdstore_linux.go`:
- Around line 49-55: Rename the local variable port to slot in
frontListenerStoreName, including its declaration, error-path context if
applicable, and concatenation when constructing the returned store name;
preserve the existing behavior and use of frontListenerDescriptorStoreSlot.
- Around line 30-37: Update storeFrontListener to return the result of
notifySystemdDescriptorStore directly, removing the unused notified variable and
redundant conditional while preserving the existing error propagation.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 52756c29-4034-4de0-ab96-c1461bb5f596

📥 Commits

Reviewing files that changed from the base of the PR and between f120780 and 3783fd2.

📒 Files selected for processing (22)
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter-transport-observer/golden_options_test.go
  • cmd/subrouter/deploy_scripts_test.go
  • cmd/subrouter/front.go
  • cmd/subrouter/front_fdstore_linux.go
  • cmd/subrouter/front_fdstore_linux_test.go
  • cmd/subrouter/front_fdstore_other.go
  • cmd/subrouter/front_handoff_other.go
  • cmd/subrouter/front_handoff_unix.go
  • cmd/subrouter/front_notify_linux.go
  • cmd/subrouter/front_notify_other.go
  • cmd/subrouter/front_takeover_test.go
  • cmd/subrouter/listener_address.go
  • cmd/subrouter/listener_transfer_unix.go
  • cmd/subrouter/listener_transfer_unix_test.go
  • cmd/subrouter/sr_test.go
  • deploy/gcp/deployment-contract.py
  • deploy/gcp/finalize-legacy-retirement.sh
  • deploy/gcp/install-front-slots.sh
  • deploy/gcp/smoke-listener-takeover.sh
  • deploy/gcp/switch-front-migration.sh
🚧 Files skipped from review as they are similar to previous changes (9)
  • cmd/subrouter/deploy_scripts_test.go
  • deploy/gcp/deployment-contract.py
  • deploy/gcp/finalize-legacy-retirement.sh
  • cmd/subrouter/listener_transfer_unix.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter-transport-observer/golden_options_test.go
  • deploy/gcp/install-front-slots.sh
  • deploy/gcp/switch-front-migration.sh
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go

Comment thread cmd/subrouter/front_handoff_unix.go
Comment thread cmd/subrouter/front_notify_other.go Outdated
Comment thread deploy/gcp/smoke-listener-takeover.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/subrouter-transport-observer/golden_acceptance_regression_test.go`:
- Around line 90-92: Update the validator checks in the relevant test around
validGoldenMigrationTransitionEvidence to first run the unmodified fixture and
fail unless the Python validator exits successfully. Then run the mutated
fixture and require a nonzero validator result, while distinguishing execution
failures from expected validation rejection so missing, invalid, or crashing
validators cannot make the test pass.

In `@cmd/subrouter/front_notify_linux.go`:
- Around line 6-10: Update the MAINPID notification flow around
notifySystemdDescriptorStore and notifySystemdBarrier so a barrier timeout or
unexpected payload remains advisory after the descriptor send succeeds. Prevent
promoteFrontSuccessor from failing before ownership confirmation due solely to
notifyFrontMainPID’s barrier error, or restore parent ownership in the rollback
path before propagating it.

In `@cmd/subrouter/front_takeover_test.go`:
- Around line 485-504: Update the test around the promoteSuccessor callback and
promotions assertion to use an atomic counter, loading it atomically from the
test goroutine. After waiting for retired, poll until the counter reaches 2
(successor promotion and parent restoration) with a timeout, rather than reading
promotions directly; include sync/atomic and preserve the existing failure
behavior. Run go test ./... before handing off.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a98d2141-fe7d-4f35-a0db-e46a6fe8a327

📥 Commits

Reviewing files that changed from the base of the PR and between 3783fd2 and 7bc0496.

📒 Files selected for processing (13)
  • cmd/subrouter-transport-observer/golden_acceptance_regression_test.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter/front.go
  • cmd/subrouter/front_fdstore_linux.go
  • cmd/subrouter/front_fdstore_linux_test.go
  • cmd/subrouter/front_handoff_other.go
  • cmd/subrouter/front_handoff_unix.go
  • cmd/subrouter/front_notify_linux.go
  • cmd/subrouter/front_takeover_test.go
  • cmd/subrouter/listener_transfer_unix_test.go
  • deploy/gcp/install-front-slots.sh
  • deploy/gcp/smoke-listener-takeover.sh
  • deploy/gcp/validate-deploy-evidence.py
🚧 Files skipped from review as they are similar to previous changes (7)
  • deploy/gcp/validate-deploy-evidence.py
  • cmd/subrouter/front_handoff_other.go
  • deploy/gcp/smoke-listener-takeover.sh
  • cmd/subrouter/front_handoff_unix.go
  • cmd/subrouter-transport-observer/golden_migration_evidence.go
  • cmd/subrouter/front.go
  • deploy/gcp/install-front-slots.sh

Comment thread cmd/subrouter-transport-observer/golden_acceptance_regression_test.go Outdated
Comment thread cmd/subrouter/front_notify_linux.go
Comment thread cmd/subrouter/front_takeover_test.go
@lawrencecchen
lawrencecchen merged commit 17eb401 into main Aug 6, 2026
6 of 7 checks passed
@lawrencecchen
lawrencecchen deleted the fix/gcp-zero-gap-cutover branch August 6, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant