Repository navigation
fix(serve): Add certified session writer handoff #7976
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
45e0382
fix(core): add certified session writer handoff
doudouOUC 257f124
fix(core): retain writer claim after failed rollback
doudouOUC dec0c56
fix(core): close session handoff claim races
doudouOUC 34e884e
fix(core): bound session handoff candidate waits
doudouOUC 4aa1abe
fix(core): reconcile session handoff proof edges
doudouOUC 577e7ca
Merge branch 'main' into agent/7752-certified-writer-handoff
qwen-code-dev-bot dd21ab4
Merge branch 'main' into agent/7752-certified-writer-handoff
wenshao 5802039
Merge branch 'main' into agent/7752-certified-writer-handoff
qwen-code-dev-bot 5fa5c35
fix(core): preserve inspect error cause and document handoff fail-clo…
qwen-code-dev-bot 0c20b1c
Merge branch 'main' into agent/7752-certified-writer-handoff
qwen-code-dev-bot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,211 @@ | ||
| # Certified session writer handoff | ||
|
|
||
| ## Problem | ||
|
|
||
| Cooperative managed shutdown currently releases each session writer lock before | ||
| the ACP child exits. That fixes the ordinary replacement path, but it cannot | ||
| distinguish a writer that deliberately stopped recording from a foreign-host | ||
| writer that disappeared without releasing its active lock. Treating hostname, | ||
| PID visibility, lock age, or transcript inactivity as proof of death would | ||
| allow two live Pods to append to the same transcript. | ||
|
|
||
| ## Scope | ||
|
|
||
| This change adds an integrity-protected handoff state for managed ACP writers. | ||
| After closing admission and durably draining accepted recorder work, a trusted | ||
| managed child can replace its active lock with a sealed record. A trusted | ||
| managed replacement can take ownership only after validating that record | ||
| against the exact transcript requested by the new Config. | ||
|
|
||
| Transcript paths must be absent or resolve to the same regular file opened for | ||
| the proof. A dangling symlink is not treated as an absent transcript. | ||
|
|
||
| The protocol remains gated by `experimental.sessionWriterLease`, which is | ||
| disabled by default and snapshotted at ACP process startup. Standalone ACP, | ||
| interactive, and headless recorders do not gain certified takeover. Normal | ||
| per-session close still releases its lock instead of leaving a sealed record. | ||
|
|
||
| This change does not reclaim an active lock left by SIGKILL, an event-loop | ||
| stall, an unhandled crash, or a storage failure before sealing completes. It | ||
| does not add TTLs, heartbeats, hostname stealing, Kubernetes API lookups, | ||
| operator force-steal endpoints, maintenance leases, or mixed-version support. | ||
| Those active-lock cases still require an authoritative external writer fence | ||
| and explicit recovery. | ||
|
|
||
| ## Lock records | ||
|
|
||
| New owners write schema v2 records. An active record retains the existing | ||
| immutable owner diagnostics and adds: | ||
|
|
||
| ```json | ||
| { | ||
| "schema_version": 2, | ||
| "state": "active" | ||
| } | ||
| ``` | ||
|
|
||
| A sealed record retains the previous owner's diagnostics and adds: | ||
|
|
||
| ```json | ||
| { | ||
| "schema_version": 2, | ||
| "state": "sealed", | ||
| "sealed_at": "2026-07-28T00:00:00.000Z", | ||
| "transcript": { | ||
| "relative_path": "<runtime-relative transcript key>", | ||
| "exists": true, | ||
| "byte_length": 1234, | ||
| "sha256": "<lowercase hex digest>" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The relative key must resolve to the transcript path already supplied by the | ||
| new Config. It is never used to select an arbitrary filesystem path. Schema v1 | ||
| records remain valid active records for compatibility during rollback, but | ||
| they can never be interpreted as sealed. | ||
|
|
||
| ## Fixed claim | ||
|
|
||
| The fixed claim path is: | ||
|
|
||
| ```text | ||
| <primary lock path>.claim | ||
| ``` | ||
|
|
||
| It serializes the two transitions that temporarily remove the primary path: | ||
| active-to-sealed and sealed-to-active. The claim is created with the existing | ||
| write-sync-and-hard-link primitive. Ambiguous link errors are reconciled | ||
| against the exact claim bytes before the transition continues or cleans up. A | ||
| claim is never reclaimed by PID, hostname, or age. Any pre-existing claim | ||
| returns `session_writer_unavailable`; manual cleanup is allowed only after an | ||
| authoritative external writer fence. | ||
|
|
||
| Ordinary acquisition checks the claim before every attempt to install a | ||
| missing primary lock. It checks again after installation and removes only its | ||
| own exact candidate if a concurrent transition acquired the claim. This keeps | ||
| the current active-lock fast path and local stale-owner recovery while making | ||
| both paths respect a handoff transition. Mixed-version writers remain | ||
| unsupported because an older writer does not know about the fixed claim. | ||
|
|
||
| An acquirer can pass its first claim check immediately before a transition | ||
| creates the claim, then install its active candidate during the transition's | ||
| primary-path gap. The transition recognizes that same-session schema-v2 active | ||
| record as a claim-aware candidate, preserves the retired predecessor, and | ||
| waits for the candidate's mandatory second check to remove it before retrying | ||
| the hard link. This wait is bounded; if the candidate stalls or exits before | ||
| its second check, the transition fails unavailable while retaining its claim | ||
| and retired predecessor. Unknown, malformed, or cross-session successors are | ||
| never removed or overwritten. | ||
|
|
||
| ## Sealing | ||
|
|
||
| Managed shutdown synchronously stops session and recorder admission, then | ||
| starts all writer terminals in parallel. A writer terminal: | ||
|
|
||
| 1. drains every recorder operation accepted before the cutoff; | ||
| 2. opens the expected transcript without following a symlink, verifies the | ||
| existing owner and transcript snapshot, and hashes the bytes through that | ||
| kept-open file descriptor; | ||
| 3. writes and syncs an owner-unique sealed candidate; | ||
| 4. acquires the fixed claim and revalidates the active owner plus the kept-open | ||
| transcript identity and metadata; | ||
| 5. renames the exact active primary to an owner-unique retired path; | ||
| 6. hard-links the sealed candidate into the now-missing primary path without | ||
| replacement; and | ||
| 7. removes only its exact retired, candidate, and claim records. | ||
|
|
||
| The primary-path transition is logically atomic for cooperating writers | ||
| because every installation respects the fixed claim, and the final hard link | ||
| cannot overwrite a lock created by another process. An error-after-effect is | ||
| reconciled from exact record bytes. The old owner never deletes or overwrites | ||
| an unknown primary. | ||
|
|
||
| A managed flush or proof failure retains the active lock. Normal per-session | ||
| close preserves the existing release behavior, including exact-owner cleanup. | ||
| If a normal release races managed shutdown and commits first, the missing | ||
| primary is already a safe handoff and the replacement performs ordinary | ||
| acquisition. | ||
|
|
||
| Failure cleanup removes the fixed claim only after proving that the exact | ||
| pre-transition primary was restored. If rollback cannot restore or verify that | ||
| record, the claim remains even when another primary appears, because that path | ||
| may be a losing ordinary-acquisition candidate that will remove itself after | ||
| observing the claim. Rollback itself is attempted only while the fixed claim | ||
| still contains this transition's exact record; a missing or replaced claim | ||
| means the current primary must not be changed. Recovery then requires the same | ||
| authoritative external writer fence as any other residual claim. | ||
|
|
||
| Failures before the primary transition starts are different: the claimant has | ||
| not created a primary-path gap or moved the predecessor, so it releases only | ||
| its own exact claim even if another certified contender already replaced the | ||
| observed sealed primary. This prevents a delayed losing contender from | ||
| stranding a claim after the winner becomes active. | ||
|
|
||
| ## Certified takeover | ||
|
|
||
| Only a Config created under a trusted managed parent enables certified | ||
| takeover. When acquisition observes a sealed record, it: | ||
|
|
||
| 1. verifies that the record's relative key matches the Config's expected | ||
| transcript; | ||
| 2. opens and hashes that transcript outside the fixed claim, retaining the | ||
| file descriptor and its identity; | ||
| 3. acquires the fixed claim; | ||
| 4. re-reads the exact sealed primary and revalidates the kept-open descriptor, | ||
| path identity, metadata, byte length, and digest; | ||
| 5. renames the sealed primary to a candidate-unique retired path; | ||
| 6. hard-links the synced active candidate from the claim into the primary | ||
| path without replacement; and | ||
| 7. removes only the exact retired record and its own claim. | ||
|
|
||
| The lease then performs the existing authoritative session reload and | ||
| transcript fence before recorder activation. Two replacements racing the same | ||
| sealed record can produce at most one active owner. A loser receives a | ||
| conflict or unavailable result depending on whether it observes the winner's | ||
| active primary or an in-progress/residual claim. | ||
|
|
||
| ## Failure contract | ||
|
|
||
| | Condition | Result | | ||
| | ---------------------------------------------------------------------------------- | ---------------------------------- | | ||
| | Valid active owner, including a foreign-host or managed dead-PID record | `session_writer_conflict` / 409 | | ||
| | Sealed proof does not match the expected transcript | `session_transcript_changed` / 409 | | ||
| | Malformed record, non-regular path, residual claim, or uncertain filesystem result | `session_writer_unavailable` / 503 | | ||
| | Current writer no longer owns its exact active record | `session_writer_lost` / 409 | | ||
|
|
||
| Public errors remain sanitized. Successful seal and takeover logs include the | ||
| session ID, previous hostname/PID, and sealing time, but never the owner token | ||
| or transcript path. | ||
|
|
||
| ## Compatibility and rollout | ||
|
|
||
| The feature gate must remain disabled during a mixed-version rollout. Enabling | ||
| or disabling it requires draining old ACP processes. A schema v2 reader still | ||
| accepts schema v1 active records, but an older reader does not understand | ||
| schema v2. Rollback therefore requires draining all new writers and confirming | ||
| that no active, sealed, or claim record from this protocol remains. | ||
|
|
||
| Hashing is intentionally performed at seal and takeover instead of adding an | ||
| incremental digest to every append. This keeps the first implementation small | ||
| and makes the proof independent of process memory. A very large transcript may | ||
| cause managed sealing to exceed the parent deadline; that fails closed with | ||
| the active lock retained and is observable as an unclean shutdown. | ||
|
|
||
| ## Verification | ||
|
|
||
| Unit and multiprocess coverage must prove: | ||
|
|
||
| - schema v1 and v2 active locks keep their existing live/stale behavior; | ||
| - managed acquisition never reclaims an active dead-PID or foreign-host lock; | ||
| - successful sealing records the exact relative key, existence, byte length, | ||
| and SHA-256 digest; | ||
| - a certified replacement reloads the sealed transcript and becomes active; | ||
| - two replacements racing one sealed record elect exactly one owner; | ||
| - transcript modification, replacement, truncation, proof corruption, | ||
| malformed sealed records, and residual claims fail closed; | ||
| - failures before and after each primary-path transition never overwrite or | ||
| delete an unknown successor; | ||
| - managed flush failure retains the active primary; | ||
| - normal recorder close releases instead of sealing; and | ||
| - the default-off and standalone ACP paths remain unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The lock directory stops being self-cleaning.
This seals every config the agent knows about, and a sealed record is removed only by a certified takeover of that exact session (
session-writer-lease.tsL1367) — there is no TTL and no sweep anywhere. Before this PR a graceful managed shutdown released every lock and left<runtime>/tmp/session-writer-locks/empty. Now it retains one file per session that was live at shutdown, permanently, for every session that is never reopened — which for a long-running daemon is most of them.Failed transitions add residue that nothing collects either:
.claim,.sealed-candidate.*,.handoff.*,.sealed.*.*,.reclaim.*,.stale.*, and.tmpfiles are only removed on paths that complete, and several fail-closed branches deliberately leave them in place.Worth a startup sweep for sealed records and orphaned transition artifacts past some age. Deleting a sealed record is safe for the same reason the release path is safe — it just means the next acquirer does ordinary acquisition and reloads the authoritative tail. Not blocking, but this is unbounded growth in a directory the daemon writes to on every session.