feat(runtime): keep shared inputs available for accepted work

[dokterdok]

Keep shared inputs available through admission and replay

A shared document can disappear from an upload cache before a Bot finishes the task that needs it. The file needs to remain available for the work that accepted it, including safe retries. This isolates the private-input custody portion of #109338 on #106742, without carrying its unrelated UX fixes or the route/Files stack.

• Retain independent, leased document copies and accept their references in the same SQLite transaction as canonical admission.
• Bind principal, session, request, final payload digest, intent and copy generation. A changed payload cannot reuse another preparation.
• Preserve exact and terminal replay without recapturing inputs or attaching unrelated copies to an old admission.
• Retain branch, lease and media holders with bounded, sealed reclamation. Images remain under canonical native-media ownership rather than becoming v3 document copies.
• Provide a trusted prepare_verified_documents boundary for verified task bytes and the complete final API payload. Source authorization and the Files receiver remain separate owners.

Read original retained inputs without recapture

Forward update d12f6041191233e7fa4f05bd7c02a02b84a7b85c adds only the Input-owned gateway/hosted_room_input_retained.py provider. It reconstructs the exact original admission binding from committed source metadata and retained custody references, verifies physical document identity and native references, and refuses missing or mismatched evidence. It does not recapture inputs, initialize another byte store, release custody or import an Output producer.

The interrupted-image correction and public nine-control Input/Q/core replay below are preserved unchanged. Retention #99107 at 031c9f3b175551bcd228fb15d8f9f704889eb820 owns the separate pending-cleanup/scan/compaction contract; Output #99159 at 93fd0cd531ffe9c3e6ec1cd80c868d0ae09e44f7 consumes both owners through its regression fixtures. The older public replay remains pinned to its stated source and does not claim to exercise this new reader.

The affected local declared composition exercised text and mixed-input retirement/replay plus task-local damaged-input isolation: 3 passed, 7 deselected (parent: 4.48s). These are the same three controls, not an additional test total. This is an isolated focused pytest diagnostic, not canonical-suite or full-suite acceptance. Its broader substrate still includes unpublished lower inputs, so this three-control proof is not yet a public-only replay. Existing native/device, protected authorization, live recovery and deployment limits remain unchanged. Exact-head hosted CI success has not been verified.

Interrupted native-image retention correction

At c061ac2cad7e699823b1f5166be795298e64b994, an interrupted mixed-input admission no longer loses its native image before the retained work can complete retirement/replay. Consumed native preparations retain the matching copy until identity-matched raw admission retirement provides positive release evidence. Missing or foreign evidence refuses collection; ordinary successful terminal release remains unchanged.

This is a normal-forward, three-file Input-only amendment: two custody modules and lower-owner lifetime regressions. It introduces no Output/driver dependency, F1 authorization change, or consumer implementation.

• The new interrupted-retention regression failed on the prior public owner and passed after the repair.
• Parent verification on this exact owner: 4 passed, 5 deselected, covering interrupted retention, absent/foreign retirement proof, and live/foreign admission identity. The worker's focused selection also retained the existing successful-release controls.
• A separate local composition passed 9 selected Files/ordinary Output/Stop-retirement cases, with the Input production and test bytes verified against this owner. That composition includes other unpublished inputs: it is not a public-only recipe, native-device acceptance, or an additional standalone test total.
• The broader base-only owner boundary run recorded 27 passed, 5 failed. Those five direct-rpc._submit authorization-refusal controls now pass when the declared Q/core dependencies below are composed with this Input head: 9 selected cases passed, comprising the five refusals and four new retention controls. Parent rerun: 9 passed in 2.61s. No guard or assertion was removed; these overlapping selections are not added together or represented as a full-suite/final-head independent-review pass.
• That minimal declared composition is now publicly reproducible: pinned replay script and instructions/source pins. Independent fresh-directory execution fetched only public objects and reproduced tree 5ff1604d08aa31d9286a79c443af94a92a7c22b1, without alternates, promisor packs or private fallbacks. It preserves Q post-preparation refusal, the core transaction guard and Input custody through three exact shared-file resolutions. The focused selection also passed on that public-only replay: 9 passed, 0 failed, repeating the same nine controls rather than adding new cases.
• Head 1b00ef7ca8926c5d5297aca86fb9bccaabe88e1a adds only the replay script and instructions; production and test source remain identical to c061ac2cad7e699823b1f5166be795298e64b994. Local proof commits are synthetic and must not be published. This still does not claim full-suite, native-device or held-scope acceptance.

Focused owner reproduction, with isolated test state and the project's resource limits:

pytest -q -o addopts= tests/gateway/test_input_custody_lifetimes.py -k 'interrupted_consumed_native_image or consumed_native_image_rejects'

Exact-head hosted CI success has not been verified. Historical evidence below remains bound to its stated revisions.

Current-runtime update

This is the input-custody prerequisite for Layer 5 of #97681, now composed against #106742 at 485d5f6848c25. File collection runs under the owning profile's home, credentials and terminal scope, before that authority is published. The existing guard still refuses collection once live ingress is available.

P2 integration correction — failed profiles must stay unavailable. Review found that a failed collection could park a secondary profile while leaving its cron execution handle and epoch registration behind. The added failure path now withdraws only that failed construction's registrations before the caller releases ownership. The valid launch profile and replacement registrations remain intact; a primary failure stays fatal and keeps its original error.

The regression uses a genuinely invalid stored reclamation marker. It failed before the correction, then passed with matching/replacement-state and cancellation controls. Cancellation does not stop a worker thread, and this does not claim a new caller-wide thread-lifetime guarantee or repair every runtime startup/hot-serve failure path.

Review corrections

Addressed the lifecycle findings in review 5205643369, with regression-led changes:

• P1 — terminal retries after image release. Replay now derives terminal state from durable admission/tombstone evidence and validates exact identity and payload digest without requiring legitimately released image bytes. Live inputs still require byte validation. Mixed and image-only tasks settle and release through the real inert drain before exact retry; neither recapture nor second execution occurs.
• P1 — orphan images after refused mixed preparation. Native targets receive durable preparation-bound reservations before publication. An absent target remains provisional until the actual no-replace publication winner is verified and bound. Refusal, expiry, callback failure or transaction rollback leave indexed collector work. Barrier tests cover an ordinary concurrent publisher winning, later settlement while the failed preparation still holds the file, expiry and SQLite reopen. Equal-byte image attachments share physical publication but retain both ordered media entries.
• P2 — lost legacy document inventory. The fixed native-only pre-v3 inventory becomes collector work, including already-initialized v3 databases. It is no longer discarded when the last legacy admission retires. The upgrade witness uses the actual old opaque hosted-document shape, without a synthetic separate native admission. Positive retirement, other holders, original physical identity and seal-before-unlink remain required.

The earlier 16-file / 50 MB task-limit correction is retained. Ordinary messages remain limited to eight files/25 MB; per-file and configured platform byte limits are unchanged.

Scope: 36 owner-only paths, +3,432/−47 against the pinned runtime. Forward updates preserve the previous public history and existing custody/lifecycle contributions. The latest amendments add interrupted native-image retention with lower-owner regressions, the public replay script and instructions, and the retained-input reader; the scoped startup integration and failure rollback remain intact. Eight setup lines across four native/API fixtures are included rather than left only in a private test assembly. No route, spool, Desktop, journal, output-producer or second-executor implementation is imported. #109338 remains intact; integrate its custody slice only once.

Required dependencies

• One gateway owns every local session: CLI, TUI, Desktop (local), API, ACP, bots and cron attach to the same live conversation #106742 at 485d5f6848c2598ca00fa7704e50e8ae66d0983a — common runtime base.
• fix(groups): keep copied Group Chats passive and protect their history #99107 at 43b9f02183444de0d48dc72170341346d58e09dc — passive safety and same-owner/post-preparation authorization safeguards.
• fix(runtime): guard new admissions and commit terminal metadata atomically #111216 at 2fb90a347b5a0d7864367b221544dc058c6cf70c — NEW-admission transaction guard.

Their implementations are not imported into this diff. Preserve both owners at admission-signature and authorization overlaps. This branch is not presented as independently runnable on the runtime base alone. #98072 supplies complementary Files ownership/provenance, not a competing custody implementation.

Verification and limits

The startup integration finding was reproduced, corrected and closed by one changed-area source re-review. This does not replace maintainer approval or claim a general review of every runtime failure path.

• Earlier pinned minimal proof: 96 distinct selected cases passed, zero failed, across 18 explicitly selected files on current One gateway owns every local session: CLI, TUI, Desktop (local), API, ACP, bots and cron attach to the same live conversation #106742 with only the declared Q/core dependencies. This includes the 12 startup/failure controls. Two workers, zero retries, private state/temp/cache paths; unsafe initialization-bearing cases remain excluded.
• Historical owner head: e671cb464b6e7c0d740855f2eabe1fd6bbe87eda. Tested minimal proof: 62d548e3a36301922f652f45ec7be32978899309. The two corrected files match the accepted broader assembly 89535feffcecfce5678725a1b9a211f47f2d6251; no R/I/O/M implementation was imported into this owner.
• The broader Files/output/private-inventory composition separately passed 716 cases with four expected OS skips. That is integration evidence, not this PR's standalone test total, and is not added to the 96.
• Historical lifecycle evidence at 39955e6063b7: 84 cases across 17 files; parent 30-case and 14-case reruns overlap that historical total.
• Changed-path Ruff and diff checks pass. Owner/proof production differences retain the existing lower-owner code, and delivered regression files match the tested copies.
• The changed-area review closed replay and legacy inventory, then exposed the concurrent-winner and duplicate-image defects. Those were corrected at the reservation/publication boundary, reproduced before the fix and checked directly by the parent afterward. This is not a claim that the reviewer has approved the final head.

Native/alias reclamation remains exclusive pre-ingress only, not online or immediate TTL deletion. Unknown physical links or replaced files refuse deletion. This does not recover inventory already discarded by older code or claim generic power-loss cleanup: loss between link and unlink can leave a private hardlink that conservatively blocks reclamation.

The bounded input-custody change is ready for review. Final-head maintainer review and the declared dependency gates remain open. At that earlier head, CI, Docker and Nix required maintainer approval with zero jobs executed. This is not a hosted-green result. Exact-head hosted CI and maintainer approval are separate gates; approval-required runs with zero jobs are not passing CI, and the local results are not every-commit hosted-green evidence. No live gateway, receiver activation, native Save, provider execution or host-loss acceptance is claimed.

[andrexibiza]

Reviewed exact head 07fc70fe7610bc2974c6dd301e98ec74618ba5d3 against its declared runtime base b36398f71929d6069905ccca59af8d1516c5ee3a, plus the exact published contracts in required #111216 (2fb90a347b5a0d7864367b221544dc058c6cf70c) and #99107 (43b9f02183444de0d48dc72170341346d58e09dc). I also traced the custody lineage back through #109338 and the digest-attested adaptation from #98072 rather than treating those as unrelated duplicates.

The direction here is substantially better than the deferred #109338 custody shape: the fixed migration cohort removes the upgrade-wide future-file hold, v3 documents are independently copied, preparation is token/generation/epoch bound, accepted refs carry principal/session/request/payload-digest/intent and compose into the canonical admission transaction, branch ownership is explicit, and reclamation seals before unlink with conservative identity/hardlink handling. The 16-item / 50 MB task-limit correction also matches the existing task contract rather than widening ordinary message limits.

I found three remaining lifecycle gaps:

1. P1 — a successfully settled image-bearing hosted task can lose exact terminal replay before the session is retired. SessionAuthority releases terminal attachments_v1 native media after settlement. lookup_request() still returns that terminal row from session_admissions with retired=False, and _reconstruct_payload() then requires the old image path to pass verified_identity(). Once settlement has legitimately released that image, retrying the same hosted task/generation fails storage_unavailable instead of returning the already-accepted terminal admission/receipt. The current mixed-input retry test retries before terminal settlement, so it does not exercise the deletion boundary. Exact replay needs to depend on durable admission/ref evidence rather than released image bytes (or the image lifetime must be extended through the replay lifetime). Please add a mixed/all-image accept → settle/release → exact retry witness proving no recapture and no second execution.

2. P1 — mixed document+image preparation can publish durable native image bytes before the final authorization/admission boundary, with no owner if admission is refused. The build_payload() callback runs during prepare_verified_documents(); for hosted mixed input it calls normalize_submission_payload(), which calls admit_attachments() / capture_native_media() before authority.submit() performs the post-preparation authorization and before the admission transaction accepts the custody handle. If membership/grant is revoked, the preparation expires, or admission otherwise refuses, the document copies are leased and collectible but the newly captured native-inputs image has no admission row or preparation reference. Native-media cleanup is admission-driven, so repeated refused mixed preparations can accumulate durable orphan bytes. Either defer canonical image capture to the accepted admission boundary or make those native refs part of the preparation/expiry ownership graph. The existing revocation test is document-only; this needs a mixed-input refusal/expiry regression that proves the image bytes are reclaimed.

3. P2 — the fixed legacy cohort can forget the only reclamation inventory for real pre-v3 hosted documents without unlinking them. On the #106742 base, hosted documents are captured under native-inputs and then represented in the admission only as prompt-text paths; they do not have a structured admission media reference. Migration records those paths in gateway_legacy_input_paths, but initialize_working_copies() / collect_legacy_input_aliases() operate on input_custody_copies populated from working-documents-v2, so a normal native-only legacy hosted document has no later collector row. When the last legacy admission is positively retired, _legacy_active() deletes gateway_legacy_input_paths outright, leaving the actual file behind with no remaining owner/index that can reclaim it. The current migration test masks that shape by creating a separate native admission via candidate() and later explicitly calling release_admission_media() on that synthetic structured owner. Please test the exact old submission_payload() shape (opaque hosted row + native document path, no separate native admission) and either convert the fixed inventory into bounded collector rows or explicitly unlink through a fail-closed reclamation state machine before clearing the inventory.

Topology / merge order: #111362 is the narrow custody carrier extracted from #109338; it should supersede only that custody slice, not #109338's unrelated fixes. #98072 is complementary Files ownership/provenance, not a competing custody implementation. #111216 supplies the NEW-admission authorization seam and #99107 supplies the same-owner/post-preparation authorization safeguards; both must compose before this carrier can claim end-to-end acceptance. #106742 remains the runtime foundation. I did not flag the absent dependency implementations in this diff as accidental omissions because the PR explicitly declares them and says this head is not independently runnable.

Acceptance: hosted proof is not green on the reviewed object. Exact-head CI 34927830877, Docker 34927830349, and Nix 34927830371 are all action_required; CI exposes zero jobs. Of the three surviving commits, ed20163c7c19e906a19c40f7aa4a13065c05711e has no workflow runs and 672e19b5b61e33037440b06e06537e8689c6d014 also has action-required hosted runs. The reported local 17-case / composed 269-case evidence is useful scoped development evidence, but this carrier is not exact-head or every-commit hosted-green yet.

This is a careful piece of state-machine work, and the core custody/ref graph is worth preserving. The remaining issues are all at lifetime boundaries where durable bytes and durable authority stop agreeing; fixing those should leave the design much tighter.

[andrexibiza]

P1 — terminal exact replay still depends on image bytes that settlement is allowed to delete. A hosted admission can remain in session_admissions with status='terminal', so lookup_request() returns it with retired=False. But settlement calls release_admission_media(), and terminal attachments_v1 images are deletion candidates. On the subsequent exact retry this line re-verifies the reconstructed native-inputs/<digest>/... path and raises storage_unavailable after that legitimate release. The mixed retry regression currently retries before settlement, so it never crosses this boundary. Please make accepted terminal replay derive from durable admission/ref evidence (or retain image bytes through that lifetime) and add accept → settle/release → exact retry coverage for mixed/all-image tasks.

[dokterdok]

Fixed in 39955e6. Terminal reconstruction now takes lifecycle state from the stored admission or tombstone, checks exact identity and payload digest, and does not require image bytes that settlement legitimately released. Live inputs still verify bytes.

The new mixed/image-only witnesses run the actual inert drain and media release before retry, return the original admission and terminal result, and prohibit both recapture and a second execution. Retired-receipt, changed prompt/digest and forged terminal-hint controls are covered too. This was a real gap in the earlier retry coverage.

[andrexibiza]

P1 — this publishes native image ownership before the final admission/authorization boundary. For a mixed document+image task, normalize_submission_payload() reaches admit_attachments() → capture_native_media() while prepare_verified_documents() is still building the final payload. authority.submit() re-authorizes only afterwards. If that authorization/admission refuses, the document copies remain under the preparation lease and can be collected, but the new durable native-inputs image has neither an admission owner nor a preparation ref; admission-driven native GC cannot find it. Repeated revoked/refused mixed preparations therefore leak durable bytes. Please either defer canonical image capture until accepted admission or track those refs in the preparation/expiry graph, and add a mixed revoke/expiry test that proves the bytes disappear.

[dokterdok]

Fixed in 39955e6. Native capture now commits a preparation-bound provisional target before publication and binds the actual verified winner afterward. A refused or failed preparation retains indexed ownership even when publication succeeds but the identity transaction rolls back.

Follow-up checks also caught a concurrent ordinary capture winning with another inode, and two equal-byte images conflicting within one cold-cache task. Both are fixed: physical publication is coalesced, but ordered media entries are preserved. Barrier regressions cover winner reuse, refusal/failure, the other admission settling while the lease holds, expiry, SQLite reopen and reclamation; live/branch/other-preparation holders remain protected.

Collection stays exclusive pre-ingress, not online or immediate TTL deletion. Unknown hardlinks still block deletion; generic power-loss temporary cleanup is not claimed.

[andrexibiza]

P2 — clearing this table can orphan the actual legacy document bytes permanently. On the base runtime, a hosted document is captured under native-inputs but then stored in the hosted admission only as an opaque prompt-text path; there is no structured media ref whose later release_admission_media() can delete it. Migration records that real path here, while the v3 collector scans input_custody_copies (and initialize_working_copies() only creates alias rows from working-documents-v2). When the final legacy admission retires, this DELETE can therefore erase the only index without unlinking the native-only legacy file. The current migration test uses a synthetic separate native admission and explicitly releases that owner, so it does not reproduce the old hosted-document shape. Please preserve/convert these inventory rows into bounded collector work (or safely unlink before dropping them) and add an exact old-submission_payload() upgrade witness.

[dokterdok]

Fixed in 39955e6. The fixed native-only path inventory is converted into bounded alias-collector work, including databases with an existing v3 marker. Retiring the last legacy admission no longer drops the only file index; deletion still requires positive retirement, holder checks, matching physical identity and a committed seal.

The regression now calls the actual old submission_payload document path and creates one opaque hosted admission, with no synthetic separate native admission. It proves live retention, settlement/retirement, reclamation and idempotent reopen, plus replacement/symlink/hardlink and unrelated-future-file controls. Inventory already discarded by older code is not guessed back from later files.

The final composed selection passes 84 cases; the parent reruns overlap it. The PR remains draft, and hosted CI/maintainer approval is still a separate gate rather than an every-commit green claim.