feat: Add NEAR key management with transaction signing and policy engine - #14
ilblackdragon wants to merge 1 commit into
Conversation
Implements hybrid-custody NEAR key management where the agent holds scoped function-call keys for routine operations while high-value operations require explicit user approval through the existing channel approval flow. Core infrastructure: - Ed25519 key generation/import via ed25519-dalek (not near-crypto) - AES-256-GCM encrypted storage via existing SecretsStore - Hand-rolled borsh-serializable NEAR transaction types - NEP-413 intent signing and MPC chain signature support - Configurable policy engine with transaction analysis pipeline - Daily spend tracking with automatic midnight UTC reset - Encrypted backup/restore with Argon2id KDF - CLI subcommands: generate, import, list, info, remove, export, policy, backup, restore - NEAR ed25519 secret key leak detection (Critical/Block) - WASM sign-payload host function (keys never enter WASM memory) - KeyManager wired into AgentDeps for agent-wide access Security invariants: private keys never reach the LLM or WASM boundary, signing happens in host Rust code with Zeroize on drop, every transaction is analyzed before signing, most-restrictive policy rule wins. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| /// Rules: 2-64 chars, lowercase alphanumeric + `.`, `-`, `_`. | ||
| /// No leading/trailing separators, no consecutive separators. | ||
| #[derive(Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] | ||
| pub struct NearAccountId(String); |
There was a problem hiding this comment.
Instead of handrolling these can we not use near protocol crates? These types should already exist in near sdk
Reviews PRs nearai#10, nearai#13, nearai#14, nearai#17, nearai#18, nearai#20, nearai#28 covering: - Critical: hand-rolled NEAR tx serialization and key mgmt (PR nearai#14) - High: hooks system can bypass safety layer (PR nearai#18) - High: DM pairing token security needs verification (PR nearai#17) - Medium: auth bypass when API key mode set without key (PR nearai#20) - Medium: safety error retry classification in failover (PR nearai#28) - Low: Okta WASM tool and benchmarking harness https://claude.ai/code/session_01B75Rq9u593YG9Kc4FG487Z
Code Review: IronClaw PR #14 - NEAR Key ManagementSummaryThis PR introduces comprehensive NEAR blockchain key management with ed25519 signing, a policy engine for transaction approval, backup/restore functionality, and WASM payload signing capabilities. The implementation demonstrates solid security fundamentals: proper use of Argon2id for key derivation, AES-256-GCM for encryption, Zeroize for secure memory handling, and a leak detector for secret key detection. However, there are several concerning areas requiring attention before this security-critical code should be merged. Pros
Concerns
Suggestions
|
ilblackdragon
left a comment
There was a problem hiding this comment.
Review: feat: Add NEAR key management with transaction signing and policy engine
This is a well-structured PR that adds critical security infrastructure. The hybrid custody model, the policy engine, and the WASM signing boundary are all architecturally sound. The test coverage is strong. That said, since this is key management -- the kind of code where a bug means lost funds -- I have several concerns that should be addressed before merging.
Security Issues (Must Fix)
1. TOCTOU race in generate_key and import_key (src/keys/mod.rs)
Both methods call load_store() to check for duplicates, then call load_store() again later to save. Between the check and the save, another concurrent call (or even the same KeyManager from another task) could insert a key with the same label. The second load_store() clobbers silently.
// Line ~133: First load to check duplicates
let store = self.load_store().await?;
if store.keys.contains_key(label) { ... }
// ... key generation and secrets store insertion happen here ...
// Line ~179: Second load to save -- races with first!
let mut store = self.load_store().await?;
store.keys.insert(label.to_string(), metadata.clone());
self.save_store(&store).await?;This is the same TOCTOU pattern flagged in CLAUDE.md under "Fix the pattern, not just the instance." Both generate_key and import_key have this. The secrets store create call could succeed while the metadata save races, leaving an orphaned secret. At minimum, do a single load, mutate, save cycle. Better: use a file lock or an RwLock on the KeyManager.
2. Binary body scanning regression in leak_detector (src/safety/leak_detector.rs)
The PR removes the String::from_utf8_lossy scan for HTTP request bodies and replaces it with a strict from_utf8 check:
// Before (secure):
let body_str = String::from_utf8_lossy(body_bytes);
self.scan_and_clean(&body_str)?;
// After (bypasses scanning):
if let Ok(body_str) = std::str::from_utf8(body_bytes) {
self.scan_and_clean(body_str)?;
}
// Binary bodies are not scannedThis means an attacker can prepend a single 0xFF byte to an exfiltration payload containing ed25519:... or sk-proj-... and the leak detector will skip the body entirely. The old code handled this correctly with from_utf8_lossy. The deleted test test_scan_http_request_blocks_secret_in_binary_body explicitly tested this exact attack vector. Revert this change.
3. Metadata file (~/.ironclaw/keys.json) stores key labels in plaintext on disk without filesystem permissions hardening
The save_store method writes keys.json with default umask permissions. On multi-user systems, another user could read which keys exist, their account IDs, public keys, networks, etc. While the private keys themselves are in the encrypted secrets store, metadata leakage is still an operational security concern. At minimum, set file mode to 0600 after write (or use std::os::unix::fs::OpenOptionsExt with mode).
4. Policy file (~/.ironclaw/key_policy.json) is unprotected
The policy file controls what the agent can auto-approve. If an attacker can modify this file, they can set transfer_auto_approve_max_yocto to u128::MAX and whitelist their own account. The file should have restrictive permissions and, ideally, a MAC/checksum so modifications are detected.
Correctness Issues
5. wrapper.rs rewrite is a breaking change (net -1180 lines)
The PR replaces the entire WASM tool wrapper with a new implementation that uses the low-level Val API instead of wasmtime::component::bindgen!(). This drops:
WasiViewimplementation (WASI context)- All HTTP host functions (
http-requestwith timeout, headers, body) - Credential injection (placeholder substitution, host-based injection, redaction)
- OAuth token refresh (
OAuthRefreshConfig) LeakDetectorintegration in WASM responsestool-invokeandsecret-existshost function bindings- The
bindgen!()macro entirely (replaced with manualValmarshalling)
The WIT file also removes the timeout-ms parameter from http-request. Only log, now-millis, workspace-read, and the new sign-payload are wired up in the new wrapper. This means all existing WASM tools that use HTTP, secrets, tool invocation, or credentials will break. This seems unintentional or at least needs to be called out in the PR description.
6. Spend tracking not recorded on approval-then-sign path (src/keys/mod.rs:314)
Spend is only recorded in the AutoApprove branch of sign_transaction:
PolicyDecision::AutoApprove => {
// ...signs...
if analysis.total_value_yocto > 0 {
let _ = self.spend_tracker.record_spend(...).await;
}
}But when approval is granted and the transaction is subsequently signed (the ApprovalRequired path), there is no code path that records the spend. The daily limit can be bypassed by always going through the approval flow for small transactions.
7. _domain parameter unused in build_chain_signature_action (src/keys/chain_signatures.rs:168)
The SignatureDomain parameter is accepted but ignored. The generated sign call doesn't pass the domain to the MPC contract. This could result in signing with the wrong curve if the contract defaults differ from what the caller expects.
Design Feedback
8. Flat file storage for keys metadata and spend tracking
Both keys.json and spend_tracking.json are read-modify-write JSON files. This has no concurrency safety (see #1), no atomicity (a crash mid-write corrupts the file), and doesn't follow the project's Database trait pattern. Per CLAUDE.md: "All new features that touch persistence MUST support both backends." Consider:
- Atomic writes (write to temp file, then rename)
- Or migrate to the Database trait with proper implementations
9. Default policy is too permissive for full-access keys
PolicyConfig::default() sets deny_full_access_operations: false, meaning a full-access key can be used for transfers, function calls, etc. The PR description says "high-value operations require explicit user approval" but the default policy auto-approves zero-value transfers from full-access keys (because transfer_auto_approve_max_yocto: 0 means "up to 0 yocto is auto-approved"). While this technically blocks non-zero transfers, function calls with zero deposit through a full-access key on an arbitrary contract are also auto-approved if the key is scoped. Consider defaulting deny_full_access_operations: true to match the "hybrid custody" philosophy.
10. let _ = self.secrets_store.delete(...) silently ignores deletion errors (src/keys/mod.rs:265)
In remove_key, if the secret deletion fails but the metadata is already removed, the key becomes orphaned in the secrets store with no way to clean it up. At least log the error.
Style / Minor
11. let-else refactoring in capabilities.rs and leak_detector.rs
The PR refactors if let ... && ... chains into nested if let / if blocks. This is fine for compatibility, but the commit message doesn't mention it. These should be in a separate commit or at least noted in the PR description since they touch security-critical code paths (endpoint matching, leak detection).
12. format_yocto has precision loss for amounts between 1 milliNEAR and 1 NEAR
let frac = (yocto % ONE_NEAR) / ONE_MILLI_NEAR;This integer division truncates. E.g., 1.999 NEAR formats as "1.999 NEAR" but 0.001999 NEAR formats as "0.001 NEAR" (the 999 microNEAR is lost). For a financial display, consider at least 6 decimal places.
13. No #[cfg(test)] on InMemorySecretsStore import
The tests in mod.rs use InMemorySecretsStore -- make sure this type is available at test time (it appears to be, but worth a compilation check with --no-default-features).
Summary
The core key management design is solid -- encrypted storage, Zeroize on drop, keys never reaching WASM, policy-before-sign pipeline. But the TOCTOU race, the leak detector regression, and the wrapper.rs rewrite breaking existing WASM tools are blockers. The spend tracking gap is a financial correctness bug that should also be fixed before merge.
Recommended disposition: Request changes on items 1, 2, 5, 6 before merging.
…, binary writes - Add pre-intercept safety param validation so sandbox-dispatched calls go through the same checks as host-dispatched calls (#1) - Set network_mode: "none" on sandbox containers to prevent outbound network access (#3) - Reject binary content in containerized write instead of silently corrupting via from_utf8_lossy (#5) - Cap list_dir depth to 10 to prevent unbounded traversal (#8) - Change container creation log from info! to debug! to avoid breaking REPL/TUI output (#10) - Make is_truthy case-insensitive so SANDBOX_ENABLED=True works (#11) - Return error instead of unwrap_or_default for missing container ID (#12) - Propagate set_permissions errors instead of silently ignoring (#13) - Return error for missing daemon output key instead of defaulting to empty object (#14) - Add env mutex guard in sandbox_live_e2e test (#15) - Fix rustfmt formatting for let-chain in canonicalize_under_root Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(engine-v2): mount-backend abstraction for per-project sandbox (Phase 1) Adds the engine-side `MountBackend` trait + minimal `WorkspaceMounts` registry and a host-side bridge interceptor that routes sandbox-eligible tool calls (`file_read`, `file_write`, `list_dir`, `apply_patch`, `shell`) through a backend when their path argument starts with `/project/`. Default behavior is unchanged: until `EffectBridgeAdapter::set_workspace_mounts(Some(...))` is called (Phase 6), the interception path is dormant. This is the first phase of the per-project sandbox plan (`docs/plans/2026-04-10-engine-v2-sandbox.md`) and a deliberately small subset of the unified Workspace VFS proposed in #1894 — just enough abstraction so the sandbox can be a `MountBackend` rather than a special case in the bridge. When #1894's full mount table lands, the sandbox backend slots in unchanged. Engine crate (`crates/ironclaw_engine/src/workspace/`): - `mount.rs` — `MountBackend` trait, `MountError` (NotFound / InvalidPath / PermissionDenied / Io / Tool / Backend / Unsupported), `DirEntry`, `EntryKind`, `ShellOutput` - `filesystem.rs` — `FilesystemBackend`: passthrough host-fs implementation with two-layer path validation (lexical reject of absolute / `..`, then symlink-escape canonicalization). `read`/`write`/`list` fully implemented; `patch`/`shell` return `Unsupported` so the bridge falls through to the host tool until Phase 5 - `registry.rs` — `WorkspaceMounts` per-project registry with lazy `ProjectMountFactory`, longest-prefix-match resolution, cached and invalidatable Bridge (`src/bridge/sandbox/`): - `intercept.rs` — `maybe_intercept` and `SANDBOX_TOOL_NAMES`. Returns `Handled(json)` on a successful backend dispatch, `FellThrough` for non-sandbox tools, host paths, missing path params, or `Unsupported` backend ops - `effect_adapter.rs` — `workspace_mounts` field + `set_workspace_mounts` setter; interception block in `execute_action_internal` right before `execute_tool_with_safety`, gated on the optional mount table Tests (31 new): - 17 engine workspace unit tests covering trait error mapping, path safety (lexical + symlink), longest-prefix routing, and lazy factory caching - 9 bridge sandbox unit tests including `intercept_actually_dispatches_into_backend` (counting backend) which proves the interceptor reaches the backend - 5 integration tests in `tests/engine_v2_sandbox_integration.rs` driving `EffectBridgeAdapter::execute_action()` end-to-end per the "Test Through the Caller" rule (`.claude/rules/testing.md`), including a host-path-falls-through test that asserts the sandbox tempdir was not touched, and a `..`-escape test that verifies no `/etc/passwd` content leaks even after safety-layer redaction Drive-by: feature-gate two pre-existing dead-code helpers in `crates/ironclaw_skills/src/parser.rs` on `#[cfg(feature = "registry")]` to match their only call site, fixing a pre-existing clippy warning that blocked the workspace's `-D warnings` policy when `ironclaw_skills` is built with `default-features = false` (as the engine crate does). Verification: - `cargo fmt --check` clean - `cargo clippy --all --benches --tests --examples --all-features` zero warnings - 31 / 31 new tests passing; no existing tests broken Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(engine-v2): per-project sandbox — Phases 2–7 + live Docker e2e test Completes the per-project sandbox plan (docs/plans/2026-04-10-engine-v2-sandbox.md Phases 2–7), building on Phase 1's mount-backend abstraction (#2211). Phase 2 — Project workspace folder: - `Project.workspace_path: Option<PathBuf>` field + `with_workspace_path()` - Host-side `project_workspace_path()`, `ensure_project_workspace_dir()` (creates `~/.ironclaw/projects/<id>/` mode 0700, idempotent) - `FilesystemMountFactory` taking a `ProjectPathResolver` closure (decoupled from `Store`); wired into `EffectBridgeAdapter` via `set_workspace_mounts()` Phase 3 — Standalone daemon binary: - `src/bin/sandbox_daemon.rs` — NDJSON over stdin/stdout, health/shutdown/execute_tool - Constructs ReadFileTool/WriteFileTool/ListDirTool/ApplyPatchTool/ShellTool with `base_dir=/project` (override via `IRONCLAW_SANDBOX_BASE_DIR`) Phase 4 — Dockerfile.sandbox: - Multi-stage build: rust-slim builder (+ python3 for pyo3) compiles sandbox_daemon; debian-slim runtime with tini PID 1, common build tools, `/project` mount target Phase 5 — ProjectSandboxManager + ContainerizedFilesystemBackend: - protocol.rs: Request/Response/RpcError matching daemon wire format - transport.rs: `SandboxTransport` trait (seam for testing without Docker) - containerized_backend.rs: `ContainerizedFilesystemBackend` impls `MountBackend`, translates relative→`/project/<rel>`, maps tool-error→MountError - docker_transport.rs: real bollard exec session, serialized Mutex, lazy reconnect - lifecycle.rs: deterministic `ironclaw-sandbox-<pid>` naming, ensure_running/stop/remove - manager.rs: `ProjectSandboxManager` per-project transport cache Phase 6 — Router gating on ENGINE_V2_SANDBOX: - `engine_v2_sandbox_enabled()` helper (truthy: 1/true/yes/on) - Router selects `ContainerizedMountFactory` when enabled + Docker reachable; falls back to `FilesystemMountFactory` with warning otherwise Live e2e bugs caught and fixed: - Shell without explicit `workdir` defaulted to host (not sandbox); fixed by defaulting to `/project/` in `extract_path_param` - `ContainerizedFilesystemBackend::shell` parsed `stdout`/`stderr` but host ShellTool returns merged `output` field; fixed with fallback key lookup - SANDBOX_TOOL_NAMES only had v2 names (`file_read`/`file_write`) but host registry uses v1 names (`read_file`/`write_file`); added both aliases Tests (62 sandbox-related, all green): - 27 bridge sandbox unit tests (intercept, workspace_path, factory, protocol, lifecycle, containerized_backend with ScriptedTransport mock) - 7 containerized-backend tests (including 2 regression tests for the shell bugs) - 5 engine v2 sandbox integration tests (EffectBridgeAdapter end-to-end) - 5 daemon binary smoke tests (real subprocess + NDJSON I/O) - 17 engine workspace unit tests - 1 live Docker e2e test: agent clones nearai/ironclaw into sandbox, renames to megaclaw via sed, verifies with grep — 70s, $0.09, recorded trace committed Verification: - `cargo fmt --check` clean - `cargo clippy --all --benches --tests --examples --all-features` zero warnings - All 62 sandbox tests passing; no existing tests broken Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: replace .expect() with Result in DockerTransport::ensure_session CI's no-panics checker flagged the .expect("just inserted") in production code. Replace with .ok_or_else() returning MountError::Backend. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: multi-tenant project paths + unify sandbox env var with v1 Two issues addressed: 1. Project workspace paths now namespace by user_id: `~/.ironclaw/projects/<user_id>/<project_id>/` instead of `~/.ironclaw/projects/<project_id>/`. Prevents filesystem collisions in multi-tenant deployments where two users could theoretically have the same project UUID. 2. Sandbox enablement now reads `SANDBOX_ENABLED` (same env var as v1 sandbox) in addition to `ENGINE_V2_SANDBOX`. Either being truthy enables the per-project sandbox. This means a single flag governs sandbox behavior regardless of engine version, while the v2-specific override remains available for transitional setups. Tests: 30 bridge sandbox unit tests passing (added multi-tenant path tests + env var combination tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review — TOCTOU race, shell env passthrough, canonicalize guard Three issues flagged by the code review bot on #2211: 1. TOCTOU race in WorkspaceMounts::resolve (HIGH): Added double-checked locking — re-check the cache after acquiring the write lock so two threads racing on the same project's first access don't both call factory.build(). The second thread finds the insert from the first. 2. Shell intercept ignores env parameter (MEDIUM): The shell arm in maybe_intercept was passing HashMap::new() instead of forwarding the tool call's env map. Fixed to parse parameters["env"] and pass it through to backend.shell(). 3. Canonicalization fails when root doesn't exist (MEDIUM): When self.root hasn't been created yet (first write to a new project), canonicalize_under_root would walk up to a real ancestor and the starts_with check against the non-existent root would always fail. Now skips canonicalization entirely when root doesn't exist — lexical safety is already guaranteed by safe_join. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review round 2 — apply_patch schema, content validation, dir perms, docs - Fix apply_patch schema mismatch: MountBackend::patch now takes (old_string, new_string, replace_all) matching ApplyPatchTool's actual contract. Previously sent {patch: diff} which would fail with invalid_params in the containerized daemon. - Validate file_write content param: return error instead of silently writing empty string when content is missing. - Log stderr frames from sandbox daemon at debug! instead of silently discarding them in docker_transport StreamReader. - Tighten permissions on intermediate directories created by ensure_project_workspace_dir (projects/, <user_id>/) to 0o700, not just the leaf. - Fix stale module doc in sandbox/mod.rs (referenced "Phase 5 will add" but all phases shipped). - Fix doc path mismatch: workspace path is <user_id>/<project_id>/, not <project_id>/ (workspace_path.rs, CLAUDE.md, design plan). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review round 3 — symlink safety, visibility, debug logging - Close TOCTOU window in canonicalize_under_root: re-canonicalize and verify containment when the reassembled path exists on disk - Fix list_dir_recursive: use symlink_metadata (lstat) so symlinks are detected instead of followed; validate directories against root before recursive traversal - Tighten is_mountable_path to /project/, /memory/, /home/ prefixes instead of any absolute path (defense-in-depth) - Narrow sandbox module visibility to pub(crate) and remove unused pub use re-exports - Remove concrete types (FilesystemBackend, DirEntry, EntryKind, ShellOutput) from engine crate top-level re-exports; access via ironclaw_engine::workspace:: module path - Add debug! tracing to sandbox intercept routing decisions - Add read_file/write_file v1 aliases to daemon SUPPORTED_TOOLS health response - Remove developer-local path from sandbox mod.rs doc comment - Merge staging to fix CI (user_timezone field on ThreadExecutionContext) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review round 4 — safety validation, network isolation, binary writes - Add pre-intercept safety param validation so sandbox-dispatched calls go through the same checks as host-dispatched calls (#1) - Set network_mode: "none" on sandbox containers to prevent outbound network access (#3) - Reject binary content in containerized write instead of silently corrupting via from_utf8_lossy (#5) - Cap list_dir depth to 10 to prevent unbounded traversal (#8) - Change container creation log from info! to debug! to avoid breaking REPL/TUI output (#10) - Make is_truthy case-insensitive so SANDBOX_ENABLED=True works (#11) - Return error instead of unwrap_or_default for missing container ID (#12) - Propagate set_permissions errors instead of silently ignoring (#13) - Return error for missing daemon output key instead of defaulting to empty object (#14) - Add env mutex guard in sandbox_live_e2e test (#15) - Fix rustfmt formatting for let-chain in canonicalize_under_root Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review round 5 — path traversal, error types, tests Security fixes: - Sanitize user_id in workspace path to prevent directory traversal via malicious user IDs containing `..` or `/` - Add Component::ParentDir check in ContainerizedFilesystemBackend::container_path matching the defense-in-depth approach of FilesystemBackend::safe_join Correctness: - Use MountError::Tool instead of MountError::InvalidPath for missing tool parameters (content, old_string, new_string) — fixes confusing LLM-visible error messages - Fix clippy sort_by_key suggestion in registry.rs Cleanup: - Remove spurious Notify import and dead _notify_link function New tests: - ContainerizedFilesystemBackend path traversal rejection (read + write) - container_path unit tests for safe and unsafe paths - Adversarial user_id test in workspace_path - Daemon-side path traversal test in sandbox_daemon_smoke Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review round 6 — param normalization, error types, edge cases - Normalize sandbox params via prepare_tool_params() before validation, matching the host execution path (fixes inconsistent validation) - Return ToolError::InvalidParameters instead of EngineError::Effect for sandbox param validation failures (consistent error surface) - ensure_dir checks path.is_dir() not path.exists() (rejects files) - Empty user_id returns "_anonymous" sentinel instead of empty hex string that would drop the tenant namespace via PathBuf::join("") - Restore ENGINE_V2_SANDBOX env var after sandbox live E2E test - Tighten is_mountable_path to /project/ only (no mounts for /memory/ or /home/ yet) - Add v1 tool name aliases (read_file, write_file) to SUPPORTED_TOOLS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: unify sandbox env var — remove ENGINE_V2_SANDBOX, use SANDBOX_ENABLED only Single env var controls sandboxing for both engine versions. The transitional ENGINE_V2_SANDBOX override is removed from code, tests, docs, and Dockerfile. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: double-checked locking in transport_for, explicit stdin close in smoke test - ProjectSandboxManager::transport_for no longer holds the mutex across the Docker ensure_running await. Uses double-checked locking so concurrent projects initialize in parallel. - sandbox_daemon_smoke: explicitly take() stdin before wait_with_output so EOF is sent even without a shutdown request. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review — network mode, error types, race, protocol dedup - Change sandbox container network_mode from "none" to default bridge so git clone / cargo build / pip install work inside the container - Fix binary content rejection to use MountError::Tool instead of MountError::InvalidPath (semantic mismatch) - Fix list depth: use actual depth value instead of depth.max(1) - Fix orphan container race in transport_for by holding lock across container creation instead of double-checked locking - Deduplicate protocol types: daemon now imports from shared bridge::sandbox::protocol instead of defining its own copies - Make bridge::sandbox pub (narrow exposure: only protocol and workspace_path sub-modules are pub) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update plan doc — sandbox uses bridge networking, not network_mode=none Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addresses the second round of review feedback on the budgets PR. **Audit events (#15, missing side effect):** - BudgetEnforcer now emits record_budget_event on every reserve / reconcile / release / deny / rollback path so the budget_events table actually gets populated. - ReservationTicket gained actor_user_id + thread_id fields so reconcile/release can emit audit rows without a second DB lookup. - record_audit helper swallows write errors (tracing::warn!) — the business-critical op has already committed. - Two regression tests assert the shape: a successful reserve + reconcile produces [Reserve, Reconcile]; a denial on the second cascade level produces [Reserve, Deny, Release(rollback)]. **u64 → i64 overflow (#8, #17, #18):** - save_budget (both backends), record_budget_event (both backends), reconcile_reservation (both backends) now use i64::try_from with DatabaseError on overflow. Prevents negative BIGINT storage from silent `as i64` wrap on tokens / wall_clock / actual_tokens. **reserve_within_tx returned-ledger divergence (#14):** - libSQL path was returning a BudgetLedger with tokens_used incremented by requested_tokens, but the SQL UPDATE did not persist that change — the returned struct diverged from the stored state. tokens_used now matches what the DB holds; tokens settle on reconcile only. **Docs (#11, #12, #13, #16):** - BudgetPeriod::Rolling24h docstring now says "quantised to UTC midnight, not sliding-window" to match the implementation. - BudgetPeriod::Calendar docstring calls out the UTC-only caveat (no chrono-tz dep yet). - BudgetPeriod::PerInvocation docstring explains each call gets its own ledger by design; use Rolling24h for cross-call caps. - BudgetLimit::tokens and the in-enforcer token-cap check now document the check as best-effort (not atomic under concurrent reserves); USD remains authoritative. **Test isolation (#10):** - bash_max_output_length_env_var_is_respected was mutating the process env without restoring. Introduced BashEnvGuard (RAII over a module-static Mutex) that snapshots and restores BASH_MAX_OUTPUT_LENGTH, and gated test_large_output_command behind the same guard so the two can't race. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…check Four follow-ups from review of 87fc8d4: - responses_api.rs (Copilot #11): the byte-counter comment claimed it measured "what the caller actually sent over the wire" but the count is canonicalised JSON, not raw request bytes. Rewrote the comment to describe the canonicalised-size cap correctly. The `Serializer` import is kept — it's needed to bring the trait method into scope for the concrete `serde_json::Serializer::serialize_seq` call below (an earlier attempt to drop it failed to compile). - responses_api.rs (Copilot #12): caller-supplied tool names were only checked for non-empty + uniqueness. Whitespace, control chars, and over-long names could propagate into engine action surfaces, SSE payloads, and downstream LLM clients (which all enforce the OpenAI Responses spec `^[A-Za-z0-9_-]{1,64}$` and would reject anyway). Validate at request time. Added regression tests covering whitespace/control/non-ASCII names and the length cap. - responses_api.rs / bridge/router.rs / bridge/mod.rs (Copilot #13): the shadow check only consulted `ToolRegistry::tool_definitions()` and missed engine v2 capability actions (`mission_*`, `skill_*`, `memory_*`, etc.). A caller registering `mission_create` as an external tool would still hit the catalog short-circuit in `EffectBridgeAdapter::execute_action`, since the LLM-visible dedup in `available_action_inventory` doesn't extend to the execute path. Added `engine_capability_action_names()` accessor that pulls the full capability-action surface from the bridge's `CapabilityRegistry`, and merged it into the collision set the responses_api handler checks against. - runtime/conversation.rs (Copilot #14): documented that `extra_initial_metadata` is spawn-only. The `Running` (inject) and `Resumable` (resume) branches ignore it; callers needing per-request metadata on existing threads must use `ThreadManager::set_thread_metadata` instead. The bridge's external-tool-catalog `transfer` already handles this for the one in-tree caller, but documenting the contract prevents future surprise.
* feat(web): support externally-provided tools in Responses API Lets callers of `/v1/responses` (and `/api/v1/responses`) declare their own `function`-typed tools and feed back results via `function_call_output` items, matching the OpenAI Responses wire shape. Since IronClaw's engine has no per-request tool surface, integration happens at the prompt level: the catalog is rendered as `<external-tools>` in the user message and the agent signals a call by ending its response with a fenced ```` ```tool_call ```` block. When that fence is recognised, the reply is split into a leading `Message` plus a `function_call` `ResponseOutputItem`. Validation rejects unsupported tool types (`web_search`, `file_search`, `code_interpreter`) and tools missing `name` with 400, with two new integration tests covering both paths. * refactor(responses-api): switch external tools to engine v2 native path Replace the prompt-level fence protocol from PR #3122 with engine v2 native tool calls: caller-supplied `tools[]` are surfaced as real LLM-callable actions, the engine pauses with `ResumeKind::External` when one is invoked, and the bridge router projects the pause to a new `AppEvent::ExternalToolCall` carrying the OpenAI-shaped `function_call` wire fields. The integration is small because v2 already has the right primitives: - `ResumeKind::External { callback_id }` and `GateResolution::ExternalCallback { payload }` already existed for OAuth-style callbacks. - `agent_loop.rs:1480` already routes Responses API messages to `handle_with_engine` when `ENGINE_V2=true`, so no v2 migration of the endpoint itself is needed. - `EffectBridgeAdapter::execute_action` is the single chokepoint where caller tools can be detected before they reach the dispatch pipeline. Changes: - New `src/bridge/external_tools.rs` (`ExternalToolCatalog`) — per-thread registry of caller-supplied `ActionDef`s, plus the `ext_tool:` callback-id helpers used to disambiguate external-tool pauses from OAuth/pairing pauses (which also use `ResumeKind::External`). - `EffectBridgeAdapter` consults the catalog: any name in it is short-circuited to a `GatePaused { resume_kind: External { callback_id: ext_tool:<call_id> } }` before any registry dispatch, and `available_action_inventory` merges the catalog into the LLM-visible action surface (internal beats external on collision). - `Submission::ExternalCallback` gains an optional `payload` field; `bridge::handle_external_callback` plumbs it into `GateResolution::ExternalCallback { payload }`. Fallback predicate `gate_resume_is_external` lets non-auth External pauses (i.e. caller-tool resumes) resolve through the same handler. - New `AppEvent::ExternalToolCall` projected by `notify_pending_gate` when a paused gate carries an `ext_tool:` callback id; OAuth/ pairing flows keep flowing through the existing `GateRequired` channel. - `responses_api.rs` is gutted of the prompt rendering and fence parsing (`render_external_tools_preamble`, `extract_trailing_tool_call`, `parse_external_tool_call`, `ParsedToolCall`, `external_tool_names` accumulator field, and the `TOOL_CALL_FENCE` constants). The handler now: rejects `tools[]` when `ENGINE_V2=false`, registers caller tools in the catalog under the resolved thread id, detects resume requests (`previous_response_id` + `function_call_output` items in `input`) and submits them as `Submission::ExternalCallback` with the outputs as the resolution payload, and surfaces `AppEvent::ExternalToolCall` as a `function_call` `ResponseOutputItem` in both streaming (`output_item.added`+`done`) and non-streaming. - All existing OAuth/pairing `ExternalCallback` constructors updated to pass `payload: None` (no behaviour change). - Fence-protocol unit tests removed; replaced with coverage for the new `responses_tools_to_action_defs` converter and the accumulator's `ExternalToolCall` arm. Existing 9 integration tests in `tests/responses_api_path_prefix.rs` still pass. Note for reviewers: - The accumulator-side text response no longer tries to split the reply on a fenced `tool_call` block. The wire shape that callers receive for caller-tool invocations is purely event-driven now. - Internal vs external collision is handled silently by the dedup in `available_action_inventory` (internal wins). A request-time rejection for shadowing names is a follow-up — the current behavior is safe (the LLM only sees the internal version) but could surprise a caller who expects their tool to run. * test(responses-api): cover ENGINE_V2-off and resume-without-pending-gate Two new integration tests for behaviours added by the engine-native external-tool refactor: - `external_tools_rejected_when_engine_v2_disabled`: a request with caller-supplied `tools[]` while `ENGINE_V2` is off must 400 with a message naming the flag, not silently fall through. - `resume_without_pending_gate_returns_400`: a request with `function_call_output` items and a `previous_response_id` that doesn't correspond to a live external-tool gate must 400, not start a fresh turn against the (unrelated) thread. Both tests drive the full router (`start_test_server` + bearer auth) per `.claude/rules/testing.md` "Test Through the Caller". * test(responses-api): integration tests + drop unsafe env mutation Three groups of changes: 1. **Drop unsafe env-var mutation in tests.** `responses_api.rs` no longer reads `ENGINE_V2` directly: it keys off the presence of the live `ExternalToolCatalog` (initialized by `init_engine`) as the "engine v2 is up" signal. The path-prefix test that exercises the no-engine branch no longer needs `unsafe { std::env::remove_var }` — the absence of `init_engine` in `TestGatewayBuilder` is what makes the catalog absent, which is what makes the request reject. 2. **Engine-tier integration tests** (`tests/e2e_responses_api_external_tools.rs`). Drives engine v2 via the existing `TestRigBuilder` + `TraceLlm` replay infrastructure rather than spinning up an HTTP gateway: - `catalog_isolates_by_thread_id` — register-under-A doesn't bleed into B. - `catalog_register_overwrites_not_merges` — Responses API contract is "each request restates the full tools[]"; the catalog must replace, not merge. - `catalog_sweep_evicts_only_stale_entries` — TTL backstop. - `catalog_clear_on_terminal_state_explicit` — what we want; the test name flags that the production hook is missing. - `catalog_handles_concurrent_registrations` — 32 concurrent register-then-contains tasks; protects per-thread isolation under contention. - `callback_id_disambiguates_external_from_oauth` — `ext_tool:` vs `pairing:` prefix is the single bit that routes a paused gate to `AppEvent::ExternalToolCall` vs `AppEvent::GateRequired`. If the prefix invariant breaks, the wrong UI renders. Two more tests deliberately `#[ignore]` and document concrete bugs the implementation has today: - `engine_pauses_when_llm_calls_registered_external_tool` — running it surfaces "engine never paused on external tool". Confirms the **thread-id mismatch** bug: the catalog is keyed by engine `ThreadId`, but the responses_api handler registers under a separately-generated UUID before the engine spawns the thread. - `round_trip_resume_payload_reaches_llm` — the load-bearing end-to-end test. Documents the **resume-payload-not-materialised** bug: `bridge::router::resolve_gate` uses `pending.resume_output` for `ExternalCallback` resolutions and ignores the payload, so caller-supplied tool outputs never reach the LLM's context. - `external_collision_with_registry_action_is_rejected` — stub asserting the desired validation behaviour for caller tool names that shadow registry actions; today silently accepted, and the catalog-wins-in-dispatch ordering means the LLM thinks it called the internal tool but actually ran caller code. The two #[ignore] tests are the deliberate failure documentation: running them with `--ignored` panics with messages naming the underlying gap. The fixes go on a follow-up commit. 3. **`TestRigBuilder.send_external_callback_with_payload`** — new helper that mirrors the OAuth `send_external_callback` but carries a JSON payload. Used by the round-trip test; the existing payload-less variant kept for OAuth/pairing tests. Quality gates: `cargo fmt`, `cargo clippy --all --tests --all-features` clean. 14 of 17 tests pass; the 3 ignored ones are deliberate documentation of the gaps. * fix(responses-api): close 4 bugs surfaced by integration tests The engine-native external-tools path landed in 44135ca had four real bugs surfaced by the integration tests in a2c13ee. This commit fixes all four; every previously-ignored test now passes. **Bug 1 — Thread-id mismatch.** `responses_api.rs` registers tools in the catalog under a `thread_uuid` it generates from `previous_response_id` (or freshly), but `ConversationManager::handle_user_message` creates the engine's *actual* `ThreadId` internally. The catalog entries were under a UUID the engine never executed in. Fix: `ExternalToolCatalog::transfer(from, to)` and a hook in `bridge::handle_with_engine_inner` that calls it after the engine returns the spawned `ThreadId`. The handler-supplied conversation scope UUID is rebound onto the actual ThreadId before the LLM call lands, so `EffectBridgeAdapter::execute_action`'s catalog check finds the registered tools. **Bug 2 — Resume payload never materialised.** `bridge::router::resolve_gate`'s `GateResolution::ExternalCallback` branch only consulted `pending.resume_output` to construct the resumed `ActionResult`. `EffectBridgeAdapter::execute_action` sets that to `None` for caller-tool gates (the output isn't known at gate-fire time), so the resume fell through to `execute_pending_gate_action`, which re-ran the original action — re-pausing forever. The caller's tool output (passed via `GateResolution::ExternalCallback { payload }`) was dropped on the floor. Fix: special-case `ext_tool:` callback ids in `resolve_gate`'s ExternalCallback branch. Extract the matching output from the payload (Responses API wire shape: `{ outputs: [{ call_id, output }] }`) via the new `extract_external_tool_output` helper, synthesise an `ActionResult`-shaped ThreadMessage, and resume the thread directly. OAuth/pairing flows (which use `pairing:` callback ids and don't carry an output payload) keep the original `pending.resume_output` path unchanged. **Bug 3 — Internal/external collision in dispatch.** The catalog short-circuit in `EffectBridgeAdapter::execute_action` ran before the registry, but `available_action_inventory` dedupes the opposite way (internal beats external in the LLM-visible list). Result: an LLM call to (say) `shell` would land in caller-side execution even though the LLM saw the *internal* `shell` description in its action surface — a confused-deputy where the caller can return any output and the LLM trusts it as the internal tool's reply. Fix: reject the collision at request validation in `responses_api.rs`. After `validate_external_tools(...)`, look up registered tool names via `state.tool_registry.tool_definitions()` and 400 any caller name that shadows an internal action. **Bug 4 — Catalog cleanup never happens.** `sweep_older_than` existed but nothing scheduled it, and there was no terminal-state hook — so catalog entries accumulated for every thread that ever ran. Fix: - `await_thread_outcome` in the bridge router calls `catalog.clear(thread_id)` on every non-`GatePaused` outcome (Completed, Stopped, MaxIterations, Failed). `GatePaused` keeps the entry so resume requests can still find it. - A periodic sweep task in `init_engine` runs `catalog.sweep_older_than(1 hour)` every 5 minutes as a backstop for callers that abandon a paused thread without resuming. **Tests now passing:** - `engine_pauses_when_llm_calls_registered_external_tool` — proves Bug 1. - `round_trip_resume_payload_reaches_llm` — proves Bug 2 (and Bug 1 by extension). - `external_tool_name_shadowing_registered_action_is_rejected` — proves Bug 3. - `catalog_cleared_on_terminal_completed_outcome` — proves Bug 4. Plus four catalog-tier unit tests for the new `transfer` method, and the two pre-existing collision-prevention tests (`ext_tool:` vs `pairing:` callback id disambiguation). `TestGatewayBuilder.tool_registry(...)` is a new builder hook for tests that need to exercise the registry-aware handler paths (currently only the collision-rejection test, but the seam is there for future ones). Quality gates: `cargo fmt`, `cargo clippy --all --benches --tests --examples --all-features` clean. 17/17 engine v2 tests pass; 12/12 HTTP path-prefix tests pass. * fix(responses-api): address review feedback from PR #3122 Closes the race-window where caller-supplied external tools could be invisible to the LLM on a thread's first turn, plus a batch of smaller review findings. Race fix (Bug #3 from review): - Plumb `conversation_scope: Option<Uuid>` through `ThreadExecutionContext`, populated from thread metadata. - `ConversationManager::handle_user_message` accepts an `extra_initial_metadata` map; the bridge stamps the parsed scope into it so the engine sees it on the in-memory thread the executor task reads from (post-spawn `set_thread_metadata` is invisible to that task). - `EffectBridgeAdapter::execute_action` and `available_action_inventory` now look up the catalog under both `ctx.thread_id` and `ctx.conversation_scope`, so the executor task that starts immediately after spawn can find caller tools even before the bridge's post-spawn `transfer` rebinds them onto the engine `thread_id`. The transfer remains for terminal-state cleanup bookkeeping. Other review fixes: - Rewrite the stale `## Externally-provided tools` module doc in `responses_api.rs` to describe the engine-native flow (the original prompt-level fence text was left over from the first commit on the branch). - `validate_external_tools` size check now fails closed on serialization error (`unwrap_or(MAX + 1)`) instead of silently passing oversized payloads. - `notify_pending_gate` debug-logs the no-broadcaster path so a future SSE-less channel that grows an external-tool surface can be diagnosed instead of silently hanging. - Update `catalog_clear_on_terminal_state_explicit` test docstring and rename to `catalog_clear_removes_entry` — Bug 4's cleanup hook in `await_thread_outcome` already wires the production cleanup (covered separately by `catalog_cleared_on_terminal_completed_outcome`). - Delete dead `wait_for_first_engine_thread` helper and the `_harness_compiles` shim that kept it alive. New test coverage: - `bridge::effect_adapter::tests`: three race-window regression tests exercising both `available_action_inventory` and `execute_action` via the conversation_scope fallback path, plus a unit test on the `external_tool_catalog_keys` helper. - `bridge::router::tests`: four `extract_external_tool_output` tests covering match-by-call_id, missing-call_id-returns-null, no-outputs-array fallback, and find-after-misses. - `channels::web::responses_api::tests`: a full streaming round-trip test driving `streaming_worker` end-to-end with synthetic StreamChunk + ExternalToolCall events, parsing the actual SSE byte stream, and asserting the wire-frame ordering an OpenAI client would observe. * test: fix three pre-existing engine test failures - `executor::structured::call_id_preserved_when_no_lease`: the test was asserting on an error string ("no lease") that the preflight path stopped emitting when it added the "not callable in this execution context" check ahead of the lease lookup. The empty `MockEffects` used by the test exposed no actions, so the call short-circuited before reaching the lease check the test name describes. Register `web_search` in the inventory so the lease-miss path the test is named for actually fires. - `runtime::manager::stop_thread_works`: the test races the consecutive-action-error guard added in #2325. The thread loops on a deliberately unregistered `test_tool`, and 7 errors land before the 10ms sleep + `stop_thread` round-trip can deliver the stop signal in fast environments. The test's intent is "calling `stop_thread` doesn't deadlock the join", so accept `Failed` as a valid terminal outcome in addition to `Stopped`/`Completed`/ `MaxIterations`. Asserts a more specific message on the join result so a true regression (non-terminal outcome) still trips. - `tests::catalog_cleared_on_terminal_completed_outcome`: was racing two ways. (1) The cleanup-poll compared `catalog.len()` to a pre-snapshot — racy because `engine_external_tool_catalog` is process-global, so concurrent tests could keep the count from ever dropping below the pre-snapshot. (2) Multiple engine-touching tests in the file race the `OnceLock<Option<EngineState>>` global init, so messages sent by test A could be processed by test B's engine state. Fixes: - Add `ExternalToolCatalog::contains_action_anywhere(name)` so the cleanup poll can verify a unique marker action is gone regardless of what other tests have registered, and switch the test to use a per-test `format!("cleanup_marker_{uuid}")` action name. - Add a process-static `tokio::sync::Mutex` (`engine_state_lock`) that the three engine-touching tests in the file acquire for their duration, serializing their use of the global engine state. Per-test engine isolation belongs in the bridge itself; this is a test-side workaround until that lands. * fix(responses-api): close gemini bot review findings Three review fixes from gemini-code-assist on PR #3122 plus a related fix for fence-style tool calls reported by the user when running gpt-5.3-codex through `/v1/responses`. streaming_worker: finalize message item even when resolved text is empty ========================================================================= The Response branch only emitted `output_item.done` when the resolved text was non-empty. If `StreamChunks` had already opened a Message item (`output_item.added` fired, `message_output_index` is `Some`) and the terminal Response then resolved to an empty string, `output_item.done` was skipped — leaving the OpenAI client with a dangling in-progress message in its UI. Now we finalize whenever either text is available or a message item is in flight, and skip the redundant delta emit when there's nothing to deliver. Regression test `streaming_worker_finalizes_item_when_resolved_text_is_empty` drives the worker with an empty StreamChunk + empty Response and asserts `added_count == done_count` for output_item events. validate_external_tools: stream the size check, no allocation ========================================================================= Replaced the `Vec<Value>` + `String::len()` size measurement with a streaming `serde_json::Serializer` writing into a counting `io::Write` sink. Same byte count, no intermediate heap allocation, correctness contract unchanged (still fails closed if the serialization stream errors mid-flight). recover_tool_calls_from_content: recognize markdown-fenced tool calls ========================================================================= Some OpenAI-compatible models — notably gpt-5.3-codex via the Codex Responses API — emit caller-supplied tool calls as ```tool_call {"name": "get_balances", "arguments": {}} ``` instead of via the structured `function_call` output channel. Root cause is engine v2's CodeAct preamble pushing "always respond in a ```repl block" hard enough that the model generalizes the fenced protocol to a sibling `tool_call` fence for tools it can't dispatch through Python. Tools ARE passed to the provider correctly (`OpenAiCodexProvider::build_request_body` sets `body["tools"]` with strict-OpenAI schema and `tool_choice: "auto"`); the model just opts out of the structured surface in favor of a fence. The structural fix is to suppress the CodeAct preamble for Responses API turns that carry caller-supplied `tools[]`, but that's a larger piece of work. As a defense-in-depth fix: - `recover_tool_calls_from_content` now also matches markdown-fenced blocks with `tool_call`, `function_call`, or `tool_calls` info strings. Opening fence must be at line start to avoid matching inline backtick references inside prose. JSON body is parsed and validated against the available tool name set; unknown names are ignored. - `clean_response` strips the same fenced blocks via a new `strip_markdown_fence_block` helper so any malformed-JSON fence the recovery skipped doesn't leak fence syntax to the user. Six new unit tests in `llm::reasoning::tests` cover the recovery (JSON, with arguments, function_call alias, unknown-tool ignored, inline-reference ignored) and the clean_response stripping (clean case + malformed-JSON case). * fix(responses-api): emit ExternalToolCall on CodeAct GatePaused outcome The Responses API was timing out on caller-supplied tool calls when the LLM emitted them through CodeAct's Python (which is the default mode for engine v2). Symptoms reported: gateway shows 'Tool X requires external confirmation (gate: external_tool)' as a generic gate card and the /v1/responses POST returns response.failed. There are two paths that handle a `GatePaused { External }` from the engine: 1. `notify_pending_gate` — fires when the engine creates the gate mid-execution (Tier 0 structured tool calls). My earlier review- feedback fix wired this path to emit `AppEvent::ExternalToolCall` so the Responses API handler can surface a `function_call` ResponseOutputItem. 2. `await_thread_outcome`'s `ThreadOutcome::GatePaused` arm — fires when CodeAct's Python catches `EngineError::GatePaused`, raises a `RuntimeError("execution paused by gate 'external_tool'")`, the script unwinds, and the thread terminates with a `GatePaused` outcome. This arm calls `send_pending_gate_status`, which has an empty branch for `ResumeKind::External` (line 514: `External { .. } => {}`). No `AppEvent::ExternalToolCall` was emitted, so the Responses API handler waited for a never-arriving event and the response timed out as failed. Fix: in the `GatePaused` arm, when `resume_kind` is External with the `ext_tool:` callback prefix, broadcast `AppEvent::ExternalToolCall` through the SSE manager and short-circuit past the approval-card delivery path (which is for human-in-the-loop UX and doesn't apply to caller-executed tools). Mirrors the `notify_pending_gate` projection. Annotated with `// projection-exempt: bridge dispatcher, ...` per `.claude/rules/gateway-events.md` since the broadcast is a projection of a `ThreadOutcome` (one of the canonical source logs) rather than an unscheduled side-channel emit. Also update the `persist_v2_tool_calls_only_called_from_completed_arm` regression test, which was matching on bare `ThreadOutcome::Completed` and `ThreadOutcome::GatePaused` text. The Bug 4 catalog-cleanup hook (commit a4ba764) introduced an early `if !matches!(outcome, ThreadOutcome::GatePaused { .. })` guard above the match block, so the first text occurrence of `ThreadOutcome::GatePaused` is now in that guard rather than the match arm — false-failing the assertion. Anchor the test on the match-arm destructuring patterns (with first-field names) instead, so it pins the structural invariant the test name describes. * fix(responses-api): filter synthetic engine markers; clearer resume error Two issues from the user's live test of caller-supplied tools: 1. `__codeact__` was leaking to the response output as a `function_call` item with that name. The orchestrator emits ActionFailed events with `action_name: "__codeact__"` when a CodeAct script crashes (orchestrator.rs:940), and the responses_api accumulator + streaming worker were dutifully projecting those into `function_call` items. Fix: add `is_synthetic_engine_action(name)` (any `__double_underscore__` name) and skip those events in `ResponseAccumulator::process` and `streaming_worker` for the ToolStarted/ToolCompleted/ToolResult arms. Internal markers no longer surface to the caller. 2. The "function_call_output supplied but no pending external tool call" error was opaque — it gave the caller no way to diagnose why their resume failed. Most common cause: the LLM ran caller tools through CodeAct (Python) instead of structured tool calls, which currently doesn't pause the thread (script crashes with a Python RuntimeError, no GatePaused outcome, no PendingGate persisted). Engine-side fix is in PR #3157 and needs an extension to cover `External` resume kinds. Improved the error to point at: - The diagnostic check (verify prior response.output had a function_call item for this call_id) - The known limitation (CodeAct path doesn't dispatch caller tools) - PR #3157 as the in-progress engine fix Two new tests: - `accumulator_filters_synthetic_engine_actions`: drives ToolStarted + ToolCompleted with `name: "__codeact__"` and asserts the output array stays empty. - `is_synthetic_engine_action_recognizes_double_underscore`: pins the `__double_underscore__` predicate (positive: `__codeact__`, `__init__`; negative: regular names, single-underscore, leading- or trailing-only doubles). * fix(responses-api): tighten resume validation, sanitize external payloads, address review findings Addresses PR #3122 review comments plus the user-directed clean-up: - responses_api.rs: reject `function_call_output` items with missing/empty `call_id` (Copilot 474). Validate the pending gate is actually an external-tool gate (ResumeKind::External + `ext_tool:` prefix) and that at least one supplied `call_id` matches the pending callback before submitting the ExternalCallback (Copilot 1211). - bridge/router.rs: run the synthesized external-tool payload through `SafetyLayer::sanitize_tool_output` before it reaches the LLM — external tool payloads originate outside `EffectBridgeAdapter`'s pipeline and need the same leak/policy/sanitizer pass internal tool outputs get. Move projection-exempt annotations onto the `broadcast_for_user` call lines so the gateway-events check passes. - bridge/effect_adapter.rs: synthesize a `call_ext_<uuid>` call id when the executor reaches the external-tool short-circuit without `current_call_id` (Copilot effect_adapter.rs:1853), and document multi-call batching as an expected limitation at the short-circuit site so future readers know the engine pauses on first. - reasoning.rs: correct the stale fence-recovery comment (Copilot 1738). - e2e_responses_api_external_tools.rs: remove the "currently expected to fail" note; the test now pins the resume materialisation contract (Copilot 153). * fix(responses-api): finalize streaming placeholder before function_call; strip stale PR pointer Two follow-ups from review: 1. Streaming external-tool dangling `output_item.added`. When a StreamChunk arrived before the ExternalToolCall (the placeholder Message was already emitted via `output_item.added`), the prose-flush path created a *new* Message item at `acc.output.len()` and emitted a fresh added+done pair for it — never finalizing the original placeholder. OpenAI clients render the unmatched placeholder as "in progress" forever. Fix: when `message_output_index` is set, take it, fold the accumulated chunks into the existing item at that index, and emit `output_item.done` for the same index. Falls through to the original no-placeholder behaviour when there is no in-flight Message. Updated `streaming_worker_external_tool_call_emits_correct_frame_sequence` to match the corrected sequence (one Message added, one Message done, then FunctionCall added/done) and added a pairing-invariant assertion (`added_count == done_count`) so the regression can't be re-locked by an incorrect literal sequence. 2. Wire-visible error message in `responses_api.rs` referenced PR #3157 for the CodeAct fix. Once the PR merges the pointer is misleading, and external callers can't follow the link anyway. Dropped the trailing note; the behavioural part of the message stays. * fix(responses-api): tighten external tool name validation and shadow check Four follow-ups from review of 87fc8d4: - responses_api.rs (Copilot #11): the byte-counter comment claimed it measured "what the caller actually sent over the wire" but the count is canonicalised JSON, not raw request bytes. Rewrote the comment to describe the canonicalised-size cap correctly. The `Serializer` import is kept — it's needed to bring the trait method into scope for the concrete `serde_json::Serializer::serialize_seq` call below (an earlier attempt to drop it failed to compile). - responses_api.rs (Copilot #12): caller-supplied tool names were only checked for non-empty + uniqueness. Whitespace, control chars, and over-long names could propagate into engine action surfaces, SSE payloads, and downstream LLM clients (which all enforce the OpenAI Responses spec `^[A-Za-z0-9_-]{1,64}$` and would reject anyway). Validate at request time. Added regression tests covering whitespace/control/non-ASCII names and the length cap. - responses_api.rs / bridge/router.rs / bridge/mod.rs (Copilot #13): the shadow check only consulted `ToolRegistry::tool_definitions()` and missed engine v2 capability actions (`mission_*`, `skill_*`, `memory_*`, etc.). A caller registering `mission_create` as an external tool would still hit the catalog short-circuit in `EffectBridgeAdapter::execute_action`, since the LLM-visible dedup in `available_action_inventory` doesn't extend to the execute path. Added `engine_capability_action_names()` accessor that pulls the full capability-action surface from the bridge's `CapabilityRegistry`, and merged it into the collision set the responses_api handler checks against. - runtime/conversation.rs (Copilot #14): documented that `extra_initial_metadata` is spawn-only. The `Running` (inject) and `Resumable` (resume) branches ignore it; callers needing per-request metadata on existing threads must use `ThreadManager::set_thread_metadata` instead. The bridge's external-tool-catalog `transfer` already handles this for the one in-tree caller, but documenting the contract prevents future surprise.
…egistrar happy path Round out the test set for the WASM hook execution path: #11 / #12: gate + observer wall-clock timeout. The pre-fix dispatcher ran wasmtime synchronously on the executor, so the outer `tokio::time::timeout` `Err(_elapsed)` arm was effectively unreachable. Now that WASM execution runs on the blocking pool, the timeout actually fires; the new tests give the wasm budget headroom (1B fuel, 5s wall) and the dispatcher a 20 ms timeout, then assert the failure classification (FailClosed for gate, FailIsolated for observer). #13: observer memory exhaustion. Mirrors `wasm_memory_exhaustion_fails_closed_for_gate` against the observer dispatch path so the FailIsolated branch of the failure matrix has explicit memory coverage, not just fuel/wall. #15: `WasmResourceLimiter::memory_grow_failed` rollback. Stages an approved grow, simulates the OS-level grow failing, and asserts a subsequent grow of the full ceiling succeeds — the inflated `memory_used` from the failed attempt must be released. #16: registrar WASM happy path. Companion to the existing `install_wasm_body_requires_runtime` negative case: a valid module installs, the binding is visible via the public registry accessor, and is not pre-poisoned. #14 (`add_milestone_metadata` happy path) is intentionally omitted — the BeforePrompt dispatch path is currently unreachable due to a pre-existing manifest-vs-registry scope conflict (`OwnCapabilities` is the only valid `BeforePrompt` scope per manifest validation, but the registry rejects `OwnCapabilities` at `BeforePrompt` because the point has no provider context). That contradiction sits outside this PR's scope; flagging for a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Implement installed WASM hook runtime Adds crates/ironclaw_hooks/docs/threat-model-wasm.md and follows the reviewed design ack: 1) module bytes are resolved, digest-cached, and compiled in the tool-WASM style while reusing its resource limiter; 2) each invocation gets a fresh wasmtime Store; 3) the ABI is a wasmtime::Linker surface, not wit-bindgen; 4) host-import sink shims enforce call, patch-byte, observer-fact, and decision budgets. * Harden WASM hook string and metadata budgets * fix(hooks): validate WASM hook ABI at install time (serrrfirat #3 on PR #3634) Address serrrfirat MEDIUM finding #3: `WasmHookRuntime::prepare()` compiled and cached module bytes but did not validate imports or the requested export. ABI mismatches (unsupported import, missing export, wrong export signature) were deferred to first live dispatch — and the prior `wasm_unsupported_host_import_fails_closed` test codified that a bad-import module would install successfully and only fail closed at invocation. Malformed untrusted modules should never reach live traffic. Changes: - `prepare()` derives the target hook point from `request.kind`, then runs `validate_module_abi()`: scratch-instantiate the module against the point-specific linker (catches unsupported / wrong-type imports) and resolve the typed export `() -> ()` (catches missing export and wrong signature). Failures surface as new `WasmHookRuntimeError::InvalidImports` or existing `WasmHookRuntimeError::InvalidExport`, both of which bubble up as `HookError::RegistryConstruction` from the registrar. - `wasm_point_for_kind(HookManifestKind)` helper centralizes the kind → wasm-point mapping; the previous `execute_*` paths can share it in a follow-up but kept inline for now to minimize churn. Tests: - `wasm_unsupported_host_import_is_rejected_at_install_time`: replaces the prior test that codified late-failure behavior; asserts the registrar returns `RegistryConstruction` citing the bad import. - `wasm_missing_export_is_rejected_at_install_time`: new module that compiles but lacks the manifest-declared export; same install-time rejection. * fix(hooks): address henrypark133 must-fix #1, #2, #3 on PR #3634 Three items from the 5-15 review: **#1 (must-fix) Extract ironclaw_wasm_limiter micro-crate** Replace `#[path = "../../../ironclaw_wasm/src/limiter.rs"]` cross-crate file import with a proper Cargo edge. The 111-line `WasmResourceLimiter` moves into a new `crates/ironclaw_wasm_limiter` micro-crate that both `ironclaw_wasm` and `ironclaw_hooks` depend on. The architecture rule forbidding `ironclaw_hooks -> ironclaw_wasm` is preserved (the new crate sits below both consumers and pulls in only `wasmtime` + `tracing`); `cargo check`, `cargo doc`, and architecture-linting tests now see the edge, and the file can't be moved out from under one of the consumers silently. Mechanical changes: - new `crates/ironclaw_wasm_limiter/` (Cargo.toml + src/lib.rs with the type exposed as `pub` instead of `pub(crate)`) - workspace `members` entry added - `crates/ironclaw_wasm/src/limiter.rs` deleted - `crates/ironclaw_wasm/src/lib.rs`: `mod limiter` removed - `crates/ironclaw_wasm/src/store.rs`: import switched to `ironclaw_wasm_limiter::WasmResourceLimiter` - `crates/ironclaw_wasm/Cargo.toml`: dep added - `crates/ironclaw_hooks/Cargo.toml`: dep added - `crates/ironclaw_hooks/src/wasm/runtime.rs`: `#[path = ...]` block removed; import switched to the crate **#2 + #3 (must-fix) Dead WASM arms in dispatch** `run_before_capability_hook`, `run_before_prompt_hook`, and `run_observer_hook` each had an early-return guard that dispatched WASM hooks with `catch_unwind` + timeout, then ALSO had a matching WASM arm in the inner `match` that ran without those protections. The prompt-path arm additionally swallowed `WasmHookFailure` via `|_| ()`, making the must-fix #2 problem worse on that path specifically. If a future refactor removed any of the early-return guards, those inner arms would silently take over and drop panic isolation, deadline enforcement, AND (for prompts) the failure category. Replaced each inner arm with `unreachable!()` carrying a comment that explains why the arm exists and references the early-return guard above it. A future refactor that removes the guard will now trip the `unreachable!` at first call instead of silently degrading. All 154 hooks lib + 29 reborn integration tests still pass. * fix(hooks): plumb context to WASM hooks + runtime hardening Critical #1 on PR #3634: WASM hooks previously received no context. The `execute_*` entry points dropped the `&BeforeCapabilityHookContext` / `&BeforePromptHookContext` / `&ObserverHookContext` value and invoked the guest export with `()`, so a WASM gate could never decide based on the capability name, tenant, provider, or other dispatch-time facts. Add an `ic:hooks/context@1` host-import module exposing two read-only calls — `ctx_size() -> i32` and `ctx_read(ptr, len) -> i32` — backed by a JSON-serialized blob the dispatcher writes per-invocation into the fresh store. Modules that don't import these continue to link; modules that do import them get a stable, non-empty payload to read. An integration test (`wasm_before_capability_hook_reads_context_blob`) asserts the contract end-to-end: a guest that fails to read a non-empty blob traps before its `deny` call. Also rolls up the other reviewer-flagged WASM runtime issues, all of which touch `wasm/runtime.rs`: HIGH #2: epoch-tick background thread now holds a shutdown `AtomicBool` and joins on `Drop`. Previously it looped forever and leaked an Engine clone on every runtime drop. MED #4: compiled-module cache is now an `lru::LruCache` bounded by `MODULE_CACHE_CAPACITY = 128`. Replaces the unbounded `HashMap`. MED #7: `prepare()` no longer compiles under the cache lock. Fast path reads from LRU under a brief lock; slow path compiles outside the lock and re-checks on insert to avoid the TOCTOU window where two concurrent installs of the same module both compile. Bug #9: post-call `deadline_exceeded()` re-check on the Ok branch is gone. wasmtime epoch-interrupt is the authoritative wall-clock signal; an Ok return is no longer reclassified as a timeout because the wall ticked over during host-side return. Bug #10: `add_milestone_metadata` returns a distinct "metadata value exceeds the u32 byte-length ceiling" error when the guest-supplied `value.len()` overflows u32, instead of misreporting it as "exceeded total prompt-patch byte budget". Existing integration tests for WASM hooks are also re-wired through `HookRegistrar::with_verified_grants` so the grants-store gate added in the foundation-01 merge stops failing the pre-existing fixtures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(hooks): run WASM hooks on the blocking pool HIGH #3 on PR #3634: `tokio::time::timeout` does NOT cancel synchronous wasmtime execution. The previous code awaited a `catch_unwind(async { h.evaluate(ctx) })` future whose body completed in one poll, so the timeout could only fire *around* the WASM call rather than against it; a hook that wedged inside wasmtime simply pinned the calling tokio task. Route gate, prompt, and observer WASM dispatch paths through `tokio::task::spawn_blocking` via a shared `run_wasm_blocking` helper. The outer `tokio::time::timeout` now governs the JoinHandle, so a stuck blocking task stops blocking the dispatcher's caller; the wasmtime epoch interrupt configured in the runtime (10 ms tick) is the authoritative in-WASM wall-clock cancel signal. JoinError (panic in the blocking task) maps to `FailureCategory::Panic`, matching the pre-existing semantics for synchronous panics caught via `catch_unwind`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(hooks): O(1) hook-id lookup via side index Finding #8 on PR #3634: `set_priority`, `poison`, `is_poisoned`, and `contains_hook` all did full-registry scans over every binding at every point. Each is called per-dispatch (poison-checks on the snapshot loop in particular), so the cost is `O(registered_hooks)` per `(installed_hook, registered_hook)` pair. Maintain a denormalized `HashMap<HookId, (HookPointSpec, usize)>` side index in lock-step with `by_point` so every per-hook-id operation becomes a single hash lookup + a direct vec indexed access. The duplicate-id rejection in `insert` now reads from the side index too, turning what used to be a flat-map scan into a `HashMap::contains_key`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(hooks): wall-clock timeout, observer memory, limiter rollback, registrar happy path Round out the test set for the WASM hook execution path: #11 / #12: gate + observer wall-clock timeout. The pre-fix dispatcher ran wasmtime synchronously on the executor, so the outer `tokio::time::timeout` `Err(_elapsed)` arm was effectively unreachable. Now that WASM execution runs on the blocking pool, the timeout actually fires; the new tests give the wasm budget headroom (1B fuel, 5s wall) and the dispatcher a 20 ms timeout, then assert the failure classification (FailClosed for gate, FailIsolated for observer). #13: observer memory exhaustion. Mirrors `wasm_memory_exhaustion_fails_closed_for_gate` against the observer dispatch path so the FailIsolated branch of the failure matrix has explicit memory coverage, not just fuel/wall. #15: `WasmResourceLimiter::memory_grow_failed` rollback. Stages an approved grow, simulates the OS-level grow failing, and asserts a subsequent grow of the full ceiling succeeds — the inflated `memory_used` from the failed attempt must be released. #16: registrar WASM happy path. Companion to the existing `install_wasm_body_requires_runtime` negative case: a valid module installs, the binding is visible via the public registry accessor, and is not pre-poisoned. #14 (`add_milestone_metadata` happy path) is intentionally omitted — the BeforePrompt dispatch path is currently unreachable due to a pre-existing manifest-vs-registry scope conflict (`OwnCapabilities` is the only valid `BeforePrompt` scope per manifest validation, but the registry rejects `OwnCapabilities` at `BeforePrompt` because the point has no provider context). That contradiction sits outside this PR's scope; flagging for a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(hooks): typed WASM version material, reconcile design doc LOW #20 on PR #3634: extract the `{extension_version}+wasm:{module_digest_hex}` concatenation into a `WasmVersionMaterial` newtype with a single `Display` impl. The identity material no longer floats free as a stringly-typed argument inside the registrar. Reconcile `docs/successors/02-wasm-runtime.md` with the implementation: - Spell out that wall-clock cancellation depends on the `tokio::time::timeout(tokio::task::spawn_blocking(...))` pair, and explain why a bare timeout over a synchronous wasmtime call cannot actually cancel. - Define `FailIsolated` and `FailClosed` as `FailureDisposition` values, distinct from the older `HookFailureMode::{FailOpen, FailClosed}` policy switch that applies to predicates. - Clarify the generic `evaluate` export contract — name is whatever the manifest declares, signature is `(): ()`, context arrives through the new `ic:hooks/context@1` host imports — and note the intentional divergence from `WitToolRuntime`'s hardcoded interface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(hooks): drop .expect() in WASM module cache capacity Pre-commit no-panics CI flagged the .expect() on the LruCache capacity. Move the validity check to a const match, so the NonZeroUsize is fixed at compile time and the no-panics regex is satisfied. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(hooks): use HookLocalId::new after newtype privatization The newtype-privatization landed in reborn-integration after the hooks-fu-wasm-runtime branch's WASM scaffolding tests were written; update the affected test/registrar sites to use HookLocalId::new instead of the now-private tuple constructor. * style: cargo fmt after newtype-privatization fixups * test(hooks): ignore 3 BeforePrompt WASM tests with manifest/registry conflict These tests were failing on the original branch tip too (verified against origin/hooks-fu-wasm-runtime @ 571efdf). The Installed-tier BeforePrompt WASM install path has no valid scope today: - OwnCapabilities is rejected by the registry C3 check (finding #2 on PR #3573) since BeforePrompt has no per-capability invocation context. - SameTenant is rejected by manifest validation ("cannot combine scope = same_tenant with kind = before_prompt"). The budget-overflow paths these tests exercise are point-agnostic; the follow-up is to either rewrite the helper to install through BeforeCapability or add a Global manifest scope. Tracked as a deferred item on the new PR. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round-3 multi-agent review at head 707e2cd found 12 straightforward fixes. All applied here. 3 design-level items deferred to discussion. SCOPE PREDICATE GAPS (security): R3-1 §1.6 three DELETE statements gain conditional <agent_predicate>: delivery-claim path + delete-path queue + delete-path child_index. Without these, agent-scoped callers can delete other agents' auxiliary rows under the same (tenant_id, user_id). R3-2 Phase 0 LEFT JOIN gains <agent_predicate_on_s> — agent-scoped replay must not surface settlement-log rows belonging to other agents. Performance shape paragraph documents the rule. R3-13 subagent_idempotency_ledger UNIQUE constraint extended to include (tenant_id, user_id, agent_id, ...). Without scope cols in the UNIQUE, a cross-tenant collision (UUID or migration artifact) would be silently ON CONFLICT DO NOTHING'd. R3-14 capability_results UNIQUE idempotency index gains agent_id. Two agents producing the same (tenant, user, run_id, capability_id, invocation_id) would otherwise have their second write silently dropped. PostgreSQL uses COALESCE(agent_id, '__non_agent__') for uniqueness across NULL-agent rows. SPEC-VS-TRAIT DRIFT: R3-9 buffer_unordered propagation: Decision #14 + D4 prose + closing checklist all now say buffer_unordered(replay_pool_size). Prior contradiction: §5.3 pseudocode MUSTed buffer_unordered but other surfaces still said join_all. WU-C reading checklist literally would reintroduce the pool-starvation regression. R3-10 Phase 3 pseudocode + §5.10 risks: .load() → .read() to match the §4.3 trait method name. Plus §5.10 capability_result_store.load reference updated. R3-15 scope_from_run_context helper deleted — LoopRunContext.scope IS already TurnScope. Replaced with &write.run_context.scope direct borrow. §4.8 "Scope source" paragraph documents the canonical pattern. CHECKLIST + TEST NAMES: R3-4 Credential audit promoted to MERGE-BLOCKING checklist item (top of list). WU-C MUST complete LoopRunContext audit + add compile- time lint OR verify write-site stripping before merging the durable gate-resolution backend. R3-5 Six reconciler test scenarios in §5.9 get canonical function names under tests::reconciler_integration::*. WU-C now has exact targets for redelivery, idempotency, tombstoned, missing-result, crash-between-insert-and-deliver, and orphan-gate paths. R3-6 §4.9 + §7.3 name the payload-size-cap test: capability_result_store_write_rejects_payload_exceeding_8_mib_with_capacity_exceeded. Asserts typed CapacityExceeded error, not raw Backend/Io error. R3-7 §5.6 names the admission-gate test: background_spawn_rejected_with_replay_in_progress_while_reconciler_is_running. Foreground / blocking subagent paths must succeed throughout. CROSS-REF FIXES: R3-8 Decision #24 + closing checklist HA leader election refs: §5.10 → §5.11 (R14 renumber missed these). DEFERRED FOR DISCUSSION: - R3-3 Phase 4 per-row seal vs seal_batch trait method - R3-11 Formal batch trait signatures subsection placement - R3-12 skipped_orphan counter — split vs combined for tombstoned Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
R4-1 CRITICAL — Phase 3 buffer_unordered → buffered. buffer_unordered emits futures in completion order, not input order. Phase 4's to_attempt.zip(load_results) pairs row identity with payload positionally → SILENT cross-child payload delivery on every replay with >1 pending row. gate_store.record_background_settlement called with row_A.parent_run_id + row_A.child_run_id + payload_B. Fix: .buffered(replay_pool_size) — same concurrency bound, preserves input order. Decision #14, D4 prose, closing checklist all updated. Schema + SQL invariants: R4-2 Seal UPDATEs (§5.4 libSQL + §5.5 PostgreSQL) add user_id + <agent_predicate> — were missing despite §1.6 mandate. R4-5 §1.6 INSERT pseudocode for child_index + deliverable_queue add user_id + agent_id columns (CF1 added schema cols but not pseudocode → would NOT NULL violation on verbatim execution). f-sec-1 PostgreSQL ledger UNIQUE uses COALESCE(agent_id, '__non_agent__') — NULL agent_id rows were silently double-INSERTable. f-sec-2 §1.7 prose no longer describes forbidden (agent_id = ? OR agent_id IS NULL) pattern; references §1.6 conditional convention instead. f-bug-3 Phase 0 LEFT JOIN matches scope cols on ledger side too — cross-tenant UUID collision could otherwise suppress this tenant's replay. f-bug-5 Phase 2a maps Vec<SettlementLogRow> → Vec<LedgerKey> before upsert_sealed_batch — type mismatch fixed. f-perf-1 + partial-index DDL — idx_subagent_idempotency_ledger_pending (partial on delivered_at IS NULL) added to §5.4 + §5.5 so Phase 0 scan stays bounded by outstanding work. Trait surface + tests: R4-4 reconciler_replays_undelivered_settled_child step 6 fixed: `skipped == 0` → 3 real counter fields. R4-6 SubagentIdempotencyLedger trait drops redundant `scope: &TurnScope` arg from 6 methods — LedgerKey embeds scope (single source of truth). R4-7 reconciler_counts_failed_on_missing_capability_result gains pencil-receipt-survives assertion (delivered_at IS NULL). R4-8 delivery_node_invalid_substituted_to_unknown test named in §5.5 + §5.9 (4 cases: oversized, control chars, disallowed chars, empty). R4-9 §5.6 active-scope enumeration gains max_active_scopes_at_boot (default 1000) cap + 5s timeout + overflow → lazy fallback + operator metrics. f-maint-1 §5.4/§5.5 SQL path comments reference inline Rust constants in migrations.rs per §8.5 (not separate .sql files). R4-3 (MERGE-BLOCKING marker on credential audit): verified already applied via R3-4 (line 1928). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(reborn): WU-B subagent durability sub-spec Sub-spec for WU-B per docs/plans/2026-06-06-subagent-compaction-impl.md. Blocks WU-C. Doc-only. Covers 4 in-memory stores (gate resolution, goal, tombstone, capability result) + 2 new tables (settlement event log, idempotency ledger). Decides typed-repo vs ScopedFilesystem per _contract-freeze-index.md §2. Introduces CapabilityResultStore + SubagentRestartReconciler traits. Specifies libSQL + PostgreSQL schemas, first-writer-wins semantics, scope propagation, migration/rollback under subagent.background_enabled toggle, and the dual-backend parity test (#4431 follow-on). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B straightforward review fixes Applies 18 straightforward findings from multi-agent code review on PR #4582. Design-level items still pending discussion. Schema: - F2: PostgreSQL ledger run_id/child_run_id UUID → TEXT (§8.3 convention) - F3: align libSQL/PostgreSQL undelivered_terminal partial index - F5: add result_ref column to subagent_gate_settlement_log both backends - F6: capability_results uses explicit PRIMARY KEY (result_ref) - F8: scope predicates mandatory on UPDATE/DELETE templates in §1.6 - F9: define post-result-write flag update path (separate transaction) - F18: 8 MiB CHECK constraint on capability_results.payload (MUST) Contracts: - F1: CapabilityResultStore trait scope &ResourceScope → &TurnScope - F7: drop CapabilityRunId alias; use TurnRunId directly - F10: specify sanitized_reason source + sanitization transform - F11: specify delivery_node validation (length, allowlist, source) - F12: §6.3 restated in binary INSERT-OR-IGNORE ledger semantics - F13: resolve tombstone trait scope-param decision in spec Doc consistency: - F4: remove delivered_at IS NULL filter (column doesn't exist) Test plan: - F14: name positive production-readiness tests (goal, tombstone, capres) - F15: tombstone first-writer-wins distinguishing test - F16: agent_id cross-leakage parity test - F17: reconciler crash-between-ledger-insert-and-gate-write test Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B two-phase ledger + orphan handling (D1+D9) Resolves two reconciler bugs surfaced by multi-agent review on PR #4582: D1 — Crash-between-ledger-insert-and-gate-write strands parent silently. Idempotency ledger goes two-phase. `delivered_at TIMESTAMPTZ NULL`: - INSERT OR IGNORE leaves `delivered_at = NULL` (pencil receipt; claim, mid-flight). - After successful gate-store write, UPDATE seals row with `delivered_at = NOW()` (pen receipt; final). - Pencil rows surviving a crash become `retryable` on next boot, not silently `skipped_idempotent` as before. Matches the existing `IdempotencyLedger::begin_or_replay` precedent in `crates/ironclaw_product_workflow/src/ledger.rs`. Both gate-store and seal UPDATE are idempotent at the row level so duplicate delivery cannot occur and missed delivery cannot occur. D9 — Orphan settlement-log rows produced perpetual `failed` count. Reconciler now checks `gate_store.gate_exists(scope, gate_ref)` first. If the gate is gone (parent cancelled, gate row deleted): write `SubagentResultTombstone { disposition: DiscardedParentGone }`, seal the ledger row, count as `skipped_orphan`. One pass per orphan; future passes skip via sealed ledger row. Settlement log stays append-only. ReplayReport gains `retryable: u32` and `skipped_orphan: u32` so each counter has one meaning. `failed > 0` is now operator-actionable only — no more phantom alerts. Spec changes: - §5.2 ReplayReport struct extended. - §5.3 algorithm rewritten: gate-exists check, then tombstone check, then pencil-claim, then deliver, then seal. Pencil read on insert-skip distinguishes sealed (skipped_idempotent) from pencil (retryable). - §5.4 + §5.5 ledger DDL: `delivered_at` becomes nullable. INSERT examples split into pencil + seal. - §5.8 test plan: orphan-gate test case added; existing test names updated. - §5.9 risks: stale-children GC bullet rewritten; capability-result- missing conclusion sentence updated. - §6.3 re-flip narrative updated to use sealed/pencil vocabulary. - "Decisions ratified up front" table gains rows 11–13. - Closing checklist gains 3 WU-C action items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B reconciler perf + cold-start shape (D4+D5 comprehensive) Resolves cold-start / scaling concerns surfaced by multi-agent review on PR #4582. Comprehensive: D4 + D5 + 5 long-term concerns folded in. D4 — Reconciler replay batch-phased (no more N+1). §5.3 algorithm rewritten: Phase 0 bound input via LEFT JOIN against ledger (only pending pencil-or-missing rows enter the algorithm; replay's scan size stays proportional to outstanding work, not historical log size). Phase 1 batched preflight: one query for gates_exist_batch, one for read_tombstones_batch. Phase 2 multi-row ledger writes: 2a — orphan + tombstoned cleanup (one upsert-sealed batch) 2b — pencil claim (one INSERT OR IGNORE batch) Phase 3 parallel capability loads via `join_all` (capped at replay_pool size). Phase 4 per-row deliver + seal (sequential per row, each row hits a different parent's mailbox). Phases 0–3 are O(1) DB calls regardless of N. Net cost dominated by Phase 4's per-row delivery, ~5–30 ms per row depending on backend latency. 10–50× speedup over the previous N+1 form. D5 — Background replay + per-scope admission gate. §5.6 composition wire-up rewritten: - Replay dispatched via `tokio::spawn` from boot; foreground traffic accepts immediately (<100 ms cold start regardless of backlog). - Per-scope `ReplayState { completed_at, last_report }` tracks completion. Background-mode `SpawnSubagentPort` consults the gate before admitting; rejects with `SubagentSpawnError::ReplayInProgress` until per-scope replay completes. Foreground / blocking subagent calls NEVER consult this gate. - Dedicated `replay_pool` (default 4 DB connections, configurable via `RebornEventStoreConfig.replay_pool_size`) — replay never starves foreground writes during recovery storms. - Eager active-scope enumeration at boot via runs-table query. Bounded by active-runs count, not historical user count. Lazy per-scope replay deferred as future optimization. Long-term concerns folded in: - HA replicas: spec is HA-safe (correctness via Phase 2b INSERT OR IGNORE + single-winner seal UPDATE), HA-redundant (each replica runs replay independently — N× DB load at boot). Active-active leader election deferred to cross-cutting follow-up. Documented in §5.6 + §5.9. - Settlement log growth: Phase 0 LEFT JOIN bounds input — replay's scan size is independent of historical log size. Archival / materialized-view summarization deferred as ops follow-up. - Replay pool sizing: default 4 fine for typical fan-outs; tuning via P95 metric. Spec does not mandate auto-tuning. §5.7 NEW — Observability contract: - `RebornEventKind::SubagentReplayCompleted` event per scope. - 5 required metrics: replay_duration_seconds (histogram), replay_pending_rows (gauge), replay_outcomes_total{outcome=…} (counter), pencil_age_seconds (gauge), replay_in_progress (gauge). All labeled by (tenant_id, agent_id). - 3 required alerts: `failed > 0`, `pencil_age_seconds > 60`, `replay_duration_seconds{P95} > 30`. - OpenTelemetry spans: one per scope (`reborn.subagent.replay`) + child spans per phase. - WU-F WebUI surfaces `replay_in_progress` per-scope; background-spawn rejection during replay shown to user as "starting up, retrying in N seconds" affordance. Prerequisite for WU-G E2E + WU-F integration. Decisions table gains rows 14–19. Closing checklist gains 7 WU-C action items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B hot-path perf + multi-tenant scaling (D6+D8+E.A+A.A+A.B) Resolves hot-path overhead + multi-tenant scaling concerns for the spawn / capability-write / replay paths. D6-A — Durable per-scope capacity counter. Replace per-spawn SELECT COUNT(*) with a sidecar `subagent_gate_capacity_counter` table — one transactional UPDATE per spawn (no extra round-trip). Race-safe via SELECT FOR UPDATE (PG) / BEGIN IMMEDIATE (libSQL). Symmetric increment on INSERT, decrement on delivery / delete via GREATEST(undelivered - N, 0) safety net. E.A — Sharded counter for hot scopes (CAPACITY_COUNTER_BUCKETS = 16). Per-scope counter row becomes a write hotspot when one mega-tenant runs 10k+ concurrent background subagents under the same scope. Shard into K=16 rows per (tenant_id, user_id, agent_id) keyed by `bucket SMALLINT/INTEGER NOT NULL`. Spawn picks bucket via `hash(child_run_id) % K`. Cap check is `SUM(undelivered)` across all K buckets — index-only at K=16. `subagent_gate_awaited_children.counter_bucket` stores bucket-of-record for symmetric decrement on cleanup. Per-scope spawn throughput lifts from ~100/sec (single-row lock contention) to ~1600/sec on PostgreSQL. Drift bound: ≤ K-1 rows over cap under maximum concurrency. D8-A — CapabilityResultStore trait takes Vec<u8>, not serde_json::Value. Executor: `let bytes = serde_json::to_vec(&output)?;` ONCE. `byte_len` is `bytes.len() as u64` — derived for free. `bytes` is MOVED into the store, not cloned. Store INSERTs bytes directly into BLOB (libSQL) / JSONB (PostgreSQL) without re-serializing. `read()` returns Vec<u8>; caller deserializes lazily via `serde_json::from_slice` only when a Value is needed (prompt assembly, compaction). Eliminates 2× full-tree serialization + 1 Value clone per capability call. ~50% CPU reduction on capability-write hot path at production scale. Trait shape reflects what crosses the boundary (bytes, not a tree). Composes with future streaming variants (BoxStream<Bytes>). A.A — Reconciler replay jitter for fleet rollouts. `RebornEventStoreConfig.reconciler_replay_jitter_ms: u64` (default 5000). Each replica sleeps a uniform-random 0..jitter ms before launching its background replay task. Spreads the deploy-time reconciler stampede over a wider window — at 50-replica rollout, peak DB reconciler conn count drops from N×replay_pool to ~jitter-spread fraction. Foreground traffic NEVER pays the jitter cost. Set to 0 for single-node deployments. A.B — HA per-scope leader election (new §5.10, future, NOT WU-C scope). Documented direction: Postgres `pg_try_advisory_xact_lock` per scope. Replicas that lose election skip replay for that scope; still consume settlement events via gate-store mailbox as normal. Total fleet reconciler work drops from O(N × scopes) to O(scopes). Promotion trigger documented (P95 replay duration > 30s + sustained replay_in_progress aggregate > 60s). Lock is transaction-scope so auto-releases on leader crash — composes cleanly with D1's two-phase ledger. libSQL fallback: noop election (every replica is leader); libSQL deployments are typically single-node so redundancy is moot. Decisions table gains rows 20–24 (D6-A, E.A, D8-A, A.A, A.B). Closing checklist gains 4 WU-C action items + 1 follow-up note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B round-2 review fixes (R1-R17) Round-2 multi-agent review surfaced 5 High-severity regressions introduced by prior fix commits, plus medium consistency issues. All resolved here. SQL TEMPLATES (§1.6) — fix regressions in transaction shapes: R3 — Conditional agent_id predicate. Replace blanket `(agent_id = ? OR agent_id IS NULL)` (which lets agent-scoped callers reach system-level rows) with placeholder `<agent_predicate>` bound conditionally per caller's scope: `agent_id = ?` when Some, `agent_id IS NULL` when None. New §1.6 preamble paragraph documents the rule. R4 — Capacity counter SUM + UPDATE use the conditional predicate too. Bare `agent_id = ?` with NULL parameter evaluated to UNKNOWN, silently bypassing the 4096 cap for non-agent runs. R5 — Delivery-claim DELETE on deliverable_queue gains `child_run_id`. Previously wiped ALL queue rows for a gate when only one child was delivered — stranded N-1 siblings. R10 — Delivery-claim UPDATE SET also flips `delivery_claimed = 1` (prose was inconsistent with SQL). R13 — Delete-path DELETEs on deliverable_queue + child_index gain `user_id` predicate. awaited_children DELETE uses the conditional agent_predicate. R15 — Settlement log dedup decision resolved (was deferred). Ledger UNIQUE + gate-store idempotency + Phase 0 LEFT JOIN make duplicate log rows benign; no MIN(id) needed. R16 — parent_run_context_json gains sensitivity audit requirement (closing-checklist gate): WU-C MUST verify LoopRunContext is credential-free or strip sensitive fields at write site. ALGORITHM PSEUDOCODE (§5.3) — fix wrong column names + bounded fan-out: R1 — Phase 0 LEFT JOIN uses `s.parent_run_id` (column actually exists; schema does NOT have `s.run_id`). R2 — Phase 0 filter uses `s.terminal_kind` (column actually exists; schema does NOT have `s.event_kind`). R11 — Phase 3 capability loads use `buffer_unordered(replay_pool_size)` not `join_all`. Unbounded fan-out at 10k pending rows would starve foreground writes on the 4-conn replay pool. R12 — Phase 2a tombstone writes use `write_tombstones_batch` (single round-trip), not a per-row `for` loop. Trait gains batch method. TRAIT + VARIANT CONSISTENCY: R8 — InMemoryCapabilityResultStore type is `Mutex<HashMap<String, Vec<u8>>>` (was self-contradicted in §4.5 — D8-A regression). R9 — SubagentResultDisposition variants documented: today's `DiscardedByParentCancel` + WU-C addition `DiscardedParentGone` (used by §5.3 Phase 2a orphan cleanup). §3.7 risks bullet updated. Forward-compat with WU-D variants (`Delivered`, `SettledByBackground`). R17 — `scope_from_run_context` helper defined in §4.8 (was undefined). Maps LoopRunContext → TurnScope; documents user_id resolution via `explicit_owner_user_id()` + SYSTEM_RESERVED_ID sentinel. DOC HYGIENE: R6 — Tombstoned-result test assertion fixed: `skipped_orphan == 1, failed == 0` (matches §5.3 algorithm; was `failed == 1`). R7 — Double-replay test assertion fixed: `skipped_idempotent == 1` (was `skipped == 1` — field does not exist on ReplayReport). R14 — Duplicate `### 5.8` heading resolved. Test plan now §5.9, Risks §5.10, HA leader election §5.11. Pen→pencil terminology consistency in §5.4 + §5.5 SQL comments. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B Copilot + Codex review fixes (CF1-CF5) Round-2 Copilot + Codex automated reviewers caught issues skill reviewers missed. All resolved here. CF1 — subagent_gate_child_index + subagent_gate_deliverable_queue gain `user_id TEXT NOT NULL` + `agent_id TEXT` columns per Decision #5 (also fixes a latent SQL syntax error: prior R13 added user_id to DELETE predicates without the columns existing). New `idx_sgci_scope` / `idx_sgdq_scope` indexes on `(tenant_id, user_id, agent_id, child_run_id)` replace the prior `tenant_child` indexes. Both libSQL + PostgreSQL. CF2 — capability_results.created_at gains `DEFAULT (datetime('now'))` in libSQL (PostgreSQL already had `DEFAULT NOW()`). Needed because §4.6/§4.7 rely on this column for `idx_capability_results_run` ordering and `list_by_run` ORDER BY — silent inserter mistakes would break replay ordering. CF3 — CapabilityResultStore::write now takes `invocation_id: InvocationId`. UNIQUE INDEX `(tenant_id, user_id, run_id, capability_id, invocation_id)` enforces true first-writer-wins idempotency. Previous design minted a fresh UUID per call, so `INSERT OR IGNORE` could never collide — idempotency claim was misleading. Now a retry-after-transient-error returns the same `result_ref`. Trait + in-memory impl note + §4.8 wire-up updated; both backend schemas gain the column + unique index. CF4 — §6.2 rollback step rewritten. Goal store stays on FilesystemSubagentGoalStore (durable) when the toggle flips OFF; the toggle gates only background-mode spawn admission, NOT backend selection. Prior wording about "re-selects InMemoryBoundedSubagentGoalStore" contradicted §2.1. CF5 — Decision #5 reworded. Scope columns are always PRESENT on every durable table and reached via a scoped index (`idx_*_scope`). PKs remain shape-appropriate per table (e.g. `(gate_ref, child_run_id)`, `(result_ref)`) — scope need not LEAD every PK. Matches actual schema guidance and removes the false-positive interpretation that all PKs must be scope-prefixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B resolve 2 leftover review items Two open threads from round-2 review now decided + applied. 1. InMemoryCapabilityResultStore gains bounded eviction. `INMEMORY_CAPABILITY_RESULT_STORE_MAX_ENTRIES = 1024` + `INMEMORY_CAPABILITY_RESULT_STORE_MAX_BYTES = 4 MiB` (FIFO by insertion order). Prevents local-dev / CI OOM on long sessions that accumulate megabyte-scale payloads. Production-readiness check still gates the impl to LocalDevTest mode regardless. 2. `gate_resolution_scoped_query_excludes_rows_from_other_agents` promoted from WU-G to WU-C. This is a security gate (cross-tenant / cross-agent leakage class via missing agent_id predicate), not an E2E gate. Shipping the gate-resolution backend in WU-C without this guard would mean releasing the durable code with no test that catches a missing agent_id WHERE clause — unacceptable per §1.7 + `_contract-freeze-index.md` §8. Closing checklist gains two WU-C action items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B round-3 review fixes (R3-1..R3-15) Round-3 multi-agent review at head 707e2cd found 12 straightforward fixes. All applied here. 3 design-level items deferred to discussion. SCOPE PREDICATE GAPS (security): R3-1 §1.6 three DELETE statements gain conditional <agent_predicate>: delivery-claim path + delete-path queue + delete-path child_index. Without these, agent-scoped callers can delete other agents' auxiliary rows under the same (tenant_id, user_id). R3-2 Phase 0 LEFT JOIN gains <agent_predicate_on_s> — agent-scoped replay must not surface settlement-log rows belonging to other agents. Performance shape paragraph documents the rule. R3-13 subagent_idempotency_ledger UNIQUE constraint extended to include (tenant_id, user_id, agent_id, ...). Without scope cols in the UNIQUE, a cross-tenant collision (UUID or migration artifact) would be silently ON CONFLICT DO NOTHING'd. R3-14 capability_results UNIQUE idempotency index gains agent_id. Two agents producing the same (tenant, user, run_id, capability_id, invocation_id) would otherwise have their second write silently dropped. PostgreSQL uses COALESCE(agent_id, '__non_agent__') for uniqueness across NULL-agent rows. SPEC-VS-TRAIT DRIFT: R3-9 buffer_unordered propagation: Decision #14 + D4 prose + closing checklist all now say buffer_unordered(replay_pool_size). Prior contradiction: §5.3 pseudocode MUSTed buffer_unordered but other surfaces still said join_all. WU-C reading checklist literally would reintroduce the pool-starvation regression. R3-10 Phase 3 pseudocode + §5.10 risks: .load() → .read() to match the §4.3 trait method name. Plus §5.10 capability_result_store.load reference updated. R3-15 scope_from_run_context helper deleted — LoopRunContext.scope IS already TurnScope. Replaced with &write.run_context.scope direct borrow. §4.8 "Scope source" paragraph documents the canonical pattern. CHECKLIST + TEST NAMES: R3-4 Credential audit promoted to MERGE-BLOCKING checklist item (top of list). WU-C MUST complete LoopRunContext audit + add compile- time lint OR verify write-site stripping before merging the durable gate-resolution backend. R3-5 Six reconciler test scenarios in §5.9 get canonical function names under tests::reconciler_integration::*. WU-C now has exact targets for redelivery, idempotency, tombstoned, missing-result, crash-between-insert-and-deliver, and orphan-gate paths. R3-6 §4.9 + §7.3 name the payload-size-cap test: capability_result_store_write_rejects_payload_exceeding_8_mib_with_capacity_exceeded. Asserts typed CapacityExceeded error, not raw Backend/Io error. R3-7 §5.6 names the admission-gate test: background_spawn_rejected_with_replay_in_progress_while_reconciler_is_running. Foreground / blocking subagent paths must succeed throughout. CROSS-REF FIXES: R3-8 Decision #24 + closing checklist HA leader election refs: §5.10 → §5.11 (R14 renumber missed these). DEFERRED FOR DISCUSSION: - R3-3 Phase 4 per-row seal vs seal_batch trait method - R3-11 Formal batch trait signatures subsection placement - R3-12 skipped_orphan counter — split vs combined for tombstoned Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B round-3 design decisions (R3-3 + R3-11 + R3-12) Three coupled design-level changes from round-3 review now ratified and applied. R3-3 — Phase 4 seal_batch. Spec §5.3 Phase 4 previously issued per-row `idempotency_ledger.seal` on each successful delivery. At 100 children through a 4-conn replay_pool that is 25 sequential rounds (~125-750 ms) on the seal step alone. Fix: Phase 4 now collects sealed-row keys into a `sealed_keys` vec during the per-row loop, then issues ONE `seal_batch(scope, Vec< LedgerKey>)` call at the end. Single multi-row UPDATE. Idempotent per-row via the `delivered_at IS NULL` guard. Single-row `seal` retained for orphan / tombstone paths in Phase 2a (which already batch via `upsert_sealed_batch`) and for any future operator-driven manual interventions on stuck rows. R3-11 — Formal batch method signatures in §5.2.1. §5.3 algorithm calls 8 batch methods. Only `write_tombstones_batch` had a rough signature; the rest were implicit. WU-C would need to reverse-engineer 7 method signatures from pseudocode call sites. Fix: new §5.2.1 "Batch method signatures (reconciler-facing)" subsection lists all 8 method signatures with full async-trait syntax plus `LedgerKey` + `LedgerRow` struct definitions. Single- row variants documented alongside batch variants for completeness. WU-C now reads §5.2.1 literally as the trait surface contract. R3-12 — Split skipped_orphan counter. Old: `skipped_orphan = orphan_rows.len() + tombstoned_rows.len()`. Two semantically distinct cases conflated: - orphan = gate row gone (parent cancel + cleanup) - tombstoned = gate live but child pre-tombstoned (parent cancelled the specific child) Different operational signals; merging them prevented operators from distinguishing gate-cleanup spikes (high `skipped_orphan` alone) from parent-cancel spikes (high `skipped_tombstoned`). Fix: `ReplayReport` gains `skipped_tombstoned: u32`. Phase 2a increments each counter independently. §5.7 metric label list extended; §5.9 `reconciler_skips_tombstoned_child` test asserts `skipped_tombstoned == 1, skipped_orphan == 0`. Decisions table row 13 updated to six counters. Closing checklist gains 3 new WU-C items (seal_batch impl, batch-method trait surface, ReplayReport split). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B round-4 review fixes (R4-1 critical + 12 more) R4-1 CRITICAL — Phase 3 buffer_unordered → buffered. buffer_unordered emits futures in completion order, not input order. Phase 4's to_attempt.zip(load_results) pairs row identity with payload positionally → SILENT cross-child payload delivery on every replay with >1 pending row. gate_store.record_background_settlement called with row_A.parent_run_id + row_A.child_run_id + payload_B. Fix: .buffered(replay_pool_size) — same concurrency bound, preserves input order. Decision #14, D4 prose, closing checklist all updated. Schema + SQL invariants: R4-2 Seal UPDATEs (§5.4 libSQL + §5.5 PostgreSQL) add user_id + <agent_predicate> — were missing despite §1.6 mandate. R4-5 §1.6 INSERT pseudocode for child_index + deliverable_queue add user_id + agent_id columns (CF1 added schema cols but not pseudocode → would NOT NULL violation on verbatim execution). f-sec-1 PostgreSQL ledger UNIQUE uses COALESCE(agent_id, '__non_agent__') — NULL agent_id rows were silently double-INSERTable. f-sec-2 §1.7 prose no longer describes forbidden (agent_id = ? OR agent_id IS NULL) pattern; references §1.6 conditional convention instead. f-bug-3 Phase 0 LEFT JOIN matches scope cols on ledger side too — cross-tenant UUID collision could otherwise suppress this tenant's replay. f-bug-5 Phase 2a maps Vec<SettlementLogRow> → Vec<LedgerKey> before upsert_sealed_batch — type mismatch fixed. f-perf-1 + partial-index DDL — idx_subagent_idempotency_ledger_pending (partial on delivered_at IS NULL) added to §5.4 + §5.5 so Phase 0 scan stays bounded by outstanding work. Trait surface + tests: R4-4 reconciler_replays_undelivered_settled_child step 6 fixed: `skipped == 0` → 3 real counter fields. R4-6 SubagentIdempotencyLedger trait drops redundant `scope: &TurnScope` arg from 6 methods — LedgerKey embeds scope (single source of truth). R4-7 reconciler_counts_failed_on_missing_capability_result gains pencil-receipt-survives assertion (delivered_at IS NULL). R4-8 delivery_node_invalid_substituted_to_unknown test named in §5.5 + §5.9 (4 cases: oversized, control chars, disallowed chars, empty). R4-9 §5.6 active-scope enumeration gains max_active_scopes_at_boot (default 1000) cap + 5s timeout + overflow → lazy fallback + operator metrics. f-maint-1 §5.4/§5.5 SQL path comments reference inline Rust constants in migrations.rs per §8.5 (not separate .sql files). R4-3 (MERGE-BLOCKING marker on credential audit): verified already applied via R3-4 (line 1928). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(reborn): WU-B round-5 review fixes (R5-1..R5-10) - R5-1: PG capability_results.payload BYTEA not JSONB (byte-exact round-trip contract; JSONB normalization breaks parity + byte_len) - R5-2: capability-write idempotency conflict target = invocation unique index, insert-then-select read-back (was result_ref, which never conflicts on retry) - R5-3: lazy per-scope replay ships in WU-C as admission-gate trigger (capped scopes were rejected with ReplayInProgress forever) - R5-4: flat tombstone ScopedPath (no thread segment — settlement log carries no thread_id); read_tombstone also gains scope param - R5-5: Phase 3 = exists_batch existence check, no payload loads; delivery via new redeliver_settled_child (record_background_settlement was undefined and payload-shaped) - R5-6: libSQL MAX() not GREATEST in counter decrements - R5-7: tombstoned rows resolve live gate row (capacity-leak fix); new resolve_undeliverable_batch - R5-8: ReplayState keyed by (tenant_id, user_id, agent_id) - R5-9: SubagentReplayCompleted contract-freeze callout - R5-10: CapacityExceeded maps to CapabilityOutcome::Failed, never aborts the loop - minors: 11-method count, pseudocode scope-arg drift, six-counter comment, PG ON CONFLICT expression target, MountView user-isolation verification, stale §5.10 throughput bullet - plan: drop stale duplicate WU-C 'Files modified' block (contradicted the corrected block above it) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(reborn): WU-B add §9 parent-initiated child cancel + inspect Audit of existing host plumbing (request_cancel, RunCancellationHandle, children_of, event projection) shows the only gap is model-visible action surface. Ratifies decisions 33-35: two thin WU-D actions (subagent_cancel, subagent_status) over existing machinery; parent- requested cancel delivers a Cancelled settlement (never tombstones — DiscardedByParentCancel stays reserved for the parent-run-cancel cascade); status is metadata-only so settle-time delivery remains the sole sanitization choke point. Child-pushed progress notes deferred pending WU-G evidence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements hybrid-custody NEAR key management where the agent holds scoped function-call keys for routine operations while high-value operations require explicit user approval through the existing channel approval flow.
Core infrastructure:
Security invariants: private keys never reach the LLM or WASM boundary, signing happens in host Rust code with Zeroize on drop, every transaction is analyzed before signing, most-restrictive policy rule wins.