Skip to content

feat: Add NEAR key management with transaction signing and policy engine - #14

Closed
ilblackdragon wants to merge 1 commit into
mainfrom
key-management
Closed

ilblackdragon wants to merge 1 commit into
mainfrom
key-management

Conversation

@ilblackdragon

Copy link
Copy Markdown
Member

Implements hybrid-custody NEAR key management where the agent holds scoped function-call keys for routine operations while high-value operations require explicit user approval through the existing channel approval flow.

Core infrastructure:

  • Ed25519 key generation/import via ed25519-dalek (not near-crypto)
  • AES-256-GCM encrypted storage via existing SecretsStore
  • Hand-rolled borsh-serializable NEAR transaction types
  • NEP-413 intent signing and MPC chain signature support
  • Configurable policy engine with transaction analysis pipeline
  • Daily spend tracking with automatic midnight UTC reset
  • Encrypted backup/restore with Argon2id KDF
  • CLI subcommands: generate, import, list, info, remove, export, policy, backup, restore
  • NEAR ed25519 secret key leak detection (Critical/Block)
  • WASM sign-payload host function (keys never enter WASM memory)
  • KeyManager wired into AgentDeps for agent-wide access

Security invariants: private keys never reach the LLM or WASM boundary, signing happens in host Rust code with Zeroize on drop, every transaction is analyzed before signing, most-restrictive policy rule wins.

Implements hybrid-custody NEAR key management where the agent holds scoped
function-call keys for routine operations while high-value operations require
explicit user approval through the existing channel approval flow.

Core infrastructure:
- Ed25519 key generation/import via ed25519-dalek (not near-crypto)
- AES-256-GCM encrypted storage via existing SecretsStore
- Hand-rolled borsh-serializable NEAR transaction types
- NEP-413 intent signing and MPC chain signature support
- Configurable policy engine with transaction analysis pipeline
- Daily spend tracking with automatic midnight UTC reset
- Encrypted backup/restore with Argon2id KDF
- CLI subcommands: generate, import, list, info, remove, export, policy, backup, restore
- NEAR ed25519 secret key leak detection (Critical/Block)
- WASM sign-payload host function (keys never enter WASM memory)
- KeyManager wired into AgentDeps for agent-wide access

Security invariants: private keys never reach the LLM or WASM boundary,
signing happens in host Rust code with Zeroize on drop, every transaction
is analyzed before signing, most-restrictive policy rule wins.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread src/keys/types.rs
/// Rules: 2-64 chars, lowercase alphanumeric + `.`, `-`, `_`.
/// No leading/trailing separators, no consecutive separators.
#[derive(Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub struct NearAccountId(String);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of handrolling these can we not use near protocol crates? These types should already exist in near sdk

serrrfirat pushed a commit to serrrfirat/ironclaw that referenced this pull request Feb 11, 2026
Reviews PRs nearai#10, nearai#13, nearai#14, nearai#17, nearai#18, nearai#20, nearai#28 covering:
- Critical: hand-rolled NEAR tx serialization and key mgmt (PR nearai#14)
- High: hooks system can bypass safety layer (PR nearai#18)
- High: DM pairing token security needs verification (PR nearai#17)
- Medium: auth bypass when API key mode set without key (PR nearai#20)
- Medium: safety error retry classification in failover (PR nearai#28)
- Low: Okta WASM tool and benchmarking harness

https://claude.ai/code/session_01B75Rq9u593YG9Kc4FG487Z
@tribendu

Copy link
Copy Markdown

Code Review: IronClaw PR #14 - NEAR Key Management

Summary

This PR introduces comprehensive NEAR blockchain key management with ed25519 signing, a policy engine for transaction approval, backup/restore functionality, and WASM payload signing capabilities. The implementation demonstrates solid security fundamentals: proper use of Argon2id for key derivation, AES-256-GCM for encryption, Zeroize for secure memory handling, and a leak detector for secret key detection. However, there are several concerning areas requiring attention before this security-critical code should be merged.

Pros

  • Strong encryption defaults: Uses Argon2id (memory-hard) for backup key derivation with AES-256-GCM (authenticated encryption), avoiding weak KDFs like PBKDF2
  • Memory safety: Implements Zeroize trait on SigningKey to clear key material from memory immediately after use
  • Defense in depth: Policy engine supports multiple layers (transfer limits, whitelists, daily spend limits, function call rules)
  • Secret leak detection: Added regex pattern for NEAR ed25519 secret keys (80-90 char base58) with Block action
  • WASM boundary security: Private keys never enter WASM memory; signing happens in host code with only signature returned
  • Proper key validation: Account IDs validated on construction, public keys validated for correct length and format
  • Comprehensive test coverage: Tests for policy evaluation, transaction analysis, key serialization, glob matching, and spend tracking

Concerns

  1. Policy bypass in scope check (src/keys/policy.rs:456-472): The function call policy check has an early return for scoped keys that may allow transactions outside the key's actual scope. The logic action.value_yocto == 0 should verify the receiver_id matches the key's scoped receiver, otherwise a function-call key scoped to contract A could potentially be used for zero-deposit calls to contract B.

  2. Incomplete Argon2 parameters (src/keys/mod.rs:853): Uses Argon2::default() which uses Argon2id but doesn't specify memory cost (m), iterations (t), or parallelism (p). The default of 2^16 KB memory may be insufficient for security-critical backup encryption. Should explicitly set: Argon2::new(Algorithm::Argon2id, Version::V0x13, Params::new(65536, 3, 4, None)?).

  3. No key rotation mechanism: The implementation stores keys indefinitely without any rotation policy. Long-lived keys increase attack surface if the secrets store is compromised.

  4. Missing rate limiting on RPC client (src/keys/rpc.rs): The RPC client makes network calls without any timeout or retry logic. Long-running requests could cause the agent to hang indefinitely.

  5. Policy stored unencrypted (src/cli/key.rs:45-48): The policy configuration at ~/.ironclaw/key_policy.json is stored in plaintext. While it doesn't contain secrets, it reveals security limits to attackers who gain filesystem access.

  6. No transaction nonce management: The implementation fetches nonce per transaction with nonce = access_key.nonce + 1 but doesn't handle concurrent transaction ordering. Multiple simultaneous transactions could result in nonce collisions.

  7. Incomplete PayloadSigner implementation: The PayloadSigner trait is defined but never actually implemented with a real key manager integration. The WASM signing capability will fail at runtime.

Suggestions

  1. Harden Argon2 parameters for backup encryption with explicit memory cost (64MB), iterations (3), and parallelism (4):

    use argon2::{Algorithm, Version, Params};
    let params = Params::new(65536, 3, 4, None)?;
    let argon2 = Argon2::new(Algorithm::Argon2id, Version::V0x13, params);
  2. Add transaction queue or nonce management to prevent concurrent transaction nonce collisions.

  3. Implement actual signer injection into WASM capabilities by connecting KeyManager as the PayloadSigner implementation.

  4. Add RPC timeouts and exponential backoff for network resilience:

    let client = reqwest::Client::builder()
        .timeout(Duration::from_secs(30))
        .build()?;
  5. Consider encrypting policy file or at minimum adding file permissions checks (0600).

  6. Add integration tests for end-to-end signing flows with actual NEAR testnet.

  7. Implement key rotation with explicit rotation policy and metadata.

@ilblackdragon ilblackdragon left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: feat: Add NEAR key management with transaction signing and policy engine

This is a well-structured PR that adds critical security infrastructure. The hybrid custody model, the policy engine, and the WASM signing boundary are all architecturally sound. The test coverage is strong. That said, since this is key management -- the kind of code where a bug means lost funds -- I have several concerns that should be addressed before merging.


Security Issues (Must Fix)

1. TOCTOU race in generate_key and import_key (src/keys/mod.rs)

Both methods call load_store() to check for duplicates, then call load_store() again later to save. Between the check and the save, another concurrent call (or even the same KeyManager from another task) could insert a key with the same label. The second load_store() clobbers silently.

// Line ~133: First load to check duplicates
let store = self.load_store().await?;
if store.keys.contains_key(label) { ... }

// ... key generation and secrets store insertion happen here ...

// Line ~179: Second load to save -- races with first!
let mut store = self.load_store().await?;
store.keys.insert(label.to_string(), metadata.clone());
self.save_store(&store).await?;

This is the same TOCTOU pattern flagged in CLAUDE.md under "Fix the pattern, not just the instance." Both generate_key and import_key have this. The secrets store create call could succeed while the metadata save races, leaving an orphaned secret. At minimum, do a single load, mutate, save cycle. Better: use a file lock or an RwLock on the KeyManager.

2. Binary body scanning regression in leak_detector (src/safety/leak_detector.rs)

The PR removes the String::from_utf8_lossy scan for HTTP request bodies and replaces it with a strict from_utf8 check:

// Before (secure):
let body_str = String::from_utf8_lossy(body_bytes);
self.scan_and_clean(&body_str)?;

// After (bypasses scanning):
if let Ok(body_str) = std::str::from_utf8(body_bytes) {
    self.scan_and_clean(body_str)?;
}
// Binary bodies are not scanned

This means an attacker can prepend a single 0xFF byte to an exfiltration payload containing ed25519:... or sk-proj-... and the leak detector will skip the body entirely. The old code handled this correctly with from_utf8_lossy. The deleted test test_scan_http_request_blocks_secret_in_binary_body explicitly tested this exact attack vector. Revert this change.

3. Metadata file (~/.ironclaw/keys.json) stores key labels in plaintext on disk without filesystem permissions hardening

The save_store method writes keys.json with default umask permissions. On multi-user systems, another user could read which keys exist, their account IDs, public keys, networks, etc. While the private keys themselves are in the encrypted secrets store, metadata leakage is still an operational security concern. At minimum, set file mode to 0600 after write (or use std::os::unix::fs::OpenOptionsExt with mode).

4. Policy file (~/.ironclaw/key_policy.json) is unprotected

The policy file controls what the agent can auto-approve. If an attacker can modify this file, they can set transfer_auto_approve_max_yocto to u128::MAX and whitelist their own account. The file should have restrictive permissions and, ideally, a MAC/checksum so modifications are detected.


Correctness Issues

5. wrapper.rs rewrite is a breaking change (net -1180 lines)

The PR replaces the entire WASM tool wrapper with a new implementation that uses the low-level Val API instead of wasmtime::component::bindgen!(). This drops:

  • WasiView implementation (WASI context)
  • All HTTP host functions (http-request with timeout, headers, body)
  • Credential injection (placeholder substitution, host-based injection, redaction)
  • OAuth token refresh (OAuthRefreshConfig)
  • LeakDetector integration in WASM responses
  • tool-invoke and secret-exists host function bindings
  • The bindgen!() macro entirely (replaced with manual Val marshalling)

The WIT file also removes the timeout-ms parameter from http-request. Only log, now-millis, workspace-read, and the new sign-payload are wired up in the new wrapper. This means all existing WASM tools that use HTTP, secrets, tool invocation, or credentials will break. This seems unintentional or at least needs to be called out in the PR description.

6. Spend tracking not recorded on approval-then-sign path (src/keys/mod.rs:314)

Spend is only recorded in the AutoApprove branch of sign_transaction:

PolicyDecision::AutoApprove => {
    // ...signs...
    if analysis.total_value_yocto > 0 {
        let _ = self.spend_tracker.record_spend(...).await;
    }
}

But when approval is granted and the transaction is subsequently signed (the ApprovalRequired path), there is no code path that records the spend. The daily limit can be bypassed by always going through the approval flow for small transactions.

7. _domain parameter unused in build_chain_signature_action (src/keys/chain_signatures.rs:168)

The SignatureDomain parameter is accepted but ignored. The generated sign call doesn't pass the domain to the MPC contract. This could result in signing with the wrong curve if the contract defaults differ from what the caller expects.


Design Feedback

8. Flat file storage for keys metadata and spend tracking

Both keys.json and spend_tracking.json are read-modify-write JSON files. This has no concurrency safety (see #1), no atomicity (a crash mid-write corrupts the file), and doesn't follow the project's Database trait pattern. Per CLAUDE.md: "All new features that touch persistence MUST support both backends." Consider:

  • Atomic writes (write to temp file, then rename)
  • Or migrate to the Database trait with proper implementations

9. Default policy is too permissive for full-access keys

PolicyConfig::default() sets deny_full_access_operations: false, meaning a full-access key can be used for transfers, function calls, etc. The PR description says "high-value operations require explicit user approval" but the default policy auto-approves zero-value transfers from full-access keys (because transfer_auto_approve_max_yocto: 0 means "up to 0 yocto is auto-approved"). While this technically blocks non-zero transfers, function calls with zero deposit through a full-access key on an arbitrary contract are also auto-approved if the key is scoped. Consider defaulting deny_full_access_operations: true to match the "hybrid custody" philosophy.

10. let _ = self.secrets_store.delete(...) silently ignores deletion errors (src/keys/mod.rs:265)

In remove_key, if the secret deletion fails but the metadata is already removed, the key becomes orphaned in the secrets store with no way to clean it up. At least log the error.


Style / Minor

11. let-else refactoring in capabilities.rs and leak_detector.rs

The PR refactors if let ... && ... chains into nested if let / if blocks. This is fine for compatibility, but the commit message doesn't mention it. These should be in a separate commit or at least noted in the PR description since they touch security-critical code paths (endpoint matching, leak detection).

12. format_yocto has precision loss for amounts between 1 milliNEAR and 1 NEAR

let frac = (yocto % ONE_NEAR) / ONE_MILLI_NEAR;

This integer division truncates. E.g., 1.999 NEAR formats as "1.999 NEAR" but 0.001999 NEAR formats as "0.001 NEAR" (the 999 microNEAR is lost). For a financial display, consider at least 6 decimal places.

13. No #[cfg(test)] on InMemorySecretsStore import

The tests in mod.rs use InMemorySecretsStore -- make sure this type is available at test time (it appears to be, but worth a compilation check with --no-default-features).


Summary

The core key management design is solid -- encrypted storage, Zeroize on drop, keys never reaching WASM, policy-before-sign pipeline. But the TOCTOU race, the leak detector regression, and the wrapper.rs rewrite breaking existing WASM tools are blockers. The spend tracking gap is a financial correctness bug that should also be fixed before merge.

Recommended disposition: Request changes on items 1, 2, 5, 6 before merging.

ilblackdragon added a commit that referenced this pull request Apr 14, 2026
…, binary writes

- Add pre-intercept safety param validation so sandbox-dispatched calls
  go through the same checks as host-dispatched calls (#1)
- Set network_mode: "none" on sandbox containers to prevent outbound
  network access (#3)
- Reject binary content in containerized write instead of silently
  corrupting via from_utf8_lossy (#5)
- Cap list_dir depth to 10 to prevent unbounded traversal (#8)
- Change container creation log from info! to debug! to avoid breaking
  REPL/TUI output (#10)
- Make is_truthy case-insensitive so SANDBOX_ENABLED=True works (#11)
- Return error instead of unwrap_or_default for missing container ID (#12)
- Propagate set_permissions errors instead of silently ignoring (#13)
- Return error for missing daemon output key instead of defaulting to
  empty object (#14)
- Add env mutex guard in sandbox_live_e2e test (#15)
- Fix rustfmt formatting for let-chain in canonicalize_under_root

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ilblackdragon added a commit that referenced this pull request Apr 19, 2026
* feat(engine-v2): mount-backend abstraction for per-project sandbox (Phase 1)

Adds the engine-side `MountBackend` trait + minimal `WorkspaceMounts` registry
and a host-side bridge interceptor that routes sandbox-eligible tool calls
(`file_read`, `file_write`, `list_dir`, `apply_patch`, `shell`) through a
backend when their path argument starts with `/project/`. Default behavior is
unchanged: until `EffectBridgeAdapter::set_workspace_mounts(Some(...))` is
called (Phase 6), the interception path is dormant.

This is the first phase of the per-project sandbox plan
(`docs/plans/2026-04-10-engine-v2-sandbox.md`) and a deliberately small subset
of the unified Workspace VFS proposed in #1894 — just enough
abstraction so the sandbox can be a `MountBackend` rather than a special case
in the bridge. When #1894's full mount table lands, the sandbox backend slots
in unchanged.

Engine crate (`crates/ironclaw_engine/src/workspace/`):
- `mount.rs` — `MountBackend` trait, `MountError` (NotFound / InvalidPath /
  PermissionDenied / Io / Tool / Backend / Unsupported), `DirEntry`,
  `EntryKind`, `ShellOutput`
- `filesystem.rs` — `FilesystemBackend`: passthrough host-fs implementation
  with two-layer path validation (lexical reject of absolute / `..`, then
  symlink-escape canonicalization). `read`/`write`/`list` fully implemented;
  `patch`/`shell` return `Unsupported` so the bridge falls through to the
  host tool until Phase 5
- `registry.rs` — `WorkspaceMounts` per-project registry with lazy
  `ProjectMountFactory`, longest-prefix-match resolution, cached and
  invalidatable

Bridge (`src/bridge/sandbox/`):
- `intercept.rs` — `maybe_intercept` and `SANDBOX_TOOL_NAMES`. Returns
  `Handled(json)` on a successful backend dispatch, `FellThrough` for
  non-sandbox tools, host paths, missing path params, or `Unsupported`
  backend ops
- `effect_adapter.rs` — `workspace_mounts` field + `set_workspace_mounts`
  setter; interception block in `execute_action_internal` right before
  `execute_tool_with_safety`, gated on the optional mount table

Tests (31 new):
- 17 engine workspace unit tests covering trait error mapping, path safety
  (lexical + symlink), longest-prefix routing, and lazy factory caching
- 9 bridge sandbox unit tests including `intercept_actually_dispatches_into_backend`
  (counting backend) which proves the interceptor reaches the backend
- 5 integration tests in `tests/engine_v2_sandbox_integration.rs` driving
  `EffectBridgeAdapter::execute_action()` end-to-end per the
  "Test Through the Caller" rule (`.claude/rules/testing.md`), including
  a host-path-falls-through test that asserts the sandbox tempdir was
  not touched, and a `..`-escape test that verifies no `/etc/passwd`
  content leaks even after safety-layer redaction

Drive-by: feature-gate two pre-existing dead-code helpers in
`crates/ironclaw_skills/src/parser.rs` on `#[cfg(feature = "registry")]` to
match their only call site, fixing a pre-existing clippy warning that blocked
the workspace's `-D warnings` policy when `ironclaw_skills` is built with
`default-features = false` (as the engine crate does).

Verification:
- `cargo fmt --check` clean
- `cargo clippy --all --benches --tests --examples --all-features` zero warnings
- 31 / 31 new tests passing; no existing tests broken

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine-v2): per-project sandbox — Phases 2–7 + live Docker e2e test

Completes the per-project sandbox plan (docs/plans/2026-04-10-engine-v2-sandbox.md
Phases 2–7), building on Phase 1's mount-backend abstraction (#2211).

Phase 2 — Project workspace folder:
- `Project.workspace_path: Option<PathBuf>` field + `with_workspace_path()`
- Host-side `project_workspace_path()`, `ensure_project_workspace_dir()` (creates
  `~/.ironclaw/projects/<id>/` mode 0700, idempotent)
- `FilesystemMountFactory` taking a `ProjectPathResolver` closure (decoupled from
  `Store`); wired into `EffectBridgeAdapter` via `set_workspace_mounts()`

Phase 3 — Standalone daemon binary:
- `src/bin/sandbox_daemon.rs` — NDJSON over stdin/stdout, health/shutdown/execute_tool
- Constructs ReadFileTool/WriteFileTool/ListDirTool/ApplyPatchTool/ShellTool with
  `base_dir=/project` (override via `IRONCLAW_SANDBOX_BASE_DIR`)

Phase 4 — Dockerfile.sandbox:
- Multi-stage build: rust-slim builder (+ python3 for pyo3) compiles sandbox_daemon;
  debian-slim runtime with tini PID 1, common build tools, `/project` mount target

Phase 5 — ProjectSandboxManager + ContainerizedFilesystemBackend:
- protocol.rs: Request/Response/RpcError matching daemon wire format
- transport.rs: `SandboxTransport` trait (seam for testing without Docker)
- containerized_backend.rs: `ContainerizedFilesystemBackend` impls `MountBackend`,
  translates relative→`/project/<rel>`, maps tool-error→MountError
- docker_transport.rs: real bollard exec session, serialized Mutex, lazy reconnect
- lifecycle.rs: deterministic `ironclaw-sandbox-<pid>` naming, ensure_running/stop/remove
- manager.rs: `ProjectSandboxManager` per-project transport cache

Phase 6 — Router gating on ENGINE_V2_SANDBOX:
- `engine_v2_sandbox_enabled()` helper (truthy: 1/true/yes/on)
- Router selects `ContainerizedMountFactory` when enabled + Docker reachable;
  falls back to `FilesystemMountFactory` with warning otherwise

Live e2e bugs caught and fixed:
- Shell without explicit `workdir` defaulted to host (not sandbox); fixed by
  defaulting to `/project/` in `extract_path_param`
- `ContainerizedFilesystemBackend::shell` parsed `stdout`/`stderr` but host
  ShellTool returns merged `output` field; fixed with fallback key lookup
- SANDBOX_TOOL_NAMES only had v2 names (`file_read`/`file_write`) but host
  registry uses v1 names (`read_file`/`write_file`); added both aliases

Tests (62 sandbox-related, all green):
- 27 bridge sandbox unit tests (intercept, workspace_path, factory, protocol,
  lifecycle, containerized_backend with ScriptedTransport mock)
- 7 containerized-backend tests (including 2 regression tests for the shell bugs)
- 5 engine v2 sandbox integration tests (EffectBridgeAdapter end-to-end)
- 5 daemon binary smoke tests (real subprocess + NDJSON I/O)
- 17 engine workspace unit tests
- 1 live Docker e2e test: agent clones nearai/ironclaw into sandbox, renames
  to megaclaw via sed, verifies with grep — 70s, $0.09, recorded trace committed

Verification:
- `cargo fmt --check` clean
- `cargo clippy --all --benches --tests --examples --all-features` zero warnings
- All 62 sandbox tests passing; no existing tests broken

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: replace .expect() with Result in DockerTransport::ensure_session

CI's no-panics checker flagged the .expect("just inserted") in production
code. Replace with .ok_or_else() returning MountError::Backend.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: multi-tenant project paths + unify sandbox env var with v1

Two issues addressed:

1. Project workspace paths now namespace by user_id:
   `~/.ironclaw/projects/<user_id>/<project_id>/` instead of
   `~/.ironclaw/projects/<project_id>/`. Prevents filesystem collisions
   in multi-tenant deployments where two users could theoretically have
   the same project UUID.

2. Sandbox enablement now reads `SANDBOX_ENABLED` (same env var as v1
   sandbox) in addition to `ENGINE_V2_SANDBOX`. Either being truthy
   enables the per-project sandbox. This means a single flag governs
   sandbox behavior regardless of engine version, while the v2-specific
   override remains available for transitional setups.

Tests: 30 bridge sandbox unit tests passing (added multi-tenant path
tests + env var combination tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review — TOCTOU race, shell env passthrough, canonicalize guard

Three issues flagged by the code review bot on #2211:

1. TOCTOU race in WorkspaceMounts::resolve (HIGH): Added double-checked
   locking — re-check the cache after acquiring the write lock so two
   threads racing on the same project's first access don't both call
   factory.build(). The second thread finds the insert from the first.

2. Shell intercept ignores env parameter (MEDIUM): The shell arm in
   maybe_intercept was passing HashMap::new() instead of forwarding
   the tool call's env map. Fixed to parse parameters["env"] and pass
   it through to backend.shell().

3. Canonicalization fails when root doesn't exist (MEDIUM): When
   self.root hasn't been created yet (first write to a new project),
   canonicalize_under_root would walk up to a real ancestor and the
   starts_with check against the non-existent root would always fail.
   Now skips canonicalization entirely when root doesn't exist — lexical
   safety is already guaranteed by safe_join.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review round 2 — apply_patch schema, content validation, dir perms, docs

- Fix apply_patch schema mismatch: MountBackend::patch now takes
  (old_string, new_string, replace_all) matching ApplyPatchTool's
  actual contract. Previously sent {patch: diff} which would fail
  with invalid_params in the containerized daemon.
- Validate file_write content param: return error instead of silently
  writing empty string when content is missing.
- Log stderr frames from sandbox daemon at debug! instead of silently
  discarding them in docker_transport StreamReader.
- Tighten permissions on intermediate directories created by
  ensure_project_workspace_dir (projects/, <user_id>/) to 0o700,
  not just the leaf.
- Fix stale module doc in sandbox/mod.rs (referenced "Phase 5 will
  add" but all phases shipped).
- Fix doc path mismatch: workspace path is <user_id>/<project_id>/,
  not <project_id>/ (workspace_path.rs, CLAUDE.md, design plan).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review round 3 — symlink safety, visibility, debug logging

- Close TOCTOU window in canonicalize_under_root: re-canonicalize and
  verify containment when the reassembled path exists on disk
- Fix list_dir_recursive: use symlink_metadata (lstat) so symlinks are
  detected instead of followed; validate directories against root before
  recursive traversal
- Tighten is_mountable_path to /project/, /memory/, /home/ prefixes
  instead of any absolute path (defense-in-depth)
- Narrow sandbox module visibility to pub(crate) and remove unused
  pub use re-exports
- Remove concrete types (FilesystemBackend, DirEntry, EntryKind,
  ShellOutput) from engine crate top-level re-exports; access via
  ironclaw_engine::workspace:: module path
- Add debug! tracing to sandbox intercept routing decisions
- Add read_file/write_file v1 aliases to daemon SUPPORTED_TOOLS health
  response
- Remove developer-local path from sandbox mod.rs doc comment
- Merge staging to fix CI (user_timezone field on ThreadExecutionContext)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review round 4 — safety validation, network isolation, binary writes

- Add pre-intercept safety param validation so sandbox-dispatched calls
  go through the same checks as host-dispatched calls (#1)
- Set network_mode: "none" on sandbox containers to prevent outbound
  network access (#3)
- Reject binary content in containerized write instead of silently
  corrupting via from_utf8_lossy (#5)
- Cap list_dir depth to 10 to prevent unbounded traversal (#8)
- Change container creation log from info! to debug! to avoid breaking
  REPL/TUI output (#10)
- Make is_truthy case-insensitive so SANDBOX_ENABLED=True works (#11)
- Return error instead of unwrap_or_default for missing container ID (#12)
- Propagate set_permissions errors instead of silently ignoring (#13)
- Return error for missing daemon output key instead of defaulting to
  empty object (#14)
- Add env mutex guard in sandbox_live_e2e test (#15)
- Fix rustfmt formatting for let-chain in canonicalize_under_root

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review round 5 — path traversal, error types, tests

Security fixes:
- Sanitize user_id in workspace path to prevent directory traversal via
  malicious user IDs containing `..` or `/`
- Add Component::ParentDir check in ContainerizedFilesystemBackend::container_path
  matching the defense-in-depth approach of FilesystemBackend::safe_join

Correctness:
- Use MountError::Tool instead of MountError::InvalidPath for missing
  tool parameters (content, old_string, new_string) — fixes confusing
  LLM-visible error messages
- Fix clippy sort_by_key suggestion in registry.rs

Cleanup:
- Remove spurious Notify import and dead _notify_link function

New tests:
- ContainerizedFilesystemBackend path traversal rejection (read + write)
- container_path unit tests for safe and unsafe paths
- Adversarial user_id test in workspace_path
- Daemon-side path traversal test in sandbox_daemon_smoke

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review round 6 — param normalization, error types, edge cases

- Normalize sandbox params via prepare_tool_params() before validation,
  matching the host execution path (fixes inconsistent validation)
- Return ToolError::InvalidParameters instead of EngineError::Effect for
  sandbox param validation failures (consistent error surface)
- ensure_dir checks path.is_dir() not path.exists() (rejects files)
- Empty user_id returns "_anonymous" sentinel instead of empty hex string
  that would drop the tenant namespace via PathBuf::join("")
- Restore ENGINE_V2_SANDBOX env var after sandbox live E2E test
- Tighten is_mountable_path to /project/ only (no mounts for /memory/
  or /home/ yet)
- Add v1 tool name aliases (read_file, write_file) to SUPPORTED_TOOLS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: unify sandbox env var — remove ENGINE_V2_SANDBOX, use SANDBOX_ENABLED only

Single env var controls sandboxing for both engine versions. The
transitional ENGINE_V2_SANDBOX override is removed from code, tests,
docs, and Dockerfile.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: double-checked locking in transport_for, explicit stdin close in smoke test

- ProjectSandboxManager::transport_for no longer holds the mutex across
  the Docker ensure_running await. Uses double-checked locking so
  concurrent projects initialize in parallel.
- sandbox_daemon_smoke: explicitly take() stdin before wait_with_output
  so EOF is sent even without a shutdown request.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review — network mode, error types, race, protocol dedup

- Change sandbox container network_mode from "none" to default bridge
  so git clone / cargo build / pip install work inside the container
- Fix binary content rejection to use MountError::Tool instead of
  MountError::InvalidPath (semantic mismatch)
- Fix list depth: use actual depth value instead of depth.max(1)
- Fix orphan container race in transport_for by holding lock across
  container creation instead of double-checked locking
- Deduplicate protocol types: daemon now imports from shared
  bridge::sandbox::protocol instead of defining its own copies
- Make bridge::sandbox pub (narrow exposure: only protocol and
  workspace_path sub-modules are pub)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update plan doc — sandbox uses bridge networking, not network_mode=none

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ilblackdragon added a commit that referenced this pull request Apr 24, 2026
Addresses the second round of review feedback on the budgets PR.

**Audit events (#15, missing side effect):**
- BudgetEnforcer now emits record_budget_event on every reserve /
  reconcile / release / deny / rollback path so the budget_events
  table actually gets populated.
- ReservationTicket gained actor_user_id + thread_id fields so
  reconcile/release can emit audit rows without a second DB lookup.
- record_audit helper swallows write errors (tracing::warn!) — the
  business-critical op has already committed.
- Two regression tests assert the shape: a successful reserve +
  reconcile produces [Reserve, Reconcile]; a denial on the second
  cascade level produces [Reserve, Deny, Release(rollback)].

**u64 → i64 overflow (#8, #17, #18):**
- save_budget (both backends), record_budget_event (both backends),
  reconcile_reservation (both backends) now use i64::try_from with
  DatabaseError on overflow. Prevents negative BIGINT storage from
  silent `as i64` wrap on tokens / wall_clock / actual_tokens.

**reserve_within_tx returned-ledger divergence (#14):**
- libSQL path was returning a BudgetLedger with tokens_used
  incremented by requested_tokens, but the SQL UPDATE did not
  persist that change — the returned struct diverged from the
  stored state. tokens_used now matches what the DB holds; tokens
  settle on reconcile only.

**Docs (#11, #12, #13, #16):**
- BudgetPeriod::Rolling24h docstring now says "quantised to UTC
  midnight, not sliding-window" to match the implementation.
- BudgetPeriod::Calendar docstring calls out the UTC-only caveat
  (no chrono-tz dep yet).
- BudgetPeriod::PerInvocation docstring explains each call gets
  its own ledger by design; use Rolling24h for cross-call caps.
- BudgetLimit::tokens and the in-enforcer token-cap check now
  document the check as best-effort (not atomic under concurrent
  reserves); USD remains authoritative.

**Test isolation (#10):**
- bash_max_output_length_env_var_is_respected was mutating the
  process env without restoring. Introduced BashEnvGuard (RAII
  over a module-static Mutex) that snapshots and restores
  BASH_MAX_OUTPUT_LENGTH, and gated test_large_output_command
  behind the same guard so the two can't race.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
errol-t3 referenced this pull request in Terminal-3/t3-claw May 14, 2026
…#14)

Ported from 88351c3 on origin/main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ilblackdragon added a commit that referenced this pull request May 16, 2026
…check

Four follow-ups from review of 87fc8d4:

- responses_api.rs (Copilot #11): the byte-counter comment claimed it
  measured "what the caller actually sent over the wire" but the count
  is canonicalised JSON, not raw request bytes. Rewrote the comment to
  describe the canonicalised-size cap correctly. The `Serializer`
  import is kept — it's needed to bring the trait method into scope
  for the concrete `serde_json::Serializer::serialize_seq` call below
  (an earlier attempt to drop it failed to compile).

- responses_api.rs (Copilot #12): caller-supplied tool names were only
  checked for non-empty + uniqueness. Whitespace, control chars, and
  over-long names could propagate into engine action surfaces, SSE
  payloads, and downstream LLM clients (which all enforce the OpenAI
  Responses spec `^[A-Za-z0-9_-]{1,64}$` and would reject anyway).
  Validate at request time. Added regression tests covering
  whitespace/control/non-ASCII names and the length cap.

- responses_api.rs / bridge/router.rs / bridge/mod.rs (Copilot #13):
  the shadow check only consulted `ToolRegistry::tool_definitions()`
  and missed engine v2 capability actions (`mission_*`, `skill_*`,
  `memory_*`, etc.). A caller registering `mission_create` as an
  external tool would still hit the catalog short-circuit in
  `EffectBridgeAdapter::execute_action`, since the LLM-visible dedup
  in `available_action_inventory` doesn't extend to the execute path.
  Added `engine_capability_action_names()` accessor that pulls the
  full capability-action surface from the bridge's
  `CapabilityRegistry`, and merged it into the collision set the
  responses_api handler checks against.

- runtime/conversation.rs (Copilot #14): documented that
  `extra_initial_metadata` is spawn-only. The `Running` (inject) and
  `Resumable` (resume) branches ignore it; callers needing
  per-request metadata on existing threads must use
  `ThreadManager::set_thread_metadata` instead. The bridge's
  external-tool-catalog `transfer` already handles this for the one
  in-tree caller, but documenting the contract prevents future
  surprise.
ilblackdragon added a commit that referenced this pull request May 16, 2026
* feat(web): support externally-provided tools in Responses API

Lets callers of `/v1/responses` (and `/api/v1/responses`) declare their
own `function`-typed tools and feed back results via
`function_call_output` items, matching the OpenAI Responses wire shape.

Since IronClaw's engine has no per-request tool surface, integration
happens at the prompt level: the catalog is rendered as
`<external-tools>` in the user message and the agent signals a call by
ending its response with a fenced ```` ```tool_call ```` block. When
that fence is recognised, the reply is split into a leading `Message`
plus a `function_call` `ResponseOutputItem`.

Validation rejects unsupported tool types (`web_search`, `file_search`,
`code_interpreter`) and tools missing `name` with 400, with two new
integration tests covering both paths.

* refactor(responses-api): switch external tools to engine v2 native path

Replace the prompt-level fence protocol from PR #3122 with engine v2
native tool calls: caller-supplied `tools[]` are surfaced as real
LLM-callable actions, the engine pauses with `ResumeKind::External`
when one is invoked, and the bridge router projects the pause to a
new `AppEvent::ExternalToolCall` carrying the OpenAI-shaped
`function_call` wire fields.

The integration is small because v2 already has the right primitives:

- `ResumeKind::External { callback_id }` and
  `GateResolution::ExternalCallback { payload }` already existed for
  OAuth-style callbacks.
- `agent_loop.rs:1480` already routes Responses API messages to
  `handle_with_engine` when `ENGINE_V2=true`, so no v2 migration of
  the endpoint itself is needed.
- `EffectBridgeAdapter::execute_action` is the single chokepoint
  where caller tools can be detected before they reach the dispatch
  pipeline.

Changes:

- New `src/bridge/external_tools.rs` (`ExternalToolCatalog`) — per-thread
  registry of caller-supplied `ActionDef`s, plus the `ext_tool:`
  callback-id helpers used to disambiguate external-tool pauses from
  OAuth/pairing pauses (which also use `ResumeKind::External`).
- `EffectBridgeAdapter` consults the catalog: any name in it is
  short-circuited to a `GatePaused { resume_kind: External {
  callback_id: ext_tool:<call_id> } }` before any registry dispatch,
  and `available_action_inventory` merges the catalog into the
  LLM-visible action surface (internal beats external on collision).
- `Submission::ExternalCallback` gains an optional `payload` field;
  `bridge::handle_external_callback` plumbs it into
  `GateResolution::ExternalCallback { payload }`. Fallback predicate
  `gate_resume_is_external` lets non-auth External pauses (i.e.
  caller-tool resumes) resolve through the same handler.
- New `AppEvent::ExternalToolCall` projected by `notify_pending_gate`
  when a paused gate carries an `ext_tool:` callback id; OAuth/
  pairing flows keep flowing through the existing `GateRequired`
  channel.
- `responses_api.rs` is gutted of the prompt rendering and fence
  parsing (`render_external_tools_preamble`, `extract_trailing_tool_call`,
  `parse_external_tool_call`, `ParsedToolCall`, `external_tool_names`
  accumulator field, and the `TOOL_CALL_FENCE` constants). The handler
  now: rejects `tools[]` when `ENGINE_V2=false`, registers caller
  tools in the catalog under the resolved thread id, detects resume
  requests (`previous_response_id` + `function_call_output` items in
  `input`) and submits them as `Submission::ExternalCallback` with
  the outputs as the resolution payload, and surfaces
  `AppEvent::ExternalToolCall` as a `function_call` `ResponseOutputItem`
  in both streaming (`output_item.added`+`done`) and non-streaming.
- All existing OAuth/pairing `ExternalCallback` constructors updated
  to pass `payload: None` (no behaviour change).
- Fence-protocol unit tests removed; replaced with coverage for the
  new `responses_tools_to_action_defs` converter and the accumulator's
  `ExternalToolCall` arm.

Existing 9 integration tests in `tests/responses_api_path_prefix.rs`
still pass.

Note for reviewers:
- The accumulator-side text response no longer tries to split the
  reply on a fenced `tool_call` block. The wire shape that callers
  receive for caller-tool invocations is purely event-driven now.
- Internal vs external collision is handled silently by the dedup in
  `available_action_inventory` (internal wins). A request-time
  rejection for shadowing names is a follow-up — the current behavior
  is safe (the LLM only sees the internal version) but could surprise
  a caller who expects their tool to run.

* test(responses-api): cover ENGINE_V2-off and resume-without-pending-gate

Two new integration tests for behaviours added by the engine-native
external-tool refactor:

- `external_tools_rejected_when_engine_v2_disabled`: a request with
  caller-supplied `tools[]` while `ENGINE_V2` is off must 400 with a
  message naming the flag, not silently fall through.
- `resume_without_pending_gate_returns_400`: a request with
  `function_call_output` items and a `previous_response_id` that
  doesn't correspond to a live external-tool gate must 400, not start
  a fresh turn against the (unrelated) thread.

Both tests drive the full router (`start_test_server` + bearer auth)
per `.claude/rules/testing.md` "Test Through the Caller".

* test(responses-api): integration tests + drop unsafe env mutation

Three groups of changes:

1. **Drop unsafe env-var mutation in tests.** `responses_api.rs` no
   longer reads `ENGINE_V2` directly: it keys off the presence of the
   live `ExternalToolCatalog` (initialized by `init_engine`) as the
   "engine v2 is up" signal. The path-prefix test that exercises the
   no-engine branch no longer needs `unsafe { std::env::remove_var }`
   — the absence of `init_engine` in `TestGatewayBuilder` is what
   makes the catalog absent, which is what makes the request reject.

2. **Engine-tier integration tests** (`tests/e2e_responses_api_external_tools.rs`).
   Drives engine v2 via the existing `TestRigBuilder` + `TraceLlm`
   replay infrastructure rather than spinning up an HTTP gateway:

   - `catalog_isolates_by_thread_id` — register-under-A doesn't bleed
     into B.
   - `catalog_register_overwrites_not_merges` — Responses API contract
     is "each request restates the full tools[]"; the catalog must
     replace, not merge.
   - `catalog_sweep_evicts_only_stale_entries` — TTL backstop.
   - `catalog_clear_on_terminal_state_explicit` — what we want; the
     test name flags that the production hook is missing.
   - `catalog_handles_concurrent_registrations` — 32 concurrent
     register-then-contains tasks; protects per-thread isolation
     under contention.
   - `callback_id_disambiguates_external_from_oauth` — `ext_tool:`
     vs `pairing:` prefix is the single bit that routes a paused gate
     to `AppEvent::ExternalToolCall` vs `AppEvent::GateRequired`. If
     the prefix invariant breaks, the wrong UI renders.

   Two more tests deliberately `#[ignore]` and document concrete bugs
   the implementation has today:

   - `engine_pauses_when_llm_calls_registered_external_tool` — running
     it surfaces "engine never paused on external tool". Confirms the
     **thread-id mismatch** bug: the catalog is keyed by engine
     `ThreadId`, but the responses_api handler registers under a
     separately-generated UUID before the engine spawns the thread.
   - `round_trip_resume_payload_reaches_llm` — the load-bearing
     end-to-end test. Documents the **resume-payload-not-materialised**
     bug: `bridge::router::resolve_gate` uses `pending.resume_output`
     for `ExternalCallback` resolutions and ignores the payload, so
     caller-supplied tool outputs never reach the LLM's context.
   - `external_collision_with_registry_action_is_rejected` — stub
     asserting the desired validation behaviour for caller tool
     names that shadow registry actions; today silently accepted,
     and the catalog-wins-in-dispatch ordering means the LLM thinks
     it called the internal tool but actually ran caller code.

   The two #[ignore] tests are the deliberate failure documentation:
   running them with `--ignored` panics with messages naming the
   underlying gap. The fixes go on a follow-up commit.

3. **`TestRigBuilder.send_external_callback_with_payload`** — new
   helper that mirrors the OAuth `send_external_callback` but carries
   a JSON payload. Used by the round-trip test; the existing
   payload-less variant kept for OAuth/pairing tests.

Quality gates: `cargo fmt`, `cargo clippy --all --tests --all-features`
clean. 14 of 17 tests pass; the 3 ignored ones are deliberate
documentation of the gaps.

* fix(responses-api): close 4 bugs surfaced by integration tests

The engine-native external-tools path landed in 44135ca had four
real bugs surfaced by the integration tests in a2c13ee. This commit
fixes all four; every previously-ignored test now passes.

**Bug 1 — Thread-id mismatch.** `responses_api.rs` registers tools in
the catalog under a `thread_uuid` it generates from `previous_response_id`
(or freshly), but `ConversationManager::handle_user_message` creates
the engine's *actual* `ThreadId` internally. The catalog entries were
under a UUID the engine never executed in.

Fix: `ExternalToolCatalog::transfer(from, to)` and a hook in
`bridge::handle_with_engine_inner` that calls it after the engine
returns the spawned `ThreadId`. The handler-supplied conversation
scope UUID is rebound onto the actual ThreadId before the LLM call
lands, so `EffectBridgeAdapter::execute_action`'s catalog check
finds the registered tools.

**Bug 2 — Resume payload never materialised.** `bridge::router::resolve_gate`'s
`GateResolution::ExternalCallback` branch only consulted
`pending.resume_output` to construct the resumed `ActionResult`.
`EffectBridgeAdapter::execute_action` sets that to `None` for
caller-tool gates (the output isn't known at gate-fire time), so
the resume fell through to `execute_pending_gate_action`, which
re-ran the original action — re-pausing forever. The caller's tool
output (passed via `GateResolution::ExternalCallback { payload }`)
was dropped on the floor.

Fix: special-case `ext_tool:` callback ids in `resolve_gate`'s
ExternalCallback branch. Extract the matching output from the
payload (Responses API wire shape: `{ outputs: [{ call_id, output }] }`)
via the new `extract_external_tool_output` helper, synthesise an
`ActionResult`-shaped ThreadMessage, and resume the thread directly.
OAuth/pairing flows (which use `pairing:` callback ids and don't
carry an output payload) keep the original `pending.resume_output`
path unchanged.

**Bug 3 — Internal/external collision in dispatch.** The catalog
short-circuit in `EffectBridgeAdapter::execute_action` ran before
the registry, but `available_action_inventory` dedupes the opposite
way (internal beats external in the LLM-visible list). Result: an
LLM call to (say) `shell` would land in caller-side execution even
though the LLM saw the *internal* `shell` description in its action
surface — a confused-deputy where the caller can return any output
and the LLM trusts it as the internal tool's reply.

Fix: reject the collision at request validation in `responses_api.rs`.
After `validate_external_tools(...)`, look up registered tool names
via `state.tool_registry.tool_definitions()` and 400 any caller name
that shadows an internal action.

**Bug 4 — Catalog cleanup never happens.** `sweep_older_than` existed
but nothing scheduled it, and there was no terminal-state hook —
so catalog entries accumulated for every thread that ever ran.

Fix:
- `await_thread_outcome` in the bridge router calls
  `catalog.clear(thread_id)` on every non-`GatePaused` outcome
  (Completed, Stopped, MaxIterations, Failed). `GatePaused` keeps
  the entry so resume requests can still find it.
- A periodic sweep task in `init_engine` runs
  `catalog.sweep_older_than(1 hour)` every 5 minutes as a backstop
  for callers that abandon a paused thread without resuming.

**Tests now passing:**
- `engine_pauses_when_llm_calls_registered_external_tool` — proves Bug 1.
- `round_trip_resume_payload_reaches_llm` — proves Bug 2 (and
  Bug 1 by extension).
- `external_tool_name_shadowing_registered_action_is_rejected` — proves Bug 3.
- `catalog_cleared_on_terminal_completed_outcome` — proves Bug 4.

Plus four catalog-tier unit tests for the new `transfer` method, and
the two pre-existing collision-prevention tests
(`ext_tool:` vs `pairing:` callback id disambiguation).

`TestGatewayBuilder.tool_registry(...)` is a new builder hook for
tests that need to exercise the registry-aware handler paths
(currently only the collision-rejection test, but the seam is there
for future ones).

Quality gates: `cargo fmt`, `cargo clippy --all --benches --tests
--examples --all-features` clean. 17/17 engine v2 tests pass; 12/12
HTTP path-prefix tests pass.

* fix(responses-api): address review feedback from PR #3122

Closes the race-window where caller-supplied external tools could be
invisible to the LLM on a thread's first turn, plus a batch of smaller
review findings.

Race fix (Bug #3 from review):
- Plumb `conversation_scope: Option<Uuid>` through
  `ThreadExecutionContext`, populated from thread metadata.
- `ConversationManager::handle_user_message` accepts an
  `extra_initial_metadata` map; the bridge stamps the parsed scope into
  it so the engine sees it on the in-memory thread the executor task
  reads from (post-spawn `set_thread_metadata` is invisible to that
  task).
- `EffectBridgeAdapter::execute_action` and
  `available_action_inventory` now look up the catalog under both
  `ctx.thread_id` and `ctx.conversation_scope`, so the executor task
  that starts immediately after spawn can find caller tools even
  before the bridge's post-spawn `transfer` rebinds them onto the
  engine `thread_id`. The transfer remains for terminal-state
  cleanup bookkeeping.

Other review fixes:
- Rewrite the stale `## Externally-provided tools` module doc in
  `responses_api.rs` to describe the engine-native flow (the original
  prompt-level fence text was left over from the first commit on the
  branch).
- `validate_external_tools` size check now fails closed on
  serialization error (`unwrap_or(MAX + 1)`) instead of silently
  passing oversized payloads.
- `notify_pending_gate` debug-logs the no-broadcaster path so a
  future SSE-less channel that grows an external-tool surface can
  be diagnosed instead of silently hanging.
- Update `catalog_clear_on_terminal_state_explicit` test docstring
  and rename to `catalog_clear_removes_entry` — Bug 4's cleanup hook
  in `await_thread_outcome` already wires the production cleanup
  (covered separately by `catalog_cleared_on_terminal_completed_outcome`).
- Delete dead `wait_for_first_engine_thread` helper and the
  `_harness_compiles` shim that kept it alive.

New test coverage:
- `bridge::effect_adapter::tests`: three race-window regression tests
  exercising both `available_action_inventory` and `execute_action`
  via the conversation_scope fallback path, plus a unit test on the
  `external_tool_catalog_keys` helper.
- `bridge::router::tests`: four `extract_external_tool_output` tests
  covering match-by-call_id, missing-call_id-returns-null,
  no-outputs-array fallback, and find-after-misses.
- `channels::web::responses_api::tests`: a full streaming round-trip
  test driving `streaming_worker` end-to-end with synthetic
  StreamChunk + ExternalToolCall events, parsing the actual SSE
  byte stream, and asserting the wire-frame ordering an OpenAI
  client would observe.

* test: fix three pre-existing engine test failures

- `executor::structured::call_id_preserved_when_no_lease`: the test
  was asserting on an error string ("no lease") that the preflight
  path stopped emitting when it added the "not callable in this
  execution context" check ahead of the lease lookup. The empty
  `MockEffects` used by the test exposed no actions, so the call
  short-circuited before reaching the lease check the test name
  describes. Register `web_search` in the inventory so the lease-miss
  path the test is named for actually fires.

- `runtime::manager::stop_thread_works`: the test races the
  consecutive-action-error guard added in #2325. The thread loops on
  a deliberately unregistered `test_tool`, and 7 errors land before
  the 10ms sleep + `stop_thread` round-trip can deliver the stop
  signal in fast environments. The test's intent is "calling
  `stop_thread` doesn't deadlock the join", so accept `Failed` as a
  valid terminal outcome in addition to `Stopped`/`Completed`/
  `MaxIterations`. Asserts a more specific message on the join
  result so a true regression (non-terminal outcome) still trips.

- `tests::catalog_cleared_on_terminal_completed_outcome`: was racing
  two ways. (1) The cleanup-poll compared `catalog.len()` to a
  pre-snapshot — racy because `engine_external_tool_catalog` is
  process-global, so concurrent tests could keep the count from ever
  dropping below the pre-snapshot. (2) Multiple engine-touching
  tests in the file race the `OnceLock<Option<EngineState>>` global
  init, so messages sent by test A could be processed by test B's
  engine state.

  Fixes:
  - Add `ExternalToolCatalog::contains_action_anywhere(name)` so the
    cleanup poll can verify a unique marker action is gone regardless
    of what other tests have registered, and switch the test to use
    a per-test `format!("cleanup_marker_{uuid}")` action name.
  - Add a process-static `tokio::sync::Mutex` (`engine_state_lock`)
    that the three engine-touching tests in the file acquire for
    their duration, serializing their use of the global engine state.
    Per-test engine isolation belongs in the bridge itself; this is a
    test-side workaround until that lands.

* fix(responses-api): close gemini bot review findings

Three review fixes from gemini-code-assist on PR #3122 plus a related
fix for fence-style tool calls reported by the user when running
gpt-5.3-codex through `/v1/responses`.

streaming_worker: finalize message item even when resolved text is empty
=========================================================================
The Response branch only emitted `output_item.done` when the resolved
text was non-empty. If `StreamChunks` had already opened a Message
item (`output_item.added` fired, `message_output_index` is `Some`)
and the terminal Response then resolved to an empty string,
`output_item.done` was skipped — leaving the OpenAI client with a
dangling in-progress message in its UI. Now we finalize whenever
either text is available or a message item is in flight, and skip
the redundant delta emit when there's nothing to deliver.

Regression test `streaming_worker_finalizes_item_when_resolved_text_is_empty`
drives the worker with an empty StreamChunk + empty Response and
asserts `added_count == done_count` for output_item events.

validate_external_tools: stream the size check, no allocation
=========================================================================
Replaced the `Vec<Value>` + `String::len()` size measurement with a
streaming `serde_json::Serializer` writing into a counting `io::Write`
sink. Same byte count, no intermediate heap allocation, correctness
contract unchanged (still fails closed if the serialization stream
errors mid-flight).

recover_tool_calls_from_content: recognize markdown-fenced tool calls
=========================================================================
Some OpenAI-compatible models — notably gpt-5.3-codex via the Codex
Responses API — emit caller-supplied tool calls as

    ```tool_call
    {"name": "get_balances", "arguments": {}}
    ```

instead of via the structured `function_call` output channel. Root
cause is engine v2's CodeAct preamble pushing "always respond in a
```repl block" hard enough that the model generalizes the fenced
protocol to a sibling `tool_call` fence for tools it can't dispatch
through Python. Tools ARE passed to the provider correctly
(`OpenAiCodexProvider::build_request_body` sets `body["tools"]` with
strict-OpenAI schema and `tool_choice: "auto"`); the model just opts
out of the structured surface in favor of a fence.

The structural fix is to suppress the CodeAct preamble for Responses
API turns that carry caller-supplied `tools[]`, but that's a larger
piece of work. As a defense-in-depth fix:

- `recover_tool_calls_from_content` now also matches markdown-fenced
  blocks with `tool_call`, `function_call`, or `tool_calls` info
  strings. Opening fence must be at line start to avoid matching
  inline backtick references inside prose. JSON body is parsed and
  validated against the available tool name set; unknown names are
  ignored.
- `clean_response` strips the same fenced blocks via a new
  `strip_markdown_fence_block` helper so any malformed-JSON fence
  the recovery skipped doesn't leak fence syntax to the user.

Six new unit tests in `llm::reasoning::tests` cover the recovery
(JSON, with arguments, function_call alias, unknown-tool ignored,
inline-reference ignored) and the clean_response stripping (clean
case + malformed-JSON case).

* fix(responses-api): emit ExternalToolCall on CodeAct GatePaused outcome

The Responses API was timing out on caller-supplied tool calls when the
LLM emitted them through CodeAct's Python (which is the default mode
for engine v2). Symptoms reported: gateway shows
'Tool X requires external confirmation (gate: external_tool)' as a
generic gate card and the /v1/responses POST returns response.failed.

There are two paths that handle a `GatePaused { External }` from the
engine:

1. `notify_pending_gate` — fires when the engine creates the gate
   mid-execution (Tier 0 structured tool calls). My earlier review-
   feedback fix wired this path to emit `AppEvent::ExternalToolCall`
   so the Responses API handler can surface a `function_call`
   ResponseOutputItem.

2. `await_thread_outcome`'s `ThreadOutcome::GatePaused` arm — fires
   when CodeAct's Python catches `EngineError::GatePaused`, raises a
   `RuntimeError("execution paused by gate 'external_tool'")`, the
   script unwinds, and the thread terminates with a `GatePaused`
   outcome. This arm calls `send_pending_gate_status`, which has an
   empty branch for `ResumeKind::External` (line 514:
   `External { .. } => {}`). No `AppEvent::ExternalToolCall` was
   emitted, so the Responses API handler waited for a never-arriving
   event and the response timed out as failed.

Fix: in the `GatePaused` arm, when `resume_kind` is External with the
`ext_tool:` callback prefix, broadcast `AppEvent::ExternalToolCall`
through the SSE manager and short-circuit past the approval-card
delivery path (which is for human-in-the-loop UX and doesn't apply to
caller-executed tools). Mirrors the `notify_pending_gate` projection.
Annotated with `// projection-exempt: bridge dispatcher, ...` per
`.claude/rules/gateway-events.md` since the broadcast is a projection
of a `ThreadOutcome` (one of the canonical source logs) rather than
an unscheduled side-channel emit.

Also update the `persist_v2_tool_calls_only_called_from_completed_arm`
regression test, which was matching on bare `ThreadOutcome::Completed`
and `ThreadOutcome::GatePaused` text. The Bug 4 catalog-cleanup hook
(commit a4ba764) introduced an early `if !matches!(outcome,
ThreadOutcome::GatePaused { .. })` guard above the match block, so the
first text occurrence of `ThreadOutcome::GatePaused` is now in that
guard rather than the match arm — false-failing the assertion. Anchor
the test on the match-arm destructuring patterns (with first-field
names) instead, so it pins the structural invariant the test name
describes.

* fix(responses-api): filter synthetic engine markers; clearer resume error

Two issues from the user's live test of caller-supplied tools:

1. `__codeact__` was leaking to the response output as a `function_call`
   item with that name. The orchestrator emits ActionFailed events with
   `action_name: "__codeact__"` when a CodeAct script crashes
   (orchestrator.rs:940), and the responses_api accumulator + streaming
   worker were dutifully projecting those into `function_call` items.

   Fix: add `is_synthetic_engine_action(name)` (any `__double_underscore__`
   name) and skip those events in `ResponseAccumulator::process` and
   `streaming_worker` for the ToolStarted/ToolCompleted/ToolResult arms.
   Internal markers no longer surface to the caller.

2. The "function_call_output supplied but no pending external tool
   call" error was opaque — it gave the caller no way to diagnose why
   their resume failed. Most common cause: the LLM ran caller tools
   through CodeAct (Python) instead of structured tool calls, which
   currently doesn't pause the thread (script crashes with a Python
   RuntimeError, no GatePaused outcome, no PendingGate persisted).
   Engine-side fix is in PR #3157 and needs an extension to cover
   `External` resume kinds.

   Improved the error to point at:
   - The diagnostic check (verify prior response.output had a
     function_call item for this call_id)
   - The known limitation (CodeAct path doesn't dispatch caller tools)
   - PR #3157 as the in-progress engine fix

Two new tests:
- `accumulator_filters_synthetic_engine_actions`: drives ToolStarted +
  ToolCompleted with `name: "__codeact__"` and asserts the output
  array stays empty.
- `is_synthetic_engine_action_recognizes_double_underscore`: pins the
  `__double_underscore__` predicate (positive: `__codeact__`,
  `__init__`; negative: regular names, single-underscore, leading- or
  trailing-only doubles).

* fix(responses-api): tighten resume validation, sanitize external payloads, address review findings

Addresses PR #3122 review comments plus the user-directed clean-up:

- responses_api.rs: reject `function_call_output` items with missing/empty
  `call_id` (Copilot 474). Validate the pending gate is actually an
  external-tool gate (ResumeKind::External + `ext_tool:` prefix) and that
  at least one supplied `call_id` matches the pending callback before
  submitting the ExternalCallback (Copilot 1211).
- bridge/router.rs: run the synthesized external-tool payload through
  `SafetyLayer::sanitize_tool_output` before it reaches the LLM —
  external tool payloads originate outside `EffectBridgeAdapter`'s
  pipeline and need the same leak/policy/sanitizer pass internal tool
  outputs get. Move projection-exempt annotations onto the
  `broadcast_for_user` call lines so the gateway-events check passes.
- bridge/effect_adapter.rs: synthesize a `call_ext_<uuid>` call id when
  the executor reaches the external-tool short-circuit without
  `current_call_id` (Copilot effect_adapter.rs:1853), and document
  multi-call batching as an expected limitation at the short-circuit
  site so future readers know the engine pauses on first.
- reasoning.rs: correct the stale fence-recovery comment (Copilot 1738).
- e2e_responses_api_external_tools.rs: remove the "currently expected
  to fail" note; the test now pins the resume materialisation contract
  (Copilot 153).

* fix(responses-api): finalize streaming placeholder before function_call; strip stale PR pointer

Two follow-ups from review:

1. Streaming external-tool dangling `output_item.added`. When a StreamChunk
   arrived before the ExternalToolCall (the placeholder Message was
   already emitted via `output_item.added`), the prose-flush path created
   a *new* Message item at `acc.output.len()` and emitted a fresh
   added+done pair for it — never finalizing the original placeholder.
   OpenAI clients render the unmatched placeholder as "in progress"
   forever. Fix: when `message_output_index` is set, take it, fold the
   accumulated chunks into the existing item at that index, and emit
   `output_item.done` for the same index. Falls through to the original
   no-placeholder behaviour when there is no in-flight Message.

   Updated `streaming_worker_external_tool_call_emits_correct_frame_sequence`
   to match the corrected sequence (one Message added, one Message done,
   then FunctionCall added/done) and added a pairing-invariant
   assertion (`added_count == done_count`) so the regression can't be
   re-locked by an incorrect literal sequence.

2. Wire-visible error message in `responses_api.rs` referenced PR #3157
   for the CodeAct fix. Once the PR merges the pointer is misleading,
   and external callers can't follow the link anyway. Dropped the
   trailing note; the behavioural part of the message stays.

* fix(responses-api): tighten external tool name validation and shadow check

Four follow-ups from review of 87fc8d4:

- responses_api.rs (Copilot #11): the byte-counter comment claimed it
  measured "what the caller actually sent over the wire" but the count
  is canonicalised JSON, not raw request bytes. Rewrote the comment to
  describe the canonicalised-size cap correctly. The `Serializer`
  import is kept — it's needed to bring the trait method into scope
  for the concrete `serde_json::Serializer::serialize_seq` call below
  (an earlier attempt to drop it failed to compile).

- responses_api.rs (Copilot #12): caller-supplied tool names were only
  checked for non-empty + uniqueness. Whitespace, control chars, and
  over-long names could propagate into engine action surfaces, SSE
  payloads, and downstream LLM clients (which all enforce the OpenAI
  Responses spec `^[A-Za-z0-9_-]{1,64}$` and would reject anyway).
  Validate at request time. Added regression tests covering
  whitespace/control/non-ASCII names and the length cap.

- responses_api.rs / bridge/router.rs / bridge/mod.rs (Copilot #13):
  the shadow check only consulted `ToolRegistry::tool_definitions()`
  and missed engine v2 capability actions (`mission_*`, `skill_*`,
  `memory_*`, etc.). A caller registering `mission_create` as an
  external tool would still hit the catalog short-circuit in
  `EffectBridgeAdapter::execute_action`, since the LLM-visible dedup
  in `available_action_inventory` doesn't extend to the execute path.
  Added `engine_capability_action_names()` accessor that pulls the
  full capability-action surface from the bridge's
  `CapabilityRegistry`, and merged it into the collision set the
  responses_api handler checks against.

- runtime/conversation.rs (Copilot #14): documented that
  `extra_initial_metadata` is spawn-only. The `Running` (inject) and
  `Resumable` (resume) branches ignore it; callers needing
  per-request metadata on existing threads must use
  `ThreadManager::set_thread_metadata` instead. The bridge's
  external-tool-catalog `transfer` already handles this for the one
  in-tree caller, but documenting the contract prevents future
  surprise.
zmanian added a commit that referenced this pull request May 20, 2026
…egistrar happy path

Round out the test set for the WASM hook execution path:

#11 / #12: gate + observer wall-clock timeout. The pre-fix dispatcher
ran wasmtime synchronously on the executor, so the outer
`tokio::time::timeout` `Err(_elapsed)` arm was effectively unreachable.
Now that WASM execution runs on the blocking pool, the timeout actually
fires; the new tests give the wasm budget headroom (1B fuel, 5s wall)
and the dispatcher a 20 ms timeout, then assert the failure
classification (FailClosed for gate, FailIsolated for observer).

#13: observer memory exhaustion. Mirrors
`wasm_memory_exhaustion_fails_closed_for_gate` against the observer
dispatch path so the FailIsolated branch of the failure matrix has
explicit memory coverage, not just fuel/wall.

#15: `WasmResourceLimiter::memory_grow_failed` rollback. Stages an
approved grow, simulates the OS-level grow failing, and asserts a
subsequent grow of the full ceiling succeeds — the inflated
`memory_used` from the failed attempt must be released.

#16: registrar WASM happy path. Companion to the existing
`install_wasm_body_requires_runtime` negative case: a valid module
installs, the binding is visible via the public registry accessor, and
is not pre-poisoned.

#14 (`add_milestone_metadata` happy path) is intentionally omitted —
the BeforePrompt dispatch path is currently unreachable due to a
pre-existing manifest-vs-registry scope conflict (`OwnCapabilities` is
the only valid `BeforePrompt` scope per manifest validation, but the
registry rejects `OwnCapabilities` at `BeforePrompt` because the point
has no provider context). That contradiction sits outside this PR's
scope; flagging for a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zmanian added a commit that referenced this pull request May 23, 2026
* Implement installed WASM hook runtime

Adds crates/ironclaw_hooks/docs/threat-model-wasm.md and follows the reviewed design ack: 1) module bytes are resolved, digest-cached, and compiled in the tool-WASM style while reusing its resource limiter; 2) each invocation gets a fresh wasmtime Store; 3) the ABI is a wasmtime::Linker surface, not wit-bindgen; 4) host-import sink shims enforce call, patch-byte, observer-fact, and decision budgets.

* Harden WASM hook string and metadata budgets

* fix(hooks): validate WASM hook ABI at install time (serrrfirat #3 on PR #3634)

Address serrrfirat MEDIUM finding #3: `WasmHookRuntime::prepare()` compiled
and cached module bytes but did not validate imports or the requested
export. ABI mismatches (unsupported import, missing export, wrong export
signature) were deferred to first live dispatch — and the prior
`wasm_unsupported_host_import_fails_closed` test codified that a
bad-import module would install successfully and only fail closed at
invocation. Malformed untrusted modules should never reach live traffic.

Changes:
- `prepare()` derives the target hook point from `request.kind`, then
  runs `validate_module_abi()`: scratch-instantiate the module against
  the point-specific linker (catches unsupported / wrong-type imports)
  and resolve the typed export `() -> ()` (catches missing export and
  wrong signature). Failures surface as new
  `WasmHookRuntimeError::InvalidImports` or existing
  `WasmHookRuntimeError::InvalidExport`, both of which bubble up as
  `HookError::RegistryConstruction` from the registrar.
- `wasm_point_for_kind(HookManifestKind)` helper centralizes the
  kind → wasm-point mapping; the previous `execute_*` paths can share
  it in a follow-up but kept inline for now to minimize churn.

Tests:
- `wasm_unsupported_host_import_is_rejected_at_install_time`: replaces
  the prior test that codified late-failure behavior; asserts the
  registrar returns `RegistryConstruction` citing the bad import.
- `wasm_missing_export_is_rejected_at_install_time`: new module that
  compiles but lacks the manifest-declared export; same install-time
  rejection.

* fix(hooks): address henrypark133 must-fix #1, #2, #3 on PR #3634

Three items from the 5-15 review:

**#1 (must-fix) Extract ironclaw_wasm_limiter micro-crate**
Replace `#[path = "../../../ironclaw_wasm/src/limiter.rs"]` cross-crate
file import with a proper Cargo edge. The 111-line `WasmResourceLimiter`
moves into a new `crates/ironclaw_wasm_limiter` micro-crate that both
`ironclaw_wasm` and `ironclaw_hooks` depend on. The architecture rule
forbidding `ironclaw_hooks -> ironclaw_wasm` is preserved (the new
crate sits below both consumers and pulls in only `wasmtime` +
`tracing`); `cargo check`, `cargo doc`, and architecture-linting tests
now see the edge, and the file can't be moved out from under one of
the consumers silently.

Mechanical changes:
- new `crates/ironclaw_wasm_limiter/` (Cargo.toml + src/lib.rs with the
  type exposed as `pub` instead of `pub(crate)`)
- workspace `members` entry added
- `crates/ironclaw_wasm/src/limiter.rs` deleted
- `crates/ironclaw_wasm/src/lib.rs`: `mod limiter` removed
- `crates/ironclaw_wasm/src/store.rs`: import switched to
  `ironclaw_wasm_limiter::WasmResourceLimiter`
- `crates/ironclaw_wasm/Cargo.toml`: dep added
- `crates/ironclaw_hooks/Cargo.toml`: dep added
- `crates/ironclaw_hooks/src/wasm/runtime.rs`: `#[path = ...]` block
  removed; import switched to the crate

**#2 + #3 (must-fix) Dead WASM arms in dispatch**
`run_before_capability_hook`, `run_before_prompt_hook`, and
`run_observer_hook` each had an early-return guard that dispatched
WASM hooks with `catch_unwind` + timeout, then ALSO had a matching
WASM arm in the inner `match` that ran without those protections. The
prompt-path arm additionally swallowed `WasmHookFailure` via `|_| ()`,
making the must-fix #2 problem worse on that path specifically.

If a future refactor removed any of the early-return guards, those
inner arms would silently take over and drop panic isolation, deadline
enforcement, AND (for prompts) the failure category. Replaced each
inner arm with `unreachable!()` carrying a comment that explains
why the arm exists and references the early-return guard above it.
A future refactor that removes the guard will now trip the
`unreachable!` at first call instead of silently degrading.

All 154 hooks lib + 29 reborn integration tests still pass.

* fix(hooks): plumb context to WASM hooks + runtime hardening

Critical #1 on PR #3634: WASM hooks previously received no context. The
`execute_*` entry points dropped the `&BeforeCapabilityHookContext` /
`&BeforePromptHookContext` / `&ObserverHookContext` value and invoked
the guest export with `()`, so a WASM gate could never decide based on
the capability name, tenant, provider, or other dispatch-time facts. Add
an `ic:hooks/context@1` host-import module exposing two read-only
calls — `ctx_size() -> i32` and `ctx_read(ptr, len) -> i32` — backed by
a JSON-serialized blob the dispatcher writes per-invocation into the
fresh store. Modules that don't import these continue to link; modules
that do import them get a stable, non-empty payload to read. An
integration test (`wasm_before_capability_hook_reads_context_blob`)
asserts the contract end-to-end: a guest that fails to read a non-empty
blob traps before its `deny` call.

Also rolls up the other reviewer-flagged WASM runtime issues, all of
which touch `wasm/runtime.rs`:

HIGH #2: epoch-tick background thread now holds a shutdown
`AtomicBool` and joins on `Drop`. Previously it looped forever and
leaked an Engine clone on every runtime drop.

MED #4: compiled-module cache is now an `lru::LruCache` bounded by
`MODULE_CACHE_CAPACITY = 128`. Replaces the unbounded `HashMap`.

MED #7: `prepare()` no longer compiles under the cache lock. Fast
path reads from LRU under a brief lock; slow path compiles outside
the lock and re-checks on insert to avoid the TOCTOU window where
two concurrent installs of the same module both compile.

Bug #9: post-call `deadline_exceeded()` re-check on the Ok branch
is gone. wasmtime epoch-interrupt is the authoritative wall-clock
signal; an Ok return is no longer reclassified as a timeout because
the wall ticked over during host-side return.

Bug #10: `add_milestone_metadata` returns a distinct
"metadata value exceeds the u32 byte-length ceiling" error when the
guest-supplied `value.len()` overflows u32, instead of misreporting it
as "exceeded total prompt-patch byte budget".

Existing integration tests for WASM hooks are also re-wired through
`HookRegistrar::with_verified_grants` so the grants-store gate added in
the foundation-01 merge stops failing the pre-existing fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(hooks): run WASM hooks on the blocking pool

HIGH #3 on PR #3634: `tokio::time::timeout` does NOT cancel synchronous
wasmtime execution. The previous code awaited a `catch_unwind(async { h.evaluate(ctx) })`
future whose body completed in one poll, so the timeout could only fire
*around* the WASM call rather than against it; a hook that wedged inside
wasmtime simply pinned the calling tokio task.

Route gate, prompt, and observer WASM dispatch paths through
`tokio::task::spawn_blocking` via a shared `run_wasm_blocking` helper.
The outer `tokio::time::timeout` now governs the JoinHandle, so a stuck
blocking task stops blocking the dispatcher's caller; the wasmtime
epoch interrupt configured in the runtime (10 ms tick) is the
authoritative in-WASM wall-clock cancel signal. JoinError (panic in
the blocking task) maps to `FailureCategory::Panic`, matching the
pre-existing semantics for synchronous panics caught via
`catch_unwind`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(hooks): O(1) hook-id lookup via side index

Finding #8 on PR #3634: `set_priority`, `poison`, `is_poisoned`, and
`contains_hook` all did full-registry scans over every binding at every
point. Each is called per-dispatch (poison-checks on the snapshot loop
in particular), so the cost is `O(registered_hooks)` per
`(installed_hook, registered_hook)` pair.

Maintain a denormalized `HashMap<HookId, (HookPointSpec, usize)>` side
index in lock-step with `by_point` so every per-hook-id operation
becomes a single hash lookup + a direct vec indexed access. The
duplicate-id rejection in `insert` now reads from the side index too,
turning what used to be a flat-map scan into a `HashMap::contains_key`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(hooks): wall-clock timeout, observer memory, limiter rollback, registrar happy path

Round out the test set for the WASM hook execution path:

#11 / #12: gate + observer wall-clock timeout. The pre-fix dispatcher
ran wasmtime synchronously on the executor, so the outer
`tokio::time::timeout` `Err(_elapsed)` arm was effectively unreachable.
Now that WASM execution runs on the blocking pool, the timeout actually
fires; the new tests give the wasm budget headroom (1B fuel, 5s wall)
and the dispatcher a 20 ms timeout, then assert the failure
classification (FailClosed for gate, FailIsolated for observer).

#13: observer memory exhaustion. Mirrors
`wasm_memory_exhaustion_fails_closed_for_gate` against the observer
dispatch path so the FailIsolated branch of the failure matrix has
explicit memory coverage, not just fuel/wall.

#15: `WasmResourceLimiter::memory_grow_failed` rollback. Stages an
approved grow, simulates the OS-level grow failing, and asserts a
subsequent grow of the full ceiling succeeds — the inflated
`memory_used` from the failed attempt must be released.

#16: registrar WASM happy path. Companion to the existing
`install_wasm_body_requires_runtime` negative case: a valid module
installs, the binding is visible via the public registry accessor, and
is not pre-poisoned.

#14 (`add_milestone_metadata` happy path) is intentionally omitted —
the BeforePrompt dispatch path is currently unreachable due to a
pre-existing manifest-vs-registry scope conflict (`OwnCapabilities` is
the only valid `BeforePrompt` scope per manifest validation, but the
registry rejects `OwnCapabilities` at `BeforePrompt` because the point
has no provider context). That contradiction sits outside this PR's
scope; flagging for a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(hooks): typed WASM version material, reconcile design doc

LOW #20 on PR #3634: extract the
`{extension_version}+wasm:{module_digest_hex}` concatenation into a
`WasmVersionMaterial` newtype with a single `Display` impl. The
identity material no longer floats free as a stringly-typed argument
inside the registrar.

Reconcile `docs/successors/02-wasm-runtime.md` with the implementation:

- Spell out that wall-clock cancellation depends on the
  `tokio::time::timeout(tokio::task::spawn_blocking(...))` pair, and
  explain why a bare timeout over a synchronous wasmtime call cannot
  actually cancel.
- Define `FailIsolated` and `FailClosed` as `FailureDisposition`
  values, distinct from the older `HookFailureMode::{FailOpen,
  FailClosed}` policy switch that applies to predicates.
- Clarify the generic `evaluate` export contract — name is whatever
  the manifest declares, signature is `(): ()`, context arrives
  through the new `ic:hooks/context@1` host imports — and note the
  intentional divergence from `WitToolRuntime`'s hardcoded interface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(hooks): drop .expect() in WASM module cache capacity

Pre-commit no-panics CI flagged the .expect() on the LruCache capacity.
Move the validity check to a const match, so the NonZeroUsize is fixed at
compile time and the no-panics regex is satisfied.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(hooks): use HookLocalId::new after newtype privatization

The newtype-privatization landed in reborn-integration after the
hooks-fu-wasm-runtime branch's WASM scaffolding tests were written;
update the affected test/registrar sites to use HookLocalId::new
instead of the now-private tuple constructor.

* style: cargo fmt after newtype-privatization fixups

* test(hooks): ignore 3 BeforePrompt WASM tests with manifest/registry conflict

These tests were failing on the original branch tip too (verified against
origin/hooks-fu-wasm-runtime @ 571efdf). The Installed-tier BeforePrompt
WASM install path has no valid scope today:
  - OwnCapabilities is rejected by the registry C3 check (finding #2 on
    PR #3573) since BeforePrompt has no per-capability invocation
    context.
  - SameTenant is rejected by manifest validation ("cannot combine
    scope = same_tenant with kind = before_prompt").

The budget-overflow paths these tests exercise are point-agnostic; the
follow-up is to either rewrite the helper to install through
BeforeCapability or add a Global manifest scope. Tracked as a deferred
item on the new PR.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
henrypark133 added a commit that referenced this pull request Jun 9, 2026
Round-3 multi-agent review at head 707e2cd found 12 straightforward
fixes. All applied here. 3 design-level items deferred to discussion.

SCOPE PREDICATE GAPS (security):
R3-1  §1.6 three DELETE statements gain conditional <agent_predicate>:
      delivery-claim path + delete-path queue + delete-path child_index.
      Without these, agent-scoped callers can delete other agents'
      auxiliary rows under the same (tenant_id, user_id).
R3-2  Phase 0 LEFT JOIN gains <agent_predicate_on_s> — agent-scoped
      replay must not surface settlement-log rows belonging to other
      agents. Performance shape paragraph documents the rule.
R3-13 subagent_idempotency_ledger UNIQUE constraint extended to include
      (tenant_id, user_id, agent_id, ...). Without scope cols in the
      UNIQUE, a cross-tenant collision (UUID or migration artifact)
      would be silently ON CONFLICT DO NOTHING'd.
R3-14 capability_results UNIQUE idempotency index gains agent_id. Two
      agents producing the same (tenant, user, run_id, capability_id,
      invocation_id) would otherwise have their second write silently
      dropped. PostgreSQL uses COALESCE(agent_id, '__non_agent__') for
      uniqueness across NULL-agent rows.

SPEC-VS-TRAIT DRIFT:
R3-9  buffer_unordered propagation: Decision #14 + D4 prose + closing
      checklist all now say buffer_unordered(replay_pool_size). Prior
      contradiction: §5.3 pseudocode MUSTed buffer_unordered but other
      surfaces still said join_all. WU-C reading checklist literally
      would reintroduce the pool-starvation regression.
R3-10 Phase 3 pseudocode + §5.10 risks: .load() → .read() to match the
      §4.3 trait method name. Plus §5.10 capability_result_store.load
      reference updated.
R3-15 scope_from_run_context helper deleted — LoopRunContext.scope IS
      already TurnScope. Replaced with &write.run_context.scope direct
      borrow. §4.8 "Scope source" paragraph documents the canonical
      pattern.

CHECKLIST + TEST NAMES:
R3-4  Credential audit promoted to MERGE-BLOCKING checklist item (top
      of list). WU-C MUST complete LoopRunContext audit + add compile-
      time lint OR verify write-site stripping before merging the
      durable gate-resolution backend.
R3-5  Six reconciler test scenarios in §5.9 get canonical function
      names under tests::reconciler_integration::*. WU-C now has exact
      targets for redelivery, idempotency, tombstoned, missing-result,
      crash-between-insert-and-deliver, and orphan-gate paths.
R3-6  §4.9 + §7.3 name the payload-size-cap test:
      capability_result_store_write_rejects_payload_exceeding_8_mib_with_capacity_exceeded.
      Asserts typed CapacityExceeded error, not raw Backend/Io error.
R3-7  §5.6 names the admission-gate test:
      background_spawn_rejected_with_replay_in_progress_while_reconciler_is_running.
      Foreground / blocking subagent paths must succeed throughout.

CROSS-REF FIXES:
R3-8  Decision #24 + closing checklist HA leader election refs:
      §5.10 → §5.11 (R14 renumber missed these).

DEFERRED FOR DISCUSSION:
- R3-3  Phase 4 per-row seal vs seal_batch trait method
- R3-11 Formal batch trait signatures subsection placement
- R3-12 skipped_orphan counter — split vs combined for tombstoned

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
henrypark133 added a commit that referenced this pull request Jun 9, 2026
R4-1 CRITICAL — Phase 3 buffer_unordered → buffered.
  buffer_unordered emits futures in completion order, not input order.
  Phase 4's to_attempt.zip(load_results) pairs row identity with
  payload positionally → SILENT cross-child payload delivery on every
  replay with >1 pending row. gate_store.record_background_settlement
  called with row_A.parent_run_id + row_A.child_run_id + payload_B.

  Fix: .buffered(replay_pool_size) — same concurrency bound, preserves
  input order. Decision #14, D4 prose, closing checklist all updated.

Schema + SQL invariants:
R4-2  Seal UPDATEs (§5.4 libSQL + §5.5 PostgreSQL) add user_id +
      <agent_predicate> — were missing despite §1.6 mandate.
R4-5  §1.6 INSERT pseudocode for child_index + deliverable_queue add
      user_id + agent_id columns (CF1 added schema cols but not
      pseudocode → would NOT NULL violation on verbatim execution).
f-sec-1  PostgreSQL ledger UNIQUE uses COALESCE(agent_id,
         '__non_agent__') — NULL agent_id rows were silently
         double-INSERTable.
f-sec-2  §1.7 prose no longer describes forbidden (agent_id = ? OR
         agent_id IS NULL) pattern; references §1.6 conditional
         convention instead.
f-bug-3  Phase 0 LEFT JOIN matches scope cols on ledger side too —
         cross-tenant UUID collision could otherwise suppress this
         tenant's replay.
f-bug-5  Phase 2a maps Vec<SettlementLogRow> → Vec<LedgerKey> before
         upsert_sealed_batch — type mismatch fixed.
f-perf-1 + partial-index DDL — idx_subagent_idempotency_ledger_pending
         (partial on delivered_at IS NULL) added to §5.4 + §5.5 so
         Phase 0 scan stays bounded by outstanding work.

Trait surface + tests:
R4-4  reconciler_replays_undelivered_settled_child step 6 fixed:
      `skipped == 0` → 3 real counter fields.
R4-6  SubagentIdempotencyLedger trait drops redundant `scope:
      &TurnScope` arg from 6 methods — LedgerKey embeds scope (single
      source of truth).
R4-7  reconciler_counts_failed_on_missing_capability_result gains
      pencil-receipt-survives assertion (delivered_at IS NULL).
R4-8  delivery_node_invalid_substituted_to_unknown test named in §5.5
      + §5.9 (4 cases: oversized, control chars, disallowed chars,
      empty).
R4-9  §5.6 active-scope enumeration gains max_active_scopes_at_boot
      (default 1000) cap + 5s timeout + overflow → lazy fallback +
      operator metrics.
f-maint-1  §5.4/§5.5 SQL path comments reference inline Rust
           constants in migrations.rs per §8.5 (not separate .sql
           files).

R4-3 (MERGE-BLOCKING marker on credential audit): verified already
applied via R3-4 (line 1928).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
henrypark133 added a commit that referenced this pull request Jun 9, 2026
* docs(reborn): WU-B subagent durability sub-spec

Sub-spec for WU-B per docs/plans/2026-06-06-subagent-compaction-impl.md.
Blocks WU-C. Doc-only.

Covers 4 in-memory stores (gate resolution, goal, tombstone, capability
result) + 2 new tables (settlement event log, idempotency ledger).
Decides typed-repo vs ScopedFilesystem per _contract-freeze-index.md §2.
Introduces CapabilityResultStore + SubagentRestartReconciler traits.
Specifies libSQL + PostgreSQL schemas, first-writer-wins semantics,
scope propagation, migration/rollback under subagent.background_enabled
toggle, and the dual-backend parity test (#4431 follow-on).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B straightforward review fixes

Applies 18 straightforward findings from multi-agent code review on PR
#4582. Design-level items still pending discussion.

Schema:
- F2: PostgreSQL ledger run_id/child_run_id UUID → TEXT (§8.3 convention)
- F3: align libSQL/PostgreSQL undelivered_terminal partial index
- F5: add result_ref column to subagent_gate_settlement_log both backends
- F6: capability_results uses explicit PRIMARY KEY (result_ref)
- F8: scope predicates mandatory on UPDATE/DELETE templates in §1.6
- F9: define post-result-write flag update path (separate transaction)
- F18: 8 MiB CHECK constraint on capability_results.payload (MUST)

Contracts:
- F1: CapabilityResultStore trait scope &ResourceScope → &TurnScope
- F7: drop CapabilityRunId alias; use TurnRunId directly
- F10: specify sanitized_reason source + sanitization transform
- F11: specify delivery_node validation (length, allowlist, source)
- F12: §6.3 restated in binary INSERT-OR-IGNORE ledger semantics
- F13: resolve tombstone trait scope-param decision in spec

Doc consistency:
- F4: remove delivered_at IS NULL filter (column doesn't exist)

Test plan:
- F14: name positive production-readiness tests (goal, tombstone, capres)
- F15: tombstone first-writer-wins distinguishing test
- F16: agent_id cross-leakage parity test
- F17: reconciler crash-between-ledger-insert-and-gate-write test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B two-phase ledger + orphan handling (D1+D9)

Resolves two reconciler bugs surfaced by multi-agent review on PR #4582:

D1 — Crash-between-ledger-insert-and-gate-write strands parent silently.
  Idempotency ledger goes two-phase. `delivered_at TIMESTAMPTZ NULL`:
  - INSERT OR IGNORE leaves `delivered_at = NULL` (pencil receipt;
    claim, mid-flight).
  - After successful gate-store write, UPDATE seals row with
    `delivered_at = NOW()` (pen receipt; final).
  - Pencil rows surviving a crash become `retryable` on next boot,
    not silently `skipped_idempotent` as before.
  Matches the existing `IdempotencyLedger::begin_or_replay` precedent in
  `crates/ironclaw_product_workflow/src/ledger.rs`. Both gate-store and
  seal UPDATE are idempotent at the row level so duplicate delivery
  cannot occur and missed delivery cannot occur.

D9 — Orphan settlement-log rows produced perpetual `failed` count.
  Reconciler now checks `gate_store.gate_exists(scope, gate_ref)` first.
  If the gate is gone (parent cancelled, gate row deleted): write
  `SubagentResultTombstone { disposition: DiscardedParentGone }`, seal
  the ledger row, count as `skipped_orphan`. One pass per orphan; future
  passes skip via sealed ledger row. Settlement log stays append-only.

ReplayReport gains `retryable: u32` and `skipped_orphan: u32` so each
counter has one meaning. `failed > 0` is now operator-actionable only —
no more phantom alerts.

Spec changes:
- §5.2 ReplayReport struct extended.
- §5.3 algorithm rewritten: gate-exists check, then tombstone check,
  then pencil-claim, then deliver, then seal. Pencil read on
  insert-skip distinguishes sealed (skipped_idempotent) from
  pencil (retryable).
- §5.4 + §5.5 ledger DDL: `delivered_at` becomes nullable. INSERT
  examples split into pencil + seal.
- §5.8 test plan: orphan-gate test case added; existing test names
  updated.
- §5.9 risks: stale-children GC bullet rewritten; capability-result-
  missing conclusion sentence updated.
- §6.3 re-flip narrative updated to use sealed/pencil vocabulary.
- "Decisions ratified up front" table gains rows 11–13.
- Closing checklist gains 3 WU-C action items.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B reconciler perf + cold-start shape (D4+D5 comprehensive)

Resolves cold-start / scaling concerns surfaced by multi-agent review on
PR #4582. Comprehensive: D4 + D5 + 5 long-term concerns folded in.

D4 — Reconciler replay batch-phased (no more N+1).
  §5.3 algorithm rewritten:
    Phase 0  bound input via LEFT JOIN against ledger (only pending
             pencil-or-missing rows enter the algorithm; replay's scan
             size stays proportional to outstanding work, not historical
             log size).
    Phase 1  batched preflight: one query for gates_exist_batch, one for
             read_tombstones_batch.
    Phase 2  multi-row ledger writes:
               2a — orphan + tombstoned cleanup (one upsert-sealed batch)
               2b — pencil claim (one INSERT OR IGNORE batch)
    Phase 3  parallel capability loads via `join_all` (capped at
             replay_pool size).
    Phase 4  per-row deliver + seal (sequential per row, each row hits a
             different parent's mailbox).
  Phases 0–3 are O(1) DB calls regardless of N. Net cost dominated by
  Phase 4's per-row delivery, ~5–30 ms per row depending on backend
  latency. 10–50× speedup over the previous N+1 form.

D5 — Background replay + per-scope admission gate.
  §5.6 composition wire-up rewritten:
    - Replay dispatched via `tokio::spawn` from boot; foreground traffic
      accepts immediately (<100 ms cold start regardless of backlog).
    - Per-scope `ReplayState { completed_at, last_report }` tracks
      completion. Background-mode `SpawnSubagentPort` consults the gate
      before admitting; rejects with `SubagentSpawnError::ReplayInProgress`
      until per-scope replay completes. Foreground / blocking subagent
      calls NEVER consult this gate.
    - Dedicated `replay_pool` (default 4 DB connections, configurable via
      `RebornEventStoreConfig.replay_pool_size`) — replay never starves
      foreground writes during recovery storms.
    - Eager active-scope enumeration at boot via runs-table query.
      Bounded by active-runs count, not historical user count. Lazy
      per-scope replay deferred as future optimization.

Long-term concerns folded in:
  - HA replicas: spec is HA-safe (correctness via Phase 2b INSERT OR
    IGNORE + single-winner seal UPDATE), HA-redundant (each replica
    runs replay independently — N× DB load at boot). Active-active
    leader election deferred to cross-cutting follow-up. Documented in
    §5.6 + §5.9.
  - Settlement log growth: Phase 0 LEFT JOIN bounds input — replay's
    scan size is independent of historical log size. Archival /
    materialized-view summarization deferred as ops follow-up.
  - Replay pool sizing: default 4 fine for typical fan-outs; tuning
    via P95 metric. Spec does not mandate auto-tuning.

§5.7 NEW — Observability contract:
  - `RebornEventKind::SubagentReplayCompleted` event per scope.
  - 5 required metrics: replay_duration_seconds (histogram),
    replay_pending_rows (gauge), replay_outcomes_total{outcome=…}
    (counter), pencil_age_seconds (gauge), replay_in_progress (gauge).
    All labeled by (tenant_id, agent_id).
  - 3 required alerts: `failed > 0`, `pencil_age_seconds > 60`,
    `replay_duration_seconds{P95} > 30`.
  - OpenTelemetry spans: one per scope (`reborn.subagent.replay`) +
    child spans per phase.
  - WU-F WebUI surfaces `replay_in_progress` per-scope; background-spawn
    rejection during replay shown to user as "starting up, retrying in
    N seconds" affordance.
  Prerequisite for WU-G E2E + WU-F integration.

Decisions table gains rows 14–19. Closing checklist gains 7 WU-C action
items.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B hot-path perf + multi-tenant scaling (D6+D8+E.A+A.A+A.B)

Resolves hot-path overhead + multi-tenant scaling concerns for the
spawn / capability-write / replay paths.

D6-A — Durable per-scope capacity counter.
  Replace per-spawn SELECT COUNT(*) with a sidecar
  `subagent_gate_capacity_counter` table — one transactional UPDATE per
  spawn (no extra round-trip). Race-safe via SELECT FOR UPDATE (PG) /
  BEGIN IMMEDIATE (libSQL). Symmetric increment on INSERT, decrement on
  delivery / delete via GREATEST(undelivered - N, 0) safety net.

E.A — Sharded counter for hot scopes (CAPACITY_COUNTER_BUCKETS = 16).
  Per-scope counter row becomes a write hotspot when one mega-tenant
  runs 10k+ concurrent background subagents under the same scope.
  Shard into K=16 rows per (tenant_id, user_id, agent_id) keyed by
  `bucket SMALLINT/INTEGER NOT NULL`. Spawn picks bucket via
  `hash(child_run_id) % K`. Cap check is `SUM(undelivered)` across all
  K buckets — index-only at K=16.
  `subagent_gate_awaited_children.counter_bucket` stores bucket-of-record
  for symmetric decrement on cleanup. Per-scope spawn throughput lifts
  from ~100/sec (single-row lock contention) to ~1600/sec on PostgreSQL.
  Drift bound: ≤ K-1 rows over cap under maximum concurrency.

D8-A — CapabilityResultStore trait takes Vec<u8>, not serde_json::Value.
  Executor: `let bytes = serde_json::to_vec(&output)?;` ONCE. `byte_len`
  is `bytes.len() as u64` — derived for free. `bytes` is MOVED into the
  store, not cloned. Store INSERTs bytes directly into BLOB (libSQL) /
  JSONB (PostgreSQL) without re-serializing. `read()` returns Vec<u8>;
  caller deserializes lazily via `serde_json::from_slice` only when a
  Value is needed (prompt assembly, compaction).
  Eliminates 2× full-tree serialization + 1 Value clone per capability
  call. ~50% CPU reduction on capability-write hot path at production
  scale. Trait shape reflects what crosses the boundary (bytes, not a
  tree). Composes with future streaming variants (BoxStream<Bytes>).

A.A — Reconciler replay jitter for fleet rollouts.
  `RebornEventStoreConfig.reconciler_replay_jitter_ms: u64` (default
  5000). Each replica sleeps a uniform-random 0..jitter ms before
  launching its background replay task. Spreads the deploy-time
  reconciler stampede over a wider window — at 50-replica rollout, peak
  DB reconciler conn count drops from N×replay_pool to ~jitter-spread
  fraction. Foreground traffic NEVER pays the jitter cost. Set to 0
  for single-node deployments.

A.B — HA per-scope leader election (new §5.10, future, NOT WU-C scope).
  Documented direction: Postgres `pg_try_advisory_xact_lock` per scope.
  Replicas that lose election skip replay for that scope; still consume
  settlement events via gate-store mailbox as normal. Total fleet
  reconciler work drops from O(N × scopes) to O(scopes). Promotion
  trigger documented (P95 replay duration > 30s + sustained
  replay_in_progress aggregate > 60s). Lock is transaction-scope so
  auto-releases on leader crash — composes cleanly with D1's two-phase
  ledger. libSQL fallback: noop election (every replica is leader);
  libSQL deployments are typically single-node so redundancy is moot.

Decisions table gains rows 20–24 (D6-A, E.A, D8-A, A.A, A.B).
Closing checklist gains 4 WU-C action items + 1 follow-up note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B round-2 review fixes (R1-R17)

Round-2 multi-agent review surfaced 5 High-severity regressions
introduced by prior fix commits, plus medium consistency issues.
All resolved here.

SQL TEMPLATES (§1.6) — fix regressions in transaction shapes:
  R3 — Conditional agent_id predicate. Replace blanket
       `(agent_id = ? OR agent_id IS NULL)` (which lets agent-scoped
       callers reach system-level rows) with placeholder
       `<agent_predicate>` bound conditionally per caller's scope:
       `agent_id = ?` when Some, `agent_id IS NULL` when None.
       New §1.6 preamble paragraph documents the rule.
  R4 — Capacity counter SUM + UPDATE use the conditional predicate too.
       Bare `agent_id = ?` with NULL parameter evaluated to UNKNOWN,
       silently bypassing the 4096 cap for non-agent runs.
  R5 — Delivery-claim DELETE on deliverable_queue gains `child_run_id`.
       Previously wiped ALL queue rows for a gate when only one child
       was delivered — stranded N-1 siblings.
  R10 — Delivery-claim UPDATE SET also flips `delivery_claimed = 1`
        (prose was inconsistent with SQL).
  R13 — Delete-path DELETEs on deliverable_queue + child_index gain
        `user_id` predicate. awaited_children DELETE uses the
        conditional agent_predicate.
  R15 — Settlement log dedup decision resolved (was deferred). Ledger
        UNIQUE + gate-store idempotency + Phase 0 LEFT JOIN make
        duplicate log rows benign; no MIN(id) needed.
  R16 — parent_run_context_json gains sensitivity audit requirement
        (closing-checklist gate): WU-C MUST verify LoopRunContext is
        credential-free or strip sensitive fields at write site.

ALGORITHM PSEUDOCODE (§5.3) — fix wrong column names + bounded fan-out:
  R1 — Phase 0 LEFT JOIN uses `s.parent_run_id` (column actually exists;
       schema does NOT have `s.run_id`).
  R2 — Phase 0 filter uses `s.terminal_kind` (column actually exists;
       schema does NOT have `s.event_kind`).
  R11 — Phase 3 capability loads use `buffer_unordered(replay_pool_size)`
        not `join_all`. Unbounded fan-out at 10k pending rows would
        starve foreground writes on the 4-conn replay pool.
  R12 — Phase 2a tombstone writes use `write_tombstones_batch` (single
        round-trip), not a per-row `for` loop. Trait gains batch method.

TRAIT + VARIANT CONSISTENCY:
  R8 — InMemoryCapabilityResultStore type is `Mutex<HashMap<String,
       Vec<u8>>>` (was self-contradicted in §4.5 — D8-A regression).
  R9 — SubagentResultDisposition variants documented: today's
       `DiscardedByParentCancel` + WU-C addition `DiscardedParentGone`
       (used by §5.3 Phase 2a orphan cleanup). §3.7 risks bullet
       updated. Forward-compat with WU-D variants (`Delivered`,
       `SettledByBackground`).
  R17 — `scope_from_run_context` helper defined in §4.8 (was undefined).
        Maps LoopRunContext → TurnScope; documents user_id resolution
        via `explicit_owner_user_id()` + SYSTEM_RESERVED_ID sentinel.

DOC HYGIENE:
  R6 — Tombstoned-result test assertion fixed: `skipped_orphan == 1,
       failed == 0` (matches §5.3 algorithm; was `failed == 1`).
  R7 — Double-replay test assertion fixed: `skipped_idempotent == 1`
       (was `skipped == 1` — field does not exist on ReplayReport).
  R14 — Duplicate `### 5.8` heading resolved. Test plan now §5.9, Risks
        §5.10, HA leader election §5.11.
  Pen→pencil terminology consistency in §5.4 + §5.5 SQL comments.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B Copilot + Codex review fixes (CF1-CF5)

Round-2 Copilot + Codex automated reviewers caught issues skill
reviewers missed. All resolved here.

CF1 — subagent_gate_child_index + subagent_gate_deliverable_queue
       gain `user_id TEXT NOT NULL` + `agent_id TEXT` columns per
       Decision #5 (also fixes a latent SQL syntax error: prior R13
       added user_id to DELETE predicates without the columns
       existing). New `idx_sgci_scope` / `idx_sgdq_scope` indexes on
       `(tenant_id, user_id, agent_id, child_run_id)` replace the
       prior `tenant_child` indexes. Both libSQL + PostgreSQL.

CF2 — capability_results.created_at gains
       `DEFAULT (datetime('now'))` in libSQL (PostgreSQL already had
       `DEFAULT NOW()`). Needed because §4.6/§4.7 rely on this column
       for `idx_capability_results_run` ordering and `list_by_run`
       ORDER BY — silent inserter mistakes would break replay
       ordering.

CF3 — CapabilityResultStore::write now takes
       `invocation_id: InvocationId`. UNIQUE INDEX
       `(tenant_id, user_id, run_id, capability_id, invocation_id)`
       enforces true first-writer-wins idempotency. Previous design
       minted a fresh UUID per call, so `INSERT OR IGNORE` could
       never collide — idempotency claim was misleading. Now a
       retry-after-transient-error returns the same `result_ref`.
       Trait + in-memory impl note + §4.8 wire-up updated; both
       backend schemas gain the column + unique index.

CF4 — §6.2 rollback step rewritten. Goal store stays on
       FilesystemSubagentGoalStore (durable) when the toggle flips
       OFF; the toggle gates only background-mode spawn admission,
       NOT backend selection. Prior wording about
       "re-selects InMemoryBoundedSubagentGoalStore" contradicted
       §2.1.

CF5 — Decision #5 reworded. Scope columns are always PRESENT on
       every durable table and reached via a scoped index
       (`idx_*_scope`). PKs remain shape-appropriate per table
       (e.g. `(gate_ref, child_run_id)`, `(result_ref)`) — scope
       need not LEAD every PK. Matches actual schema guidance and
       removes the false-positive interpretation that all PKs must
       be scope-prefixed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B resolve 2 leftover review items

Two open threads from round-2 review now decided + applied.

1. InMemoryCapabilityResultStore gains bounded eviction.
   `INMEMORY_CAPABILITY_RESULT_STORE_MAX_ENTRIES = 1024` +
   `INMEMORY_CAPABILITY_RESULT_STORE_MAX_BYTES = 4 MiB` (FIFO by
   insertion order). Prevents local-dev / CI OOM on long sessions
   that accumulate megabyte-scale payloads. Production-readiness
   check still gates the impl to LocalDevTest mode regardless.

2. `gate_resolution_scoped_query_excludes_rows_from_other_agents`
   promoted from WU-G to WU-C. This is a security gate (cross-tenant
   / cross-agent leakage class via missing agent_id predicate), not
   an E2E gate. Shipping the gate-resolution backend in WU-C without
   this guard would mean releasing the durable code with no test
   that catches a missing agent_id WHERE clause — unacceptable per
   §1.7 + `_contract-freeze-index.md` §8.

Closing checklist gains two WU-C action items.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B round-3 review fixes (R3-1..R3-15)

Round-3 multi-agent review at head 707e2cd found 12 straightforward
fixes. All applied here. 3 design-level items deferred to discussion.

SCOPE PREDICATE GAPS (security):
R3-1  §1.6 three DELETE statements gain conditional <agent_predicate>:
      delivery-claim path + delete-path queue + delete-path child_index.
      Without these, agent-scoped callers can delete other agents'
      auxiliary rows under the same (tenant_id, user_id).
R3-2  Phase 0 LEFT JOIN gains <agent_predicate_on_s> — agent-scoped
      replay must not surface settlement-log rows belonging to other
      agents. Performance shape paragraph documents the rule.
R3-13 subagent_idempotency_ledger UNIQUE constraint extended to include
      (tenant_id, user_id, agent_id, ...). Without scope cols in the
      UNIQUE, a cross-tenant collision (UUID or migration artifact)
      would be silently ON CONFLICT DO NOTHING'd.
R3-14 capability_results UNIQUE idempotency index gains agent_id. Two
      agents producing the same (tenant, user, run_id, capability_id,
      invocation_id) would otherwise have their second write silently
      dropped. PostgreSQL uses COALESCE(agent_id, '__non_agent__') for
      uniqueness across NULL-agent rows.

SPEC-VS-TRAIT DRIFT:
R3-9  buffer_unordered propagation: Decision #14 + D4 prose + closing
      checklist all now say buffer_unordered(replay_pool_size). Prior
      contradiction: §5.3 pseudocode MUSTed buffer_unordered but other
      surfaces still said join_all. WU-C reading checklist literally
      would reintroduce the pool-starvation regression.
R3-10 Phase 3 pseudocode + §5.10 risks: .load() → .read() to match the
      §4.3 trait method name. Plus §5.10 capability_result_store.load
      reference updated.
R3-15 scope_from_run_context helper deleted — LoopRunContext.scope IS
      already TurnScope. Replaced with &write.run_context.scope direct
      borrow. §4.8 "Scope source" paragraph documents the canonical
      pattern.

CHECKLIST + TEST NAMES:
R3-4  Credential audit promoted to MERGE-BLOCKING checklist item (top
      of list). WU-C MUST complete LoopRunContext audit + add compile-
      time lint OR verify write-site stripping before merging the
      durable gate-resolution backend.
R3-5  Six reconciler test scenarios in §5.9 get canonical function
      names under tests::reconciler_integration::*. WU-C now has exact
      targets for redelivery, idempotency, tombstoned, missing-result,
      crash-between-insert-and-deliver, and orphan-gate paths.
R3-6  §4.9 + §7.3 name the payload-size-cap test:
      capability_result_store_write_rejects_payload_exceeding_8_mib_with_capacity_exceeded.
      Asserts typed CapacityExceeded error, not raw Backend/Io error.
R3-7  §5.6 names the admission-gate test:
      background_spawn_rejected_with_replay_in_progress_while_reconciler_is_running.
      Foreground / blocking subagent paths must succeed throughout.

CROSS-REF FIXES:
R3-8  Decision #24 + closing checklist HA leader election refs:
      §5.10 → §5.11 (R14 renumber missed these).

DEFERRED FOR DISCUSSION:
- R3-3  Phase 4 per-row seal vs seal_batch trait method
- R3-11 Formal batch trait signatures subsection placement
- R3-12 skipped_orphan counter — split vs combined for tombstoned

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B round-3 design decisions (R3-3 + R3-11 + R3-12)

Three coupled design-level changes from round-3 review now ratified
and applied.

R3-3 — Phase 4 seal_batch.
  Spec §5.3 Phase 4 previously issued per-row `idempotency_ledger.seal`
  on each successful delivery. At 100 children through a 4-conn
  replay_pool that is 25 sequential rounds (~125-750 ms) on the seal
  step alone.

  Fix: Phase 4 now collects sealed-row keys into a `sealed_keys` vec
  during the per-row loop, then issues ONE `seal_batch(scope, Vec<
  LedgerKey>)` call at the end. Single multi-row UPDATE. Idempotent
  per-row via the `delivered_at IS NULL` guard.

  Single-row `seal` retained for orphan / tombstone paths in Phase 2a
  (which already batch via `upsert_sealed_batch`) and for any future
  operator-driven manual interventions on stuck rows.

R3-11 — Formal batch method signatures in §5.2.1.
  §5.3 algorithm calls 8 batch methods. Only `write_tombstones_batch`
  had a rough signature; the rest were implicit. WU-C would need to
  reverse-engineer 7 method signatures from pseudocode call sites.

  Fix: new §5.2.1 "Batch method signatures (reconciler-facing)"
  subsection lists all 8 method signatures with full async-trait
  syntax plus `LedgerKey` + `LedgerRow` struct definitions. Single-
  row variants documented alongside batch variants for completeness.
  WU-C now reads §5.2.1 literally as the trait surface contract.

R3-12 — Split skipped_orphan counter.
  Old: `skipped_orphan = orphan_rows.len() + tombstoned_rows.len()`.
  Two semantically distinct cases conflated:
    - orphan       = gate row gone (parent cancel + cleanup)
    - tombstoned   = gate live but child pre-tombstoned (parent
                     cancelled the specific child)

  Different operational signals; merging them prevented operators
  from distinguishing gate-cleanup spikes (high `skipped_orphan`
  alone) from parent-cancel spikes (high `skipped_tombstoned`).

  Fix: `ReplayReport` gains `skipped_tombstoned: u32`. Phase 2a
  increments each counter independently. §5.7 metric label list
  extended; §5.9 `reconciler_skips_tombstoned_child` test asserts
  `skipped_tombstoned == 1, skipped_orphan == 0`. Decisions table
  row 13 updated to six counters.

Closing checklist gains 3 new WU-C items (seal_batch impl,
batch-method trait surface, ReplayReport split).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B round-4 review fixes (R4-1 critical + 12 more)

R4-1 CRITICAL — Phase 3 buffer_unordered → buffered.
  buffer_unordered emits futures in completion order, not input order.
  Phase 4's to_attempt.zip(load_results) pairs row identity with
  payload positionally → SILENT cross-child payload delivery on every
  replay with >1 pending row. gate_store.record_background_settlement
  called with row_A.parent_run_id + row_A.child_run_id + payload_B.

  Fix: .buffered(replay_pool_size) — same concurrency bound, preserves
  input order. Decision #14, D4 prose, closing checklist all updated.

Schema + SQL invariants:
R4-2  Seal UPDATEs (§5.4 libSQL + §5.5 PostgreSQL) add user_id +
      <agent_predicate> — were missing despite §1.6 mandate.
R4-5  §1.6 INSERT pseudocode for child_index + deliverable_queue add
      user_id + agent_id columns (CF1 added schema cols but not
      pseudocode → would NOT NULL violation on verbatim execution).
f-sec-1  PostgreSQL ledger UNIQUE uses COALESCE(agent_id,
         '__non_agent__') — NULL agent_id rows were silently
         double-INSERTable.
f-sec-2  §1.7 prose no longer describes forbidden (agent_id = ? OR
         agent_id IS NULL) pattern; references §1.6 conditional
         convention instead.
f-bug-3  Phase 0 LEFT JOIN matches scope cols on ledger side too —
         cross-tenant UUID collision could otherwise suppress this
         tenant's replay.
f-bug-5  Phase 2a maps Vec<SettlementLogRow> → Vec<LedgerKey> before
         upsert_sealed_batch — type mismatch fixed.
f-perf-1 + partial-index DDL — idx_subagent_idempotency_ledger_pending
         (partial on delivered_at IS NULL) added to §5.4 + §5.5 so
         Phase 0 scan stays bounded by outstanding work.

Trait surface + tests:
R4-4  reconciler_replays_undelivered_settled_child step 6 fixed:
      `skipped == 0` → 3 real counter fields.
R4-6  SubagentIdempotencyLedger trait drops redundant `scope:
      &TurnScope` arg from 6 methods — LedgerKey embeds scope (single
      source of truth).
R4-7  reconciler_counts_failed_on_missing_capability_result gains
      pencil-receipt-survives assertion (delivered_at IS NULL).
R4-8  delivery_node_invalid_substituted_to_unknown test named in §5.5
      + §5.9 (4 cases: oversized, control chars, disallowed chars,
      empty).
R4-9  §5.6 active-scope enumeration gains max_active_scopes_at_boot
      (default 1000) cap + 5s timeout + overflow → lazy fallback +
      operator metrics.
f-maint-1  §5.4/§5.5 SQL path comments reference inline Rust
           constants in migrations.rs per §8.5 (not separate .sql
           files).

R4-3 (MERGE-BLOCKING marker on credential audit): verified already
applied via R3-4 (line 1928).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(reborn): WU-B round-5 review fixes (R5-1..R5-10)

- R5-1: PG capability_results.payload BYTEA not JSONB (byte-exact
  round-trip contract; JSONB normalization breaks parity + byte_len)
- R5-2: capability-write idempotency conflict target = invocation
  unique index, insert-then-select read-back (was result_ref, which
  never conflicts on retry)
- R5-3: lazy per-scope replay ships in WU-C as admission-gate trigger
  (capped scopes were rejected with ReplayInProgress forever)
- R5-4: flat tombstone ScopedPath (no thread segment — settlement log
  carries no thread_id); read_tombstone also gains scope param
- R5-5: Phase 3 = exists_batch existence check, no payload loads;
  delivery via new redeliver_settled_child (record_background_settlement
  was undefined and payload-shaped)
- R5-6: libSQL MAX() not GREATEST in counter decrements
- R5-7: tombstoned rows resolve live gate row (capacity-leak fix);
  new resolve_undeliverable_batch
- R5-8: ReplayState keyed by (tenant_id, user_id, agent_id)
- R5-9: SubagentReplayCompleted contract-freeze callout
- R5-10: CapacityExceeded maps to CapabilityOutcome::Failed, never
  aborts the loop
- minors: 11-method count, pseudocode scope-arg drift, six-counter
  comment, PG ON CONFLICT expression target, MountView user-isolation
  verification, stale §5.10 throughput bullet
- plan: drop stale duplicate WU-C 'Files modified' block (contradicted
  the corrected block above it)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(reborn): WU-B add §9 parent-initiated child cancel + inspect

Audit of existing host plumbing (request_cancel, RunCancellationHandle,
children_of, event projection) shows the only gap is model-visible
action surface. Ratifies decisions 33-35: two thin WU-D actions
(subagent_cancel, subagent_status) over existing machinery; parent-
requested cancel delivers a Cancelled settlement (never tombstones —
DiscardedByParentCancel stays reserved for the parent-run-cancel
cascade); status is metadata-only so settle-time delivery remains the
sole sanitization choke point. Child-pushed progress notes deferred
pending WU-G evidence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chenzhongsong93 pushed a commit to chenzhongsong93/ironclaw that referenced this pull request Sep 26, 2026
chenzhongsong93 pushed a commit to chenzhongsong93/ironclaw that referenced this pull request Sep 26, 2026
chenzhongsong93 pushed a commit to chenzhongsong93/ironclaw that referenced this pull request Sep 26, 2026
chenzhongsong93 pushed a commit to chenzhongsong93/ironclaw that referenced this pull request Sep 26, 2026
chenzhongsong93 pushed a commit to chenzhongsong93/ironclaw that referenced this pull request Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants