From 96d0d4608eb083b9c26b5c441d9b8614fa0ec9bb Mon Sep 17 00:00:00 2001 From: BenKurrek Date: Mon, 3 Aug 2026 20:14:41 -0400 Subject: [PATCH 1/2] refactor(host-runtime): split obligations into its three chartered owners (WS3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `crates/ironclaw_host_runtime/src/obligations.rs` was 3,122 lines fusing the three owners PROPOSAL §6.5.9 charters separately, held apart only by an `// arch-exempt: large_file` waiver. It is now one module per owner: - `obligations::handler` — which obligations apply and what each does before/after dispatch, plus the audit/redaction/ceiling/mount validation. - `obligations::staged_handoffs` — material staged for a later consumer: the runtime-secret and network-policy stores and the credential-account resolver port. - `obligations::process_store` — post-start handoff discard and reservation reconciliation. - `obligations::mod` — only `BuiltinObligationServices`, the assembly seam, and deliberately the one place naming all three at once. Every module is under the 1,500-line gate, so the waiver is deleted rather than carried forward: re-fusing the owners now trips `pre-commit-safety.sh`. `mod obligations;` stays private and the crate's `pub use obligations::{…}` names are unchanged, so no consumer outside the crate sees this. Behavior-free. Cross-owner access is `pub(super)` (three methods), not `pub(crate)`. The split revealed one narrowing in the other direction: `secret_present` was `pub(crate)` with no caller outside its own file and is now private. Also from the same CHECKLIST row, the bounded half of "shrink `services/builder.rs` toward composition-facing factories": three builder methods whose only callers are inside the crate's `src` narrow to `pub(crate)`. The rest of that clause is measured and deferred in the CHECKLIST amendment — 17 methods need a `test-support` cargo feature, three are callerless and belong to WS8, and the remaining 33 are a redesign of the fluent surface rather than a shrink of it. `+production_wiring` is refuted there: it is readiness diagnostics, not assembly. Two loud path-keyed gates fired and were repointed, not relaxed: `reborn_host_runtime_services_do_not_expose_lower_substrate_handles` now scans the whole `obligations/` directory and asserts it read ≥ 4 files (`collect_runtime_rs` returns a count; both its callers now assert non-zero), and `reborn_struct_test_support_ratchet`'s frozen per-file count moves to `staged_handoffs.rs` with its count unchanged at 1. Test accounting (un-masking discipline): `cargo test -p ironclaw_host_runtime --all-targets -- --list` is 1,246 before and 1,246 after, name-by-name identical — zero added, removed or renamed. `LAYER_MATRIX_EXCEPTIONS` is 10 before and after; an intra-crate split cannot move the register. Co-Authored-By: Claude Opus 5 --- .../tests/reborn_dependency_boundaries.rs | 68 +- .../reborn_struct_test_support_ratchet.rs | 5 +- crates/ironclaw_host_runtime/AGENTS.md | 6 +- crates/ironclaw_host_runtime/CLAUDE.md | 1 + .../ironclaw_host_runtime/src/obligations.rs | 3122 ----------------- .../src/obligations/handler.rs | 1276 +++++++ .../src/obligations/mod.rs | 217 ++ .../src/obligations/process_store.rs | 586 ++++ .../src/obligations/staged_handoffs.rs | 500 +++ .../src/obligations/tests.rs | 650 ++++ .../src/services/builder.rs | 6 +- docs/reborn/target-architecture/CHECKLIST.md | 15 +- docs/reborn/target-architecture/PROPOSAL.md | 6 +- 13 files changed, 3304 insertions(+), 3154 deletions(-) delete mode 100644 crates/ironclaw_host_runtime/src/obligations.rs create mode 100644 crates/ironclaw_host_runtime/src/obligations/handler.rs create mode 100644 crates/ironclaw_host_runtime/src/obligations/mod.rs create mode 100644 crates/ironclaw_host_runtime/src/obligations/process_store.rs create mode 100644 crates/ironclaw_host_runtime/src/obligations/staged_handoffs.rs create mode 100644 crates/ironclaw_host_runtime/src/obligations/tests.rs diff --git a/crates/ironclaw_architecture/tests/reborn_dependency_boundaries.rs b/crates/ironclaw_architecture/tests/reborn_dependency_boundaries.rs index 1cb77e4c26f..e30f4f28716 100644 --- a/crates/ironclaw_architecture/tests/reborn_dependency_boundaries.rs +++ b/crates/ironclaw_architecture/tests/reborn_dependency_boundaries.rs @@ -980,7 +980,13 @@ fn reborn_cli_binary_crate_stays_separate_from_v1_root() { let runtime_dir = root.join("crates/ironclaw_reborn_cli/src/runtime"); let mut cli_runtime_source = String::new(); - collect_runtime_rs(&runtime_dir, &mut cli_runtime_source); + let cli_runtime_files = collect_runtime_rs(&runtime_dir, &mut cli_runtime_source); + assert!( + cli_runtime_files > 0, + "expected the Reborn CLI runtime module tree at {}; found no .rs files, so this scan \ + would be enforcing nothing", + runtime_dir.display() + ); assert!( cli_runtime_source.contains("build_reborn_runtime"), "Reborn CLI should enter the assembled runtime through ironclaw_reborn_composition::build_reborn_runtime" @@ -1008,9 +1014,23 @@ fn reborn_host_runtime_services_do_not_expose_lower_substrate_handles() { let services = std::fs::read_to_string(root.join("crates/ironclaw_host_runtime/src/services.rs")) .expect("host runtime services.rs must be readable"); - let obligations = - std::fs::read_to_string(root.join("crates/ironclaw_host_runtime/src/obligations.rs")) - .expect("host runtime obligations.rs must be readable"); + // WS3 split `obligations.rs` into `obligations/{mod,handler,staged_handoffs, + // process_store,tests}.rs` (one module per chartered owner). This gate is + // about the *escape hatches* the obligation code may not make public, and + // those can now be written in any of those modules — so it scans the whole + // directory rather than one file. The file count is asserted so a future + // move that empties the directory fails here instead of scanning nothing + // and reporting success (the loud-vs-silent distinction WS10 tracks). + let obligations_dir = root.join("crates/ironclaw_host_runtime/src/obligations"); + let mut obligations = String::new(); + let obligations_files = collect_runtime_rs(&obligations_dir, &mut obligations); + assert!( + obligations_files >= 4, + "expected the host-runtime obligation owners under {} (mod + the three chartered \ + modules at minimum); found {obligations_files} .rs files, so this gate would be \ + scanning nothing", + obligations_dir.display() + ); let host_runtime_contract = std::fs::read_to_string(root.join("docs/reborn/contracts/host-runtime.md")) .expect("host runtime contract must be readable"); @@ -4586,35 +4606,37 @@ fn assert_no_normal_workspace_deps<'a>( } /// Recursively concatenate every `.rs` file under `dir` into `out`, -/// descending into subdirectories. Matches the recursion pattern used by -/// `collect_forbidden_*` walkers above so future boundary checks over -/// `runtime/` can reuse the same helper. Used by -/// `reborn_cli_binary_crate_stays_separate_from_v1_root` to scan the -/// entire `runtime/` module tree for forbidden imports. -fn collect_runtime_rs(dir: &std::path::Path, out: &mut String) { - for entry in std::fs::read_dir(dir).unwrap_or_else(|err| { - panic!( - "Reborn CLI runtime directory must be readable at {}: {err}", - dir.display() - ) - }) { +/// descending into subdirectories, and return how many files were read. +/// Matches the recursion pattern used by `collect_forbidden_*` walkers above so +/// future boundary checks over a module tree can reuse the same helper. Used by +/// `reborn_cli_binary_crate_stays_separate_from_v1_root` to scan the entire +/// `runtime/` module tree for forbidden imports, and by +/// `reborn_host_runtime_services_do_not_expose_lower_substrate_handles` to scan +/// the obligation owners. +/// +/// The count is returned so a caller can assert it actually read something: a +/// path-keyed scan that silently walks an empty directory reports success while +/// enforcing nothing. +fn collect_runtime_rs(dir: &std::path::Path, out: &mut String) -> usize { + let mut files = 0usize; + for entry in std::fs::read_dir(dir) + .unwrap_or_else(|err| panic!("directory must be readable at {}: {err}", dir.display())) + { let path = entry.expect("dir entry").path(); if path.is_dir() { - collect_runtime_rs(&path, out); + files += collect_runtime_rs(&path, out); continue; } if path.extension().and_then(|s| s.to_str()) != Some("rs") { continue; } - let content = std::fs::read_to_string(&path).unwrap_or_else(|err| { - panic!( - "Reborn CLI runtime file {} unreadable: {err}", - path.display() - ) - }); + let content = std::fs::read_to_string(&path) + .unwrap_or_else(|err| panic!("file {} unreadable: {err}", path.display())); out.push_str(&content); out.push('\n'); + files += 1; } + files } fn collect_forbidden_runtime_network_uses( diff --git a/crates/ironclaw_architecture/tests/reborn_struct_test_support_ratchet.rs b/crates/ironclaw_architecture/tests/reborn_struct_test_support_ratchet.rs index 9d92e0135af..1e1d9cb2645 100644 --- a/crates/ironclaw_architecture/tests/reborn_struct_test_support_ratchet.rs +++ b/crates/ironclaw_architecture/tests/reborn_struct_test_support_ratchet.rs @@ -300,10 +300,13 @@ const FROZEN_PATH_COUNTS: &[FrozenPathCount] = &[ path: "crates/ironclaw_host_runtime/src/sandbox_process/credential_firewall.rs", count: 1, }, + // WS3 split `obligations.rs` into one module per chartered owner; the + // single frozen test-support method travelled to the staged-handoff owner. + // Repointed, not re-baselined: the count is unchanged at 1. FrozenPathCount { category: "test-support", item_kind: "method", - path: "crates/ironclaw_host_runtime/src/obligations.rs", + path: "crates/ironclaw_host_runtime/src/obligations/staged_handoffs.rs", count: 1, }, FrozenPathCount { diff --git a/crates/ironclaw_host_runtime/AGENTS.md b/crates/ironclaw_host_runtime/AGENTS.md index a0c49b103fd..b7ce1b5afda 100644 --- a/crates/ironclaw_host_runtime/AGENTS.md +++ b/crates/ironclaw_host_runtime/AGENTS.md @@ -16,7 +16,11 @@ - Capability surface: the capability-surface policy `CapabilitySurfacePolicy`/`VisibleCapability`/`VisibleCapabilityAccess` (`surface`); the hot capability catalog `HotCapabilityCatalog`/`HotCapabilityRecord`/`publish_hot_capability_catalog` (`capability_catalog`). - First-party capabilities: the `FirstPartyCapabilityRegistry`/handler/request/result (`first_party`) and the builtin tool set `BuiltinFirstPartyTools` with capability IDs (echo/time/json/http/shell/read_file/write_file/list_dir/glob/grep/apply_patch) and `builtin_first_party_handlers`/`_package` (`first_party_tools`). - Host-owned extension contract discovery: `default_host_api_contract_registry`, `default_host_port_catalog`, `discover_extensions_with_default_host_api_contracts*` (`extension_contracts`). Product-specific manifest contracts are added by the owning composition/product layer. -- Obligation handling: `BuiltinObligationHandler`/`BuiltinObligationServices`, `ProcessObligationLifecycleStore`, and the secret-injection/network-policy stores (`obligations`). +- Obligation handling (`obligations`), split along its **three chartered owners** so no single file fuses them again (PROPOSAL §6.5.9, CHECKLIST WS3). Put new obligation code in the owner it belongs to, never in `mod.rs`: + - `obligations::handler` — which obligations apply and what each one does before/after dispatch: `BuiltinObligationHandler`, the `CapabilityObligationHandler` impl, and the audit / redaction / resource-ceiling / mount validation behind them. + - `obligations::staged_handoffs` — material staged for a *later* consumer: `RuntimeSecretInjectionStore`, `NetworkObligationPolicyStore`, and the `RuntimeCredentialAccountResolver` port that feeds them. + - `obligations::process_store` — `ProcessObligationLifecycleStore`: discarding staged handoffs and reconciling or releasing a prepared reservation once a capability process has started. + - `obligations::mod` holds only `BuiltinObligationServices`, the assembly seam that binds the three for composition, and is deliberately the only place naming all three at once. - The runtime process port `RuntimeProcessPort`/`HostProcessPort` + command execution types (`process_port`) and memory-context builders (`memory_context`). - Production validation of the `TurnRunWakeNotifier` handle consumed by `ironclaw_runner` (`ProductionWiringComponent::TurnRunWakeNotifier`, `src/services/production_wiring.rs`); scheduler/executor ownership lives in that runner-side crate. - Low-level mediation by composing `ironclaw_network`/`ironclaw_secrets`/`ironclaw_resources` (egress, redaction, secret leases, accounting) — never duplicating that logic in runtime crates. diff --git a/crates/ironclaw_host_runtime/CLAUDE.md b/crates/ironclaw_host_runtime/CLAUDE.md index 5e093b79dd4..4ce9a577652 100644 --- a/crates/ironclaw_host_runtime/CLAUDE.md +++ b/crates/ironclaw_host_runtime/CLAUDE.md @@ -4,6 +4,7 @@ - Keep runtime-specific request shapes in the runtime crates; adapters should translate into host API contracts and delegate here. - Compose low-level services such as `ironclaw_network` and `ironclaw_secrets`; do not duplicate URL parsing, DNS checks, private-IP filtering, HTTP clients, secret stores, or redaction logic in runtime crates. - Host HTTP egress lives under `src/egress/`: keep request validation/sanitization, credential-source resolution, staged network-policy lookup, staged secret injection, transport dispatch, response sanitization, and response-body storage as separate pipeline steps instead of rebuilding a monolithic service method. +- Obligation code lives under `src/obligations/` and stays in its owner: `handler` decides and executes obligations, `staged_handoffs` owns the secret/network material staged for a later consumer, `process_store` owns post-start cleanup and reservation reconciliation. `mod.rs` holds only `BuiltinObligationServices` — the assembly seam. A change that needs all three at once is a signal the split is being undone, not a reason to add code to `mod.rs`. Cross-owner access is `pub(super)`, never `pub(crate)`, so widening it is a deliberate edit. - Production host HTTP egress must be constructed with staged `NetworkObligationPolicyStore` and `RuntimeSecretInjectionStore` handoffs. Request-carried policy and direct `SecretStoreLease` sources are legacy/test compatibility paths only. - Preserve the accounting invariant: `network_egress_bytes` is outbound request bytes only, with response bytes tracked separately. - Keep raw secret material inside the narrow lease/injection path. Reject runtime-supplied manual credentials, scan raw and percent-decoded URL forms, redact leased values from runtime-visible errors and responses, strip sensitive response headers, and block credential-shaped runtime requests/responses before they reach external services or runtime callers. diff --git a/crates/ironclaw_host_runtime/src/obligations.rs b/crates/ironclaw_host_runtime/src/obligations.rs deleted file mode 100644 index 0c8493d45ae..00000000000 --- a/crates/ironclaw_host_runtime/src/obligations.rs +++ /dev/null @@ -1,3122 +0,0 @@ -// arch-exempt: large_file, canonical obligation orchestration remains co-located; resolved-source optimization only, plan #5499 -use std::{ - collections::{HashMap, HashSet}, - fmt, - sync::{ - Arc, Mutex, - atomic::{AtomicBool, Ordering}, - }, - time::{Duration, Instant}, -}; - -use async_trait::async_trait; -use chrono::Utc; -use ironclaw_capabilities::{ - CapabilityObligationAbortRequest, CapabilityObligationCompletionRequest, - CapabilityObligationError, CapabilityObligationFailureKind, CapabilityObligationHandler, - CapabilityObligationOutcome, CapabilityObligationPhase, CapabilityObligationRequest, -}; -use ironclaw_events::{ - AuditSink, EventSink, RuntimeEvent, SecurityAuditEvent, SecurityAuditSink, SecurityBoundary, - SecurityDecision, -}; -use ironclaw_host_api::{ - Timestamp, - action::NetworkPolicy, - audit::{ActionResultSummary, ActionSummary, AuditEnvelope, AuditStage, DecisionSummary}, - capability::{EffectKind, RuntimeCredentialAccountSetup}, - decision::{Obligation, RuntimeCredentialAuthRequirement}, - dispatch::{CapabilityDispatchResult, CredentialStageError}, - http::RuntimeHttpEgress, - ids::{AuditEventId, CapabilityId, ExtensionId, ProcessId, SecretHandle, VendorId}, - mount::MountView, - resource::{ - ResourceCeiling, ResourceEstimate, ResourceReservation, ResourceScope, ResourceUsage, - SandboxQuota, - }, -}; -use ironclaw_network::NetworkHttpEgress; -use ironclaw_processes::{ - ProcessError, ProcessJournalCommit, ProcessJournalCommitObserver, ProcessJournalKind, - ProcessKind, ProcessRecord, ProcessRuntimePort, ProcessStart, ProcessSubmissionLifecycle, - capability_process_record, complete_capability_process, fail_capability_process, - process_record_from_snapshot, submit_capability_process, -}; -use ironclaw_resources::{ResourceError, ResourceGovernor}; -use ironclaw_safety::LeakDetector; -use ironclaw_secrets::{ - SecretLease, SecretLeaseId, SecretMaterial, SecretMetadata, SecretStoreError, SecretStorePort, -}; -use secrecy::ExposeSecret; - -use crate::{ - ToolCallHttpEgress, - http_body::{RuntimeHttpBodyStore, UnsupportedRuntimeHttpBodyStore}, -}; - -/// Default maximum lifetime for one-shot runtime secret material staged in memory. -pub(crate) const DEFAULT_RUNTIME_SECRET_INJECTION_TTL: Duration = Duration::from_secs(300); - -#[derive(Debug)] -pub struct RuntimeCredentialAccountRequest<'a> { - pub scope: &'a ResourceScope, - pub provider: &'a VendorId, - pub setup: &'a RuntimeCredentialAccountSetup, - pub provider_scopes: &'a [String], - pub requester_extension: &'a ExtensionId, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct RuntimeCredentialAccessSecret { - pub scope: ResourceScope, - pub handle: SecretHandle, -} - -#[async_trait] -pub trait RuntimeCredentialAccountResolver: Send + Sync + fmt::Debug { - /// Resolve the access-secret source for the requested product-auth account. - /// - /// Returns [`CredentialStageError::AuthRequired`] when the account is - /// missing/unconfigured/expired/revoked (user must re-authenticate), or - /// [`CredentialStageError::Backend`] for internal failures not attributable - /// to user credentials. Shares its error vocabulary with the rest of the - /// staged-credential surface (`ProductAuthCredentialStageError`, - /// `GsuiteCredentialStageError`) so no per-layer error mapping is needed. - async fn resolve_access_secret( - &self, - request: RuntimeCredentialAccountRequest<'_>, - ) -> Result; -} - -/// Runtime secret material staged after `InjectSecretOnce` lease consumption. -/// -/// The store is keyed by scoped invocation, capability, and handle. Runtime adapters -/// borrow staged material during dispatch; `complete_dispatch`/`abort` removes it -/// after the scoped capability finishes. Entries also expire after a short TTL so -/// abandoned handoffs from setup failures, cancellation, or adapter bugs cannot -/// remain usable indefinitely. -#[derive(Clone)] -pub(crate) struct RuntimeSecretInjectionStore { - state: Arc, -} - -struct RuntimeSecretInjectionState { - secrets: Mutex>, - ttl: Duration, -} - -struct RuntimeSecretInjectionEntry { - material: SecretMaterial, - expires_at: Instant, -} - -impl RuntimeSecretInjectionStore { - pub(crate) fn new() -> Self { - Self::default() - } - - pub(crate) fn with_ttl(ttl: Duration) -> Self { - Self { - state: Arc::new(RuntimeSecretInjectionState { - secrets: Mutex::new(HashMap::new()), - ttl, - }), - } - } - - pub(crate) fn insert( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - handle: &SecretHandle, - material: SecretMaterial, - ) -> Result<(), RuntimeSecretInjectionStoreError> { - let now = Instant::now(); - let expires_at = now.checked_add(self.state.ttl).unwrap_or(now); - let mut secrets = self.lock()?; - prune_expired_entries(&mut secrets, now); - secrets.insert( - RuntimeSecretInjectionKey::new(scope, capability_id, handle), - RuntimeSecretInjectionEntry { - material, - expires_at, - }, - ); - Ok(()) - } - - pub(crate) fn take( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - handle: &SecretHandle, - ) -> Result, RuntimeSecretInjectionStoreError> { - let now = Instant::now(); - let mut secrets = self.lock()?; - prune_expired_entries(&mut secrets, now); - Ok(secrets - .remove(&RuntimeSecretInjectionKey::new( - scope, - capability_id, - handle, - )) - .map(|entry| entry.material)) - } - - pub(crate) fn clone_material( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - handle: &SecretHandle, - ) -> Result, RuntimeSecretInjectionStoreError> { - let now = Instant::now(); - let mut secrets = self.lock()?; - prune_expired_entries(&mut secrets, now); - Ok(secrets - .get(&RuntimeSecretInjectionKey::new( - scope, - capability_id, - handle, - )) - .map(|entry| SecretMaterial::from(entry.material.expose_secret()))) - } - - /// Discard all staged secrets for a scoped capability before process ownership exists. - /// - /// Background process lifecycle cleanup is guarded by a single-active-handoff - /// invariant for the scoped capability; this method remains the abort/inline cleanup seam. - pub(crate) fn discard_for_capability( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - ) -> Result<(), RuntimeSecretInjectionStoreError> { - let scope_key = RuntimeSecretInjectionScopeKey::new(scope, capability_id); - let mut secrets = self.lock()?; - prune_expired_entries(&mut secrets, Instant::now()); - secrets.retain(|key, _| !key.matches_scope(&scope_key)); - Ok(()) - } - - fn has_for_capability( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - ) -> Result { - let scope_key = RuntimeSecretInjectionScopeKey::new(scope, capability_id); - let mut secrets = self.lock()?; - prune_expired_entries(&mut secrets, Instant::now()); - Ok(secrets.keys().any(|key| key.matches_scope(&scope_key))) - } - - #[cfg(test)] - fn prune_expired(&self) -> Result { - let mut secrets = self.lock()?; - Ok(prune_expired_entries(&mut secrets, Instant::now())) - } - - fn lock( - &self, - ) -> Result< - std::sync::MutexGuard<'_, HashMap>, - RuntimeSecretInjectionStoreError, - > { - self.state - .secrets - .lock() - .map_err(|_| RuntimeSecretInjectionStoreError::Unavailable) - } -} - -impl Default for RuntimeSecretInjectionStore { - fn default() -> Self { - Self::with_ttl(DEFAULT_RUNTIME_SECRET_INJECTION_TTL) - } -} - -impl fmt::Debug for RuntimeSecretInjectionStore { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter - .debug_struct("RuntimeSecretInjectionStore") - .field("secrets", &"[REDACTED]") - .field("ttl", &self.state.ttl) - .finish() - } -} - -fn prune_expired_entries( - secrets: &mut HashMap, - now: Instant, -) -> usize { - let before = secrets.len(); - secrets.retain(|_, entry| entry.expires_at > now); - before.saturating_sub(secrets.len()) -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub(crate) enum RuntimeSecretInjectionStoreError { - Unavailable, -} - -impl fmt::Display for RuntimeSecretInjectionStoreError { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::Unavailable => formatter.write_str("runtime secret injection store unavailable"), - } - } -} - -impl std::error::Error for RuntimeSecretInjectionStoreError {} - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct RuntimeSecretInjectionKey { - tenant_id: String, - user_id: String, - agent_id: Option, - project_id: Option, - mission_id: Option, - thread_id: Option, - invocation_id: String, - capability_id: String, - handle: String, -} - -impl RuntimeSecretInjectionKey { - fn new(scope: &ResourceScope, capability_id: &CapabilityId, handle: &SecretHandle) -> Self { - Self { - tenant_id: scope.tenant_id.as_str().to_string(), - user_id: scope.user_id.as_str().to_string(), - agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), - project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), - mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), - thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), - invocation_id: scope.invocation_id.to_string(), - capability_id: capability_id.as_str().to_string(), - handle: handle.as_str().to_string(), - } - } - - fn matches_scope(&self, scope: &RuntimeSecretInjectionScopeKey) -> bool { - self.tenant_id == scope.tenant_id - && self.user_id == scope.user_id - && self.agent_id == scope.agent_id - && self.project_id == scope.project_id - && self.mission_id == scope.mission_id - && self.thread_id == scope.thread_id - && self.invocation_id == scope.invocation_id - && self.capability_id == scope.capability_id - } -} - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct RuntimeSecretInjectionScopeKey { - tenant_id: String, - user_id: String, - agent_id: Option, - project_id: Option, - mission_id: Option, - thread_id: Option, - invocation_id: String, - capability_id: String, -} - -impl RuntimeSecretInjectionScopeKey { - fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { - Self { - tenant_id: scope.tenant_id.as_str().to_string(), - user_id: scope.user_id.as_str().to_string(), - agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), - project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), - mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), - thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), - invocation_id: scope.invocation_id.to_string(), - capability_id: capability_id.as_str().to_string(), - } - } -} - -/// In-memory policy handoff from obligation handling to runtime adapters. -/// -/// Policies are keyed by tenant/user/project/mission/thread/invocation scope and -/// capability id. Runtime adapters and host egress borrow the staged policy for -/// every network operation in the invocation; obligation completion/abort or -/// process lifecycle cleanup owns the final discard. -#[derive(Debug, Clone, Default)] -pub(crate) struct NetworkObligationPolicyStore { - policies: Arc>>, -} - -impl NetworkObligationPolicyStore { - pub(crate) fn new() -> Self { - Self::default() - } - - pub(crate) fn insert( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - policy: NetworkPolicy, - ) { - self.policies - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .insert(NetworkPolicyKey::new(scope, capability_id), policy); - } - - pub(crate) fn get( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - ) -> Option { - self.policies - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .get(&NetworkPolicyKey::new(scope, capability_id)) - .cloned() - } - - pub(crate) fn take( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - ) -> Option { - self.policies - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .remove(&NetworkPolicyKey::new(scope, capability_id)) - } - - /// Discard a staged policy for a scoped capability before process ownership exists. - /// - /// Background process lifecycle cleanup is guarded by a single-active-handoff - /// invariant for the scoped capability; this method remains the abort/inline cleanup seam. - pub(crate) fn discard_for_capability( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - ) { - let _ = self.take(scope, capability_id); - } - - fn contains(&self, scope: &ResourceScope, capability_id: &CapabilityId) -> bool { - self.policies - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .contains_key(&NetworkPolicyKey::new(scope, capability_id)) - } -} - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct NetworkPolicyKey { - tenant_id: String, - user_id: String, - agent_id: Option, - project_id: Option, - mission_id: Option, - thread_id: Option, - invocation_id: String, - capability_id: String, -} - -impl NetworkPolicyKey { - fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { - Self { - tenant_id: scope.tenant_id.as_str().to_string(), - user_id: scope.user_id.as_str().to_string(), - agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), - project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), - mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), - thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), - invocation_id: scope.invocation_id.to_string(), - capability_id: capability_id.as_str().to_string(), - } - } -} - -/// Host-runtime-owned backing services for a fully configured built-in obligation handler. -/// -/// This value is the production composition seam for obligation handling. It -/// keeps the in-memory network-policy and runtime-secret handoff stores alive -/// outside the handler so runtime adapters can consume the exact staged state -/// that [`BuiltinObligationHandler`] prepares before dispatch. -#[derive(Clone)] -pub struct BuiltinObligationServices { - audit_sink: Arc, - network_policies: Arc, - secret_store: Arc, - secret_injections: Arc, - resource_governor: Arc, - credential_account_resolver: Option>, -} - -impl BuiltinObligationServices { - pub fn new( - audit_sink: Arc, - secret_store: Arc, - resource_governor: Arc, - ) -> Self { - Self::with_handoff_stores( - audit_sink, - Arc::new(NetworkObligationPolicyStore::new()), - secret_store, - Arc::new(RuntimeSecretInjectionStore::new()), - resource_governor, - ) - } - - pub(crate) fn with_handoff_stores( - audit_sink: Arc, - network_policies: Arc, - secret_store: Arc, - secret_injections: Arc, - resource_governor: Arc, - ) -> Self { - Self { - audit_sink, - network_policies, - secret_store, - secret_injections, - resource_governor, - credential_account_resolver: None, - } - } - - pub fn with_credential_account_resolver(mut self, resolver: Arc) -> Self - where - T: RuntimeCredentialAccountResolver + 'static, - { - self.credential_account_resolver = Some(resolver); - self - } - - pub fn with_credential_account_resolver_dyn( - mut self, - resolver: Arc, - ) -> Self { - self.credential_account_resolver = Some(resolver); - self - } - - pub fn audit_sink(&self) -> Arc { - self.audit_sink.clone() - } - - pub fn secret_store(&self) -> Arc { - self.secret_store.clone() - } - - pub fn resource_governor(&self) -> Arc { - self.resource_governor.clone() - } - - /// Builds host HTTP egress over this service graph's private handoff stores. - /// Callers can supply concrete network transport without receiving mutable - /// access to staged policy or secret material. - pub fn host_http_egress( - &self, - network: N, - ) -> impl RuntimeHttpEgress + ToolCallHttpEgress + use - where - N: NetworkHttpEgress + 'static, - { - self.host_http_egress_with_body_store(network, Arc::new(UnsupportedRuntimeHttpBodyStore)) - } - - pub fn host_http_egress_with_body_store( - &self, - network: N, - body_store: Arc, - ) -> impl RuntimeHttpEgress + ToolCallHttpEgress + use - where - N: NetworkHttpEgress + 'static, - T: RuntimeHttpBodyStore + 'static, - { - let body_store: Arc = body_store; - crate::HostHttpEgressService::production( - network, - SharedSecretStore(self.secret_store.clone()), - self.network_policies.clone(), - self.secret_injections.clone(), - body_store, - ) - } - - pub fn process_obligation_lifecycle_store( - &self, - inner: Arc, - ) -> ProcessObligationLifecycleStore - where - S: ProcessRuntimePort + 'static, - { - ProcessObligationLifecycleStore::new( - inner, - self.network_policies.clone(), - self.secret_injections.clone(), - self.resource_governor.clone(), - ) - } - - pub fn process_obligation_lifecycle_store_dyn( - &self, - inner: Arc, - ) -> ProcessObligationLifecycleStore { - ProcessObligationLifecycleStore::from_dyn( - inner, - self.network_policies.clone(), - self.secret_injections.clone(), - self.resource_governor.clone(), - ) - } - - pub fn obligation_handler(&self) -> BuiltinObligationHandler { - let handler = BuiltinObligationHandler::new() - .with_audit_sink_dyn(self.audit_sink.clone()) - .with_network_policy_store(self.network_policies.clone()) - .with_secret_store_dyn(self.secret_store.clone()) - .with_secret_injection_store(self.secret_injections.clone()) - .with_resource_governor_dyn(self.resource_governor.clone()); - match &self.credential_account_resolver { - Some(resolver) => handler.with_credential_account_resolver_dyn(Arc::clone(resolver)), - None => handler, - } - } -} - -impl fmt::Debug for BuiltinObligationServices { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - formatter - .debug_struct("BuiltinObligationServices") - .field("audit_sink", &"") - .field("network_policies", &self.network_policies) - .field("secret_store", &"[REDACTED]") - .field("secret_injections", &self.secret_injections) - .field("resource_governor", &"") - .field( - "credential_account_resolver", - &self - .credential_account_resolver - .as_ref() - .map(|_| ""), - ) - .finish() - } -} - -#[derive(Clone)] -pub(crate) struct SharedSecretStore(pub(crate) Arc); - -#[async_trait] -impl SecretStorePort for SharedSecretStore { - async fn put( - &self, - scope: ResourceScope, - handle: SecretHandle, - material: SecretMaterial, - expires_at: Option, - ) -> Result { - self.0.put(scope, handle, material, expires_at).await - } - - async fn metadata( - &self, - scope: &ResourceScope, - handle: &SecretHandle, - ) -> Result, SecretStoreError> { - self.0.metadata(scope, handle).await - } - - async fn metadata_for_scope( - &self, - scope: &ResourceScope, - ) -> Result, SecretStoreError> { - self.0.metadata_for_scope(scope).await - } - - async fn delete( - &self, - scope: &ResourceScope, - handle: &SecretHandle, - ) -> Result { - self.0.delete(scope, handle).await - } - - async fn lease_once( - &self, - scope: &ResourceScope, - handle: &SecretHandle, - ) -> Result { - self.0.lease_once(scope, handle).await - } - - async fn consume( - &self, - scope: &ResourceScope, - lease_id: SecretLeaseId, - ) -> Result { - self.0.consume(scope, lease_id).await - } - - async fn revoke( - &self, - scope: &ResourceScope, - lease_id: SecretLeaseId, - ) -> Result { - self.0.revoke(scope, lease_id).await - } - - async fn leases_for_scope( - &self, - scope: &ResourceScope, - ) -> Result, SecretStoreError> { - self.0.leases_for_scope(scope).await - } -} - -/// Process-store wrapper that owns spawn-phase obligation handoffs after -/// process submission succeeds. -/// -/// `CapabilityHost` aborts prepared effects when process start fails. Once -/// start succeeds, this wrapper becomes responsible for discarding staged -/// network/secret handoffs and reconciling or releasing a prepared resource -/// reservation when the process reaches a terminal state. -pub struct ProcessObligationLifecycleStore { - processes: Arc, - network_policies: Arc, - secret_injections: Arc, - resource_governor: Mutex>, - event_sink: Mutex>>, - observer_registered: AtomicBool, - active_process_handoffs: Mutex>, - cleaned_process_handoffs: Mutex>, -} - -impl ProcessObligationLifecycleStore { - pub(crate) fn new( - inner: Arc, - network_policies: Arc, - secret_injections: Arc, - resource_governor: Arc, - ) -> Self - where - S: ProcessRuntimePort + 'static, - { - let inner: Arc = inner; - Self::from_dyn( - inner, - network_policies, - secret_injections, - resource_governor, - ) - } - - pub(crate) fn from_dyn( - processes: Arc, - network_policies: Arc, - secret_injections: Arc, - resource_governor: Arc, - ) -> Self { - Self { - processes, - network_policies, - secret_injections, - resource_governor: Mutex::new(resource_governor), - event_sink: Mutex::new(None), - observer_registered: AtomicBool::new(false), - active_process_handoffs: Mutex::new(HashMap::new()), - cleaned_process_handoffs: Mutex::new(HashSet::new()), - } - } - - pub(crate) fn set_resource_governor(&self, resource_governor: Arc) { - match self.resource_governor.lock() { - Ok(mut slot) => { - *slot = resource_governor; - } - Err(error) => { - tracing::error!(error = %error, "process resource governor registry unavailable"); - } - } - } - - #[doc(hidden)] - pub fn register_journal_observer( - self: &Arc, - runtime: &dyn ProcessRuntimePort, - ) -> Result<(), String> { - if self - .observer_registered - .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) - .is_err() - { - return Ok(()); - } - let observer: Arc = self.clone(); - if let Err(error) = runtime.subscribe_process_observer(observer) { - self.observer_registered.store(false, Ordering::Release); - return Err(error); - } - Ok(()) - } - - /// Attaches a best-effort event sink for process lifecycle transitions. - pub fn set_event_sink(&self, event_sink: Arc) { - match self.event_sink.lock() { - Ok(mut slot) => { - *slot = Some(event_sink); - } - Err(error) => { - tracing::debug!( - error = %error, - "process lifecycle event sink registry unavailable" - ); - } - } - } - - async fn emit_process_event(&self, event: RuntimeEvent) { - let event_sink = match self.event_sink.lock() { - Ok(slot) => slot.clone(), - Err(error) => { - tracing::debug!( - error = %error, - "process lifecycle event sink registry unavailable" - ); - None - } - }; - if let Some(event_sink) = event_sink - && let Err(error) = event_sink.emit(event).await - { - tracing::debug!(?error, "best-effort process lifecycle event emit failed"); - } - } - - /// Discards staged obligation handoffs and closes any reservation for an - /// executor that finished but could not publish its result record. - pub async fn cleanup_process_obligations( - &self, - scope: &ResourceScope, - process_id: ProcessId, - reconcile: bool, - ) -> Result<(), ProcessError> { - if let Some(record) = - capability_process_record(self.processes.as_ref(), scope, process_id).await? - { - self.cleanup_record_obligations(&record, reconcile)?; - self.release_active_process_handoff(&record)?; - self.mark_process_handoff_cleaned(&record)?; - } - Ok(()) - } - - fn has_process_obligations(&self, start: &ProcessStart) -> Result { - let has_secret_handoff = self - .secret_injections - .has_for_capability(&start.scope, &start.capability_id) - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff lookup failed".to_string(), - })?; - Ok(start.resource_reservation_id.is_some() - || self - .network_policies - .contains(&start.scope, &start.capability_id) - || has_secret_handoff) - } - - fn claim_active_process_handoff(&self, start: &ProcessStart) -> Result { - if !self.has_process_obligations(start)? { - return Ok(false); - } - - let key = ProcessObligationHandoffKey::new(&start.scope, &start.capability_id); - let mut active = - self.active_process_handoffs - .lock() - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff registry unavailable".to_string(), - })?; - if let Some(existing_process_id) = active.get(&key) { - return Err(ProcessError::InvalidStoredRecord { - reason: format!( - "process obligation handoff already active for scoped capability: {existing_process_id}" - ), - }); - } - active.insert(key, start.process_id); - Ok(true) - } - - fn release_claimed_process_handoff( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - process_id: ProcessId, - ) -> Result<(), ProcessError> { - let key = ProcessObligationHandoffKey::new(scope, capability_id); - let mut active = - self.active_process_handoffs - .lock() - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff registry unavailable".to_string(), - })?; - if active.get(&key) == Some(&process_id) { - active.remove(&key); - } - Ok(()) - } - - fn release_active_process_handoff(&self, record: &ProcessRecord) -> Result<(), ProcessError> { - self.release_claimed_process_handoff( - &record.scope, - &record.capability_id, - record.process_id, - ) - } - - fn has_active_process_handoff(&self, record: &ProcessRecord) -> Result { - let key = ProcessObligationHandoffKey::new(&record.scope, &record.capability_id); - let active = - self.active_process_handoffs - .lock() - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff registry unavailable".to_string(), - })?; - Ok(active.get(&key) == Some(&record.process_id)) - } - - fn process_handoff_cleaned(&self, record: &ProcessRecord) -> Result { - let key = ProcessObligationProcessKey::new(&record.scope, record.process_id); - let cleaned = self.cleaned_process_handoffs.lock().map_err(|_| { - ProcessError::InvalidStoredRecord { - reason: "process obligation cleanup registry unavailable".to_string(), - } - })?; - Ok(cleaned.contains(&key)) - } - - fn mark_process_handoff_cleaned(&self, record: &ProcessRecord) -> Result<(), ProcessError> { - let key = ProcessObligationProcessKey::new(&record.scope, record.process_id); - let mut cleaned = self.cleaned_process_handoffs.lock().map_err(|_| { - ProcessError::InvalidStoredRecord { - reason: "process obligation cleanup registry unavailable".to_string(), - } - })?; - cleaned.insert(key); - Ok(()) - } - - fn has_staged_handoffs(&self, record: &ProcessRecord) -> Result { - let has_secret_handoff = self - .secret_injections - .has_for_capability(&record.scope, &record.capability_id) - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff lookup failed".to_string(), - })?; - Ok(self - .network_policies - .contains(&record.scope, &record.capability_id) - || has_secret_handoff) - } - - fn cleanup_terminal( - &self, - record: &ProcessRecord, - reconcile: bool, - ) -> Result<(), ProcessError> { - if let Err(error) = self.cleanup_record_obligations(record, reconcile) { - tracing::warn!( - process_id = %record.process_id, - tenant_id = %record.scope.tenant_id, - user_id = %record.scope.user_id, - reconcile, - error = %error, - "process obligation cleanup failed after terminal transition" - ); - return Err(error); - } - self.release_active_process_handoff(record)?; - self.mark_process_handoff_cleaned(record)?; - Ok(()) - } - - fn cleanup_record_obligations( - &self, - record: &ProcessRecord, - reconcile: bool, - ) -> Result<(), ProcessError> { - if self.process_handoff_cleaned(record)? { - return Ok(()); - } - let should_cleanup_handoffs = self.has_active_process_handoff(record)? - || record.resource_reservation_id.is_some() - || self.has_staged_handoffs(record)?; - if should_cleanup_handoffs { - self.network_policies - .discard_for_capability(&record.scope, &record.capability_id); - self.secret_injections - .discard_for_capability(&record.scope, &record.capability_id) - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process obligation handoff cleanup failed".to_string(), - })?; - } - if let Some(reservation_id) = record.resource_reservation_id { - let governor = - self.resource_governor - .lock() - .map_err(|_| ProcessError::InvalidStoredRecord { - reason: "process resource governor registry unavailable".to_string(), - })?; - if reconcile { - close_reservation_once( - governor.reconcile(reservation_id, ResourceUsage::default()), - )?; - } else { - close_reservation_once(governor.release(reservation_id))?; - } - } - Ok(()) - } -} - -#[async_trait] -impl ProcessJournalCommitObserver for ProcessObligationLifecycleStore { - fn process_observer_id(&self) -> &'static str { - "process-obligation-lifecycle-v1" - } - - async fn observe_process_commit(&self, commit: ProcessJournalCommit) -> Result<(), String> { - if commit.state.process_kind != ProcessKind::CapabilityInvocation { - return Ok(()); - } - let record = - process_record_from_snapshot(commit.state).map_err(|error| error.to_string())?; - match commit.kind { - ProcessJournalKind::Completed => { - self.emit_process_event(RuntimeEvent::process_completed( - record.scope.clone(), - record.capability_id.clone(), - record.extension_id.clone(), - record.runtime, - record.process_id, - )) - .await; - self.cleanup_terminal(&record, true) - .map_err(|error| error.to_string())?; - } - ProcessJournalKind::Failed => { - self.emit_process_event(RuntimeEvent::process_failed( - record.scope.clone(), - record.capability_id.clone(), - record.extension_id.clone(), - record.runtime, - record.process_id, - record - .error_kind - .clone() - .unwrap_or_else(|| "unknown".to_string()), - )) - .await; - self.cleanup_terminal(&record, false) - .map_err(|error| error.to_string())?; - } - ProcessJournalKind::Stopped - | ProcessJournalKind::Cancelled - | ProcessJournalKind::Killed - | ProcessJournalKind::RecoveryRequired => { - self.emit_process_event(RuntimeEvent::process_killed( - record.scope.clone(), - record.capability_id.clone(), - record.extension_id.clone(), - record.runtime, - record.process_id, - )) - .await; - self.cleanup_terminal(&record, false) - .map_err(|error| error.to_string())?; - } - _ => {} - } - Ok(()) - } -} - -#[async_trait] -impl ProcessSubmissionLifecycle for ProcessObligationLifecycleStore { - async fn before_submit(&self, start: &ProcessStart) -> Result<(), ProcessError> { - self.claim_active_process_handoff(start).map(|_| ()) - } - - async fn submit_failed(&self, start: &ProcessStart) -> Result<(), ProcessError> { - self.release_claimed_process_handoff(&start.scope, &start.capability_id, start.process_id) - } - - async fn submitted(&self, record: &ProcessRecord) -> Result<(), ProcessError> { - self.emit_process_event(RuntimeEvent::process_started( - record.scope.clone(), - record.capability_id.clone(), - record.extension_id.clone(), - record.runtime, - record.process_id, - )) - .await; - Ok(()) - } -} - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct ProcessObligationHandoffKey { - tenant_id: String, - user_id: String, - agent_id: Option, - project_id: Option, - mission_id: Option, - thread_id: Option, - invocation_id: String, - capability_id: String, -} - -impl ProcessObligationHandoffKey { - fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { - Self { - tenant_id: scope.tenant_id.as_str().to_string(), - user_id: scope.user_id.as_str().to_string(), - agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), - project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), - mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), - thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), - invocation_id: scope.invocation_id.to_string(), - capability_id: capability_id.as_str().to_string(), - } - } -} - -#[derive(Debug, Clone, PartialEq, Eq, Hash)] -struct ProcessObligationProcessKey { - tenant_id: String, - user_id: String, - agent_id: Option, - project_id: Option, - mission_id: Option, - thread_id: Option, - process_id: ProcessId, -} - -impl ProcessObligationProcessKey { - fn new(scope: &ResourceScope, process_id: ProcessId) -> Self { - Self { - tenant_id: scope.tenant_id.as_str().to_string(), - user_id: scope.user_id.as_str().to_string(), - agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), - project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), - mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), - thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), - process_id, - } - } -} - -impl ProcessObligationLifecycleStore { - pub fn process_runtime(&self) -> Arc { - Arc::clone(&self.processes) - } - - pub async fn start(&self, start: ProcessStart) -> Result { - let claimed = self.claim_active_process_handoff(&start)?; - let process_id = start.process_id; - let scope = start.scope.clone(); - let capability_id = start.capability_id.clone(); - match submit_capability_process(self.processes.as_ref(), start).await { - Ok(record) => { - self.emit_process_event(RuntimeEvent::process_started( - record.scope.clone(), - record.capability_id.clone(), - record.extension_id.clone(), - record.runtime, - record.process_id, - )) - .await; - Ok(record) - } - Err(error) => { - if claimed { - self.release_claimed_process_handoff(&scope, &capability_id, process_id)?; - } - Err(error) - } - } - } - - pub async fn complete( - &self, - scope: &ResourceScope, - process_id: ProcessId, - ) -> Result { - let record = - complete_capability_process(self.processes.as_ref(), scope, process_id).await?; - self.cleanup_terminal(&record, true)?; - Ok(record) - } - - pub async fn fail( - &self, - scope: &ResourceScope, - process_id: ProcessId, - error_kind: String, - ) -> Result { - let record = - fail_capability_process(self.processes.as_ref(), scope, process_id, error_kind).await?; - self.cleanup_terminal(&record, false)?; - Ok(record) - } - - pub async fn kill( - &self, - scope: &ResourceScope, - process_id: ProcessId, - ) -> Result { - let result = self - .processes - .kill_process(ironclaw_processes::KillProcessRequest { - scope: scope.clone(), - process_id, - operation_id: None, - reason: None, - }) - .await - .map_err(|error| ProcessError::InvalidStoredRecord { - reason: error.to_string(), - })?; - let record = process_record_from_snapshot(result.state)?; - self.cleanup_terminal(&record, false)?; - Ok(record) - } - - pub async fn get( - &self, - scope: &ResourceScope, - process_id: ProcessId, - ) -> Result, ProcessError> { - capability_process_record(self.processes.as_ref(), scope, process_id).await - } - - pub async fn records_for_scope( - &self, - scope: &ResourceScope, - ) -> Result, ProcessError> { - self.processes - .process_snapshots(scope) - .await - .map_err(|error| ProcessError::InvalidStoredRecord { - reason: error.to_string(), - })? - .into_iter() - .filter(|snapshot| snapshot.process_kind == ProcessKind::CapabilityInvocation) - .map(process_record_from_snapshot) - .collect() - } -} - -fn close_reservation_once(result: Result) -> Result<(), ProcessError> { - match result { - Ok(_) => Ok(()), - Err(ResourceError::ReservationClosed { .. }) => Ok(()), - Err(ResourceError::UnknownReservation { .. }) => Ok(()), - Err(error) => Err(error.into()), - } -} - -/// Built-in obligation handler for the current host-runtime slice. -#[derive(Clone, Default)] -pub struct BuiltinObligationHandler { - audit_sink: Option>, - security_audit_sink: Option>, - network_policies: Option>, - secret_store: Option>, - secret_injections: Option>, - resource_governor: Option>, - credential_account_resolver: Option>, -} - -struct ResolvedSecretInjection { - handle: SecretHandle, - source_scope: ResourceScope, -} - -impl BuiltinObligationHandler { - pub fn new() -> Self { - Self::default() - } - - pub fn with_audit_sink(mut self, sink: Arc) -> Self - where - T: AuditSink + 'static, - { - let sink: Arc = sink; - self.audit_sink = Some(sink); - self - } - - pub fn with_audit_sink_dyn(mut self, sink: Arc) -> Self { - self.audit_sink = Some(sink); - self - } - - /// Wire in a [`SecurityAuditSink`] for boundary-decision recording. - /// - /// Currently consumed by the output-redaction (leak-detector) path in - /// [`Self::complete_dispatch`]. Additional boundaries inside this handler - /// will adopt the same sink in follow-up PRs; the wiring is intentionally - /// optional so unconfigured callers keep working unchanged. - pub fn with_security_audit_sink(mut self, sink: Arc) -> Self { - self.security_audit_sink = Some(sink); - self - } - - pub(crate) fn with_network_policy_store( - mut self, - store: Arc, - ) -> Self { - self.network_policies = Some(store); - self - } - - pub fn with_secret_store(mut self, store: Arc) -> Self - where - T: SecretStorePort + 'static, - { - let store: Arc = store; - self.secret_store = Some(store); - self - } - - pub fn with_secret_store_dyn(mut self, store: Arc) -> Self { - self.secret_store = Some(store); - self - } - - pub(crate) fn with_secret_injection_store( - mut self, - store: Arc, - ) -> Self { - self.secret_injections = Some(store); - self - } - - pub fn with_resource_governor(mut self, governor: Arc) -> Self - where - T: ResourceGovernor + 'static, - { - let governor: Arc = governor; - self.resource_governor = Some(governor); - self - } - - pub fn with_resource_governor_dyn(mut self, governor: Arc) -> Self { - self.resource_governor = Some(governor); - self - } - - pub fn with_credential_account_resolver(mut self, resolver: Arc) -> Self - where - T: RuntimeCredentialAccountResolver + 'static, - { - let resolver: Arc = resolver; - self.credential_account_resolver = Some(resolver); - self - } - - pub fn with_credential_account_resolver_dyn( - mut self, - resolver: Arc, - ) -> Self { - self.credential_account_resolver = Some(resolver); - self - } - - async fn emit_audit_before( - &self, - request: &CapabilityObligationRequest<'_>, - ) -> Result<(), CapabilityObligationError> { - let Some(audit_sink) = &self.audit_sink else { - return Err(CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Audit, - }); - }; - - audit_sink - .emit_audit(audit_before_record(request)) - .await - .map_err(|_| CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Audit, - }) - } - - async fn preflight_secret_injection( - &self, - request: &CapabilityObligationRequest<'_>, - handles: &[SecretHandle], - ) -> Result, CapabilityObligationError> { - if handles.is_empty() { - return Ok(Vec::new()); - } - let Some(secret_store) = &self.secret_store else { - return Err(secret_obligation_failed()); - }; - if self.secret_injections.is_none() { - return Err(secret_obligation_failed()); - } - let mut resolved = Vec::with_capacity(handles.len()); - for handle in handles { - // Fail closed on a store error: the dispatch-time backstop must never - // let an uncredentialed call through on a transient failure. Preserve the - // cause as a server-side trail (`SecretStoreError` Display carries no raw - // secret material — handles/reasons only); the caller still receives the - // opaque, sanitized secret-obligation failure. - let owner = match secret_owner_scope( - secret_store.as_ref(), - &request.context.resource_scope, - handle, - ) - .await - { - Ok(owner) => owner, - Err(error) => { - tracing::debug!( - secret_handle = handle.as_str(), - error = %error, - "dispatch-time secret presence probe failed; failing closed at the obligation backstop" - ); - return Err(secret_obligation_failed()); - } - }; - let Some(source_scope) = owner else { - return Err(CapabilityObligationError::AuthRequired { - credential_requirements: Vec::new(), - }); - }; - resolved.push(ResolvedSecretInjection { - handle: handle.clone(), - source_scope, - }); - } - Ok(resolved) - } - - async fn inject_secrets( - &self, - request: &CapabilityObligationRequest<'_>, - resolved: &[ResolvedSecretInjection], - ) -> Result<(), CapabilityObligationError> { - if resolved.is_empty() { - return Ok(()); - } - let Some(secret_store) = &self.secret_store else { - return Err(secret_obligation_failed()); - }; - let Some(secret_injections) = &self.secret_injections else { - return Err(secret_obligation_failed()); - }; - - let mut material = Vec::with_capacity(resolved.len()); - for resolved in resolved { - // Use the same source scope the presence probe accepted: the caller's - // own secret if present, else the tenant-shared admin-managed secret - // (#5459). The injection target below stays the caller's invocation - // slot regardless of where the source material came from. The - // lease/consume operations remain authoritative if the secret - // vanishes between preflight and here. - // Every arm below fails closed; the bound error is logged first so a - // shared-secret lease/consume fault (e.g. an AAD/scope regression on - // the cross-scope read) leaves a server-side trail. `SecretStoreError` - // Display carries no raw secret material — handles/reasons only. - let lease = secret_store - .lease_once(&resolved.source_scope, &resolved.handle) - .await - .map_err(|error| { - tracing::debug!( - secret_handle = resolved.handle.as_str(), - error = %error, - "secret injection: lease failed; failing closed" - ); - secret_obligation_failed() - })?; - let secret = secret_store - .consume(&resolved.source_scope, lease.id) - .await - .map_err(|error| { - tracing::debug!( - secret_handle = resolved.handle.as_str(), - error = %error, - "secret injection: lease consume failed; failing closed" - ); - secret_obligation_failed() - })?; - material.push((resolved.handle.clone(), secret)); - } - - for (handle, secret) in material { - secret_injections - .insert( - &request.context.resource_scope, - request.capability_id, - &handle, - secret, - ) - .map_err(|error| { - tracing::debug!( - secret_handle = handle.as_str(), - error = %error, - "secret injection: injection-slot insert failed; failing closed" - ); - secret_obligation_failed() - })?; - } - Ok(()) - } - - async fn inject_credential_accounts( - &self, - request: &CapabilityObligationRequest<'_>, - ) -> Result<(), CapabilityObligationError> { - let account_obligations = credential_account_injection_obligations(request.obligations); - if account_obligations.is_empty() { - return Ok(()); - } - let Some(resolver) = &self.credential_account_resolver else { - return Err(secret_obligation_failed()); - }; - let Some(secret_store) = &self.secret_store else { - return Err(secret_obligation_failed()); - }; - let Some(secret_injections) = &self.secret_injections else { - return Err(secret_obligation_failed()); - }; - - for obligation in account_obligations { - let access_secret = resolver - .resolve_access_secret(RuntimeCredentialAccountRequest { - scope: &request.context.resource_scope, - provider: obligation.provider, - setup: obligation.setup, - provider_scopes: obligation.provider_scopes, - requester_extension: obligation.requester_extension, - }) - .await - .map_err(|error| { - credential_stage_error_to_obligation_error(error, Some(&obligation)) - })?; - // Retrieve and stage the resolved credential under the obligation's injection handle. - // The access_secret names the material in the secret store; obligation.handle is - // the slot name the WASM guest expects. - stage_credential_material( - secret_store.as_ref(), - secret_injections, - &access_secret.scope, - &request.context.resource_scope, - request.capability_id, - &access_secret.handle, - obligation.handle, - ) - .await - .map_err(|error| { - credential_stage_error_to_obligation_error(error, Some(&obligation)) - })?; - } - - Ok(()) - } - - fn reserve_resource_obligation( - &self, - request: &CapabilityObligationRequest<'_>, - ) -> Result, CapabilityObligationError> { - let mut reservation_id = None; - for obligation in request.obligations { - if let Obligation::ReserveResources { reservation_id: id } = obligation { - if reservation_id.is_some() { - return Err(resource_obligation_failed()); - } - reservation_id = Some(*id); - } - } - let Some(reservation_id) = reservation_id else { - return Ok(None); - }; - let Some(governor) = &self.resource_governor else { - return Err(resource_obligation_failed()); - }; - governor - .reserve_with_id( - request.context.resource_scope.clone(), - request.estimate.clone(), - reservation_id, - ) - .map(Some) - .map_err(|_| resource_obligation_failed()) - } - - fn preflight_resource_ceiling( - &self, - request: &CapabilityObligationRequest<'_>, - ) -> Result<(), CapabilityObligationError> { - let Some(ceiling) = resource_ceiling_obligation(request.obligations)? else { - return Ok(()); - }; - validate_supported_resource_ceiling(ceiling)?; - validate_estimate_within_ceiling(request.estimate, ceiling) - } - - async fn finish_prepare( - &self, - request: &CapabilityObligationRequest<'_>, - resolved_secret_injections: &[ResolvedSecretInjection], - network_policy: Option, - ) -> Result<(), CapabilityObligationError> { - if request - .obligations - .iter() - .any(|obligation| matches!(obligation, Obligation::AuditBefore)) - { - self.emit_audit_before(request).await?; - } - - self.inject_secrets(request, resolved_secret_injections) - .await?; - self.inject_credential_accounts(request).await?; - - if let Some(policy) = network_policy { - let Some(store) = &self.network_policies else { - return Err(network_obligation_failed()); - }; - store.insert( - &request.context.resource_scope, - request.capability_id, - policy, - ); - } - - Ok(()) - } - - async fn emit_audit_after( - &self, - request: &CapabilityObligationCompletionRequest<'_>, - output_bytes: u64, - ) -> Result<(), CapabilityObligationError> { - let Some(audit_sink) = &self.audit_sink else { - return Err(CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Audit, - }); - }; - - audit_sink - .emit_audit(audit_after_record(request, output_bytes)) - .await - .map_err(|_| CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Audit, - }) - } -} - -#[async_trait] -impl CapabilityObligationHandler for BuiltinObligationHandler { - async fn satisfy( - &self, - request: CapabilityObligationRequest<'_>, - ) -> Result<(), CapabilityObligationError> { - // `satisfy` is the direct one-shot path for callers that need staged - // network/secret handoff but do not need to pass prepared mounts or a - // reservation downstream. Resource reservations are released without - // discarding staged handoffs because successful callers still need the - // network/secret material handed to runtime adapters. CapabilityHost - // uses `prepare`/`complete`/`abort` directly instead. Post-dispatch - // obligations fail closed here because this path has no dispatch result - // to redact, limit, or audit. - let post_dispatch = post_dispatch_obligations(request.obligations); - if !post_dispatch.is_empty() { - return Err(CapabilityObligationError::Unsupported { - obligations: post_dispatch, - }); - } - let outcome = self - .prepare(CapabilityObligationRequest { - phase: request.phase, - context: request.context, - capability_id: request.capability_id, - estimate: request.estimate, - obligations: request.obligations, - }) - .await?; - if let Some(reservation) = &outcome.resource_reservation - && let Err(error) = self.release_resource_reservation(reservation) - { - if let Err(cleanup_error) = self.discard_staged_handoffs( - &request.context.resource_scope, - request.capability_id, - request.obligations, - ) { - tracing::debug!(error = ?cleanup_error, "best-effort discard of staged handoffs failed"); - } - return Err(error); - } - Ok(()) - } - - async fn prepare( - &self, - request: CapabilityObligationRequest<'_>, - ) -> Result { - let unsupported = unsupported_obligations(request.phase, request.obligations); - if !unsupported.is_empty() { - return Err(CapabilityObligationError::Unsupported { - obligations: unsupported, - }); - } - - let network_policy = network_policy_obligation(request.obligations)?; - if network_policy.is_some() && self.network_policies.is_none() { - return Err(network_obligation_failed()); - } - let scoped_mounts = scoped_mount_obligation(request.context, request.obligations)?; - let secret_handles = secret_injection_handles(request.obligations); - let resolved_secret_injections = self - .preflight_secret_injection(&request, &secret_handles) - .await?; - self.preflight_resource_ceiling(&request)?; - let resource_reservation = self.reserve_resource_obligation(&request)?; - let outcome = CapabilityObligationOutcome { - mounts: scoped_mounts, - resource_reservation, - }; - - if let Err(error) = self - .finish_prepare(&request, &resolved_secret_injections, network_policy) - .await - { - self.abort(CapabilityObligationAbortRequest { - phase: request.phase, - context: request.context, - capability_id: request.capability_id, - estimate: request.estimate, - obligations: request.obligations, - outcome: &outcome, - }) - .await?; - return Err(error); - } - - Ok(outcome) - } - - async fn abort( - &self, - request: CapabilityObligationAbortRequest<'_>, - ) -> Result<(), CapabilityObligationError> { - self.discard_staged_handoffs( - &request.context.resource_scope, - request.capability_id, - request.obligations, - )?; - - if let Some(reservation) = &request.outcome.resource_reservation { - self.release_resource_reservation(reservation)?; - } - Ok(()) - } - - async fn complete_dispatch( - &self, - request: CapabilityObligationCompletionRequest<'_>, - ) -> Result { - let unsupported = unsupported_completion_obligations(request.phase, request.obligations); - if !unsupported.is_empty() { - return Err(CapabilityObligationError::Unsupported { - obligations: unsupported, - }); - } - - let mut dispatch = request.dispatch.clone(); - // Turn any base64 document payload into extracted text before redaction - // and the output-size obligations run, so the model gets bounded text - // (leak-scanned, size-checked) and the large base64 never survives. - dispatch.output = crate::document_output::extract_documents_in_output( - dispatch.capability_id.as_str(), - dispatch.output, - ); - if request - .obligations - .iter() - .any(|obligation| matches!(obligation, Obligation::RedactOutput)) - { - dispatch.output = match redact_output(dispatch.output) { - Ok(value) => value, - Err(error) => { - // Leak-detector blocked: record the boundary decision - // before propagating. The event is payload-free by - // construction — only the boundary, decision, and a - // stable code reach the sink. The original output never - // leaves the type system. - if let Some(sink) = &self.security_audit_sink { - let event = SecurityAuditEvent::new( - SecurityBoundary::LeakDetector, - SecurityDecision::Blocked, - LEAK_REDACT_FAILED_CODE, - ) - .with_capability_id(request.capability_id.clone()) - .with_scope(request.context.resource_scope.clone()); - sink.record(event); - } - return Err(error); - } - }; - dispatch.display_preview = None; - } - - let output_bytes = dispatch_output_bytes(&dispatch.output)?; - for obligation in request.obligations { - if let Obligation::EnforceResourceCeiling { ceiling } = obligation { - validate_supported_resource_ceiling(ceiling)?; - validate_usage_within_ceiling(&dispatch.usage, output_bytes, ceiling)?; - } - } - for obligation in request.obligations { - if let Obligation::EnforceOutputLimit { bytes } = obligation - && output_bytes > *bytes - { - return Err(output_obligation_failed()); - } - } - - self.discard_staged_handoffs( - &request.context.resource_scope, - request.capability_id, - request.obligations, - )?; - - if request - .obligations - .iter() - .any(|obligation| matches!(obligation, Obligation::AuditAfter)) - { - self.emit_audit_after(&request, output_bytes).await?; - } - - Ok(dispatch) - } -} - -impl BuiltinObligationHandler { - fn release_resource_reservation( - &self, - reservation: &ResourceReservation, - ) -> Result<(), CapabilityObligationError> { - let Some(governor) = &self.resource_governor else { - return Err(resource_obligation_failed()); - }; - governor - .release(reservation.id) - .map(|_| ()) - .map_err(|_| resource_obligation_failed()) - } - - fn discard_staged_handoffs( - &self, - scope: &ResourceScope, - capability_id: &CapabilityId, - obligations: &[Obligation], - ) -> Result<(), CapabilityObligationError> { - if obligations - .iter() - .any(|obligation| matches!(obligation, Obligation::ApplyNetworkPolicy { .. })) - && let Some(store) = &self.network_policies - { - let _ = store.take(scope, capability_id); - } - - if let Some(store) = &self.secret_injections { - for handle in staged_secret_injection_handles(obligations) { - let _ = store - .take(scope, capability_id, &handle) - .map_err(|_| secret_obligation_failed())?; - } - } - - Ok(()) - } -} - -fn post_dispatch_obligations(obligations: &[Obligation]) -> Vec { - obligations - .iter() - .filter(|obligation| { - matches!( - obligation, - Obligation::AuditAfter - | Obligation::RedactOutput - | Obligation::EnforceResourceCeiling { .. } - | Obligation::EnforceOutputLimit { .. } - ) - }) - .cloned() - .collect() -} - -fn unsupported_obligations( - phase: CapabilityObligationPhase, - obligations: &[Obligation], -) -> Vec { - obligations - .iter() - .filter(|obligation| !obligation_supported_before_dispatch(phase, obligation)) - .cloned() - .collect() -} - -fn obligation_supported_before_dispatch( - phase: CapabilityObligationPhase, - obligation: &Obligation, -) -> bool { - match obligation { - Obligation::AuditBefore - | Obligation::ApplyNetworkPolicy { .. } - | Obligation::FirstPartyCredentialStagedViaHostPort { .. } - | Obligation::InjectCredentialAccountOnce { .. } - | Obligation::InjectSecretOnce { .. } - | Obligation::ReserveResources { .. } - | Obligation::UseScopedMounts { .. } => true, - Obligation::EnforceResourceCeiling { .. } => { - !matches!(phase, CapabilityObligationPhase::Spawn) - } - Obligation::AuditAfter - | Obligation::RedactOutput - | Obligation::EnforceOutputLimit { .. } => { - !matches!(phase, CapabilityObligationPhase::Spawn) - } - } -} - -fn unsupported_completion_obligations( - phase: CapabilityObligationPhase, - obligations: &[Obligation], -) -> Vec { - obligations - .iter() - .filter(|obligation| !obligation_supported_after_dispatch(phase, obligation)) - .cloned() - .collect() -} - -fn obligation_supported_after_dispatch( - phase: CapabilityObligationPhase, - obligation: &Obligation, -) -> bool { - match obligation { - Obligation::AuditBefore - | Obligation::ApplyNetworkPolicy { .. } - | Obligation::FirstPartyCredentialStagedViaHostPort { .. } - | Obligation::InjectCredentialAccountOnce { .. } - | Obligation::InjectSecretOnce { .. } - | Obligation::ReserveResources { .. } - | Obligation::UseScopedMounts { .. } => true, - Obligation::EnforceResourceCeiling { .. } => { - !matches!(phase, CapabilityObligationPhase::Spawn) - } - Obligation::AuditAfter - | Obligation::RedactOutput - | Obligation::EnforceOutputLimit { .. } => { - !matches!(phase, CapabilityObligationPhase::Spawn) - } - } -} - -fn secret_injection_handles(obligations: &[Obligation]) -> Vec { - obligations - .iter() - .filter_map(|obligation| match obligation { - Obligation::InjectSecretOnce { handle } => Some(handle.clone()), - _ => None, - }) - .collect() -} - -struct CredentialAccountInjectionObligation<'a> { - handle: &'a SecretHandle, - provider: &'a VendorId, - setup: &'a RuntimeCredentialAccountSetup, - provider_scopes: &'a [String], - requester_extension: &'a ExtensionId, -} - -fn credential_account_injection_obligations( - obligations: &[Obligation], -) -> Vec> { - obligations - .iter() - .filter_map(|obligation| match obligation { - Obligation::InjectCredentialAccountOnce { - handle, - provider, - setup, - provider_scopes, - requester_extension, - } => Some(CredentialAccountInjectionObligation { - handle, - provider, - setup, - provider_scopes, - requester_extension, - }), - _ => None, - }) - .collect() -} - -fn staged_secret_injection_handles(obligations: &[Obligation]) -> Vec { - obligations - .iter() - .filter_map(|obligation| match obligation { - Obligation::InjectSecretOnce { handle } - | Obligation::InjectCredentialAccountOnce { handle, .. } => Some(handle.clone()), - _ => None, - }) - .collect() -} - -/// Map the canonical staged-credential error to the obligation-handler error type. -/// -/// Used by both [`inject_credential_accounts`] (resolver-side errors) and -/// [`stage_credential_material`] (storage-side errors) so the WASM -/// `InjectCredentialAccountOnce` path and the first-party stager path share -/// the same AuthRequired/Backend semantics. -fn credential_stage_error_to_obligation_error( - error: CredentialStageError, - credential_obligation: Option<&CredentialAccountInjectionObligation<'_>>, -) -> CapabilityObligationError { - match error { - CredentialStageError::AuthRequired => CapabilityObligationError::AuthRequired { - credential_requirements: credential_obligation - .map(|obligation| { - vec![RuntimeCredentialAuthRequirement { - provider: obligation.provider.clone(), - setup: obligation.setup.clone(), - requester_extension: obligation.requester_extension.clone(), - provider_scopes: obligation.provider_scopes.to_vec(), - }] - }) - .unwrap_or_default(), - }, - CredentialStageError::Backend => secret_obligation_failed(), - } -} - -/// Retrieve `source` from the secret store and stage the material under `target` -/// in the injection store for the given capability invocation. -/// -/// Used when the secret store key (`source`) differs from the runtime injection slot -/// (`target`) — for example, when a product-auth account's backing secret is resolved -/// to a concrete handle before being injected under the WASM guest's declared slot name. -/// Lease → consume → insert the staged credential material. -/// -/// Mirrors [`crate::services::ProductAuthProviderRuntimePorts::stage_secret_once`] -/// so the WASM `InjectCredentialAccountOnce` path and the first-party stager path -/// (e.g. `ProductAuthRuntimeGsuiteCredentialStager`) share identical lease/consume -/// semantics and `CredentialStageError` mapping. `SecretStoreError` variants for -/// unknown/expired/revoked/consumed material map to -/// [`CredentialStageError::AuthRequired`] via [`crate::services::stage_secret_error`]; -/// other failures map to [`CredentialStageError::Backend`]. -async fn stage_credential_material( - secret_store: &dyn SecretStorePort, - secret_injections: &RuntimeSecretInjectionStore, - source_scope: &ResourceScope, - target_scope: &ResourceScope, - capability_id: &CapabilityId, - source: &SecretHandle, - target: &SecretHandle, -) -> Result<(), CredentialStageError> { - let lease = secret_store - .lease_once(source_scope, source) - .await - .map_err(|e| { - tracing::debug!(err = %e, "stage_credential_material: lease_once failed"); - crate::services::stage_secret_error(e) - })?; - let secret = secret_store - .consume(source_scope, lease.id) - .await - .map_err(|e| { - tracing::debug!(err = %e, "stage_credential_material: consume failed"); - crate::services::stage_secret_error(e) - })?; - secret_injections - .insert(target_scope, capability_id, target, secret) - .map_err(|e| { - tracing::debug!(err = %e, "stage_credential_material: insert failed"); - CredentialStageError::Backend - }) -} - -fn network_policy_obligation( - obligations: &[Obligation], -) -> Result, CapabilityObligationError> { - let mut policy = None; - for obligation in obligations { - if let Obligation::ApplyNetworkPolicy { policy: next } = obligation { - if policy.is_some() { - return Err(network_obligation_failed()); - } - validate_network_policy_metadata(next)?; - policy = Some(next.clone()); - } - } - Ok(policy) -} - -fn scoped_mount_obligation( - context: &ironclaw_host_api::scope::ExecutionContext, - obligations: &[Obligation], -) -> Result, CapabilityObligationError> { - let mut mounts = None; - for obligation in obligations { - if let Obligation::UseScopedMounts { mounts: next } = obligation { - if mounts.is_some() { - return Err(mount_obligation_failed()); - } - next.validate().map_err(|_| mount_obligation_failed())?; - if !next.is_subset_of(&context.mounts) { - return Err(mount_obligation_failed()); - } - mounts = Some(next.clone()); - } - } - Ok(mounts) -} - -fn resource_ceiling_obligation( - obligations: &[Obligation], -) -> Result, CapabilityObligationError> { - let mut ceiling = None; - for obligation in obligations { - if let Obligation::EnforceResourceCeiling { ceiling: next } = obligation { - if ceiling.is_some() { - return Err(resource_obligation_failed()); - } - ceiling = Some(next); - } - } - Ok(ceiling) -} - -fn validate_supported_resource_ceiling( - ceiling: &ResourceCeiling, -) -> Result<(), CapabilityObligationError> { - if ceiling.max_wall_clock_ms.is_some() { - return Err(resource_obligation_failed()); - } - if let Some(sandbox) = &ceiling.sandbox { - validate_supported_sandbox_quota(sandbox)?; - } - Ok(()) -} - -fn validate_supported_sandbox_quota( - sandbox: &SandboxQuota, -) -> Result<(), CapabilityObligationError> { - if sandbox.cpu_time_ms.is_some() - || sandbox.memory_bytes.is_some() - || sandbox.disk_bytes.is_some() - || sandbox.network_egress_bytes.is_some() - || sandbox.process_count.is_some() - { - return Err(resource_obligation_failed()); - } - Ok(()) -} - -fn validate_estimate_within_ceiling( - estimate: &ResourceEstimate, - ceiling: &ResourceCeiling, -) -> Result<(), CapabilityObligationError> { - check_optional_decimal_ceiling(estimate.usd, ceiling.max_usd)?; - check_required_integer_ceiling(estimate.input_tokens, ceiling.max_input_tokens)?; - check_required_integer_ceiling(estimate.output_tokens, ceiling.max_output_tokens)?; - Ok(()) -} - -fn validate_usage_within_ceiling( - usage: &ResourceUsage, - output_bytes: u64, - ceiling: &ResourceCeiling, -) -> Result<(), CapabilityObligationError> { - check_decimal_ceiling(usage.usd, ceiling.max_usd)?; - check_integer_ceiling(usage.input_tokens, ceiling.max_input_tokens)?; - check_integer_ceiling(usage.output_tokens, ceiling.max_output_tokens)?; - check_output_bytes_ceiling(output_bytes, ceiling.max_output_bytes)?; - Ok(()) -} - -fn check_output_bytes_ceiling( - actual: u64, - ceiling: Option, -) -> Result<(), CapabilityObligationError> { - if let Some(ceiling) = ceiling - && actual > ceiling - { - return Err(output_obligation_failed()); - } - Ok(()) -} - -fn check_optional_decimal_ceiling( - actual: Option, - ceiling: Option, -) -> Result<(), CapabilityObligationError> { - let Some(ceiling) = ceiling else { - return Ok(()); - }; - let Some(actual) = actual else { - return Err(resource_obligation_failed()); - }; - check_decimal_ceiling(actual, Some(ceiling)) -} - -fn check_decimal_ceiling( - actual: rust_decimal::Decimal, - ceiling: Option, -) -> Result<(), CapabilityObligationError> { - if let Some(ceiling) = ceiling - && actual > ceiling - { - return Err(resource_obligation_failed()); - } - Ok(()) -} - -fn check_required_integer_ceiling( - actual: Option, - ceiling: Option, -) -> Result<(), CapabilityObligationError> { - let Some(ceiling) = ceiling else { - return Ok(()); - }; - let Some(actual) = actual else { - return Err(resource_obligation_failed()); - }; - check_integer_ceiling(actual, Some(ceiling)) -} - -fn check_integer_ceiling( - actual: u64, - ceiling: Option, -) -> Result<(), CapabilityObligationError> { - if let Some(ceiling) = ceiling - && actual > ceiling - { - return Err(resource_obligation_failed()); - } - Ok(()) -} - -fn validate_network_policy_metadata( - policy: &NetworkPolicy, -) -> Result<(), CapabilityObligationError> { - if policy.allowed_targets.is_empty() { - return Err(network_obligation_failed()); - } - Ok(()) -} - -fn network_obligation_failed() -> CapabilityObligationError { - CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Network, - } -} - -fn secret_obligation_failed() -> CapabilityObligationError { - CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Secret, - } -} - -/// Single source of truth for "is this required secret present in `scope`". -/// -/// Both the credential pre-flight (ordering — `DefaultHostRuntime:: -/// credential_preflight_check`) and the dispatch-time obligation backstop -/// (enforcement — [`BuiltinObligationHandler::preflight_secret_injection`]) -/// consult this one rule so "what counts as a present credential" cannot drift -/// between the two call sites. Each caller decides how to treat a store `Err` -/// (the pre-flight fails open and skips; the obligation backstop fails closed). -pub(crate) async fn secret_present( - store: &dyn SecretStorePort, - scope: &ResourceScope, - handle: &SecretHandle, -) -> Result { - Ok(store.metadata(scope, handle).await?.is_some()) -} - -/// Resolve which scope owns `handle` for this caller, honoring tenant-shared, -/// admin-managed credentials (#5459). A caller's OWN secret wins; otherwise the -/// tenant-shared admin-managed scope ([`ResourceScope::tenant_shared_managed_scope`]), -/// so one admin-set key satisfies every user of the tenant. `Ok(None)` means the -/// secret is absent in both scopes. -/// -/// Single source of truth for BOTH "is this required secret present" (callers map -/// to `.is_some()`) and "where does the lease read from" — the pre-flight ordering -/// probe (`credential_preflight_check`) and the dispatch-time backstop -/// (`preflight_secret_injection`) consult this rule, then the injection lease -/// (`inject_secrets`) consumes the resolved source scope. Each caller decides how -/// to treat a store `Err` (the pre-flight fails open and skips; the obligation -/// backstop and lease fail closed). -pub(crate) async fn secret_owner_scope( - store: &dyn SecretStorePort, - caller_scope: &ResourceScope, - handle: &SecretHandle, -) -> Result, SecretStoreError> { - if secret_present(store, caller_scope, handle).await? { - return Ok(Some(caller_scope.clone())); - } - let shared = caller_scope.tenant_shared_managed_scope(); - if secret_present(store, &shared, handle).await? { - return Ok(Some(shared)); - } - Ok(None) -} - -fn resource_obligation_failed() -> CapabilityObligationError { - CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Resource, - } -} - -fn mount_obligation_failed() -> CapabilityObligationError { - CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Mount, - } -} - -fn output_obligation_failed() -> CapabilityObligationError { - CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Output, - } -} - -fn dispatch_output_bytes(output: &serde_json::Value) -> Result { - serde_json::to_vec(output) - .map(|bytes| bytes.len() as u64) - .map_err(|_| output_obligation_failed()) -} - -/// Security-audit reason code emitted when [`redact_output`] rejects output -/// because the leak detector matched. Stable grep target for SRE pattern -/// matching across durable security-audit logs. -pub const LEAK_REDACT_FAILED_CODE: &str = "leak_redact_failed"; - -fn redact_output( - output: serde_json::Value, -) -> Result { - match output { - serde_json::Value::String(value) => { - redact_output_string(value).map(serde_json::Value::String) - } - serde_json::Value::Array(values) => values - .into_iter() - .map(redact_output) - .collect::, _>>() - .map(serde_json::Value::Array), - serde_json::Value::Object(entries) => { - let mut redacted = serde_json::Map::with_capacity(entries.len()); - for (key, value) in entries { - let key = redact_output_string(key)?; - let value = redact_output(value)?; - if redacted.insert(key, value).is_some() { - return Err(output_obligation_failed()); - } - } - Ok(serde_json::Value::Object(redacted)) - } - value => Ok(value), - } -} - -fn redact_output_string(value: String) -> Result { - LeakDetector::new() - .scan_and_clean(&value) - .map_err(|_| output_obligation_failed()) -} - -fn audit_before_record(request: &CapabilityObligationRequest<'_>) -> AuditEnvelope { - AuditEnvelope { - event_id: AuditEventId::new(), - correlation_id: request.context.correlation_id, - stage: AuditStage::Before, - timestamp: Utc::now(), - tenant_id: request.context.tenant_id.clone(), - user_id: request.context.user_id.clone(), - agent_id: request.context.agent_id.clone(), - project_id: request.context.project_id.clone(), - mission_id: request.context.mission_id.clone(), - thread_id: request.context.thread_id.clone(), - invocation_id: request.context.invocation_id, - process_id: request.context.process_id, - approval_request_id: None, - extension_id: Some(request.context.extension_id.clone()), - action: ActionSummary { - kind: capability_action_kind(request.phase).to_string(), - target: Some(request.capability_id.as_str().to_string()), - effects: capability_action_effects(request.phase), - }, - decision: DecisionSummary { - kind: "obligation_satisfied".to_string(), - reason: None, - actor: None, - }, - result: Some(ActionResultSummary { - success: true, - status: Some(obligation_status(request.obligations)), - output_bytes: None, - }), - } -} - -fn audit_after_record( - request: &CapabilityObligationCompletionRequest<'_>, - output_bytes: u64, -) -> AuditEnvelope { - AuditEnvelope { - event_id: AuditEventId::new(), - correlation_id: request.context.correlation_id, - stage: AuditStage::After, - timestamp: Utc::now(), - tenant_id: request.context.tenant_id.clone(), - user_id: request.context.user_id.clone(), - agent_id: request.context.agent_id.clone(), - project_id: request.context.project_id.clone(), - mission_id: request.context.mission_id.clone(), - thread_id: request.context.thread_id.clone(), - invocation_id: request.context.invocation_id, - process_id: request.context.process_id, - approval_request_id: None, - extension_id: Some(request.context.extension_id.clone()), - action: ActionSummary { - kind: capability_action_kind(request.phase).to_string(), - target: Some(request.capability_id.as_str().to_string()), - effects: capability_action_effects(request.phase), - }, - decision: DecisionSummary { - kind: "obligation_satisfied".to_string(), - reason: None, - actor: None, - }, - result: Some(ActionResultSummary { - success: true, - status: Some(obligation_status(request.obligations)), - output_bytes: Some(output_bytes), - }), - } -} - -fn capability_action_kind(phase: CapabilityObligationPhase) -> &'static str { - match phase { - CapabilityObligationPhase::Invoke => "capability_invoke", - CapabilityObligationPhase::Resume => "capability_resume", - CapabilityObligationPhase::Spawn => "capability_spawn", - } -} - -fn capability_action_effects(phase: CapabilityObligationPhase) -> Vec { - match phase { - CapabilityObligationPhase::Invoke | CapabilityObligationPhase::Resume => { - vec![EffectKind::DispatchCapability] - } - CapabilityObligationPhase::Spawn => { - vec![EffectKind::DispatchCapability, EffectKind::SpawnProcess] - } - } -} - -fn obligation_status(obligations: &[Obligation]) -> String { - obligations - .iter() - .filter_map(obligation_label) - .collect::>() - .join(",") -} - -fn obligation_label(obligation: &Obligation) -> Option<&'static str> { - match obligation { - Obligation::AuditBefore => Some("audit_before"), - Obligation::AuditAfter => Some("audit_after"), - Obligation::RedactOutput => Some("redact_output"), - Obligation::ApplyNetworkPolicy { .. } => Some("apply_network_policy"), - Obligation::InjectSecretOnce { .. } => Some("inject_secret_once"), - Obligation::InjectCredentialAccountOnce { .. } => Some("inject_credential_account_once"), - Obligation::FirstPartyCredentialStagedViaHostPort { .. } => { - Some("first_party_credential_staged_via_host_port") - } - Obligation::EnforceOutputLimit { .. } => Some("enforce_output_limit"), - Obligation::ReserveResources { .. } => Some("reserve_resources"), - Obligation::UseScopedMounts { .. } => Some("use_scoped_mounts"), - Obligation::EnforceResourceCeiling { .. } => Some("enforce_resource_ceiling"), - } -} - -/// **Finding H2 — compile-time regression guard.** -/// -/// The original H2 claim was that `RuntimeSecretInjectionStore`'s -/// `HashMap<_, RuntimeSecretInjectionEntry>` would bitwise-copy plaintext out -/// of the old bucket array on rehash and free it without zeroization. On -/// closer inspection that does *not* happen, because `SecretMaterial = -/// secrecy::SecretBox`: the rehash moves a `Box` pointer plus the -/// `Instant`, while the actual buffer stays at its original heap address -/// until `SecretBox::drop` zeroizes it. -/// -/// The protection is real but depends on the staged entry's `material` field -/// being a `ZeroizeOnDrop` carrier. If it ever swaps to a non-zeroizing type -/// (plain `String`, `Vec`, etc.), the bitwise-copy concern returns. This -/// `const _: fn(...) = ...` references the field through a -/// `ZeroizeOnDrop`-bounded helper, so the swap is rejected at compile time -/// rather than only failing a test run. The function is never called — only -/// type-checked. -const _: fn(&RuntimeSecretInjectionEntry) = |entry| { - fn require_zeroize_on_drop(_: &T) {} - require_zeroize_on_drop(&entry.material); -}; - -#[cfg(test)] -mod tests { - use std::{sync::Arc, time::Duration}; - - use ironclaw_events::InMemoryAuditSink; - use ironclaw_host_api::{ - action::{NetworkScheme, NetworkTargetPattern}, - capability::CapabilitySet, - dispatch::CapabilityDisplayOutputPreview, - ids::{ - AgentId, CorrelationId, ExtensionId, InvocationId, ProjectId, ResourceReservationId, - TenantId, UserId, - }, - runtime::{RuntimeKind, TrustClass}, - scope::ExecutionContext, - }; - use ironclaw_resources::{InMemoryResourceGovernor, ResourceAccount}; - use ironclaw_secrets::SecretStore; - - use super::*; - - #[tokio::test] - async fn runtime_secret_injection_store_prunes_expired_handoffs() { - let store = RuntimeSecretInjectionStore::with_ttl(Duration::from_millis(5)); - let scope = resource_scope_with_agent("agent-a"); - let capability_id = capability_id(); - let handle = SecretHandle::new("api_token").unwrap(); - - store - .insert( - &scope, - &capability_id, - &handle, - SecretMaterial::from("runtime-secret"), - ) - .unwrap(); - tokio::time::sleep(Duration::from_millis(20)).await; - - assert_eq!(store.prune_expired().unwrap(), 1); - assert!( - store - .take(&scope, &capability_id, &handle) - .unwrap() - .is_none() - ); - } - - #[test] - fn network_obligation_policy_store_isolates_agent_scope() { - let store = NetworkObligationPolicyStore::new(); - let (agent_a, agent_b) = same_invocation_agent_scopes(); - let capability_id = capability_id(); - - store.insert(&agent_a, &capability_id, allowed_network_policy()); - - assert!(store.take(&agent_b, &capability_id).is_none()); - assert!(store.take(&agent_a, &capability_id).is_some()); - } - - #[test] - fn runtime_secret_injection_store_isolates_agent_scope() { - let store = RuntimeSecretInjectionStore::new(); - let (agent_a, agent_b) = same_invocation_agent_scopes(); - let capability_id = capability_id(); - let handle = SecretHandle::new("api_token").unwrap(); - - store - .insert( - &agent_a, - &capability_id, - &handle, - SecretMaterial::from("runtime-secret"), - ) - .unwrap(); - - assert!( - store - .take(&agent_b, &capability_id, &handle) - .unwrap() - .is_none() - ); - assert!( - store - .take(&agent_a, &capability_id, &handle) - .unwrap() - .is_some() - ); - } - - #[tokio::test] - async fn builtin_obligation_handler_satisfy_release_preserves_staged_handoffs() { - let network_policies = Arc::new(NetworkObligationPolicyStore::new()); - let secret_injections = Arc::new(RuntimeSecretInjectionStore::new()); - let secret_store = Arc::new(SecretStore::ephemeral()); - let governor = Arc::new(InMemoryResourceGovernor::new()); - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - network_policies.clone(), - secret_store.clone(), - secret_injections.clone(), - governor.clone(), - ); - let handler = services.obligation_handler(); - let context = execution_context(); - let account = ResourceAccount::tenant(context.resource_scope.tenant_id.clone()); - let capability_id = capability_id(); - let handle = SecretHandle::new("api_token").unwrap(); - let estimate = ResourceEstimate::default().set_concurrency_slots(1); - secret_store - .put( - context.resource_scope.clone(), - handle.clone(), - SecretMaterial::from("runtime-secret"), - None, - ) - .await - .unwrap(); - let obligations = vec![ - Obligation::ApplyNetworkPolicy { - policy: allowed_network_policy(), - }, - Obligation::InjectSecretOnce { - handle: handle.clone(), - }, - Obligation::ReserveResources { - reservation_id: ResourceReservationId::new(), - }, - ]; - - handler - .satisfy(CapabilityObligationRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - }) - .await - .unwrap(); - - assert_eq!(governor.reserved_for(&account).concurrency_slots, 0); - assert!( - network_policies - .take(&context.resource_scope, &capability_id) - .is_some() - ); - assert!( - secret_injections - .take(&context.resource_scope, &capability_id, &handle) - .unwrap() - .is_some() - ); - } - - // #5459 tenant-shared credential resolution: a caller's own secret wins; - // otherwise the tenant-shared admin-managed scope; otherwise absent. - #[tokio::test] - async fn secret_owner_scope_prefers_caller_then_tenant_shared_then_none() { - let handle = SecretHandle::new("market_data_api_key").unwrap(); - let caller = execution_context().resource_scope; - let shared = caller.tenant_shared_managed_scope(); - - // Absent in both scopes -> None (dispatch then gates with AuthRequired). - let store = SecretStore::ephemeral(); - assert_eq!( - secret_owner_scope(&store, &caller, &handle).await.unwrap(), - None, - ); - - // Present ONLY at the tenant-shared admin-managed scope -> resolves there, - // so one admin-set key satisfies a caller who never provisioned it. - let store = SecretStore::ephemeral(); - store - .put( - shared.clone(), - handle.clone(), - SecretMaterial::from("shared-admin-key"), - None, - ) - .await - .unwrap(); - assert_eq!( - secret_owner_scope(&store, &caller, &handle) - .await - .unwrap() - .as_ref(), - Some(&shared), - ); - - // Present at BOTH scopes -> the caller's OWN secret wins over the shared one. - let store = SecretStore::ephemeral(); - store - .put( - caller.clone(), - handle.clone(), - SecretMaterial::from("caller-own-key"), - None, - ) - .await - .unwrap(); - store - .put( - shared.clone(), - handle.clone(), - SecretMaterial::from("shared-admin-key"), - None, - ) - .await - .unwrap(); - assert_eq!( - secret_owner_scope(&store, &caller, &handle) - .await - .unwrap() - .as_ref(), - Some(&caller), - ); - } - - // Through the caller: InjectSecretOnce is satisfied by an admin-set - // tenant-shared key even when the caller has no personal secret, and the - // material is staged at the caller's own invocation slot (#5459). - #[tokio::test] - async fn inject_secret_once_falls_back_to_tenant_shared_admin_key() { - let secret_store = Arc::new(SecretStore::ephemeral()); - let secret_injections = Arc::new(RuntimeSecretInjectionStore::new()); - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - Arc::new(NetworkObligationPolicyStore::new()), - secret_store.clone(), - secret_injections.clone(), - Arc::new(InMemoryResourceGovernor::new()), - ); - let handler = services.obligation_handler(); - let context = execution_context(); - let capability_id = capability_id(); - let handle = SecretHandle::new("market_data_api_key").unwrap(); - let estimate = ResourceEstimate::default(); - - // Admin set the key ONLY at the tenant-shared scope; the caller has none. - secret_store - .put( - context.resource_scope.tenant_shared_managed_scope(), - handle.clone(), - SecretMaterial::from("shared-admin-key"), - None, - ) - .await - .unwrap(); - - let obligations = vec![Obligation::InjectSecretOnce { - handle: handle.clone(), - }]; - handler - .satisfy(CapabilityObligationRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - }) - .await - .expect( - "tenant-shared key satisfies InjectSecretOnce for a caller with no personal secret", - ); - - assert!( - secret_injections - .take(&context.resource_scope, &capability_id, &handle) - .unwrap() - .is_some(), - "shared-sourced secret must be staged at the caller's own invocation slot", - ); - } - - #[tokio::test] - async fn redact_output_clears_display_preview_side_channel() { - use ironclaw_host_api::{ - resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, - runtime::RuntimeKind, - }; - - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - Arc::new(NetworkObligationPolicyStore::new()), - Arc::new(SecretStore::ephemeral()), - Arc::new(RuntimeSecretInjectionStore::new()), - Arc::new(InMemoryResourceGovernor::new()), - ); - let handler = services.obligation_handler(); - let context = execution_context(); - let capability_id = capability_id(); - let estimate = ResourceEstimate::default(); - let obligations = vec![Obligation::RedactOutput]; - let dispatch = CapabilityDispatchResult { - capability_id: capability_id.clone(), - provider: context.extension_id.clone(), - runtime: RuntimeKind::Wasm, - output: serde_json::json!({"secret": "sk-secret", "safe": "ok"}), - display_preview: Some(CapabilityDisplayOutputPreview { - output_summary: Some("contains secret".to_string()), - output_preview: "sk-secret".to_string(), - output_kind: "text".to_string(), - subtitle: None, - truncated: false, - }), - usage: ResourceUsage::default(), - receipt: ResourceReceipt { - id: ResourceReservationId::new(), - scope: context.resource_scope.clone(), - status: ReservationStatus::Released, - estimate: ResourceEstimate::default(), - actual: None, - }, - }; - - let completed = handler - .complete_dispatch(CapabilityObligationCompletionRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - dispatch: &dispatch, - }) - .await - .expect("redacted dispatch completes"); - - assert!(completed.display_preview.is_none()); - assert_eq!(completed.output["safe"], serde_json::json!("ok")); - } - - #[tokio::test] - async fn complete_dispatch_extracts_base64_document_into_text() { - use base64::Engine as _; - use ironclaw_host_api::{ - resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, - runtime::RuntimeKind, - }; - - // Drive the *caller* (`complete_dispatch`), not the helper: a dispatch - // result carrying `content_base64` + `mime_type` must come back with the - // extracted text in `content` and no base64 left for the model to see. - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - Arc::new(NetworkObligationPolicyStore::new()), - Arc::new(SecretStore::ephemeral()), - Arc::new(RuntimeSecretInjectionStore::new()), - Arc::new(InMemoryResourceGovernor::new()), - ); - let handler = services.obligation_handler(); - let context = execution_context(); - // The document-extraction transform is capability-gated, so this test - // must dispatch a capability that opts in (`google-drive.download_file`) - // — the shared `echo.say` helper id would pass through untouched. - let capability_id = CapabilityId::new("google-drive.download_file").unwrap(); - let estimate = ResourceEstimate::default(); - let obligations = vec![Obligation::RedactOutput]; - let encoded = base64::engine::general_purpose::STANDARD.encode(b"name,age\nAlice,30"); - let dispatch = CapabilityDispatchResult { - capability_id: capability_id.clone(), - provider: context.extension_id.clone(), - runtime: RuntimeKind::Wasm, - output: serde_json::json!({ - "file_id": "f1", - "name": "data.csv", - "mime_type": "text/csv", - "content_base64": encoded, - }), - display_preview: None, - usage: ResourceUsage::default(), - receipt: ResourceReceipt { - id: ResourceReservationId::new(), - scope: context.resource_scope.clone(), - status: ReservationStatus::Released, - estimate: ResourceEstimate::default(), - actual: None, - }, - }; - - let completed = handler - .complete_dispatch(CapabilityObligationCompletionRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - dispatch: &dispatch, - }) - .await - .expect("base64 document dispatch completes"); - - assert_eq!( - completed.output["content"], - serde_json::json!("name,age\nAlice,30") - ); - assert!( - completed.output.get("content_base64").is_none(), - "base64 must be stripped before the result reaches the model" - ); - } - - #[tokio::test] - async fn leak_detector_block_records_security_audit_event_through_complete_dispatch() { - use ironclaw_events::{ - InMemorySecurityAuditSink, SecurityAuditSink, SecurityBoundary, SecurityDecision, - }; - use ironclaw_host_api::{ - resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, - runtime::RuntimeKind, - }; - - // Build a handler with both an audit sink (unused here — we hit the - // redact branch, not the AuditAfter branch) and a recording - // security-audit sink. Other backing stores are not exercised by - // the redact-only path, but the handler requires them to be set - // for safety; we install minimal in-memory ones. - let security_sink: Arc = - Arc::new(InMemorySecurityAuditSink::new()); - let security_sink_dyn: Arc = security_sink.clone(); - - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - Arc::new(NetworkObligationPolicyStore::new()), - Arc::new(SecretStore::ephemeral()), - Arc::new(RuntimeSecretInjectionStore::new()), - Arc::new(InMemoryResourceGovernor::new()), - ); - let handler = services - .obligation_handler() - .with_security_audit_sink(security_sink_dyn); - - let context = execution_context(); - let capability_id = capability_id(); - let estimate = ResourceEstimate::default(); - let obligations = vec![Obligation::RedactOutput]; - - // An AWS access-key shaped string is a built-in BLOCK pattern in - // `ironclaw_safety::LeakDetector` (`AKIA[0-9A-Z]{16}`). Per the - // module invariant we drive the *caller* (`complete_dispatch`), - // not the helper, and assert the recorded event: - // - boundary == LeakDetector - // - decision == Blocked - // - code == LEAK_REDACT_FAILED_CODE - // - capability_id + scope are populated - // - no payload (the offending string never appears in the event) - let leaky_payload = - serde_json::Value::String("hello AKIAIOSFODNN7EXAMPLE goodbye".to_string()); - let dispatch = CapabilityDispatchResult { - capability_id: capability_id.clone(), - provider: context.extension_id.clone(), - runtime: RuntimeKind::Wasm, - output: leaky_payload, - display_preview: None, - usage: ResourceUsage::default(), - receipt: ResourceReceipt { - id: ResourceReservationId::new(), - scope: context.resource_scope.clone(), - status: ReservationStatus::Released, - estimate: ResourceEstimate::default(), - actual: None, - }, - }; - - let request = CapabilityObligationCompletionRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - dispatch: &dispatch, - }; - - let result = handler.complete_dispatch(request).await; - assert!( - matches!( - result, - Err(CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Output - }) - ), - "expected output-obligation failure, got {result:?}" - ); - - let events = security_sink.snapshot(); - assert_eq!( - events.len(), - 1, - "exactly one boundary decision should have been recorded, got {events:?}" - ); - let event = &events[0]; - assert_eq!(event.boundary, SecurityBoundary::LeakDetector); - assert_eq!(event.decision, SecurityDecision::Blocked); - assert_eq!(event.code, LEAK_REDACT_FAILED_CODE); - assert_eq!(event.code, "leak_redact_failed"); // stability lock - assert_eq!(event.capability_id.as_ref(), Some(&capability_id)); - assert_eq!(event.scope.as_ref(), Some(&context.resource_scope)); - - // The `SecurityAuditEvent` shape has no free-form payload field. - // That invariant is enforced at the type level by the absence of - // a `String` member on the struct. The check below is therefore a - // documentation-only assertion: it locks the field set at the - // value level for future readers, but the real guard is the type - // shape in `ironclaw_events::security_audit`. - // - // pub struct SecurityAuditEvent { - // pub boundary: SecurityBoundary, - // pub decision: SecurityDecision, - // pub capability_id: Option, - // pub scope: Option, - // pub timestamp: SystemTime, - // pub code: &'static str, - // } - } - - #[tokio::test] - async fn leak_detector_block_without_security_sink_does_not_panic() { - use ironclaw_host_api::{ - resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, - runtime::RuntimeKind, - }; - - let services = BuiltinObligationServices::with_handoff_stores( - Arc::new(InMemoryAuditSink::new()), - Arc::new(NetworkObligationPolicyStore::new()), - Arc::new(SecretStore::ephemeral()), - Arc::new(RuntimeSecretInjectionStore::new()), - Arc::new(InMemoryResourceGovernor::new()), - ); - // No `.with_security_audit_sink(...)` — confirms the sink is - // optional and the original failure semantics are preserved. - let handler = services.obligation_handler(); - - let context = execution_context(); - let capability_id = capability_id(); - let estimate = ResourceEstimate::default(); - let obligations = vec![Obligation::RedactOutput]; - let dispatch = CapabilityDispatchResult { - capability_id: capability_id.clone(), - provider: context.extension_id.clone(), - runtime: RuntimeKind::Wasm, - output: serde_json::Value::String("leak AKIAIOSFODNN7EXAMPLE".to_string()), - display_preview: None, - usage: ResourceUsage::default(), - receipt: ResourceReceipt { - id: ResourceReservationId::new(), - scope: context.resource_scope.clone(), - status: ReservationStatus::Released, - estimate: ResourceEstimate::default(), - actual: None, - }, - }; - - let result = handler - .complete_dispatch(CapabilityObligationCompletionRequest { - phase: CapabilityObligationPhase::Invoke, - context: &context, - capability_id: &capability_id, - estimate: &estimate, - obligations: &obligations, - dispatch: &dispatch, - }) - .await; - assert!(matches!( - result, - Err(CapabilityObligationError::Failed { - kind: CapabilityObligationFailureKind::Output - }) - )); - } - - fn same_invocation_agent_scopes() -> (ResourceScope, ResourceScope) { - let mut agent_a = resource_scope_with_agent("agent-a"); - agent_a.invocation_id = InvocationId::new(); - let mut agent_b = agent_a.clone(); - agent_b.agent_id = Some(AgentId::new("agent-b").unwrap()); - (agent_a, agent_b) - } - - fn resource_scope_with_agent(agent_id: &str) -> ResourceScope { - ResourceScope { - tenant_id: TenantId::new("tenant1").unwrap(), - user_id: UserId::new("user1").unwrap(), - agent_id: Some(AgentId::new(agent_id).unwrap()), - project_id: Some(ProjectId::new("project1").unwrap()), - mission_id: None, - thread_id: None, - invocation_id: InvocationId::new(), - } - } - - fn execution_context() -> ExecutionContext { - let invocation_id = InvocationId::new(); - let resource_scope = ResourceScope { - tenant_id: TenantId::new("tenant1").unwrap(), - user_id: UserId::new("user1").unwrap(), - agent_id: Some(AgentId::new("agent-a").unwrap()), - project_id: Some(ProjectId::new("project1").unwrap()), - mission_id: None, - thread_id: None, - invocation_id, - }; - ExecutionContext { - run_id: None, - origin: None, - invocation_id, - correlation_id: CorrelationId::new(), - process_id: None, - parent_process_id: None, - tenant_id: resource_scope.tenant_id.clone(), - user_id: resource_scope.user_id.clone(), - authenticated_actor_user_id: None, - agent_id: resource_scope.agent_id.clone(), - project_id: resource_scope.project_id.clone(), - mission_id: resource_scope.mission_id.clone(), - thread_id: resource_scope.thread_id.clone(), - extension_id: ExtensionId::new("caller").unwrap(), - runtime: RuntimeKind::Wasm, - trust: TrustClass::Sandbox, - grants: CapabilitySet::default(), - mounts: MountView::default(), - resource_scope, - } - } - - fn capability_id() -> CapabilityId { - CapabilityId::new("echo.say").unwrap() - } - - fn allowed_network_policy() -> NetworkPolicy { - NetworkPolicy { - allowed_targets: vec![NetworkTargetPattern { - scheme: Some(NetworkScheme::Https), - host_pattern: "api.example.test".to_string(), - port: None, - }], - deny_private_ip_ranges: true, - max_egress_bytes: Some(1024), - } - } -} diff --git a/crates/ironclaw_host_runtime/src/obligations/handler.rs b/crates/ironclaw_host_runtime/src/obligations/handler.rs new file mode 100644 index 00000000000..e376688d0db --- /dev/null +++ b/crates/ironclaw_host_runtime/src/obligations/handler.rs @@ -0,0 +1,1276 @@ +//! Obligation handling — one of the three chartered owners of the obligation +//! module (PROPOSAL §6.5.9, CHECKLIST WS3). +//! +//! This module owns the decision half: which obligations this host runtime +//! supports, what each one does before and after dispatch, and the audit, +//! redaction, resource-ceiling and mount validation that back them. It reads +//! and writes the staging stores in [`super::staged_handoffs`] but does not own +//! them, and it hands post-start cleanup to [`super::process_store`]. + +use std::sync::Arc; + +use async_trait::async_trait; +use chrono::Utc; +use ironclaw_capabilities::{ + CapabilityObligationAbortRequest, CapabilityObligationCompletionRequest, + CapabilityObligationError, CapabilityObligationFailureKind, CapabilityObligationHandler, + CapabilityObligationOutcome, CapabilityObligationPhase, CapabilityObligationRequest, +}; +use ironclaw_events::{ + AuditSink, SecurityAuditEvent, SecurityAuditSink, SecurityBoundary, SecurityDecision, +}; +use ironclaw_host_api::{ + action::NetworkPolicy, + audit::{ActionResultSummary, ActionSummary, AuditEnvelope, AuditStage, DecisionSummary}, + capability::{EffectKind, RuntimeCredentialAccountSetup}, + decision::{Obligation, RuntimeCredentialAuthRequirement}, + dispatch::{CapabilityDispatchResult, CredentialStageError}, + ids::{AuditEventId, CapabilityId, ExtensionId, SecretHandle, VendorId}, + mount::MountView, + resource::{ + ResourceCeiling, ResourceEstimate, ResourceReservation, ResourceScope, ResourceUsage, + SandboxQuota, + }, +}; +use ironclaw_resources::ResourceGovernor; +use ironclaw_safety::LeakDetector; +use ironclaw_secrets::{SecretStoreError, SecretStorePort}; + +use super::staged_handoffs::{ + NetworkObligationPolicyStore, RuntimeCredentialAccountRequest, + RuntimeCredentialAccountResolver, RuntimeSecretInjectionStore, +}; + +/// Built-in obligation handler for the current host-runtime slice. +#[derive(Clone, Default)] +pub struct BuiltinObligationHandler { + audit_sink: Option>, + security_audit_sink: Option>, + network_policies: Option>, + secret_store: Option>, + secret_injections: Option>, + resource_governor: Option>, + credential_account_resolver: Option>, +} + +struct ResolvedSecretInjection { + handle: SecretHandle, + source_scope: ResourceScope, +} + +impl BuiltinObligationHandler { + pub fn new() -> Self { + Self::default() + } + + pub fn with_audit_sink(mut self, sink: Arc) -> Self + where + T: AuditSink + 'static, + { + let sink: Arc = sink; + self.audit_sink = Some(sink); + self + } + + pub fn with_audit_sink_dyn(mut self, sink: Arc) -> Self { + self.audit_sink = Some(sink); + self + } + + /// Wire in a [`SecurityAuditSink`] for boundary-decision recording. + /// + /// Currently consumed by the output-redaction (leak-detector) path in + /// [`Self::complete_dispatch`]. Additional boundaries inside this handler + /// will adopt the same sink in follow-up PRs; the wiring is intentionally + /// optional so unconfigured callers keep working unchanged. + pub fn with_security_audit_sink(mut self, sink: Arc) -> Self { + self.security_audit_sink = Some(sink); + self + } + + pub(crate) fn with_network_policy_store( + mut self, + store: Arc, + ) -> Self { + self.network_policies = Some(store); + self + } + + pub fn with_secret_store(mut self, store: Arc) -> Self + where + T: SecretStorePort + 'static, + { + let store: Arc = store; + self.secret_store = Some(store); + self + } + + pub fn with_secret_store_dyn(mut self, store: Arc) -> Self { + self.secret_store = Some(store); + self + } + + pub(crate) fn with_secret_injection_store( + mut self, + store: Arc, + ) -> Self { + self.secret_injections = Some(store); + self + } + + pub fn with_resource_governor(mut self, governor: Arc) -> Self + where + T: ResourceGovernor + 'static, + { + let governor: Arc = governor; + self.resource_governor = Some(governor); + self + } + + pub fn with_resource_governor_dyn(mut self, governor: Arc) -> Self { + self.resource_governor = Some(governor); + self + } + + pub fn with_credential_account_resolver(mut self, resolver: Arc) -> Self + where + T: RuntimeCredentialAccountResolver + 'static, + { + let resolver: Arc = resolver; + self.credential_account_resolver = Some(resolver); + self + } + + pub fn with_credential_account_resolver_dyn( + mut self, + resolver: Arc, + ) -> Self { + self.credential_account_resolver = Some(resolver); + self + } + + async fn emit_audit_before( + &self, + request: &CapabilityObligationRequest<'_>, + ) -> Result<(), CapabilityObligationError> { + let Some(audit_sink) = &self.audit_sink else { + return Err(CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Audit, + }); + }; + + audit_sink + .emit_audit(audit_before_record(request)) + .await + .map_err(|_| CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Audit, + }) + } + + async fn preflight_secret_injection( + &self, + request: &CapabilityObligationRequest<'_>, + handles: &[SecretHandle], + ) -> Result, CapabilityObligationError> { + if handles.is_empty() { + return Ok(Vec::new()); + } + let Some(secret_store) = &self.secret_store else { + return Err(secret_obligation_failed()); + }; + if self.secret_injections.is_none() { + return Err(secret_obligation_failed()); + } + let mut resolved = Vec::with_capacity(handles.len()); + for handle in handles { + // Fail closed on a store error: the dispatch-time backstop must never + // let an uncredentialed call through on a transient failure. Preserve the + // cause as a server-side trail (`SecretStoreError` Display carries no raw + // secret material — handles/reasons only); the caller still receives the + // opaque, sanitized secret-obligation failure. + let owner = match secret_owner_scope( + secret_store.as_ref(), + &request.context.resource_scope, + handle, + ) + .await + { + Ok(owner) => owner, + Err(error) => { + tracing::debug!( + secret_handle = handle.as_str(), + error = %error, + "dispatch-time secret presence probe failed; failing closed at the obligation backstop" + ); + return Err(secret_obligation_failed()); + } + }; + let Some(source_scope) = owner else { + return Err(CapabilityObligationError::AuthRequired { + credential_requirements: Vec::new(), + }); + }; + resolved.push(ResolvedSecretInjection { + handle: handle.clone(), + source_scope, + }); + } + Ok(resolved) + } + + async fn inject_secrets( + &self, + request: &CapabilityObligationRequest<'_>, + resolved: &[ResolvedSecretInjection], + ) -> Result<(), CapabilityObligationError> { + if resolved.is_empty() { + return Ok(()); + } + let Some(secret_store) = &self.secret_store else { + return Err(secret_obligation_failed()); + }; + let Some(secret_injections) = &self.secret_injections else { + return Err(secret_obligation_failed()); + }; + + let mut material = Vec::with_capacity(resolved.len()); + for resolved in resolved { + // Use the same source scope the presence probe accepted: the caller's + // own secret if present, else the tenant-shared admin-managed secret + // (#5459). The injection target below stays the caller's invocation + // slot regardless of where the source material came from. The + // lease/consume operations remain authoritative if the secret + // vanishes between preflight and here. + // Every arm below fails closed; the bound error is logged first so a + // shared-secret lease/consume fault (e.g. an AAD/scope regression on + // the cross-scope read) leaves a server-side trail. `SecretStoreError` + // Display carries no raw secret material — handles/reasons only. + let lease = secret_store + .lease_once(&resolved.source_scope, &resolved.handle) + .await + .map_err(|error| { + tracing::debug!( + secret_handle = resolved.handle.as_str(), + error = %error, + "secret injection: lease failed; failing closed" + ); + secret_obligation_failed() + })?; + let secret = secret_store + .consume(&resolved.source_scope, lease.id) + .await + .map_err(|error| { + tracing::debug!( + secret_handle = resolved.handle.as_str(), + error = %error, + "secret injection: lease consume failed; failing closed" + ); + secret_obligation_failed() + })?; + material.push((resolved.handle.clone(), secret)); + } + + for (handle, secret) in material { + secret_injections + .insert( + &request.context.resource_scope, + request.capability_id, + &handle, + secret, + ) + .map_err(|error| { + tracing::debug!( + secret_handle = handle.as_str(), + error = %error, + "secret injection: injection-slot insert failed; failing closed" + ); + secret_obligation_failed() + })?; + } + Ok(()) + } + + async fn inject_credential_accounts( + &self, + request: &CapabilityObligationRequest<'_>, + ) -> Result<(), CapabilityObligationError> { + let account_obligations = credential_account_injection_obligations(request.obligations); + if account_obligations.is_empty() { + return Ok(()); + } + let Some(resolver) = &self.credential_account_resolver else { + return Err(secret_obligation_failed()); + }; + let Some(secret_store) = &self.secret_store else { + return Err(secret_obligation_failed()); + }; + let Some(secret_injections) = &self.secret_injections else { + return Err(secret_obligation_failed()); + }; + + for obligation in account_obligations { + let access_secret = resolver + .resolve_access_secret(RuntimeCredentialAccountRequest { + scope: &request.context.resource_scope, + provider: obligation.provider, + setup: obligation.setup, + provider_scopes: obligation.provider_scopes, + requester_extension: obligation.requester_extension, + }) + .await + .map_err(|error| { + credential_stage_error_to_obligation_error(error, Some(&obligation)) + })?; + // Retrieve and stage the resolved credential under the obligation's injection handle. + // The access_secret names the material in the secret store; obligation.handle is + // the slot name the WASM guest expects. + stage_credential_material( + secret_store.as_ref(), + secret_injections, + &access_secret.scope, + &request.context.resource_scope, + request.capability_id, + &access_secret.handle, + obligation.handle, + ) + .await + .map_err(|error| { + credential_stage_error_to_obligation_error(error, Some(&obligation)) + })?; + } + + Ok(()) + } + + fn reserve_resource_obligation( + &self, + request: &CapabilityObligationRequest<'_>, + ) -> Result, CapabilityObligationError> { + let mut reservation_id = None; + for obligation in request.obligations { + if let Obligation::ReserveResources { reservation_id: id } = obligation { + if reservation_id.is_some() { + return Err(resource_obligation_failed()); + } + reservation_id = Some(*id); + } + } + let Some(reservation_id) = reservation_id else { + return Ok(None); + }; + let Some(governor) = &self.resource_governor else { + return Err(resource_obligation_failed()); + }; + governor + .reserve_with_id( + request.context.resource_scope.clone(), + request.estimate.clone(), + reservation_id, + ) + .map(Some) + .map_err(|_| resource_obligation_failed()) + } + + fn preflight_resource_ceiling( + &self, + request: &CapabilityObligationRequest<'_>, + ) -> Result<(), CapabilityObligationError> { + let Some(ceiling) = resource_ceiling_obligation(request.obligations)? else { + return Ok(()); + }; + validate_supported_resource_ceiling(ceiling)?; + validate_estimate_within_ceiling(request.estimate, ceiling) + } + + async fn finish_prepare( + &self, + request: &CapabilityObligationRequest<'_>, + resolved_secret_injections: &[ResolvedSecretInjection], + network_policy: Option, + ) -> Result<(), CapabilityObligationError> { + if request + .obligations + .iter() + .any(|obligation| matches!(obligation, Obligation::AuditBefore)) + { + self.emit_audit_before(request).await?; + } + + self.inject_secrets(request, resolved_secret_injections) + .await?; + self.inject_credential_accounts(request).await?; + + if let Some(policy) = network_policy { + let Some(store) = &self.network_policies else { + return Err(network_obligation_failed()); + }; + store.insert( + &request.context.resource_scope, + request.capability_id, + policy, + ); + } + + Ok(()) + } + + async fn emit_audit_after( + &self, + request: &CapabilityObligationCompletionRequest<'_>, + output_bytes: u64, + ) -> Result<(), CapabilityObligationError> { + let Some(audit_sink) = &self.audit_sink else { + return Err(CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Audit, + }); + }; + + audit_sink + .emit_audit(audit_after_record(request, output_bytes)) + .await + .map_err(|_| CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Audit, + }) + } +} + +#[async_trait] +impl CapabilityObligationHandler for BuiltinObligationHandler { + async fn satisfy( + &self, + request: CapabilityObligationRequest<'_>, + ) -> Result<(), CapabilityObligationError> { + // `satisfy` is the direct one-shot path for callers that need staged + // network/secret handoff but do not need to pass prepared mounts or a + // reservation downstream. Resource reservations are released without + // discarding staged handoffs because successful callers still need the + // network/secret material handed to runtime adapters. CapabilityHost + // uses `prepare`/`complete`/`abort` directly instead. Post-dispatch + // obligations fail closed here because this path has no dispatch result + // to redact, limit, or audit. + let post_dispatch = post_dispatch_obligations(request.obligations); + if !post_dispatch.is_empty() { + return Err(CapabilityObligationError::Unsupported { + obligations: post_dispatch, + }); + } + let outcome = self + .prepare(CapabilityObligationRequest { + phase: request.phase, + context: request.context, + capability_id: request.capability_id, + estimate: request.estimate, + obligations: request.obligations, + }) + .await?; + if let Some(reservation) = &outcome.resource_reservation + && let Err(error) = self.release_resource_reservation(reservation) + { + if let Err(cleanup_error) = self.discard_staged_handoffs( + &request.context.resource_scope, + request.capability_id, + request.obligations, + ) { + tracing::debug!(error = ?cleanup_error, "best-effort discard of staged handoffs failed"); + } + return Err(error); + } + Ok(()) + } + + async fn prepare( + &self, + request: CapabilityObligationRequest<'_>, + ) -> Result { + let unsupported = unsupported_obligations(request.phase, request.obligations); + if !unsupported.is_empty() { + return Err(CapabilityObligationError::Unsupported { + obligations: unsupported, + }); + } + + let network_policy = network_policy_obligation(request.obligations)?; + if network_policy.is_some() && self.network_policies.is_none() { + return Err(network_obligation_failed()); + } + let scoped_mounts = scoped_mount_obligation(request.context, request.obligations)?; + let secret_handles = secret_injection_handles(request.obligations); + let resolved_secret_injections = self + .preflight_secret_injection(&request, &secret_handles) + .await?; + self.preflight_resource_ceiling(&request)?; + let resource_reservation = self.reserve_resource_obligation(&request)?; + let outcome = CapabilityObligationOutcome { + mounts: scoped_mounts, + resource_reservation, + }; + + if let Err(error) = self + .finish_prepare(&request, &resolved_secret_injections, network_policy) + .await + { + self.abort(CapabilityObligationAbortRequest { + phase: request.phase, + context: request.context, + capability_id: request.capability_id, + estimate: request.estimate, + obligations: request.obligations, + outcome: &outcome, + }) + .await?; + return Err(error); + } + + Ok(outcome) + } + + async fn abort( + &self, + request: CapabilityObligationAbortRequest<'_>, + ) -> Result<(), CapabilityObligationError> { + self.discard_staged_handoffs( + &request.context.resource_scope, + request.capability_id, + request.obligations, + )?; + + if let Some(reservation) = &request.outcome.resource_reservation { + self.release_resource_reservation(reservation)?; + } + Ok(()) + } + + async fn complete_dispatch( + &self, + request: CapabilityObligationCompletionRequest<'_>, + ) -> Result { + let unsupported = unsupported_completion_obligations(request.phase, request.obligations); + if !unsupported.is_empty() { + return Err(CapabilityObligationError::Unsupported { + obligations: unsupported, + }); + } + + let mut dispatch = request.dispatch.clone(); + // Turn any base64 document payload into extracted text before redaction + // and the output-size obligations run, so the model gets bounded text + // (leak-scanned, size-checked) and the large base64 never survives. + dispatch.output = crate::document_output::extract_documents_in_output( + dispatch.capability_id.as_str(), + dispatch.output, + ); + if request + .obligations + .iter() + .any(|obligation| matches!(obligation, Obligation::RedactOutput)) + { + dispatch.output = match redact_output(dispatch.output) { + Ok(value) => value, + Err(error) => { + // Leak-detector blocked: record the boundary decision + // before propagating. The event is payload-free by + // construction — only the boundary, decision, and a + // stable code reach the sink. The original output never + // leaves the type system. + if let Some(sink) = &self.security_audit_sink { + let event = SecurityAuditEvent::new( + SecurityBoundary::LeakDetector, + SecurityDecision::Blocked, + LEAK_REDACT_FAILED_CODE, + ) + .with_capability_id(request.capability_id.clone()) + .with_scope(request.context.resource_scope.clone()); + sink.record(event); + } + return Err(error); + } + }; + dispatch.display_preview = None; + } + + let output_bytes = dispatch_output_bytes(&dispatch.output)?; + for obligation in request.obligations { + if let Obligation::EnforceResourceCeiling { ceiling } = obligation { + validate_supported_resource_ceiling(ceiling)?; + validate_usage_within_ceiling(&dispatch.usage, output_bytes, ceiling)?; + } + } + for obligation in request.obligations { + if let Obligation::EnforceOutputLimit { bytes } = obligation + && output_bytes > *bytes + { + return Err(output_obligation_failed()); + } + } + + self.discard_staged_handoffs( + &request.context.resource_scope, + request.capability_id, + request.obligations, + )?; + + if request + .obligations + .iter() + .any(|obligation| matches!(obligation, Obligation::AuditAfter)) + { + self.emit_audit_after(&request, output_bytes).await?; + } + + Ok(dispatch) + } +} + +impl BuiltinObligationHandler { + fn release_resource_reservation( + &self, + reservation: &ResourceReservation, + ) -> Result<(), CapabilityObligationError> { + let Some(governor) = &self.resource_governor else { + return Err(resource_obligation_failed()); + }; + governor + .release(reservation.id) + .map(|_| ()) + .map_err(|_| resource_obligation_failed()) + } + + fn discard_staged_handoffs( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + obligations: &[Obligation], + ) -> Result<(), CapabilityObligationError> { + if obligations + .iter() + .any(|obligation| matches!(obligation, Obligation::ApplyNetworkPolicy { .. })) + && let Some(store) = &self.network_policies + { + let _ = store.take(scope, capability_id); + } + + if let Some(store) = &self.secret_injections { + for handle in staged_secret_injection_handles(obligations) { + let _ = store + .take(scope, capability_id, &handle) + .map_err(|_| secret_obligation_failed())?; + } + } + + Ok(()) + } +} + +fn post_dispatch_obligations(obligations: &[Obligation]) -> Vec { + obligations + .iter() + .filter(|obligation| { + matches!( + obligation, + Obligation::AuditAfter + | Obligation::RedactOutput + | Obligation::EnforceResourceCeiling { .. } + | Obligation::EnforceOutputLimit { .. } + ) + }) + .cloned() + .collect() +} + +fn unsupported_obligations( + phase: CapabilityObligationPhase, + obligations: &[Obligation], +) -> Vec { + obligations + .iter() + .filter(|obligation| !obligation_supported_before_dispatch(phase, obligation)) + .cloned() + .collect() +} + +fn obligation_supported_before_dispatch( + phase: CapabilityObligationPhase, + obligation: &Obligation, +) -> bool { + match obligation { + Obligation::AuditBefore + | Obligation::ApplyNetworkPolicy { .. } + | Obligation::FirstPartyCredentialStagedViaHostPort { .. } + | Obligation::InjectCredentialAccountOnce { .. } + | Obligation::InjectSecretOnce { .. } + | Obligation::ReserveResources { .. } + | Obligation::UseScopedMounts { .. } => true, + Obligation::EnforceResourceCeiling { .. } => { + !matches!(phase, CapabilityObligationPhase::Spawn) + } + Obligation::AuditAfter + | Obligation::RedactOutput + | Obligation::EnforceOutputLimit { .. } => { + !matches!(phase, CapabilityObligationPhase::Spawn) + } + } +} + +fn unsupported_completion_obligations( + phase: CapabilityObligationPhase, + obligations: &[Obligation], +) -> Vec { + obligations + .iter() + .filter(|obligation| !obligation_supported_after_dispatch(phase, obligation)) + .cloned() + .collect() +} + +fn obligation_supported_after_dispatch( + phase: CapabilityObligationPhase, + obligation: &Obligation, +) -> bool { + match obligation { + Obligation::AuditBefore + | Obligation::ApplyNetworkPolicy { .. } + | Obligation::FirstPartyCredentialStagedViaHostPort { .. } + | Obligation::InjectCredentialAccountOnce { .. } + | Obligation::InjectSecretOnce { .. } + | Obligation::ReserveResources { .. } + | Obligation::UseScopedMounts { .. } => true, + Obligation::EnforceResourceCeiling { .. } => { + !matches!(phase, CapabilityObligationPhase::Spawn) + } + Obligation::AuditAfter + | Obligation::RedactOutput + | Obligation::EnforceOutputLimit { .. } => { + !matches!(phase, CapabilityObligationPhase::Spawn) + } + } +} + +fn secret_injection_handles(obligations: &[Obligation]) -> Vec { + obligations + .iter() + .filter_map(|obligation| match obligation { + Obligation::InjectSecretOnce { handle } => Some(handle.clone()), + _ => None, + }) + .collect() +} + +struct CredentialAccountInjectionObligation<'a> { + handle: &'a SecretHandle, + provider: &'a VendorId, + setup: &'a RuntimeCredentialAccountSetup, + provider_scopes: &'a [String], + requester_extension: &'a ExtensionId, +} + +fn credential_account_injection_obligations( + obligations: &[Obligation], +) -> Vec> { + obligations + .iter() + .filter_map(|obligation| match obligation { + Obligation::InjectCredentialAccountOnce { + handle, + provider, + setup, + provider_scopes, + requester_extension, + } => Some(CredentialAccountInjectionObligation { + handle, + provider, + setup, + provider_scopes, + requester_extension, + }), + _ => None, + }) + .collect() +} + +fn staged_secret_injection_handles(obligations: &[Obligation]) -> Vec { + obligations + .iter() + .filter_map(|obligation| match obligation { + Obligation::InjectSecretOnce { handle } + | Obligation::InjectCredentialAccountOnce { handle, .. } => Some(handle.clone()), + _ => None, + }) + .collect() +} + +/// Map the canonical staged-credential error to the obligation-handler error type. +/// +/// Used by both [`inject_credential_accounts`] (resolver-side errors) and +/// [`stage_credential_material`] (storage-side errors) so the WASM +/// `InjectCredentialAccountOnce` path and the first-party stager path share +/// the same AuthRequired/Backend semantics. +fn credential_stage_error_to_obligation_error( + error: CredentialStageError, + credential_obligation: Option<&CredentialAccountInjectionObligation<'_>>, +) -> CapabilityObligationError { + match error { + CredentialStageError::AuthRequired => CapabilityObligationError::AuthRequired { + credential_requirements: credential_obligation + .map(|obligation| { + vec![RuntimeCredentialAuthRequirement { + provider: obligation.provider.clone(), + setup: obligation.setup.clone(), + requester_extension: obligation.requester_extension.clone(), + provider_scopes: obligation.provider_scopes.to_vec(), + }] + }) + .unwrap_or_default(), + }, + CredentialStageError::Backend => secret_obligation_failed(), + } +} + +/// Retrieve `source` from the secret store and stage the material under `target` +/// in the injection store for the given capability invocation. +/// +/// Used when the secret store key (`source`) differs from the runtime injection slot +/// (`target`) — for example, when a product-auth account's backing secret is resolved +/// to a concrete handle before being injected under the WASM guest's declared slot name. +/// Lease → consume → insert the staged credential material. +/// +/// Mirrors [`crate::services::ProductAuthProviderRuntimePorts::stage_secret_once`] +/// so the WASM `InjectCredentialAccountOnce` path and the first-party stager path +/// (e.g. `ProductAuthRuntimeGsuiteCredentialStager`) share identical lease/consume +/// semantics and `CredentialStageError` mapping. `SecretStoreError` variants for +/// unknown/expired/revoked/consumed material map to +/// [`CredentialStageError::AuthRequired`] via [`crate::services::stage_secret_error`]; +/// other failures map to [`CredentialStageError::Backend`]. +async fn stage_credential_material( + secret_store: &dyn SecretStorePort, + secret_injections: &RuntimeSecretInjectionStore, + source_scope: &ResourceScope, + target_scope: &ResourceScope, + capability_id: &CapabilityId, + source: &SecretHandle, + target: &SecretHandle, +) -> Result<(), CredentialStageError> { + let lease = secret_store + .lease_once(source_scope, source) + .await + .map_err(|e| { + tracing::debug!(err = %e, "stage_credential_material: lease_once failed"); + crate::services::stage_secret_error(e) + })?; + let secret = secret_store + .consume(source_scope, lease.id) + .await + .map_err(|e| { + tracing::debug!(err = %e, "stage_credential_material: consume failed"); + crate::services::stage_secret_error(e) + })?; + secret_injections + .insert(target_scope, capability_id, target, secret) + .map_err(|e| { + tracing::debug!(err = %e, "stage_credential_material: insert failed"); + CredentialStageError::Backend + }) +} + +fn network_policy_obligation( + obligations: &[Obligation], +) -> Result, CapabilityObligationError> { + let mut policy = None; + for obligation in obligations { + if let Obligation::ApplyNetworkPolicy { policy: next } = obligation { + if policy.is_some() { + return Err(network_obligation_failed()); + } + validate_network_policy_metadata(next)?; + policy = Some(next.clone()); + } + } + Ok(policy) +} + +fn scoped_mount_obligation( + context: &ironclaw_host_api::scope::ExecutionContext, + obligations: &[Obligation], +) -> Result, CapabilityObligationError> { + let mut mounts = None; + for obligation in obligations { + if let Obligation::UseScopedMounts { mounts: next } = obligation { + if mounts.is_some() { + return Err(mount_obligation_failed()); + } + next.validate().map_err(|_| mount_obligation_failed())?; + if !next.is_subset_of(&context.mounts) { + return Err(mount_obligation_failed()); + } + mounts = Some(next.clone()); + } + } + Ok(mounts) +} + +fn resource_ceiling_obligation( + obligations: &[Obligation], +) -> Result, CapabilityObligationError> { + let mut ceiling = None; + for obligation in obligations { + if let Obligation::EnforceResourceCeiling { ceiling: next } = obligation { + if ceiling.is_some() { + return Err(resource_obligation_failed()); + } + ceiling = Some(next); + } + } + Ok(ceiling) +} + +fn validate_supported_resource_ceiling( + ceiling: &ResourceCeiling, +) -> Result<(), CapabilityObligationError> { + if ceiling.max_wall_clock_ms.is_some() { + return Err(resource_obligation_failed()); + } + if let Some(sandbox) = &ceiling.sandbox { + validate_supported_sandbox_quota(sandbox)?; + } + Ok(()) +} + +fn validate_supported_sandbox_quota( + sandbox: &SandboxQuota, +) -> Result<(), CapabilityObligationError> { + if sandbox.cpu_time_ms.is_some() + || sandbox.memory_bytes.is_some() + || sandbox.disk_bytes.is_some() + || sandbox.network_egress_bytes.is_some() + || sandbox.process_count.is_some() + { + return Err(resource_obligation_failed()); + } + Ok(()) +} + +fn validate_estimate_within_ceiling( + estimate: &ResourceEstimate, + ceiling: &ResourceCeiling, +) -> Result<(), CapabilityObligationError> { + check_optional_decimal_ceiling(estimate.usd, ceiling.max_usd)?; + check_required_integer_ceiling(estimate.input_tokens, ceiling.max_input_tokens)?; + check_required_integer_ceiling(estimate.output_tokens, ceiling.max_output_tokens)?; + Ok(()) +} + +fn validate_usage_within_ceiling( + usage: &ResourceUsage, + output_bytes: u64, + ceiling: &ResourceCeiling, +) -> Result<(), CapabilityObligationError> { + check_decimal_ceiling(usage.usd, ceiling.max_usd)?; + check_integer_ceiling(usage.input_tokens, ceiling.max_input_tokens)?; + check_integer_ceiling(usage.output_tokens, ceiling.max_output_tokens)?; + check_output_bytes_ceiling(output_bytes, ceiling.max_output_bytes)?; + Ok(()) +} + +fn check_output_bytes_ceiling( + actual: u64, + ceiling: Option, +) -> Result<(), CapabilityObligationError> { + if let Some(ceiling) = ceiling + && actual > ceiling + { + return Err(output_obligation_failed()); + } + Ok(()) +} + +fn check_optional_decimal_ceiling( + actual: Option, + ceiling: Option, +) -> Result<(), CapabilityObligationError> { + let Some(ceiling) = ceiling else { + return Ok(()); + }; + let Some(actual) = actual else { + return Err(resource_obligation_failed()); + }; + check_decimal_ceiling(actual, Some(ceiling)) +} + +fn check_decimal_ceiling( + actual: rust_decimal::Decimal, + ceiling: Option, +) -> Result<(), CapabilityObligationError> { + if let Some(ceiling) = ceiling + && actual > ceiling + { + return Err(resource_obligation_failed()); + } + Ok(()) +} + +fn check_required_integer_ceiling( + actual: Option, + ceiling: Option, +) -> Result<(), CapabilityObligationError> { + let Some(ceiling) = ceiling else { + return Ok(()); + }; + let Some(actual) = actual else { + return Err(resource_obligation_failed()); + }; + check_integer_ceiling(actual, Some(ceiling)) +} + +fn check_integer_ceiling( + actual: u64, + ceiling: Option, +) -> Result<(), CapabilityObligationError> { + if let Some(ceiling) = ceiling + && actual > ceiling + { + return Err(resource_obligation_failed()); + } + Ok(()) +} + +fn validate_network_policy_metadata( + policy: &NetworkPolicy, +) -> Result<(), CapabilityObligationError> { + if policy.allowed_targets.is_empty() { + return Err(network_obligation_failed()); + } + Ok(()) +} + +fn network_obligation_failed() -> CapabilityObligationError { + CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Network, + } +} + +fn secret_obligation_failed() -> CapabilityObligationError { + CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Secret, + } +} + +/// Single source of truth for "is this required secret present in `scope`". +/// +/// Both the credential pre-flight (ordering — `DefaultHostRuntime:: +/// credential_preflight_check`) and the dispatch-time obligation backstop +/// (enforcement — [`BuiltinObligationHandler::preflight_secret_injection`]) +/// consult this one rule so "what counts as a present credential" cannot drift +/// between the two call sites. Each caller decides how to treat a store `Err` +/// (the pre-flight fails open and skips; the obligation backstop fails closed). +async fn secret_present( + store: &dyn SecretStorePort, + scope: &ResourceScope, + handle: &SecretHandle, +) -> Result { + Ok(store.metadata(scope, handle).await?.is_some()) +} + +/// Resolve which scope owns `handle` for this caller, honoring tenant-shared, +/// admin-managed credentials (#5459). A caller's OWN secret wins; otherwise the +/// tenant-shared admin-managed scope ([`ResourceScope::tenant_shared_managed_scope`]), +/// so one admin-set key satisfies every user of the tenant. `Ok(None)` means the +/// secret is absent in both scopes. +/// +/// Single source of truth for BOTH "is this required secret present" (callers map +/// to `.is_some()`) and "where does the lease read from" — the pre-flight ordering +/// probe (`credential_preflight_check`) and the dispatch-time backstop +/// (`preflight_secret_injection`) consult this rule, then the injection lease +/// (`inject_secrets`) consumes the resolved source scope. Each caller decides how +/// to treat a store `Err` (the pre-flight fails open and skips; the obligation +/// backstop and lease fail closed). +pub(crate) async fn secret_owner_scope( + store: &dyn SecretStorePort, + caller_scope: &ResourceScope, + handle: &SecretHandle, +) -> Result, SecretStoreError> { + if secret_present(store, caller_scope, handle).await? { + return Ok(Some(caller_scope.clone())); + } + let shared = caller_scope.tenant_shared_managed_scope(); + if secret_present(store, &shared, handle).await? { + return Ok(Some(shared)); + } + Ok(None) +} + +fn resource_obligation_failed() -> CapabilityObligationError { + CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Resource, + } +} + +fn mount_obligation_failed() -> CapabilityObligationError { + CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Mount, + } +} + +fn output_obligation_failed() -> CapabilityObligationError { + CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Output, + } +} + +fn dispatch_output_bytes(output: &serde_json::Value) -> Result { + serde_json::to_vec(output) + .map(|bytes| bytes.len() as u64) + .map_err(|_| output_obligation_failed()) +} + +/// Security-audit reason code emitted when [`redact_output`] rejects output +/// because the leak detector matched. Stable grep target for SRE pattern +/// matching across durable security-audit logs. +pub const LEAK_REDACT_FAILED_CODE: &str = "leak_redact_failed"; + +fn redact_output( + output: serde_json::Value, +) -> Result { + match output { + serde_json::Value::String(value) => { + redact_output_string(value).map(serde_json::Value::String) + } + serde_json::Value::Array(values) => values + .into_iter() + .map(redact_output) + .collect::, _>>() + .map(serde_json::Value::Array), + serde_json::Value::Object(entries) => { + let mut redacted = serde_json::Map::with_capacity(entries.len()); + for (key, value) in entries { + let key = redact_output_string(key)?; + let value = redact_output(value)?; + if redacted.insert(key, value).is_some() { + return Err(output_obligation_failed()); + } + } + Ok(serde_json::Value::Object(redacted)) + } + value => Ok(value), + } +} + +fn redact_output_string(value: String) -> Result { + LeakDetector::new() + .scan_and_clean(&value) + .map_err(|_| output_obligation_failed()) +} + +fn audit_before_record(request: &CapabilityObligationRequest<'_>) -> AuditEnvelope { + AuditEnvelope { + event_id: AuditEventId::new(), + correlation_id: request.context.correlation_id, + stage: AuditStage::Before, + timestamp: Utc::now(), + tenant_id: request.context.tenant_id.clone(), + user_id: request.context.user_id.clone(), + agent_id: request.context.agent_id.clone(), + project_id: request.context.project_id.clone(), + mission_id: request.context.mission_id.clone(), + thread_id: request.context.thread_id.clone(), + invocation_id: request.context.invocation_id, + process_id: request.context.process_id, + approval_request_id: None, + extension_id: Some(request.context.extension_id.clone()), + action: ActionSummary { + kind: capability_action_kind(request.phase).to_string(), + target: Some(request.capability_id.as_str().to_string()), + effects: capability_action_effects(request.phase), + }, + decision: DecisionSummary { + kind: "obligation_satisfied".to_string(), + reason: None, + actor: None, + }, + result: Some(ActionResultSummary { + success: true, + status: Some(obligation_status(request.obligations)), + output_bytes: None, + }), + } +} + +fn audit_after_record( + request: &CapabilityObligationCompletionRequest<'_>, + output_bytes: u64, +) -> AuditEnvelope { + AuditEnvelope { + event_id: AuditEventId::new(), + correlation_id: request.context.correlation_id, + stage: AuditStage::After, + timestamp: Utc::now(), + tenant_id: request.context.tenant_id.clone(), + user_id: request.context.user_id.clone(), + agent_id: request.context.agent_id.clone(), + project_id: request.context.project_id.clone(), + mission_id: request.context.mission_id.clone(), + thread_id: request.context.thread_id.clone(), + invocation_id: request.context.invocation_id, + process_id: request.context.process_id, + approval_request_id: None, + extension_id: Some(request.context.extension_id.clone()), + action: ActionSummary { + kind: capability_action_kind(request.phase).to_string(), + target: Some(request.capability_id.as_str().to_string()), + effects: capability_action_effects(request.phase), + }, + decision: DecisionSummary { + kind: "obligation_satisfied".to_string(), + reason: None, + actor: None, + }, + result: Some(ActionResultSummary { + success: true, + status: Some(obligation_status(request.obligations)), + output_bytes: Some(output_bytes), + }), + } +} + +fn capability_action_kind(phase: CapabilityObligationPhase) -> &'static str { + match phase { + CapabilityObligationPhase::Invoke => "capability_invoke", + CapabilityObligationPhase::Resume => "capability_resume", + CapabilityObligationPhase::Spawn => "capability_spawn", + } +} + +fn capability_action_effects(phase: CapabilityObligationPhase) -> Vec { + match phase { + CapabilityObligationPhase::Invoke | CapabilityObligationPhase::Resume => { + vec![EffectKind::DispatchCapability] + } + CapabilityObligationPhase::Spawn => { + vec![EffectKind::DispatchCapability, EffectKind::SpawnProcess] + } + } +} + +fn obligation_status(obligations: &[Obligation]) -> String { + obligations + .iter() + .filter_map(obligation_label) + .collect::>() + .join(",") +} + +fn obligation_label(obligation: &Obligation) -> Option<&'static str> { + match obligation { + Obligation::AuditBefore => Some("audit_before"), + Obligation::AuditAfter => Some("audit_after"), + Obligation::RedactOutput => Some("redact_output"), + Obligation::ApplyNetworkPolicy { .. } => Some("apply_network_policy"), + Obligation::InjectSecretOnce { .. } => Some("inject_secret_once"), + Obligation::InjectCredentialAccountOnce { .. } => Some("inject_credential_account_once"), + Obligation::FirstPartyCredentialStagedViaHostPort { .. } => { + Some("first_party_credential_staged_via_host_port") + } + Obligation::EnforceOutputLimit { .. } => Some("enforce_output_limit"), + Obligation::ReserveResources { .. } => Some("reserve_resources"), + Obligation::UseScopedMounts { .. } => Some("use_scoped_mounts"), + Obligation::EnforceResourceCeiling { .. } => Some("enforce_resource_ceiling"), + } +} diff --git a/crates/ironclaw_host_runtime/src/obligations/mod.rs b/crates/ironclaw_host_runtime/src/obligations/mod.rs new file mode 100644 index 00000000000..70472fdb0a1 --- /dev/null +++ b/crates/ironclaw_host_runtime/src/obligations/mod.rs @@ -0,0 +1,217 @@ +//! Mediated obligation handling for the kernel service graph. +//! +//! Obligation work has three distinct owners, and this module is split along +//! them so that no single file fuses them again (PROPOSAL §6.5.9, CHECKLIST +//! WS3 — "splits internally into its three chartered owners"): +//! +//! | Owner | Module | Responsibility | +//! |---|---|---| +//! | obligation handling | [`handler`] | which obligations apply, and what each does before/after dispatch | +//! | staged secret/network handoffs | [`staged_handoffs`] | one-shot secret material and per-invocation network policy staged for a later consumer | +//! | process-obligation store | [`process_store`] | discarding handoffs and reconciling reservations once a capability process has started | +//! +//! [`BuiltinObligationServices`] below is the assembly seam that binds the +//! three together for composition; it is deliberately the only place that names +//! all three owners at once. + +use std::{fmt, sync::Arc}; + +use ironclaw_events::AuditSink; +use ironclaw_host_api::http::RuntimeHttpEgress; +use ironclaw_network::NetworkHttpEgress; +use ironclaw_processes::ProcessRuntimePort; +use ironclaw_resources::ResourceGovernor; +use ironclaw_secrets::SecretStorePort; + +use crate::{ + ToolCallHttpEgress, + http_body::{RuntimeHttpBodyStore, UnsupportedRuntimeHttpBodyStore}, +}; + +mod handler; +mod process_store; +mod staged_handoffs; + +#[cfg(test)] +mod tests; + +pub use handler::{BuiltinObligationHandler, LEAK_REDACT_FAILED_CODE}; +pub use process_store::ProcessObligationLifecycleStore; +pub use staged_handoffs::{ + RuntimeCredentialAccessSecret, RuntimeCredentialAccountRequest, + RuntimeCredentialAccountResolver, +}; + +pub(crate) use handler::secret_owner_scope; +pub(crate) use staged_handoffs::{ + NetworkObligationPolicyStore, RuntimeSecretInjectionStore, SharedSecretStore, +}; + +/// Host-runtime-owned backing services for a fully configured built-in obligation handler. +/// +/// This value is the production composition seam for obligation handling. It +/// keeps the in-memory network-policy and runtime-secret handoff stores alive +/// outside the handler so runtime adapters can consume the exact staged state +/// that [`BuiltinObligationHandler`] prepares before dispatch. +#[derive(Clone)] +pub struct BuiltinObligationServices { + audit_sink: Arc, + network_policies: Arc, + secret_store: Arc, + secret_injections: Arc, + resource_governor: Arc, + credential_account_resolver: Option>, +} + +impl BuiltinObligationServices { + pub fn new( + audit_sink: Arc, + secret_store: Arc, + resource_governor: Arc, + ) -> Self { + Self::with_handoff_stores( + audit_sink, + Arc::new(NetworkObligationPolicyStore::new()), + secret_store, + Arc::new(RuntimeSecretInjectionStore::new()), + resource_governor, + ) + } + + pub(crate) fn with_handoff_stores( + audit_sink: Arc, + network_policies: Arc, + secret_store: Arc, + secret_injections: Arc, + resource_governor: Arc, + ) -> Self { + Self { + audit_sink, + network_policies, + secret_store, + secret_injections, + resource_governor, + credential_account_resolver: None, + } + } + + pub fn with_credential_account_resolver(mut self, resolver: Arc) -> Self + where + T: RuntimeCredentialAccountResolver + 'static, + { + self.credential_account_resolver = Some(resolver); + self + } + + pub fn with_credential_account_resolver_dyn( + mut self, + resolver: Arc, + ) -> Self { + self.credential_account_resolver = Some(resolver); + self + } + + pub fn audit_sink(&self) -> Arc { + self.audit_sink.clone() + } + + pub fn secret_store(&self) -> Arc { + self.secret_store.clone() + } + + pub fn resource_governor(&self) -> Arc { + self.resource_governor.clone() + } + + /// Builds host HTTP egress over this service graph's private handoff stores. + /// Callers can supply concrete network transport without receiving mutable + /// access to staged policy or secret material. + pub fn host_http_egress( + &self, + network: N, + ) -> impl RuntimeHttpEgress + ToolCallHttpEgress + use + where + N: NetworkHttpEgress + 'static, + { + self.host_http_egress_with_body_store(network, Arc::new(UnsupportedRuntimeHttpBodyStore)) + } + + pub fn host_http_egress_with_body_store( + &self, + network: N, + body_store: Arc, + ) -> impl RuntimeHttpEgress + ToolCallHttpEgress + use + where + N: NetworkHttpEgress + 'static, + T: RuntimeHttpBodyStore + 'static, + { + let body_store: Arc = body_store; + crate::HostHttpEgressService::production( + network, + SharedSecretStore(self.secret_store.clone()), + self.network_policies.clone(), + self.secret_injections.clone(), + body_store, + ) + } + + pub fn process_obligation_lifecycle_store( + &self, + inner: Arc, + ) -> ProcessObligationLifecycleStore + where + S: ProcessRuntimePort + 'static, + { + ProcessObligationLifecycleStore::new( + inner, + self.network_policies.clone(), + self.secret_injections.clone(), + self.resource_governor.clone(), + ) + } + + pub fn process_obligation_lifecycle_store_dyn( + &self, + inner: Arc, + ) -> ProcessObligationLifecycleStore { + ProcessObligationLifecycleStore::from_dyn( + inner, + self.network_policies.clone(), + self.secret_injections.clone(), + self.resource_governor.clone(), + ) + } + + pub fn obligation_handler(&self) -> BuiltinObligationHandler { + let handler = BuiltinObligationHandler::new() + .with_audit_sink_dyn(self.audit_sink.clone()) + .with_network_policy_store(self.network_policies.clone()) + .with_secret_store_dyn(self.secret_store.clone()) + .with_secret_injection_store(self.secret_injections.clone()) + .with_resource_governor_dyn(self.resource_governor.clone()); + match &self.credential_account_resolver { + Some(resolver) => handler.with_credential_account_resolver_dyn(Arc::clone(resolver)), + None => handler, + } + } +} + +impl fmt::Debug for BuiltinObligationServices { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("BuiltinObligationServices") + .field("audit_sink", &"") + .field("network_policies", &self.network_policies) + .field("secret_store", &"[REDACTED]") + .field("secret_injections", &self.secret_injections) + .field("resource_governor", &"") + .field( + "credential_account_resolver", + &self + .credential_account_resolver + .as_ref() + .map(|_| ""), + ) + .finish() + } +} diff --git a/crates/ironclaw_host_runtime/src/obligations/process_store.rs b/crates/ironclaw_host_runtime/src/obligations/process_store.rs new file mode 100644 index 00000000000..8a12a3e7d8f --- /dev/null +++ b/crates/ironclaw_host_runtime/src/obligations/process_store.rs @@ -0,0 +1,586 @@ +//! The process-obligation store — one of the three chartered owners of the +//! obligation module (PROPOSAL §6.5.9, CHECKLIST WS3). +//! +//! Once a capability process starts, obligation cleanup stops being the +//! handler's business and becomes the process lifecycle's: staged handoffs are +//! discarded and a prepared resource reservation is reconciled or released when +//! the process reaches a terminal state. This module owns that wrapper. It +//! stages nothing itself (see [`super::staged_handoffs`]) and decides nothing +//! about which obligations apply (see [`super::handler`]). + +use std::{ + collections::{HashMap, HashSet}, + sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, + }, +}; + +use async_trait::async_trait; +use ironclaw_events::{EventSink, RuntimeEvent}; +use ironclaw_host_api::{ + ids::{CapabilityId, ProcessId}, + resource::{ResourceScope, ResourceUsage}, +}; +use ironclaw_processes::{ + ProcessError, ProcessJournalCommit, ProcessJournalCommitObserver, ProcessJournalKind, + ProcessKind, ProcessRecord, ProcessRuntimePort, ProcessStart, ProcessSubmissionLifecycle, + capability_process_record, complete_capability_process, fail_capability_process, + process_record_from_snapshot, submit_capability_process, +}; +use ironclaw_resources::{ResourceError, ResourceGovernor}; + +use super::staged_handoffs::{NetworkObligationPolicyStore, RuntimeSecretInjectionStore}; + +/// Process-store wrapper that owns spawn-phase obligation handoffs after +/// process submission succeeds. +/// +/// `CapabilityHost` aborts prepared effects when process start fails. Once +/// start succeeds, this wrapper becomes responsible for discarding staged +/// network/secret handoffs and reconciling or releasing a prepared resource +/// reservation when the process reaches a terminal state. +pub struct ProcessObligationLifecycleStore { + processes: Arc, + network_policies: Arc, + secret_injections: Arc, + resource_governor: Mutex>, + event_sink: Mutex>>, + observer_registered: AtomicBool, + active_process_handoffs: Mutex>, + cleaned_process_handoffs: Mutex>, +} + +impl ProcessObligationLifecycleStore { + pub(crate) fn new( + inner: Arc, + network_policies: Arc, + secret_injections: Arc, + resource_governor: Arc, + ) -> Self + where + S: ProcessRuntimePort + 'static, + { + let inner: Arc = inner; + Self::from_dyn( + inner, + network_policies, + secret_injections, + resource_governor, + ) + } + + pub(crate) fn from_dyn( + processes: Arc, + network_policies: Arc, + secret_injections: Arc, + resource_governor: Arc, + ) -> Self { + Self { + processes, + network_policies, + secret_injections, + resource_governor: Mutex::new(resource_governor), + event_sink: Mutex::new(None), + observer_registered: AtomicBool::new(false), + active_process_handoffs: Mutex::new(HashMap::new()), + cleaned_process_handoffs: Mutex::new(HashSet::new()), + } + } + + pub(crate) fn set_resource_governor(&self, resource_governor: Arc) { + match self.resource_governor.lock() { + Ok(mut slot) => { + *slot = resource_governor; + } + Err(error) => { + tracing::error!(error = %error, "process resource governor registry unavailable"); + } + } + } + + #[doc(hidden)] + pub fn register_journal_observer( + self: &Arc, + runtime: &dyn ProcessRuntimePort, + ) -> Result<(), String> { + if self + .observer_registered + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return Ok(()); + } + let observer: Arc = self.clone(); + if let Err(error) = runtime.subscribe_process_observer(observer) { + self.observer_registered.store(false, Ordering::Release); + return Err(error); + } + Ok(()) + } + + /// Attaches a best-effort event sink for process lifecycle transitions. + pub fn set_event_sink(&self, event_sink: Arc) { + match self.event_sink.lock() { + Ok(mut slot) => { + *slot = Some(event_sink); + } + Err(error) => { + tracing::debug!( + error = %error, + "process lifecycle event sink registry unavailable" + ); + } + } + } + + async fn emit_process_event(&self, event: RuntimeEvent) { + let event_sink = match self.event_sink.lock() { + Ok(slot) => slot.clone(), + Err(error) => { + tracing::debug!( + error = %error, + "process lifecycle event sink registry unavailable" + ); + None + } + }; + if let Some(event_sink) = event_sink + && let Err(error) = event_sink.emit(event).await + { + tracing::debug!(?error, "best-effort process lifecycle event emit failed"); + } + } + + /// Discards staged obligation handoffs and closes any reservation for an + /// executor that finished but could not publish its result record. + pub async fn cleanup_process_obligations( + &self, + scope: &ResourceScope, + process_id: ProcessId, + reconcile: bool, + ) -> Result<(), ProcessError> { + if let Some(record) = + capability_process_record(self.processes.as_ref(), scope, process_id).await? + { + self.cleanup_record_obligations(&record, reconcile)?; + self.release_active_process_handoff(&record)?; + self.mark_process_handoff_cleaned(&record)?; + } + Ok(()) + } + + fn has_process_obligations(&self, start: &ProcessStart) -> Result { + let has_secret_handoff = self + .secret_injections + .has_for_capability(&start.scope, &start.capability_id) + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff lookup failed".to_string(), + })?; + Ok(start.resource_reservation_id.is_some() + || self + .network_policies + .contains(&start.scope, &start.capability_id) + || has_secret_handoff) + } + + fn claim_active_process_handoff(&self, start: &ProcessStart) -> Result { + if !self.has_process_obligations(start)? { + return Ok(false); + } + + let key = ProcessObligationHandoffKey::new(&start.scope, &start.capability_id); + let mut active = + self.active_process_handoffs + .lock() + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff registry unavailable".to_string(), + })?; + if let Some(existing_process_id) = active.get(&key) { + return Err(ProcessError::InvalidStoredRecord { + reason: format!( + "process obligation handoff already active for scoped capability: {existing_process_id}" + ), + }); + } + active.insert(key, start.process_id); + Ok(true) + } + + fn release_claimed_process_handoff( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + process_id: ProcessId, + ) -> Result<(), ProcessError> { + let key = ProcessObligationHandoffKey::new(scope, capability_id); + let mut active = + self.active_process_handoffs + .lock() + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff registry unavailable".to_string(), + })?; + if active.get(&key) == Some(&process_id) { + active.remove(&key); + } + Ok(()) + } + + fn release_active_process_handoff(&self, record: &ProcessRecord) -> Result<(), ProcessError> { + self.release_claimed_process_handoff( + &record.scope, + &record.capability_id, + record.process_id, + ) + } + + fn has_active_process_handoff(&self, record: &ProcessRecord) -> Result { + let key = ProcessObligationHandoffKey::new(&record.scope, &record.capability_id); + let active = + self.active_process_handoffs + .lock() + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff registry unavailable".to_string(), + })?; + Ok(active.get(&key) == Some(&record.process_id)) + } + + fn process_handoff_cleaned(&self, record: &ProcessRecord) -> Result { + let key = ProcessObligationProcessKey::new(&record.scope, record.process_id); + let cleaned = self.cleaned_process_handoffs.lock().map_err(|_| { + ProcessError::InvalidStoredRecord { + reason: "process obligation cleanup registry unavailable".to_string(), + } + })?; + Ok(cleaned.contains(&key)) + } + + fn mark_process_handoff_cleaned(&self, record: &ProcessRecord) -> Result<(), ProcessError> { + let key = ProcessObligationProcessKey::new(&record.scope, record.process_id); + let mut cleaned = self.cleaned_process_handoffs.lock().map_err(|_| { + ProcessError::InvalidStoredRecord { + reason: "process obligation cleanup registry unavailable".to_string(), + } + })?; + cleaned.insert(key); + Ok(()) + } + + fn has_staged_handoffs(&self, record: &ProcessRecord) -> Result { + let has_secret_handoff = self + .secret_injections + .has_for_capability(&record.scope, &record.capability_id) + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff lookup failed".to_string(), + })?; + Ok(self + .network_policies + .contains(&record.scope, &record.capability_id) + || has_secret_handoff) + } + + fn cleanup_terminal( + &self, + record: &ProcessRecord, + reconcile: bool, + ) -> Result<(), ProcessError> { + if let Err(error) = self.cleanup_record_obligations(record, reconcile) { + tracing::warn!( + process_id = %record.process_id, + tenant_id = %record.scope.tenant_id, + user_id = %record.scope.user_id, + reconcile, + error = %error, + "process obligation cleanup failed after terminal transition" + ); + return Err(error); + } + self.release_active_process_handoff(record)?; + self.mark_process_handoff_cleaned(record)?; + Ok(()) + } + + fn cleanup_record_obligations( + &self, + record: &ProcessRecord, + reconcile: bool, + ) -> Result<(), ProcessError> { + if self.process_handoff_cleaned(record)? { + return Ok(()); + } + let should_cleanup_handoffs = self.has_active_process_handoff(record)? + || record.resource_reservation_id.is_some() + || self.has_staged_handoffs(record)?; + if should_cleanup_handoffs { + self.network_policies + .discard_for_capability(&record.scope, &record.capability_id); + self.secret_injections + .discard_for_capability(&record.scope, &record.capability_id) + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process obligation handoff cleanup failed".to_string(), + })?; + } + if let Some(reservation_id) = record.resource_reservation_id { + let governor = + self.resource_governor + .lock() + .map_err(|_| ProcessError::InvalidStoredRecord { + reason: "process resource governor registry unavailable".to_string(), + })?; + if reconcile { + close_reservation_once( + governor.reconcile(reservation_id, ResourceUsage::default()), + )?; + } else { + close_reservation_once(governor.release(reservation_id))?; + } + } + Ok(()) + } +} + +#[async_trait] +impl ProcessJournalCommitObserver for ProcessObligationLifecycleStore { + fn process_observer_id(&self) -> &'static str { + "process-obligation-lifecycle-v1" + } + + async fn observe_process_commit(&self, commit: ProcessJournalCommit) -> Result<(), String> { + if commit.state.process_kind != ProcessKind::CapabilityInvocation { + return Ok(()); + } + let record = + process_record_from_snapshot(commit.state).map_err(|error| error.to_string())?; + match commit.kind { + ProcessJournalKind::Completed => { + self.emit_process_event(RuntimeEvent::process_completed( + record.scope.clone(), + record.capability_id.clone(), + record.extension_id.clone(), + record.runtime, + record.process_id, + )) + .await; + self.cleanup_terminal(&record, true) + .map_err(|error| error.to_string())?; + } + ProcessJournalKind::Failed => { + self.emit_process_event(RuntimeEvent::process_failed( + record.scope.clone(), + record.capability_id.clone(), + record.extension_id.clone(), + record.runtime, + record.process_id, + record + .error_kind + .clone() + .unwrap_or_else(|| "unknown".to_string()), + )) + .await; + self.cleanup_terminal(&record, false) + .map_err(|error| error.to_string())?; + } + ProcessJournalKind::Stopped + | ProcessJournalKind::Cancelled + | ProcessJournalKind::Killed + | ProcessJournalKind::RecoveryRequired => { + self.emit_process_event(RuntimeEvent::process_killed( + record.scope.clone(), + record.capability_id.clone(), + record.extension_id.clone(), + record.runtime, + record.process_id, + )) + .await; + self.cleanup_terminal(&record, false) + .map_err(|error| error.to_string())?; + } + _ => {} + } + Ok(()) + } +} + +#[async_trait] +impl ProcessSubmissionLifecycle for ProcessObligationLifecycleStore { + async fn before_submit(&self, start: &ProcessStart) -> Result<(), ProcessError> { + self.claim_active_process_handoff(start).map(|_| ()) + } + + async fn submit_failed(&self, start: &ProcessStart) -> Result<(), ProcessError> { + self.release_claimed_process_handoff(&start.scope, &start.capability_id, start.process_id) + } + + async fn submitted(&self, record: &ProcessRecord) -> Result<(), ProcessError> { + self.emit_process_event(RuntimeEvent::process_started( + record.scope.clone(), + record.capability_id.clone(), + record.extension_id.clone(), + record.runtime, + record.process_id, + )) + .await; + Ok(()) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct ProcessObligationHandoffKey { + tenant_id: String, + user_id: String, + agent_id: Option, + project_id: Option, + mission_id: Option, + thread_id: Option, + invocation_id: String, + capability_id: String, +} + +impl ProcessObligationHandoffKey { + fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { + Self { + tenant_id: scope.tenant_id.as_str().to_string(), + user_id: scope.user_id.as_str().to_string(), + agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), + project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), + mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), + thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), + invocation_id: scope.invocation_id.to_string(), + capability_id: capability_id.as_str().to_string(), + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct ProcessObligationProcessKey { + tenant_id: String, + user_id: String, + agent_id: Option, + project_id: Option, + mission_id: Option, + thread_id: Option, + process_id: ProcessId, +} + +impl ProcessObligationProcessKey { + fn new(scope: &ResourceScope, process_id: ProcessId) -> Self { + Self { + tenant_id: scope.tenant_id.as_str().to_string(), + user_id: scope.user_id.as_str().to_string(), + agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), + project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), + mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), + thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), + process_id, + } + } +} + +impl ProcessObligationLifecycleStore { + pub fn process_runtime(&self) -> Arc { + Arc::clone(&self.processes) + } + + pub async fn start(&self, start: ProcessStart) -> Result { + let claimed = self.claim_active_process_handoff(&start)?; + let process_id = start.process_id; + let scope = start.scope.clone(); + let capability_id = start.capability_id.clone(); + match submit_capability_process(self.processes.as_ref(), start).await { + Ok(record) => { + self.emit_process_event(RuntimeEvent::process_started( + record.scope.clone(), + record.capability_id.clone(), + record.extension_id.clone(), + record.runtime, + record.process_id, + )) + .await; + Ok(record) + } + Err(error) => { + if claimed { + self.release_claimed_process_handoff(&scope, &capability_id, process_id)?; + } + Err(error) + } + } + } + + pub async fn complete( + &self, + scope: &ResourceScope, + process_id: ProcessId, + ) -> Result { + let record = + complete_capability_process(self.processes.as_ref(), scope, process_id).await?; + self.cleanup_terminal(&record, true)?; + Ok(record) + } + + pub async fn fail( + &self, + scope: &ResourceScope, + process_id: ProcessId, + error_kind: String, + ) -> Result { + let record = + fail_capability_process(self.processes.as_ref(), scope, process_id, error_kind).await?; + self.cleanup_terminal(&record, false)?; + Ok(record) + } + + pub async fn kill( + &self, + scope: &ResourceScope, + process_id: ProcessId, + ) -> Result { + let result = self + .processes + .kill_process(ironclaw_processes::KillProcessRequest { + scope: scope.clone(), + process_id, + operation_id: None, + reason: None, + }) + .await + .map_err(|error| ProcessError::InvalidStoredRecord { + reason: error.to_string(), + })?; + let record = process_record_from_snapshot(result.state)?; + self.cleanup_terminal(&record, false)?; + Ok(record) + } + + pub async fn get( + &self, + scope: &ResourceScope, + process_id: ProcessId, + ) -> Result, ProcessError> { + capability_process_record(self.processes.as_ref(), scope, process_id).await + } + + pub async fn records_for_scope( + &self, + scope: &ResourceScope, + ) -> Result, ProcessError> { + self.processes + .process_snapshots(scope) + .await + .map_err(|error| ProcessError::InvalidStoredRecord { + reason: error.to_string(), + })? + .into_iter() + .filter(|snapshot| snapshot.process_kind == ProcessKind::CapabilityInvocation) + .map(process_record_from_snapshot) + .collect() + } +} + +fn close_reservation_once(result: Result) -> Result<(), ProcessError> { + match result { + Ok(_) => Ok(()), + Err(ResourceError::ReservationClosed { .. }) => Ok(()), + Err(ResourceError::UnknownReservation { .. }) => Ok(()), + Err(error) => Err(error.into()), + } +} diff --git a/crates/ironclaw_host_runtime/src/obligations/staged_handoffs.rs b/crates/ironclaw_host_runtime/src/obligations/staged_handoffs.rs new file mode 100644 index 00000000000..06c5d1d14de --- /dev/null +++ b/crates/ironclaw_host_runtime/src/obligations/staged_handoffs.rs @@ -0,0 +1,500 @@ +//! Staged secret and network handoffs — one of the three chartered owners of +//! the obligation module (PROPOSAL §6.5.9, CHECKLIST WS3). +//! +//! Obligation *preparation* stages material that a later, separate actor +//! consumes: one-shot runtime secret material and per-invocation network +//! policy. This module owns those staging stores and the credential-account +//! resolver port that feeds them. It performs no obligation orchestration +//! (see [`super::handler`]) and no process-lifecycle bookkeeping (see +//! [`super::process_store`]). + +use std::{ + collections::HashMap, + fmt, + sync::{Arc, Mutex}, + time::{Duration, Instant}, +}; + +use async_trait::async_trait; +use ironclaw_host_api::{ + Timestamp, + action::NetworkPolicy, + capability::RuntimeCredentialAccountSetup, + dispatch::CredentialStageError, + ids::{CapabilityId, ExtensionId, SecretHandle, VendorId}, + resource::ResourceScope, +}; +use ironclaw_secrets::{ + SecretLease, SecretLeaseId, SecretMaterial, SecretMetadata, SecretStoreError, SecretStorePort, +}; +use secrecy::ExposeSecret; + +/// Default maximum lifetime for one-shot runtime secret material staged in memory. +pub(crate) const DEFAULT_RUNTIME_SECRET_INJECTION_TTL: Duration = Duration::from_secs(300); + +#[derive(Debug)] +pub struct RuntimeCredentialAccountRequest<'a> { + pub scope: &'a ResourceScope, + pub provider: &'a VendorId, + pub setup: &'a RuntimeCredentialAccountSetup, + pub provider_scopes: &'a [String], + pub requester_extension: &'a ExtensionId, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RuntimeCredentialAccessSecret { + pub scope: ResourceScope, + pub handle: SecretHandle, +} + +#[async_trait] +pub trait RuntimeCredentialAccountResolver: Send + Sync + fmt::Debug { + /// Resolve the access-secret source for the requested product-auth account. + /// + /// Returns [`CredentialStageError::AuthRequired`] when the account is + /// missing/unconfigured/expired/revoked (user must re-authenticate), or + /// [`CredentialStageError::Backend`] for internal failures not attributable + /// to user credentials. Shares its error vocabulary with the rest of the + /// staged-credential surface (`ProductAuthCredentialStageError`, + /// `GsuiteCredentialStageError`) so no per-layer error mapping is needed. + async fn resolve_access_secret( + &self, + request: RuntimeCredentialAccountRequest<'_>, + ) -> Result; +} + +/// Runtime secret material staged after `InjectSecretOnce` lease consumption. +/// +/// The store is keyed by scoped invocation, capability, and handle. Runtime adapters +/// borrow staged material during dispatch; `complete_dispatch`/`abort` removes it +/// after the scoped capability finishes. Entries also expire after a short TTL so +/// abandoned handoffs from setup failures, cancellation, or adapter bugs cannot +/// remain usable indefinitely. +#[derive(Clone)] +pub(crate) struct RuntimeSecretInjectionStore { + state: Arc, +} + +struct RuntimeSecretInjectionState { + secrets: Mutex>, + ttl: Duration, +} + +struct RuntimeSecretInjectionEntry { + material: SecretMaterial, + expires_at: Instant, +} + +impl RuntimeSecretInjectionStore { + pub(crate) fn new() -> Self { + Self::default() + } + + pub(crate) fn with_ttl(ttl: Duration) -> Self { + Self { + state: Arc::new(RuntimeSecretInjectionState { + secrets: Mutex::new(HashMap::new()), + ttl, + }), + } + } + + pub(crate) fn insert( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + handle: &SecretHandle, + material: SecretMaterial, + ) -> Result<(), RuntimeSecretInjectionStoreError> { + let now = Instant::now(); + let expires_at = now.checked_add(self.state.ttl).unwrap_or(now); + let mut secrets = self.lock()?; + prune_expired_entries(&mut secrets, now); + secrets.insert( + RuntimeSecretInjectionKey::new(scope, capability_id, handle), + RuntimeSecretInjectionEntry { + material, + expires_at, + }, + ); + Ok(()) + } + + pub(crate) fn take( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + handle: &SecretHandle, + ) -> Result, RuntimeSecretInjectionStoreError> { + let now = Instant::now(); + let mut secrets = self.lock()?; + prune_expired_entries(&mut secrets, now); + Ok(secrets + .remove(&RuntimeSecretInjectionKey::new( + scope, + capability_id, + handle, + )) + .map(|entry| entry.material)) + } + + pub(crate) fn clone_material( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + handle: &SecretHandle, + ) -> Result, RuntimeSecretInjectionStoreError> { + let now = Instant::now(); + let mut secrets = self.lock()?; + prune_expired_entries(&mut secrets, now); + Ok(secrets + .get(&RuntimeSecretInjectionKey::new( + scope, + capability_id, + handle, + )) + .map(|entry| SecretMaterial::from(entry.material.expose_secret()))) + } + + /// Discard all staged secrets for a scoped capability before process ownership exists. + /// + /// Background process lifecycle cleanup is guarded by a single-active-handoff + /// invariant for the scoped capability; this method remains the abort/inline cleanup seam. + pub(crate) fn discard_for_capability( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + ) -> Result<(), RuntimeSecretInjectionStoreError> { + let scope_key = RuntimeSecretInjectionScopeKey::new(scope, capability_id); + let mut secrets = self.lock()?; + prune_expired_entries(&mut secrets, Instant::now()); + secrets.retain(|key, _| !key.matches_scope(&scope_key)); + Ok(()) + } + + pub(super) fn has_for_capability( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + ) -> Result { + let scope_key = RuntimeSecretInjectionScopeKey::new(scope, capability_id); + let mut secrets = self.lock()?; + prune_expired_entries(&mut secrets, Instant::now()); + Ok(secrets.keys().any(|key| key.matches_scope(&scope_key))) + } + + #[cfg(test)] + pub(super) fn prune_expired(&self) -> Result { + let mut secrets = self.lock()?; + Ok(prune_expired_entries(&mut secrets, Instant::now())) + } + + fn lock( + &self, + ) -> Result< + std::sync::MutexGuard<'_, HashMap>, + RuntimeSecretInjectionStoreError, + > { + self.state + .secrets + .lock() + .map_err(|_| RuntimeSecretInjectionStoreError::Unavailable) + } +} + +impl Default for RuntimeSecretInjectionStore { + fn default() -> Self { + Self::with_ttl(DEFAULT_RUNTIME_SECRET_INJECTION_TTL) + } +} + +impl fmt::Debug for RuntimeSecretInjectionStore { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("RuntimeSecretInjectionStore") + .field("secrets", &"[REDACTED]") + .field("ttl", &self.state.ttl) + .finish() + } +} + +fn prune_expired_entries( + secrets: &mut HashMap, + now: Instant, +) -> usize { + let before = secrets.len(); + secrets.retain(|_, entry| entry.expires_at > now); + before.saturating_sub(secrets.len()) +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum RuntimeSecretInjectionStoreError { + Unavailable, +} + +impl fmt::Display for RuntimeSecretInjectionStoreError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Unavailable => formatter.write_str("runtime secret injection store unavailable"), + } + } +} + +impl std::error::Error for RuntimeSecretInjectionStoreError {} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct RuntimeSecretInjectionKey { + tenant_id: String, + user_id: String, + agent_id: Option, + project_id: Option, + mission_id: Option, + thread_id: Option, + invocation_id: String, + capability_id: String, + handle: String, +} + +impl RuntimeSecretInjectionKey { + fn new(scope: &ResourceScope, capability_id: &CapabilityId, handle: &SecretHandle) -> Self { + Self { + tenant_id: scope.tenant_id.as_str().to_string(), + user_id: scope.user_id.as_str().to_string(), + agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), + project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), + mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), + thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), + invocation_id: scope.invocation_id.to_string(), + capability_id: capability_id.as_str().to_string(), + handle: handle.as_str().to_string(), + } + } + + fn matches_scope(&self, scope: &RuntimeSecretInjectionScopeKey) -> bool { + self.tenant_id == scope.tenant_id + && self.user_id == scope.user_id + && self.agent_id == scope.agent_id + && self.project_id == scope.project_id + && self.mission_id == scope.mission_id + && self.thread_id == scope.thread_id + && self.invocation_id == scope.invocation_id + && self.capability_id == scope.capability_id + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct RuntimeSecretInjectionScopeKey { + tenant_id: String, + user_id: String, + agent_id: Option, + project_id: Option, + mission_id: Option, + thread_id: Option, + invocation_id: String, + capability_id: String, +} + +impl RuntimeSecretInjectionScopeKey { + fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { + Self { + tenant_id: scope.tenant_id.as_str().to_string(), + user_id: scope.user_id.as_str().to_string(), + agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), + project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), + mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), + thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), + invocation_id: scope.invocation_id.to_string(), + capability_id: capability_id.as_str().to_string(), + } + } +} + +/// In-memory policy handoff from obligation handling to runtime adapters. +/// +/// Policies are keyed by tenant/user/project/mission/thread/invocation scope and +/// capability id. Runtime adapters and host egress borrow the staged policy for +/// every network operation in the invocation; obligation completion/abort or +/// process lifecycle cleanup owns the final discard. +#[derive(Debug, Clone, Default)] +pub(crate) struct NetworkObligationPolicyStore { + policies: Arc>>, +} + +impl NetworkObligationPolicyStore { + pub(crate) fn new() -> Self { + Self::default() + } + + pub(crate) fn insert( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + policy: NetworkPolicy, + ) { + self.policies + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .insert(NetworkPolicyKey::new(scope, capability_id), policy); + } + + pub(crate) fn get( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + ) -> Option { + self.policies + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .get(&NetworkPolicyKey::new(scope, capability_id)) + .cloned() + } + + pub(crate) fn take( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + ) -> Option { + self.policies + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .remove(&NetworkPolicyKey::new(scope, capability_id)) + } + + /// Discard a staged policy for a scoped capability before process ownership exists. + /// + /// Background process lifecycle cleanup is guarded by a single-active-handoff + /// invariant for the scoped capability; this method remains the abort/inline cleanup seam. + pub(crate) fn discard_for_capability( + &self, + scope: &ResourceScope, + capability_id: &CapabilityId, + ) { + let _ = self.take(scope, capability_id); + } + + pub(super) fn contains(&self, scope: &ResourceScope, capability_id: &CapabilityId) -> bool { + self.policies + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .contains_key(&NetworkPolicyKey::new(scope, capability_id)) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct NetworkPolicyKey { + tenant_id: String, + user_id: String, + agent_id: Option, + project_id: Option, + mission_id: Option, + thread_id: Option, + invocation_id: String, + capability_id: String, +} + +impl NetworkPolicyKey { + fn new(scope: &ResourceScope, capability_id: &CapabilityId) -> Self { + Self { + tenant_id: scope.tenant_id.as_str().to_string(), + user_id: scope.user_id.as_str().to_string(), + agent_id: scope.agent_id.as_ref().map(|id| id.as_str().to_string()), + project_id: scope.project_id.as_ref().map(|id| id.as_str().to_string()), + mission_id: scope.mission_id.as_ref().map(|id| id.as_str().to_string()), + thread_id: scope.thread_id.as_ref().map(|id| id.as_str().to_string()), + invocation_id: scope.invocation_id.to_string(), + capability_id: capability_id.as_str().to_string(), + } + } +} + +#[derive(Clone)] +pub(crate) struct SharedSecretStore(pub(crate) Arc); + +#[async_trait] +impl SecretStorePort for SharedSecretStore { + async fn put( + &self, + scope: ResourceScope, + handle: SecretHandle, + material: SecretMaterial, + expires_at: Option, + ) -> Result { + self.0.put(scope, handle, material, expires_at).await + } + + async fn metadata( + &self, + scope: &ResourceScope, + handle: &SecretHandle, + ) -> Result, SecretStoreError> { + self.0.metadata(scope, handle).await + } + + async fn metadata_for_scope( + &self, + scope: &ResourceScope, + ) -> Result, SecretStoreError> { + self.0.metadata_for_scope(scope).await + } + + async fn delete( + &self, + scope: &ResourceScope, + handle: &SecretHandle, + ) -> Result { + self.0.delete(scope, handle).await + } + + async fn lease_once( + &self, + scope: &ResourceScope, + handle: &SecretHandle, + ) -> Result { + self.0.lease_once(scope, handle).await + } + + async fn consume( + &self, + scope: &ResourceScope, + lease_id: SecretLeaseId, + ) -> Result { + self.0.consume(scope, lease_id).await + } + + async fn revoke( + &self, + scope: &ResourceScope, + lease_id: SecretLeaseId, + ) -> Result { + self.0.revoke(scope, lease_id).await + } + + async fn leases_for_scope( + &self, + scope: &ResourceScope, + ) -> Result, SecretStoreError> { + self.0.leases_for_scope(scope).await + } +} + +/// **Finding H2 — compile-time regression guard.** +/// +/// The original H2 claim was that `RuntimeSecretInjectionStore`'s +/// `HashMap<_, RuntimeSecretInjectionEntry>` would bitwise-copy plaintext out +/// of the old bucket array on rehash and free it without zeroization. On +/// closer inspection that does *not* happen, because `SecretMaterial = +/// secrecy::SecretBox`: the rehash moves a `Box` pointer plus the +/// `Instant`, while the actual buffer stays at its original heap address +/// until `SecretBox::drop` zeroizes it. +/// +/// The protection is real but depends on the staged entry's `material` field +/// being a `ZeroizeOnDrop` carrier. If it ever swaps to a non-zeroizing type +/// (plain `String`, `Vec`, etc.), the bitwise-copy concern returns. This +/// `const _: fn(...) = ...` references the field through a +/// `ZeroizeOnDrop`-bounded helper, so the swap is rejected at compile time +/// rather than only failing a test run. The function is never called — only +/// type-checked. +const _: fn(&RuntimeSecretInjectionEntry) = |entry| { + fn require_zeroize_on_drop(_: &T) {} + require_zeroize_on_drop(&entry.material); +}; diff --git a/crates/ironclaw_host_runtime/src/obligations/tests.rs b/crates/ironclaw_host_runtime/src/obligations/tests.rs new file mode 100644 index 00000000000..2b5fa92fbde --- /dev/null +++ b/crates/ironclaw_host_runtime/src/obligations/tests.rs @@ -0,0 +1,650 @@ +//! Behavioural coverage for the three obligation owners. +//! +//! The suite is deliberately kept whole rather than partitioned by module: every +//! case here drives obligation handling *through* the staged-handoff stores and +//! the process-obligation store, which is the interaction the split must +//! preserve. + +use std::{sync::Arc, time::Duration}; + +use ironclaw_capabilities::{ + CapabilityObligationCompletionRequest, CapabilityObligationError, + CapabilityObligationFailureKind, CapabilityObligationHandler, CapabilityObligationPhase, + CapabilityObligationRequest, +}; +use ironclaw_events::InMemoryAuditSink; +use ironclaw_host_api::{ + action::{NetworkPolicy, NetworkScheme, NetworkTargetPattern}, + capability::CapabilitySet, + decision::Obligation, + dispatch::{CapabilityDispatchResult, CapabilityDisplayOutputPreview}, + ids::{ + AgentId, CapabilityId, CorrelationId, ExtensionId, InvocationId, ProjectId, + ResourceReservationId, SecretHandle, TenantId, UserId, + }, + mount::MountView, + resource::{ResourceEstimate, ResourceScope}, + runtime::{RuntimeKind, TrustClass}, + scope::ExecutionContext, +}; +use ironclaw_resources::{InMemoryResourceGovernor, ResourceAccount}; +use ironclaw_secrets::{SecretMaterial, SecretStore, SecretStorePort}; + +use super::*; + +#[tokio::test] +async fn runtime_secret_injection_store_prunes_expired_handoffs() { + let store = RuntimeSecretInjectionStore::with_ttl(Duration::from_millis(5)); + let scope = resource_scope_with_agent("agent-a"); + let capability_id = capability_id(); + let handle = SecretHandle::new("api_token").unwrap(); + + store + .insert( + &scope, + &capability_id, + &handle, + SecretMaterial::from("runtime-secret"), + ) + .unwrap(); + tokio::time::sleep(Duration::from_millis(20)).await; + + assert_eq!(store.prune_expired().unwrap(), 1); + assert!( + store + .take(&scope, &capability_id, &handle) + .unwrap() + .is_none() + ); +} + +#[test] +fn network_obligation_policy_store_isolates_agent_scope() { + let store = NetworkObligationPolicyStore::new(); + let (agent_a, agent_b) = same_invocation_agent_scopes(); + let capability_id = capability_id(); + + store.insert(&agent_a, &capability_id, allowed_network_policy()); + + assert!(store.take(&agent_b, &capability_id).is_none()); + assert!(store.take(&agent_a, &capability_id).is_some()); +} + +#[test] +fn runtime_secret_injection_store_isolates_agent_scope() { + let store = RuntimeSecretInjectionStore::new(); + let (agent_a, agent_b) = same_invocation_agent_scopes(); + let capability_id = capability_id(); + let handle = SecretHandle::new("api_token").unwrap(); + + store + .insert( + &agent_a, + &capability_id, + &handle, + SecretMaterial::from("runtime-secret"), + ) + .unwrap(); + + assert!( + store + .take(&agent_b, &capability_id, &handle) + .unwrap() + .is_none() + ); + assert!( + store + .take(&agent_a, &capability_id, &handle) + .unwrap() + .is_some() + ); +} + +#[tokio::test] +async fn builtin_obligation_handler_satisfy_release_preserves_staged_handoffs() { + let network_policies = Arc::new(NetworkObligationPolicyStore::new()); + let secret_injections = Arc::new(RuntimeSecretInjectionStore::new()); + let secret_store = Arc::new(SecretStore::ephemeral()); + let governor = Arc::new(InMemoryResourceGovernor::new()); + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + network_policies.clone(), + secret_store.clone(), + secret_injections.clone(), + governor.clone(), + ); + let handler = services.obligation_handler(); + let context = execution_context(); + let account = ResourceAccount::tenant(context.resource_scope.tenant_id.clone()); + let capability_id = capability_id(); + let handle = SecretHandle::new("api_token").unwrap(); + let estimate = ResourceEstimate::default().set_concurrency_slots(1); + secret_store + .put( + context.resource_scope.clone(), + handle.clone(), + SecretMaterial::from("runtime-secret"), + None, + ) + .await + .unwrap(); + let obligations = vec![ + Obligation::ApplyNetworkPolicy { + policy: allowed_network_policy(), + }, + Obligation::InjectSecretOnce { + handle: handle.clone(), + }, + Obligation::ReserveResources { + reservation_id: ResourceReservationId::new(), + }, + ]; + + handler + .satisfy(CapabilityObligationRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + }) + .await + .unwrap(); + + assert_eq!(governor.reserved_for(&account).concurrency_slots, 0); + assert!( + network_policies + .take(&context.resource_scope, &capability_id) + .is_some() + ); + assert!( + secret_injections + .take(&context.resource_scope, &capability_id, &handle) + .unwrap() + .is_some() + ); +} + +// #5459 tenant-shared credential resolution: a caller's own secret wins; +// otherwise the tenant-shared admin-managed scope; otherwise absent. +#[tokio::test] +async fn secret_owner_scope_prefers_caller_then_tenant_shared_then_none() { + let handle = SecretHandle::new("market_data_api_key").unwrap(); + let caller = execution_context().resource_scope; + let shared = caller.tenant_shared_managed_scope(); + + // Absent in both scopes -> None (dispatch then gates with AuthRequired). + let store = SecretStore::ephemeral(); + assert_eq!( + secret_owner_scope(&store, &caller, &handle).await.unwrap(), + None, + ); + + // Present ONLY at the tenant-shared admin-managed scope -> resolves there, + // so one admin-set key satisfies a caller who never provisioned it. + let store = SecretStore::ephemeral(); + store + .put( + shared.clone(), + handle.clone(), + SecretMaterial::from("shared-admin-key"), + None, + ) + .await + .unwrap(); + assert_eq!( + secret_owner_scope(&store, &caller, &handle) + .await + .unwrap() + .as_ref(), + Some(&shared), + ); + + // Present at BOTH scopes -> the caller's OWN secret wins over the shared one. + let store = SecretStore::ephemeral(); + store + .put( + caller.clone(), + handle.clone(), + SecretMaterial::from("caller-own-key"), + None, + ) + .await + .unwrap(); + store + .put( + shared.clone(), + handle.clone(), + SecretMaterial::from("shared-admin-key"), + None, + ) + .await + .unwrap(); + assert_eq!( + secret_owner_scope(&store, &caller, &handle) + .await + .unwrap() + .as_ref(), + Some(&caller), + ); +} + +// Through the caller: InjectSecretOnce is satisfied by an admin-set +// tenant-shared key even when the caller has no personal secret, and the +// material is staged at the caller's own invocation slot (#5459). +#[tokio::test] +async fn inject_secret_once_falls_back_to_tenant_shared_admin_key() { + let secret_store = Arc::new(SecretStore::ephemeral()); + let secret_injections = Arc::new(RuntimeSecretInjectionStore::new()); + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + Arc::new(NetworkObligationPolicyStore::new()), + secret_store.clone(), + secret_injections.clone(), + Arc::new(InMemoryResourceGovernor::new()), + ); + let handler = services.obligation_handler(); + let context = execution_context(); + let capability_id = capability_id(); + let handle = SecretHandle::new("market_data_api_key").unwrap(); + let estimate = ResourceEstimate::default(); + + // Admin set the key ONLY at the tenant-shared scope; the caller has none. + secret_store + .put( + context.resource_scope.tenant_shared_managed_scope(), + handle.clone(), + SecretMaterial::from("shared-admin-key"), + None, + ) + .await + .unwrap(); + + let obligations = vec![Obligation::InjectSecretOnce { + handle: handle.clone(), + }]; + handler + .satisfy(CapabilityObligationRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + }) + .await + .expect( + "tenant-shared key satisfies InjectSecretOnce for a caller with no personal secret", + ); + + assert!( + secret_injections + .take(&context.resource_scope, &capability_id, &handle) + .unwrap() + .is_some(), + "shared-sourced secret must be staged at the caller's own invocation slot", + ); +} + +#[tokio::test] +async fn redact_output_clears_display_preview_side_channel() { + use ironclaw_host_api::{ + resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, + runtime::RuntimeKind, + }; + + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + Arc::new(NetworkObligationPolicyStore::new()), + Arc::new(SecretStore::ephemeral()), + Arc::new(RuntimeSecretInjectionStore::new()), + Arc::new(InMemoryResourceGovernor::new()), + ); + let handler = services.obligation_handler(); + let context = execution_context(); + let capability_id = capability_id(); + let estimate = ResourceEstimate::default(); + let obligations = vec![Obligation::RedactOutput]; + let dispatch = CapabilityDispatchResult { + capability_id: capability_id.clone(), + provider: context.extension_id.clone(), + runtime: RuntimeKind::Wasm, + output: serde_json::json!({"secret": "sk-secret", "safe": "ok"}), + display_preview: Some(CapabilityDisplayOutputPreview { + output_summary: Some("contains secret".to_string()), + output_preview: "sk-secret".to_string(), + output_kind: "text".to_string(), + subtitle: None, + truncated: false, + }), + usage: ResourceUsage::default(), + receipt: ResourceReceipt { + id: ResourceReservationId::new(), + scope: context.resource_scope.clone(), + status: ReservationStatus::Released, + estimate: ResourceEstimate::default(), + actual: None, + }, + }; + + let completed = handler + .complete_dispatch(CapabilityObligationCompletionRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + dispatch: &dispatch, + }) + .await + .expect("redacted dispatch completes"); + + assert!(completed.display_preview.is_none()); + assert_eq!(completed.output["safe"], serde_json::json!("ok")); +} + +#[tokio::test] +async fn complete_dispatch_extracts_base64_document_into_text() { + use base64::Engine as _; + use ironclaw_host_api::{ + resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, + runtime::RuntimeKind, + }; + + // Drive the *caller* (`complete_dispatch`), not the helper: a dispatch + // result carrying `content_base64` + `mime_type` must come back with the + // extracted text in `content` and no base64 left for the model to see. + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + Arc::new(NetworkObligationPolicyStore::new()), + Arc::new(SecretStore::ephemeral()), + Arc::new(RuntimeSecretInjectionStore::new()), + Arc::new(InMemoryResourceGovernor::new()), + ); + let handler = services.obligation_handler(); + let context = execution_context(); + // The document-extraction transform is capability-gated, so this test + // must dispatch a capability that opts in (`google-drive.download_file`) + // — the shared `echo.say` helper id would pass through untouched. + let capability_id = CapabilityId::new("google-drive.download_file").unwrap(); + let estimate = ResourceEstimate::default(); + let obligations = vec![Obligation::RedactOutput]; + let encoded = base64::engine::general_purpose::STANDARD.encode(b"name,age\nAlice,30"); + let dispatch = CapabilityDispatchResult { + capability_id: capability_id.clone(), + provider: context.extension_id.clone(), + runtime: RuntimeKind::Wasm, + output: serde_json::json!({ + "file_id": "f1", + "name": "data.csv", + "mime_type": "text/csv", + "content_base64": encoded, + }), + display_preview: None, + usage: ResourceUsage::default(), + receipt: ResourceReceipt { + id: ResourceReservationId::new(), + scope: context.resource_scope.clone(), + status: ReservationStatus::Released, + estimate: ResourceEstimate::default(), + actual: None, + }, + }; + + let completed = handler + .complete_dispatch(CapabilityObligationCompletionRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + dispatch: &dispatch, + }) + .await + .expect("base64 document dispatch completes"); + + assert_eq!( + completed.output["content"], + serde_json::json!("name,age\nAlice,30") + ); + assert!( + completed.output.get("content_base64").is_none(), + "base64 must be stripped before the result reaches the model" + ); +} + +#[tokio::test] +async fn leak_detector_block_records_security_audit_event_through_complete_dispatch() { + use ironclaw_events::{ + InMemorySecurityAuditSink, SecurityAuditSink, SecurityBoundary, SecurityDecision, + }; + use ironclaw_host_api::{ + resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, + runtime::RuntimeKind, + }; + + // Build a handler with both an audit sink (unused here — we hit the + // redact branch, not the AuditAfter branch) and a recording + // security-audit sink. Other backing stores are not exercised by + // the redact-only path, but the handler requires them to be set + // for safety; we install minimal in-memory ones. + let security_sink: Arc = Arc::new(InMemorySecurityAuditSink::new()); + let security_sink_dyn: Arc = security_sink.clone(); + + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + Arc::new(NetworkObligationPolicyStore::new()), + Arc::new(SecretStore::ephemeral()), + Arc::new(RuntimeSecretInjectionStore::new()), + Arc::new(InMemoryResourceGovernor::new()), + ); + let handler = services + .obligation_handler() + .with_security_audit_sink(security_sink_dyn); + + let context = execution_context(); + let capability_id = capability_id(); + let estimate = ResourceEstimate::default(); + let obligations = vec![Obligation::RedactOutput]; + + // An AWS access-key shaped string is a built-in BLOCK pattern in + // `ironclaw_safety::LeakDetector` (`AKIA[0-9A-Z]{16}`). Per the + // module invariant we drive the *caller* (`complete_dispatch`), + // not the helper, and assert the recorded event: + // - boundary == LeakDetector + // - decision == Blocked + // - code == LEAK_REDACT_FAILED_CODE + // - capability_id + scope are populated + // - no payload (the offending string never appears in the event) + let leaky_payload = serde_json::Value::String("hello AKIAIOSFODNN7EXAMPLE goodbye".to_string()); + let dispatch = CapabilityDispatchResult { + capability_id: capability_id.clone(), + provider: context.extension_id.clone(), + runtime: RuntimeKind::Wasm, + output: leaky_payload, + display_preview: None, + usage: ResourceUsage::default(), + receipt: ResourceReceipt { + id: ResourceReservationId::new(), + scope: context.resource_scope.clone(), + status: ReservationStatus::Released, + estimate: ResourceEstimate::default(), + actual: None, + }, + }; + + let request = CapabilityObligationCompletionRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + dispatch: &dispatch, + }; + + let result = handler.complete_dispatch(request).await; + assert!( + matches!( + result, + Err(CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Output + }) + ), + "expected output-obligation failure, got {result:?}" + ); + + let events = security_sink.snapshot(); + assert_eq!( + events.len(), + 1, + "exactly one boundary decision should have been recorded, got {events:?}" + ); + let event = &events[0]; + assert_eq!(event.boundary, SecurityBoundary::LeakDetector); + assert_eq!(event.decision, SecurityDecision::Blocked); + assert_eq!(event.code, LEAK_REDACT_FAILED_CODE); + assert_eq!(event.code, "leak_redact_failed"); // stability lock + assert_eq!(event.capability_id.as_ref(), Some(&capability_id)); + assert_eq!(event.scope.as_ref(), Some(&context.resource_scope)); + + // The `SecurityAuditEvent` shape has no free-form payload field. + // That invariant is enforced at the type level by the absence of + // a `String` member on the struct. The check below is therefore a + // documentation-only assertion: it locks the field set at the + // value level for future readers, but the real guard is the type + // shape in `ironclaw_events::security_audit`. + // + // pub struct SecurityAuditEvent { + // pub boundary: SecurityBoundary, + // pub decision: SecurityDecision, + // pub capability_id: Option, + // pub scope: Option, + // pub timestamp: SystemTime, + // pub code: &'static str, + // } +} + +#[tokio::test] +async fn leak_detector_block_without_security_sink_does_not_panic() { + use ironclaw_host_api::{ + resource::{ReservationStatus, ResourceReceipt, ResourceUsage}, + runtime::RuntimeKind, + }; + + let services = BuiltinObligationServices::with_handoff_stores( + Arc::new(InMemoryAuditSink::new()), + Arc::new(NetworkObligationPolicyStore::new()), + Arc::new(SecretStore::ephemeral()), + Arc::new(RuntimeSecretInjectionStore::new()), + Arc::new(InMemoryResourceGovernor::new()), + ); + // No `.with_security_audit_sink(...)` — confirms the sink is + // optional and the original failure semantics are preserved. + let handler = services.obligation_handler(); + + let context = execution_context(); + let capability_id = capability_id(); + let estimate = ResourceEstimate::default(); + let obligations = vec![Obligation::RedactOutput]; + let dispatch = CapabilityDispatchResult { + capability_id: capability_id.clone(), + provider: context.extension_id.clone(), + runtime: RuntimeKind::Wasm, + output: serde_json::Value::String("leak AKIAIOSFODNN7EXAMPLE".to_string()), + display_preview: None, + usage: ResourceUsage::default(), + receipt: ResourceReceipt { + id: ResourceReservationId::new(), + scope: context.resource_scope.clone(), + status: ReservationStatus::Released, + estimate: ResourceEstimate::default(), + actual: None, + }, + }; + + let result = handler + .complete_dispatch(CapabilityObligationCompletionRequest { + phase: CapabilityObligationPhase::Invoke, + context: &context, + capability_id: &capability_id, + estimate: &estimate, + obligations: &obligations, + dispatch: &dispatch, + }) + .await; + assert!(matches!( + result, + Err(CapabilityObligationError::Failed { + kind: CapabilityObligationFailureKind::Output + }) + )); +} + +fn same_invocation_agent_scopes() -> (ResourceScope, ResourceScope) { + let mut agent_a = resource_scope_with_agent("agent-a"); + agent_a.invocation_id = InvocationId::new(); + let mut agent_b = agent_a.clone(); + agent_b.agent_id = Some(AgentId::new("agent-b").unwrap()); + (agent_a, agent_b) +} + +fn resource_scope_with_agent(agent_id: &str) -> ResourceScope { + ResourceScope { + tenant_id: TenantId::new("tenant1").unwrap(), + user_id: UserId::new("user1").unwrap(), + agent_id: Some(AgentId::new(agent_id).unwrap()), + project_id: Some(ProjectId::new("project1").unwrap()), + mission_id: None, + thread_id: None, + invocation_id: InvocationId::new(), + } +} + +fn execution_context() -> ExecutionContext { + let invocation_id = InvocationId::new(); + let resource_scope = ResourceScope { + tenant_id: TenantId::new("tenant1").unwrap(), + user_id: UserId::new("user1").unwrap(), + agent_id: Some(AgentId::new("agent-a").unwrap()), + project_id: Some(ProjectId::new("project1").unwrap()), + mission_id: None, + thread_id: None, + invocation_id, + }; + ExecutionContext { + run_id: None, + origin: None, + invocation_id, + correlation_id: CorrelationId::new(), + process_id: None, + parent_process_id: None, + tenant_id: resource_scope.tenant_id.clone(), + user_id: resource_scope.user_id.clone(), + authenticated_actor_user_id: None, + agent_id: resource_scope.agent_id.clone(), + project_id: resource_scope.project_id.clone(), + mission_id: resource_scope.mission_id.clone(), + thread_id: resource_scope.thread_id.clone(), + extension_id: ExtensionId::new("caller").unwrap(), + runtime: RuntimeKind::Wasm, + trust: TrustClass::Sandbox, + grants: CapabilitySet::default(), + mounts: MountView::default(), + resource_scope, + } +} + +fn capability_id() -> CapabilityId { + CapabilityId::new("echo.say").unwrap() +} + +fn allowed_network_policy() -> NetworkPolicy { + NetworkPolicy { + allowed_targets: vec![NetworkTargetPattern { + scheme: Some(NetworkScheme::Https), + host_pattern: "api.example.test".to_string(), + port: None, + }], + deny_private_ip_ranges: true, + max_egress_bytes: Some(1024), + } +} diff --git a/crates/ironclaw_host_runtime/src/services/builder.rs b/crates/ironclaw_host_runtime/src/services/builder.rs index adb1912efb8..07457eb4df3 100644 --- a/crates/ironclaw_host_runtime/src/services/builder.rs +++ b/crates/ironclaw_host_runtime/src/services/builder.rs @@ -511,7 +511,7 @@ where self } - pub fn with_credential_account_store(mut self, store: Arc) -> Self + pub(crate) fn with_credential_account_store(mut self, store: Arc) -> Self where T: CredentialAccountStore + 'static, { @@ -520,7 +520,7 @@ where self } - pub fn with_credential_session_store(mut self, store: Arc) -> Self + pub(crate) fn with_credential_session_store(mut self, store: Arc) -> Self where T: CredentialSessionStore + 'static, { @@ -652,7 +652,7 @@ where /// Attaches the host HTTP egress shape required for production runtime /// adapters. The service must use staged network-policy handoffs and secret /// injection handoffs, not request-local/test policy fallback. - pub fn with_host_http_egress_service( + pub(crate) fn with_host_http_egress_service( mut self, runtime_http_egress: Arc>, ) -> Self diff --git a/docs/reborn/target-architecture/CHECKLIST.md b/docs/reborn/target-architecture/CHECKLIST.md index f3f75941c78..3de76596119 100644 --- a/docs/reborn/target-architecture/CHECKLIST.md +++ b/docs/reborn/target-architecture/CHECKLIST.md @@ -139,7 +139,20 @@ Conventions: every code item lands with its tests and its guidance updates in th - [ ] Move `host_runtime/first_party_tools/**` (http, shell, time, json, echo, schemas, outbound-delivery, memory tools, trigger management, skill management/url-install, trace_commons, spawn-subagent stub) into `extensions/ironclaw_extension_support/` via the existing `FirstPartyHandlerRegistrar` pattern; host_runtime keeps only the registrar port. - [ ] Create `lanes/ironclaw_sandbox` by merging `process_sandbox` (plan contract) + `host_runtime/sandbox_process/**` (Docker/broker/credential-firewall/CA) + the `scripts` Docker backend; delete `ironclaw_scripts` and `ironclaw_process_sandbox`; route all process spawning through the transport seam (fixing scripts' direct `std::process` bypass). No production behavior change (all pieces currently unwired/test-only — re-verify at land time). -- [ ] Split `host_runtime/obligations.rs` internally into its three chartered owners (obligation handling ∣ staged secret/network handoffs ∣ process-obligation store); shrink `services/builder.rs`+`production_wiring` toward composition-facing factories. +- [~] Split `host_runtime/obligations.rs` internally into its three chartered owners (obligation handling ∣ staged secret/network handoffs ∣ process-obligation store); shrink `services/builder.rs`+`production_wiring` toward composition-facing factories. + ✎ **Amended 2026-08-03 (WS3 obligations/builder PR) — the split half is DONE; the builder half is measured, partly executed, and partly refuted.** + - **The three-way split landed as written, and it is now held by a gate rather than by discipline.** `obligations.rs` (**3,122** lines at `0f897e9366` — PROPOSAL §2.4's "3,097" is stale by 25) became `obligations/{mod,handler,staged_handoffs,process_store,tests}.rs` at 217 / 1,276 / 500 / 586 / 650 lines. The point of those numbers is the file's first line: it carried `// arch-exempt: large_file, canonical obligation orchestration remains co-located` — a standing waiver against `scripts/pre-commit-safety.sh`'s 1,500-line threshold. **Every module is now under that threshold, so the waiver was deleted rather than carried forward**, and re-fusing the owners re-trips the gate. `mod obligations;` is private in `lib.rs` and the crate's seven `pub use obligations::{…}` names are unchanged, so **no consumer outside `ironclaw_host_runtime` sees this at all**. + - **The measured cost of the split is three visibility widenings, and it is `pub(super)`, not `pub(crate)`.** `RuntimeSecretInjectionStore::{has_for_capability, prune_expired}` and `NetworkObligationPolicyStore::contains` were file-private and are read by the process store (a sibling module now) and the test module. `pub(super)` from inside `staged_handoffs` reaches `obligations` and its descendants and stops there — the crate cannot see them, so widening further stays a deliberate edit. Recorded in the crate's `CLAUDE.md` as the rule for future cross-owner access. The split also *revealed* one narrowing in the other direction: `secret_present` was `pub(crate)` with no caller outside its own file and is now private. + - **Test accounting, un-masking discipline: 1,246 → 1,246, name-by-name identical.** `cargo test -p ironclaw_host_runtime --all-targets -- --list`, unfiltered, sorted and `comm`-diffed on a quiescent tree: zero added, zero removed, zero renamed. The `#[cfg(test)] mod tests` block moved whole to `obligations/tests.rs` rather than being partitioned by owner — every case there drives the handler *through* the staging stores and the process store, which is precisely the interaction the split must not break, so partitioning it would have weakened the coverage the split needs. Its `use super::*` inheritance of the parent module's imports had to become explicit imports; no test body changed. + - **Zero `LAYER_MATRIX_EXCEPTIONS` moved, and that is the expected result** — the register is 10 before and after (counted with Python between `const LAYER_MATRIX_EXCEPTIONS` and its closing `];`; the file holds **15** `LayerMatrixException {` occurrences in total, the extra five being the struct definition and four test fixtures). An intra-crate module split changes no crate's layer and no crate's manifest, so the register cannot see it. Same lesson as Wave 2's item 4, in a different shape. + - **Two path-keyed gates fired, both loud, both repointed rather than re-baselined.** `reborn_struct_test_support_ratchet.rs` freezes test-support/dead-code member counts *per file path*, and its `crates/ironclaw_host_runtime/src/obligations.rs` entry (count 1) became a new-occurrence report against `obligations/staged_handoffs.rs` the moment the method travelled. The entry was **repointed with its count unchanged at 1** — a split must not be a re-baselining opportunity. Worth recording as a class alongside the gate below: a per-file frozen count is a path-keyed gate too, and an intra-crate module split trips it exactly like a cross-crate move would. + - **One path-keyed gate fired, and it is a data point for WS10's loud-inventory row.** `reborn_host_runtime_services_do_not_expose_lower_substrate_handles` read `crates/ironclaw_host_runtime/src/obligations.rs` by name and panicked on `NotFound` the moment the file became a directory — i.e. it is in the **loud** half of the WS10 inventory, and it behaved as designed. It was **repointed, not relaxed**: the same forbidden-pattern set now scans every `.rs` under `src/obligations/`, and `collect_runtime_rs` returns a file count so the call site can assert it read ≥ 4 files. Both of that helper's callers (this gate and the CLI-runtime scan) now assert non-zero, closing the "walks an empty directory, reports success" shape for the helper rather than for this one gate. Worth noting for WS7 planning: the loud half's cost is one repoint per move, which is cheap — the expensive half remains the silent gates #6996 closed. + - **`+production_wiring` is refuted: it is already what the row asks for.** `services/production_wiring.rs` (372 lines) declares `ProductionWiringConfig`/`Component`/`IssueKind`/`Issue`/`Report` and the component-type classifier — a *diagnostic* vocabulary that composition consumes through `RebornReadinessDiagnostic::from_production_wiring_report` (`readiness.rs:312`). There is no assembly in it to shrink into a factory. The row should not have paired it with `builder.rs`. + - **`shrink services/builder.rs` — 3 of 50 executed; the rest is a cargo-feature question and a redesign, both out of scope for a split PR.** `builder.rs` is 887 lines and 50 public `with_*`/`try_with_*` methods over `HostRuntimeServices`, with **602** call sites in **45** files that name the type (`ironclaw_host_runtime` 520, `ironclaw_reborn_composition` 35, root `tests/` 34, `ironclaw_extension_manager` 10, `ironclaw_extension_host` 2, `ironclaw_runner` 1). Dispositions: + - **Executed — 3 narrowed `pub` → `pub(crate)`**: `with_credential_account_store` and `with_credential_session_store` (called only by `with_credential_broker`, three lines below them in the same file) and `with_host_http_egress_service` (called by `try_with_host_http_egress_internal` and one in-`src` test). The composition-facing surface is 3 methods smaller and nothing outside the crate moved. + - **Not narrowable, and the reason is a finding: 3 more are src-only because they are *callerless*.** `with_postgres_root_filesystem` and `with_reborn_event_stores` have **zero** callers anywhere in the tree (the sole textual hit for the latter is a `docs/plans/` note about a *different* method, `with_reborn_event_stores_verified`), and `HostRuntimeServices::with_trust_policy_dyn` has zero — the call at `services.rs:764` is `DefaultHostRuntime`'s same-named method (`production.rs:223`), which is what makes this one look live in a grep. `pub(crate)` on a callerless method is `dead_code` and fails `-D warnings`, so the correct disposition is deletion under WS8's un-masking discipline, not a visibility edit inside a split PR (principles 2 and 4). Filed as an issue against the WS8 "Modules" row. + - **Deferred with measurement — 17 methods are crate-only but *integration-test*-visible.** Their only callers outside `src` are in `crates/ironclaw_host_runtime/tests/**`, which links the crate as an external consumer, so narrowing them requires a `test-support` cargo feature. `.claude/rules/cargo-features.md` sanctions that name (bar #4, dev-only seam) but it is a build-shape change, not a narrowing, and it belongs with whoever owns the crate's test seam. `with_script_runtime` alone accounts for 38 of those call sites. + - **The remaining 33 are the actual composition-facing set**, and turning a 50-method fluent chain into "composition-facing factories" is a redesign of that surface, not a shrink of it — the same shape as, and the same reason as, the runner-sheds lane's deferred `build_*` clause (WS4). Sized here so the next slice starts from evidence rather than from the row's one-line phrasing. - [ ] Move host_runtime's extension binding/catalog-default logic → `extension_host` (§6.5.9). - [ ] `mcp` drops the registry dep (consume `extension_contracts`); confirm the estimate/usage vocabulary it needs lives in `host_api::resource`. ✎ **Annotated 2026-07-31 (#6930) — the flip target is unchanged; the payload under it grew.** Re-verified at `2e6522580`: the import list is byte-identical — `use ironclaw_extensions::{ExtensionPackage, ExtensionRuntime, HostedMcpDiscoveredTool, HostedMcpDiscoveredToolAnnotations};` (`crates/ironclaw_mcp/src/lib.rs:20-22`) — so the four DTOs this row hands to `extension_contracts` are still exactly four, and Wave 1's `mcp → extensions` exception annotations stand as written. What changed is their **shape** and their **company**: `ExtensionPackage` swapped `root: VirtualPath` for `root_binding: PackageRootBinding` (`crates/ironclaw_extensions/src/package.rs:20`, enum at `resolved.rs:39`) and `HostedMcpDiscoveredToolAnnotations` gained three fields (`hosted_mcp_discovery.rs:27,31,32`), so the carve-out moves more surface than the name count suggests; and the lane picked up a **second** hosted-MCP vocabulary source, `host_api::hosted_mcp::McpAuthChallenge` (`lib.rs:27`). Plan the flip for two contracts modules, not one — and note that `host_api::hosted_mcp` is itself mutually bound to `package_lifecycle` (PROPOSAL §6.1.1), so where it lands is decided by the WS1 `extension_contracts`/`product_contracts` slots, not here. - [ ] Re-layer `processes` → kernel. Internal `capabilities/host.rs` split along its six workflows (module charter only). diff --git a/docs/reborn/target-architecture/PROPOSAL.md b/docs/reborn/target-architecture/PROPOSAL.md index e0e4898dfcd..db9a78e40e6 100644 --- a/docs/reborn/target-architecture/PROPOSAL.md +++ b/docs/reborn/target-architecture/PROPOSAL.md @@ -80,7 +80,7 @@ The four are still the four — but two of them **shrank** and two **grew**, and - **`ironclaw_reborn_composition`** (✎ **68.3k** lines, was 77.0k): still ~30% assembly / ~70% behavior. ✎ The tree formerly called `runtime/local_dev/**` is now `runtime/capability_host/**` + `capability_authorization` + `runtime_mounts` (#6691 retired the misnomer in the module names and the typename ratchet — CHECKLIST WS6 item, landed); it is ~11.6k lines and still the **production** capability/delivery/approval/skill path — `RuntimeSubstrate` still has only `None|ProductionShaped` variants and `runtime.rs:3016` still hardcodes `let local_runtime = Some(&services);` (the local *variable* name survived the rename). Behavior inventory still resident, with owners named in §9: approval/authorization policy (`capability_authorization`), trigger poller + trusted submit (~4.3k across `automation/trigger_poller*`, `trigger_fire_access.rs`, `trigger_creation_assembly.rs`), admin-user directory (322), trace capture (1.2k + a 350-line hooks projection), system-prompt content (`root/default_system_prompt.rs`), HTTP route mounts (`llm_admin/{openai_compat_serve,nearai_login_serve}.rs`), project filesystem reader (453), blocked-auth resume fan-out, Google OAuth secret store (155), NEAR-AI MCP (336). §6.10.1 carries the eviction reconciliation. `build_reborn_services` — still cited by `crates/Architecture.md` — **does not exist**. - **`ironclaw_extension_host`** (✎ **50.7k** lines, ✎ 88 files, was 45.5k/78, no guidance file): a generic lifecycle/binding core (#6116) + channel ingress router/verifier (vendor-blind, manifest-recipe driven — the system's cleanest seam) + delivery/egress transports + the product-serve wiring landed from composition (#6616/#6669) + non-extension strays (`skill_learning.rs`, `bundled_skills.rs`+`build.rs`, Axum pairing routes). ✎ It grew again in #6691, which handed it the generic bundled-skill tree migration and composition's extension capability surface — more evidence for the §6.8.3 split, not less. Unpoliced `include_str!` edges into `../ironclaw_first_party_extensions/assets/…`. ✎ **Re-measured 2026-07-31 at `2e6522580`: 56,940 lines / 93 files** — of which #6930 contributed **+3,367 / +4 files**, a whole new sub-owner: the hosted-MCP *registration pipeline* (`hosted_mcp_admission.rs`, `hosted_mcp_manifest.rs`, `hosted_mcp_preparation.rs` — declared private at `lib.rs:60,62,63` — plus the public `mcp_catalog_safety.rs` at `:74`, beside the pre-existing `hosted_mcp_discovery_authority.rs`). See §6.8.2 for what that means for the split. ✎ **Re-measured 2026-08-02 at `3be5f056e` (Wave 2 truth audit): 48,791 lines / 75 files** — WS2.4 carved `ironclaw_extension_manager` out of it (§6.8.3), so this entry's peak figure of 56,940/93 is now the *pre-split* high-water mark, not the current state. It is still a god crate by mass and still first in this list; the split took the product face, not the bulk. -- **`ironclaw_host_runtime`** (✎ **48.5k** lines, was 46.5k; 31 internal deps incl. `libsql`, `deadpool-postgres`, `bollard`, `rcgen`): kernel pipeline construction (sole `CapabilityHost` construction site) + closed lane executor + mediated egress/secret staging/obligations (✎ `obligations.rs` 3,097, fuses three owners) + `first_party_tools/` (incl. the `trace_commons` product feature) + pure assembly + an unwired Docker/CA sandbox subsystem ("ships unwired until W6"). +- **`ironclaw_host_runtime`** (✎ **48.5k** lines, was 46.5k; 31 internal deps incl. `libsql`, `deadpool-postgres`, `bollard`, `rcgen`): kernel pipeline construction (sole `CapabilityHost` construction site) + closed lane executor + mediated egress/secret staging/obligations (✎ `obligations.rs` 3,097, fuses three owners — ✎ **superseded 2026-08-03 (WS3 obligations/builder PR): the file was 3,122 lines at `0f897e9366` and no longer exists.** It is `obligations/{mod,handler,staged_handoffs,process_store,tests}.rs` at 217/1,276/500/586/650, one module per chartered owner, and the `arch-exempt: large_file` waiver it carried is deleted rather than moved — every module is under the 1,500-line gate. The remaining god-module in this crate is `first_party_tools/`, which the WS3 first-party row owns.) + `first_party_tools/` (incl. the `trace_commons` product feature) + pure assembly + an unwired Docker/CA sandbox subsystem ("ships unwired until W6"). - **`ironclaw_runner`** (✎ **33.1k** lines, was 38.3k): ✎ #6696 took the scheduler — `turn_scheduler` fell from 2,003 lines to **292**, and what remains is explicitly "an agent-turn projection over the generic process supervisor" (`ironclaw_processes::ProcessSupervisor`). ✎ The `subagent/` tree fell 7.7k → 4.9k, but **`subagent/await_edge/` survives at 2.9k** — the await-edge machinery was reworked, not deleted (see §2.7 and §6.7.3). Still resident: a ✎ 28-`with_*` loop-host factory and `runtime.rs` `build_*` functions composition reaches into. ✎ **The model gateway (3.9k) and tool disclosure (5.6k) are gone as of 2026-08-03 (WS3 runner sheds) — both moved to `ironclaw_loop_host`, taking `model_routes`, the driver-host port adapters, and the crate's whole `ironclaw_llm`/`ironclaw_common`/`base64`/`jsonschema` dependency set with them. The crate is **22.0k**, was 33.2k; `ironclaw_loop_host` is **49.7k**, was 38.5k.** ### 2.5 Contract-vocabulary pooling (`CURRENT`) @@ -578,7 +578,7 @@ Compact entries (all: layer `substrates`; forbidden = anything ≥ kernel unless - **6.5.6 `ironclaw_capabilities`** — retain. The caller-facing authority path: `CapabilityHost` (concrete struct; 6 workflows invoke/resume/auth-resume/decline/resume-spawn/spawn), the authorization fold, obligation seams (`CapabilityObligationHandler`), replay-payload store, process re-mint port, and `RuntimeDispatcher` (the sole `CapabilityDispatcher` impl). Never: lane mechanics, product workflow, approval resolution. Internal: split the 4,534-line `host.rs` along its six workflows (module charter). Stage: the membrane — every privileged effect crosses here. Why a crate: **the** loop/host security boundary; single construction site preserved (`host_runtime`). - **6.5.7 `ironclaw_processes`** — retain, widen (**widening LANDED 2026-07-29 via #6696; the `DIRECTION` marking is discharged**). ✎ It is now the **general durable lifecycle authority** the target described: row-native journal (`journal.rs` + `journal_store/{command,migration,observer,rows,state,validation}`), `ProcessSupervisor` (claim/lease/heartbeat/recovery/panic containment/shutdown), process kinds as registered executors (`ProcessKind::AgentTurn` registered by the turn runner, capability invocation by host_runtime), checkpoint payload rows, immutable process input, `result_store`. 2.7k → 12.8k lines; ✎ 8 consumers. Never: scheduling *policy*, model behavior, approval authority (journal records "waiting on approval X", approvals decides). Stage: durable lifecycle + recovery authority. Why a crate: the one-lifecycle invariant needs one owner — now demonstrated rather than argued. ✎ Remaining target work is unchanged and unrelated to the collapse: the `processes → resources` W7 exception still stands and still dissolves by re-layering this crate to `kernel` (§8.3). - **6.5.8 `ironclaw_turns`** — retain, narrow. The turn admission kernel: `TurnCoordinator` (accept/resume/cancel, one-active-run-per-thread, idempotency), `TurnStateRowStore`, `LoopExitApplier` + evidence validation, turn lifecycle events. Sheds: ID/scope vocabulary (already `host_api`'s), `run_profile/` (→ `loop_contracts`), `external_tool_catalog` (→ product, its self-described owner), the `product_adapter` compatibility re-export. ✎ **The predicted internal consolidation landed** (#6696): the `turn_state_row_store/**` engine is gone and the turn store is a projection/adapter over `processes` — 33.7k → 26.0k lines, and the crate did not move, as specified. Consumers drop from ✎ 18 to ~4 (assistant, composition, turn_runner, loop_host); the shed of `run_profile/` (✎ still 14.5k, unshrunk) is now the dominant remaining item. Why a crate: admission + exit-validation authority ("LoopExit is a claim, not truth" lives here). -- **6.5.9 `ironclaw_host_runtime`** — retain, narrow (the kernel service graph). Keeps: `DefaultHostRuntime` (+ the `HostRuntime` port for upper tiers), CapabilityHost construction, the closed `RuntimeLaneExecutor` + lane adapters, mediated egress pipeline (policy+secret staging+sanitize), `BuiltinObligationHandler` + staged handoff stores, process executors, memory-service resolution (provider-agnostic), invocation services. Sheds (with owners): `first_party_tools/` → first-party package (§6.8.4); skill-management *domain execution* → `skills` (+ manager adapter), while the thin builtin *tool handlers* that front it register from `first_party` like every builtin tool; extension binding/catalog defaults → `extension_host`; `sandbox_process/**` → `lanes/ironclaw_sandbox`; pure assembly (`builder.rs`/`production_wiring`) shrinks into composition-facing factories; `obligations.rs` splits internally into its three chartered owners (obligation handling ∣ staged handoffs ∣ process-obligation store). Never: vendor names, product features, DB drivers beyond what mediation itself needs. Why a crate: the privileged service graph — the thing `kernel-boundary.md` names as "the current concrete composition crate for kernel-facing services"; after narrowing it is exactly that and nothing else. +- **6.5.9 `ironclaw_host_runtime`** — retain, narrow (the kernel service graph). Keeps: `DefaultHostRuntime` (+ the `HostRuntime` port for upper tiers), CapabilityHost construction, the closed `RuntimeLaneExecutor` + lane adapters, mediated egress pipeline (policy+secret staging+sanitize), `BuiltinObligationHandler` + staged handoff stores, process executors, memory-service resolution (provider-agnostic), invocation services. Sheds (with owners): `first_party_tools/` → first-party package (§6.8.4); skill-management *domain execution* → `skills` (+ manager adapter), while the thin builtin *tool handlers* that front it register from `first_party` like every builtin tool; extension binding/catalog defaults → `extension_host`; `sandbox_process/**` → `lanes/ironclaw_sandbox`; pure assembly (`builder.rs`/`production_wiring`) shrinks into composition-facing factories; `obligations.rs` splits internally into its three chartered owners (obligation handling ∣ staged handoffs ∣ process-obligation store). ✎ **Amended 2026-08-03 (WS3 obligations/builder PR): the obligations clause is executed; the assembly clause is half-refuted.** The split landed as `obligations/{handler,staged_handoffs,process_store}` behind a `mod.rs` that holds only `BuiltinObligationServices` (the assembly seam), with three `pub(super)` widenings as its whole cost and the crate's public names unchanged. **`production_wiring` does not belong in that clause**: it is 372 lines of readiness *diagnostics* that composition already consumes through `RebornReadinessDiagnostic::from_production_wiring_report`, with no assembly in it to move — the pairing with `builder.rs` was a misreading of what the file is. `builder.rs` itself measured 887 lines / 50 public builder methods / 602 call sites in 45 files, of which only three could be narrowed without either a `test-support` cargo feature (17 methods reachable only from the crate's own `tests/**`) or a redesign of the fluent surface (the 33 genuinely composition-facing ones); three more are narrowable-looking but simply callerless and belong to WS8. CHECKLIST WS3's amended row carries the per-method disposition. Never: vendor names, product features, DB drivers beyond what mediation itself needs. Why a crate: the privileged service graph — the thing `kernel-boundary.md` names as "the current concrete composition crate for kernel-facing services"; after narrowing it is exactly that and nothing else. - **6.5.10 `ironclaw_run_state`** — ~~retain transitional, then delete~~ — **RETIRED 2026-07-30: the deletion landed.** *Kept as a dated tombstone because this document is a decision record and the entry was load-bearing for §9 and the CHECKLIST.* What it described (verified at authoring): `RunRecord`/`RunStatus` duplicating process statuses under the same `InvocationId` with one real consumer (capabilities); `BlockedApproval`/`BlockedAuth` duplicated verbatim in `TurnStatus`; three parallel "what is blocked" handles. #6696 resolved it exactly as specified — the crate is gone, its approval and gate record stores are `approvals::approval_store`, and invocation state is a projection over the process journal. The kernel family is nine crates in code as well as in the target; no successor entry is needed. ### 6.6 `crates/lanes/` — execution mechanisms @@ -964,7 +964,7 @@ Every current workspace package (66) plus excluded packages. Disposition vocabul **Legacy-v1 classification:** with the enclave already deleted from `main`, the only v1 remnants are *inside* live crates and are handled as deletions above: `auth::loopback_oauth`, `llm::reasoning`, `skills::{registry,catalog,v2,gating}` + its v1 lib.rs doc, `ironclaw_embeddings`, and the stale v1 references across guidance (§11.5). Nothing else qualifies. -**Explicitly identified anti-pattern inventory (per the deliverable checklist):** compatibility shims — `dispatcher`, `turns::{ids,scope,product_adapter}` re-exports, memory_native's six path shims, traces' two re-export modules, product's ~120-symbol facade; transitional bridges — ~~`run_state`~~ (✎ deleted 2026-07-29 with #6696), `first_party_extension_ports` (until W7 shed), config's parse-only `SlackSection`; god-crate modules — composition `runtime.rs`/`factory.rs`/✎`runtime/capability_host/**` (ex `local_dev/**`), extension_host's #6616/#6669 arrivals, host_runtime `obligations.rs`/`first_party_tools/`, runner ✎`subagent/await_edge`+`model_gateway`+`tool_disclosure`, product `reborn_services/**`, loop_host `capability_port.rs`, traces `contribution.rs`, webui `handlers.rs`; backend duplication — product/openai-compat LibSql/Postgres newtype wrappers over the already-backend-neutral fabric (collapse to the generic form), triggers/hooks hand-written SQL (ADR-or-converge); vendor fragmentation — slack across 3 locations, telegram across 2 crates + CLI googlisms + config vendor sections (all resolved into `packages/`); accidental trait/DTO seams — the ~17 single-impl product ports (relocated, not deleted — they are real inversions in the wrong crate), `ToolPermissionOverrideStorePort` & `RouteCurrentRunFinalReply` & memory-native `EmbeddingProvider` (deleted — no inversion), the `ExternalActorRef`/`ExternalConversationRef`/`AttachmentRef`/`SessionThreadService`/`EventStreamManager` name collisions (renamed/unified). +**Explicitly identified anti-pattern inventory (per the deliverable checklist):** compatibility shims — `dispatcher`, `turns::{ids,scope,product_adapter}` re-exports, memory_native's six path shims, traces' two re-export modules, product's ~120-symbol facade; transitional bridges — ~~`run_state`~~ (✎ deleted 2026-07-29 with #6696), `first_party_extension_ports` (until W7 shed), config's parse-only `SlackSection`; god-crate modules — composition `runtime.rs`/`factory.rs`/✎`runtime/capability_host/**` (ex `local_dev/**`), extension_host's #6616/#6669 arrivals, host_runtime ~~`obligations.rs`~~ (✎ split into its three owners 2026-08-03, WS3)/`first_party_tools/`, runner ✎`subagent/await_edge`+`model_gateway`+`tool_disclosure`, product `reborn_services/**`, loop_host `capability_port.rs`, traces `contribution.rs`, webui `handlers.rs`; backend duplication — product/openai-compat LibSql/Postgres newtype wrappers over the already-backend-neutral fabric (collapse to the generic form), triggers/hooks hand-written SQL (ADR-or-converge); vendor fragmentation — slack across 3 locations, telegram across 2 crates + CLI googlisms + config vendor sections (all resolved into `packages/`); accidental trait/DTO seams — the ~17 single-impl product ports (relocated, not deleted — they are real inversions in the wrong crate), `ToolPermissionOverrideStorePort` & `RouteCurrentRunFinalReply` & memory-native `EmbeddingProvider` (deleted — no inversion), the `ExternalActorRef`/`ExternalConversationRef`/`AttachmentRef`/`SessionThreadService`/`EventStreamManager` name collisions (renamed/unified). --- From 043bc6c19b96c2ba96680dbe80dca2e47a672e72 Mon Sep 17 00:00:00 2001 From: BenKurrek Date: Mon, 3 Aug 2026 21:19:58 -0400 Subject: [PATCH 2/2] docs(host-runtime): state the obligation visibility rule as it holds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review catch (#7090): the guardrail sentence promised "cross-owner access is `pub(super)`, never `pub(crate)`", which is stronger than the code. Verified: `RuntimeSecretInjectionStore::{insert, take, clone_material, discard_for_capability}`, `NetworkObligationPolicyStore::{insert, get, take, discard_for_capability}` and both constructors are `pub(crate)` and must stay so — `src/egress/{mod,host_port,credential}.rs` call them, and that is host-runtime composition outside `obligations/`. The rule is restated as the property that actually holds: a method whose only callers are inside `obligations/` is `pub(super)` (the three that are), and `pub(crate)` is what the stores expose to the egress pipeline they exist to serve. A future agent reading the old sentence would have read the existing `pub(crate)` methods as violations. Guidance-only; no code change. Co-Authored-By: Claude Opus 5 --- crates/ironclaw_host_runtime/CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/ironclaw_host_runtime/CLAUDE.md b/crates/ironclaw_host_runtime/CLAUDE.md index 4ce9a577652..6df90a942db 100644 --- a/crates/ironclaw_host_runtime/CLAUDE.md +++ b/crates/ironclaw_host_runtime/CLAUDE.md @@ -4,7 +4,7 @@ - Keep runtime-specific request shapes in the runtime crates; adapters should translate into host API contracts and delegate here. - Compose low-level services such as `ironclaw_network` and `ironclaw_secrets`; do not duplicate URL parsing, DNS checks, private-IP filtering, HTTP clients, secret stores, or redaction logic in runtime crates. - Host HTTP egress lives under `src/egress/`: keep request validation/sanitization, credential-source resolution, staged network-policy lookup, staged secret injection, transport dispatch, response sanitization, and response-body storage as separate pipeline steps instead of rebuilding a monolithic service method. -- Obligation code lives under `src/obligations/` and stays in its owner: `handler` decides and executes obligations, `staged_handoffs` owns the secret/network material staged for a later consumer, `process_store` owns post-start cleanup and reservation reconciliation. `mod.rs` holds only `BuiltinObligationServices` — the assembly seam. A change that needs all three at once is a signal the split is being undone, not a reason to add code to `mod.rs`. Cross-owner access is `pub(super)`, never `pub(crate)`, so widening it is a deliberate edit. +- Obligation code lives under `src/obligations/` and stays in its owner: `handler` decides and executes obligations, `staged_handoffs` owns the secret/network material staged for a later consumer, `process_store` owns post-start cleanup and reservation reconciliation. `mod.rs` holds only `BuiltinObligationServices` — the assembly seam. A change that needs all three at once is a signal the split is being undone, not a reason to add code to `mod.rs`. Access needed **only** by a sibling owner is `pub(super)` — `RuntimeSecretInjectionStore::{has_for_capability, prune_expired}` and `NetworkObligationPolicyStore::contains` — so widening one of those is a deliberate edit. `pub(crate)` on the staged-handoff stores is *not* a violation of that rule: `insert`/`take`/`clone_material`/`get`/`discard_for_capability` are also called from `src/egress/**`, which is host-runtime composition outside `obligations/` and is the reason those stores exist. The test is what the caller set actually is, not which keyword appears: if a method's only callers are inside `obligations/`, it is `pub(super)`. - Production host HTTP egress must be constructed with staged `NetworkObligationPolicyStore` and `RuntimeSecretInjectionStore` handoffs. Request-carried policy and direct `SecretStoreLease` sources are legacy/test compatibility paths only. - Preserve the accounting invariant: `network_egress_bytes` is outbound request bytes only, with response bytes tracked separately. - Keep raw secret material inside the narrow lease/injection path. Reject runtime-supplied manual credentials, scan raw and percent-decoded URL forms, redact leased values from runtime-visible errors and responses, strip sensitive response headers, and block credential-shaped runtime requests/responses before they reach external services or runtime callers.