diff --git a/.claude/rules/database.md b/.claude/rules/database.md index 07accf07917..44bd3baa720 100644 --- a/.claude/rules/database.md +++ b/.claude/rules/database.md @@ -18,7 +18,8 @@ See `src/db/CLAUDE.md` for full schema, dialect differences, and libSQL limitati 4. Implement in `src/db/libsql/.rs` (use `self.connect().await?` per operation) 5. Add migration if needed: - PostgreSQL: new `migrations/VN__description.sql` - - libSQL: add `CREATE TABLE IF NOT EXISTS` to `libsql_migrations.rs` + - libSQL: add entry to `INCREMENTAL_MIGRATIONS` in `libsql_migrations.rs` + - **Version numbering**: always number after the highest version on `staging`/`main` — those migrations may already be in production. Check with `git ls-tree origin/staging migrations/` and staging's `INCREMENTAL_MIGRATIONS`. Never reuse or insert before an existing version. 6. Test feature isolation: ```bash cargo check # postgres (default) @@ -58,6 +59,10 @@ Multi-step operations (INSERT+INSERT, UPDATE+DELETE, read-modify-write) MUST be `LibSqlBackend::connect()` creates a fresh connection per operation with `PRAGMA busy_timeout = 5000`. This is intentional -- no pool exists. Never hold connections open across `await` points. Satellite stores (`LibSqlSecretsStore`, `LibSqlWasmToolStore`) receive `Arc` via `shared_db()` and call `.connect()` themselves -- never pass a live `Connection`. +## Never Delete LLM Output Data + +All LLM execution data — thread messages, steps, events, tool call parameters and results — must **never** be deleted from the database. This is the most valuable data in the system. No `DELETE` statements, no `DROP`, no truncation of LLM-generated content. In-memory caches (HashMaps in `HybridStore`) may evict entries for memory pressure, but database rows are permanent. Load methods must fall back to the database on a cache miss. + ## Fix the Pattern, Not the Instance When fixing a bug in one backend's SQL, always grep for the same pattern in the other. A fix to `postgres.rs` that doesn't also fix `libsql/jobs.rs` is half a fix. Same applies to satellite stores. diff --git a/.claude/rules/skills.md b/.claude/rules/skills.md index ded26de98b2..1dc797dda0e 100644 --- a/.claude/rules/skills.md +++ b/.claude/rules/skills.md @@ -31,16 +31,19 @@ activation: tags: - "devops" max_context_tokens: 2000 -metadata: - openclaw: - requires: - bins: [docker, kubectl] - env: [KUBECONFIG] +requires: + bins: [docker, kubectl] + env: [KUBECONFIG] --- # Skill instructions here... ``` +Only the top-level `requires:` block is supported. The legacy nested shape +`metadata.openclaw.requires` is unsupported and ignored by the current parser, +so older external skills must be migrated instead of relying on silent +compatibility. + ## Selection Pipeline 1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing diff --git a/.claude/rules/testing.md b/.claude/rules/testing.md index 3d50b3ea67b..263c890f1b7 100644 --- a/.claude/rules/testing.md +++ b/.claude/rules/testing.md @@ -23,3 +23,44 @@ Run `bash scripts/check-boundaries.sh` to verify test tier gating. - Use `tempfile` crate for test directories, never hardcode `/tmp/` - Regression test with every bug fix (enforced by commit-msg hook) - Integration tests (`--test workspace_integration`) require PostgreSQL; skipped if DB is unreachable + +## Test Through the Caller, Not Just the Helper + +**When a helper gates a side-effecting flow, the test must go through the caller — not just the helper in isolation.** + +A whole class of bugs in this repo has the same shape: a wrapper function silently loses one of its inputs, and the unit test for the helper passes because it never crosses the layer where the input gets dropped. + +Real examples (do not let these recur): + +| Bug | Helper | What got lost | How a caller-level test would have caught it | +|-----|--------|--------------|------------------------------------------------| +| nearai/ironclaw#1948 | `McpServerConfig::has_custom_auth_header()` | Helper existed but `requires_auth()` never consulted it, so MCP triggered OAuth/DCR even with a user-set `Authorization` header | A test driving `mcp::factory::create_client_from_config()` with a header-bearing config and asserting zero OAuth-state side effects | +| nearai/ironclaw#1921 | `derive_activation_status(ext, has_owner_binding)` | Wrapper hardcodes the underlying classifier's `has_paired` axis to `false`, even though `classify_wasm_channel_activation` takes both bools | A test driving `extensions_list_handler` against a DB with a real `channel_identities` row and asserting `Active`, not `Pairing` | +| nearai/ironclaw#1502 | `window.open` mock `(url) => { window._lastOpenedUrl = url }` | Mock captured only the URL, silently swallowing `target` and `windowFeatures`; a regression to same-tab open would not fail | A mock capturing all three args plus an assert that `target === '_blank'` | + +### When the rule applies + +You must add a caller-level test (not just a helper-level unit test) when **all** of the following are true: + +1. The helper is a **predicate, classifier, or transform** whose return value gates a side effect (HTTP call, DB write, UI mutation, OAuth flow, secret read, tool execution, sandbox launch, etc.). +2. There is **at least one wrapper or call site** between the helper and the side effect. +3. The helper has **more than one input** *or* its caller computes any of the inputs from the surrounding context. + +If all three are true, a unit test on the helper alone is **not sufficient regression coverage**. You must additionally either: + +- Add a test that drives the call site (`*_handler`, `factory::create_*`, `manager::*`), **or** +- Inline the helper into its single caller so there is no wrapper to silently drop an input. + +### Where the test belongs + +Most of these gaps are above unit-test scope and below e2e scope. Default to the **integration tier** (`cargo test --features integration`): + +- `tests/_integration.rs` for Rust integration tests against the public handler/factory surface +- `tests/multi_tenant_integration.rs` when the lost axis is per-user state +- `tests/e2e/scenarios/test_*.py` when the lost axis is browser-visible + +Unit tests in `mod tests {}` are still fine for the helper itself, but they do not satisfy this rule. + +### Mock hygiene corollary + +When you mock a browser/runtime API in a test, the mock's signature must match the production call site's signature, and assertions should cover **every argument** the production code passes. A `(url) => {}` stub for a `window.open(url, target, features)` call site is a silent argument-loss bug waiting to happen. diff --git a/.claude/rules/tools.md b/.claude/rules/tools.md index a35d9e237e2..fa58f599677 100644 --- a/.claude/rules/tools.md +++ b/.claude/rules/tools.md @@ -2,6 +2,8 @@ paths: - "src/tools/**" - "tools-src/**" + - "src/channels/**" + - "src/cli/**" --- # Tool Architecture @@ -37,3 +39,102 @@ impl Tool for MyTool { fn requires_sanitization(&self) -> bool { true } // External data } ``` + +## Everything Goes Through Tools + +**All actions originating from any non-agent caller — gateway handlers, CLI +commands, routine engine, WASM channels, future channel extensions — MUST +go through `ToolDispatcher::dispatch()`, never directly through the +database, workspace, or domain managers.** + +This is the core design principle behind #2049. The reasons are concrete: + +1. **Audit trail.** Every dispatched call creates an `ActionRecord` linked + to a system job, so UI-initiated mutations are visible in job history + alongside agent-initiated ones. Direct DB calls bypass this entirely. +2. **Safety pipeline parity.** The dispatcher runs the same pipeline as + `Worker::execute_tool`: parameter normalization, schema validation, + `sensitive_params()` redaction, per-tool timeout, output sanitization. + Direct calls skip all of it and risk leaking secrets into logs or + persisting unsafe content. +3. **Channel-agnostic.** Channels are interchangeable extensions (gateway, + CLI, telegram, WASM, future custom channels). Routing through a single + dispatch function means new channels inherit the full pipeline for free. +4. **Agent parity.** The agent can do anything channels can do (and vice + versa), because both call the same tools. No more "the UI can install + extensions but the agent can only list them" gaps. + +### Required pattern + +```rust +// In any gateway handler, CLI command, or routine engine callback: +use crate::tools::dispatch::{DispatchSource, ToolDispatcher}; + +let dispatcher: &ToolDispatcher = state + .tool_dispatcher + .as_ref() + .ok_or((StatusCode::SERVICE_UNAVAILABLE, "dispatcher unavailable"))?; + +let output = dispatcher + .dispatch( + "memory_write", + serde_json::json!({ "target": path, "content": content }), + &user.user_id, + DispatchSource::Channel("gateway".into()), + ) + .await + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?; +``` + +### Forbidden pattern + +```rust +// DO NOT do this in a gateway handler, CLI command, or routine callback: +let store = state.store.as_ref().ok_or(...)?; +store.set_setting(&user.user_id, &key, &value).await?; // BYPASSES dispatch + +let workspace = resolve_workspace(&state, &user).await?; +workspace.write(path, content).await?; // BYPASSES dispatch + safety pipeline + +let ext_mgr = state.extension_manager.as_ref().ok_or(...)?; +ext_mgr.install(name, url, kind, &user.user_id).await?; // BYPASSES audit trail +``` + +### When direct access IS allowed + +The dispatch principle applies to **non-agent callers** acting on behalf of +a user. These are exempt: + +| Layer | Why exempt | +|---|---| +| `Worker::execute_tool()` (agent loop) | Has its own atomic sequence-numbered audit trail; the dispatcher would conflict | +| `EffectBridgeAdapter::execute_action()` (v2 engine) | Same — its own audit via `ThreadEvent` event sourcing | +| The tool implementations themselves | Tools are the leaves; they need direct `Workspace`, `Database`, etc. handles to do their work | +| Background jobs (scheduler, hygiene, mission runner) inside the engine | These ARE the engine; they emit their own events | +| Pure read endpoints that need to JOIN/aggregate from multiple sources | A single tool call cannot express "list all jobs across users with filters X, Y, Z" — these are queries, not actions, and the audit value is low | + +### Annotating intentional exceptions + +If a handler legitimately needs direct access (rare — usually only for +read aggregation), suppress the pre-commit check with a trailing comment +on the offending line: + +```rust +let rows = state.store.list_agent_jobs().await?; // dispatch-exempt: read-only aggregation +``` + +The pre-commit hook (`scripts/pre-commit-safety.sh`) flags any newly +added line in `src/channels/web/handlers/*.rs` or `src/cli/*.rs` that +touches `state.{store,workspace,workspace_pool,extension_manager, +skill_registry,session_manager}.*` without a trailing +`// dispatch-exempt: ` comment on the same line. The check only +looks at added lines (`+` lines in the diff), so existing untouched code +doesn't trip it during incremental migration. + +### Migration status + +As of #2049, `ToolDispatcher` is wired into `GatewayState` but per-handler +migration is incomplete. New handlers MUST use the dispatcher. Existing +handlers should be migrated incrementally; each handler family +(settings, memory, extensions, skills, routines, jobs, threads) is its +own follow-up PR. diff --git a/.claude/skills/mintlify-docs/SKILL.md b/.claude/skills/mintlify-docs/SKILL.md new file mode 100644 index 00000000000..334e45fd009 --- /dev/null +++ b/.claude/skills/mintlify-docs/SKILL.md @@ -0,0 +1,328 @@ +--- +name: mintlify +description: Build and maintain documentation sites with Mintlify. Use when creating docs pages, configuring navigation, adding components, or setting up API references. +license: MIT +compatibility: Requires Node.js for CLI. Works with any Git-based workflow. +metadata: + author: mintlify + version: "1.0" +--- + +# Mintlify best practices + +**Always consult [mintlify.com/docs](https://mintlify.com/docs) for components, configuration, and latest features.** + +If you are not already connected to the Mintlify MCP server, https://mintlify.com/docs/mcp, add it so that you can search more efficiently. + +**Always** favor searching the current Mintlify documentation over whatever is in your training data about Mintlify. + +Mintlify is a documentation platform that transforms MDX files into documentation sites. Configure site-wide settings in the `docs.json` file, write content in MDX with YAML frontmatter, and favor built-in components over custom components. + +Full schema at [mintlify.com/docs.json](https://mintlify.com/docs.json). + +## Before you write + +### Understand the project + +Read `docs.json` in the project root. This file defines the entire site: navigation structure, theme, colors, links, API and specs. + +Understanding the project tells you: + +- What pages exist and how they're organized +- What navigation groups are used (and their naming conventions) +- How the site navigation is structured +- What theme and configuration the site uses + +### Check for existing content + +Search the docs before creating new pages. You may need to: +- Update an existing page instead of creating a new one +- Add a section to an existing page +- Link to existing content rather than duplicating + +### Read surrounding content + +Before writing, read 2-3 similar pages to understand the site's voice, structure, formatting conventions, and level of detail. + +### Understand Mintlify components + +Review the Mintlify [components](https://www.mintlify.com/docs/components) to select and use any relevant components for the documentation request that you are working on. + +## Quick reference + +### CLI commands +- `npm i -g mint` - Install the Mintlify CLI +- `mint dev` - Local preview at localhost:3000 +- `mint broken-links` - Check internal links +- `mint a11y` - Check for accessibility issues in content +- `mint validate` - Validate documentation builds + +### Required files +- `docs.json` - Site configuration (navigation, theme, integrations, etc.). See [global settings](https://mintlify.com/docs/settings/global) for all options. +- `*.mdx` files - Documentation pages with YAML frontmatter + +### Example file structure +``` +project/ +├── docs.json # Site configuration +├── introduction.mdx +├── quickstart.mdx +├── guides/ +│ └── example.mdx +├── openapi.yml # API specification +├── images/ # Static assets +│ └── example.png +└── snippets/ # Reusable components + └── component.jsx +``` + +## Page frontmatter + +Every page requires `title` in its frontmatter. Include `description` for SEO and navigation. + +```yaml +--- +title: "Clear, descriptive title" +description: "Concise summary for SEO and navigation." +--- +``` + +Optional frontmatter fields: +- `sidebarTitle`: Short title for sidebar navigation. +- `icon`: Lucide or Font Awesome icon name, URL, or file path. +- `tag`: Label next to the page title in the sidebar (for example, "NEW"). +- `mode`: Page layout mode (`default`, `wide`, `custom`). +- `keywords`: Array of terms related to the page content for local search and SEO. +- Any custom YAML fields for use with personalization or conditional content. + +## File conventions + +- Match existing naming patterns in the directory +- If there are no existing files or inconsistent file naming patterns, use kebab-case: `getting-started.mdx`, `api-reference.mdx` +- Use root-relative paths without file extensions for internal links: `/getting-started/quickstart` +- Do not use relative paths (`../`) or absolute URLs for internal pages +- When you create a new page, add it to `docs.json` navigation or it won't appear in the sidebar + +## Organize content + +When a user asks about anything related to site-wide configurations, start by understanding the [global settings](https://www.mintlify.com/docs/organize/settings). See if a setting in the `docs.json` file can be updated to achieve what the user wants. + +### Navigation + +The `navigation` property in `docs.json` controls site structure. Choose one primary pattern at the root level, then nest others within it. + +**Choose your primary pattern:** + +| Pattern | When to use | +|---------|-------------| +| **Groups** | Default. Single audience, straightforward hierarchy | +| **Tabs** | Distinct sections with different audiences (Guides vs API Reference) or content types | +| **Anchors** | Want persistent section links at sidebar top. Good for separating docs from external resources | +| **Dropdowns** | Multiple doc sections users switch between, but not distinct enough for tabs | +| **Products** | Multi-product company with separate documentation per product | +| **Versions** | Maintaining docs for multiple API/product versions simultaneously | +| **Languages** | Localized content | + +**Within your primary pattern:** + +- **Groups** - Organize related pages. Can nest groups within groups, but keep hierarchy shallow +- **Menus** - Add dropdown navigation within tabs for quick jumps to specific pages +- **`expanded: false`** - Collapse nested groups by default. Use for reference sections users browse selectively +- **`openapi`** - Auto-generate pages from OpenAPI spec. Add at group/tab level to inherit + +**Common combinations:** +- Tabs containing groups (most common for docs with API reference) +- Products containing tabs (multi-product SaaS) +- Versions containing tabs (versioned API docs) +- Anchors containing groups (simple docs with external resource links) + +### Links and paths + +- **Internal links:** Root-relative, no extension: `/getting-started/quickstart` +- **Images:** Store in `/images`, reference as `/images/example.png` +- **External links:** Use full URLs, they open in new tabs automatically + +## Customize docs sites + +**What to customize where:** +- **Brand colors, fonts, logo** → `docs.json`. See [global settings](https://mintlify.com/docs/settings/global) +- **Component styling, layout tweaks** → `custom.css` at project root +- **Dark mode** → Enabled by default. Only disable with `"appearance": "light"` in `docs.json` if brand requires it + +Start with `docs.json`. Only add `custom.css` when you need styling that config doesn't support. + +## Write content + +### Components + +The [components overview](https://mintlify.com/docs/components) organizes all components by purpose: structure content, draw attention, show/hide content, document APIs, link to pages, and add visual context. Start there to find the right component. + +**Common decision points:** + +| Need | Use | +|------|-----| +| Hide optional details | `` | +| Long code examples | `` | +| User chooses one option | `` | +| Linked navigation cards | `` in `` | +| Sequential instructions | `` | +| Code in multiple languages | `` | +| API parameters | `` | +| API response fields | `` | + +**Callouts by severity:** +- `` - Supplementary info, safe to skip +- `` - Helpful context such as permissions +- `` - Recommendations or best practices +- `` - Potentially destructive actions +- `` - Success confirmation + +### Reusable content + +**When to use snippets:** +- Exact content appears on more than one page +- Complex components you want to maintain in one place +- Shared content across teams/repos + +**When NOT to use snippets:** +- Slight variations needed per page (leads to complex props) + +Import snippets with `import { Component } from "/path/to/snippet-name.jsx"`. + +## Writing standards + +### Voice and structure + +- Second-person voice ("you") +- Active voice, direct language +- Sentence case for headings ("Getting started", not "Getting Started") +- Sentence case for code block titles ("Expandable example", not "Expandable Example") +- Lead with context: explain what something is before how to use it +- Prerequisites at the start of procedural content + +### What to avoid + +**Never use:** +- Marketing language ("powerful", "seamless", "robust", "cutting-edge") +- Filler phrases ("it's important to note", "in order to") +- Excessive conjunctions ("moreover", "furthermore", "additionally") +- Editorializing ("obviously", "simply", "just", "easily") + +**Watch for AI-typical patterns:** +- Overly formal or stilted phrasing +- Unnecessary repetition of concepts +- Generic introductions that don't add value +- Concluding summaries that restate what was just said + +### Formatting + +- All code blocks must have language tags +- All images and media must have descriptive alt text +- Use bold and italics only when they serve the reader's understanding--never use text styling just for decoration +- No decorative formatting or emoji + +### Code examples + +- Keep examples simple and practical +- Use realistic values (not "foo" or "bar") +- One clear example is better than multiple variations +- Test that code works before including it + +## Document APIs + +**Choose your approach:** +- **Have an OpenAPI spec?** → Add to `docs.json` with `"openapi": ["openapi.yaml"]`. Pages auto-generate. Reference in navigation as `GET /endpoint` +- **No spec?** → Write endpoints manually with `api: "POST /users"` in frontmatter. More work but full control +- **Hybrid** → Use OpenAPI for most endpoints, manual pages for complex workflows + +Encourage users to generate endpoint pages from an OpenAPI spec. It is the most efficient and easiest to maintain option. + +## Deploy + +Mintlify deploys automatically when changes are pushed to the connected Git repository. + +**What agents can configure:** +- **Redirects** → Add to `docs.json` with `"redirects": [{"source": "/old", "destination": "/new"}]` +- **SEO indexing** → Control with `"seo": {"indexing": "all"}` to include hidden pages in search + +**Requires dashboard setup (human task):** +- Custom domains and subdomains +- Preview deployment settings +- DNS configuration + +For `/docs` subpath hosting with Vercel or Cloudflare, agents can help configure rewrite rules. See [/docs subpath](https://mintlify.com/docs/deploy/vercel). + +## Workflow + +### 1. Understand the task + +Identify what needs to be documented, which pages are affected, and what the reader should accomplish afterward. If any of these are unclear, ask. + +### 2. Research + +- Read `docs.json` to understand the site structure +- Search existing docs for related content +- Read similar pages to match the site's style + +### 3. Plan + +- Synthesize what the reader should accomplish after reading the docs and the current content +- Propose any updates or new content +- Verify that your proposed changes will help readers be successful + +### 4. Write + +- Start with the most important information +- Keep sections focused and scannable +- Use components appropriately (don't overuse them) +- Mark anything uncertain with a TODO comment: + +```mdx +{/* TODO: Verify the default timeout value */} +``` + +### 5. Update navigation + +If you created a new page, add it to the appropriate group in `docs.json`. + +### 6. Verify + +Before submitting: + +- [ ] Frontmatter includes title and description +- [ ] All code blocks have language tags +- [ ] Internal links use root-relative paths without file extensions +- [ ] New pages are added to `docs.json` navigation +- [ ] Content matches the style of surrounding pages +- [ ] No marketing language or filler phrases +- [ ] TODOs are clearly marked for anything uncertain +- [ ] Run `mint broken-links` to check links +- [ ] Run `mint validate` to find any errors + +## Edge cases + +### Migrations + +If a user asks about migrating to Mintlify, ask if they are using ReadMe or Docusaurus. If they are, use the [@mintlify/scraping](https://www.npmjs.com/package/@mintlify/scraping) CLI to migrate content. If they are using a different platform to host their documentation, help them manually convert their content to MDX pages using Mintlify components. + +### Hidden pages + +Any page that is not included in the `docs.json` navigation is hidden. Use hidden pages for content that should be accessible by URL or indexed for the assistant or search, but not discoverable through the sidebar navigation. + +### Exclude pages + +The `.mintignore` file is used to exclude files from a documentation repository from being processed. + +## Common gotchas + +1. **Component imports** - JSX components need explicit import, MDX components don't +2. **Frontmatter required** - Every MDX file needs `title` at minimum +3. **Code block language** - Always specify language identifier +4. **Never use `mint.json`** - `mint.json` is deprecated. Only ever use `docs.json` + +## Resources + +- [Documentation](https://mintlify.com/docs) +- [Configuration schema](https://mintlify.com/docs.json) +- [Feature requests](https://github.com/orgs/mintlify/discussions/categories/feature-requests) +- [Bugs and feedback](https://github.com/orgs/mintlify/discussions/categories/bugs-feedback) diff --git a/.dockerignore b/.dockerignore index 32b9468cabd..09a7c89ec53 100644 --- a/.dockerignore +++ b/.dockerignore @@ -5,4 +5,3 @@ target/ *.md !CLAUDE.md node_modules/ -tools-src/ diff --git a/.env.example b/.env.example index ce3e312409d..f81b1d69766 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,6 @@ # Database Configuration DATABASE_URL=postgres://localhost/ironclaw -DATABASE_POOL_SIZE=10 +DATABASE_POOL_SIZE=30 # multi-tenant default; reduce to 5-10 for single-user or low-resource deployments # LLM Provider # LLM_BACKEND=nearai # default @@ -42,6 +42,9 @@ DATABASE_POOL_SIZE=10 # Base URL defaults to https://private.near.ai # 2. API key: Set NEARAI_API_KEY to use API key auth from cloud.near.ai. # Base URL defaults to https://cloud-api.near.ai +# When both NEARAI_BASE_URL and NEARAI_API_KEY are set at startup, +# IronClaw also bootstraps a persisted `nearai` MCP server using the +# same base URL and Authorization header. NEARAI_MODEL=Qwen/Qwen3.5-122B-A10B NEARAI_BASE_URL=https://private.near.ai NEARAI_AUTH_URL=https://private.near.ai @@ -191,10 +194,9 @@ HEARTBEAT_NOTIFY_CHANNEL=cli HEARTBEAT_NOTIFY_USER=default # Memory hygiene settings (automatic cleanup of stale workspace documents) -# Runs on each heartbeat tick; identity files (IDENTITY.md, SOUL.md) are never deleted +# Runs on each heartbeat tick; discovers cleanup targets from .config metadata # MEMORY_HYGIENE_ENABLED=true -# MEMORY_HYGIENE_DAILY_RETENTION_DAYS=30 # delete daily/ docs older than this many days -# MEMORY_HYGIENE_CONVERSATION_RETENTION_DAYS=7 # delete conversations/ docs older than this many days +# MEMORY_HYGIENE_VERSION_KEEP_COUNT=50 # max versions to keep per document # MEMORY_HYGIENE_CADENCE_HOURS=12 # minimum hours between cleanup passes # Docker Sandbox @@ -209,6 +211,12 @@ HEARTBEAT_NOTIFY_USER=default # SANDBOX_TIMEOUT_SECS=120 # SANDBOX_MEMORY_LIMIT_MB=2048 +# ACP (Agent Client Protocol) agents +# ACP_ENABLED=false # Enable ACP agent sandbox mode +# ACP_MEMORY_LIMIT_MB=4096 # Memory limit for ACP containers +# ACP_TIMEOUT_SECS=1800 # Maximum session timeout +# Configure agents via CLI: ironclaw acp add goose --command goose --arg "--stdio" + # Safety settings SAFETY_MAX_OUTPUT_LENGTH=100000 SAFETY_INJECTION_CHECK_ENABLED=true @@ -220,5 +228,63 @@ SAFETY_INJECTION_CHECK_ENABLED=true # IRONCLAW_RESTART_DELAY=5 # default wait before exit (seconds, range: 1-30) # IRONCLAW_MAX_FAILURES=10 # max consecutive failures before container exits +# ─── OAuth / Social Login ──────────────────────────────────────────────── +# Enable direct OAuth login (Google, GitHub). Disabled by default. +# OAUTH_ENABLED=true + +# Base URL for OAuth callback URLs. Defaults to http://localhost:{GATEWAY_PORT}. +# Set this to your public URL in production (e.g., https://myapp.example.com). +# OAUTH_BASE_URL=https://myapp.example.com + +# Restrict OAuth login to specific email domains (comma-separated). +# When set, only users with verified emails from these domains can log in. +# Applies to all OAuth providers and OIDC. Leave unset to allow all domains. +# OAUTH_ALLOWED_DOMAINS=company.com,partner.org + +# Google OAuth — Create credentials at https://console.cloud.google.com/apis/credentials +# 1. Create an OAuth 2.0 Client ID (Web application type) +# 2. Add authorized redirect URI: {OAUTH_BASE_URL}/auth/callback/google +# 3. Copy Client ID and Client Secret below +# GOOGLE_CLIENT_ID= +# GOOGLE_CLIENT_SECRET= + +# Restrict Google login to a specific Workspace (G Suite) domain. +# Adds the `hd` parameter to the authorization URL and validates server-side. +# GOOGLE_ALLOWED_HD=company.com + +# Apple Sign In — Configure in https://developer.apple.com/account/resources/identifiers +# 1. Register a Services ID (e.g. com.example.myapp) under Identifiers +# 2. Enable "Sign In with Apple" and configure the return URL: {OAUTH_BASE_URL}/auth/callback/apple +# 3. Create a key (Keys section), enable "Sign In with Apple", download the .p8 file +# 4. Note your Team ID (top right of developer portal) and Key ID +# APPLE_CLIENT_ID=com.example.myapp +# APPLE_TEAM_ID=XXXXXXXXXX +# APPLE_KEY_ID=YYYYYYYYYY +# APPLE_PRIVATE_KEY_PATH=/path/to/AuthKey_YYYYYYYYYY.p8 +# Or inline: APPLE_PRIVATE_KEY_PEM="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" + +# GitHub OAuth — Create an OAuth App at https://github.com/settings/developers +# 1. Create a new OAuth App +# 2. Set Authorization callback URL to: {OAUTH_BASE_URL}/auth/callback/github +# 3. Copy Client ID and generate a Client Secret below +# GITHUB_CLIENT_ID= +# GITHUB_CLIENT_SECRET= + +# NEAR Wallet — No external setup needed. Users sign in with any NEAR wallet +# (HOT, Meteor, MyNearWallet, etc.) via the near-connect SDK. +# NEAR_AUTH_ENABLED=true +# NEAR_AUTH_NETWORK=mainnet # or testnet +# NEAR_AUTH_RPC_URL=https://rpc.mainnet.near.org # auto-detected from network + +# ─── OIDC / SSO (Okta, Cognito, etc.) ────────────────────────────────── +# For reverse-proxy SSO (e.g., AWS ALB + Okta). The gateway validates JWTs +# from the configured header. See also OAUTH_ALLOWED_DOMAINS above, which +# applies to OIDC logins too. +# GATEWAY_OIDC_ENABLED=true +# GATEWAY_OIDC_JWKS_URL=https://your-idp.example.com/.well-known/jwks.json +# GATEWAY_OIDC_HEADER=x-amzn-oidc-data +# GATEWAY_OIDC_ISSUER=https://your-idp.example.com +# GATEWAY_OIDC_AUDIENCE=your-client-id + # Logging RUST_LOG=ironclaw=debug,tower_http=debug diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 0abd640a823..5a833d4d2fc 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -22,3 +22,16 @@ if $NEEDS_CHECK; then exit 1 fi fi + +# i18n parity: when any language pack changes, all languages must stay in sync. +if echo "$STAGED" | grep -qE '^crates/ironclaw_gateway/static/i18n/.*\.js$'; then + echo "pre-commit: checking i18n parity..." + if ! ./scripts/check-i18n-parity.sh; then + echo "" + echo "Commit blocked: i18n parity check failed." + echo "Every key added to en.js must also be added to all other language files (zh-CN.js, ko.js, ...)." + echo "Placeholder tokens like {name} must match across all languages." + echo "To bypass: git commit --no-verify" + exit 1 + fi +fi diff --git a/.github/ISSUE_TEMPLATE/qa-bug.yml b/.github/ISSUE_TEMPLATE/qa-bug.yml new file mode 100644 index 00000000000..ef1d912a642 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/qa-bug.yml @@ -0,0 +1,94 @@ +name: QA Bug Report +description: Bug found during QA testing on staging or hosted environments +title: "[QA] " +labels: ["qa-bug"] +body: + - type: dropdown + id: environment + attributes: + label: Environment + description: Where was this bug found? + options: + - hosted-staging (crab shack) + - hosted-production + - local (cloned ironclaw) + - railway-staging + validations: + required: true + + - type: input + id: version + attributes: + label: Version / Commit Hash + description: Paste the commit hash from staging at time of discovery (run `git rev-parse HEAD` or find it on the Railway deploy) + placeholder: "e.g. abcdef1" + validations: + required: true + + - type: input + id: qa-date + attributes: + label: QA Test Date + description: Date you discovered this (YYYY-MM-DD) + placeholder: "e.g. 2026-04-12" + validations: + required: true + + - type: input + id: feature-area + attributes: + label: Feature Area + description: What part of the app? (e.g. Google Suite extension, Telegram pairing, auth flow) + placeholder: "e.g. Extensions → Google Suite install" + validations: + required: true + + - type: textarea + id: steps + attributes: + label: Steps to Reproduce + description: Exact steps — numbered, specific, no summaries + placeholder: | + 1. Open extensions tab + 2. Click "Install Google Suite" + 3. Fill in credentials and click Save + 4. ... + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What should happen? + validations: + required: true + + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened? Include the exact error message/text. + placeholder: "Error: 'Failed to authenticate with Google: invalid_grant' shown in red toast" + validations: + required: true + + - type: textarea + id: logs + attributes: + label: Logs / Screenshots + description: Paste relevant logs, error output, or attach screenshots. Drag files here. + validations: + required: false + + - type: checkboxes + id: checklist + attributes: + label: Pre-submit checklist + options: + - label: Title is specific (not "fix Google Suite" but "Google Suite install throws invalid_grant on OAuth step") + required: true + - label: Commit hash is filled in + required: true + - label: Steps are numbered and reproducible + required: true diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 00000000000..472089ed5b1 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,48 @@ +version: 2 +updates: + - package-ecosystem: cargo + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 10 + groups: + tokio-ecosystem: + patterns: + - "tokio*" + - "hyper*" + - "axum*" + - "tower*" + serialization: + patterns: + - "serde*" + - "prost*" + wasm: + patterns: + - "wasmtime*" + - "wit-*" + - "wasm-*" + - "cargo-component*" + everything-else: + patterns: + - "*" + exclude-patterns: + - "tokio*" + - "hyper*" + - "axum*" + - "tower*" + - "serde*" + - "prost*" + - "wasmtime*" + - "wit-*" + - "wasm-*" + - "cargo-component*" + + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 5 + groups: + actions: + patterns: + - "*" diff --git a/.github/labeler.yml b/.github/labeler.yml index fd7da0be2fd..6ac08552b00 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -1,5 +1,5 @@ -# Scope labels for actions/labeler@v6 -# Maps file path globs to scope labels. Multiple labels can apply per PR. +# Labels for actions/labeler@v6 +# Maps file path globs to labels. Multiple labels can apply per PR. "scope: agent": - changed-files: @@ -164,3 +164,9 @@ - any-glob-to-any-file: - Cargo.toml - Cargo.lock + +"DB MIGRATION": + - changed-files: + - any-glob-to-any-file: + - migrations/** + - src/db/libsql_migrations.rs diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 4fc7cbf233b..e6fe6128f82 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,7 +6,7 @@ ## Change Type - + - [ ] Bug fix - [ ] New feature @@ -18,16 +18,19 @@ ## Linked Issue - + ## Validation -- [ ] `cargo fmt` -- [ ] `cargo clippy --all --benches --tests --examples --all-features` +- [ ] `cargo fmt --all -- --check` +- [ ] `cargo clippy --all --benches --tests --examples --all-features -- -D warnings` +- [ ] `cargo build` - [ ] Relevant tests pass: +- [ ] `cargo test --features integration` if database-backed or integration behavior changed - [ ] Manual testing: +- [ ] If a coding agent was used and supports it, `review-pr` or `pr-shepherd --fix` was run before requesting review ## Security Impact @@ -45,6 +48,10 @@ +## Review Follow-Through + + + --- -**Review track**: +**Review track**: diff --git a/.github/scripts/create-labels.sh b/.github/scripts/create-labels.sh index 66f07ea9ce1..6b6d10d3cd1 100755 --- a/.github/scripts/create-labels.sh +++ b/.github/scripts/create-labels.sh @@ -62,6 +62,9 @@ create "scope: ci" "546E7A" "CI/CD workflows" create "scope: docs" "78909C" "Documentation" create "scope: dependencies" "90A4AE" "Dependency updates" +echo "==> Creating coordination labels..." +create "DB MIGRATION" "C62828" "PR adds or modifies PostgreSQL or libSQL migration definitions" + echo "==> Creating workflow labels..." create "skip-regression-check" "9E9E9E" "Acknowledged: fix without regression test" diff --git a/.github/scripts/pr-labeler.sh b/.github/scripts/pr-labeler.sh index 96dc0fa7483..78d41a344a5 100755 --- a/.github/scripts/pr-labeler.sh +++ b/.github/scripts/pr-labeler.sh @@ -43,7 +43,10 @@ classify_size() { local total total=$(gh api "repos/${REPO}/pulls/${PR_NUMBER}/files" \ --paginate --jq ' - [.[] | select(.filename | test("\\.(md|txt|rst|adoc)$") | not) | .changes] + [.[] + | select(.filename | test("\\.(md|txt|rst|adoc)$") | not) + | select(.filename | test("^tests/|_test\\.rs$|_tests\\.rs$|/tests/|\\.test\\.[jt]sx?$|\\.spec\\.[jt]sx?$") | not) + | .changes] | add // 0 ') diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index 26c15d8928a..a792ea0d95f 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -20,12 +20,13 @@ jobs: if: contains(github.event.pull_request.labels.*.name, 'staging-promotion') runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 + persist-credentials: false - name: Run Claude Code review - uses: anthropics/claude-code-action@v1 + uses: anthropics/claude-code-action@1eddb334cfa79fdb21ecbe2180ca1a016e8e7d47 # v1 with: anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} allowed_bots: "ironclaw-ci[bot]" diff --git a/.github/workflows/code_style.yml b/.github/workflows/code_style.yml index f89161d9285..614611d3424 100644 --- a/.github/workflows/code_style.yml +++ b/.github/workflows/code_style.yml @@ -2,15 +2,20 @@ name: Code Style on: pull_request: +permissions: + contents: read + jobs: format: name: Formatting runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: components: rustfmt - name: Check formatting @@ -21,9 +26,11 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - name: Run cargo deny - uses: EmbarkStudios/cargo-deny-action@v2 + uses: EmbarkStudios/cargo-deny-action@3fd3802e88374d3fe9159b834c7714ec57d6c979 # v2 clippy: name: Clippy (${{ matrix.name }}) @@ -40,12 +47,14 @@ jobs: flags: "--no-default-features --features libsql" steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: components: clippy - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: clippy-${{ matrix.name }} - name: Check lints @@ -67,12 +76,14 @@ jobs: flags: "--no-default-features --features libsql" steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: components: clippy - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: clippy-windows-${{ matrix.name }} - name: Check lints @@ -83,10 +94,11 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - - uses: actions/setup-python@v5 + persist-credentials: false + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: "3.12" - name: Check for .unwrap(), .expect(), assert!() in production code diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 2f885b169e2..074433d232d 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -32,13 +32,15 @@ on: branches: [main] permissions: - id-token: write contents: read jobs: coverage: name: Coverage (${{ matrix.name }}) runs-on: ubuntu-latest + permissions: + id-token: write + contents: read strategy: fail-fast: false matrix: @@ -67,19 +69,21 @@ jobs: --health-timeout 5s --health-retries 5 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: components: llvm-tools-preview targets: wasm32-wasip2 - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: coverage-${{ matrix.name }} - name: Install cargo-llvm-cov - uses: taiki-e/install-action@cargo-llvm-cov + uses: taiki-e/install-action@62b0f2dec647a8e604c6a0fda0e38530180dce20 # cargo-llvm-cov - name: Install cargo-component run: | @@ -113,7 +117,7 @@ jobs: run: cargo llvm-cov ${{ matrix.flags }} --workspace --lcov --output-path lcov.info - name: Upload to Codecov - uses: codecov/codecov-action@v5 + uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5 with: files: lcov.info flags: ${{ matrix.name }} @@ -125,20 +129,25 @@ jobs: name: E2E Coverage runs-on: ubuntu-latest timeout-minutes: 30 + permissions: + id-token: write + contents: read steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: components: llvm-tools-preview targets: wasm32-wasip2 - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: e2e-coverage - name: Install cargo-llvm-cov - uses: taiki-e/install-action@cargo-llvm-cov + uses: taiki-e/install-action@62b0f2dec647a8e604c6a0fda0e38530180dce20 # cargo-llvm-cov - name: Install cargo-component run: | @@ -162,7 +171,7 @@ jobs: - name: Build instrumented binary run: cargo build --no-default-features --features libsql - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: "3.12" @@ -197,7 +206,7 @@ jobs: - name: Upload to Codecov if: always() - uses: codecov/codecov-action@v5 + uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5 with: files: e2e-coverage.info flags: e2e @@ -207,7 +216,7 @@ jobs: - name: Upload screenshots on failure if: failure() - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: e2e-screenshots path: tests/e2e/screenshots/ diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml new file mode 100644 index 00000000000..9601165fa41 --- /dev/null +++ b/.github/workflows/docker.yml @@ -0,0 +1,173 @@ +name: Docker Image + +on: + # Called by release.yml or other workflows + workflow_call: + inputs: + tag: + description: "Image tag override (leave empty for auto-detect)" + required: false + type: string + default: "" + # On-demand builds + workflow_dispatch: + inputs: + tag: + description: "Image tag override (leave empty for auto-detect)" + required: false + type: string + default: "" + # Daily staging build from the staging branch + schedule: + - cron: '0 6 * * *' + +env: + IMAGE_NAME: nearaidev/ironclaw + WORKER_IMAGE_NAME: nearaidev/ironclaw-worker + +jobs: + build: + name: Build & Push + runs-on: ubuntu-24.04 + permissions: + contents: read + packages: read + actions: write + steps: + - name: Checkout + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + with: + ref: ${{ github.event_name == 'schedule' && 'staging' || '' }} + persist-credentials: false + + - name: Extract version from Cargo.toml + id: version + run: | + VERSION=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/') + echo "version=${VERSION}" >> "$GITHUB_OUTPUT" + echo "Detected version: ${VERSION}" + + - name: Determine tags + id: tags + env: + VERSION: ${{ steps.version.outputs.version }} + EVENT_NAME: ${{ github.event_name }} + INPUT_TAG: ${{ inputs.tag }} + run: | + SHA="sha-${GITHUB_SHA::7}" + echo "sha_tag=${SHA}" >> "$GITHUB_OUTPUT" + + if [[ "${EVENT_NAME}" == "workflow_call" ]]; then + # Release: :version + :latest + :sha-xxx + TAGS="${IMAGE_NAME}:${VERSION}" + TAGS="${TAGS},${IMAGE_NAME}:latest" + TAGS="${TAGS},${IMAGE_NAME}:${SHA}" + WORKER_TAGS="${WORKER_IMAGE_NAME}:${VERSION}" + WORKER_TAGS="${WORKER_TAGS},${WORKER_IMAGE_NAME}:latest" + WORKER_TAGS="${WORKER_TAGS},${WORKER_IMAGE_NAME}:${SHA}" + elif [[ "${EVENT_NAME}" == "schedule" ]]; then + # Daily staging: :staging + :sha-xxx + TAGS="${IMAGE_NAME}:staging" + TAGS="${TAGS},${IMAGE_NAME}:${SHA}" + WORKER_TAGS="${WORKER_IMAGE_NAME}:staging" + WORKER_TAGS="${WORKER_TAGS},${WORKER_IMAGE_NAME}:${SHA}" + else + # Manual dispatch: :sha-xxx only + TAGS="${IMAGE_NAME}:${SHA}" + WORKER_TAGS="${WORKER_IMAGE_NAME}:${SHA}" + fi + + # Manual override adds an extra tag (e.g. "staging") + if [[ -n "${INPUT_TAG}" ]]; then + TAGS="${TAGS},${IMAGE_NAME}:${INPUT_TAG}" + WORKER_TAGS="${WORKER_TAGS},${WORKER_IMAGE_NAME}:${INPUT_TAG}" + fi + echo "tags=${TAGS}" >> "$GITHUB_OUTPUT" + echo "worker_tags=${WORKER_TAGS}" >> "$GITHUB_OUTPUT" + + # Staging builds get pre-bundled WASM extensions + if [[ "${EVENT_NAME}" == "schedule" || "${INPUT_TAG}" == "staging" ]]; then + echo "target=runtime-staging" >> "$GITHUB_OUTPUT" + else + echo "target=runtime" >> "$GITHUB_OUTPUT" + fi + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + + - name: Log in to Docker Hub + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + with: + username: ${{ vars.DOCKER_REGISTRY_USER }} + password: ${{ secrets.DOCKER_REGISTRY_TOKEN }} + + - name: Build and push (ironclaw) + uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 + with: + context: . + push: true + tags: ${{ steps.tags.outputs.tags }} + target: ${{ steps.tags.outputs.target }} + platforms: linux/amd64 + cache-from: type=gha + cache-to: type=gha,mode=max + + - name: Build and push (ironclaw-worker) + uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 + with: + context: . + file: Dockerfile.worker + push: true + tags: ${{ steps.tags.outputs.worker_tags }} + platforms: linux/amd64 + cache-from: type=gha,scope=worker + cache-to: type=gha,mode=max,scope=worker + + - name: Create releases-manager app token + id: app-token + continue-on-error: true + uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2 + with: + app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }} + private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }} + owner: nearai + repositories: ironclaw-dind + + - name: Trigger ironclaw-dind Build & Push + if: steps.app-token.outcome == 'success' + continue-on-error: true + env: + GH_TOKEN: ${{ steps.app-token.outputs.token }} + EVENT_NAME: ${{ github.event_name }} + INPUT_TAG: ${{ inputs.tag }} + VERSION: ${{ steps.version.outputs.version }} + run: | + if [[ "${EVENT_NAME}" == "workflow_call" && -n "${VERSION}" ]]; then + gh api repos/nearai/ironclaw-dind/dispatches \ + --method POST \ + -f event_type="ironclaw_image_published" \ + -f client_payload[version]="${VERSION}" + elif [[ "${EVENT_NAME}" == "schedule" ]] || [[ "${INPUT_TAG}" == "staging" ]]; then + gh api repos/nearai/ironclaw-dind/dispatches \ + --method POST \ + -f event_type="ironclaw_image_published" + fi + + - name: Summary + run: | + { + echo "## Docker Images" + echo "" + echo "**ironclaw:**" + echo '```' + echo "${{ steps.tags.outputs.tags }}" | tr ',' '\n' + echo '```' + echo "" + echo "**ironclaw-worker:**" + echo '```' + echo "${{ steps.tags.outputs.worker_tags }}" | tr ',' '\n' + echo '```' + echo "" + echo "- version: \`${{ steps.version.outputs.version }}\`" + echo "- sha: \`${GITHUB_SHA::7}\`" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index bc705df7280..9ed4df9cad9 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -1,6 +1,11 @@ name: E2E Tests on: workflow_call: + inputs: + ref: + description: Commit SHA or ref to test + required: false + type: string schedule: - cron: "0 6 * * 1" # Weekly Monday 6 AM UTC workflow_dispatch: @@ -11,6 +16,9 @@ on: - "src/channels/web/**" - "tests/e2e/**" +permissions: + contents: read + jobs: # ── Step 1: compile once ────────────────────────────────────────────────── build: @@ -18,11 +26,14 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - - uses: dtolnay/rust-toolchain@stable + - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - - uses: actions/cache@v4 + - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4 with: path: | target @@ -33,7 +44,7 @@ jobs: run: cargo build --no-default-features --features libsql - name: Upload binary - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: ironclaw-e2e-binary path: target/debug/ironclaw @@ -54,14 +65,17 @@ jobs: - group: features files: "tests/e2e/scenarios/test_skills.py tests/e2e/scenarios/test_tool_approval.py tests/e2e/scenarios/test_webhook.py" - group: extensions - files: "tests/e2e/scenarios/test_extensions.py tests/e2e/scenarios/test_extension_oauth.py tests/e2e/scenarios/test_oauth_url_parameters.py tests/e2e/scenarios/test_telegram_token_validation.py tests/e2e/scenarios/test_telegram_hot_activation.py tests/e2e/scenarios/test_wasm_lifecycle.py tests/e2e/scenarios/test_tool_execution.py tests/e2e/scenarios/test_pairing.py tests/e2e/scenarios/test_mcp_auth_flow.py tests/e2e/scenarios/test_oauth_credential_fallback.py tests/e2e/scenarios/test_routine_oauth_credential_injection.py" + files: "tests/e2e/scenarios/test_extensions.py tests/e2e/scenarios/test_extension_oauth.py tests/e2e/scenarios/test_oauth_url_parameters.py tests/e2e/scenarios/test_telegram_token_validation.py tests/e2e/scenarios/test_telegram_hot_activation.py tests/e2e/scenarios/test_wasm_lifecycle.py tests/e2e/scenarios/test_tool_execution.py tests/e2e/scenarios/test_agent_loop_recovery.py tests/e2e/scenarios/test_pairing.py tests/e2e/scenarios/test_mcp_auth_flow.py tests/e2e/scenarios/test_oauth_credential_fallback.py tests/e2e/scenarios/test_routine_oauth_credential_injection.py" - group: routines files: "tests/e2e/scenarios/test_owner_scope.py tests/e2e/scenarios/test_routine_event_batch.py" steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Download binary - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: name: ironclaw-e2e-binary path: target/debug/ @@ -69,7 +83,7 @@ jobs: - name: Make binary executable run: chmod +x target/debug/ironclaw - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 with: python-version: "3.12" @@ -84,7 +98,7 @@ jobs: - name: Upload screenshots on failure if: failure() - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: e2e-screenshots-${{ matrix.group }} path: tests/e2e/screenshots/ diff --git a/.github/workflows/pr-label-classify.yml b/.github/workflows/pr-label-classify.yml index 90f141de717..7d0ee97a9ac 100644 --- a/.github/workflows/pr-label-classify.yml +++ b/.github/workflows/pr-label-classify.yml @@ -14,9 +14,10 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout base branch - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: ${{ github.event.pull_request.base.ref }} + persist-credentials: false - name: Classify PR env: diff --git a/.github/workflows/pr-label-scope.yml b/.github/workflows/pr-label-scope.yml index 1c3885612e7..b8a282472ba 100644 --- a/.github/workflows/pr-label-scope.yml +++ b/.github/workflows/pr-label-scope.yml @@ -6,13 +6,20 @@ on: permissions: contents: read + issues: write pull-requests: write jobs: scope: runs-on: ubuntu-latest steps: - - uses: actions/labeler@v5 + - name: Ensure DB MIGRATION label exists + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + run: gh label create "DB MIGRATION" --repo "$REPO" --color C62828 --description "PR adds or modifies PostgreSQL or libSQL migration definitions" --force + + - uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5 with: configuration-path: .github/labeler.yml sync-labels: false # additive only — never remove scope labels diff --git a/.github/workflows/regression-test-check.yml b/.github/workflows/regression-test-check.yml index 75b8eb55304..d06301b3787 100644 --- a/.github/workflows/regression-test-check.yml +++ b/.github/workflows/regression-test-check.yml @@ -3,29 +3,37 @@ name: Regression Test Check on: pull_request: +permissions: + contents: read + jobs: regression-test: name: Regression test enforcement runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: fetch-depth: 0 + persist-credentials: false - name: Fetch PR head and base + env: + BASE_REF: ${{ github.event.pull_request.base.ref }} + PR_NUMBER: ${{ github.event.pull_request.number }} run: | - git fetch origin ${{ github.event.pull_request.base.ref }} - git fetch origin pull/${{ github.event.pull_request.number }}/head:pr-head + git fetch origin -- "$BASE_REF" + git fetch origin -- "pull/${PR_NUMBER}/head:pr-head" - name: Check for regression tests env: PR_TITLE: ${{ github.event.pull_request.title }} PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }} + PR_BASE_REF: ${{ github.event.pull_request.base.ref }} run: | set -euo pipefail - BASE_REF="origin/${{ github.event.pull_request.base.ref }}" + BASE_REF="origin/${PR_BASE_REF}" # Use the actual PR head, not the merge commit that actions/checkout checks out HEAD_REF="pr-head" diff --git a/.github/workflows/release-plz-batch-summary.yml b/.github/workflows/release-plz-batch-summary.yml index 0e1067362fd..8e01ec40e21 100644 --- a/.github/workflows/release-plz-batch-summary.yml +++ b/.github/workflows/release-plz-batch-summary.yml @@ -29,11 +29,12 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout base branch - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: ref: ${{ github.event_name == 'workflow_dispatch' && 'main' || github.event.pull_request.base.ref }} fetch-depth: 0 fetch-tags: true + persist-credentials: false - name: Update release-plz PR body with staging batch summary env: diff --git a/.github/workflows/release-plz.yml b/.github/workflows/release-plz.yml index d1be9004e68..cfff0e59720 100644 --- a/.github/workflows/release-plz.yml +++ b/.github/workflows/release-plz.yml @@ -17,18 +17,18 @@ jobs: steps: - &checkout name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 persist-credentials: false - &install-rust name: Install Rust toolchain - uses: dtolnay/rust-toolchain@stable - - uses: Swatinem/rust-cache@v2 + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 # Generating a GitHub token, so that PRs and tags created by # the release-plz-action can trigger actions workflows. - name: Generate GitHub token - uses: actions/create-github-app-token@v2 + uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2 id: generate-token with: # GitHub App ID secret name @@ -36,7 +36,7 @@ jobs: # GitHub App private key secret name private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }} - name: Run release-plz - uses: release-plz/action@v0.5 + uses: release-plz/action@1528104d2ca23787631a1c1f022abb64b34c1e11 # v0.5 with: command: release env: @@ -57,15 +57,15 @@ jobs: steps: - *checkout - *install-rust - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 - name: Generate GitHub token - uses: actions/create-github-app-token@v2 + uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2 id: generate-token with: app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }} private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }} - name: Run release-plz - uses: release-plz/action@v0.5 + uses: release-plz/action@1528104d2ca23787631a1c1f022abb64b34c1e11 # v0.5 with: command: release-pr env: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c4a4f416d53..7d23449e00f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -15,7 +15,7 @@ name: Release permissions: - "contents": "write" + contents: read # This task will run whenever you push a git tag that looks like a version # like "1.0.0", "v0.1.0-prerelease.1", "my-app/0.1.0", "releases/v1.0.0", etc. @@ -41,7 +41,7 @@ permissions: on: push: tags: - - '**[0-9]+.[0-9]+.[0-9]+*' + - 'ironclaw-v[0-9]+.[0-9]+.[0-9]+*' jobs: # Run 'dist plan' (or host) to determine what tasks we need to do @@ -55,7 +55,7 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive @@ -65,7 +65,7 @@ jobs: shell: bash run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.3/cargo-dist-installer.sh | sh" - name: Cache dist - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: cargo-dist-cache path: ~/.cargo/bin/dist @@ -75,13 +75,20 @@ jobs: # (PRs run on the *source* but secrets are usually on the *target* -- that's *good* # but also really annoying to build CI around when it needs secrets to work right.) - id: plan + env: + IS_PUSH: ${{ !github.event.pull_request }} + REF_NAME: ${{ github.ref_name }} run: | - dist ${{ (!github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name)) || 'plan' }} --output-format=json > plan-dist-manifest.json + if [ "$IS_PUSH" = "true" ]; then + dist host --steps=create --tag="$REF_NAME" --output-format=json > plan-dist-manifest.json + else + dist plan --output-format=json > plan-dist-manifest.json + fi echo "dist ran successfully" cat plan-dist-manifest.json echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT" - name: "Upload dist-manifest.json" - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: artifacts-plan-dist-manifest path: plan-dist-manifest.json @@ -117,7 +124,7 @@ jobs: - name: enable windows longpaths run: | git config --global core.longpaths true - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive @@ -128,7 +135,7 @@ jobs: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y echo "$HOME/.cargo/bin" >> $GITHUB_PATH fi - - uses: swatinem/rust-cache@v2 + - uses: swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: ${{ join(matrix.targets, '-') }} cache-provider: ${{ matrix.cache_provider }} @@ -136,7 +143,7 @@ jobs: run: ${{ matrix.install_dist.run }} # Get the dist-manifest - name: Fetch local artifacts - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: artifacts-* path: target/distrib/ @@ -180,9 +187,13 @@ jobs: run: | ${{ matrix.packages_install }} - name: Build artifacts + env: + TAG_FLAG: ${{ needs.plan.outputs.tag-flag }} + DIST_ARGS: ${{ matrix.dist_args }} run: | # Actually do builds and make zips and whatnot - dist build ${{ needs.plan.outputs.tag-flag }} --print=linkage --output-format=json ${{ matrix.dist_args }} > dist-manifest.json + # shellcheck disable=SC2086 # TAG_FLAG/DIST_ARGS may contain multiple args + dist build $TAG_FLAG --print=linkage --output-format=json $DIST_ARGS > dist-manifest.json echo "dist ran successfully" - id: cargo-dist name: Post-build @@ -198,7 +209,7 @@ jobs: cp dist-manifest.json "$BUILD_MANIFEST_NAME" - name: "Upload artifacts" - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: artifacts-build-local-${{ join(matrix.targets, '_') }} path: | @@ -215,27 +226,30 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive - name: Install cached dist - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: name: cargo-dist-cache path: ~/.cargo/bin/ - run: chmod +x ~/.cargo/bin/dist # Get all the local artifacts for the global tasks to use (for e.g. checksums) - name: Fetch local artifacts - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: artifacts-* path: target/distrib/ merge-multiple: true - id: cargo-dist shell: bash + env: + TAG_FLAG: ${{ needs.plan.outputs.tag-flag }} run: | - dist build ${{ needs.plan.outputs.tag-flag }} --output-format=json "--artifacts=global" > dist-manifest.json + # shellcheck disable=SC2086 # TAG_FLAG may expand to '--tag=X' or empty + dist build $TAG_FLAG --output-format=json "--artifacts=global" > dist-manifest.json echo "dist ran successfully" # Parse out what we just built and upload it to scratch storage @@ -245,7 +259,7 @@ jobs: cp dist-manifest.json "$BUILD_MANIFEST_NAME" - name: "Upload artifacts" - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: artifacts-build-global path: | @@ -260,7 +274,7 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive @@ -268,7 +282,7 @@ jobs: run: | rustup target add wasm32-wasip2 cargo install cargo-component --locked || true - - uses: swatinem/rust-cache@v2 + - uses: swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: wasm-extensions - name: Build and package WASM extensions @@ -374,7 +388,7 @@ jobs: echo "=== WASM bundles built ===" ls -la target/wasm-bundles/ - name: "Upload WASM bundles" - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: artifacts-wasm-extensions path: | @@ -390,45 +404,50 @@ jobs: - build-wasm-extensions # Only run if we're "publishing", and only if plan, local, global, and wasm didn't fail (skipped is fine) if: ${{ always() && needs.plan.result == 'success' && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') && (needs.build-wasm-extensions.result == 'skipped' || needs.build-wasm-extensions.result == 'success') }} + permissions: + contents: write env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} runs-on: "ubuntu-22.04" outputs: val: ${{ steps.host.outputs.manifest }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive - name: Install cached dist - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: name: cargo-dist-cache path: ~/.cargo/bin/ - run: chmod +x ~/.cargo/bin/dist # Fetch artifacts from scratch-storage - name: Fetch artifacts - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: artifacts-* path: target/distrib/ merge-multiple: true - id: host shell: bash + env: + TAG_FLAG: ${{ needs.plan.outputs.tag-flag }} run: | - dist host ${{ needs.plan.outputs.tag-flag }} --steps=upload --steps=release --output-format=json > dist-manifest.json + # shellcheck disable=SC2086 # TAG_FLAG may expand to '--tag=X' or empty + dist host $TAG_FLAG --steps=upload --steps=release --output-format=json > dist-manifest.json echo "artifacts uploaded and released successfully" cat dist-manifest.json echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT" - name: "Upload dist-manifest.json" - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: # Overwrite the previous copy name: artifacts-dist-manifest path: dist-manifest.json # Create a GitHub Release while uploading all files to it - name: "Download GitHub Artifacts" - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: artifacts-* path: artifacts @@ -443,11 +462,24 @@ jobs: ANNOUNCEMENT_TITLE: "${{ fromJson(steps.host.outputs.manifest).announcement_title }}" ANNOUNCEMENT_BODY: "${{ fromJson(steps.host.outputs.manifest).announcement_github_body }}" RELEASE_COMMIT: "${{ github.sha }}" + RELEASE_TAG: ${{ needs.plan.outputs.tag }} run: | # Write and read notes from a file to avoid quoting breaking things - echo "$ANNOUNCEMENT_BODY" > $RUNNER_TEMP/notes.txt + echo "$ANNOUNCEMENT_BODY" > "$RUNNER_TEMP/notes.txt" + + # shellcheck disable=SC2086 # PRERELEASE_FLAG is '--prerelease' or empty + gh release create "$RELEASE_TAG" --target "$RELEASE_COMMIT" $PRERELEASE_FLAG --title "$ANNOUNCEMENT_TITLE" --notes-file "$RUNNER_TEMP/notes.txt" artifacts/* - gh release create "${{ needs.plan.outputs.tag }}" --target "$RELEASE_COMMIT" $PRERELEASE_FLAG --title "$ANNOUNCEMENT_TITLE" --notes-file "$RUNNER_TEMP/notes.txt" artifacts/* + # Build and push Docker Hub images (:version, :latest, :sha-*) after the GitHub Release exists. + docker-image: + needs: host + if: ${{ always() && needs.host.result == 'success' }} + permissions: + contents: read + packages: read + actions: write + uses: ./.github/workflows/docker.yml + secrets: inherit # Commit patched manifest SHA256 checksums back to main so the repo # stays in sync with the released artifacts. @@ -464,11 +496,12 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: main + # persist-credentials kept enabled — job pushes a checksum-update branch. - name: Fetch WASM checksums - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: name: artifacts-wasm-extensions path: target/wasm-bundles/ @@ -537,7 +570,7 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: persist-credentials: false submodules: recursive diff --git a/.github/workflows/staging-ci.yml b/.github/workflows/staging-ci.yml index 2df7bf6f70d..5b8cc1abfe1 100644 --- a/.github/workflows/staging-ci.yml +++ b/.github/workflows/staging-ci.yml @@ -15,10 +15,7 @@ on: default: false permissions: - contents: write - issues: write - pull-requests: write - checks: read + contents: read concurrency: group: staging-ci @@ -29,6 +26,9 @@ jobs: resolve-promotion-base: name: Resolve promotion base runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: promotion_base: ${{ steps.resolve.outputs.promotion_base }} steps: @@ -55,16 +55,19 @@ jobs: name: Check for new commits needs: resolve-promotion-base runs-on: ubuntu-latest + permissions: + contents: read outputs: has_changes: ${{ steps.check.outputs.has_changes }} current_head: ${{ steps.check.outputs.current_head }} diff_range: ${{ steps.check.outputs.diff_range }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: - ref: staging + ref: ${{ github.sha }} fetch-depth: 0 fetch-tags: true + persist-credentials: false - name: Check for changes since last tested id: check @@ -117,6 +120,8 @@ jobs: needs: check-changes if: needs.check-changes.outputs.has_changes == 'true' uses: ./.github/workflows/test.yml + with: + ref: ${{ needs.check-changes.outputs.current_head }} # ── Run E2E browser tests ──────────────────────────────────────── e2e: @@ -124,6 +129,8 @@ jobs: needs: check-changes if: needs.check-changes.outputs.has_changes == 'true' uses: ./.github/workflows/e2e.yml + with: + ref: ${{ needs.check-changes.outputs.current_head }} # ── Create promotion PR (triggers claude-review.yml on the PR) ── create-promotion-pr: @@ -131,22 +138,26 @@ jobs: needs: [resolve-promotion-base, check-changes] if: needs.check-changes.outputs.has_changes == 'true' runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write outputs: pr_number: ${{ steps.create-pr.outputs.pr_number }} promotion_branch: ${{ steps.branch.outputs.branch }} steps: - - uses: actions/checkout@v6 - with: - ref: staging - fetch-depth: 0 - - name: Generate GitHub App token id: app-token - uses: actions/create-github-app-token@v2 + uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2 with: app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }} private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }} + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ needs.check-changes.outputs.current_head }} + fetch-depth: 0 + token: ${{ steps.app-token.outputs.token }} + - name: Set token id: token run: | @@ -163,7 +174,7 @@ jobs: PROMOTION_BASE: ${{ needs.resolve-promotion-base.outputs.promotion_base }} run: | git fetch origin "${PROMOTION_BASE}" - AHEAD=$(git rev-list --count "origin/${PROMOTION_BASE}..origin/staging") + AHEAD=$(git rev-list --count "origin/${PROMOTION_BASE}..HEAD") echo "commits_ahead=${AHEAD}" >> "$GITHUB_OUTPUT" if [ "$AHEAD" -eq 0 ]; then echo "Staging is not ahead of ${PROMOTION_BASE}. Nothing to promote." @@ -247,18 +258,24 @@ jobs: needs.create-promotion-pr.result == 'success' runs-on: ubuntu-latest timeout-minutes: 25 + permissions: + contents: write + pull-requests: write + issues: write + checks: read outputs: gate_passed: ${{ steps.evaluate.outputs.passed }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: ref: staging # Need full history to recompute the final promoted range before merge. fetch-depth: 0 + persist-credentials: false - name: Generate GitHub App token id: app-token - uses: actions/create-github-app-token@v2 + uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2 with: app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }} private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }} @@ -489,11 +506,14 @@ jobs: needs.e2e.result == 'success' && needs.create-promotion-pr.result == 'success' runs-on: ubuntu-latest + permissions: + contents: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: ref: staging fetch-depth: 0 + # persist-credentials kept enabled — job pushes the staging-tested tag. - name: Update staging-tested tag run: | @@ -507,6 +527,8 @@ jobs: needs: [check-changes, tests, e2e, create-promotion-pr, gate, update-tag] if: always() && needs.check-changes.outputs.has_changes == 'true' runs-on: ubuntu-latest + permissions: + contents: read steps: - name: Summary run: | diff --git a/.github/workflows/staging-promotion-metadata.yml b/.github/workflows/staging-promotion-metadata.yml index 76b8326b29c..3017e97061a 100644 --- a/.github/workflows/staging-promotion-metadata.yml +++ b/.github/workflows/staging-promotion-metadata.yml @@ -20,7 +20,6 @@ on: permissions: contents: read - pull-requests: write jobs: refresh-single-pr: @@ -30,15 +29,19 @@ jobs: startsWith(github.event.pull_request.head.ref, 'staging-promote/')) || github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: write steps: - name: Checkout workflow source - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: # For chained promotion PRs, the script lives on the trusted PR head, # not necessarily on the older promotion branch used as the PR base. ref: ${{ github.event_name == 'workflow_dispatch' && 'main' || github.event.pull_request.head.sha }} fetch-depth: 0 fetch-tags: true + persist-credentials: false - name: Refresh staging promotion PR body env: @@ -51,13 +54,17 @@ jobs: refresh-open-prs-after-main-push: if: github.event_name == 'push' runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: write steps: - name: Checkout main - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: ref: main fetch-depth: 0 fetch-tags: true + persist-credentials: false - name: Refresh all open staging promotion PR bodies env: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 00488c70fca..67aef9f1b62 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1,6 +1,11 @@ name: Run Tests on: workflow_call: + inputs: + ref: + description: Commit SHA or ref to test + required: false + type: string pull_request: branches: - main @@ -8,10 +13,14 @@ on: branches: - main +permissions: + contents: read + jobs: tests: name: Tests (${{ matrix.name }}) runs-on: ubuntu-latest + timeout-minutes: 45 strategy: fail-fast: false matrix: @@ -27,12 +36,15 @@ jobs: flags: "--no-default-features --features libsql" steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: targets: wasm32-wasip2 - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: ${{ matrix.name }} - name: Install cargo-component @@ -40,27 +52,37 @@ jobs: - name: Build WASM channels (for integration tests) run: ./scripts/build-wasm-extensions.sh --channels - name: Run Tests - run: cargo test ${{ matrix.flags }} -- --nocapture + run: | + timeout --signal=INT --kill-after=30s 40m \ + cargo test ${{ matrix.flags }} -- --nocapture heavy-integration-tests: name: Heavy Integration Tests runs-on: ubuntu-latest + timeout-minutes: 20 steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: targets: wasm32-wasip2 - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: heavy-integration - name: Build Telegram WASM channel run: cargo build --manifest-path channels-src/telegram/Cargo.toml --target wasm32-wasip2 --release - name: Run thread scheduling integration tests - run: cargo test --no-default-features --features libsql,integration --test e2e_thread_scheduling -- --nocapture + run: | + timeout --signal=INT --kill-after=30s 15m \ + cargo test --no-default-features --features libsql,integration --test e2e_thread_scheduling -- --nocapture - name: Run Telegram thread-scope regression test - run: cargo test --features integration --test telegram_auth_integration test_private_messages_use_chat_id_as_thread_scope -- --exact + run: | + timeout --signal=INT --kill-after=30s 10m \ + cargo test --features integration --test telegram_auth_integration test_private_messages_use_chat_id_as_thread_scope -- --exact telegram-tests: name: Telegram Channel Tests @@ -68,14 +90,20 @@ jobs: github.event_name != 'pull_request' || github.base_ref != 'staging' runs-on: ubuntu-latest + timeout-minutes: 15 steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable - - uses: Swatinem/rust-cache@v2 + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 - name: Run Telegram Channel Tests - run: cargo test --manifest-path channels-src/telegram/Cargo.toml -- --nocapture + run: | + timeout --signal=INT --kill-after=30s 10m \ + cargo test --manifest-path channels-src/telegram/Cargo.toml -- --nocapture windows-build: name: Windows Build (${{ matrix.name }}) @@ -95,10 +123,13 @@ jobs: flags: "--no-default-features --features libsql" steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable - - uses: Swatinem/rust-cache@v2 + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: windows-${{ matrix.name }} - name: Check compilation @@ -110,14 +141,18 @@ jobs: github.event_name != 'pull_request' || github.base_ref != 'staging' runs-on: ubuntu-latest + timeout-minutes: 30 steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable with: targets: wasm32-wasip2 - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: wasm-extensions - name: Install cargo-component @@ -125,17 +160,22 @@ jobs: - name: Build all WASM extensions against current WIT run: ./scripts/build-wasm-extensions.sh - name: Instantiation test (host linker compatibility) - run: cargo test --all-features wit_compat -- --nocapture + run: | + timeout --signal=INT --kill-after=30s 20m \ + cargo test --all-features wit_compat -- --nocapture bench-compile: name: Benchmark Compilation runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Install Rust - uses: dtolnay/rust-toolchain@stable - - uses: Swatinem/rust-cache@v2 + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: key: bench - name: Compile benchmarks @@ -149,9 +189,12 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false - name: Build Docker image - run: docker build -t ironclaw-test:ci . + run: docker build --target runtime -t ironclaw-test:ci . version-check: name: Version Bump Check @@ -159,8 +202,10 @@ jobs: if: github.event_name == 'pull_request' steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: + ref: ${{ inputs.ref || github.sha }} + persist-credentials: false fetch-depth: 0 - name: Check version bumps for changed extensions env: diff --git a/.gitignore b/.gitignore index 2577b4a278d..d83d6b97b96 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,7 @@ target/ # Python __pycache__/ *.pyc +/tests/e2e/.venv/ # Benchmark results (local runs, not committed) bench-results/ @@ -39,3 +40,4 @@ __pycache__/ *.pyc *.pyo *.pyd +engine_trace_*.json diff --git a/AGENTS.md b/AGENTS.md index cc5e7cff5d6..90ad4bf5229 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,6 +77,7 @@ Start with these deeper docs as needed: - If you change implementation status for any feature tracked in `FEATURE_PARITY.md`, update that file in the same branch. - Do not open a PR that changes feature behavior without checking `FEATURE_PARITY.md` for needed status updates (`❌`, `🚧`, `✅`, notes, and priorities). - Add the narrowest tests that validate the change: unit tests for local logic, integration tests for runtime/DB/routing behavior, and E2E or trace coverage for gateway, approvals, extensions, or other user-visible flows. +- **Test through the caller, not just the helper.** When a predicate/classifier/transform helper gates a side effect (HTTP, DB write, OAuth flow, UI mutation, tool execution) and has any wrapper or computed input between it and that side effect, a unit test on the helper alone is not sufficient regression coverage. Add a test that drives the actual call site (`*_handler`, `factory::create_*`, `manager::*`) at the integration tier or higher. Mocks of multi-arg runtime APIs must capture every argument the production caller passes. See `.claude/rules/testing.md` for the full rule and bug examples. ## Risk and Change Discipline diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aad499357c..589ee6d9bde 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,324 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.25.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.24.0...ironclaw-v0.25.0) - 2026-04-11 + +### Added + +- *(tools)* production-grade coding tools, file history, and skills ([#2025](https://github.com/nearai/ironclaw/pull/2025)) +- add extensible deployment profiles (IRONCLAW_PROFILE) ([#2203](https://github.com/nearai/ironclaw/pull/2203)) +- *(skills)* commitments system — active intake for personal AI assistant ([#1736](https://github.com/nearai/ironclaw/pull/1736)) +- add native Composio tool for third-party app integrations ([#920](https://github.com/nearai/ironclaw/pull/920)) +- *(gateway)* extract gateway frontend into ironclaw_gateway crate with widget system ([#1725](https://github.com/nearai/ironclaw/pull/1725)) +- *(railway)* build staging target with pre-bundled WASM extensions ([#2219](https://github.com/nearai/ironclaw/pull/2219)) +- *(docker)* pre-bundle WASM extensions in staging image ([#2210](https://github.com/nearai/ironclaw/pull/2210)) +- *(tui)* ship TUI in default binary ([#2195](https://github.com/nearai/ironclaw/pull/2195)) +- *(admin)* admin tool policy to disable tools for users ([#2154](https://github.com/nearai/ironclaw/pull/2154)) +- *(web)* add scroll-to-bottom arrow in gateway chat ([#2202](https://github.com/nearai/ironclaw/pull/2202)) +- unified tool dispatch + schema-validated workspace ([#2049](https://github.com/nearai/ironclaw/pull/2049)) +- *(workspace)* admin system prompt shared with all users ([#2109](https://github.com/nearai/ironclaw/pull/2109)) +- *(engine)* restage skill repair learning loop on staging ([#1962](https://github.com/nearai/ironclaw/pull/1962)) +- *(tui)* port full-featured Ratatui terminal UI onto staging ([#1973](https://github.com/nearai/ironclaw/pull/1973)) +- *(slack)* implement on_broadcast and fix message tool hints ([#2113](https://github.com/nearai/ironclaw/pull/2113)) +- *(i18n)* add Korean translation, fix zh-CN drift, and prevent future drift via pre-commit hook ([#2065](https://github.com/nearai/ironclaw/pull/2065)) +- NEAR AI MCP server ([#2009](https://github.com/nearai/ironclaw/pull/2009)) +- *(test)* dual-mode live/replay test harness with LLM judge ([#2039](https://github.com/nearai/ironclaw/pull/2039)) +- add AWS Bedrock embeddings provider ([#1568](https://github.com/nearai/ironclaw/pull/1568)) +- *(ownership)* centralized ownership model with typed identities, DB-backed pairing, and OwnershipCache ([#1898](https://github.com/nearai/ironclaw/pull/1898)) +- *(tools)* persistent per-user tool permission system ([#1911](https://github.com/nearai/ironclaw/pull/1911)) +- *(engine)* Unified Thread-Capability-CodeAct execution engine (v2 architecture) ([#1557](https://github.com/nearai/ironclaw/pull/1557)) +- *(auth)* direct OAuth/social login with Google, GitHub, Apple, and NEAR wallet ([#1798](https://github.com/nearai/ironclaw/pull/1798)) +- Add ACP (Agent Client Protocol) job mode for delegating to any compatible coding agent ([#1600](https://github.com/nearai/ironclaw/pull/1600)) +- *(workspace)* metadata-driven indexing/hygiene, document versioning, and patch ([#1723](https://github.com/nearai/ironclaw/pull/1723)) +- *(jobs)* per-job MCP server filtering and max_iterations cap ([#1243](https://github.com/nearai/ironclaw/pull/1243)) +- *(config)* unify all settings to DB > env > default priority ([#1722](https://github.com/nearai/ironclaw/pull/1722)) +- *(telegram)* add sendVoice support for audio/ogg attachments ([#1314](https://github.com/nearai/ironclaw/pull/1314)) +- *(setup)* build ironclaw-worker Docker image in setup wizard ([#1757](https://github.com/nearai/ironclaw/pull/1757)) + +### Fixed + +- *(ci)* bump 5 channel versions + fix lifetime desync in panics check ([#2300](https://github.com/nearai/ironclaw/pull/2300)) +- *(test)* case-insensitive hint matching in TraceLlm step_matches ([#2292](https://github.com/nearai/ironclaw/pull/2292)) +- *(v2)* tool naming, auth gates, schema flatten, WASM traps, workspace race ([#2209](https://github.com/nearai/ironclaw/pull/2209)) +- *(ci)* resolve 4 staging test failures ([#2273](https://github.com/nearai/ironclaw/pull/2273)) +- *(docker)* copy profiles/ into build stages ([#2289](https://github.com/nearai/ironclaw/pull/2289)) +- *(engine)* mission cron scheduling + timezone propagation ([#1944](https://github.com/nearai/ironclaw/pull/1944)) ([#1957](https://github.com/nearai/ironclaw/pull/1957)) +- *(oauth)* use localhost for redirect URI when bound to 0.0.0.0 ([#2247](https://github.com/nearai/ironclaw/pull/2247)) +- *(bridge)* sanitize auth_url on engine v2 path ([#2206](https://github.com/nearai/ironclaw/pull/2206)) ([#2215](https://github.com/nearai/ironclaw/pull/2215)) +- *(docs)* explain in more details `activation` block & installation steps for skills ([#2216](https://github.com/nearai/ironclaw/pull/2216)) +- *(docker)* consume CACHE_BUST arg so BuildKit invalidates cache +- *(gateway)* suppress duplicate text response during auth flow and unify extension config modal ([#2172](https://github.com/nearai/ironclaw/pull/2172)) +- *(agent)* stop intercepting bare yes/no/always as approval when nothing pending ([#2178](https://github.com/nearai/ironclaw/pull/2178)) +- *(ci)* resolve 3 staging test failures ([#2207](https://github.com/nearai/ironclaw/pull/2207)) +- *(wasm)* upgrade Wasmtime to 43.0.1 and restore CI ([#2224](https://github.com/nearai/ironclaw/pull/2224)) +- fix(auth) first-pass Gmail OAuth auth prompt in chat ([#2038](https://github.com/nearai/ironclaw/pull/2038)) +- *(db)* repair V6 migration checksum and guard against re-modification ([#1328](https://github.com/nearai/ironclaw/pull/1328)) ([#2101](https://github.com/nearai/ironclaw/pull/2101)) +- *(ci)* target wasm32-wasip2 in WASM build script ([#2175](https://github.com/nearai/ironclaw/pull/2175)) +- *(test)* use canonical extension name in setup submit test ([#2158](https://github.com/nearai/ironclaw/pull/2158)) +- fix (skills) installs for invalid catalog names ([#2040](https://github.com/nearai/ironclaw/pull/2040)) +- universal engine-version tool visibility filtering ([#2132](https://github.com/nearai/ironclaw/pull/2132)) +- *(ownership)* remove silent cross-tenant credential fallback ([#2099](https://github.com/nearai/ironclaw/pull/2099)) +- *(e2e)* canonicalize extension names + fix remaining test failures ([#2129](https://github.com/nearai/ironclaw/pull/2129)) +- *(ownership)* unify ownership checks via Owned trait and fix mission visibility bug ([#2126](https://github.com/nearai/ironclaw/pull/2126)) +- *(web)* intercept approval text input in chat ([#2124](https://github.com/nearai/ironclaw/pull/2124)) +- *(staging)* repair 4 categories of CI test failures ([#2091](https://github.com/nearai/ironclaw/pull/2091)) +- *(web)* emit Done after response — SSE ordering fix ([#2079](https://github.com/nearai/ironclaw/pull/2079)) ([#2104](https://github.com/nearai/ironclaw/pull/2104)) +- *(tools)* gate claude_code and acp modes behind enabled flags ([#2003](https://github.com/nearai/ironclaw/pull/2003)) +- *(acp)* propagate follow-up prompt failures as job errors ([#1981](https://github.com/nearai/ironclaw/pull/1981)) +- color for tools use ([#2096](https://github.com/nearai/ironclaw/pull/2096)) +- *(registry)* use canonical underscore names in manifests to fix WASM install ([#2029](https://github.com/nearai/ironclaw/pull/2029)) +- *(safety)* add credential patterns and sensitive path blocklist ([#1675](https://github.com/nearai/ironclaw/pull/1675)) +- *(channels)* allow telegram wasm channel name ([#2051](https://github.com/nearai/ironclaw/pull/2051)) +- *(staging)* repair broken test build and macOS-incompatible SSRF tests ([#2064](https://github.com/nearai/ironclaw/pull/2064)) +- honor auto-approve tools in engine v2 ([#2013](https://github.com/nearai/ironclaw/pull/2013)) +- *(bridge)* sanitize orphaned tool results in v2 adapter ([#1975](https://github.com/nearai/ironclaw/pull/1975)) +- *(docker)* ensure ironclaw runtime home exists ([#1918](https://github.com/nearai/ironclaw/pull/1918)) +- *(agent)* prevent self-repair notification spam for stuck jobs ([#1867](https://github.com/nearai/ironclaw/pull/1867)) +- *(self-repair)* skip built-in tools in broken tool detection and repair ([#1991](https://github.com/nearai/ironclaw/pull/1991)) +- unblock bootstrap ownership on dynamic_tools ([#2005](https://github.com/nearai/ironclaw/pull/2005)) +- *(llm)* invert reasoning default — unknown models skip think/final tags ([#1952](https://github.com/nearai/ironclaw/pull/1952)) +- *(llm)* add sanitize_tool_messages to OpenAiCodexProvider ([#1971](https://github.com/nearai/ironclaw/pull/1971)) +- update CLI help snapshots for --auto-approve and acp command ([#1966](https://github.com/nearai/ironclaw/pull/1966)) +- *(docker)* switch to glibc to fix libSQL segfault on DB reopen ([#1930](https://github.com/nearai/ironclaw/pull/1930)) +- *(db)* swap V16/V17 to match production PG (document_versions before user_identities) ([#1931](https://github.com/nearai/ironclaw/pull/1931)) +- *(db)* keep V15=conversation_source_channel to match production PG ([#1928](https://github.com/nearai/ironclaw/pull/1928)) +- *(db)* resolve V15 migration numbering conflict ([#1923](https://github.com/nearai/ironclaw/pull/1923)) +- *(routines)* add bounded retry for transient lightweight failures ([#1471](https://github.com/nearai/ironclaw/pull/1471)) +- *(relay)* thread responses under original message in Slack channels ([#1848](https://github.com/nearai/ironclaw/pull/1848)) +- *(worker)* Improve command execution parameter validation ([#1692](https://github.com/nearai/ironclaw/pull/1692)) +- *(telegram)* auto-generate webhook secret during setup ([#1536](https://github.com/nearai/ironclaw/pull/1536)) +- *(builder)* accept inline-table and object-map dependency formats from LLM ([#1748](https://github.com/nearai/ironclaw/pull/1748)) +- *(gemini)* preserve and echo thoughtSignature for Gemini 3.x function calls ([#1752](https://github.com/nearai/ironclaw/pull/1752)) +- *(relay)* route async Slack messages to correct channel instead of DMs ([#1845](https://github.com/nearai/ironclaw/pull/1845)) +- *(security)* block cross-channel approval thread hijacking ([#1590](https://github.com/nearai/ironclaw/pull/1590)) +- *(builder)* add approval context propagation for sub-tool execution ([#1125](https://github.com/nearai/ironclaw/pull/1125)) + +### Other + +- trigger ironclaw-dind image build ([#2190](https://github.com/nearai/ironclaw/pull/2190)) +- add amazon tutorial ([#2261](https://github.com/nearai/ironclaw/pull/2261)) +- Create QA Bug Report issue template ([#2228](https://github.com/nearai/ironclaw/pull/2228)) +- [codex] Stabilize auth readiness and gate flows ([#2050](https://github.com/nearai/ironclaw/pull/2050)) +- Add mintlify docs ([#2189](https://github.com/nearai/ironclaw/pull/2189)) +- [codex] allow private local llm endpoints ([#1955](https://github.com/nearai/ironclaw/pull/1955)) +- *(ci)* add Dependabot and pin GitHub Actions by SHA ([#2043](https://github.com/nearai/ironclaw/pull/2043)) +- Fix routine Telegram notification summaries ([#2033](https://github.com/nearai/ironclaw/pull/2033)) +- *(channels)* add Slack E2E tests, integration tests, and smoke runner ([#2042](https://github.com/nearai/ironclaw/pull/2042)) +- *(engine)* rename ENGINE_V2_TRACE to IRONCLAW_RECORD_TRACE ([#2114](https://github.com/nearai/ironclaw/pull/2114)) +- fix multi-tenant inference latency (per-conversation locking + workspace indexing) ([#2127](https://github.com/nearai/ironclaw/pull/2127)) +- Improve channel onboarding and Telegram pairing flow ([#2103](https://github.com/nearai/ironclaw/pull/2103)) +- *(e2e)* expand SSE resilience coverage ([#1897](https://github.com/nearai/ironclaw/pull/1897)) +- add Telegram E2E tests and Rust integration tests ([#2037](https://github.com/nearai/ironclaw/pull/2037)) +- (fix) WASM channel HTTP SSRF protections ([#1976](https://github.com/nearai/ironclaw/pull/1976)) +- Ignore default model override and empty WASM polls ([#1914](https://github.com/nearai/ironclaw/pull/1914)) +- *(workspace)* add direct regression tests for scoped_to_user rebinding ([#1652](https://github.com/nearai/ironclaw/pull/1652)) ([#1875](https://github.com/nearai/ironclaw/pull/1875)) +- Fix turn cost footer and per-turn usage accounting ([#1951](https://github.com/nearai/ironclaw/pull/1951)) +- Publish ironclaw-worker image from Dockerfile.worker ([#1979](https://github.com/nearai/ironclaw/pull/1979)) +- [codex] Move safety benches into ironclaw_safety crate ([#1954](https://github.com/nearai/ironclaw/pull/1954)) +- Fix bootstrap paths and webhook defaults +- Only tag :latest/:version on release, allow :staging via manual dispatch [skip-regression-check] ([#1925](https://github.com/nearai/ironclaw/pull/1925)) +- Add Docker Hub workflow and optimize Dockerfile for size ([#1886](https://github.com/nearai/ironclaw/pull/1886)) +- *(e2e)* add agent loop recovery coverage ([#1854](https://github.com/nearai/ironclaw/pull/1854)) +- disable cooldown in gateway webhook workflow test ([#1889](https://github.com/nearai/ironclaw/pull/1889)) +- Expand GitHub WASM tool surface ([#1884](https://github.com/nearai/ironclaw/pull/1884)) +- *(e2e)* cover chat approval parity across channels ([#1858](https://github.com/nearai/ironclaw/pull/1858)) +- add routine coverage for issue 1781 ([#1856](https://github.com/nearai/ironclaw/pull/1856)) + +## [0.24.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.23.0...ironclaw-v0.24.0) - 2026-03-31 + +### Added + +- *(gateway)* OIDC JWT authentication for reverse-proxy deployments ([#1463](https://github.com/nearai/ironclaw/pull/1463)) +- support custom LLM provider configuration via web UI ([#1340](https://github.com/nearai/ironclaw/pull/1340)) +- *(skills)* recursive bundle directory scanning for skill discovery ([#1667](https://github.com/nearai/ironclaw/pull/1667)) +- *(discord)* add gateway channel flow in wasm ([#944](https://github.com/nearai/ironclaw/pull/944)) +- DB-backed user management, admin secrets provisioning, and multi-tenant isolation ([#1626](https://github.com/nearai/ironclaw/pull/1626)) +- *(gateway)* add OpenAI Responses API endpoints ([#1656](https://github.com/nearai/ironclaw/pull/1656)) + +### Fixed + +- *(routines)* clone Arc before await in web handler event cache refresh ([#1756](https://github.com/nearai/ironclaw/pull/1756)) +- *(slack)* respond to thread replies without requiring @mention ([#1405](https://github.com/nearai/ironclaw/pull/1405)) +- resolve 11 test failures from multi-tenant bootstrap and sandbox gate regressions ([#1746](https://github.com/nearai/ironclaw/pull/1746)) +- *(auth)* make shared Google tool status scope-aware ([#1532](https://github.com/nearai/ironclaw/pull/1532)) +- *(wasm)* inject Content-Length: 0 for bodyless mutating HTTP requests ([#1529](https://github.com/nearai/ironclaw/pull/1529)) +- *(bedrock)* strip tool blocks from messages when toolConfig is absent ([#1630](https://github.com/nearai/ironclaw/pull/1630)) +- prevent UTF-8 panics in byte-index string truncation ([#1688](https://github.com/nearai/ironclaw/pull/1688)) +- *(gemini)* preserve thought signatures on all tool calls ([#1565](https://github.com/nearai/ironclaw/pull/1565)) +- pin staging ci jobs to a single tested sha ([#1628](https://github.com/nearai/ironclaw/pull/1628)) +- *(routines)* complete full_job execution reliability overhaul ([#1650](https://github.com/nearai/ironclaw/pull/1650)) +- *(worker)* treat empty LLM response after text output as completion ([#1677](https://github.com/nearai/ironclaw/pull/1677)) +- *(worker)* replace script -qfc with pty-process for injection-safe PTY ([#1678](https://github.com/nearai/ironclaw/pull/1678)) +- *(web)* redact database error details from API responses ([#1711](https://github.com/nearai/ironclaw/pull/1711)) +- *(oauth)* tighten legacy state validation and fallback handling ([#1701](https://github.com/nearai/ironclaw/pull/1701)) +- *(db)* add tracing warn for naive timestamp fallback and improve parse_timestamp tests ([#1700](https://github.com/nearai/ironclaw/pull/1700)) +- *(wasm)* use typed WASM schema as advertised schema when available ([#1699](https://github.com/nearai/ironclaw/pull/1699)) +- sanitize tool error results before llm injection ([#1639](https://github.com/nearai/ironclaw/pull/1639)) +- require Feishu webhook authentication ([#1638](https://github.com/nearai/ironclaw/pull/1638)) +- *(llm)* prevent UTF-8 panic in line_bounds() (fixes #1669) ([#1679](https://github.com/nearai/ironclaw/pull/1679)) +- downgrade excessive debug logging in hot path (closes #1686) ([#1694](https://github.com/nearai/ironclaw/pull/1694)) + +### Other + +- Stabilize MCP refresh regression tests ([#1772](https://github.com/nearai/ironclaw/pull/1772)) +- Fix hosted MCP OAuth refresh flow ([#1767](https://github.com/nearai/ironclaw/pull/1767)) +- Track routine verification state across updates ([#1716](https://github.com/nearai/ironclaw/pull/1716)) +- *(e2e)* align WASM reinstall expectation with uninstall cleanup ([#1762](https://github.com/nearai/ironclaw/pull/1762)) +- Handle empty tool completions in autonomous jobs ([#1720](https://github.com/nearai/ironclaw/pull/1720)) +- Clarify message tool vs channel setup guidance ([#1715](https://github.com/nearai/ironclaw/pull/1715)) +- tighten contribution and PR guidance ([#1704](https://github.com/nearai/ironclaw/pull/1704)) +- Clean up extension credentials on uninstall ([#1718](https://github.com/nearai/ironclaw/pull/1718)) + +## [0.23.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.22.0...ironclaw-v0.23.0) - 2026-03-27 + +### Added + +- complete multi-tenant isolation — phases 2–4 ([#1614](https://github.com/nearai/ironclaw/pull/1614)) + +### Fixed + +- *(routines)* recover delete name after failed update fallback ([#1108](https://github.com/nearai/ironclaw/pull/1108)) +- *(mcp)* handle 202 Accepted and wire session manager for Streamable HTTP ([#1437](https://github.com/nearai/ironclaw/pull/1437)) +- *(extensions)* channel-relay auth dead-end, observability, and URL override ([#1681](https://github.com/nearai/ironclaw/pull/1681)) +- *(agent)* discard truncated tool calls when finish_reason == Length ([#1631](https://github.com/nearai/ironclaw/pull/1631)) ([#1632](https://github.com/nearai/ironclaw/pull/1632)) +- *(llm)* filter XML tool-call recovery by context ([#1641](https://github.com/nearai/ironclaw/pull/1641)) + +### Other + +- Support direct hosted OAuth callbacks with proxy auth token ([#1684](https://github.com/nearai/ironclaw/pull/1684)) + +## [0.22.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.21.0...ironclaw-v0.22.0) - 2026-03-25 + +### Added + +- *(agent)* thread per-tool reasoning through provider, session, and all surfaces ([#1513](https://github.com/nearai/ironclaw/pull/1513)) +- *(cli)* show credential auth status in tool info ([#1572](https://github.com/nearai/ironclaw/pull/1572)) +- multi-tenant auth with per-user workspace isolation ([#1118](https://github.com/nearai/ironclaw/pull/1118)) +- *(cli)* add ironclaw models subcommands (list/status/set/set-provider) ([#1043](https://github.com/nearai/ironclaw/pull/1043)) +- *(workspace)* multi-scope workspace reads ([#1117](https://github.com/nearai/ironclaw/pull/1117)) +- *(ux)* complete UX overhaul — design system, onboarding, web polish ([#1277](https://github.com/nearai/ironclaw/pull/1277)) +- *(gemini_oauth)* full Gemini CLI OAuth integration with Cloud Code API ([#1356](https://github.com/nearai/ironclaw/pull/1356)) +- *(shell)* add Low/Medium/High risk levels for graduated command approval (closes #172) ([#368](https://github.com/nearai/ironclaw/pull/368)) +- *(agent)* queue and merge messages during active turns ([#1412](https://github.com/nearai/ironclaw/pull/1412)) +- *(cli)* add `ironclaw hooks list` subcommand ([#1023](https://github.com/nearai/ironclaw/pull/1023)) +- *(extensions)* support text setup fields in web configure modal ([#496](https://github.com/nearai/ironclaw/pull/496)) +- *(llm)* add GitHub Copilot as LLM provider ([#1512](https://github.com/nearai/ironclaw/pull/1512)) +- *(workspace)* layered memory with sensitivity-based privacy redirect ([#1112](https://github.com/nearai/ironclaw/pull/1112)) +- *(webhooks)* add public webhook trigger endpoint for routines ([#736](https://github.com/nearai/ironclaw/pull/736)) +- *(llm)* Add OpenAI Codex (ChatGPT subscription) as LLM provider ([#1461](https://github.com/nearai/ironclaw/pull/1461)) +- *(web)* add light theme with dark/light/system toggle ([#1457](https://github.com/nearai/ironclaw/pull/1457)) +- *(agent)* activate stuck_threshold for time-based stuck job detection ([#1234](https://github.com/nearai/ironclaw/pull/1234)) +- chat onboarding and routine advisor ([#927](https://github.com/nearai/ironclaw/pull/927)) + +### Fixed + +- ensure LLM calls always end with user message (closes #763) ([#1259](https://github.com/nearai/ironclaw/pull/1259)) +- restore owner-scoped gateway startup ([#1625](https://github.com/nearai/ironclaw/pull/1625)) +- remove stale stream_token gate from channel-relay activation ([#1623](https://github.com/nearai/ironclaw/pull/1623)) +- *(agent)* case-insensitive channel match and user_id filter for event triggers ([#1211](https://github.com/nearai/ironclaw/pull/1211)) +- *(routines)* normalize status display across web and CLI ([#1469](https://github.com/nearai/ironclaw/pull/1469)) +- *(tunnel)* managed tunnels target wrong port and die from SIGPIPE ([#1093](https://github.com/nearai/ironclaw/pull/1093)) +- *(agent)* persist /model selection to .env, TOML, and DB ([#1581](https://github.com/nearai/ironclaw/pull/1581)) +- post-merge review sweep — 8 fixes across security, perf, and correctness ([#1550](https://github.com/nearai/ironclaw/pull/1550)) +- generate Mistral-compatible 9-char alphanumeric tool call IDs ([#1242](https://github.com/nearai/ironclaw/pull/1242)) +- *(mcp)* handle empty 202 notification acknowledgements ([#1539](https://github.com/nearai/ironclaw/pull/1539)) +- *(tests)* eliminate env mutex poison cascade ([#1558](https://github.com/nearai/ironclaw/pull/1558)) +- *(safety)* escape tool output XML content and remove misleading sanitized attr ([#1067](https://github.com/nearai/ironclaw/pull/1067)) +- *(oauth)* reject malformed ic2.* states in decode_hosted_oauth_state ([#1441](https://github.com/nearai/ironclaw/pull/1441)) ([#1454](https://github.com/nearai/ironclaw/pull/1454)) +- parameter coercion and validation for oneOf/anyOf/allOf schemas ([#1397](https://github.com/nearai/ironclaw/pull/1397)) +- persist startup-loaded MCP clients in ExtensionManager ([#1509](https://github.com/nearai/ironclaw/pull/1509)) +- *(deps)* patch rustls-webpki vulnerability (RUSTSEC-2026-0049) +- *(routines)* add missing extension_manager field in trigger_manual EngineContext +- *(ci)* serialize env-mutating OAuth wildcard tests with ENV_MUTEX ([#1280](https://github.com/nearai/ironclaw/pull/1280)) ([#1468](https://github.com/nearai/ironclaw/pull/1468)) +- *(setup)* remove redundant LLM config and API keys from bootstrap .env ([#1448](https://github.com/nearai/ironclaw/pull/1448)) +- resolve wasm broadcast merge conflicts with staging ([#395](https://github.com/nearai/ironclaw/pull/395)) ([#1460](https://github.com/nearai/ironclaw/pull/1460)) +- skip credential validation for Bedrock backend ([#1011](https://github.com/nearai/ironclaw/pull/1011)) +- register sandbox jobs in ContextManager for query tool visibility ([#1426](https://github.com/nearai/ironclaw/pull/1426)) +- prefer execution-local message routing metadata ([#1449](https://github.com/nearai/ironclaw/pull/1449)) +- *(security)* validate embedding base URLs to prevent SSRF ([#1221](https://github.com/nearai/ironclaw/pull/1221)) +- f32→f64 precision artifact in temperature causes provider 400 errors ([#1450](https://github.com/nearai/ironclaw/pull/1450)) +- *(routines)* surface errors when sandbox unavailable for full_job routines ([#769](https://github.com/nearai/ironclaw/pull/769)) +- restore libSQL vector search with dynamic dimensions ([#1393](https://github.com/nearai/ironclaw/pull/1393)) +- staging CI triage — consolidate retry parsing, fix flaky tests, add docs ([#1427](https://github.com/nearai/ironclaw/pull/1427)) + +### Other + +- Merge branch 'main' into staging-promote/455f543b-23329172268 +- Merge pull request #1655 from nearai/codex/fix-staging-promotion-1451-version-bumps +- Merge pull request #1499 from nearai/staging-promote/9603fefd-23364438978 +- Fix libsql prompt scope regressions ([#1651](https://github.com/nearai/ironclaw/pull/1651)) +- Normalize cron schedules on routine create ([#1648](https://github.com/nearai/ironclaw/pull/1648)) +- Fix MCP lifecycle trace user scope ([#1646](https://github.com/nearai/ironclaw/pull/1646)) +- Fix REPL single-message hang and cap CI test duration ([#1643](https://github.com/nearai/ironclaw/pull/1643)) +- extract AppEvent to crates/ironclaw_common ([#1615](https://github.com/nearai/ironclaw/pull/1615)) +- Fix hosted OAuth refresh via proxy ([#1602](https://github.com/nearai/ironclaw/pull/1602)) +- *(agent)* optimize approval thread resolution (UUID parsing + lock contention) ([#1592](https://github.com/nearai/ironclaw/pull/1592)) +- *(tools)* auto-compact WASM tool schemas, add descriptions, improve credential prompts ([#1525](https://github.com/nearai/ironclaw/pull/1525)) +- Default new lightweight routines to tools-enabled ([#1573](https://github.com/nearai/ironclaw/pull/1573)) +- Google OAuth URL broken when initiated from Telegram channel ([#1165](https://github.com/nearai/ironclaw/pull/1165)) +- add gitcgr code graph badge ([#1563](https://github.com/nearai/ironclaw/pull/1563)) +- Fix owner-scoped message routing fallbacks ([#1574](https://github.com/nearai/ironclaw/pull/1574)) +- *(tools)* remove unconditional params clone in shared execution (fix #893) ([#926](https://github.com/nearai/ironclaw/pull/926)) +- *(llm)* move transcription module into src/llm/ ([#1559](https://github.com/nearai/ironclaw/pull/1559)) +- *(agent)* avoid preview allocations for non-truncated strings (fix #894) ([#924](https://github.com/nearai/ironclaw/pull/924)) +- Expand AGENTS.md with coding agents guidance ([#1392](https://github.com/nearai/ironclaw/pull/1392)) +- Fix CI approval flows and stale fixtures ([#1478](https://github.com/nearai/ironclaw/pull/1478)) +- Use live owner tool scope for autonomous routines and jobs ([#1453](https://github.com/nearai/ironclaw/pull/1453)) +- use Arc in embedding cache to avoid clones on miss path ([#1438](https://github.com/nearai/ironclaw/pull/1438)) +- Add owner-scoped permissions for full-job routines ([#1440](https://github.com/nearai/ironclaw/pull/1440)) + +## [0.21.0](https://github.com/nearai/ironclaw/compare/v0.20.0...v0.21.0) - 2026-03-20 + +### Added + +- structured fallback deliverables for failed/stuck jobs ([#236](https://github.com/nearai/ironclaw/pull/236)) +- LRU embedding cache for workspace search ([#1423](https://github.com/nearai/ironclaw/pull/1423)) +- receive relay events via webhook callbacks ([#1254](https://github.com/nearai/ironclaw/pull/1254)) + +### Fixed + +- bump Feishu channel version for promotion +- *(approval)* make "always" auto-approve work for credentialed HTTP requests ([#1257](https://github.com/nearai/ironclaw/pull/1257)) +- skip NEAR AI session check when backend is not nearai ([#1413](https://github.com/nearai/ironclaw/pull/1413)) + +### Other + +- Make hosted OAuth and MCP auth generic ([#1375](https://github.com/nearai/ironclaw/pull/1375)) + +## [0.20.0](https://github.com/nearai/ironclaw/compare/v0.19.0...v0.20.0) - 2026-03-19 + +### Added + +- *(self-repair)* wire stuck_threshold, store, and builder ([#712](https://github.com/nearai/ironclaw/pull/712)) +- *(testing)* add FaultInjector framework for StubLlm ([#1233](https://github.com/nearai/ironclaw/pull/1233)) +- *(gateway)* unified settings page with subtabs ([#1191](https://github.com/nearai/ironclaw/pull/1191)) +- upgrade MiniMax default model to M2.7 ([#1357](https://github.com/nearai/ironclaw/pull/1357)) + +### Fixed + +- navigate telegram E2E tests to channels subtab ([#1408](https://github.com/nearai/ironclaw/pull/1408)) +- add missing `builder` field and update E2E extensions tab navigation ([#1400](https://github.com/nearai/ironclaw/pull/1400)) +- remove debug_assert guards that panic on valid error paths ([#1385](https://github.com/nearai/ironclaw/pull/1385)) +- address valid review comments from PR #1359 ([#1380](https://github.com/nearai/ironclaw/pull/1380)) +- full_job routine runs stay running until linked job completion ([#1374](https://github.com/nearai/ironclaw/pull/1374)) +- full_job routine concurrency tracks linked job lifetime ([#1372](https://github.com/nearai/ironclaw/pull/1372)) +- remove -x from coverage pytest to prevent suite-blocking failures ([#1360](https://github.com/nearai/ironclaw/pull/1360)) +- add debug_assert invariant guards to critical code paths ([#1312](https://github.com/nearai/ironclaw/pull/1312)) +- *(mcp)* retry after missing session id errors ([#1355](https://github.com/nearai/ironclaw/pull/1355)) +- *(telegram)* preserve polling after secret-blocked updates ([#1353](https://github.com/nearai/ironclaw/pull/1353)) +- *(llm)* cap retry-after delays ([#1351](https://github.com/nearai/ironclaw/pull/1351)) +- *(setup)* remove nonexistent webhook secret command hint ([#1349](https://github.com/nearai/ironclaw/pull/1349)) +- Rate limiter returns retry after None instead of a duration ([#1269](https://github.com/nearai/ironclaw/pull/1269)) + +### Other + +- bump telegram channel version to 0.2.5 ([#1410](https://github.com/nearai/ironclaw/pull/1410)) +- *(ci)* enforce test requirement for state machine and resilience changes ([#1230](https://github.com/nearai/ironclaw/pull/1230)) ([#1304](https://github.com/nearai/ironclaw/pull/1304)) +- Fix duplicate LLM responses for matched event routines ([#1275](https://github.com/nearai/ironclaw/pull/1275)) +- add Japanese README ([#1306](https://github.com/nearai/ironclaw/pull/1306)) +- *(ci)* add coverage gates via codecov.yml ([#1228](https://github.com/nearai/ironclaw/pull/1228)) ([#1291](https://github.com/nearai/ironclaw/pull/1291)) +- Redesign routine create requests for LLMs ([#1147](https://github.com/nearai/ironclaw/pull/1147)) + ## [0.19.0](https://github.com/nearai/ironclaw/compare/v0.18.0...v0.19.0) - 2026-03-17 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index e2d84c1eee3..aa0a47121e9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,6 +24,9 @@ E2E tests: see `tests/e2e/CLAUDE.md`. - Prefer strong types over strings (enums, newtypes) - Keep functions focused, extract helpers when logic is reused - Comments for non-obvious logic only +- **Prompt templates live in files, not Rust code**: Multi-line prompt strings (mission goals, system prompts, CodeAct preambles) go in `crates/ironclaw_engine/prompts/*.md` and are loaded via `include_str!()`. Never inline large prompt templates as Rust string constants — they're hard to read, review, and iterate on. Single-line format strings are fine inline. +- **Logging levels matter for REPL/TUI**: `info!` and `warn!` output appears in the REPL and corrupts the terminal UI. Use `debug!` for internal diagnostics (trace analysis, reflection results, engine internals). Reserve `info!` for user-facing status that the REPL intentionally renders. Background tasks (reflection, trace analysis) must NEVER use `info!` — it breaks the interactive display. +- **Test through the caller, not just the helper**: When a predicate/classifier/transform helper gates a side effect (HTTP, DB write, OAuth, UI mutation, tool execution) and has any wrapper or computed input between it and that side effect, a unit test on the helper alone is *not* sufficient regression coverage. Add a test that drives the call site — typically a `*_handler`, `factory::create_*`, or `manager::*` — at the integration tier (`cargo test --features integration`) or higher. The same applies to test mocks: if you mock a multi-arg runtime API like `window.open(url, target, features)`, the mock must capture every argument the production caller passes. See `.claude/rules/testing.md` ("Test Through the Caller, Not Just the Helper") for the full rule and the bug examples that motivated it. ## Architecture @@ -33,9 +36,11 @@ Key traits for extensibility: `Database`, `Channel`, `Tool`, `LlmProvider`, `Suc All I/O is async with tokio. Use `Arc` for shared state, `RwLock` for concurrent access. +**LLM data is never deleted.** All LLM output — context fed to the model, reasoning, tool calls, messages, events, steps — is the most valuable data in the system. Never strip, truncate, or delete it from the database. Mark with timestamps, make filterable, but always retain. In-memory HashMaps are caches; the database (via Workspace) is the source of truth. "Cleanup" means evicting from in-memory caches, never deleting database rows. + ## Extracted Crates -Safety logic lives in `crates/ironclaw_safety/`. The `src/safety/mod.rs` shim re-exports everything for backward compatibility, but **new code should import from `ironclaw_safety` directly** (e.g. `use ironclaw_safety::SafetyLayer`). When touching a file that still uses `crate::safety::*`, migrate its imports to `ironclaw_safety::*`. +Safety logic lives in `crates/ironclaw_safety/`, skills in `crates/ironclaw_skills/`. **Import directly from the extracted crate** (e.g. `use ironclaw_safety::SafetyLayer`, `use ironclaw_skills::SkillRegistry`). Do not use `crate::safety::` or `crate::skills::` for types that originate in extracted crates — `src/safety/mod.rs` and `src/skills/mod.rs` no longer glob-re-export. Local items defined in those modules (e.g. `crate::skills::attenuate_tools`) are fine. ## Project Structure @@ -191,14 +196,15 @@ When modifying a module with a spec, read the spec first. Code follows spec; spe | `src/setup/` | `src/setup/README.md` | | `src/tools/` | `src/tools/README.md` | | `src/workspace/` | `src/workspace/README.md` | +| `crates/ironclaw_engine/` | `crates/ironclaw_engine/CLAUDE.md` | | `tests/e2e/` | `tests/e2e/CLAUDE.md` | ## Job State Machine ``` Pending -> InProgress -> Completed -> Submitted -> Accepted - \-> Failed - \-> Stuck -> InProgress (recovery) + \ \-> Failed + \-> Failed \-> Stuck -> InProgress (recovery) \-> Failed ``` @@ -221,6 +227,34 @@ See `.env.example` for all environment variables. LLM backends (`nearai`, `opena 3. Add config in `src/config/channels.rs` 4. Wire up in `src/app.rs` channel setup section +## Everything Goes Through Tools + +**Core principle**: all actions originating from gateway handlers, CLI +commands, routine engine, WASM channels, or any other non-agent caller +MUST go through `ToolDispatcher::dispatch()` — never directly through +`state.store`, `workspace`, `extension_manager`, `skill_registry`, or +`session_manager`. + +This gives every UI-initiated mutation the same audit trail +(`ActionRecord`), safety pipeline (param validation, sensitive-param +redaction, output sanitization), and channel-agnostic surface as +agent-initiated tool calls. Channels are interchangeable extensions; +routing through one dispatch function means new channels inherit the +full pipeline for free. + +The pre-commit hook (`scripts/pre-commit-safety.sh`) flags newly-added +lines in handler/CLI files that touch +`state.{store,workspace,extension_manager,skill_registry,session_manager}.*` +directly. Annotate intentional exceptions (rare — usually only read +aggregation across multiple users) with a trailing +`// dispatch-exempt: ` comment on the same line. The check only +sees added lines, so existing untouched code doesn't trip during +incremental migration. + +See `.claude/rules/tools.md` for the full pattern, allowed exemptions, +and migration status. The dispatcher itself lives in +`src/tools/dispatch.rs`. + ## Workspace & Memory Persistent memory with hybrid search (FTS + vector via RRF). Four tools: `memory_search`, `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into system prompt. Heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings. See `src/workspace/README.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1c5c6d88194..51b20d349dd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,6 +10,42 @@ cd ironclaw This installs the Rust toolchain, WASM targets, git hooks, and runs initial checks. +## How to Contribute + +- Bug fixes, docs improvements, and focused cleanup tied to a concrete problem are welcome. +- Search existing issues and PRs before opening a new one to avoid duplicates. +- Keep changes scoped. One bug, one feature, or one documentation improvement per PR. + +### Creating Issues + +Open an issue when you are reporting a bug, proposing a feature, or documenting a gap in behavior. + +For bug reports, include: + +- What you expected to happen +- What actually happened +- Clear reproduction steps +- Relevant logs, screenshots, or error output +- Environment details when they matter (OS, database backend, feature flags, commit/branch) + +For feature requests: + +- Open an issue first before writing code +- Explain the problem being solved, not just the implementation idea +- Wait for maintainer feedback before investing in a large PR + +We require an issue for new features so maintainers can prioritize the work and confirm it fits the roadmap before anyone spends time implementing it. + +### Fixing Bugs + +- Small, targeted bug-fix PRs are welcome +- If there is already an issue, link it in your PR +- If the bug is non-trivial, security-sensitive, or changes behavior across subsystems, open or confirm an issue first so the approach can be aligned before implementation + +### Refactor-Only PRs + +Refactor-only PRs are not accepted from contributors outside the core team. If a refactor is necessary to land a bug fix or approved feature, keep it minimal and clearly tied to that change. + ## Development Workflow ```bash @@ -19,6 +55,45 @@ cargo test # unit tests cargo test --features integration # + PostgreSQL tests ``` +These commands are for day-to-day iteration while you are developing locally. The pre-submission checks below are intentionally stricter and use CI-style flags so you can catch formatting drift and clippy warnings before requesting review. + +## Before You Open a PR + +Run the local validation checks required before requesting a review. These are stricter than the commands for iterative development: + +```bash +cargo fmt --all -- --check +cargo clippy --all --benches --tests --examples --all-features -- -D warnings +cargo build +cargo test +``` + +Also run this when your change touches database-backed or integration behavior: + +```bash +cargo test --features integration +``` + +Before asking for review: + +- Build and exercise the changed path locally, not just the narrowest unit test +- Keep the PR focused and avoid mixing unrelated concerns +- Fill out the PR template with a clear summary, validation notes, and impact assessment +- If your change affects tracked behavior, update `FEATURE_PARITY.md` in the same branch +- If onboarding or setup behavior changes, update the relevant setup docs in the same branch +- If you are using a coding agent and it supports them, run `review-pr` or `pr-shepherd --fix` before opening or updating the PR +- `codex review --base origin/main` is also encouraged before requesting review + +## Review Follow-Through + +Review conversations are author-owned. + +- Address each review comment with a code change or a clear explanation +- Resolve conversations you have handled; leave them open only when reviewer judgment is still needed +- Do not leave review cleanup for maintainers when the follow-through belongs to the author + +If a PR is stale for more than 48 hours after review feedback is posted, maintainers may take over the follow-up work and land the changes needed to accomplish the original PR or issue intent. + ## Code Style - Zero clippy warnings policy @@ -46,7 +121,7 @@ All PRs follow a risk-based review process: | Track | Scope | Requirements | |-------|-------|-------------| | **A** | Docs, tests, chore, dependency bumps | 1 approval + CI green | -| **B** | Features, refactors, new tools/channels | 1 approval + CI green + test evidence | +| **B** | Features, maintainer-requested refactors, new tools/channels | 1 approval + CI green + test evidence | | **C** | Security (`src/safety/`, `src/secrets/`), runtime (`src/agent/`, `src/worker/`), database schema, CI workflows | 2 approvals + rollback plan documented | Select the appropriate track in the PR template based on what your changes touch. @@ -58,3 +133,33 @@ IronClaw uses dual-backend persistence (PostgreSQL + libSQL). All new persistenc ## Adding Dependencies Run `cargo deny check` before adding new dependencies to verify license compatibility and check for known advisories. + +## Document your Changes + +- The folder `/docs` contains user-facing documentation for technical savvy users, developers and operators. It is built with Mintlify and rendered on the website. +- For features, update the relevant capability doc in `docs/capabilities/` +- For channels, update the relevant channel doc in `docs/channels/` +- For extensions / tools, update the relevant doc in `docs/extensions/` +- Core features live in `docs/capabilities` + +In case you want to document the library itself (i.e. reference documentation) for other core contributors, use the `docs/internal/` folder + +If you use your Claude Code to "plan" and want to leave a record of it, use the `docs/plans` folder. + +### Skills +Read the `.claude/skills/mintlify-docs` for guidelines on how to generate documentation with mintlify. + +### Test the Docs +To make sure the documentation still works, do: + +```bash +cd docs +mint dev +``` + +To make sure you did not break any internal links, do: + +```bash +cd docs +mint broken-links +``` diff --git a/Cargo.lock b/Cargo.lock index a813ef2b10b..7c656642c6d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,9 +4,9 @@ version = 4 [[package]] name = "addr2line" -version = "0.24.2" +version = "0.26.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1" +checksum = "59317f77929f0e679d39364702289274de2f0f0b22cbf50b2b8cff2169a0b27a" dependencies = [ "gimli", ] @@ -61,6 +61,37 @@ dependencies = [ "subtle", ] +[[package]] +name = "agent-client-protocol" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c56a59cf6315e99f874d2c1f96c69d2da5ffe0087d211297fc4a41f849770a2" +dependencies = [ + "agent-client-protocol-schema", + "anyhow", + "async-broadcast", + "async-trait", + "derive_more", + "futures", + "log", + "serde", + "serde_json", +] + +[[package]] +name = "agent-client-protocol-schema" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0497b9a95a404e35799904835c57c6f8c69b9d08ccfd3cb5b7d746425cd6789" +dependencies = [ + "anyhow", + "derive_more", + "schemars 1.2.1", + "serde", + "serde_json", + "strum 0.28.0", +] + [[package]] name = "ahash" version = "0.7.8" @@ -80,9 +111,11 @@ checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" dependencies = [ "cfg-if", "const-random", + "getrandom 0.3.4", "once_cell", + "serde", "version_check", - "zerocopy 0.8.42", + "zerocopy 0.8.48", ] [[package]] @@ -123,9 +156,9 @@ checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" [[package]] name = "anstream" -version = "0.6.21" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43d5b281e737544384e969a5ccad3f1cdd24b48086a0fc1b2a5262a26b8f4f4a" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" dependencies = [ "anstyle", "anstyle-parse", @@ -138,15 +171,15 @@ dependencies = [ [[package]] name = "anstyle" -version = "1.0.13" +version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5192cca8006f1fd4f7237516f40fa183bb07f8fbdfedaa0036de5ea9b0b45e78" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" [[package]] name = "anstyle-parse" -version = "0.2.7" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e7644824f0aa2c7b9384579234ef10eb7efb6a0deb83f9630a49594dd9c15c2" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" dependencies = [ "utf8parse", ] @@ -157,7 +190,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -168,7 +201,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -178,21 +211,32 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" [[package]] -name = "ar_archive_writer" -version = "0.5.1" +name = "arbitrary" +version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7eb93bbb63b9c227414f6eb3a0adfddca591a8ce1e9b60661bb08969b87e340b" +checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" dependencies = [ - "object 0.37.3", + "derive_arbitrary", ] [[package]] -name = "arbitrary" -version = "1.4.2" +name = "arboard" +version = "3.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" +checksum = "0348a1c054491f4bfe6ab86a7b6ab1e44e45d899005de92f58b3df180b36ddaf" dependencies = [ - "derive_arbitrary", + "clipboard-win", + "image", + "log", + "objc2", + "objc2-app-kit", + "objc2-core-foundation", + "objc2-core-graphics", + "objc2-foundation", + "parking_lot", + "percent-encoding", + "windows-sys 0.60.2", + "x11rb", ] [[package]] @@ -386,12 +430,51 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "atomic-polyfill" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cf2bce30dfe09ef0bfaef228b9d414faaf7e563035494d7fe092dba54b300f4" +dependencies = [ + "critical-section", +] + [[package]] name = "atomic-waker" version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" +[[package]] +name = "attribute-derive" +version = "0.10.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05832cdddc8f2650cc2cc187cc2e952b8c133a48eb055f35211f61ee81502d77" +dependencies = [ + "attribute-derive-macro", + "derive-where", + "manyhow", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "attribute-derive-macro" +version = "0.10.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0a7cdbbd4bd005c5d3e2e9c885e6fa575db4f4a3572335b974d8db853b6beb61" +dependencies = [ + "collection_literals", + "interpolator", + "manyhow", + "proc-macro-utils", + "proc-macro2", + "quote", + "quote-use", + "syn 2.0.117", +] + [[package]] name = "autocfg" version = "1.5.0" @@ -442,9 +525,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.16.1" +version = "1.16.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94bffc006df10ac2a68c83692d734a465f8ee6c5b384d8545a636f81d858f4bf" +checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc" dependencies = [ "aws-lc-sys", "zeroize", @@ -452,9 +535,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.38.0" +version = "0.39.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4321e568ed89bb5a7d291a7f37997c2c0df89809d7b6d12062c81ddb54aa782e" +checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399" dependencies = [ "cc", "cmake", @@ -490,9 +573,9 @@ dependencies = [ [[package]] name = "aws-sdk-bedrockruntime" -version = "1.127.0" +version = "1.128.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7dcd5ccbed3bd50d342077d3f731de46d9608340386c87d07566c4c507891eda" +checksum = "3949d34a5c329ed83e7146d2fc1ffc06473fdc9bcbc5fa3d3534abeb950569c5" dependencies = [ "aws-credential-types", "aws-runtime", @@ -517,9 +600,9 @@ dependencies = [ [[package]] name = "aws-sdk-sso" -version = "1.96.0" +version = "1.97.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f64a6eded248c6b453966e915d32aeddb48ea63ad17932682774eb026fbef5b1" +checksum = "9aadc669e184501caaa6beafb28c6267fc1baef0810fb58f9b205485ca3f2567" dependencies = [ "aws-credential-types", "aws-runtime", @@ -541,9 +624,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.98.0" +version = "1.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db96d720d3c622fcbe08bae1c4b04a72ce6257d8b0584cb5418da00ae20a344f" +checksum = "1342a7db8f358d3de0aed2007a0b54e875458e39848d54cc1d46700b2bfcb0a8" dependencies = [ "aws-credential-types", "aws-runtime", @@ -565,9 +648,9 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.100.0" +version = "1.101.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fafbdda43b93f57f699c5dfe8328db590b967b8a820a13ccdd6687355dfcc7ca" +checksum = "ab41ad64e4051ecabeea802d6a17845a91e83287e1dd249e6963ea1ba78c428a" dependencies = [ "aws-credential-types", "aws-runtime", @@ -757,9 +840,9 @@ dependencies = [ [[package]] name = "aws-smithy-types" -version = "1.4.6" +version = "1.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2b1117b3b2bbe166d11199b540ceed0d0f7676e36e7b962b5a437a9971eac75" +checksum = "9d73dbfbaa8e4bc57b9045137680b958d274823509a360abfd8e1d514d40c95c" dependencies = [ "base64-simd", "bytes", @@ -964,6 +1047,21 @@ dependencies = [ "which", ] +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + [[package]] name = "bitflags" version = "1.3.2" @@ -976,6 +1074,15 @@ version = "2.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" +[[package]] +name = "bitmaps" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "031043d04099746d8db04daf1fa424b2bc8bd69d92b25962dcde24da39ab64a2" +dependencies = [ + "typenum", +] + [[package]] name = "bitvec" version = "1.0.1" @@ -1083,21 +1190,28 @@ dependencies = [ "serde_with", ] +[[package]] +name = "borrow-or-share" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc0b364ead1874514c8c2855ab558056ebfeb775653e7ae45ff72f28f8f3166c" + [[package]] name = "borsh" -version = "1.6.0" +version = "1.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1da5ab77c1437701eeff7c88d968729e7766172279eab0676857b3d63af7a6f" +checksum = "cfd1e3f8955a5d7de9fab72fc8373fade9fb8a703968cb200ae3dc6cf08e185a" dependencies = [ "borsh-derive", + "bytes", "cfg_aliases", ] [[package]] name = "borsh-derive" -version = "1.6.0" +version = "1.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0686c856aa6aac0c4498f936d7d6a02df690f614c03e4d906d1018062b5c5e2c" +checksum = "bfcfdc083699101d5a7965e49925975f2f55060f94f9a05e7187be95d530ca59" dependencies = [ "once_cell", "proc-macro-crate", @@ -1106,6 +1220,26 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "bs58" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf88ba1141d185c399bee5288d850d63b8369520c1eafc32a0430b5b6c287bf4" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "bstr" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" +dependencies = [ + "memchr", + "regex-automata", + "serde", +] + [[package]] name = "bumpalo" version = "3.20.2" @@ -1137,12 +1271,44 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "bytecount" +version = "0.6.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" + +[[package]] +name = "bytemuck" +version = "1.25.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" +dependencies = [ + "bytemuck_derive", +] + +[[package]] +name = "bytemuck_derive" +version = "1.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "byteorder" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" +[[package]] +name = "byteorder-lite" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495" + [[package]] name = "bytes" version = "1.11.1" @@ -1240,12 +1406,27 @@ dependencies = [ "winx", ] +[[package]] +name = "cassowary" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df8670b8c7b9dae1793364eafadf7239c40d669904660c5960d74cfd80b46a53" + [[package]] name = "cast" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" +[[package]] +name = "castaway" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a" +dependencies = [ + "rustversion", +] + [[package]] name = "cbc" version = "0.1.2" @@ -1257,9 +1438,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.56" +version = "1.2.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aebf35691d1bfb0ac386a69bac2fde4dd276fb618cf8bf4f5318fe285e821bb2" +checksum = "e1e928d4b69e3077709075a938a05ffbedfa53a84c8f766efbf8220bb1ff60e1" dependencies = [ "find-msvc-tools", "jobserver", @@ -1362,9 +1543,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.60" +version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2797f34da339ce31042b27d23607e051786132987f595b02ba4f6a6dffb7030a" +checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" dependencies = [ "clap_builder", "clap_derive", @@ -1372,9 +1553,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.60" +version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24a241312cea5059b13574bb9b3861cabf758b879c15190b37b6d6fd63ab6876" +checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" dependencies = [ "anstream", "anstyle", @@ -1384,18 +1565,18 @@ dependencies = [ [[package]] name = "clap_complete" -version = "4.5.66" +version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c757a3b7e39161a4e56f9365141ada2a6c915a8622c408ab6bb4b5d047371031" +checksum = "19c9f1dde76b736e3681f28cec9d5a61299cbaae0fce80a68e43724ad56031eb" dependencies = [ "clap", ] [[package]] name = "clap_derive" -version = "4.5.55" +version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a92793da1a46a5f2a02a6f4c46c6496b28c43638adea8306fcb0caa1634f24e5" +checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" dependencies = [ "heck", "proc-macro2", @@ -1405,9 +1586,9 @@ dependencies = [ [[package]] name = "clap_lex" -version = "1.0.0" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a822ea5bc7590f9d40f1ba12c0dc3c2760f3482c6984db1573ad11031420831" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" [[package]] name = "clipboard-win" @@ -1420,9 +1601,9 @@ dependencies = [ [[package]] name = "cmake" -version = "0.1.57" +version = "0.1.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" dependencies = [ "cc", ] @@ -1436,11 +1617,45 @@ dependencies = [ "thiserror 2.0.18", ] +[[package]] +name = "collection_literals" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2550f75b8cfac212855f6b1885455df8eaee8fe8e246b647d69146142e016084" + [[package]] name = "colorchoice" -version = "1.0.4" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "compact_str" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b79c4069c6cad78e2e0cdfcbd26275770669fb39fd308a752dc110e83b9af32" +dependencies = [ + "castaway", + "cfg-if", + "itoa", + "rustversion", + "ryu", + "static_assertions", +] + +[[package]] +name = "compact_str" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75" +checksum = "3fdb1325a1cece981e8a296ab8f0f9b63ae357bd0784a9faaf548cc7b480707a" +dependencies = [ + "castaway", + "cfg-if", + "itoa", + "rustversion", + "ryu", + "static_assertions", +] [[package]] name = "concurrent-queue" @@ -1453,14 +1668,13 @@ dependencies = [ [[package]] name = "console" -version = "0.15.11" +version = "0.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8" +checksum = "d64e8af5551369d19cf50138de61f1c42074ab970f74e99be916646777f8fc87" dependencies = [ "encode_unicode", "libc", - "once_cell", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -1504,13 +1718,23 @@ dependencies = [ "unicode-segmentation", ] +[[package]] +name = "cookie" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747" +dependencies = [ + "time", + "version_check", +] + [[package]] name = "coolor" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "980c2afde4af43d6a05c5be738f9eae595cff86dce1f38f88b95058a98c027f3" dependencies = [ - "crossterm", + "crossterm 0.29.0", ] [[package]] @@ -1557,32 +1781,53 @@ dependencies = [ "libc", ] +[[package]] +name = "cranelift-assembler-x64" +version = "0.130.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "046d4b584c3bb9b5eb500c8f29549bec36be11000f1ba2a927cef3d1a9875691" +dependencies = [ + "cranelift-assembler-x64-meta", +] + +[[package]] +name = "cranelift-assembler-x64-meta" +version = "0.130.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9b194a7870becb1490366fc0ae392ccd188065ff35f8391e77ac659db6fb977" +dependencies = [ + "cranelift-srcgen", +] + [[package]] name = "cranelift-bforest" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88c1d02b72b6c411c0a2e92b25ed791ad5d071184193c08a34aa0fdcdf000b72" +checksum = "bb6a4ab44c6b371e661846b97dab687387a60ac4e2f864e2d4257284aad9e889" dependencies = [ "cranelift-entity", + "wasmtime-internal-core", ] [[package]] name = "cranelift-bitset" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "720b93bd86ebbb23ebfb2db1ed44d54b2ecbdbb2d034d485bc64aa605ee787ab" +checksum = "b8b7a44150c2f471a94023482bda1902710746e4bed9f9973d60c5a94319b06d" dependencies = [ "serde", "serde_derive", + "wasmtime-internal-core", ] [[package]] name = "cranelift-codegen" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aed3d2d9914d30b460eedd7fd507720203023997bef71452ce84873f9c93537c" +checksum = "01b06598133b1dd76758b8b95f8d6747c124124aade50cea96a3d88b962da9fa" dependencies = [ "bumpalo", + "cranelift-assembler-x64", "cranelift-bforest", "cranelift-bitset", "cranelift-codegen-meta", @@ -1591,55 +1836,63 @@ dependencies = [ "cranelift-entity", "cranelift-isle", "gimli", - "hashbrown 0.14.5", + "hashbrown 0.16.1", + "libm", "log", + "pulley-interpreter", "regalloc2", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "serde", "smallvec", "target-lexicon", + "wasmtime-internal-core", ] [[package]] name = "cranelift-codegen-meta" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "888c188d32263ec9e048873ff0b68c700933600d553f4412417916828be25f8e" +checksum = "6190e2e7bcf0a678da2f715363d34ed530fedf7a2f0ab75edaefef72a70465ff" dependencies = [ + "cranelift-assembler-x64-meta", "cranelift-codegen-shared", + "cranelift-srcgen", + "heck", + "pulley-interpreter", ] [[package]] name = "cranelift-codegen-shared" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ddd5f4114d04ce7e073dd74e2ad16541fc61970726fcc8b2d5644a154ee4127" +checksum = "f583cf203d1aa8b79560e3b01f929bdacf9070b015eec4ea9c46e22a3f83e4a0" [[package]] name = "cranelift-control" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92cc4c98d6a4256a1600d93ccd3536f3e77da9b4ca2c279de786ac22876e67d6" +checksum = "803159df35cc398ae54473c150b16d6c77e92ab2948be638488de126a3328fbc" dependencies = [ "arbitrary", ] [[package]] name = "cranelift-entity" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "760af4b5e051b5f82097a27274b917e3751736369fa73660513488248d27f23d" +checksum = "3109e417257082d88087f5bcce677525bdaa8322b88dd7f175ed1a1fd41d546c" dependencies = [ "cranelift-bitset", "serde", "serde_derive", + "wasmtime-internal-core", ] [[package]] name = "cranelift-frontend" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0bf77ec0f470621655ec7539860b5c620d4f91326654ab21b075b83900f8831" +checksum = "14db6b0e0e4994c581092df78d837be2072578f7cb2528f96a6cf895e56dee63" dependencies = [ "cranelift-codegen", "log", @@ -1649,21 +1902,27 @@ dependencies = [ [[package]] name = "cranelift-isle" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b665d0a6932c421620be184f9fc7f7adaf1b0bc2fa77bb7ac5177c49abf645b" +checksum = "ec66ea5025c7317383699778282ac98741d68444f956e3b1d7b62f12b7216e67" [[package]] name = "cranelift-native" -version = "0.115.1" +version = "0.130.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb2e75d1bd43dfec10924798f15e6474f1dbf63b0024506551aa19394dbe72ab" +checksum = "373ade56438e6232619d85678477d0a88a31b3581936e0503e61e96b546b0800" dependencies = [ "cranelift-codegen", "libc", "target-lexicon", ] +[[package]] +name = "cranelift-srcgen" +version = "0.130.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef53619d3cd5c78fd998c6d9420547af26b72e6456f94c2a8a2334cb76b42baa" + [[package]] name = "crc" version = "3.4.0" @@ -1724,6 +1983,12 @@ dependencies = [ "itertools 0.10.5", ] +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crokey" version = "1.4.0" @@ -1731,7 +1996,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04a63daf06a168535c74ab97cdba3ed4fa5d4f32cb36e437dcceb83d66854b7c" dependencies = [ "crokey-proc_macros", - "crossterm", + "crossterm 0.29.0", "once_cell", "serde", "strict", @@ -1743,7 +2008,7 @@ version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "847f11a14855fc490bd5d059821895c53e77eeb3c2b73ee3dded7ce77c93b231" dependencies = [ - "crossterm", + "crossterm 0.29.0", "proc-macro2", "quote", "strict", @@ -1817,6 +2082,22 @@ version = "0.8.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" +[[package]] +name = "crossterm" +version = "0.28.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "829d955a0bb380ef178a640b91779e3987da38c9aea133b20614cfed8cdea9c6" +dependencies = [ + "bitflags 2.11.0", + "crossterm_winapi", + "mio", + "parking_lot", + "rustix 0.38.44", + "signal-hook", + "signal-hook-mio", + "winapi", +] + [[package]] name = "crossterm" version = "0.29.0" @@ -1922,9 +2203,9 @@ dependencies = [ [[package]] name = "darling" -version = "0.21.3" +version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0" +checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" dependencies = [ "darling_core", "darling_macro", @@ -1932,11 +2213,10 @@ dependencies = [ [[package]] name = "darling_core" -version = "0.21.3" +version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4" +checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0" dependencies = [ - "fnv", "ident_case", "proc-macro2", "quote", @@ -1946,9 +2226,9 @@ dependencies = [ [[package]] name = "darling_macro" -version = "0.21.3" +version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81" +checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" dependencies = [ "darling_core", "quote", @@ -2038,6 +2318,17 @@ dependencies = [ "serde_core", ] +[[package]] +name = "derive-where" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d08b3a0bcc0d079199cd476b2cae8435016ec11d1c0986c6901c5ac223041534" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "derive_arbitrary" version = "1.4.2" @@ -2069,6 +2360,7 @@ dependencies = [ "quote", "rustc_version", "syn 2.0.117", + "unicode-xid", ] [[package]] @@ -2098,33 +2390,13 @@ dependencies = [ "dirs-sys-next", ] -[[package]] -name = "dirs" -version = "4.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca3aa72a6f96ea37bbc5aa912f6788242832f75369bdfdadcb0e38423f100059" -dependencies = [ - "dirs-sys 0.3.7", -] - [[package]] name = "dirs" version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e" dependencies = [ - "dirs-sys 0.5.0", -] - -[[package]] -name = "dirs-sys" -version = "0.3.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b1d1d91c932ef41c0f2663aa8b0ca0342d444d842c06914aa0a7e352d0bada6" -dependencies = [ - "libc", - "redox_users 0.4.6", - "winapi", + "dirs-sys", ] [[package]] @@ -2136,7 +2408,7 @@ dependencies = [ "libc", "option-ext", "redox_users 0.5.2", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2151,8 +2423,18 @@ dependencies = [ ] [[package]] -name = "displaydoc" -version = "0.2.5" +name = "dispatch2" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" +dependencies = [ + "bitflags 2.11.0", + "objc2", +] + +[[package]] +name = "displaydoc" +version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ @@ -2250,6 +2532,15 @@ version = "1.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" +[[package]] +name = "email_address" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449" +dependencies = [ + "serde", +] + [[package]] name = "embedded-io" version = "0.4.0" @@ -2323,7 +2614,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -2402,12 +2693,43 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" +[[package]] +name = "fancy-regex" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72cf461f865c862bb7dc573f643dd6a2b6842f7c30b07882b56bd148cc2761b8" +dependencies = [ + "bit-set", + "regex-automata", + "regex-syntax", +] + [[package]] name = "fastrand" version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" +[[package]] +name = "fax" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05de7d48f37cd6730705cbca900770cab77a89f413d23e100ad7fad7795a0ab" +dependencies = [ + "fax_derive", +] + +[[package]] +name = "fax_derive" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0aca10fb742cb43f9e7bb8467c91aa9bcb8e3ffbc6a6f7389bb93ffc920577d" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "fd-lock" version = "4.0.4" @@ -2419,6 +2741,15 @@ dependencies = [ "windows-sys 0.59.0", ] +[[package]] +name = "fdeflate" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c" +dependencies = [ + "simd-adler32", +] + [[package]] name = "fiat-crypto" version = "0.2.9" @@ -2442,6 +2773,12 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" +[[package]] +name = "fixedbitset" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" + [[package]] name = "flagset" version = "0.4.7" @@ -2458,6 +2795,17 @@ dependencies = [ "miniz_oxide", ] +[[package]] +name = "fluent-uri" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc74ac4d8359ae70623506d512209619e5cf8f347124910440dbc221714b328e" +dependencies = [ + "borrow-or-share", + "ref-cast", + "serde", +] + [[package]] name = "fnv" version = "1.0.7" @@ -2485,6 +2833,16 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fraction" +version = "0.15.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f158e3ff0a1b334408dc9fb811cd99b446986f4d8b741bb08f9df1604085ae7" +dependencies = [ + "lazy_static", + "num", +] + [[package]] name = "fs-set-times" version = "0.20.3" @@ -2635,25 +2993,17 @@ dependencies = [ "slab", ] -[[package]] -name = "fxhash" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c31b6d751ae2c7f11320402d34e41349dd1016f8d5d45e48c4312bc8625af50c" -dependencies = [ - "byteorder", -] - [[package]] name = "fxprof-processed-profile" -version = "0.6.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27d12c0aed7f1e24276a241aadc4cb8ea9f83000f34bc062b7cc2d51e3b0fabd" +checksum = "25234f20a3ec0a962a61770cfe39ecf03cb529a6e474ad8cff025ed497eda557" dependencies = [ "bitflags 2.11.0", "debugid", - "fxhash", + "rustc-hash 2.1.2", "serde", + "serde_derive", "serde_json", ] @@ -2667,13 +3017,47 @@ dependencies = [ "version_check", ] +[[package]] +name = "get-size-derive2" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2b6d1e2f75c16bfbcd0f95d84f99858a6e2f885c2287d1f5c3a96e8444a34b4" +dependencies = [ + "attribute-derive", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "get-size2" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49cf31a6d70300cf81461098f7797571362387ef4bf85d32ac47eaa59b3a5a1a" +dependencies = [ + "compact_str 0.9.0", + "get-size-derive2", + "hashbrown 0.16.1", + "ordermap", + "smallvec", +] + +[[package]] +name = "gethostname" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bd49230192a3797a9a4d6abe9b3eed6f7fa4c8a8a4947977c6f80025f92cbd8" +dependencies = [ + "rustix 1.1.4", + "windows-link", +] + [[package]] name = "getopts" version = "0.2.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfe4fbac503b8d1f88e6676011885f34b7174f46e59956bba534ba83abded4df" dependencies = [ - "unicode-width 0.2.2", + "unicode-width 0.2.0", ] [[package]] @@ -2728,11 +3112,12 @@ dependencies = [ [[package]] name = "gimli" -version = "0.31.1" +version = "0.33.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f" +checksum = "0bf7f043f89559805f8c7cacc432749b2fa0d0a0a9ee46ce47164ed5ba7f126c" dependencies = [ - "fallible-iterator 0.3.0", + "fnv", + "hashbrown 0.16.1", "indexmap 2.13.0", "stable_deref_trait", ] @@ -2789,7 +3174,16 @@ checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" dependencies = [ "cfg-if", "crunchy", - "zerocopy 0.8.42", + "zerocopy 0.8.48", +] + +[[package]] +name = "hash32" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0c35f58762feb77d74ebe43bdbc3210f09be9fe6742234d573bacc26ed92b67" +dependencies = [ + "byteorder", ] [[package]] @@ -2809,7 +3203,6 @@ checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" dependencies = [ "ahash 0.8.12", "allocator-api2", - "serde", ] [[package]] @@ -2818,8 +3211,9 @@ version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ + "allocator-api2", + "equivalent", "foldhash 0.1.5", - "serde", ] [[package]] @@ -2831,6 +3225,8 @@ dependencies = [ "allocator-api2", "equivalent", "foldhash 0.2.0", + "serde", + "serde_core", ] [[package]] @@ -2842,6 +3238,20 @@ dependencies = [ "hashbrown 0.14.5", ] +[[package]] +name = "heapless" +version = "0.7.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdc6457c0eb62c71aac4bc17216026d8410337c4126773b9c5daba343f17964f" +dependencies = [ + "atomic-polyfill", + "hash32", + "rustc_version", + "serde", + "spin", + "stable_deref_trait", +] + [[package]] name = "heck" version = "0.5.0" @@ -2898,16 +3308,16 @@ dependencies = [ [[package]] name = "html-to-markdown-rs" -version = "2.28.2" +version = "2.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f9377e16af590b764fd98fd176027cf8831c5335f8964f3f643753e38913a4e" +checksum = "7ea41945a2fd834381642a000ef75b03f0030f3023f3dd3291fc5c372d3dda33" dependencies = [ "ahash 0.8.12", "astral-tl", "base64 0.22.1", "html-escape", - "html5ever 0.38.0", - "lru", + "html5ever 0.39.0", + "lru 0.16.3", "once_cell", "regex", "serde", @@ -2935,6 +3345,16 @@ dependencies = [ "markup5ever 0.38.0", ] +[[package]] +name = "html5ever" +version = "0.39.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46a1761807faccc9a19e86944bbf40610014066306f96edcdedc2fb714bcb7b8" +dependencies = [ + "log", + "markup5ever 0.39.0", +] + [[package]] name = "http" version = "0.2.12" @@ -3150,7 +3570,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.3", "system-configuration", "tokio", "tower-service", @@ -3311,6 +3731,34 @@ dependencies = [ "icu_properties", ] +[[package]] +name = "im-rc" +version = "15.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af1955a75fa080c677d3972822ec4bad316169ab1cfc6c257a942c2265dbe5fe" +dependencies = [ + "bitmaps", + "rand_core 0.6.4", + "rand_xoshiro", + "sized-chunks", + "typenum", + "version_check", +] + +[[package]] +name = "image" +version = "0.25.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85ab80394333c02fe689eaf900ab500fbd0c2213da414687ebf995a65d5a6104" +dependencies = [ + "bytemuck", + "byteorder-lite", + "moxcms", + "num-traits", + "png", + "tiff", +] + [[package]] name = "indexmap" version = "1.9.3" @@ -3334,6 +3782,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "indoc" +version = "2.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706" +dependencies = [ + "rustversion", +] + [[package]] name = "inout" version = "0.1.4" @@ -3346,9 +3803,9 @@ dependencies = [ [[package]] name = "insta" -version = "1.46.3" +version = "1.47.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e82db8c87c7f1ccecb34ce0c24399b8a73081427f3c7c50a5d597925356115e4" +checksum = "99322078b2c076829a1db959d49da554fabc4342257fc0ba5a070a1eb3a01cd8" dependencies = [ "console", "once_cell", @@ -3356,6 +3813,25 @@ dependencies = [ "tempfile", ] +[[package]] +name = "instability" +version = "0.3.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb2d60ef19920a3a9193c3e371f726ec1dafc045dac788d0fb3704272458971" +dependencies = [ + "darling", + "indoc", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "interpolator" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71dd52191aae121e8611f1e8dc3e324dd0dd1dee1e6dd91d10ee07a3cfb4d9d8" + [[package]] name = "io-extras" version = "0.18.4" @@ -3380,9 +3856,9 @@ checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" [[package]] name = "iri-string" -version = "0.7.10" +version = "0.7.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a" +checksum = "d8e7418f59cc01c88316161279a7f665217ae316b388e58a0d10e29f54f1e5eb" dependencies = [ "memchr", "serde", @@ -3390,9 +3866,10 @@ dependencies = [ [[package]] name = "ironclaw" -version = "0.19.0" +version = "0.25.0" dependencies = [ "aes-gcm", + "agent-client-protocol", "aho-corasick", "anyhow", "async-trait", @@ -3403,22 +3880,24 @@ dependencies = [ "base64 0.22.1", "blake3", "bollard", + "bs58", "bytes", "chrono", "chrono-tz", "clap", "clap_complete", - "criterion", + "cookie", "cron", - "crossterm", + "crossterm 0.29.0", "deadpool-postgres", - "dirs 6.0.0", + "dirs", "dotenvy", "ed25519-dalek", "eventsource-stream", "flate2", "fs4", "futures", + "glob", "hex", "hkdf", "hmac", @@ -3428,16 +3907,24 @@ dependencies = [ "hyper-util", "iana-time-zone", "insta", + "ironclaw_common", + "ironclaw_engine", + "ironclaw_gateway", "ironclaw_safety", + "ironclaw_skills", + "ironclaw_tui", "json5", + "jsonschema", + "jsonwebtoken", "libsql", - "lru", + "lru 0.16.3", "mime_guess", "open", "pdf-extract", "pgvector", "postgres-types", "pretty_assertions", + "pty-process", "rand 0.8.5", "readabilityrs", "refinery", @@ -3469,7 +3956,8 @@ dependencies = [ "tokio-stream", "tokio-test", "tokio-tungstenite 0.26.2", - "toml", + "tokio-util", + "toml 0.8.23", "tower 0.5.3", "tower-http 0.6.8", "tracing", @@ -3478,18 +3966,61 @@ dependencies = [ "url", "urlencoding", "uuid", - "wasmparser 0.220.1", + "wasmparser 0.245.1", "wasmtime", "wasmtime-wasi", + "webpki-roots 0.26.11", "zbus", "zip", ] [[package]] -name = "ironclaw_safety" +name = "ironclaw_common" +version = "0.2.0" +dependencies = [ + "chrono-tz", + "serde", + "serde_json", + "tracing", +] + +[[package]] +name = "ironclaw_engine" +version = "0.1.0" +dependencies = [ + "async-trait", + "chrono", + "cron", + "ironclaw_common", + "ironclaw_skills", + "monty", + "pretty_assertions", + "regex", + "serde", + "serde_json", + "sha2", + "thiserror 2.0.18", + "tokio", + "tracing", + "uuid", +] + +[[package]] +name = "ironclaw_gateway" version = "0.1.0" +dependencies = [ + "serde", + "serde_json", + "thiserror 2.0.18", + "tracing", +] + +[[package]] +name = "ironclaw_safety" +version = "0.2.1" dependencies = [ "aho-corasick", + "criterion", "regex", "serde_json", "thiserror 2.0.18", @@ -3497,6 +4028,43 @@ dependencies = [ "url", ] +[[package]] +name = "ironclaw_skills" +version = "0.1.0" +dependencies = [ + "chrono", + "futures", + "regex", + "reqwest", + "serde", + "serde_json", + "serde_yml", + "sha2", + "tempfile", + "thiserror 2.0.18", + "tokio", + "tracing", + "urlencoding", +] + +[[package]] +name = "ironclaw_tui" +version = "0.1.0" +dependencies = [ + "arboard", + "chrono", + "image", + "pulldown-cmark", + "ratatui", + "serde", + "serde_json", + "thiserror 2.0.18", + "tokio", + "tracing", + "tui-textarea", + "unicode-width 0.2.0", +] + [[package]] name = "is-docker" version = "0.2.0" @@ -3506,6 +4074,18 @@ dependencies = [ "once_cell", ] +[[package]] +name = "is-macro" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d57a3e447e24c22647738e4607f1df1e0ec6f72e16182c4cd199f647cdfb0e4" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "is-terminal" version = "0.4.17" @@ -3514,7 +4094,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46" dependencies = [ "hermit-abi", "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3552,10 +4132,28 @@ dependencies = [ ] [[package]] -name = "itoa" -version = "1.0.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" +name = "itertools" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" +dependencies = [ + "either", +] + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "ittapi" @@ -3577,6 +4175,21 @@ dependencies = [ "cc", ] +[[package]] +name = "jiter" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "020ba671987d7444d251d3ee5340be1bf4606cd6c0b53e6f4066b5a1ee376b22" +dependencies = [ + "ahash 0.8.12", + "bitvec", + "lexical-parse-float", + "num-bigint", + "num-traits", + "pyo3", + "smallvec", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -3589,10 +4202,12 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.91" +version = "0.3.92" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c" +checksum = "cc4c90f45aa2e6eacbe8645f77fdea542ac97a494bcd117a67df9ff4d611f995" dependencies = [ + "cfg-if", + "futures-util", "once_cell", "wasm-bindgen", ] @@ -3608,6 +4223,48 @@ dependencies = [ "serde", ] +[[package]] +name = "jsonschema" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f29616f6e19415398eb186964fb7cbbeef572c79bede3622a8277667924bbe3" +dependencies = [ + "ahash 0.8.12", + "bytecount", + "data-encoding", + "email_address", + "fancy-regex", + "fraction", + "getrandom 0.3.4", + "idna", + "itoa", + "num-cmp", + "num-traits", + "percent-encoding", + "referencing", + "regex", + "regex-syntax", + "serde", + "serde_json", + "unicode-general-category", + "uuid-simd", +] + +[[package]] +name = "jsonwebtoken" +version = "9.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a87cc7a48537badeae96744432de36f4be2b4a34a05a5ef32e9dd8a1c169dde" +dependencies = [ + "base64 0.22.1", + "js-sys", + "pem", + "ring", + "serde", + "serde_json", + "simple_asn1", +] + [[package]] name = "kuchikikiki" version = "0.9.2" @@ -3670,6 +4327,31 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + [[package]] name = "libc" version = "0.2.183" @@ -3694,9 +4376,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libredox" -version = "0.1.14" +version = "0.1.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1744e39d1d6a9948f4f388969627434e31128196de472883b39f148769bfe30a" +checksum = "7ddbf48fd451246b1f8c2610bd3b4ac0cc6e149d89832867093ab69a17194f08" dependencies = [ "bitflags 2.11.0", "libc", @@ -3903,6 +4585,15 @@ dependencies = [ "weezl", ] +[[package]] +name = "lru" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" +dependencies = [ + "hashbrown 0.15.5", +] + [[package]] name = "lru" version = "0.16.3" @@ -3933,6 +4624,29 @@ dependencies = [ "libc", ] +[[package]] +name = "manyhow" +version = "0.11.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b33efb3ca6d3b07393750d4030418d594ab1139cee518f0dc88db70fec873587" +dependencies = [ + "manyhow-macros", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "manyhow-macros" +version = "0.11.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46fce34d199b78b6e6073abf984c9cf5fd3e9330145a93ee0738a7443e371495" +dependencies = [ + "proc-macro-utils", + "proc-macro2", + "quote", +] + [[package]] name = "markup5ever" version = "0.36.1" @@ -3955,6 +4669,17 @@ dependencies = [ "web_atoms", ] +[[package]] +name = "markup5ever" +version = "0.39.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7122d987ec5f704ee56f6e5b41a7d93722e9aae27ae07cafa4036c4d3f9757de" +dependencies = [ + "log", + "tendril 0.5.0", + "web_atoms", +] + [[package]] name = "matchers" version = "0.2.0" @@ -4059,9 +4784,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" dependencies = [ "libc", "log", @@ -4069,6 +4794,44 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "monty" +version = "0.0.11" +source = "git+https://github.com/pydantic/monty.git?tag=v0.0.11#2e9df4b508e8a9ac80f3a6a26ed680242d1f460d" +dependencies = [ + "ahash 0.8.12", + "bytemuck", + "chrono", + "fancy-regex", + "hashbrown 0.16.1", + "indexmap 2.13.0", + "itertools 0.14.0", + "jiter", + "libm", + "num-bigint", + "num-integer", + "num-traits", + "postcard", + "pyo3-build-config", + "ruff_python_ast", + "ruff_python_parser", + "ruff_text_size", + "serde", + "smallvec", + "speedate", + "strum 0.27.2", +] + +[[package]] +name = "moxcms" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb85c154ba489f01b25c0d36ae69a87e4a1c73a72631fc6c0eb6dde34a73e44b" +dependencies = [ + "num-traits", + "pxfm", +] + [[package]] name = "nanoid" version = "0.4.0" @@ -4134,7 +4897,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4159,8 +4922,15 @@ checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" dependencies = [ "num-integer", "num-traits", + "serde", ] +[[package]] +name = "num-cmp" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63335b2e2c34fae2fb0aa2cecfd9f0832a1e24b3b32ecec612c3426d46dc8aaa" + [[package]] name = "num-complex" version = "0.4.6" @@ -4172,9 +4942,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.2.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" +checksum = "c6673768db2d862beb9b39a78fdcb1a69439615d5794a1be50caa9bc92c81967" [[package]] name = "num-integer" @@ -4226,6 +4996,27 @@ dependencies = [ "libc", ] +[[package]] +name = "objc2" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a12a8ed07aefc768292f076dc3ac8c48f3781c8f2d5851dd3d98950e8c5a89f" +dependencies = [ + "objc2-encode", +] + +[[package]] +name = "objc2-app-kit" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d49e936b501e5c5bf01fda3a9452ff86dc3ea98ad5f283e1455153142d97518c" +dependencies = [ + "bitflags 2.11.0", + "objc2", + "objc2-core-graphics", + "objc2-foundation", +] + [[package]] name = "objc2-core-foundation" version = "0.3.2" @@ -4233,43 +5024,77 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" dependencies = [ "bitflags 2.11.0", + "dispatch2", + "objc2", ] [[package]] -name = "objc2-system-configuration" +name = "objc2-core-graphics" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396" +checksum = "e022c9d066895efa1345f8e33e584b9f958da2fd4cd116792e15e07e4720a807" dependencies = [ + "bitflags 2.11.0", + "dispatch2", + "objc2", "objc2-core-foundation", + "objc2-io-surface", ] [[package]] -name = "object" -version = "0.36.7" +name = "objc2-encode" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33" + +[[package]] +name = "objc2-foundation" +version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87" +checksum = "e3e0adef53c21f888deb4fa59fc59f7eb17404926ee8a6f59f5df0fd7f9f3272" dependencies = [ - "crc32fast", - "hashbrown 0.15.5", - "indexmap 2.13.0", - "memchr", + "bitflags 2.11.0", + "objc2", + "objc2-core-foundation", +] + +[[package]] +name = "objc2-io-surface" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "180788110936d59bab6bd83b6060ffdfffb3b922ba1396b312ae795e1de9d81d" +dependencies = [ + "bitflags 2.11.0", + "objc2", + "objc2-core-foundation", +] + +[[package]] +name = "objc2-system-configuration" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396" +dependencies = [ + "objc2-core-foundation", ] [[package]] name = "object" -version = "0.37.3" +version = "0.38.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" +checksum = "271638cd5fa9cca89c4c304675ca658efc4e64a66c716b7cfe1afb4b9611dbbc" dependencies = [ + "crc32fast", + "hashbrown 0.16.1", + "indexmap 2.13.0", "memchr", ] [[package]] name = "once_cell" -version = "1.21.3" +version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "once_cell_polyfill" @@ -4320,9 +5145,9 @@ checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" [[package]] name = "ordered-float" -version = "5.1.0" +version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f4779c6901a562440c3786d08192c6fbda7c1c2060edd10006b05ee35d10f2d" +checksum = "b7d950ca161dc355eaf28f82b11345ed76c6e1f6eb1f4f4479e0323b9e2fbd0e" dependencies = [ "num-traits", ] @@ -4337,6 +5162,15 @@ dependencies = [ "pin-project-lite", ] +[[package]] +name = "ordermap" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfa78c92071bbd3628c22b1a964f7e0eb201dc1456555db072beb1662ecd6715" +dependencies = [ + "indexmap 2.13.0", +] + [[package]] name = "outref" version = "0.5.2" @@ -4430,6 +5264,16 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19b17cddbe7ec3f8bc800887bab5e717348c95ea2ca0b1bf0837fb964dc67099" +[[package]] +name = "pem" +version = "3.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" +dependencies = [ + "base64 0.22.1", + "serde_core", +] + [[package]] name = "percent-encoding" version = "2.3.2" @@ -4479,6 +5323,16 @@ dependencies = [ "sha2", ] +[[package]] +name = "petgraph" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4c5cc86750666a3ed20bdaf5ca2a0344f9c67674cae0515bec2da16fbaa47db" +dependencies = [ + "fixedbitset", + "indexmap 2.13.0", +] + [[package]] name = "pgvector" version = "0.4.1" @@ -4692,6 +5546,19 @@ dependencies = [ "plotters-backend", ] +[[package]] +name = "png" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61" +dependencies = [ + "bitflags 2.11.0", + "crc32fast", + "fdeflate", + "flate2", + "miniz_oxide", +] + [[package]] name = "polling" version = "3.11.0" @@ -4724,6 +5591,12 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60f6ce597ecdcc9a098e7fddacb1065093a3d66446fa16c675e7e71d1b5c28e6" +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "postcard" version = "1.1.3" @@ -4733,6 +5606,7 @@ dependencies = [ "cobs", "embedded-io 0.4.0", "embedded-io 0.6.1", + "heapless", "serde", ] @@ -4796,7 +5670,7 @@ version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" dependencies = [ - "zerocopy 0.8.42", + "zerocopy 0.8.48", ] [[package]] @@ -4831,7 +5705,18 @@ version = "3.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" dependencies = [ - "toml_edit 0.25.4+spec-1.1.0", + "toml_edit 0.25.8+spec-1.1.0", +] + +[[package]] +name = "proc-macro-utils" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eeaf08a13de400bc215877b5bdc088f241b12eb42f0a548d3390dc1c56bb7071" +dependencies = [ + "proc-macro2", + "quote", + "smallvec", ] [[package]] @@ -4866,16 +5751,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "psm" -version = "0.1.30" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3852766467df634d74f0b2d7819bf8dc483a0eb2e3b0f50f756f9cfe8b0d18d8" -dependencies = [ - "ar_archive_writer", - "cc", -] - [[package]] name = "ptr_meta" version = "0.1.4" @@ -4896,17 +5771,122 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "pty-process" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71cec9e2670207c5ebb9e477763c74436af3b9091dd550b9fb3c1bec7f3ea266" +dependencies = [ + "rustix 1.1.4", + "tokio", +] + +[[package]] +name = "pulldown-cmark" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f86ba2052aebccc42cbbb3ed234b8b13ce76f75c3551a303cb2bcffcff12bb14" +dependencies = [ + "bitflags 2.11.0", + "memchr", + "unicase", +] + [[package]] name = "pulley-interpreter" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8324e531de91a3c25021a30fb7862d39cc516b61fbb801176acb5ff279ea887b" +checksum = "010dec3755eb61b2f1051ecb3611b718460b7a74c131e474de2af20a845938af" dependencies = [ "cranelift-bitset", "log", - "sptr", + "pulley-macros", + "wasmtime-internal-core", +] + +[[package]] +name = "pulley-macros" +version = "43.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ad360c32e85ca4b083ac0e2b6856e8f11c3d5060dafa7d5dc57b370857fa3018" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "pxfm" +version = "0.1.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5a041e753da8b807c9255f28de81879c78c876392ff2469cde94799b2896b9d" + +[[package]] +name = "pyo3" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91fd8e38a3b50ed1167fb981cd6fd60147e091784c427b8f7183a7ee32c31c12" +dependencies = [ + "libc", + "num-bigint", + "num-traits", + "once_cell", + "portable-atomic", + "pyo3-build-config", + "pyo3-ffi", + "pyo3-macros", +] + +[[package]] +name = "pyo3-build-config" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e" +dependencies = [ + "target-lexicon", +] + +[[package]] +name = "pyo3-ffi" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f29e10af80b1f7ccaf7f69eace800a03ecd13e883acfacc1e5d0988605f651e" +dependencies = [ + "libc", + "pyo3-build-config", +] + +[[package]] +name = "pyo3-macros" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df6e520eff47c45997d2fc7dd8214b25dd1310918bbb2642156ef66a67f29813" +dependencies = [ + "proc-macro2", + "pyo3-macros-backend", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "pyo3-macros-backend" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4cdc218d835738f81c2338f822078af45b4afdf8b2e33cbb5916f108b813acb" +dependencies = [ + "heck", + "proc-macro2", + "pyo3-build-config", + "quote", + "syn 2.0.117", ] +[[package]] +name = "quick-error" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3" + [[package]] name = "quinn" version = "0.11.9" @@ -4918,9 +5898,9 @@ dependencies = [ "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "rustls 0.23.37", - "socket2 0.5.10", + "socket2 0.6.3", "thiserror 2.0.18", "tokio", "tracing", @@ -4938,7 +5918,7 @@ dependencies = [ "lru-slab", "rand 0.9.2", "ring", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "rustls 0.23.37", "rustls-pki-types", "slab", @@ -4957,18 +5937,40 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.3", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.60.2", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "quote-use" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9619db1197b497a36178cfc736dc96b271fe918875fbf1344c436a7e93d0321e" +dependencies = [ + "quote", + "quote-use-macros", ] [[package]] -name = "quote" -version = "1.0.45" +name = "quote-use-macros" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +checksum = "82ebfb7faafadc06a7ab141a6f67bcfb24cb8beb158c6fe933f2f035afa99f35" dependencies = [ + "proc-macro-utils", "proc-macro2", + "quote", + "syn 2.0.117", ] [[package]] @@ -5058,12 +6060,42 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_xoshiro" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f97cdb2a36ed4183de61b2f824cc45c9f1037f28afe0a322e9fff4c108b5aaa" +dependencies = [ + "rand_core 0.6.4", +] + [[package]] name = "rangemap" version = "1.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "973443cf09a9c8656b574a866ab68dfa19f0867d0340648c7d2f6a71b8a8ea68" +[[package]] +name = "ratatui" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eabd94c2f37801c20583fc49dd5cd6b0ba68c716787c2dd6ed18571e1e63117b" +dependencies = [ + "bitflags 2.11.0", + "cassowary", + "compact_str 0.8.1", + "crossterm 0.28.1", + "indoc", + "instability", + "itertools 0.13.0", + "lru 0.12.5", + "paste", + "strum 0.26.3", + "unicode-segmentation", + "unicode-truncate", + "unicode-width 0.2.0", +] + [[package]] name = "rayon" version = "1.11.0" @@ -5171,6 +6203,21 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "referencing" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8a618c14f8ba29d8193bb55e2bf13e4fb2b1115313ecb7ae94b43100c7ac7d5" +dependencies = [ + "ahash 0.8.12", + "fluent-uri", + "getrandom 0.3.4", + "hashbrown 0.16.1", + "parking_lot", + "percent-encoding", + "serde_json", +] + [[package]] name = "refinery" version = "0.8.16" @@ -5197,7 +6244,7 @@ dependencies = [ "time", "tokio", "tokio-postgres", - "toml", + "toml 0.8.23", "url", "walkdir", ] @@ -5218,15 +6265,15 @@ dependencies = [ [[package]] name = "regalloc2" -version = "0.11.2" +version = "0.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc06e6b318142614e4a48bc725abbf08ff166694835c43c9dae5a9009704639a" +checksum = "952ddbfc6f9f64d006c3efd8c9851a6ba2f2b944ba94730db255d55006e0ffda" dependencies = [ "allocator-api2", "bumpalo", "hashbrown 0.15.5", "log", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "smallvec", ] @@ -5395,11 +6442,77 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "ruff_python_ast" +version = "0.0.0" +source = "git+https://github.com/astral-sh/ruff.git?rev=6ded4bed1651e30b34dd04cdaa50c763036abb0d#6ded4bed1651e30b34dd04cdaa50c763036abb0d" +dependencies = [ + "aho-corasick", + "bitflags 2.11.0", + "compact_str 0.9.0", + "get-size2", + "is-macro", + "memchr", + "ruff_python_trivia", + "ruff_source_file", + "ruff_text_size", + "rustc-hash 2.1.2", + "thiserror 2.0.18", +] + +[[package]] +name = "ruff_python_parser" +version = "0.0.0" +source = "git+https://github.com/astral-sh/ruff.git?rev=6ded4bed1651e30b34dd04cdaa50c763036abb0d#6ded4bed1651e30b34dd04cdaa50c763036abb0d" +dependencies = [ + "bitflags 2.11.0", + "bstr", + "compact_str 0.9.0", + "get-size2", + "memchr", + "ruff_python_ast", + "ruff_python_trivia", + "ruff_text_size", + "rustc-hash 2.1.2", + "static_assertions", + "unicode-ident", + "unicode-normalization", + "unicode_names2", +] + +[[package]] +name = "ruff_python_trivia" +version = "0.0.0" +source = "git+https://github.com/astral-sh/ruff.git?rev=6ded4bed1651e30b34dd04cdaa50c763036abb0d#6ded4bed1651e30b34dd04cdaa50c763036abb0d" +dependencies = [ + "itertools 0.14.0", + "ruff_source_file", + "ruff_text_size", + "unicode-ident", +] + +[[package]] +name = "ruff_source_file" +version = "0.0.0" +source = "git+https://github.com/astral-sh/ruff.git?rev=6ded4bed1651e30b34dd04cdaa50c763036abb0d#6ded4bed1651e30b34dd04cdaa50c763036abb0d" +dependencies = [ + "memchr", + "ruff_text_size", +] + +[[package]] +name = "ruff_text_size" +version = "0.0.0" +source = "git+https://github.com/astral-sh/ruff.git?rev=6ded4bed1651e30b34dd04cdaa50c763036abb0d#6ded4bed1651e30b34dd04cdaa50c763036abb0d" +dependencies = [ + "get-size2", +] + [[package]] name = "rust_decimal" -version = "1.40.0" +version = "1.41.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61f703d19852dbf87cbc513643fa81428361eb6940f1ac14fd58155d295a3eb0" +checksum = "2ce901f9a19d251159075a4c37af514c3b8ef99c22e02dd8c19161cf397ee94a" dependencies = [ "arrayvec", "borsh", @@ -5410,6 +6523,7 @@ dependencies = [ "rkyv", "serde", "serde_json", + "wasm-bindgen", ] [[package]] @@ -5436,9 +6550,9 @@ checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" [[package]] name = "rustc-hash" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" [[package]] name = "rustc_version" @@ -5472,7 +6586,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.12.1", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -5627,7 +6741,7 @@ dependencies = [ "radix_trie", "rustyline-derive", "unicode-segmentation", - "unicode-width 0.2.2", + "unicode-width 0.2.0", "utf8parse", "windows-sys 0.60.2", ] @@ -5820,7 +6934,7 @@ dependencies = [ "phf 0.13.1", "phf_codegen 0.13.1", "precomputed-hash", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "servo_arc", "smallvec", ] @@ -5839,7 +6953,7 @@ dependencies = [ "phf 0.13.1", "phf_codegen 0.13.1", "precomputed-hash", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "servo_arc", "smallvec", ] @@ -5939,6 +7053,15 @@ dependencies = [ "serde", ] +[[package]] +name = "serde_spanned" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6662b5879511e06e8999a8a235d848113e942c9124f211511b16466ee2995f26" +dependencies = [ + "serde_core", +] + [[package]] name = "serde_urlencoded" version = "0.7.1" @@ -5953,9 +7076,9 @@ dependencies = [ [[package]] name = "serde_with" -version = "3.17.0" +version = "3.18.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "381b283ce7bc6b476d903296fb59d0d36633652b633b27f64db4fb46dcbfc3b9" +checksum = "dd5414fad8e6907dbdd5bc441a50ae8d6e26151a03b1de04d89a5576de61d01f" dependencies = [ "base64 0.22.1", "chrono", @@ -5972,9 +7095,9 @@ dependencies = [ [[package]] name = "serde_with_macros" -version = "3.17.0" +version = "3.18.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6d4e30573c8cb306ed6ab1dca8423eec9a463ea0e155f45399455e0368b27e0" +checksum = "d3db8978e608f1fe7357e211969fd9abdcae80bac1ba7a3369bb7eb6b404eb65" dependencies = [ "darling", "proc-macro2", @@ -5982,6 +7105,19 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.13.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "serde_yml" version = "0.0.12" @@ -6043,15 +7179,6 @@ dependencies = [ "lazy_static", ] -[[package]] -name = "shellexpand" -version = "2.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ccc8076840c4da029af4f87e4e8daeb0fca6b87bbb02e10cb60b791450e11e4" -dependencies = [ - "dirs 4.0.0", -] - [[package]] name = "shlex" version = "1.3.0" @@ -6100,9 +7227,9 @@ dependencies = [ [[package]] name = "simd-adler32" -version = "0.3.8" +version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" [[package]] name = "simdutf8" @@ -6116,12 +7243,34 @@ version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa" +[[package]] +name = "simple_asn1" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d" +dependencies = [ + "num-bigint", + "num-traits", + "thiserror 2.0.18", + "time", +] + [[package]] name = "siphasher" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e" +[[package]] +name = "sized-chunks" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "16d69225bde7a69b235da73377861095455d298f2b970996eec25ddbb42b3d1e" +dependencies = [ + "bitmaps", + "typenum", +] + [[package]] name = "slab" version = "0.4.12" @@ -6154,7 +7303,27 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", +] + +[[package]] +name = "speedate" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aba069c070b5e213f2a094deb7e5ed50ecb092be36102a4f4042e8d2056d060e" +dependencies = [ + "lexical-parse-float", + "strum 0.27.2", + "strum_macros 0.27.2", +] + +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +dependencies = [ + "lock_api", ] [[package]] @@ -6167,12 +7336,6 @@ dependencies = [ "der", ] -[[package]] -name = "sptr" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b9b39299b249ad65f3b7e96443bad61c02ca5cd3589f46cb6d610a0fd6c0d6a" - [[package]] name = "stable_deref_trait" version = "1.2.1" @@ -6255,6 +7418,70 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" +dependencies = [ + "strum_macros 0.26.4", +] + +[[package]] +name = "strum" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" +dependencies = [ + "strum_macros 0.27.2", +] + +[[package]] +name = "strum" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd" +dependencies = [ + "strum_macros 0.28.0", +] + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.117", +] + +[[package]] +name = "strum_macros" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "strum_macros" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "subtle" version = "2.6.1" @@ -6365,9 +7592,9 @@ dependencies = [ [[package]] name = "target-lexicon" -version = "0.12.16" +version = "0.13.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61c41af27dd6d1e27b1b16b489db798443478cef1f06a660c96db617ba5de3b1" +checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" [[package]] name = "tempfile" @@ -6379,7 +7606,7 @@ dependencies = [ "getrandom 0.4.2", "once_cell", "rustix 1.1.4", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -6515,6 +7742,20 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "tiff" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b63feaf3343d35b6ca4d50483f94843803b0f51634937cc2ec519fc32232bc52" +dependencies = [ + "fax", + "flate2", + "half", + "quick-error", + "weezl", + "zune-jpeg", +] + [[package]] name = "time" version = "0.3.47" @@ -6577,9 +7818,9 @@ dependencies = [ [[package]] name = "tinyvec" -version = "1.10.0" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa5fdc3bce6191a1dbc8c02d5c8bffcf557bafa17c124c5264a458f1b0613fa" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" dependencies = [ "tinyvec_macros", ] @@ -6768,7 +8009,11 @@ checksum = "7a9daff607c6d2bf6c16fd681ccb7eecc83e4e2cdc1ca067ffaadfca5de7f084" dependencies = [ "futures-util", "log", + "rustls 0.23.37", + "rustls-native-certs 0.8.3", + "rustls-pki-types", "tokio", + "tokio-rustls 0.26.4", "tungstenite 0.26.2", ] @@ -6792,6 +8037,7 @@ checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" dependencies = [ "bytes", "futures-core", + "futures-io", "futures-sink", "pin-project-lite", "tokio", @@ -6804,11 +8050,26 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362" dependencies = [ "serde", - "serde_spanned", + "serde_spanned 0.6.9", "toml_datetime 0.6.11", "toml_edit 0.22.27", ] +[[package]] +name = "toml" +version = "0.9.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863" +dependencies = [ + "indexmap 2.13.0", + "serde_core", + "serde_spanned 1.1.1", + "toml_datetime 0.7.5+spec-1.1.0", + "toml_parser", + "toml_writer", + "winnow 0.7.15", +] + [[package]] name = "toml_datetime" version = "0.6.11" @@ -6820,9 +8081,18 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "1.0.0+spec-1.1.0" +version = "0.7.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_datetime" +version = "1.1.0+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32c2555c699578a4f59f0cc68e5116c8d7cabbd45e1409b989d4be085b53f13e" +checksum = "97251a7c317e03ad83774a8752a7e81fb6067740609f75ea2b585b569a59198f" dependencies = [ "serde_core", ] @@ -6835,31 +8105,31 @@ checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" dependencies = [ "indexmap 2.13.0", "serde", - "serde_spanned", + "serde_spanned 0.6.9", "toml_datetime 0.6.11", "toml_write", - "winnow", + "winnow 0.7.15", ] [[package]] name = "toml_edit" -version = "0.25.4+spec-1.1.0" +version = "0.25.8+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7193cbd0ce53dc966037f54351dbbcf0d5a642c7f0038c382ef9e677ce8c13f2" +checksum = "16bff38f1d86c47f9ff0647e6838d7bb362522bdf44006c7068c2b1e606f1f3c" dependencies = [ "indexmap 2.13.0", - "toml_datetime 1.0.0+spec-1.1.0", + "toml_datetime 1.1.0+spec-1.1.0", "toml_parser", - "winnow", + "winnow 1.0.0", ] [[package]] name = "toml_parser" -version = "1.0.9+spec-1.1.0" +version = "1.1.0+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "702d4415e08923e7e1ef96cd5727c0dfed80b4d2fa25db9647fe5eb6f7c5a4c4" +checksum = "2334f11ee363607eb04df9b8fc8a13ca1715a72ba8662a26ac285c98aabb4011" dependencies = [ - "winnow", + "winnow 1.0.0", ] [[package]] @@ -6868,6 +8138,12 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801" +[[package]] +name = "toml_writer" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "756daf9b1013ebe47a8776667b466417e2d4c5679d441c26230efd9ef78692db" + [[package]] name = "tonic" version = "0.11.0" @@ -6982,6 +8258,7 @@ dependencies = [ "futures-util", "http 1.4.0", "http-body 1.0.1", + "http-body-util", "iri-string", "pin-project-lite", "tower 0.5.3", @@ -7070,9 +8347,9 @@ dependencies = [ [[package]] name = "tracing-subscriber" -version = "0.3.22" +version = "0.3.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f30143827ddab0d256fd843b7a66d164e9f271cfa0dde49142c5ca0ca291f1e" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" dependencies = [ "matchers", "nu-ansi-term", @@ -7116,6 +8393,17 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "tui-textarea" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0a5318dd619ed73c52a9417ad19046724effc1287fb75cdcc4eca1d6ac1acbae" +dependencies = [ + "crossterm 0.28.1", + "ratatui", + "unicode-width 0.2.0", +] + [[package]] name = "tungstenite" version = "0.26.2" @@ -7128,6 +8416,8 @@ dependencies = [ "httparse", "log", "rand 0.9.2", + "rustls 0.23.37", + "rustls-pki-types", "sha1", "thiserror 2.0.18", "utf-8", @@ -7152,9 +8442,9 @@ dependencies = [ [[package]] name = "type1-encoding-parser" -version = "0.1.0" +version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3d6cc09e1a99c7e01f2afe4953789311a1c50baebbdac5b477ecf78e2e92a5b" +checksum = "fa10c302f5a53b7ad27fd42a3996e23d096ba39b5b8dd6d9e683a05b01bee749" dependencies = [ "pom", ] @@ -7179,7 +8469,7 @@ checksum = "f2f6fb2847f6742cd76af783a2a2c49e9375d0a111c7bef6f71cd9e738c72d6e" dependencies = [ "memoffset", "tempfile", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -7203,6 +8493,12 @@ version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.24" @@ -7226,9 +8522,20 @@ checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" [[package]] name = "unicode-segmentation" -version = "1.12.0" +version = "1.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" + +[[package]] +name = "unicode-truncate" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" +checksum = "b3644627a5af5fa321c95b9b235a72fd24cd29c648c2c379431e6628655627bf" +dependencies = [ + "itertools 0.13.0", + "unicode-segmentation", + "unicode-width 0.1.14", +] [[package]] name = "unicode-width" @@ -7238,15 +8545,37 @@ checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af" [[package]] name = "unicode-width" -version = "0.2.2" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +checksum = "1fc81956842c57dac11422a97c3b8195a1ff727f06e85c84ed2e8aa277c9a0fd" [[package]] name = "unicode-xid" version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unicode_names2" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1673eca9782c84de5f81b82e4109dcfb3611c8ba0d52930ec4a9478f547b2dd" +dependencies = [ + "phf 0.11.3", + "unicode_names2_generator", +] + +[[package]] +name = "unicode_names2_generator" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b91e5b84611016120197efd7dc93ef76774f4e084cd73c9fb3ea4a86c570c56e" +dependencies = [ + "getopts", + "log", + "phf_codegen 0.11.3", + "rand 0.8.5", +] [[package]] name = "universal-hash" @@ -7258,6 +8587,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" @@ -7309,9 +8644,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.22.0" +version = "1.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37" +checksum = "5ac8b6f42ead25368cf5b098aeb3dc8a1a2c05a3eee8a9a1a68c640edbfc79d9" dependencies = [ "getrandom 0.4.2", "js-sys", @@ -7320,6 +8655,16 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "uuid-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b082222b4f6619906941c17eb2297fff4c2fb96cb60164170522942a200bd8" +dependencies = [ + "outref", + "vsimd", +] + [[package]] name = "v_htmlescape" version = "0.15.8" @@ -7407,36 +8752,33 @@ dependencies = [ [[package]] name = "wasm-bindgen" -version = "0.2.114" +version = "0.2.115" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e" +checksum = "6523d69017b7633e396a89c5efab138161ed5aafcbc8d3e5c5a42ae38f50495a" dependencies = [ "cfg-if", "once_cell", "rustversion", + "serde", "wasm-bindgen-macro", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-futures" -version = "0.4.64" +version = "0.4.65" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8" +checksum = "2d1faf851e778dfa54db7cd438b70758eba9755cb47403f3496edd7c8fc212f0" dependencies = [ - "cfg-if", - "futures-util", "js-sys", - "once_cell", "wasm-bindgen", - "web-sys", ] [[package]] name = "wasm-bindgen-macro" -version = "0.2.114" +version = "0.2.115" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6" +checksum = "4e3a6c758eb2f701ed3d052ff5737f5bfe6614326ea7f3bbac7156192dc32e67" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -7444,9 +8786,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.114" +version = "0.2.115" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3" +checksum = "921de2737904886b52bcbb237301552d05969a6f9c40d261eb0533c8b055fedf" dependencies = [ "bumpalo", "proc-macro2", @@ -7457,21 +8799,32 @@ dependencies = [ [[package]] name = "wasm-bindgen-shared" -version = "0.2.114" +version = "0.2.115" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16" +checksum = "a93e946af942b58934c604527337bad9ae33ba1d5c6900bbb41c2c07c2364a93" dependencies = [ "unicode-ident", ] [[package]] -name = "wasm-encoder" -version = "0.221.3" +name = "wasm-compose" +version = "0.245.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc8444fe4920de80a4fe5ab564fff2ae58b6b73166b89751f8c6c93509da32e5" +checksum = "5fd23d12cc95c451c1306db5bc63075fbebb612bb70c53b4237b1ce5bc178343" dependencies = [ - "leb128", - "wasmparser 0.221.3", + "anyhow", + "heck", + "im-rc", + "indexmap 2.13.0", + "log", + "petgraph", + "serde", + "serde_derive", + "serde_yaml", + "smallvec", + "wasm-encoder 0.245.1", + "wasmparser 0.245.1", + "wat", ] [[package]] @@ -7519,33 +8872,6 @@ dependencies = [ "web-sys", ] -[[package]] -name = "wasmparser" -version = "0.220.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d07b6a3b550fefa1a914b6d54fc175dd11c3392da11eee604e6ffc759805d25" -dependencies = [ - "ahash 0.8.12", - "bitflags 2.11.0", - "hashbrown 0.14.5", - "indexmap 2.13.0", - "semver", - "serde", -] - -[[package]] -name = "wasmparser" -version = "0.221.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d06bfa36ab3ac2be0dee563380147a5b81ba10dd8885d7fbbc9eb574be67d185" -dependencies = [ - "bitflags 2.11.0", - "hashbrown 0.15.5", - "indexmap 2.13.0", - "semver", - "serde", -] - [[package]] name = "wasmparser" version = "0.244.0" @@ -7565,135 +8891,166 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4f08c9adee0428b7bddf3890fc27e015ac4b761cc608c822667102b8bfd6995e" dependencies = [ "bitflags 2.11.0", + "hashbrown 0.16.1", "indexmap 2.13.0", "semver", + "serde", ] [[package]] name = "wasmprinter" -version = "0.221.3" +version = "0.245.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7343c42a97f2926c7819ff81b64012092ae954c5d83ddd30c9fcdefd97d0b283" +checksum = "5f41517a3716fbb8ccf46daa9c1325f760fcbff5168e75c7392288e410b91ac8" dependencies = [ "anyhow", "termcolor", - "wasmparser 0.221.3", + "wasmparser 0.245.1", ] [[package]] name = "wasmtime" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edd30973c65eceb0f37dfcc430d83abd5eb24015fdfcab6912f52949287e04f0" +checksum = "ce205cd643d661b5ba5ba4717e13730262e8cdbc8f2eacbc7b906d45c1a74026" dependencies = [ "addr2line", - "anyhow", "async-trait", "bitflags 2.11.0", "bumpalo", "cc", "cfg-if", "encoding_rs", + "futures", "fxprof-processed-profile", "gimli", - "hashbrown 0.14.5", - "indexmap 2.13.0", "ittapi", "libc", - "libm", "log", "mach2", "memfd", - "object 0.36.7", + "object", "once_cell", - "paste", "postcard", - "psm", "pulley-interpreter", "rayon", - "rustix 0.38.44", + "rustix 1.1.4", "semver", "serde", "serde_derive", "serde_json", "smallvec", - "sptr", "target-lexicon", - "wasm-encoder 0.221.3", - "wasmparser 0.221.3", - "wasmtime-asm-macros", - "wasmtime-cache", - "wasmtime-component-macro", - "wasmtime-component-util", - "wasmtime-cranelift", + "tempfile", + "wasm-compose", + "wasm-encoder 0.245.1", + "wasmparser 0.245.1", "wasmtime-environ", - "wasmtime-fiber", - "wasmtime-jit-debug", - "wasmtime-jit-icache-coherence", - "wasmtime-slab", - "wasmtime-versioned-export-macros", - "wasmtime-winch", + "wasmtime-internal-cache", + "wasmtime-internal-component-macro", + "wasmtime-internal-component-util", + "wasmtime-internal-core", + "wasmtime-internal-cranelift", + "wasmtime-internal-fiber", + "wasmtime-internal-jit-debug", + "wasmtime-internal-jit-icache-coherence", + "wasmtime-internal-unwinder", + "wasmtime-internal-versioned-export-macros", + "wasmtime-internal-winch", "wat", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] -name = "wasmtime-asm-macros" -version = "28.0.1" +name = "wasmtime-environ" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6c21dd30d1f3f93ee390ac1a7ec304ecdbfdab6390e1add41a1f52727b0992b" +checksum = "0b8b78abf3677d4a0a5db82e5015b4d085ff3a1b8b472cbb8c70d4b769f019ce" dependencies = [ - "cfg-if", + "anyhow", + "cpp_demangle", + "cranelift-bforest", + "cranelift-bitset", + "cranelift-entity", + "gimli", + "hashbrown 0.16.1", + "indexmap 2.13.0", + "log", + "object", + "postcard", + "rustc-demangle", + "semver", + "serde", + "serde_derive", + "sha2", + "smallvec", + "target-lexicon", + "wasm-encoder 0.245.1", + "wasmparser 0.245.1", + "wasmprinter", + "wasmtime-internal-component-util", + "wasmtime-internal-core", ] [[package]] -name = "wasmtime-cache" -version = "28.0.1" +name = "wasmtime-internal-cache" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cabd563cfbfe75c5bf514081f624ca8d18391a37520d8c794abce702474e688c" +checksum = "8e4fd4103ba413c0da2e636f73490c6c8e446d708cbde7573703941bc3d6a448" dependencies = [ - "anyhow", - "base64 0.21.7", + "base64 0.22.1", "directories-next", "log", "postcard", - "rustix 0.38.44", + "rustix 1.1.4", "serde", "serde_derive", "sha2", - "toml", - "windows-sys 0.59.0", + "toml 0.9.12+spec-1.1.0", + "wasmtime-environ", + "windows-sys 0.61.2", "zstd", ] [[package]] -name = "wasmtime-component-macro" -version = "28.0.1" +name = "wasmtime-internal-component-macro" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f948a6ef3119d52c9f12936970de28ddf3f9bea04bc65571f4a92d2e5ab38f4" +checksum = "0d3d6914f34be2f9d78d8ee9f422e834dfc204e71ccce697205fae95fed87892" dependencies = [ "anyhow", "proc-macro2", "quote", "syn 2.0.117", - "wasmtime-component-util", - "wasmtime-wit-bindgen", - "wit-parser 0.221.3", + "wasmtime-internal-component-util", + "wasmtime-internal-wit-bindgen", + "wit-parser 0.245.1", ] [[package]] -name = "wasmtime-component-util" -version = "28.0.1" +name = "wasmtime-internal-component-util" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9275aa01ceaaa2fa6c0ecaa5267518d80b9d6e9ae7c7ea42f4c6e073e6a69ef" +checksum = "3751b0616b914fdd87fe1bf804694a078f321b000338e6476bc48a4d6e454f21" [[package]] -name = "wasmtime-cranelift" -version = "28.0.1" +name = "wasmtime-internal-core" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0701a44a323267aae4499672dae422b266cee3135a23b640972ec8c0e10a44a2" +checksum = "22632b187e1b0716f1b9ac57ad29013bed33175fcb19e10bb6896126f82fac67" dependencies = [ "anyhow", + "hashbrown 0.16.1", + "libm", + "serde", +] + +[[package]] +name = "wasmtime-internal-cranelift" +version = "43.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b3ca07b3e0bb3429674b173b5800577719d600774dd81bff58f775c0aaa64ee" +dependencies = [ "cfg-if", "cranelift-codegen", "cranelift-control", @@ -7701,93 +9058,77 @@ dependencies = [ "cranelift-frontend", "cranelift-native", "gimli", - "itertools 0.12.1", + "itertools 0.14.0", "log", - "object 0.36.7", + "object", + "pulley-interpreter", "smallvec", "target-lexicon", - "thiserror 1.0.69", - "wasmparser 0.221.3", + "thiserror 2.0.18", + "wasmparser 0.245.1", "wasmtime-environ", - "wasmtime-versioned-export-macros", -] - -[[package]] -name = "wasmtime-environ" -version = "28.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "264c968c1b81d340355ece2be0bc31a10f567ccb6ce08512c3b7d10e26f3cbe5" -dependencies = [ - "anyhow", - "cpp_demangle", - "cranelift-bitset", - "cranelift-entity", - "gimli", - "indexmap 2.13.0", - "log", - "object 0.36.7", - "postcard", - "rustc-demangle", - "semver", - "serde", - "serde_derive", - "smallvec", - "target-lexicon", - "wasm-encoder 0.221.3", - "wasmparser 0.221.3", - "wasmprinter", - "wasmtime-component-util", + "wasmtime-internal-core", + "wasmtime-internal-unwinder", + "wasmtime-internal-versioned-export-macros", ] [[package]] -name = "wasmtime-fiber" -version = "28.0.1" +name = "wasmtime-internal-fiber" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78505221fd5bd7b07b4e1fa2804edea49dc231e626ad6861adc8f531812973e6" +checksum = "20c8b2c9704eb1f33ead025ec16038277ccb63d0a14c31e99d5b765d7c36da55" dependencies = [ - "anyhow", "cc", "cfg-if", - "rustix 0.38.44", - "wasmtime-asm-macros", - "wasmtime-versioned-export-macros", - "windows-sys 0.59.0", + "libc", + "rustix 1.1.4", + "wasmtime-environ", + "wasmtime-internal-versioned-export-macros", + "windows-sys 0.61.2", ] [[package]] -name = "wasmtime-jit-debug" -version = "28.0.1" +name = "wasmtime-internal-jit-debug" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cec0a8e5620ae71bfcaaec78e3076be5b6ebf869f4e6191925d73242224a915" +checksum = "d950310d07391d34369f62c48336ebb14eacbd4d6f772bb5f349c24e838e0664" dependencies = [ - "object 0.36.7", - "rustix 0.38.44", - "wasmtime-versioned-export-macros", + "cc", + "object", + "rustix 1.1.4", + "wasmtime-internal-versioned-export-macros", ] [[package]] -name = "wasmtime-jit-icache-coherence" -version = "28.0.1" +name = "wasmtime-internal-jit-icache-coherence" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bedb677ca1b549d98f95e9e1f9251b460090d99a2c196a0614228c064bf2e59" +checksum = "3606662c156962d096be3127b8b8ae8ee2f8be3f896dad29259ff01ddb64abfd" dependencies = [ - "anyhow", "cfg-if", "libc", - "windows-sys 0.59.0", + "wasmtime-internal-core", + "windows-sys 0.61.2", ] [[package]] -name = "wasmtime-slab" -version = "28.0.1" +name = "wasmtime-internal-unwinder" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "564905638c132c275d365c1fa074f0b499790568f43148d29de84ccecfb5cb31" +checksum = "75eef0747e52dc545b075f64fd0e0cc237ae738e641266b1970e07e2d744bc32" +dependencies = [ + "cfg-if", + "cranelift-codegen", + "log", + "object", + "wasmtime-environ", +] [[package]] -name = "wasmtime-versioned-export-macros" -version = "28.0.1" +name = "wasmtime-internal-versioned-export-macros" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e91092e6cf77390eeccee273846a9327f3e8f91c3c6280f60f37809f0e62d29" +checksum = "d8b0a5dab02a8fb527f547855ecc0e05f9fdc3d5bd57b8b080349408f9a6cece" dependencies = [ "proc-macro2", "quote", @@ -7795,12 +9136,41 @@ dependencies = [ ] [[package]] -name = "wasmtime-wasi" -version = "28.0.1" +name = "wasmtime-internal-winch" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a8e04b9a4c68ad018b330a4f4914b82b01dc3582d715ce21a93564c7f26b19f" +checksum = "8007342bd12ff400293a817973f7ecd6f1d9a8549a53369a9c1af357166f1f1e" +dependencies = [ + "cranelift-codegen", + "gimli", + "log", + "object", + "target-lexicon", + "wasmparser 0.245.1", + "wasmtime-environ", + "wasmtime-internal-cranelift", + "winch-codegen", +] + +[[package]] +name = "wasmtime-internal-wit-bindgen" +version = "43.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7900c3e3c1d6e475bc225d73b02d6d5484815f260022e6964dca9558e50dd01a" dependencies = [ "anyhow", + "bitflags 2.11.0", + "heck", + "indexmap 2.13.0", + "wit-parser 0.245.1", +] + +[[package]] +name = "wasmtime-wasi" +version = "43.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed3e3ddcfad69e9eb025bd19bff70dad45bafe1d6eacd134c0ffdfc4c161d045" +dependencies = [ "async-trait", "bitflags 2.11.0", "bytes", @@ -7813,44 +9183,29 @@ dependencies = [ "futures", "io-extras", "io-lifetimes", - "rustix 0.38.44", + "rustix 1.1.4", "system-interface", - "thiserror 1.0.69", + "thiserror 2.0.18", "tokio", "tracing", "url", "wasmtime", + "wasmtime-wasi-io", "wiggle", - "windows-sys 0.59.0", -] - -[[package]] -name = "wasmtime-winch" -version = "28.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b111d909dc604c741bd8ac2f4af373eaa5c68c34b5717271bcb687688212cef8" -dependencies = [ - "anyhow", - "cranelift-codegen", - "gimli", - "object 0.36.7", - "target-lexicon", - "wasmparser 0.221.3", - "wasmtime-cranelift", - "wasmtime-environ", - "winch-codegen", + "windows-sys 0.61.2", ] [[package]] -name = "wasmtime-wit-bindgen" -version = "28.0.1" +name = "wasmtime-wasi-io" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f38f7a5eb2f06f53fe943e7fb8bf4197f7cf279f1bc52c0ce56e9d3ffd750a4" +checksum = "3ca5dd3b9f04a851c422d05f333366722742da46bff9369ae0191f32cf83565a" dependencies = [ - "anyhow", - "heck", - "indexmap 2.13.0", - "wit-parser 0.221.3", + "async-trait", + "bytes", + "futures", + "tracing", + "wasmtime", ] [[package]] @@ -7871,7 +9226,7 @@ dependencies = [ "bumpalo", "leb128fmt", "memchr", - "unicode-width 0.2.2", + "unicode-width 0.2.0", "wasm-encoder 0.245.1", ] @@ -7886,9 +9241,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.91" +version = "0.3.92" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "854ba17bb104abfb26ba36da9729addc7ce7f06f5c0f90f3c391f8461cca21f9" +checksum = "84cde8507f4d7cfcb1185b8cb5890c494ffea65edbe1ba82cfd63661c805ed94" dependencies = [ "js-sys", "wasm-bindgen", @@ -7967,39 +9322,37 @@ dependencies = [ [[package]] name = "wiggle" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b23e3dc273d1e35cab9f38a5f76487aeeedcfa6a3fb594e209ee7b6f8b41dcc" +checksum = "cc1b1135efc8e5a008971897bea8d41ca56d8d501d4efb807842ae0a1c78f639" dependencies = [ - "anyhow", - "async-trait", "bitflags 2.11.0", - "thiserror 1.0.69", + "thiserror 2.0.18", "tracing", "wasmtime", + "wasmtime-environ", "wiggle-macro", ] [[package]] name = "wiggle-generate" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8738c5a7ef3a9de0fae10f8b84091a2aa4e059d8fef23de202ab689812b6bc6e" +checksum = "a7bc2b0d50ec8773b44fbfe1da6cb5cc44a92deaf8483233dcf0831e6db33172" dependencies = [ - "anyhow", "heck", "proc-macro2", "quote", - "shellexpand", "syn 2.0.117", + "wasmtime-environ", "witx", ] [[package]] name = "wiggle-macro" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e882267ac583e013a38a5aaeb83a49b219456ba3aa6e6772440f7213b176e8ff" +checksum = "2d6c7d44ea552e1fbfdcd7a2cd83f5c2d1e803d5b1a11e3462c06888b77f455f" dependencies = [ "proc-macro2", "quote", @@ -8029,7 +9382,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -8040,19 +9393,21 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "winch-codegen" -version = "28.0.1" +version = "43.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6232f40a795be2ce10fc761ed3b403825126a60d12491ac556ea104a932fd18a" +checksum = "eb9f45f7172a2628c8317766e427babc0a400f9d10b1c0f0b0617c5ed5b79de6" dependencies = [ - "anyhow", + "cranelift-assembler-x64", "cranelift-codegen", "gimli", "regalloc2", "smallvec", "target-lexicon", - "wasmparser 0.221.3", - "wasmtime-cranelift", + "thiserror 2.0.18", + "wasmparser 0.245.1", "wasmtime-environ", + "wasmtime-internal-core", + "wasmtime-internal-cranelift", ] [[package]] @@ -8365,6 +9720,15 @@ dependencies = [ "memchr", ] +[[package]] +name = "winnow" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a90e88e4667264a994d34e6d1ab2d26d398dcdca8b7f52bec8668957517fc7d8" +dependencies = [ + "memchr", +] + [[package]] name = "winx" version = "0.36.4" @@ -8447,9 +9811,9 @@ dependencies = [ [[package]] name = "wit-parser" -version = "0.221.3" +version = "0.244.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "896112579ed56b4a538b07a3d16e562d101ff6265c46b515ce0c701eef16b2ac" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" dependencies = [ "anyhow", "id-arena", @@ -8460,16 +9824,17 @@ dependencies = [ "serde_derive", "serde_json", "unicode-xid", - "wasmparser 0.221.3", + "wasmparser 0.244.0", ] [[package]] name = "wit-parser" -version = "0.244.0" +version = "0.245.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +checksum = "330698718e82983499419494dd1e3d7811a457a9bf9f69734e8c5f07a2547929" dependencies = [ "anyhow", + "hashbrown 0.16.1", "id-arena", "indexmap 2.13.0", "log", @@ -8478,7 +9843,7 @@ dependencies = [ "serde_derive", "serde_json", "unicode-xid", - "wasmparser 0.244.0", + "wasmparser 0.245.1", ] [[package]] @@ -8508,6 +9873,23 @@ dependencies = [ "tap", ] +[[package]] +name = "x11rb" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9993aa5be5a26815fe2c3eacfc1fde061fc1a1f094bf1ad2a18bf9c495dd7414" +dependencies = [ + "gethostname", + "rustix 1.1.4", + "x11rb-protocol", +] + +[[package]] +name = "x11rb-protocol" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea6fc2961e4ef194dcbfe56bb845534d0dc8098940c7e5c012a258bfec6701bd" + [[package]] name = "x509-cert" version = "0.2.5" @@ -8650,11 +10032,11 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2578b716f8a7a858b7f02d5bd870c14bf4ddbbcf3a4c05414ba6503640505e3" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" dependencies = [ - "zerocopy-derive 0.8.42", + "zerocopy-derive 0.8.48", ] [[package]] @@ -8670,9 +10052,9 @@ dependencies = [ [[package]] name = "zerocopy-derive" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e6cc098ea4d3bd6246687de65af3f920c430e236bee1e3bf2e441463f08a02f" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" dependencies = [ "proc-macro2", "quote", @@ -8816,6 +10198,21 @@ dependencies = [ "pkg-config", ] +[[package]] +name = "zune-core" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb8a0807f7c01457d0379ba880ba6322660448ddebc890ce29bb64da71fb40f9" + +[[package]] +name = "zune-jpeg" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296" +dependencies = [ + "zune-core", +] + [[package]] name = "zvariant" version = "4.2.0" diff --git a/Cargo.toml b/Cargo.toml index 99992a40ee2..2bb3b4e2ca0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,10 +1,11 @@ [workspace] -members = [".", "crates/ironclaw_safety"] +members = [".", "crates/ironclaw_common", "crates/ironclaw_safety", "crates/ironclaw_skills", "crates/ironclaw_engine", "crates/ironclaw_gateway", "crates/ironclaw_tui"] exclude = [ "channels-src/discord", "channels-src/telegram", "channels-src/slack", "channels-src/whatsapp", + "tools-src/composio", "tools-src/github", "tools-src/gmail", "tools-src/google-calendar", @@ -20,7 +21,7 @@ exclude = [ [package] name = "ironclaw" -version = "0.19.0" +version = "0.25.0" edition = "2024" rust-version = "1.92" description = "Secure personal AI assistant that protects your data and expands its capabilities on the fly" @@ -39,9 +40,14 @@ eula = false # Async runtime tokio = { version = "1", features = ["full"] } tokio-stream = { version = "0.1", features = ["sync"] } +tokio-util = { version = "0.7", features = ["compat"] } futures = "0.3" +tokio-tungstenite = { version = "0.26", features = ["rustls-tls-native-roots"] } eventsource-stream = "0.2" +# Agent Client Protocol (ACP) — standard communication with coding agents +agent-client-protocol = "0.10" + # HTTP client reqwest = { version = "0.12", default-features = false, features = ["json", "multipart", "rustls-tls-native-roots", "stream"] } @@ -57,6 +63,7 @@ refinery = { version = "0.8", features = ["tokio-postgres"], optional = true } tokio-postgres-rustls = { version = "0.13", optional = true } rustls = { version = "0.23", optional = true, default-features = false } rustls-native-certs = { version = "0.8", optional = true } +webpki-roots = { version = "0.26", optional = true } # Database - libSQL/Turso (optional embedded database) libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication", "remote", "tls"] } @@ -95,13 +102,25 @@ termimad = "0.34" # Channel integrations axum = { version = "0.8", features = ["ws"] } tower = "0.5" -tower-http = { version = "0.6", features = ["trace", "cors", "set-header"] } +tower-http = { version = "0.6", features = ["trace", "cors", "set-header", "catch-panic"] } # Cron scheduling for routines cron = "0.13" +# JSON Schema validation (workspace document metadata). +# default-features disabled to avoid pulling a second `reqwest` major version +# for remote $ref resolution (we only validate against in-memory schemas). +jsonschema = { version = "0.45", default-features = false } + +# Shared types +ironclaw_common = { path = "crates/ironclaw_common", version = "0.2.0" } + # Safety/sanitization -ironclaw_safety = { path = "crates/ironclaw_safety", version = "0.1.0" } +ironclaw_engine = { path = "crates/ironclaw_engine", version = "0.1.0" } +ironclaw_gateway = { path = "crates/ironclaw_gateway", version = "0.1.0" } +ironclaw_safety = { path = "crates/ironclaw_safety", version = "0.2.1" } +ironclaw_skills = { path = "crates/ironclaw_skills", version = "0.1.0" } +ironclaw_tui = { path = "crates/ironclaw_tui", optional = true, version = "0.1.0" } regex = "1" aho-corasick = "1" @@ -111,6 +130,7 @@ serde_yml = "0.0.12" # Filesystem paths dirs = "6" fs4 = "0.6" +glob = "0.3" # Semantic versioning semver = "1" @@ -130,9 +150,9 @@ open = "5" pgvector = { version = "0.4", features = ["postgres"], optional = true } # WASM sandbox for untrusted tool execution -wasmtime = { version = "28", features = ["component-model"] } -wasmtime-wasi = "28" # WASI support for component model -wasmparser = "0.220" # WASM binary parsing for validation +wasmtime = { version = "43.0.1", features = ["component-model"] } +wasmtime-wasi = "43.0.1" # WASI support for component model +wasmparser = "0.245.1" # WASM binary parsing for validation # Cryptography for secrets management aes-gcm = "0.10" @@ -168,6 +188,8 @@ hyper-util = { version = "0.1", features = ["server", "tokio", "http1", "http2"] http-body-util = "0.1" bytes = "1" base64 = "0.22.1" +cookie = "0.18" +jsonwebtoken = "9" mime_guess = "2.0.5" clap_complete = "4.5.0" lru = "0.16.3" @@ -176,6 +198,7 @@ lru = "0.16.3" html-to-markdown-rs = { version = "2.3", optional = true } readabilityrs = { version = "0.1.2", optional = true } ed25519-dalek = { version = "2.2.0", features = ["std"] } +bs58 = "0.5" hex = "0.4.3" # OpenClaw import (feature gated) @@ -185,37 +208,35 @@ json5 = { version = "0.4", optional = true } [target.'cfg(target_os = "macos")'.dependencies] security-framework = "3" +# PTY allocation for Claude CLI stdout buffering fix (Unix only) +[target.'cfg(unix)'.dependencies] +pty-process = { version = "0.5", features = ["async"] } + # Linux secret-service (GNOME Keyring, KWallet) [target.'cfg(target_os = "linux")'.dependencies] secret-service = { version = "4", features = ["rt-tokio-crypto-rust"] } zbus = "4" +[build-dependencies] +serde_json = "1" + [dev-dependencies] tokio-test = "0.4" tracing-test = "0.2" -tokio-tungstenite = "0.26" testcontainers-modules = { version = "0.11", features = ["postgres"] } pretty_assertions = "1" tempfile = "3" insta = "1.46.3" -criterion = "0.5" - -[[bench]] -name = "safety_check" -harness = false - -[[bench]] -name = "safety_pipeline" -harness = false [features] -default = ["postgres", "libsql", "html-to-markdown"] +default = ["postgres", "libsql", "html-to-markdown", "tui"] postgres = [ "dep:deadpool-postgres", "dep:tokio-postgres", "dep:tokio-postgres-rustls", "dep:rustls", "dep:rustls-native-certs", + "dep:webpki-roots", "dep:postgres-types", "dep:refinery", "dep:pgvector", @@ -227,6 +248,7 @@ libsql = ["dep:libsql"] integration = [] html-to-markdown = ["dep:html-to-markdown-rs", "dep:readabilityrs"] bedrock = ["dep:aws-config", "dep:aws-sdk-bedrockruntime", "dep:aws-smithy-types"] +tui = ["dep:ironclaw_tui"] import = ["dep:json5", "libsql"] [[test]] @@ -243,8 +265,7 @@ strip = true # Remove debug symbols from release binaries # The profile that 'cargo dist' will build with [profile.dist] inherits = "release" -lto = "fat" # Full cross-crate LTO (slow build, better codegen) -codegen-units = 1 # Single codegen unit for maximum optimization +lto = "thin" # Config for 'dist' [workspace.metadata.dist] diff --git a/Dockerfile b/Dockerfile index a2c2610d6f1..2f1c208e2e7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,56 +1,154 @@ # Multi-stage Dockerfile for the IronClaw agent (cloud deployment). # +# Uses cargo-chef for dependency caching — only rebuilds deps when +# Cargo.toml/Cargo.lock change, not on every source edit. +# +# Debian-based build + runtime. The bundled libSQL/SQLite C code has +# threading issues when statically linked against musl (segfault on +# database reopen), so we use glibc. +# # Build: -# docker build --platform linux/amd64 -t ironclaw:latest . +# docker build --platform linux/amd64 --target runtime -t ironclaw:latest . # # Run: # docker run --env-file .env -p 3000:3000 ironclaw:latest -# Stage 1: Build -FROM rust:1.92-slim-bookworm AS builder +# Stage 1: Install cargo-chef +FROM rust:1.92-bookworm AS chef -RUN apt-get update && apt-get install -y --no-install-recommends \ - pkg-config libssl-dev cmake gcc g++ \ - && rm -rf /var/lib/apt/lists/* \ - && rustup target add wasm32-wasip2 \ - && cargo install wasm-tools +RUN rustup target add wasm32-wasip2 \ + && cargo install cargo-chef@0.1.77 wasm-tools@1.246.1 WORKDIR /app -# Copy manifests first for layer caching +# Stage 2: Generate the dependency recipe (changes only when Cargo.toml/lock change) +FROM chef AS planner + COPY Cargo.toml Cargo.lock ./ COPY crates/ crates/ +COPY build.rs build.rs +COPY src/ src/ +COPY tests/ tests/ +COPY migrations/ migrations/ +COPY registry/ registry/ +COPY channels-src/ channels-src/ +COPY tools-src/ tools-src/ +COPY wit/ wit/ +COPY providers.json providers.json + +RUN cargo chef prepare --recipe-path recipe.json + +# Stage 3: Build dependencies (cached unless Cargo.toml/lock change) +FROM chef AS deps + +# Docker-only overrides for the dist profile (not in Cargo.toml because +# cargo-dist uses dist for release binaries that need unwinding). +ENV CARGO_PROFILE_DIST_PANIC=abort \ + CARGO_PROFILE_DIST_CODEGEN_UNITS=1 + +COPY --from=planner /app/recipe.json recipe.json +RUN cargo chef cook --profile dist --recipe-path recipe.json + +# Stage 4: Build the actual binary (only recompiles ironclaw source) +FROM deps AS builder -# Copy source, build script, tests, and supporting directories +COPY Cargo.toml Cargo.lock ./ +COPY crates/ crates/ COPY build.rs build.rs COPY src/ src/ COPY tests/ tests/ COPY migrations/ migrations/ COPY registry/ registry/ COPY channels-src/ channels-src/ +COPY tools-src/ tools-src/ COPY wit/ wit/ COPY providers.json providers.json -# [[bench]] entries in Cargo.toml require bench sources to exist for cargo to parse the manifest -COPY benches/ benches/ +COPY profiles/ profiles/ + +RUN cargo build --profile dist --bin ironclaw + +# Stage 4b: Build all WASM extensions from source (only used by runtime-staging) +# +# Inherits from chef (not builder) so WASM extensions only rebuild when +# tools-src/, channels-src/, registry/, or wit/ change — not on every +# src/ edit. The extensions are standalone crates with their own lockfiles. +FROM chef AS wasm-builder -RUN cargo build --release --bin ironclaw +RUN apt-get update && apt-get install -y --no-install-recommends jq && rm -rf /var/lib/apt/lists/* -# Stage 2: Runtime -FROM debian:bookworm-slim +COPY tools-src/ tools-src/ +COPY channels-src/ channels-src/ +COPY registry/ registry/ +COPY wit/ wit/ -RUN apt-get update && apt-get install -y --no-install-recommends \ - ca-certificates libssl3 \ +RUN set -eux; \ + mkdir -p /app/wasm-bundles/tools /app/wasm-bundles/channels; \ + for manifest in registry/tools/*.json registry/channels/*.json; do \ + [ -f "$manifest" ] || continue; \ + kind=$(jq -r '.kind' "$manifest"); \ + ext_name=$(jq -r '.name' "$manifest"); \ + source_dir=$(jq -r '.source.dir' "$manifest"); \ + caps_file=$(jq -r '.source.capabilities' "$manifest"); \ + crate_name=$(jq -r '.source.crate_name' "$manifest"); \ + [ -d "$source_dir" ] || continue; \ + # Telegram is embedded in the binary at build time; skip it + [ "$ext_name" = "telegram" ] && continue; \ + echo "=== Building $ext_name from $source_dir ==="; \ + if [ -f "$source_dir/Cargo.lock" ]; then \ + CARGO_TARGET_DIR=/app/target cargo build --locked --release --target wasm32-wasip2 \ + --manifest-path "$source_dir/Cargo.toml" || { echo "WARN: build failed for $ext_name"; continue; }; \ + else \ + CARGO_TARGET_DIR=/app/target cargo build --release --target wasm32-wasip2 \ + --manifest-path "$source_dir/Cargo.toml" || { echo "WARN: build failed for $ext_name"; continue; }; \ + fi; \ + wasm_artifact=$(echo "${crate_name}" | tr '-' '_'); \ + raw_wasm="/app/target/wasm32-wasip2/release/${wasm_artifact}.wasm"; \ + [ -f "$raw_wasm" ] || continue; \ + dest_dir="/app/wasm-bundles/tools"; \ + [ "$kind" = "channel" ] && dest_dir="/app/wasm-bundles/channels"; \ + wasm-tools component new "$raw_wasm" -o "$dest_dir/${ext_name}.wasm" 2>/dev/null \ + || cp "$raw_wasm" "$dest_dir/${ext_name}.wasm"; \ + wasm-tools strip "$dest_dir/${ext_name}.wasm" -o "$dest_dir/${ext_name}.wasm.tmp" 2>/dev/null \ + && mv "$dest_dir/${ext_name}.wasm.tmp" "$dest_dir/${ext_name}.wasm" \ + || true; \ + [ -f "$source_dir/$caps_file" ] && cp "$source_dir/$caps_file" "$dest_dir/${ext_name}.capabilities.json"; \ + echo " -> $dest_dir/${ext_name}.wasm"; \ + done; \ + count=$(find /app/wasm-bundles -name '*.wasm' | wc -l); \ + echo "Built $count WASM extensions"; \ + [ "$count" -gt 0 ] || { echo "ERROR: No WASM extensions were built"; exit 1; } + +# Stage 5a: Shared runtime base +FROM debian:bookworm-slim AS runtime-base + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates \ && rm -rf /var/lib/apt/lists/* -COPY --from=builder /app/target/release/ironclaw /usr/local/bin/ironclaw +COPY --from=builder /app/target/dist/ironclaw /usr/local/bin/ironclaw COPY --from=builder /app/migrations /app/migrations # Non-root user -RUN useradd -m -u 1000 -s /bin/bash ironclaw -USER ironclaw +ENV HOME=/home/ironclaw +RUN useradd -m -d /home/ironclaw -u 1000 ironclaw \ + && mkdir -p /home/ironclaw/.ironclaw \ + && chown -R ironclaw:ironclaw /home/ironclaw +WORKDIR /home/ironclaw EXPOSE 3000 ENV RUST_LOG=ironclaw=info ENTRYPOINT ["ironclaw"] + +# Stage 5b: Production runtime (no pre-bundled extensions) +FROM runtime-base AS runtime +USER ironclaw + +# Stage 5c: Staging runtime (with pre-built WASM extensions) +# Last stage = default target. Railway doesn't support --target, so this +# must be last for Railway deploys. CI uses explicit --target flags. +FROM runtime-base AS runtime-staging +COPY --from=wasm-builder --chown=ironclaw:ironclaw /app/wasm-bundles/tools/ /home/ironclaw/.ironclaw/tools/ +COPY --from=wasm-builder --chown=ironclaw:ironclaw /app/wasm-bundles/channels/ /home/ironclaw/.ironclaw/channels/ +USER ironclaw diff --git a/FEATURE_PARITY.md b/FEATURE_PARITY.md index ad2db551177..7be7b0c2829 100644 --- a/FEATURE_PARITY.md +++ b/FEATURE_PARITY.md @@ -38,7 +38,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O | HTTP endpoints for Control UI | ✅ | ✅ | Web dashboard with chat, memory, jobs, logs, extensions | | Channel connection lifecycle | ✅ | ✅ | ChannelManager + WebSocket tracker | | Session management/routing | ✅ | ✅ | SessionManager exists | -| Configuration hot-reload | ✅ | ❌ | | +| Configuration hot-reload | ✅ | 🚧 | LLM backend/model settings hot-reload via web UI when the reload handle is available; broader config reload still pending | | Network modes (loopback/LAN/remote) | ✅ | 🚧 | HTTP only | | OpenAI-compatible HTTP API | ✅ | ✅ | /v1/chat/completions, per-request `model` override | | Canvas hosting | ✅ | ❌ | Agent-driven UI | @@ -69,8 +69,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O | REPL (simple) | ✅ | ✅ | - | For testing | | WASM channels | ❌ | ✅ | - | IronClaw innovation; host resolves owner scope vs sender identity | | WhatsApp | ✅ | ❌ | P1 | Baileys (Web), same-phone mode with echo detection | -| Telegram | ✅ | ✅ | - | WASM channel(MTProto), DM pairing, caption, /start, bot_username, DM topics, setup-time owner auto-verification, owner-scoped persistence | -| Discord | ✅ | ❌ | P2 | discord.js, thread parent binding inheritance | +| Telegram | ✅ | ✅ | - | WASM channel(MTProto), polling-first setup, DM pairing, caption, /start, bot_username, DM topics, web/UI ownership claim flow, owner-scoped persistence | +| Discord | ✅ | 🚧 | P2 | Gateway `MESSAGE_CREATE` intake restored via websocket queue + WASM poll; Gateway DMs now respect pairing; thread parent binding inheritance and reply/thread parity still incomplete | | Signal | ✅ | ✅ | P2 | signal-cli daemonPC, SSE listener HTTP/JSON-R, user/group allowlists, DM pairing | | Slack | ✅ | ✅ | - | WASM tool | | iMessage | ✅ | ❌ | P3 | BlueBubbles or Linq recommended | @@ -97,6 +97,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O | Cron/heartbeat topic targeting | ✅ | ❌ | Messages land in correct topic | | DM topics support | ✅ | ❌ | Agent/topic bindings in DMs and agent-scoped SessionKeys | | Persistent ACP topic binding | ✅ | ❌ | ACP harness sessions can pin to Telegram forum or DM topics | +| sendVoice (voice note replies) | ✅ | ✅ | audio/ogg attachments sent as voice notes; prerequisite for TTS (#90) | ### Discord-Specific Features (since Feb 2025) @@ -112,7 +113,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O |---------|----------|----------|-------| | Streaming draft replies | ✅ | ❌ | Partial replies via draft message updates | | Configurable stream modes | ✅ | ❌ | Per-channel stream behavior | -| Thread ownership | ✅ | ❌ | Thread-level ownership tracking plus reply participation memory | +| Thread ownership | ✅ | 🚧 | Reply participation memory now persists with TTL-bounded tracking; full thread-level ownership tracking is still missing | | Download-file action | ✅ | ❌ | On-demand attachment downloads via message actions | ### Mattermost-Specific Features (since Mar 2026) @@ -349,6 +350,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O | MMR re-ranking | ✅ | ❌ | Maximal marginal relevance for result diversity | | LLM-based query expansion | ✅ | ❌ | Expand FTS queries via LLM | | OpenAI embeddings | ✅ | ✅ | | +| Bedrock embeddings | ❌ | ✅ | Reuses Bedrock region/profile auth for Titan Text Embeddings V2 | | Gemini embeddings | ✅ | ❌ | | | Local embeddings | ✅ | ❌ | | | SQLite-vec backend | ✅ | ❌ | IronClaw uses PostgreSQL | @@ -558,7 +560,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O ### P1 - High Priority - ❌ Slack channel (real implementation) -- ✅ Telegram channel (WASM, DM pairing, caption, /start) +- ✅ Telegram channel (WASM, polling-first setup, DM pairing, caption, /start) - ❌ WhatsApp channel - ✅ Multi-provider failover (`FailoverProvider` with retryable error classification) - ✅ Hooks system (core lifecycle hooks + bundled/plugin/workspace hooks + outbound webhooks) diff --git a/README.ja.md b/README.ja.md index 887cf67e5f4..cc6e31b4110 100644 --- a/README.ja.md +++ b/README.ja.md @@ -18,7 +18,8 @@ English | 简体中文 | Русский | - 日本語 + 日本語 | + 한국어

@@ -180,7 +181,7 @@ LLM_API_KEY=sk-or-... LLM_MODEL=anthropic/claude-sonnet-4 ``` -完全なプロバイダーガイドは[docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md)をご覧ください。 +完全なプロバイダーガイドは[docs/capabilities/llm-providers.md](docs/capabilities/llm-providers.md)をご覧ください。 ## セキュリティ @@ -306,7 +307,7 @@ cargo test cargo test test_name ``` -- **Telegramチャネル**: セットアップとDMペアリングについては[docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md)を参照してください。 +- **チャネル**: Telegram、Discord、その他のチャネルの設定は[docs/channels/overview.mdx](docs/channels/overview.mdx)を参照してください。 - **チャネルソースの変更**: `cargo build`の前に`./channels-src/telegram/build.sh`を実行して、更新されたWASMをバンドルしてください。 ## OpenClawの系譜 diff --git a/README.ko.md b/README.ko.md new file mode 100644 index 00000000000..903b8d1c2a1 --- /dev/null +++ b/README.ko.md @@ -0,0 +1,338 @@ +

+ IronClaw +

+ +

IronClaw

+ +

+ 언제나 당신 편인 안전한 개인 AI 어시스턴트 +

+ +

+ License: MIT OR Apache-2.0 + Telegram: @ironclawAI + Reddit: r/ironclawAI + + gitcgr + +

+ +

+ English | + 简体中文 | + Русский | + 日本語 | + 한국어 +

+ +

+ 철학 • + 기능 • + 설치 • + 설정 • + 보안 • + 아키텍처 +

+ +--- + +## 철학 + +IronClaw는 단순한 원칙 위에 만들어졌습니다: **AI 어시스턴트는 당신을 위해 일해야 하며, 당신을 거슬러서는 안 됩니다**. + +AI 시스템이 데이터 처리에 대해 점점 더 불투명해지고 기업의 이익에 맞춰지는 세상에서, IronClaw는 다른 접근 방식을 취합니다: + +- **데이터는 당신의 것** - 모든 정보는 로컬에 저장되고 암호화되며, 절대 당신의 통제를 벗어나지 않습니다 +- **설계에 의한 투명성** - 오픈 소스, 감사 가능, 숨겨진 텔레메트리나 데이터 수집 없음 +- **자가 확장 기능** - 공급업체의 업데이트를 기다리지 않고 즉석에서 새로운 도구를 만들 수 있습니다 +- **심층 방어** - 프롬프트 인젝션 및 데이터 유출로부터 보호하는 다중 보안 계층 + +IronClaw는 개인적, 직업적 삶에서 실제로 신뢰할 수 있는 AI 어시스턴트입니다. + +## 기능 + +### 보안 우선 + +- **WASM 샌드박스** - 신뢰할 수 없는 도구는 권한 기반의 격리된 WebAssembly 컨테이너에서 실행됩니다 +- **자격 증명 보호** - 비밀은 도구에 노출되지 않고, 누출 감지와 함께 호스트 경계에서 주입됩니다 +- **프롬프트 인젝션 방어** - 패턴 감지, 콘텐츠 정화, 정책 시행 +- **엔드포인트 화이트리스트** - HTTP 요청은 명시적으로 승인된 호스트와 경로로만 전송됩니다 + +### 항상 사용 가능 + +- **다중 채널** - REPL, HTTP 웹훅, WASM 채널 (Telegram, Slack), 웹 게이트웨이 +- **Docker 샌드박스** - 작업별 토큰과 오케스트레이터/워커 패턴을 사용한 격리된 컨테이너 실행 +- **웹 게이트웨이** - 실시간 SSE/WebSocket 스트리밍이 있는 브라우저 UI +- **루틴** - 백그라운드 자동화를 위한 cron 일정, 이벤트 트리거, 웹훅 핸들러 +- **하트비트 시스템** - 모니터링 및 유지 보수 작업을 위한 사전 백그라운드 실행 +- **병렬 작업** - 격리된 컨텍스트로 여러 요청을 동시에 처리합니다 +- **자가 복구** - 중단된 작업의 자동 감지 및 복구 + +### 자가 확장 + +- **동적 도구 빌드** - 필요한 것을 설명하면 IronClaw가 WASM 도구로 만들어 줍니다 +- **MCP 프로토콜** - 추가 기능을 위해 Model Context Protocol 서버에 연결합니다 +- **플러그인 아키텍처** - 재시작 없이 새로운 WASM 도구와 채널을 추가할 수 있습니다 + +### 영구 메모리 + +- **하이브리드 검색** - Reciprocal Rank Fusion을 사용한 전체 텍스트 + 벡터 검색 +- **워크스페이스 파일시스템** - 노트, 로그, 컨텍스트를 위한 유연한 경로 기반 저장소 +- **아이덴티티 파일** - 세션 간 일관된 성격과 선호도를 유지합니다 + +## 설치 + +### 사전 요구 사항 + +- Rust 1.85+ +- [pgvector](https://github.com/pgvector/pgvector) 확장이 있는 PostgreSQL 15+ +- NEAR AI 계정 (인증은 설정 마법사를 통해 처리됨) + +## 다운로드 또는 빌드 + +[릴리스 페이지](https://github.com/nearai/ironclaw/releases/)를 방문하여 최신 업데이트를 확인하세요. + +
+ Windows 인스톨러로 설치 (Windows) + +[Windows 인스톨러](https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-x86_64-pc-windows-msvc.msi)를 다운로드하여 실행하세요. + +
+ +
+ PowerShell 스크립트로 설치 (Windows) + +```sh +irm https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.ps1 | iex +``` + +
+ +
+ 셸 스크립트로 설치 (macOS, Linux, Windows/WSL) + +```sh +curl --proto '=https' --tlsv1.2 -LsSf https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.sh | sh +``` +
+ +
+ Homebrew로 설치 (macOS/Linux) + +```sh +brew install ironclaw +``` + +
+ +
+ 소스 코드 컴파일 (Windows, Linux, macOS의 Cargo) + +`cargo`로 설치하세요. 컴퓨터에 [Rust](https://rustup.rs)가 설치되어 있는지 확인하세요. + +```bash +# 저장소 복제 +git clone https://github.com/nearai/ironclaw.git +cd ironclaw + +# 빌드 +cargo build --release + +# 테스트 실행 +cargo test +``` + +**전체 릴리스**의 경우 (채널 소스를 수정한 후), `./scripts/build-all.sh`를 실행하여 채널을 먼저 다시 빌드하세요. + +
+ +### 데이터베이스 설정 + +```bash +# 데이터베이스 생성 +createdb ironclaw + +# pgvector 활성화 +psql ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;" +``` + +## 설정 + +설정 마법사를 실행하여 IronClaw를 구성하세요: + +```bash +ironclaw onboard +``` + +마법사는 데이터베이스 연결, NEAR AI 인증 (브라우저 OAuth를 통해), +그리고 비밀 암호화 (시스템 키체인 사용)를 처리합니다. 설정은 연결된 +데이터베이스에 저장됩니다. 부트스트랩 변수 (예: `DATABASE_URL`, `LLM_BACKEND`)는 +데이터베이스가 연결되기 전에 사용할 수 있도록 `~/.ironclaw/.env`에 기록됩니다. + +### 대체 LLM 공급자 + +IronClaw는 기본적으로 NEAR AI를 사용하지만 많은 LLM 공급자를 기본 지원합니다. +내장 공급자에는 **Anthropic**, **OpenAI**, **GitHub Copilot**, **Google Gemini**, **MiniMax**, +**Mistral**, **Ollama** (로컬)이 포함됩니다. **OpenRouter** +(300+ 모델), **Together AI**, **Fireworks AI**, 자체 호스팅 서버 (**vLLM**, +**LiteLLM**) 같은 OpenAI 호환 서비스도 지원됩니다. + +마법사에서 공급자를 선택하거나 환경 변수를 직접 설정하세요: + +```env +# 예: MiniMax (내장, 204K 컨텍스트) +LLM_BACKEND=minimax +MINIMAX_API_KEY=... + +# 예: OpenAI 호환 엔드포인트 +LLM_BACKEND=openai_compatible +LLM_BASE_URL=https://openrouter.ai/api/v1 +LLM_API_KEY=sk-or-... +LLM_MODEL=anthropic/claude-sonnet-4 +``` + +전체 공급자 가이드는 [docs/capabilities/llm-providers.md](docs/capabilities/llm-providers.md)를 참조하세요. + +## 보안 + +IronClaw는 데이터를 보호하고 오용을 방지하기 위해 심층 방어를 구현합니다. + +### WASM 샌드박스 + +신뢰할 수 없는 모든 도구는 격리된 WebAssembly 컨테이너에서 실행됩니다: + +- **권한 기반 권한** - HTTP, 비밀, 도구 호출에 대한 명시적 옵트인 +- **엔드포인트 화이트리스트** - HTTP 요청은 승인된 호스트/경로로만 전송됩니다 +- **자격 증명 주입** - 비밀은 호스트 경계에서 주입되며, WASM 코드에 절대 노출되지 않습니다 +- **누출 감지** - 비밀 유출 시도에 대해 요청과 응답을 스캔합니다 +- **속도 제한** - 남용을 방지하기 위한 도구별 요청 제한 +- **리소스 제한** - 메모리, CPU, 실행 시간 제약 + +``` +WASM ──► 화이트리스트 ──► 누출 스캔 ──► 자격 증명 ──► 실행 ──► 누출 스캔 ──► WASM + 검증기 (요청) 주입기 요청 (응답) +``` + +### 프롬프트 인젝션 방어 + +외부 콘텐츠는 여러 보안 계층을 통과합니다: + +- 인젝션 시도의 패턴 기반 감지 +- 콘텐츠 정화 및 이스케이핑 +- 심각도 수준이 있는 정책 규칙 (차단/경고/검토/정화) +- 안전한 LLM 컨텍스트 주입을 위한 도구 출력 래핑 + +### 데이터 보호 + +- 모든 데이터는 로컬 PostgreSQL 데이터베이스에 저장됩니다 +- 비밀은 AES-256-GCM으로 암호화됩니다 +- 텔레메트리, 분석, 데이터 공유 없음 +- 모든 도구 실행에 대한 전체 감사 로그 + +## 아키텍처 + +``` +┌────────────────────────────────────────────────────────────────┐ +│ 채널 │ +│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ REPL │ │ HTTP │ │ WASM 채널 │ │ 웹 게이트웨이│ │ +│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │ │ +│ │ │ │ └──────┬──────┘ │ +│ └─────────┴──────────────┴────────────────┘ │ +│ │ │ +│ ┌─────────▼─────────┐ │ +│ │ 에이전트 루프 │ 의도 라우팅 │ +│ └────┬──────────┬───┘ │ +│ │ │ │ +│ ┌──────────▼────┐ ┌──▼───────────────┐ │ +│ │ 스케줄러 │ │ 루틴 엔진 │ │ +│ │ (병렬 작업) │ │ (cron, 이벤트, wh)│ │ +│ └──────┬────────┘ └────────┬─────────┘ │ +│ │ │ │ +│ ┌─────────────┼────────────────────┘ │ +│ │ │ │ +│ ┌───▼─────┐ ┌────▼────────────────┐ │ +│ │ 로컬 │ │ 오케스트레이터 │ │ +│ │ 워커 │ │ ┌───────────────┐ │ │ +│ │(인프로세스)│ │ │Docker 샌드박스│ │ │ +│ └───┬─────┘ │ │ 컨테이너 │ │ │ +│ │ │ │ ┌───────────┐ │ │ │ +│ │ │ │ │Worker / CC│ │ │ │ +│ │ │ │ └───────────┘ │ │ │ +│ │ │ └───────────────┘ │ │ +│ │ └─────────┬───────────┘ │ +│ └──────────────────┤ │ +│ │ │ +│ ┌───────────▼──────────┐ │ +│ │ 도구 레지스트리 │ │ +│ │ 내장, MCP, WASM │ │ +│ └──────────────────────┘ │ +└────────────────────────────────────────────────────────────────┘ +``` + +### 핵심 구성 요소 + +| 구성 요소 | 목적 | +|-----------|---------| +| **에이전트 루프** | 주요 메시지 처리 및 작업 조정 | +| **라우터** | 사용자 의도 분류 (명령, 쿼리, 작업) | +| **스케줄러** | 우선순위가 있는 병렬 작업 실행 관리 | +| **워커** | LLM 추론과 도구 호출로 작업 실행 | +| **오케스트레이터** | 컨테이너 라이프사이클, LLM 프록시, 작업별 인증 | +| **웹 게이트웨이** | 채팅, 메모리, 작업, 로그, 확장, 루틴이 있는 브라우저 UI | +| **루틴 엔진** | 예약된 (cron) 및 반응형 (이벤트, 웹훅) 백그라운드 작업 | +| **워크스페이스** | 하이브리드 검색이 있는 영구 메모리 | +| **안전 계층** | 프롬프트 인젝션 방어 및 콘텐츠 정화 | + +## 사용법 + +```bash +# 첫 설정 (데이터베이스, 인증 등 구성) +ironclaw onboard + +# 대화형 REPL 시작 +cargo run + +# 디버그 로깅 사용 +RUST_LOG=ironclaw=debug cargo run +``` + +## 개발 + +```bash +# 코드 포맷 +cargo fmt + +# 린트 +cargo clippy --all --benches --tests --examples --all-features + +# 테스트 실행 +createdb ironclaw_test +cargo test + +# 특정 테스트 실행 +cargo test test_name +``` + +- **채널**: Telegram, Discord 및 기타 채널 설정은 [docs/channels/overview.mdx](docs/channels/overview.mdx)를 참조하세요. +- **채널 소스 변경**: 업데이트된 WASM이 번들되도록 `cargo build` 전에 `./channels-src/telegram/build.sh`를 실행하세요. + +## OpenClaw 역사 + +IronClaw는 [OpenClaw](https://github.com/openclaw/openclaw)에서 영감을 받은 Rust 재구현입니다. 전체 추적 매트릭스는 [FEATURE_PARITY.md](FEATURE_PARITY.md)를 참조하세요. + +주요 차이점: + +- **Rust vs TypeScript** - 네이티브 성능, 메모리 안전, 단일 바이너리 +- **WASM 샌드박스 vs Docker** - 가벼운 권한 기반 보안 +- **PostgreSQL vs SQLite** - 프로덕션 준비된 영속성 +- **보안 우선 설계** - 다중 방어 계층, 자격 증명 보호 + +## 라이선스 + +다음 중 하나를 선택하여 라이선스가 부여됩니다: + +- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE)) +- MIT License ([LICENSE-MIT](LICENSE-MIT)) + +원하는 대로 선택할 수 있습니다. diff --git a/README.md b/README.md index cb759236be6..c99e0f4b561 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,8 @@ English | 简体中文 | Русский | - 日本語 + 日本語 | + 한국어

@@ -190,7 +191,7 @@ LLM_API_KEY=sk-or-... LLM_MODEL=anthropic/claude-sonnet-4 ``` -See [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) for a full provider guide. +See [docs/capabilities/llm-providers.md](docs/capabilities/llm-providers.md) for a full provider guide. ## Security @@ -313,7 +314,7 @@ cargo test cargo test test_name ``` -- **Telegram channel**: See [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) for setup and DM pairing. +- **Channels**: See [docs/channels/overview.mdx](docs/channels/overview.mdx) for setup of Telegram, Discord, and other channels. - **Changing channel sources**: Run `./channels-src/telegram/build.sh` before `cargo build` so the updated WASM is bundled. ## OpenClaw Heritage diff --git a/README.ru.md b/README.ru.md index 0546e7f44c5..06689c04d59 100644 --- a/README.ru.md +++ b/README.ru.md @@ -18,7 +18,8 @@ English | 简体中文 | Русский | - 日本語 + 日本語 | + 한국어

@@ -184,7 +185,7 @@ LLM_API_KEY=sk-or-... LLM_MODEL=anthropic/claude-sonnet-4 ``` -Смотрите [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) для получения полного руководства по провайдерам. +Смотрите [docs/capabilities/llm-providers.md](docs/capabilities/llm-providers.md) для получения полного руководства по провайдерам. ## Безопасность @@ -308,7 +309,7 @@ cargo test cargo test название_теста ``` -- **Telegram-канал**: Смотрите [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) для настройки и привязки аккаунта. +- **Каналы**: Смотрите [docs/channels/overview.mdx](docs/channels/overview.mdx) для настройки Telegram, Discord и других каналов. - **Изменение исходников каналов**: Перед `cargo build` выполните `./channels-src/telegram/build.sh`, чтобы обновить встроенный WASM. ## Наследие OpenClaw diff --git a/README.zh-CN.md b/README.zh-CN.md index d818872acfa..d840793b618 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -18,7 +18,8 @@ English | 简体中文 | Русский | - 日本語 + 日本語 | + 한국어

@@ -181,7 +182,7 @@ LLM_API_KEY=sk-or-... LLM_MODEL=anthropic/claude-sonnet-4 ``` -详见 [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) 获取完整的提供商指南。 +详见 [docs/capabilities/llm-providers.md](docs/capabilities/llm-providers.md) 获取完整的提供商指南。 ## 安全机制 @@ -304,7 +305,7 @@ cargo test cargo test test_name ``` -- **Telegram 渠道**：参见 [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) 了解设置和私信配对。 +- **渠道**：参见 [docs/channels/overview.mdx](docs/channels/overview.mdx) 了解 Telegram、Discord 和其他渠道的设置。 - **修改渠道源码**：在 `cargo build` 之前运行 `./channels-src/telegram/build.sh` 以便打包更新后的 WASM。 ## OpenClaw 传承 diff --git a/build.rs b/build.rs index c2b939237e6..9a2877dc5b0 100644 --- a/build.rs +++ b/build.rs @@ -20,6 +20,9 @@ fn main() { // ── Embed registry manifests ──────────────────────────────────────── embed_registry_catalog(&root); + // ── Embed bundled skills ──────────────────────────────────────────── + embed_skills(&root); + // ── Build Telegram channel WASM ───────────────────────────────────── let channel_dir = root.join("channels-src/telegram"); let wasm_out = channel_dir.join("telegram.wasm"); @@ -121,11 +124,15 @@ fn embed_registry_catalog(root: &Path) { let registry_dir = root.join("registry"); - // Rerun if the bundles file changes (per-file watches for tools/channels - // are emitted inside collect_json_files to track content changes reliably). + // Directory-level watches ensure Cargo reruns build.rs when new files are + // added or removed. Per-file watches (emitted inside collect_json_files) + // cover content changes to existing files. println!("cargo:rerun-if-changed=registry/_bundles.json"); + println!("cargo:rerun-if-changed=registry/tools"); + println!("cargo:rerun-if-changed=registry/channels"); + println!("cargo:rerun-if-changed=registry/mcp-servers"); - let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); + let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); // safety: build script let out_path = out_dir.join("embedded_catalog.json"); if !registry_dir.is_dir() { @@ -177,7 +184,60 @@ fn embed_registry_catalog(root: &Path) { bundles_raw, ); - fs::write(&out_path, catalog).unwrap(); + fs::write(&out_path, catalog).unwrap(); // safety: build script +} + +/// Collect all `skills/*/SKILL.md` files into an embedded JSON blob. +/// +/// Output: `$OUT_DIR/embedded_skills.json` — a JSON array of `{"name": "...", "content": "..."}`. +/// These are loaded at runtime as bundled skills (lowest discovery priority, Trusted trust level). +fn embed_skills(root: &Path) { + use std::fs; + + let skills_dir = root.join("skills"); + + // Rerun when any skill changes + println!("cargo:rerun-if-changed=skills"); + + let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); // safety: build script panics on failure + let out_path = out_dir.join("embedded_skills.json"); + + if !skills_dir.is_dir() { + fs::write(&out_path, "[]").unwrap(); // safety: build script + return; + } + + let mut skills: Vec = Vec::new(); + + let mut entries: Vec<_> = fs::read_dir(&skills_dir) + .unwrap() // safety: build script + .filter_map(|e| e.ok()) + .filter(|e| e.path().is_dir()) + .collect(); + entries.sort_by_key(|e| e.file_name()); + + for entry in entries { + let skill_md = entry.path().join("SKILL.md"); + if !skill_md.is_file() { + continue; + } + // Emit per-file watch + println!("cargo:rerun-if-changed={}", skill_md.display()); + + let name = entry.file_name().to_string_lossy().to_string(); + if let Ok(content) = fs::read_to_string(&skill_md) { + // Escape for JSON embedding + let name_json = serde_json::to_string(&name).unwrap(); // safety: build script + let content_json = serde_json::to_string(&content).unwrap(); // safety: build script + skills.push(format!( + r#"{{"name":{},"content":{}}}"#, + name_json, content_json + )); + } + } + + let catalog = format!("[{}]", skills.join(",")); + fs::write(&out_path, catalog).unwrap(); // safety: build script } /// Read all .json files from a directory and push their raw contents into `out`. diff --git a/channels-src/discord/Cargo.lock b/channels-src/discord/Cargo.lock index f25ce5511b5..f6e4a814278 100644 --- a/channels-src/discord/Cargo.lock +++ b/channels-src/discord/Cargo.lock @@ -20,162 +20,33 @@ version = "1.0.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" -[[package]] -name = "base64ct" -version = "1.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" - [[package]] name = "bitflags" version = "2.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" -[[package]] -name = "block-buffer" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" -dependencies = [ - "generic-array", -] - [[package]] name = "cfg-if" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" -[[package]] -name = "const-oid" -version = "0.9.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" - -[[package]] -name = "cpufeatures" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] - -[[package]] -name = "crypto-common" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" -dependencies = [ - "generic-array", - "typenum", -] - -[[package]] -name = "curve25519-dalek" -version = "4.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" -dependencies = [ - "cfg-if", - "cpufeatures", - "curve25519-dalek-derive", - "digest", - "fiat-crypto", - "rustc_version", - "subtle", - "zeroize", -] - -[[package]] -name = "curve25519-dalek-derive" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "der" -version = "0.7.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" -dependencies = [ - "const-oid", - "zeroize", -] - -[[package]] -name = "digest" -version = "0.10.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" -dependencies = [ - "block-buffer", - "crypto-common", -] - [[package]] name = "discord-channel" -version = "0.2.0" +version = "0.2.1" dependencies = [ - "ed25519-dalek", - "hex", "serde", "serde_json", "wit-bindgen", ] -[[package]] -name = "ed25519" -version = "2.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115531babc129696a58c64a4fef0a8bf9e9698629fb97e9e40767d235cfbcd53" -dependencies = [ - "pkcs8", - "signature", -] - -[[package]] -name = "ed25519-dalek" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e796c081cee67dc755e1a36a0a172b897fab85fc3f6bc48307991f64e4eca9" -dependencies = [ - "curve25519-dalek", - "ed25519", - "serde", - "sha2", - "subtle", - "zeroize", -] - [[package]] name = "equivalent" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" -[[package]] -name = "fiat-crypto" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" - -[[package]] -name = "generic-array" -version = "0.14.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" -dependencies = [ - "typenum", - "version_check", -] - [[package]] name = "hashbrown" version = "0.14.5" @@ -197,12 +68,6 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" -[[package]] -name = "hex" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" - [[package]] name = "id-arena" version = "2.3.0" @@ -223,9 +88,9 @@ dependencies = [ [[package]] name = "itoa" -version = "1.0.17" +version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "leb128" @@ -233,12 +98,6 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "884e2677b40cc8c339eaefcb701c32ef1fd2493d71118dc0ca4b6a736c93bd67" -[[package]] -name = "libc" -version = "0.2.182" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6800badb6cb2082ffd7b6a67e6125bb39f18782f793520caee8cb8846be06112" - [[package]] name = "log" version = "0.4.29" @@ -253,19 +112,9 @@ checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" [[package]] name = "once_cell" -version = "1.21.3" +version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" - -[[package]] -name = "pkcs8" -version = "0.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" -dependencies = [ - "der", - "spki", -] +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "prettyplease" @@ -288,22 +137,13 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.44" +version = "1.0.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" dependencies = [ "proc-macro2", ] -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - [[package]] name = "semver" version = "1.0.27" @@ -353,23 +193,6 @@ dependencies = [ "zmij", ] -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if", - "cpufeatures", - "digest", -] - -[[package]] -name = "signature" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" - [[package]] name = "smallvec" version = "1.15.1" @@ -385,22 +208,6 @@ dependencies = [ "smallvec", ] -[[package]] -name = "spki" -version = "0.7.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" -dependencies = [ - "base64ct", - "der", -] - -[[package]] -name = "subtle" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" - [[package]] name = "syn" version = "2.0.117" @@ -412,12 +219,6 @@ dependencies = [ "unicode-ident", ] -[[package]] -name = "typenum" -version = "1.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" - [[package]] name = "unicode-ident" version = "1.0.24" @@ -575,30 +376,24 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.39" +version = "0.8.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db6d35d663eadb6c932438e763b262fe1a70987f9ae936e60158176d710cae4a" +checksum = "efbb2a062be311f2ba113ce66f697a4dc589f85e78a4aea276200804cea0ed87" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.39" +version = "0.8.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4122cd3169e94605190e77839c9a40d40ed048d305bfdc146e7df40ab0f3e517" +checksum = "0e8bc7269b54418e7aeeef514aa68f8690b8c0489a06b0136e5f57c4c5ccab89" dependencies = [ "proc-macro2", "quote", "syn", ] -[[package]] -name = "zeroize" -version = "1.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" - [[package]] name = "zmij" version = "1.0.21" diff --git a/channels-src/discord/Cargo.toml b/channels-src/discord/Cargo.toml index a2892494a84..6388178c0a1 100644 --- a/channels-src/discord/Cargo.toml +++ b/channels-src/discord/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "discord-channel" -version = "0.2.0" +version = "0.2.1" edition = "2021" description = "Discord channel for IronClaw" license = "MIT OR Apache-2.0" @@ -10,8 +10,6 @@ publish = false serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" wit-bindgen = "0.36" -ed25519-dalek = { version = "2", default-features = false, features = ["alloc", "fast", "zeroize"] } -hex = "0.4" [lib] crate-type = ["cdylib"] diff --git a/channels-src/discord/README.md b/channels-src/discord/README.md index 333e7670db0..927d2fae9b4 100644 --- a/channels-src/discord/README.md +++ b/channels-src/discord/README.md @@ -86,6 +86,24 @@ If an internal error occurs (e.g., metadata serialization failure), the tool att Check the host logs for detailed error information. ## Advanced Usage +### Gateway Mode + +The Discord channel now defaults to Discord Gateway transport for inbound message intake. +The bundled identify payload requests intents `4609`, which expands to: + +- `GUILDS` (`1`) +- `GUILD_MESSAGES` (`512`) +- `DIRECT_MESSAGES` (`4096`) + +Gateway DMs now follow the same pairing policy as webhook DMs. Unpaired users receive a pairing +instruction reply in the DM channel before the message is allowed through to the agent. If you +want stricter access control than pairing, set `owner_id`; that lock still applies to both +webhook and Gateway traffic. + +Gateway presence simply reflects a successful authenticated Gateway connection and advertises +`online`. Pairing still controls whether DMs are allowed through to the agent, but it no longer +changes the visible Discord status. + ### Mention Polling The Discord channel can also poll configured channels for `@bot` mentions. @@ -110,6 +128,7 @@ Example channel config: - `owner_id`: when set, only that Discord user can interact with the bot. - `dm_policy`: `open` allows all DMs; `pairing` requires approval. - `allow_from`: allowlist entries for DM pairing checks (`*`, user id, or username). +- Gateway DMs respect `dm_policy` and pairing just like webhook DMs. ### Embeds diff --git a/channels-src/discord/discord.capabilities.json b/channels-src/discord/discord.capabilities.json index 9ff7a8905d6..00ee25858eb 100644 --- a/channels-src/discord/discord.capabilities.json +++ b/channels-src/discord/discord.capabilities.json @@ -1,5 +1,5 @@ { - "version": "0.2.0", + "version": "0.2.1", "wit_version": "0.3.0", "type": "channel", "name": "discord", @@ -22,7 +22,8 @@ "capabilities": { "http": { "allowlist": [ - { "host": "discord.com", "path_prefix": "/api/v10" } + { "host": "discord.com", "path_prefix": "/api/v10" }, + { "host": "gateway.discord.gg", "path_prefix": "/", "methods": ["GET"] } ], "credentials": { "discord_bot_token": { @@ -36,6 +37,20 @@ "requests_per_hour": 3600 } }, + "websocket": { + "url": "wss://gateway.discord.gg/?v=10&encoding=json", + "connect_on_start": true, + "identify_secret_name": "discord_bot_token", + "identify": { + "_intents_doc": "GUILDS(1) + GUILD_MESSAGES(512) + DIRECT_MESSAGES(4096)", + "intents": 4609, + "properties": { + "os": "linux", + "browser": "ironclaw", + "device": "ironclaw" + } + } + }, "secrets": { "allowed_names": ["discord_bot_token", "discord_*"] }, diff --git a/channels-src/discord/src/lib.rs b/channels-src/discord/src/lib.rs index cdb6c515077..e06736c7453 100644 --- a/channels-src/discord/src/lib.rs +++ b/channels-src/discord/src/lib.rs @@ -10,11 +10,11 @@ //! - Message event parsing (@mentions, DMs) //! - Thread support for conversations //! - Response posting via Discord Web API -//! - Automatic message truncation (> 2000 chars) +//! - Markdown attachment fallback for oversized replies //! //! # Security //! -//! - Signature validation is handled in-channel using Discord's Ed25519 headers +//! - Signature validation is handled by the host (webhook secrets) //! - Bot token is injected by host during HTTP requests //! - WASM never sees raw credentials @@ -23,17 +23,18 @@ wit_bindgen::generate!({ path: "../../wit/channel.wit", }); -use std::{cmp::Ordering, collections::HashMap}; - -use ed25519_dalek::{Signature, Verifier, VerifyingKey}; use serde::{Deserialize, Serialize}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{SystemTime, UNIX_EPOCH}; use exports::near::agent::channel::{ AgentResponse, ChannelConfig, Guest, HttpEndpointConfig, IncomingHttpRequest, - OutgoingHttpResponse, PollConfig, StatusUpdate, + OutgoingHttpResponse, PollConfig, StatusType, StatusUpdate, }; use near::agent::channel_host::{self, EmittedMessage}; +const DISCORD_API_BASE: &str = "https://discord.com/api/v10"; + /// Discord interaction wrapper. #[derive(Debug, Deserialize)] struct DiscordInteraction { @@ -108,146 +109,453 @@ struct DiscordMessage { author: DiscordUser, } -#[derive(Debug, Deserialize)] -struct DiscordChannelMessage { - id: String, - content: String, +/// Deserialize a String that may be null or missing (backward compat with old Option fields). +fn deserialize_nullable_string<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + Option::::deserialize(deserializer).map(|opt| opt.unwrap_or_default()) +} + +/// Metadata stored with emitted messages for response routing. +#[derive(Debug, Serialize, Deserialize)] +struct DiscordMessageMetadata { + /// Discord channel ID channel_id: String, - author: DiscordChannelAuthor, - #[serde(default)] - mentions: Vec, + + /// Interaction ID for followups + #[serde(default, deserialize_with = "deserialize_nullable_string")] + interaction_id: String, + + /// Interaction token for responding + #[serde(default, deserialize_with = "deserialize_nullable_string")] + token: String, + + /// Application ID + #[serde(default, deserialize_with = "deserialize_nullable_string")] + application_id: String, + + /// Source message ID when handling mention-poll events. #[serde(default)] - webhook_id: Option, + source_message_id: Option, + + /// Thread ID (for forum threads) + thread_id: Option, } -#[derive(Debug, Deserialize)] -struct DiscordChannelAuthor { - id: String, - username: String, - global_name: Option, - #[serde(default)] - bot: bool, +#[derive(Debug, PartialEq, Eq)] +enum DiscordResponseRoute { + InteractionWebhook(String), + ChannelMessage(String), } -#[derive(Debug, Clone, Serialize, Deserialize)] -struct DiscordRuntimeConfig { - #[serde(default = "default_require_signature_verification")] - require_signature_verification: bool, - #[serde(default)] - webhook_secret: Option, - #[serde(default)] - polling_enabled: bool, - #[serde(default = "default_poll_interval_ms")] - poll_interval_ms: u32, - #[serde(default)] - mention_channel_ids: Vec, - #[serde(default)] - owner_id: Option, - #[serde(default = "default_dm_policy")] - dm_policy: String, - #[serde(default)] - allow_from: Vec, +fn response_route_for_metadata(metadata: &DiscordMessageMetadata) -> DiscordResponseRoute { + if !metadata.application_id.is_empty() && !metadata.token.is_empty() { + DiscordResponseRoute::InteractionWebhook(format!( + "{DISCORD_API_BASE}/webhooks/{}/{}/messages/@original", + metadata.application_id, metadata.token + )) + } else { + DiscordResponseRoute::ChannelMessage(format!( + "{DISCORD_API_BASE}/channels/{}/messages", + metadata.channel_id + )) + } } -fn default_poll_interval_ms() -> u32 { - 30_000 +fn typing_request_url_for_update(update: &StatusUpdate) -> Option { + if update.status != StatusType::Thinking { + return None; + } + + let metadata: DiscordMessageMetadata = serde_json::from_str(&update.metadata_json).ok()?; + if metadata.channel_id.is_empty() { + return None; + } + + Some(format!( + "{DISCORD_API_BASE}/channels/{}/typing", + metadata.channel_id + )) } -fn default_require_signature_verification() -> bool { - true +const DISCORD_MESSAGE_CHAR_LIMIT: usize = 2000; +const DISCORD_MULTIPART_BOUNDARY: &str = "ironclaw-discord-response-boundary"; +const DISCORD_ATTACHMENT_FILENAME: &str = "response.md"; +const DISCORD_ATTACHMENT_NOTICE: &str = "Response too long for Discord; attached as response.md."; +static MULTIPART_BOUNDARY_COUNTER: AtomicU64 = AtomicU64::new(0); + +#[derive(Debug, PartialEq, Eq)] +struct DiscordHttpRequest { + headers_json: String, + body: Vec, +} + +#[derive(Debug, PartialEq, Eq)] +enum DiscordReplyPlan { + Inline(DiscordHttpRequest), + Attachment { + upload: DiscordHttpRequest, + fallback: DiscordHttpRequest, + }, +} + +fn embeds_from_metadata_json(metadata_json: &str) -> Option { + serde_json::from_str::(metadata_json) + .ok()? + .get("embeds") + .cloned() } -fn default_dm_policy() -> String { - "pairing".to_string() +fn build_discord_json_request( + content: &str, + embeds: Option<&serde_json::Value>, +) -> Result { + let mut payload = serde_json::json!({ + "content": content, + }); + + if let Some(embeds) = embeds { + payload["embeds"] = embeds.clone(); + } + + Ok(DiscordHttpRequest { + headers_json: serde_json::json!({ + "Content-Type": "application/json" + }) + .to_string(), + body: serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize: {}", e))?, + }) +} + +fn build_discord_attachment_request( + content: &str, + embeds: Option<&serde_json::Value>, +) -> Result { + let boundary = next_multipart_boundary(); + let mut payload = serde_json::json!({ + "content": DISCORD_ATTACHMENT_NOTICE, + }); + + if let Some(embeds) = embeds { + payload["embeds"] = embeds.clone(); + } + + let payload_json = + serde_json::to_string(&payload).map_err(|e| format!("Failed to serialize: {}", e))?; + + let mut body = Vec::new(); + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"payload_json\"\r\nContent-Type: application/json\r\n\r\n{payload_json}\r\n", + boundary = boundary, + ) + .as_bytes(), + ); + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"files[0]\"; filename=\"{filename}\"\r\nContent-Type: text/markdown\r\n\r\n", + boundary = boundary, + filename = DISCORD_ATTACHMENT_FILENAME, + ) + .as_bytes(), + ); + body.extend_from_slice(content.as_bytes()); + body.extend_from_slice(format!("\r\n--{}--\r\n", boundary).as_bytes()); + + Ok(DiscordHttpRequest { + headers_json: serde_json::json!({ + "Content-Type": format!( + "multipart/form-data; boundary={}", + boundary + ) + }) + .to_string(), + body, + }) +} + +fn next_multipart_boundary() -> String { + let counter = MULTIPART_BOUNDARY_COUNTER.fetch_add(1, Ordering::Relaxed); + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|duration| duration.as_nanos()) + .unwrap_or(0); + format!("{}-{:x}-{:x}", DISCORD_MULTIPART_BOUNDARY, nanos, counter) +} + +fn build_discord_reply_plan(response: &AgentResponse) -> Result { + let embeds = embeds_from_metadata_json(&response.metadata_json); + + if response.content.chars().count() <= DISCORD_MESSAGE_CHAR_LIMIT { + return build_discord_json_request(&response.content, embeds.as_ref()) + .map(DiscordReplyPlan::Inline); + } + + Ok(DiscordReplyPlan::Attachment { + upload: build_discord_attachment_request(&response.content, embeds.as_ref())?, + fallback: build_discord_json_request( + &truncate_message(&response.content), + embeds.as_ref(), + )?, + }) } -fn default_runtime_config() -> DiscordRuntimeConfig { - DiscordRuntimeConfig { - require_signature_verification: default_require_signature_verification(), - webhook_secret: None, - polling_enabled: false, - poll_interval_ms: default_poll_interval_ms(), - mention_channel_ids: Vec::new(), - owner_id: None, - dm_policy: default_dm_policy(), - allow_from: Vec::new(), +fn send_discord_request( + method: &str, + url: &str, + request: &DiscordHttpRequest, +) -> Result<(), String> { + match channel_host::http_request( + method, + url, + &request.headers_json, + Some(&request.body), + None, + ) { + Ok(http_response) => { + if http_response.status >= 200 && http_response.status < 300 { + channel_host::log(channel_host::LogLevel::Debug, "Posted followup to Discord"); + Ok(()) + } else { + let body_str = String::from_utf8_lossy(&http_response.body); + Err(format!( + "Discord API error: {} - {}", + http_response.status, body_str + )) + } + } + Err(e) => Err(format!("HTTP request failed: {}", e)), } } /// Workspace path for persisting owner_id across WASM callbacks. const OWNER_ID_PATH: &str = "state/owner_id"; +/// Workspace path for persisting polling_enabled flag. +const POLLING_ENABLED_PATH: &str = "state/polling_enabled"; +/// Workspace path for persisting mention channel IDs (JSON array). +const MENTION_CHANNEL_IDS_PATH: &str = "state/mention_channel_ids"; /// Workspace path for persisting dm_policy across WASM callbacks. const DM_POLICY_PATH: &str = "state/dm_policy"; /// Workspace path for persisting allow_from (JSON array) across WASM callbacks. const ALLOW_FROM_PATH: &str = "state/allow_from"; +/// Workspace path for the current gateway text-frame batch prepared by the host runtime. +const GATEWAY_EVENT_QUEUE_PATH: &str = "state/gateway_event_queue_processing"; +/// Workspace path for persisting the bot user id learned from READY dispatches. +const BOT_USER_ID_PATH: &str = "state/bot_user_id"; /// Channel name for pairing store (used by pairing host APIs). const CHANNEL_NAME: &str = "discord"; -/// Metadata stored with emitted messages for response routing. -#[derive(Debug, Serialize, Deserialize)] -struct DiscordMessageMetadata { - /// Discord channel ID - channel_id: String, +#[derive(Debug, Deserialize)] +struct DiscordGatewayEvent { + op: u64, + #[serde(default)] + t: Option, + #[serde(default)] + d: serde_json::Value, +} - /// Interaction ID for followups +#[derive(Debug, Deserialize)] +struct DiscordGatewayReady { + user: DiscordGatewayAuthor, +} + +#[derive(Debug, Deserialize, Clone)] +struct DiscordGatewayAuthor { + id: String, + username: String, + global_name: Option, #[serde(default)] - interaction_id: Option, + bot: bool, +} - /// Interaction token for responding +#[derive(Debug, Deserialize)] +struct DiscordGatewayMessageCreate { + channel_id: String, #[serde(default)] - token: Option, + guild_id: Option, + content: String, + author: DiscordGatewayAuthor, +} - /// Application ID +/// A message returned by the Discord REST channel-messages endpoint. +#[derive(Debug, Deserialize)] +struct DiscordChannelMessage { + id: String, + content: String, + channel_id: String, + author: DiscordChannelAuthor, #[serde(default)] - application_id: Option, + mentions: Vec, + #[serde(default)] + webhook_id: Option, +} - /// Source message ID when handling mention-poll events. +/// Author sub-object for REST channel messages. +#[derive(Debug, Deserialize)] +struct DiscordChannelAuthor { + id: String, + username: String, + global_name: Option, #[serde(default)] - source_message_id: Option, + bot: bool, +} - /// Thread ID (for forum threads) - thread_id: Option, +#[derive(Debug, PartialEq, Eq)] +struct ParsedGatewayMessage { + user_id: String, + user_name: String, + channel_id: String, + content: String, + is_dm: bool, } -struct DiscordChannel; +#[derive(Debug, Default, PartialEq, Eq)] +struct GatewayPollResult { + bot_user_id: Option, + messages: Vec, +} -impl Guest for DiscordChannel { - fn on_start(config_json: String) -> Result { - channel_host::log(channel_host::LogLevel::Info, "Discord channel starting"); +fn parse_gateway_event_queue( + queue_json: &str, + known_bot_user_id: Option<&str>, +) -> GatewayPollResult { + let frames: Vec = match serde_json::from_str(queue_json) { + Ok(v) => v, + Err(e) => { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Failed to deserialize gateway event queue: {}", e), + ); + return GatewayPollResult::default(); + } + }; + let mut result = GatewayPollResult::default(); + let mut bot_user_id = known_bot_user_id.map(ToOwned::to_owned); - let config = - serde_json::from_str::(&config_json).unwrap_or_else(|e| { - channel_host::log( - channel_host::LogLevel::Warn, - &format!("Invalid config JSON, using defaults: {}", e), - ); - default_runtime_config() - }); + for frame in frames { + let event: DiscordGatewayEvent = match serde_json::from_str(&frame) { + Ok(value) => value, + Err(_) => continue, + }; - if let Ok(serialized) = serde_json::to_string(&config) { - let _ = channel_host::workspace_write("config.json", &serialized); + if event.op != 0 { + continue; } - if config.require_signature_verification - && config - .webhook_secret - .as_deref() - .map(str::trim) - .filter(|s| !s.is_empty()) - .is_none() - { - channel_host::log( - channel_host::LogLevel::Error, - "Discord channel misconfigured: require_signature_verification=true but webhook_secret is empty", - ); - } else if !config.require_signature_verification { - channel_host::log( - channel_host::LogLevel::Warn, - "Discord signature verification is disabled; webhook endpoint is unprotected", - ); + match event.t.as_deref() { + Some("READY") => { + if let Ok(ready) = serde_json::from_value::(event.d) { + if !ready.user.id.is_empty() { + bot_user_id = Some(ready.user.id); + } + } + } + Some("MESSAGE_CREATE") => { + let message = match serde_json::from_value::(event.d) { + Ok(value) => value, + Err(_) => continue, + }; + + let active_bot_user_id = bot_user_id.as_deref().or(known_bot_user_id); + if message.author.bot + || active_bot_user_id.is_some_and(|bot_id| message.author.id == bot_id) + { + continue; + } + + let is_dm = message.guild_id.is_none(); + let content = + match gateway_content_for_agent(&message.content, active_bot_user_id, is_dm) { + Some(value) => value, + None => continue, + }; + + result.messages.push(ParsedGatewayMessage { + user_id: message.author.id, + user_name: message + .author + .global_name + .unwrap_or(message.author.username), + channel_id: message.channel_id, + content, + is_dm, + }); + } + _ => {} + } + } + + result.bot_user_id = bot_user_id; + result +} + +fn gateway_content_for_agent( + content: &str, + bot_user_id: Option<&str>, + is_dm: bool, +) -> Option { + let trimmed = content.trim(); + if trimmed.is_empty() { + return None; + } + + if is_dm { + return Some(trimmed.to_string()); + } + + let bot_user_id = bot_user_id?; + for mention in [ + format!("<@{}>", bot_user_id), + format!("<@!{}>", bot_user_id), + ] { + if let Some(stripped) = trimmed.strip_prefix(&mention) { + let cleaned = stripped.trim(); + return if cleaned.is_empty() { + None + } else { + Some(cleaned.to_string()) + }; } + } + + None +} + +fn default_poll_interval_ms() -> u32 { + 30_000 +} + +/// Channel configuration from capabilities file. +#[derive(Debug, Deserialize)] +struct DiscordConfig { + #[serde(default)] + #[allow(dead_code)] + require_signature_verification: bool, + #[serde(default)] + owner_id: Option, + #[serde(default)] + dm_policy: Option, + #[serde(default)] + allow_from: Option>, + #[serde(default)] + polling_enabled: bool, + #[serde(default = "default_poll_interval_ms")] + poll_interval_ms: u32, + #[serde(default)] + mention_channel_ids: Vec, +} + +struct DiscordChannel; + +impl Guest for DiscordChannel { + fn on_start(config_json: String) -> Result { + let config: DiscordConfig = serde_json::from_str(&config_json) + .map_err(|e| format!("Failed to parse config: {}", e))?; + + channel_host::log(channel_host::LogLevel::Info, "Discord channel starting"); - // Persist owner_id so subsequent callbacks can read it. + // Persist owner_id so subsequent callbacks can read it if let Some(ref owner_id) = config.owner_id { let _ = channel_host::workspace_write(OWNER_ID_PATH, owner_id); channel_host::log( @@ -258,18 +566,29 @@ impl Guest for DiscordChannel { let _ = channel_host::workspace_write(OWNER_ID_PATH, ""); } - // Persist dm_policy and allow_from for DM pairing. - let _ = channel_host::workspace_write(DM_POLICY_PATH, &config.dm_policy); - let allow_from_json = - serde_json::to_string(&config.allow_from).unwrap_or_else(|_| "[]".to_string()); + // Persist dm_policy and allow_from for DM pairing + let dm_policy = config.dm_policy.as_deref().unwrap_or("pairing"); + let _ = channel_host::workspace_write(DM_POLICY_PATH, dm_policy); + + let allow_from_json = serde_json::to_string(&config.allow_from.unwrap_or_default()) + .unwrap_or_else(|_| "[]".to_string()); let _ = channel_host::workspace_write(ALLOW_FROM_PATH, &allow_from_json); + // Persist polling config + let _ = channel_host::workspace_write( + POLLING_ENABLED_PATH, + &config.polling_enabled.to_string(), + ); + let mention_ids_json = + serde_json::to_string(&config.mention_channel_ids).unwrap_or_else(|_| "[]".to_string()); + let _ = channel_host::workspace_write(MENTION_CHANNEL_IDS_PATH, &mention_ids_json); + Ok(ChannelConfig { display_name: "Discord".to_string(), http_endpoints: vec![HttpEndpointConfig { path: "/webhook/discord".to_string(), methods: vec!["POST".to_string()], - require_secret: false, + require_secret: true, }], poll: if config.polling_enabled { Some(PollConfig { @@ -283,45 +602,6 @@ impl Guest for DiscordChannel { } fn on_http_request(req: IncomingHttpRequest) -> OutgoingHttpResponse { - let config = load_runtime_config(); - let headers: HashMap = - serde_json::from_str(&req.headers_json).unwrap_or_default(); - if config.require_signature_verification { - if config - .webhook_secret - .as_deref() - .map(str::trim) - .filter(|s| !s.is_empty()) - .is_none() - { - channel_host::log( - channel_host::LogLevel::Error, - "Discord channel misconfigured: webhook_secret not set while verification is required", - ); - return json_response( - 500, - serde_json::json!({"error": "Channel misconfigured: webhook_secret not set"}), - ); - } - - if !verify_discord_request_signature( - headers, - &req.body, - config.webhook_secret.as_deref(), - ) { - channel_host::log( - channel_host::LogLevel::Warn, - "Discord signature verification failed", - ); - return json_response(401, serde_json::json!({"error": "Invalid signature"})); - } - } else { - channel_host::log( - channel_host::LogLevel::Warn, - "Discord signature verification is disabled; accepting unverified webhook request", - ); - } - let body_str = match std::str::from_utf8(&req.body) { Ok(s) => s, Err(_) => { @@ -350,16 +630,9 @@ impl Guest for DiscordChannel { // Application Command (slash command) 2 => { if handle_slash_command(&interaction) { - json_response( - 200, - serde_json::json!({ - "type": 5, - "data": { - "content": "🤔 Thinking..." - } - }), - ) + json_response(200, serde_json::json!({"type": 5})) } else { + // Permission denied — ephemeral response json_response( 200, serde_json::json!({ @@ -395,528 +668,200 @@ impl Guest for DiscordChannel { } fn on_poll() { - poll_for_mentions(); - } - - fn on_respond(response: AgentResponse) -> Result<(), String> { - let metadata: DiscordMessageMetadata = serde_json::from_str(&response.metadata_json) - .map_err(|e| format!("Failed to parse metadata: {}", e))?; - - // Truncate content to 2000 characters to comply with Discord limits - let content = truncate_message(&response.content); - - let mut payload = serde_json::json!({ "content": content }); - - // Check for embeds in metadata - if let Ok(meta_json) = serde_json::from_str::(&response.metadata_json) { - if let Some(embeds) = meta_json.get("embeds") { - payload["embeds"] = embeds.clone(); - } - } - - let payload_bytes = - serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize: {}", e))?; - - let headers = serde_json::json!({ - "Content-Type": "application/json" - }); - - let (method, url) = if let (Some(application_id), Some(token)) = - (metadata.application_id.as_ref(), metadata.token.as_ref()) - { - ( - "PATCH", - format!( - "https://discord.com/api/v10/webhooks/{}/{}/messages/@original", - application_id, token - ), - ) - } else if let Some(source_message_id) = metadata.source_message_id.as_ref() { - payload["message_reference"] = serde_json::json!({ - "message_id": source_message_id - }); - payload["allowed_mentions"] = serde_json::json!({ - "replied_user": true - }); - let mention_payload = serde_json::to_vec(&payload) - .map_err(|e| format!("Failed to serialize mention payload: {}", e))?; - let mention_url = format!( - "https://discord.com/api/v10/channels/{}/messages", - metadata.channel_id - ); - let result = channel_host::http_request( - "POST", - &mention_url, - &discord_auth_headers_json(true), - Some(&mention_payload), - None, - ); - return map_discord_response(result); - } else { - return Err("Unsupported Discord response metadata".to_string()); - }; - - let result = channel_host::http_request( - method, - &url, - &headers.to_string(), - Some(&payload_bytes), - None, - ); - - map_discord_response(result) - } - - fn on_status(_update: StatusUpdate) {} - - fn on_broadcast(_user_id: String, _response: AgentResponse) -> Result<(), String> { - Err("broadcast not yet implemented for Discord channel".to_string()) - } - - fn on_shutdown() { - channel_host::log( - channel_host::LogLevel::Info, - "Discord channel shutting down", - ); - } -} - -fn map_discord_response( - result: Result, -) -> Result<(), String> { - match result { - Ok(http_response) => { - if http_response.status >= 200 && http_response.status < 300 { - channel_host::log(channel_host::LogLevel::Debug, "Posted response to Discord"); - Ok(()) - } else { - let body_str = String::from_utf8_lossy(&http_response.body); - Err(format!( - "Discord API error: {} - {}", - http_response.status, body_str - )) - } - } - Err(e) => Err(format!("HTTP request failed: {}", e)), - } -} - -fn load_runtime_config() -> DiscordRuntimeConfig { - channel_host::workspace_read("config.json") - .and_then(|raw| serde_json::from_str::(&raw).ok()) - .unwrap_or_else(default_runtime_config) -} - -fn poll_for_mentions() { - let config = load_runtime_config(); - if !config.polling_enabled || config.mention_channel_ids.is_empty() { - return; - } - - let bot_id = match get_or_fetch_bot_id() { - Some(id) => id, - None => { - channel_host::log( - channel_host::LogLevel::Warn, - "Skipping mention polling: failed to resolve bot user id", - ); - return; - } - }; - - for channel_id in &config.mention_channel_ids { - poll_channel_mentions(channel_id, &bot_id); - } -} - -fn get_or_fetch_bot_id() -> Option { - if let Some(id) = channel_host::workspace_read("bot_user_id.txt") { - let trimmed = id.trim(); - if !trimmed.is_empty() { - return Some(trimmed.to_string()); - } - } - - let response = channel_host::http_request( - "GET", - "https://discord.com/api/v10/users/@me", - &discord_auth_headers_json(false), - None, - Some(10_000), - ) - .ok()?; - - if !(200..300).contains(&response.status) { - return None; - } - - let value: serde_json::Value = serde_json::from_slice(&response.body).ok()?; - let id = value.get("id")?.as_str()?.to_string(); - let _ = channel_host::workspace_write("bot_user_id.txt", &id); - Some(id) -} - -fn poll_channel_mentions(channel_id: &str, bot_id: &str) { - let cursor_path = format!("cursor_{}.txt", channel_id); - let last_seen = channel_host::workspace_read(&cursor_path).map(|s| s.trim().to_string()); - - // On first run for a channel, initialize the cursor to "latest seen" and - // skip back-processing historical messages. - if last_seen.is_none() { - if let Some(latest) = fetch_latest_message_id(channel_id) { - let _ = channel_host::workspace_write(&cursor_path, &latest); - } - return; - } - - let Some(mut messages) = - fetch_messages_after_cursor(channel_id, last_seen.as_deref().unwrap_or("")) - else { - return; - }; - if messages.is_empty() { - return; - } - - messages.sort_by(|a, b| compare_message_ids(&a.id, &b.id)); - let mut max_seen = last_seen.clone(); - let mut recent_ids = load_recent_processed_ids(channel_id); - let mut dedup_updated = false; - - for msg in messages { - if is_new_message(max_seen.as_deref(), &msg.id) { - max_seen = Some(msg.id.clone()); - } + // 1. Process Gateway event queue + let queue_json = channel_host::workspace_read(GATEWAY_EVENT_QUEUE_PATH).unwrap_or_default(); + let has_gateway_events = !queue_json.trim().is_empty() && queue_json.trim() != "[]"; - if msg.webhook_id.is_some() || msg.author.bot || msg.author.id == bot_id { - continue; - } - - if !message_mentions_bot(&msg, bot_id) { - continue; - } - - if recent_ids.iter().any(|id| id == &msg.id) { - continue; - } - - let user_name = msg - .author - .global_name - .as_ref() - .filter(|s| !s.is_empty()) - .unwrap_or(&msg.author.username) - .clone(); - if !check_sender_permission(&msg.author.id, Some(&user_name), false, None) { - continue; - } - - let content = strip_bot_mention(&msg.content, bot_id); - let metadata = DiscordMessageMetadata { - channel_id: msg.channel_id.clone(), - interaction_id: None, - token: None, - application_id: None, - source_message_id: Some(msg.id.clone()), - thread_id: None, - }; + if has_gateway_events { + let known_bot_user_id = channel_host::workspace_read(BOT_USER_ID_PATH); + let parsed = parse_gateway_event_queue(&queue_json, known_bot_user_id.as_deref()); - let metadata_json = match serde_json::to_string(&metadata) { - Ok(v) => v, - Err(e) => { + if let Err(error) = channel_host::workspace_write(GATEWAY_EVENT_QUEUE_PATH, "[]") { channel_host::log( channel_host::LogLevel::Warn, - &format!("Failed to serialize mention metadata: {}", e), + &format!("Failed to clear Discord gateway queue: {}", error), ); - continue; } - }; - - channel_host::emit_message(&EmittedMessage { - user_id: msg.author.id.clone(), - user_name: Some(user_name.clone()), - content: if content.is_empty() { - "mention".to_string() - } else { - content - }, - thread_id: None, - metadata_json, - attachments: vec![], - }); - - remember_processed_id(&mut recent_ids, &msg.id); - dedup_updated = true; - } - if let Some(cursor) = max_seen { - let _ = channel_host::workspace_write(&cursor_path, &cursor); - } - if dedup_updated { - let _ = save_recent_processed_ids(channel_id, &recent_ids); - } -} - -fn fetch_latest_message_id(channel_id: &str) -> Option { - let url = format!( - "https://discord.com/api/v10/channels/{}/messages?limit=1", - channel_id - ); - let response = channel_host::http_request( - "GET", - &url, - &discord_auth_headers_json(false), - None, - Some(10_000), - ) - .ok()?; - if !(200..300).contains(&response.status) { - let body = String::from_utf8_lossy(&response.body); - channel_host::log( - channel_host::LogLevel::Warn, - &format!( - "Discord initial poll failed for channel {}: status={} body={}", - channel_id, response.status, body - ), - ); - return None; - } - let messages: Vec = serde_json::from_slice(&response.body).ok()?; - messages.first().map(|m| m.id.clone()) -} - -fn fetch_messages_after_cursor( - channel_id: &str, - last_seen: &str, -) -> Option> { - const PAGE_LIMIT: usize = 100; - const MAX_PAGES: usize = 50; - - let mut all_messages = Vec::new(); - let mut after = last_seen.to_string(); - - for page in 0..MAX_PAGES { - let url = format!( - "https://discord.com/api/v10/channels/{}/messages?limit={}&after={}", - channel_id, PAGE_LIMIT, after - ); - let response = match channel_host::http_request( - "GET", - &url, - &discord_auth_headers_json(false), - None, - Some(10_000), - ) { - Ok(r) => r, - Err(e) => { - channel_host::log( - channel_host::LogLevel::Warn, - &format!( - "Discord poll request failed for channel {}: {}", - channel_id, e - ), - ); - return None; - } - }; - - if !(200..300).contains(&response.status) { - let body = String::from_utf8_lossy(&response.body); - channel_host::log( - channel_host::LogLevel::Warn, - &format!( - "Discord poll failed for channel {}: status={} body={}", - channel_id, response.status, body - ), - ); - return None; - } - - let messages: Vec = match serde_json::from_slice(&response.body) { - Ok(v) => v, - Err(e) => { - channel_host::log( - channel_host::LogLevel::Warn, - &format!("Failed to parse polled Discord messages: {}", e), - ); - return None; + if let Some(bot_user_id) = parsed.bot_user_id.as_deref() { + if let Err(error) = channel_host::workspace_write(BOT_USER_ID_PATH, bot_user_id) { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Failed to persist Discord bot user id: {}", error), + ); + } } - }; - let page_len = messages.len(); - if messages.is_empty() { - break; - } - let page_max_id = messages - .iter() - .map(|m| m.id.as_str()) - .max_by(|a, b| compare_message_ids(a, b)) - .map(str::to_string); - - all_messages.extend(messages.into_iter()); - - if page_len < PAGE_LIMIT { - break; - } + for message in parsed.messages { + if !check_sender_permission( + &message.user_id, + Some(&message.user_name), + message.is_dm, + PermissionSource::Gateway, + Some(&PairingReplyCtx { + channel_id: message.channel_id.clone(), + application_id: String::new(), + token: String::new(), + }), + ) { + continue; + } - if let Some(max_id) = page_max_id { - if max_id == after { - break; + let metadata = DiscordMessageMetadata { + channel_id: message.channel_id, + interaction_id: String::new(), + token: String::new(), + application_id: String::new(), + source_message_id: None, + thread_id: None, + }; + + let metadata_json = match serde_json::to_string(&metadata) { + Ok(json) => json, + Err(error) => { + channel_host::log( + channel_host::LogLevel::Error, + &format!("Failed to serialize gateway metadata: {}", error), + ); + continue; + } + }; + + channel_host::emit_message(&EmittedMessage { + user_id: message.user_id, + user_name: Some(message.user_name), + content: message.content, + thread_id: None, + metadata_json, + attachments: vec![], + }); } - after = max_id; - } else { - break; } - if page + 1 == MAX_PAGES { - channel_host::log( - channel_host::LogLevel::Warn, - &format!( - "Discord poll pagination limit reached for channel {}; processing partial batch", - channel_id - ), - ); - } - } - - Some(all_messages) -} - -fn compare_message_ids(a: &str, b: &str) -> Ordering { - match (a.parse::(), b.parse::()) { - (Ok(left), Ok(right)) => left.cmp(&right), - _ => a.cmp(b), + // 2. Run mention polling if configured + poll_for_mentions(); } -} -fn dedup_ids_path(channel_id: &str) -> String { - format!("dedup_{}.json", channel_id) -} - -fn load_recent_processed_ids(channel_id: &str) -> Vec { - let path = dedup_ids_path(channel_id); - channel_host::workspace_read(&path) - .and_then(|raw| serde_json::from_str::>(&raw).ok()) - .unwrap_or_default() -} + fn on_respond(response: AgentResponse) -> Result<(), String> { + let metadata: DiscordMessageMetadata = serde_json::from_str(&response.metadata_json) + .map_err(|e| format!("Failed to parse metadata: {}", e))?; -fn save_recent_processed_ids(channel_id: &str, ids: &[String]) -> Result<(), String> { - let path = dedup_ids_path(channel_id); - let raw = - serde_json::to_string(ids).map_err(|e| format!("Failed to serialize dedup ids: {}", e))?; - channel_host::workspace_write(&path, &raw) -} + // Mention-poll replies: include message_reference so Discord renders as a reply + if let Some(ref source_id) = metadata.source_message_id { + if let DiscordResponseRoute::ChannelMessage(ref url) = + response_route_for_metadata(&metadata) + { + let embeds = embeds_from_metadata_json(&response.metadata_json); + let content = if response.content.chars().count() > DISCORD_MESSAGE_CHAR_LIMIT { + truncate_message(&response.content) + } else { + response.content.clone() + }; + + let mut payload = serde_json::json!({ + "content": content, + "message_reference": { + "message_id": source_id + }, + "allowed_mentions": { + "replied_user": true + } + }); -fn remember_processed_id(ids: &mut Vec, message_id: &str) { - const MAX_RECENT_IDS: usize = 200; - if ids.iter().any(|id| id == message_id) { - return; - } - ids.push(message_id.to_string()); - if ids.len() > MAX_RECENT_IDS { - let drop_count = ids.len() - MAX_RECENT_IDS; - ids.drain(0..drop_count); - } -} + if let Some(ref e) = embeds { + payload["embeds"] = e.clone(); + } -fn is_new_message(last_seen: Option<&str>, current: &str) -> bool { - match last_seen { - None => true, - Some(prev) => { - let prev_num = prev.parse::().ok(); - let cur_num = current.parse::().ok(); - match (prev_num, cur_num) { - (Some(p), Some(c)) => c > p, - _ => current > prev, + let headers = discord_auth_headers_json(true); + let body = serde_json::to_vec(&payload) + .map_err(|e| format!("Failed to serialize: {}", e))?; + + return send_discord_request( + "POST", + url, + &DiscordHttpRequest { + headers_json: headers, + body, + }, + ); } } - } -} -fn message_mentions_bot(msg: &DiscordChannelMessage, bot_id: &str) -> bool { - msg.mentions.iter().any(|u| u.id == bot_id) - || msg.content.contains(&format!("<@{}>", bot_id)) - || msg.content.contains(&format!("<@!{}>", bot_id)) -} + let route = response_route_for_metadata(&metadata); + let plan = build_discord_reply_plan(&response)?; -fn strip_bot_mention(content: &str, bot_id: &str) -> String { - content - .replace(&format!("<@{}>", bot_id), "") - .replace(&format!("<@!{}>", bot_id), "") - .trim() - .to_string() -} + let (method, url) = match &route { + DiscordResponseRoute::InteractionWebhook(url) => ("PATCH", url.as_str()), + DiscordResponseRoute::ChannelMessage(url) => ("POST", url.as_str()), + }; -fn discord_auth_headers_json(include_content_type: bool) -> String { - if include_content_type { - serde_json::json!({ - "Content-Type": "application/json", - "Authorization": "Bot {DISCORD_BOT_TOKEN}" - }) - .to_string() - } else { - serde_json::json!({ - "Authorization": "Bot {DISCORD_BOT_TOKEN}" - }) - .to_string() + match plan { + DiscordReplyPlan::Inline(request) => send_discord_request(method, url, &request), + DiscordReplyPlan::Attachment { upload, fallback } => { + match send_discord_request(method, url, &upload) { + Ok(()) => Ok(()), + Err(upload_error) => { + channel_host::log( + channel_host::LogLevel::Warn, + &format!( + "Discord attachment upload failed, falling back to truncated text: {}", + upload_error + ), + ); + send_discord_request(method, url, &fallback).map_err(|fallback_error| { + format!( + "Discord attachment upload failed: {}; fallback also failed: {}", + upload_error, fallback_error + ) + }) + } + } + } + } } -} -fn verify_discord_request_signature( - headers: HashMap, - body: &[u8], - public_key_hex: Option<&str>, -) -> bool { - let Some(public_key_hex) = public_key_hex.map(str::trim).filter(|s| !s.is_empty()) else { - return false; - }; - let Some(signature_hex) = header_case_insensitive(&headers, "x-signature-ed25519") else { - return false; - }; - let Some(timestamp) = header_case_insensitive(&headers, "x-signature-timestamp") else { - return false; - }; - - let public_key_bytes = match hex::decode(public_key_hex) { - Ok(v) => v, - Err(_) => return false, - }; - let public_key_arr: [u8; 32] = match public_key_bytes.try_into() { - Ok(v) => v, - Err(_) => return false, - }; - let verifying_key = match VerifyingKey::from_bytes(&public_key_arr) { - Ok(v) => v, - Err(_) => return false, - }; - - let sig_bytes = match hex::decode(signature_hex.trim()) { - Ok(v) => v, - Err(_) => return false, - }; - let sig_arr: [u8; 64] = match sig_bytes.try_into() { - Ok(v) => v, - Err(_) => return false, - }; - let signature = Signature::from_bytes(&sig_arr); + fn on_status(update: StatusUpdate) { + let Some(url) = typing_request_url_for_update(&update) else { + return; + }; - let mut signed_message = Vec::with_capacity(timestamp.len() + body.len()); - signed_message.extend_from_slice(timestamp.as_bytes()); - signed_message.extend_from_slice(body); + let headers = serde_json::json!({ + "Content-Type": "application/json" + }); - verifying_key.verify(&signed_message, &signature).is_ok() -} + match channel_host::http_request("POST", &url, &headers.to_string(), None, None) { + Ok(response) if (200..300).contains(&response.status) => {} + Ok(response) => { + channel_host::log( + channel_host::LogLevel::Warn, + &format!( + "Discord typing indicator failed with status {}", + response.status + ), + ); + } + Err(error) => { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Discord typing indicator request failed: {}", error), + ); + } + } + } + + fn on_broadcast(user_id: String, response: AgentResponse) -> Result<(), String> { + broadcast_dm(&user_id, &response.content) + } -fn header_case_insensitive<'a>( - headers: &'a HashMap, - name: &str, -) -> Option<&'a str> { - headers - .iter() - .find(|(k, _)| k.eq_ignore_ascii_case(name)) - .map(|(_, v)| v.as_str()) + fn on_shutdown() { + channel_host::log( + channel_host::LogLevel::Info, + "Discord channel shutting down", + ); + } } +/// Returns true if the message was emitted, false if permission denied. fn handle_slash_command(interaction: &DiscordInteraction) -> bool { let user = interaction .member @@ -934,13 +879,17 @@ fn handle_slash_command(interaction: &DiscordInteraction) -> bool { }) .unwrap_or_default(); - // DM if no guild member context (only direct user field set). + // DM if no guild member context (only direct user field set) let is_dm = interaction.member.is_none(); + + // Permission check if !check_sender_permission( &user_id, Some(&user_name), is_dm, + PermissionSource::Webhook, Some(&PairingReplyCtx { + channel_id: interaction.channel_id.clone().unwrap_or_default(), application_id: interaction.application_id.clone(), token: interaction.token.clone(), }), @@ -970,9 +919,9 @@ fn handle_slash_command(interaction: &DiscordInteraction) -> bool { let metadata = DiscordMessageMetadata { channel_id: channel_id.clone(), - interaction_id: Some(interaction.id.clone()), - token: Some(interaction.token.clone()), - application_id: Some(interaction.application_id.clone()), + interaction_id: interaction.id.clone(), + token: interaction.token.clone(), + application_id: interaction.application_id.clone(), source_message_id: None, thread_id: None, }; @@ -984,14 +933,13 @@ fn handle_slash_command(interaction: &DiscordInteraction) -> bool { channel_host::LogLevel::Error, &format!("Failed to serialize metadata: {}", e), ); - // Attempt to notify user of internal error let url = format!( - "https://discord.com/api/v10/webhooks/{}/{}", + "{DISCORD_API_BASE}/webhooks/{}/{}", interaction.application_id, interaction.token ); let payload = serde_json::json!({ "content": "❌ Internal Error: Failed to process command metadata.", - "flags": 64 // Ephemeral + "flags": 64 }); let _ = channel_host::http_request( "POST", @@ -1000,7 +948,7 @@ fn handle_slash_command(interaction: &DiscordInteraction) -> bool { Some(&serde_json::to_vec(&payload).unwrap_or_default()), None, ); - return true; + return true; // Error, but not a permission denial } }; @@ -1016,7 +964,6 @@ fn handle_slash_command(interaction: &DiscordInteraction) -> bool { } fn handle_message_component(interaction: &DiscordInteraction, message: &DiscordMessage) { - // Check member first (for server contexts), then user (for DMs) let user = interaction .member .as_ref() @@ -1034,7 +981,13 @@ fn handle_message_component(interaction: &DiscordInteraction, message: &DiscordM .unwrap_or_default(); let is_dm = interaction.member.is_none(); - if !check_sender_permission(&user_id, Some(&user_name), is_dm, None) { + if !check_sender_permission( + &user_id, + Some(&user_name), + is_dm, + PermissionSource::Webhook, + None, + ) { return; } @@ -1042,9 +995,9 @@ fn handle_message_component(interaction: &DiscordInteraction, message: &DiscordM let metadata = DiscordMessageMetadata { channel_id: channel_id.clone(), - interaction_id: Some(interaction.id.clone()), - token: Some(interaction.token.clone()), - application_id: Some(interaction.application_id.clone()), + interaction_id: interaction.id.clone(), + token: interaction.token.clone(), + application_id: interaction.application_id.clone(), source_message_id: None, thread_id: None, }; @@ -1070,21 +1023,39 @@ fn handle_message_component(interaction: &DiscordInteraction, message: &DiscordM }); } +// ============================================================================ +// Permission & Pairing +// ============================================================================ + /// Context needed to send a pairing reply via Discord webhook followup. struct PairingReplyCtx { + channel_id: String, application_id: String, token: String, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PermissionSource { + Webhook, + Gateway, +} + +fn should_apply_dm_pairing(_source: PermissionSource, is_dm: bool) -> bool { + // All current permission sources (Webhook, Gateway) apply DM pairing equally. + // Kept as a function for future sources that may bypass pairing (e.g. internal). + is_dm +} + /// Check if a sender is permitted to interact with the bot. /// Returns true if allowed, false if denied (pairing reply sent if applicable). fn check_sender_permission( user_id: &str, username: Option<&str>, is_dm: bool, + source: PermissionSource, reply_ctx: Option<&PairingReplyCtx>, ) -> bool { - // 1. Owner check (highest priority, applies to all contexts). + // 1. Owner check (highest priority, applies to all contexts) let owner_id = channel_host::workspace_read(OWNER_ID_PATH).filter(|s| !s.is_empty()); if let Some(ref owner) = owner_id { if user_id != owner { @@ -1100,26 +1071,28 @@ fn check_sender_permission( return true; } - // 2. DM policy (only for DMs when no owner_id). - if !is_dm { + // 2. DM policy (only for DMs when no owner_id) + if !should_apply_dm_pairing(source, is_dm) { return true; } let dm_policy = - channel_host::workspace_read(DM_POLICY_PATH).unwrap_or_else(|| default_dm_policy()); + channel_host::workspace_read(DM_POLICY_PATH).unwrap_or_else(|| "pairing".to_string()); + if dm_policy == "open" { return true; } - // 3. Build merged allow list: config allow_from + pairing store. + // 3. Build merged allow list: config allow_from + pairing store let mut allowed: Vec = channel_host::workspace_read(ALLOW_FROM_PATH) .and_then(|s| serde_json::from_str(&s).ok()) .unwrap_or_default(); + if let Ok(store_allowed) = channel_host::pairing_read_allow_from(CHANNEL_NAME) { allowed.extend(store_allowed); } - // 4. Check sender against allow list. + // 4. Check sender against allow list let is_allowed = allowed.contains(&"*".to_string()) || allowed.contains(&user_id.to_string()) || username.is_some_and(|u| allowed.contains(&u.to_string())); @@ -1128,23 +1101,22 @@ fn check_sender_permission( return true; } - // 5. Not allowed - handle by policy. + // 5. Not allowed — handle by policy if dm_policy == "pairing" { let meta = serde_json::json!({ "user_id": user_id, "username": username, }) .to_string(); + match channel_host::pairing_upsert_request(CHANNEL_NAME, user_id, &meta) { Ok(result) => { channel_host::log( channel_host::LogLevel::Info, &format!("Pairing request for user {}: code {}", user_id, result.code), ); - if result.created { - if let Some(ctx) = reply_ctx { - let _ = send_pairing_reply(ctx, &result.code); - } + if let Some(ctx) = reply_ctx { + let _ = send_pairing_reply(ctx, &result.code); } } Err(e) => { @@ -1158,29 +1130,52 @@ fn check_sender_permission( false } -/// Send a pairing code as an ephemeral Discord followup message. +fn pairing_reply_route(ctx: &PairingReplyCtx) -> DiscordResponseRoute { + if !ctx.application_id.is_empty() && !ctx.token.is_empty() { + DiscordResponseRoute::InteractionWebhook(format!( + "{DISCORD_API_BASE}/webhooks/{}/{}", + ctx.application_id, ctx.token + )) + } else { + DiscordResponseRoute::ChannelMessage(format!( + "{DISCORD_API_BASE}/channels/{}/messages", + ctx.channel_id + )) + } +} + +/// Send a pairing code reply via webhook followup or channel message. fn send_pairing_reply(ctx: &PairingReplyCtx, code: &str) -> Result<(), String> { - let url = format!( - "https://discord.com/api/v10/webhooks/{}/{}", - ctx.application_id, ctx.token - ); - let payload = serde_json::json!({ + let route = pairing_reply_route(ctx); + + let mut payload = serde_json::json!({ "content": format!( - "To pair with this bot, run: `ironclaw pairing approve discord {}`", - code - ), - "flags": 64 + "Enter this code in IronClaw to pair your discord account: `{}`. CLI fallback: `ironclaw pairing approve discord {}`", + code, code + ) }); + + if matches!(route, DiscordResponseRoute::InteractionWebhook(_)) { + payload["flags"] = serde_json::json!(64); + } + let payload_bytes = serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize: {}", e))?; + let headers = serde_json::json!({"Content-Type": "application/json"}); + let url = match &route { + DiscordResponseRoute::InteractionWebhook(url) => url, + DiscordResponseRoute::ChannelMessage(url) => url, + }; + let result = channel_host::http_request( "POST", - &url, + url, &headers.to_string(), Some(&payload_bytes), None, ); + match result { Ok(response) if response.status >= 200 && response.status < 300 => Ok(()), Ok(response) => { @@ -1194,6 +1189,387 @@ fn send_pairing_reply(ctx: &PairingReplyCtx, code: &str) -> Result<(), String> { } } +// ============================================================================ +// Mention Polling +// ============================================================================ + +/// Maximum number of processed message IDs to keep per channel for dedup. +const DEDUP_CAP: usize = 200; + +/// Poll configured channels for new messages that mention the bot. +fn poll_for_mentions() { + let enabled = channel_host::workspace_read(POLLING_ENABLED_PATH) + .map(|v| v.trim() == "true") + .unwrap_or(false); + + if !enabled { + return; + } + + let bot_id = match get_or_fetch_bot_id() { + Some(id) => id, + None => { + channel_host::log( + channel_host::LogLevel::Warn, + "Mention polling: unable to determine bot user id", + ); + return; + } + }; + + let channel_ids: Vec = channel_host::workspace_read(MENTION_CHANNEL_IDS_PATH) + .and_then(|s| serde_json::from_str(&s).ok()) + .unwrap_or_default(); + + for channel_id in &channel_ids { + poll_channel_mentions(channel_id, &bot_id); + } +} + +/// Read the bot user ID from workspace or fetch it from the Discord API. +fn get_or_fetch_bot_id() -> Option { + if let Some(id) = channel_host::workspace_read(BOT_USER_ID_PATH).filter(|s| !s.is_empty()) { + return Some(id); + } + + let headers = discord_auth_headers_json(false); + let resp = channel_host::http_request( + "GET", + "{DISCORD_API_BASE}/users/@me", + &headers, + None, + None, + ) + .ok()?; + + if resp.status < 200 || resp.status >= 300 { + return None; + } + + let body: serde_json::Value = serde_json::from_slice(&resp.body).ok()?; + let id = body["id"].as_str()?.to_string(); + + let _ = channel_host::workspace_write(BOT_USER_ID_PATH, &id); + Some(id) +} + +/// Poll a single channel for new mention messages. +fn poll_channel_mentions(channel_id: &str, bot_id: &str) { + let cursor_path = format!("state/mention_cursor/{}", channel_id); + let last_seen = channel_host::workspace_read(&cursor_path).unwrap_or_default(); + + let messages = if last_seen.is_empty() { + // First poll: initialise cursor without emitting any messages. + if let Some(latest_id) = fetch_latest_message_id(channel_id) { + let _ = channel_host::workspace_write(&cursor_path, &latest_id); + } + return; + } else { + match fetch_messages_after_cursor(channel_id, &last_seen) { + Some(msgs) => msgs, + None => return, + } + }; + + let mut processed_ids = load_recent_processed_ids(channel_id); + let mut new_cursor = last_seen.clone(); + + for msg in &messages { + if !is_new_message(&last_seen, &msg.id) { + continue; + } + if processed_ids.contains(&msg.id) { + continue; + } + if msg.author.bot || msg.author.id == bot_id { + remember_processed_id(&msg.id, &mut processed_ids); + continue; + } + if msg.webhook_id.is_some() { + remember_processed_id(&msg.id, &mut processed_ids); + continue; + } + if !message_mentions_bot(msg, bot_id) { + remember_processed_id(&msg.id, &mut processed_ids); + continue; + } + + // Permission check (API-based poll uses Webhook source) + if !check_sender_permission( + &msg.author.id, + Some(&msg.author.username), + false, + PermissionSource::Webhook, + None, + ) { + remember_processed_id(&msg.id, &mut processed_ids); + continue; + } + + let content = strip_bot_mention(&msg.content, bot_id); + if content.is_empty() { + remember_processed_id(&msg.id, &mut processed_ids); + continue; + } + + let user_name = msg + .author + .global_name + .clone() + .unwrap_or_else(|| msg.author.username.clone()); + + let metadata = DiscordMessageMetadata { + channel_id: msg.channel_id.clone(), + interaction_id: String::new(), + token: String::new(), + application_id: String::new(), + source_message_id: Some(msg.id.clone()), + thread_id: None, + }; + + let metadata_json = match serde_json::to_string(&metadata) { + Ok(json) => json, + Err(error) => { + channel_host::log( + channel_host::LogLevel::Error, + &format!("Failed to serialize mention-poll metadata: {}", error), + ); + continue; + } + }; + + channel_host::emit_message(&EmittedMessage { + user_id: msg.author.id.clone(), + user_name: Some(user_name), + content, + thread_id: None, + metadata_json, + attachments: vec![], + }); + + remember_processed_id(&msg.id, &mut processed_ids); + + if compare_message_ids(&msg.id, &new_cursor) == std::cmp::Ordering::Greater { + new_cursor = msg.id.clone(); + } + } + + if new_cursor != last_seen { + let _ = channel_host::workspace_write(&cursor_path, &new_cursor); + } + + save_recent_processed_ids(channel_id, &processed_ids); +} + +/// Fetch the latest message ID in a channel (used for cursor initialisation). +fn fetch_latest_message_id(channel_id: &str) -> Option { + let url = format!( + "{DISCORD_API_BASE}/channels/{}/messages?limit=1", + channel_id + ); + let headers = discord_auth_headers_json(false); + let resp = channel_host::http_request("GET", &url, &headers, None, None).ok()?; + + if resp.status < 200 || resp.status >= 300 { + return None; + } + + let messages: Vec = serde_json::from_slice(&resp.body).ok()?; + messages + .first() + .and_then(|m| m["id"].as_str().map(String::from)) +} + +/// Maximum number of pages to fetch when catching up on missed messages. +const MENTION_POLL_MAX_PAGES: usize = 5; + +/// Fetch messages after `last_seen` using the `after` parameter, paginating up +/// to [`MENTION_POLL_MAX_PAGES`] pages of 100 messages each. +fn fetch_messages_after_cursor( + channel_id: &str, + last_seen: &str, +) -> Option> { + let headers = discord_auth_headers_json(false); + let mut all_messages: Vec = Vec::new(); + let mut after = last_seen.to_string(); + + for _ in 0..MENTION_POLL_MAX_PAGES { + let url = format!( + "{DISCORD_API_BASE}/channels/{}/messages?after={}&limit=100", + channel_id, after + ); + let resp = channel_host::http_request("GET", &url, &headers, None, None).ok()?; + + if resp.status < 200 || resp.status >= 300 { + let body_str = String::from_utf8_lossy(&resp.body); + channel_host::log( + channel_host::LogLevel::Warn, + &format!( + "Mention poll: failed to fetch messages for channel {}: {} - {}", + channel_id, resp.status, body_str + ), + ); + return None; + } + + let page: Vec = serde_json::from_slice(&resp.body).ok()?; + let page_len = page.len(); + + if page.is_empty() { + break; + } + + // Discord returns newest-first; find the max ID for the next page cursor + let page_max_id = page + .iter() + .map(|m| m.id.as_str()) + .max_by(|a, b| compare_message_ids(a, b)) + .map(str::to_string); + + all_messages.extend(page); + + if page_len < 100 { + break; + } + + match page_max_id { + Some(max_id) if max_id != after => after = max_id, + _ => break, + } + } + + Some(all_messages) +} + +/// Compare two Discord snowflake IDs. Falls back to lexical comparison. +fn compare_message_ids(a: &str, b: &str) -> std::cmp::Ordering { + match (a.parse::(), b.parse::()) { + (Ok(a_num), Ok(b_num)) => a_num.cmp(&b_num), + _ => a.cmp(b), + } +} + +fn dedup_ids_path(channel_id: &str) -> String { + format!("state/mention_dedup/{}", channel_id) +} + +fn load_recent_processed_ids(channel_id: &str) -> Vec { + channel_host::workspace_read(&dedup_ids_path(channel_id)) + .and_then(|s| serde_json::from_str(&s).ok()) + .unwrap_or_default() +} + +fn save_recent_processed_ids(channel_id: &str, ids: &[String]) { + let json = serde_json::to_string(ids).unwrap_or_else(|_| "[]".to_string()); + let _ = channel_host::workspace_write(&dedup_ids_path(channel_id), &json); +} + +fn remember_processed_id(msg_id: &str, ids: &mut Vec) { + if ids.contains(&msg_id.to_string()) { + return; + } + ids.push(msg_id.to_string()); + if ids.len() > DEDUP_CAP { + let excess = ids.len() - DEDUP_CAP; + ids.drain(0..excess); + } +} + +/// Returns true when `current` is strictly newer than `last_seen`. +fn is_new_message(last_seen: &str, current: &str) -> bool { + compare_message_ids(current, last_seen) == std::cmp::Ordering::Greater +} + +/// Returns true if the message mentions the bot (by mention objects or content). +fn message_mentions_bot(msg: &DiscordChannelMessage, bot_id: &str) -> bool { + if msg.mentions.iter().any(|u| u.id == bot_id) { + return true; + } + let mention = format!("<@{}>", bot_id); + let mention_nick = format!("<@!{}>", bot_id); + msg.content.contains(&mention) || msg.content.contains(&mention_nick) +} + +/// Strip the bot mention prefix from content. +fn strip_bot_mention(content: &str, bot_id: &str) -> String { + let trimmed = content.trim(); + for mention in [format!("<@{}>", bot_id), format!("<@!{}>", bot_id)] { + if let Some(rest) = trimmed.strip_prefix(&mention) { + return rest.trim().to_string(); + } + } + trimmed.to_string() +} + +/// Build JSON headers string with Discord bot authorization. +/// When `include_content_type` is true, includes `Content-Type: application/json`. +fn discord_auth_headers_json(include_content_type: bool) -> String { + if include_content_type { + serde_json::json!({ + "Content-Type": "application/json" + }) + .to_string() + } else { + serde_json::json!({}).to_string() + } +} + +/// Send a DM to a Discord user by opening (or reusing) a DM channel. +fn broadcast_dm(user_id: &str, content: &str) -> Result<(), String> { + // Validate user_id is a plausible Discord snowflake (numeric, 17-20 digits) + // to avoid injecting arbitrary strings into API URLs. + if user_id.is_empty() + || !user_id.chars().all(|c| c.is_ascii_digit()) + || user_id.len() < 17 + || user_id.len() > 20 + { + return Err(format!("Invalid Discord user ID: '{}'", user_id)); + } + + // Step 1: Open (or reuse) a DM channel with the target user. + let create_dm_payload = serde_json::json!({ "recipient_id": user_id }); + let create_dm_bytes = serde_json::to_vec(&create_dm_payload) + .map_err(|e| format!("Failed to serialize DM channel request: {}", e))?; + + let dm_response = channel_host::http_request( + "POST", + &format!("{DISCORD_API_BASE}/users/@me/channels"), + &discord_auth_headers_json(true), + Some(&create_dm_bytes), + Some(10_000), + ) + .map_err(|e| format!("Failed to create DM channel: {}", e))?; + + if !(200..300).contains(&dm_response.status) { + let body = String::from_utf8_lossy(&dm_response.body); + return Err(format!( + "Discord create-DM failed: {} - {}", + dm_response.status, body + )); + } + + #[derive(Deserialize)] + struct DmChannelResponse { + id: String, + } + let dm_channel: DmChannelResponse = serde_json::from_slice(&dm_response.body) + .map_err(|e| format!("Failed to parse DM channel response: {}", e))?; + + // Step 2: Send the message to the DM channel. + let truncated = truncate_message(content); + let payload = serde_json::json!({ "content": truncated }); + let body = + serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize: {}", e))?; + send_discord_request( + "POST", + &format!("{DISCORD_API_BASE}/channels/{}/messages", dm_channel.id), + &DiscordHttpRequest { + headers_json: discord_auth_headers_json(true), + body, + }, + ) +} + fn json_response(status: u16, value: serde_json::Value) -> OutgoingHttpResponse { let body = serde_json::to_vec(&value).unwrap_or_default(); let headers = serde_json::json!({"Content-Type": "application/json"}); @@ -1208,17 +1584,12 @@ fn json_response(status: u16, value: serde_json::Value) -> OutgoingHttpResponse export!(DiscordChannel); fn truncate_message(content: &str) -> String { - if content.len() <= 2000 { + if content.chars().count() <= DISCORD_MESSAGE_CHAR_LIMIT { content.to_string() } else { - let max_bytes = 1990; - let cutoff = content - .char_indices() - .map(|(i, c)| i + c.len_utf8()) - .take_while(|&end| end <= max_bytes) - .last() - .unwrap_or(0); - let mut truncated = content[..cutoff].to_string(); + let suffix = "\n... (truncated)"; + let allowed_chars = DISCORD_MESSAGE_CHAR_LIMIT.saturating_sub(suffix.chars().count()); + let mut truncated = content.chars().take(allowed_chars).collect::(); truncated.push_str("\n... (truncated)"); truncated } @@ -1227,7 +1598,8 @@ fn truncate_message(content: &str) -> String { #[cfg(test)] mod tests { use super::*; - use ed25519_dalek::{Signer, SigningKey}; + + const DISCORD_CAPABILITIES_JSON: &str = include_str!("../discord.capabilities.json"); #[test] fn test_truncate_message() { @@ -1236,361 +1608,645 @@ mod tests { let long = "a".repeat(2005); let truncated = truncate_message(&long); - assert_eq!(truncated.len(), 2006); // 1990 + 16 chars suffix + assert_eq!(truncated.chars().count(), 2000); assert!(truncated.ends_with("\n... (truncated)")); // Test with multibyte characters (Euro sign is 3 bytes) - // 1000 chars * 3 bytes = 3000 bytes - let multi = "€".repeat(1000); + let multi = "€".repeat(2005); let truncated_multi = truncate_message(&multi); - // 1990 bytes limit. 1990 / 3 = 663 with remainder 1. - // Should truncate at 663 chars (1989 bytes). - // Suffix is 16 bytes. Total: 1989 + 16 = 2005 bytes. - assert!(truncated_multi.len() <= 2006); - assert!(truncated_multi.len() >= 2006 - 4); // Allow for max utf8 char width variance + assert_eq!(truncated_multi.chars().count(), 2000); assert!(truncated_multi.ends_with("\n... (truncated)")); let content_part = &truncated_multi[..truncated_multi.len() - 16]; assert!(content_part.chars().all(|c| c == '€')); } + #[test] + fn test_reply_plan_uses_character_count_for_attachment_threshold() { + let inline = + build_discord_reply_plan(&test_response(test_metadata_json(), "€".repeat(2000))) + .unwrap(); + + assert!(matches!(inline, DiscordReplyPlan::Inline(_))); + } + + fn test_response(metadata_json: String, content: String) -> AgentResponse { + AgentResponse { + message_id: "msg-1".to_string(), + content, + thread_id: None, + metadata_json, + attachments: vec![], + } + } + + fn test_metadata_json() -> String { + serde_json::json!({ + "channel_id": "chan-1", + "interaction_id": "int-1", + "token": "tok-1", + "application_id": "app-1", + "thread_id": null, + "embeds": [{"title": "embed title"}] + }) + .to_string() + } + + #[test] + fn test_reply_plan_threshold_uses_attachment_only_above_2000_chars() { + let inline = + build_discord_reply_plan(&test_response(test_metadata_json(), "a".repeat(2000))) + .unwrap(); + assert!(matches!(inline, DiscordReplyPlan::Inline(_))); + + let attachment = + build_discord_reply_plan(&test_response(test_metadata_json(), "a".repeat(2001))) + .unwrap(); + assert!(matches!(attachment, DiscordReplyPlan::Attachment { .. })); + } + + #[test] + fn test_reply_plan_preserves_short_message_content_and_embeds() { + let plan = build_discord_reply_plan(&test_response( + test_metadata_json(), + "short reply".to_string(), + )) + .unwrap(); + + let DiscordReplyPlan::Inline(request) = plan else { + panic!("expected inline plan"); + }; + + assert_eq!( + request.headers_json, + r#"{"Content-Type":"application/json"}"# + ); + + let payload: serde_json::Value = serde_json::from_slice(&request.body).unwrap(); + assert_eq!(payload["content"], "short reply"); + assert_eq!(payload["embeds"][0]["title"], "embed title"); + } + + #[test] + fn test_reply_plan_builds_markdown_attachment_multipart_payload() { + let content = "# Heading\n\nA long markdown reply".repeat(80); + let plan = build_discord_reply_plan(&test_response(test_metadata_json(), content.clone())) + .unwrap(); + + let DiscordReplyPlan::Attachment { upload, .. } = plan else { + panic!("expected attachment plan"); + }; + + assert!(upload + .headers_json + .contains("multipart/form-data; boundary=")); + + let body = String::from_utf8(upload.body).unwrap(); + assert!(body.contains("name=\"payload_json\"")); + assert!(body.contains("filename=\"response.md\"")); + assert!(body.contains("Content-Type: text/markdown")); + assert!(body.contains(DISCORD_ATTACHMENT_NOTICE)); + assert!(body.contains("embed title")); + assert!(body.contains(&content)); + } + + #[test] + fn test_reply_plan_uses_dynamic_multipart_boundary() { + let content = "# Heading\n\nA long markdown reply".repeat(80); + + let first = build_discord_reply_plan(&test_response(test_metadata_json(), content.clone())) + .unwrap(); + let second = + build_discord_reply_plan(&test_response(test_metadata_json(), content)).unwrap(); + + let DiscordReplyPlan::Attachment { + upload: first_upload, + .. + } = first + else { + panic!("expected attachment plan"); + }; + let DiscordReplyPlan::Attachment { + upload: second_upload, + .. + } = second + else { + panic!("expected attachment plan"); + }; + + let first_headers: serde_json::Value = + serde_json::from_str(&first_upload.headers_json).unwrap(); + let second_headers: serde_json::Value = + serde_json::from_str(&second_upload.headers_json).unwrap(); + + let first_boundary = first_headers["Content-Type"] + .as_str() + .unwrap() + .strip_prefix("multipart/form-data; boundary=") + .unwrap(); + let second_boundary = second_headers["Content-Type"] + .as_str() + .unwrap() + .strip_prefix("multipart/form-data; boundary=") + .unwrap(); + + assert!(first_boundary.starts_with(DISCORD_MULTIPART_BOUNDARY)); + assert!(second_boundary.starts_with(DISCORD_MULTIPART_BOUNDARY)); + assert_ne!(first_boundary, second_boundary); + + let first_body = String::from_utf8(first_upload.body).unwrap(); + let second_body = String::from_utf8(second_upload.body).unwrap(); + assert!(first_body.contains(&format!("--{first_boundary}\r\n"))); + assert!(second_body.contains(&format!("--{second_boundary}\r\n"))); + } + + #[test] + fn test_reply_plan_includes_truncated_text_fallback_for_attachment_failures() { + let content = "a".repeat(2400); + let plan = build_discord_reply_plan(&test_response(test_metadata_json(), content.clone())) + .unwrap(); + + let DiscordReplyPlan::Attachment { fallback, .. } = plan else { + panic!("expected attachment plan"); + }; + + let payload: serde_json::Value = serde_json::from_slice(&fallback.body).unwrap(); + assert_eq!(payload["content"], truncate_message(&content)); + assert_eq!(payload["embeds"][0]["title"], "embed title"); + } + #[test] fn test_metadata_serialization() { let metadata = DiscordMessageMetadata { channel_id: "123".into(), - interaction_id: Some("456".into()), - token: Some("abc".into()), - application_id: Some("789".into()), + interaction_id: "456".into(), + token: "abc".into(), + application_id: "789".into(), source_message_id: None, thread_id: None, }; let json = serde_json::to_string(&metadata).unwrap(); let parsed: DiscordMessageMetadata = serde_json::from_str(&json).unwrap(); assert_eq!(parsed.channel_id, "123"); - assert_eq!(parsed.interaction_id.as_deref(), Some("456")); + assert_eq!(parsed.interaction_id, "456"); } #[test] - fn test_is_new_message() { - assert!(is_new_message(None, "100")); - assert!(is_new_message(Some("100"), "200")); - assert!(!is_new_message(Some("200"), "100")); - assert!(!is_new_message(Some("100"), "100")); - assert!(is_new_message(Some("abc"), "abd")); - assert!(!is_new_message(Some("abd"), "abc")); + fn test_metadata_backward_compat_with_old_option_format() { + // Old metadata format used Option for these fields + let old_json = r#"{ + "channel_id": "123", + "interaction_id": null, + "token": null, + "application_id": null, + "thread_id": null + }"#; + let parsed: DiscordMessageMetadata = serde_json::from_str(old_json).unwrap(); + assert_eq!(parsed.channel_id, "123"); + assert!(parsed.interaction_id.is_empty()); + + // Old format without the fields at all + let minimal_json = r#"{"channel_id": "456"}"#; + let parsed: DiscordMessageMetadata = serde_json::from_str(minimal_json).unwrap(); + assert_eq!(parsed.channel_id, "456"); + assert!(parsed.interaction_id.is_empty()); + assert!(parsed.token.is_empty()); + assert!(parsed.application_id.is_empty()); } #[test] - fn test_strip_bot_mention() { - assert_eq!(strip_bot_mention("<@123> hello", "123"), "hello"); - assert_eq!(strip_bot_mention("<@!123> hello", "123"), "hello"); - assert_eq!(strip_bot_mention("<@123>", "123"), ""); + fn test_response_route_uses_webhook_for_interactions() { + let metadata = DiscordMessageMetadata { + channel_id: "123".into(), + interaction_id: "456".into(), + token: "tok".into(), + application_id: "app".into(), + source_message_id: None, + thread_id: None, + }; + assert_eq!( - strip_bot_mention("hello <@123> world <@!123>", "123"), - "hello world" + response_route_for_metadata(&metadata), + DiscordResponseRoute::InteractionWebhook( + format!("{DISCORD_API_BASE}/webhooks/app/tok/messages/@original") + ) ); } #[test] - fn test_message_mentions_bot() { - let msg = DiscordChannelMessage { - id: "1".to_string(), - content: "hello <@123>".to_string(), - channel_id: "10".to_string(), - author: DiscordChannelAuthor { - id: "u1".to_string(), - username: "alice".to_string(), - global_name: None, - bot: false, - }, - mentions: vec![], - webhook_id: None, + fn test_response_route_uses_channel_messages_for_gateway_metadata() { + let metadata = DiscordMessageMetadata { + channel_id: "chan-1".into(), + interaction_id: String::new(), + token: String::new(), + application_id: String::new(), + source_message_id: None, + thread_id: None, }; - assert!(message_mentions_bot(&msg, "123")); - assert!(!message_mentions_bot(&msg, "999")); + + assert_eq!( + response_route_for_metadata(&metadata), + DiscordResponseRoute::ChannelMessage( + format!("{DISCORD_API_BASE}/channels/chan-1/messages") + ) + ); } #[test] - fn test_message_mentions_bot_via_mentions_array() { - let msg = DiscordChannelMessage { - id: "2".to_string(), - content: "hello".to_string(), - channel_id: "10".to_string(), - author: DiscordChannelAuthor { - id: "u1".to_string(), - username: "alice".to_string(), - global_name: None, - bot: false, - }, - mentions: vec![DiscordUser { - id: "777".to_string(), - username: "bot".to_string(), - global_name: None, - }], - webhook_id: None, + fn test_typing_request_url_uses_channel_id_for_thinking_status() { + let update = StatusUpdate { + status: StatusType::Thinking, + message: "Thinking...".to_string(), + metadata_json: serde_json::json!({ + "channel_id": "chan-42", + "interaction_id": "", + "token": "", + "application_id": "", + "thread_id": null + }) + .to_string(), }; - assert!(message_mentions_bot(&msg, "777")); + + assert_eq!( + typing_request_url_for_update(&update), + Some(format!("{DISCORD_API_BASE}/channels/chan-42/typing")) + ); } #[test] - fn test_compare_message_ids_numeric_and_lexical_fallback() { - assert_eq!(compare_message_ids("100", "20"), Ordering::Greater); - assert_eq!(compare_message_ids("20", "100"), Ordering::Less); - assert_eq!(compare_message_ids("abc", "abd"), Ordering::Less); - assert_eq!(compare_message_ids("abd", "abc"), Ordering::Greater); + fn test_typing_request_url_ignores_non_thinking_status() { + let update = StatusUpdate { + status: StatusType::Done, + message: "Done".to_string(), + metadata_json: serde_json::json!({ + "channel_id": "chan-42", + "interaction_id": "", + "token": "", + "application_id": "", + "thread_id": null + }) + .to_string(), + }; + + assert_eq!(typing_request_url_for_update(&update), None); } #[test] - fn test_remember_processed_id_dedup_and_cap() { - let mut ids = Vec::new(); - for i in 0..220 { - remember_processed_id(&mut ids, &format!("{}", i)); - } - assert_eq!(ids.len(), 200); - assert_eq!(ids.first().map(String::as_str), Some("20")); - assert_eq!(ids.last().map(String::as_str), Some("219")); + fn test_typing_request_url_ignores_invalid_metadata() { + let update = StatusUpdate { + status: StatusType::Thinking, + message: "Thinking...".to_string(), + metadata_json: "not-json".to_string(), + }; - remember_processed_id(&mut ids, "219"); - assert_eq!(ids.len(), 200); - assert_eq!(ids.last().map(String::as_str), Some("219")); + assert_eq!(typing_request_url_for_update(&update), None); } #[test] - fn test_header_case_insensitive() { - let mut headers = HashMap::new(); - headers.insert("X-Signature-Timestamp".to_string(), "123".to_string()); - assert_eq!( - header_case_insensitive(&headers, "x-signature-timestamp"), - Some("123") - ); - assert_eq!(header_case_insensitive(&headers, "missing"), None); + fn test_parse_slash_command_interaction() { + // Verify that a slash command interaction deserializes correctly. + let json = r#"{ + "type": 2, + "id": "int_1", + "application_id": "app_1", + "channel_id": "ch_1", + "member": { + "user": { + "id": "user_1", + "username": "testuser", + "global_name": "Test User" + } + }, + "data": { + "id": "cmd_1", + "name": "ask", + "options": [ + {"name": "question", "value": "What is rust?"} + ] + }, + "token": "token_abc" + }"#; + + let interaction: DiscordInteraction = serde_json::from_str(json).unwrap(); + assert_eq!(interaction.interaction_type, 2); + assert!(interaction.data.is_some()); } #[test] - fn test_discord_auth_headers_json_shape() { - let with_ct: serde_json::Value = - serde_json::from_str(&discord_auth_headers_json(true)).unwrap(); + fn test_capabilities_default_to_gateway_mode() { + let caps: serde_json::Value = + serde_json::from_str(DISCORD_CAPABILITIES_JSON).expect("capabilities parse"); + let allowlist = caps["capabilities"]["http"]["allowlist"] + .as_array() + .expect("http allowlist array"); + assert_eq!( - with_ct.get("Content-Type").and_then(|v| v.as_str()), - Some("application/json") + caps["capabilities"]["channel"]["allow_polling"], + serde_json::Value::Bool(true) ); + assert!(allowlist.iter().any(|entry| { + entry["host"] == serde_json::Value::String("gateway.discord.gg".to_string()) + && entry["methods"] == serde_json::json!(["GET"]) + })); assert_eq!( - with_ct.get("Authorization").and_then(|v| v.as_str()), - Some("Bot {DISCORD_BOT_TOKEN}") + caps["capabilities"]["websocket"]["url"], + serde_json::Value::String("wss://gateway.discord.gg/?v=10&encoding=json".to_string()) + ); + assert_eq!( + caps["capabilities"]["websocket"]["connect_on_start"], + serde_json::Value::Bool(true) + ); + assert_eq!( + caps["capabilities"]["websocket"]["identify_secret_name"], + serde_json::Value::String("discord_bot_token".to_string()) ); - - let no_ct: serde_json::Value = - serde_json::from_str(&discord_auth_headers_json(false)).unwrap(); - assert!(no_ct.get("Content-Type").is_none()); assert_eq!( - no_ct.get("Authorization").and_then(|v| v.as_str()), - Some("Bot {DISCORD_BOT_TOKEN}") + caps["capabilities"]["websocket"]["identify"]["intents"], + serde_json::Value::Number(4609u64.into()) ); } #[test] - fn test_verify_discord_request_signature_valid() { - let signing_key = SigningKey::from_bytes(&[7u8; 32]); - let public_key_hex = hex::encode(signing_key.verifying_key().to_bytes()); - let timestamp = "1234567890"; - let body = br#"{"type":1}"#; - - let mut signed = Vec::new(); - signed.extend_from_slice(timestamp.as_bytes()); - signed.extend_from_slice(body); - let signature = signing_key.sign(&signed); - - let mut headers = HashMap::new(); - headers.insert( - "x-signature-ed25519".to_string(), - hex::encode(signature.to_bytes()), - ); - headers.insert("x-signature-timestamp".to_string(), timestamp.to_string()); + fn test_parse_gateway_event_queue_emits_message_create_after_ready() { + let queue_json = serde_json::json!([ + serde_json::json!({ + "op": 0, + "t": "READY", + "d": { + "user": { + "id": "bot-1", + "username": "ironclaw", + "global_name": "IronClaw", + "bot": true + } + } + }) + .to_string(), + serde_json::json!({ + "op": 0, + "t": "MESSAGE_CREATE", + "d": { + "channel_id": "chan-1", + "guild_id": "guild-1", + "content": "<@bot-1> hello from discord", + "author": { + "id": "user-1", + "username": "alice", + "global_name": "Alice", + "bot": false + } + } + }) + .to_string() + ]) + .to_string(); - assert!(verify_discord_request_signature( - headers, - body, - Some(&public_key_hex) - )); - } + let result = parse_gateway_event_queue(&queue_json, None); - #[test] - fn test_verify_discord_request_signature_tampered_body() { - let signing_key = SigningKey::from_bytes(&[9u8; 32]); - let public_key_hex = hex::encode(signing_key.verifying_key().to_bytes()); - let timestamp = "1234567890"; - let body = b"hello"; - - let mut signed = Vec::new(); - signed.extend_from_slice(timestamp.as_bytes()); - signed.extend_from_slice(body); - let signature = signing_key.sign(&signed); - - let mut headers = HashMap::new(); - headers.insert( - "x-signature-ed25519".to_string(), - hex::encode(signature.to_bytes()), + assert_eq!(result.bot_user_id.as_deref(), Some("bot-1")); + assert_eq!( + result.messages, + vec![ParsedGatewayMessage { + user_id: "user-1".to_string(), + user_name: "Alice".to_string(), + channel_id: "chan-1".to_string(), + content: "hello from discord".to_string(), + is_dm: false, + }] ); - headers.insert("x-signature-timestamp".to_string(), timestamp.to_string()); - - assert!(!verify_discord_request_signature( - headers, - b"hello-modified", - Some(&public_key_hex) - )); } #[test] - fn test_verify_discord_request_signature_wrong_public_key() { - let signing_key = SigningKey::from_bytes(&[11u8; 32]); - let wrong_key = SigningKey::from_bytes(&[12u8; 32]); - let timestamp = "1234567890"; - let body = b"payload"; - - let mut signed = Vec::new(); - signed.extend_from_slice(timestamp.as_bytes()); - signed.extend_from_slice(body); - let signature = signing_key.sign(&signed); - - let mut headers = HashMap::new(); - headers.insert( - "x-signature-ed25519".to_string(), - hex::encode(signature.to_bytes()), - ); - headers.insert("x-signature-timestamp".to_string(), timestamp.to_string()); + fn test_parse_gateway_event_queue_ignores_bot_and_unmentioned_guild_messages() { + let queue_json = serde_json::json!([ + serde_json::json!({ + "op": 0, + "t": "MESSAGE_CREATE", + "d": { + "channel_id": "chan-1", + "guild_id": "guild-1", + "content": "this should not trigger", + "author": { + "id": "user-1", + "username": "alice", + "global_name": "Alice", + "bot": false + } + } + }) + .to_string(), + serde_json::json!({ + "op": 0, + "t": "MESSAGE_CREATE", + "d": { + "channel_id": "dm-1", + "content": "bot echo", + "author": { + "id": "bot-1", + "username": "ironclaw", + "global_name": "IronClaw", + "bot": true + } + } + }) + .to_string(), + serde_json::json!({ + "op": 0, + "t": "MESSAGE_CREATE", + "d": { + "channel_id": "dm-2", + "content": "direct message", + "author": { + "id": "user-2", + "username": "bob", + "global_name": null, + "bot": false + } + } + }) + .to_string() + ]) + .to_string(); - assert!(!verify_discord_request_signature( - headers, - body, - Some(&hex::encode(wrong_key.verifying_key().to_bytes())) - )); + let result = parse_gateway_event_queue(&queue_json, Some("bot-1")); + + assert_eq!(result.bot_user_id.as_deref(), Some("bot-1")); + assert_eq!( + result.messages, + vec![ParsedGatewayMessage { + user_id: "user-2".to_string(), + user_name: "bob".to_string(), + channel_id: "dm-2".to_string(), + content: "direct message".to_string(), + is_dm: true, + }] + ); } #[test] - fn test_verify_discord_request_signature_missing_headers() { - let headers = HashMap::new(); - assert!(!verify_discord_request_signature( - headers, - b"abc", - Some("00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff") - )); + fn test_non_gateway_dm_pairing_behavior_is_unchanged() { + assert!(should_apply_dm_pairing(PermissionSource::Webhook, true)); + assert!(!should_apply_dm_pairing(PermissionSource::Webhook, false)); } #[test] - fn test_verify_discord_request_signature_invalid_signature_hex() { - let mut headers = HashMap::new(); - headers.insert("x-signature-ed25519".to_string(), "not-hex".to_string()); - headers.insert( - "x-signature-timestamp".to_string(), - "1234567890".to_string(), - ); - assert!(!verify_discord_request_signature( - headers, - b"abc", - Some("00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff") - )); + fn test_gateway_dm_pairing_behavior_matches_webhook_dm() { + assert!(should_apply_dm_pairing(PermissionSource::Gateway, true)); + assert!(!should_apply_dm_pairing(PermissionSource::Gateway, false)); } #[test] - fn test_verify_discord_request_signature_invalid_public_key_hex() { - let mut headers = HashMap::new(); - headers.insert("x-signature-ed25519".to_string(), "00".repeat(64)); - headers.insert( - "x-signature-timestamp".to_string(), - "1234567890".to_string(), + fn test_pairing_reply_route_uses_channel_messages_for_gateway_metadata() { + let route = pairing_reply_route(&PairingReplyCtx { + channel_id: "chan-1".to_string(), + application_id: String::new(), + token: String::new(), + }); + + assert_eq!( + route, + DiscordResponseRoute::ChannelMessage( + format!("{DISCORD_API_BASE}/channels/chan-1/messages") + ) ); - assert!(!verify_discord_request_signature( - headers, - b"abc", - Some("not-hex") - )); } #[test] - fn test_verify_discord_request_signature_invalid_lengths() { - let mut headers = HashMap::new(); - headers.insert("x-signature-ed25519".to_string(), "00".repeat(10)); - headers.insert( - "x-signature-timestamp".to_string(), - "1234567890".to_string(), + fn test_pairing_reply_route_uses_webhook_for_interactions() { + let route = pairing_reply_route(&PairingReplyCtx { + channel_id: "chan-1".to_string(), + application_id: "app-1".to_string(), + token: "tok-1".to_string(), + }); + + assert_eq!( + route, + DiscordResponseRoute::InteractionWebhook( + format!("{DISCORD_API_BASE}/webhooks/app-1/tok-1") + ) ); - assert!(!verify_discord_request_signature( - headers.clone(), - b"abc", - Some("00".repeat(31).as_str()) - )); - assert!(!verify_discord_request_signature( - headers, - b"abc", - Some("00".repeat(32).as_str()) - )); } - #[test] - fn test_verify_discord_request_signature_case_insensitive_headers() { - let signing_key = SigningKey::from_bytes(&[13u8; 32]); - let public_key_hex = hex::encode(signing_key.verifying_key().to_bytes()); - let timestamp = "1234567890"; - let body = b"case-header"; - - let mut signed = Vec::new(); - signed.extend_from_slice(timestamp.as_bytes()); - signed.extend_from_slice(body); - let signature = signing_key.sign(&signed); - - let mut headers = HashMap::new(); - headers.insert( - "X-Signature-Ed25519".to_string(), - hex::encode(signature.to_bytes()), - ); - headers.insert("X-Signature-Timestamp".to_string(), timestamp.to_string()); + // ====================================================================== + // Mention polling tests + // ====================================================================== - assert!(verify_discord_request_signature( - headers, - body, - Some(&public_key_hex) + #[test] + fn test_is_new_message() { + assert!(is_new_message("100", "200")); + assert!(!is_new_message("200", "100")); + assert!(!is_new_message("100", "100")); + // Large snowflake-like IDs + assert!(is_new_message("1234567890123456789", "1234567890123456790")); + assert!(!is_new_message( + "1234567890123456790", + "1234567890123456789" )); } #[test] - fn test_verify_discord_request_signature_empty_public_key() { - let mut headers = HashMap::new(); - headers.insert("x-signature-ed25519".to_string(), "00".repeat(64)); - headers.insert( - "x-signature-timestamp".to_string(), - "1234567890".to_string(), + fn test_strip_bot_mention() { + assert_eq!( + strip_bot_mention("<@bot-123> hello world", "bot-123"), + "hello world" + ); + assert_eq!( + strip_bot_mention("<@!bot-123> hi there", "bot-123"), + "hi there" + ); + // No mention prefix — return content as-is + assert_eq!( + strip_bot_mention("no mention here", "bot-123"), + "no mention here" ); - assert!(!verify_discord_request_signature(headers, b"abc", Some(""))); + // Only mention, no content after stripping + assert_eq!(strip_bot_mention("<@bot-123>", "bot-123"), ""); + assert_eq!(strip_bot_mention("<@bot-123> ", "bot-123"), ""); } #[test] - fn test_parse_slash_command_interaction() { - // Verify that a slash command interaction deserializes correctly. - let json = r#"{ - "type": 2, - "id": "int_1", - "application_id": "app_1", - "channel_id": "ch_1", - "member": { - "user": { - "id": "user_1", - "username": "testuser", - "global_name": "Test User" - } + fn test_message_mentions_bot() { + // Via mentions array + let msg = DiscordChannelMessage { + id: "1".to_string(), + content: "hello".to_string(), + channel_id: "ch-1".to_string(), + author: DiscordChannelAuthor { + id: "user-1".to_string(), + username: "alice".to_string(), + global_name: None, + bot: false, }, - "data": { - "id": "cmd_1", - "name": "ask", - "options": [ - {"name": "question", "value": "What is rust?"} - ] + mentions: vec![DiscordUser { + id: "bot-1".to_string(), + username: "ironclaw".to_string(), + global_name: None, + }], + webhook_id: None, + }; + assert!(message_mentions_bot(&msg, "bot-1")); + assert!(!message_mentions_bot(&msg, "other-bot")); + + // Via content + let msg2 = DiscordChannelMessage { + id: "2".to_string(), + content: "<@bot-2> do something".to_string(), + channel_id: "ch-1".to_string(), + author: DiscordChannelAuthor { + id: "user-1".to_string(), + username: "alice".to_string(), + global_name: None, + bot: false, }, - "token": "token_abc" - }"#; + mentions: vec![], + webhook_id: None, + }; + assert!(message_mentions_bot(&msg2, "bot-2")); + assert!(!message_mentions_bot(&msg2, "other-bot")); + } - let interaction: DiscordInteraction = serde_json::from_str(json).unwrap(); - assert_eq!(interaction.interaction_type, 2); - assert!(interaction.data.is_some()); + #[test] + fn test_compare_message_ids() { + use std::cmp::Ordering; + assert_eq!(compare_message_ids("100", "200"), Ordering::Less); + assert_eq!(compare_message_ids("200", "100"), Ordering::Greater); + assert_eq!(compare_message_ids("100", "100"), Ordering::Equal); + // Non-numeric fallback + assert_eq!(compare_message_ids("abc", "abd"), Ordering::Less); + assert_eq!(compare_message_ids("abd", "abc"), Ordering::Greater); + } + + #[test] + fn test_remember_processed_id_dedup_and_cap() { + let mut ids = Vec::new(); + + // Basic add + remember_processed_id("msg-1", &mut ids); + assert_eq!(ids, vec!["msg-1".to_string()]); + + // Duplicate is ignored + remember_processed_id("msg-1", &mut ids); + assert_eq!(ids.len(), 1); + + // Fill beyond DEDUP_CAP + for i in 2..=(DEDUP_CAP + 5) { + remember_processed_id(&format!("msg-{}", i), &mut ids); + } + assert_eq!(ids.len(), DEDUP_CAP); + // Oldest entries should have been drained + assert!(!ids.contains(&"msg-1".to_string())); + assert!(ids.contains(&format!("msg-{}", DEDUP_CAP + 5))); + } + + #[test] + fn test_discord_auth_headers_json_shape() { + let with_ct = discord_auth_headers_json(true); + let parsed: serde_json::Value = serde_json::from_str(&with_ct).unwrap(); + assert_eq!(parsed["Content-Type"], "application/json"); + + let without_ct = discord_auth_headers_json(false); + let parsed: serde_json::Value = serde_json::from_str(&without_ct).unwrap(); + assert!(parsed.get("Content-Type").is_none()); } } diff --git a/channels-src/feishu/Cargo.lock b/channels-src/feishu/Cargo.lock index 60f68fccaf5..4e95f3fe86f 100644 --- a/channels-src/feishu/Cargo.lock +++ b/channels-src/feishu/Cargo.lock @@ -44,6 +44,7 @@ version = "0.1.0" dependencies = [ "serde", "serde_json", + "subtle", "wit-bindgen", ] @@ -208,6 +209,12 @@ dependencies = [ "smallvec", ] +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + [[package]] name = "syn" version = "2.0.117" diff --git a/channels-src/feishu/Cargo.toml b/channels-src/feishu/Cargo.toml index 53b9357df38..957624108e9 100644 --- a/channels-src/feishu/Cargo.toml +++ b/channels-src/feishu/Cargo.toml @@ -15,6 +15,7 @@ wit-bindgen = "0.36" # Serialization serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" +subtle = "2.6" # Exclude from parent workspace (this is a standalone WASM component) diff --git a/channels-src/feishu/feishu.capabilities.json b/channels-src/feishu/feishu.capabilities.json index 877a293a960..cf344d74b36 100644 --- a/channels-src/feishu/feishu.capabilities.json +++ b/channels-src/feishu/feishu.capabilities.json @@ -21,13 +21,13 @@ }, { "name": "feishu_app_secret", - "prompt": "Enter your Feishu/Lark App Secret", + "prompt": "Enter your Feishu/Lark App Secret (from your app settings at open.feishu.cn)", "optional": false }, { "name": "feishu_verification_token", "prompt": "Enter your Feishu/Lark Verification Token (from Event Subscription webhook settings)", - "optional": true + "optional": false } ], "setup_url": "https://open.feishu.cn/app" @@ -63,13 +63,15 @@ }, "webhook": { "secret_header": "X-Feishu-Verification-Token", - "secret_name": "feishu_verification_token" + "secret_name": "feishu_verification_token", + "managed_by_host": false } } }, "config": { "app_id": null, "app_secret": null, + "verification_token": null, "api_base": "https://open.feishu.cn", "owner_id": null, "dm_policy": "pairing", diff --git a/channels-src/feishu/src/lib.rs b/channels-src/feishu/src/lib.rs index 62440d2c074..1774b1d0dfc 100644 --- a/channels-src/feishu/src/lib.rs +++ b/channels-src/feishu/src/lib.rs @@ -23,7 +23,8 @@ //! - App credentials (app_id, app_secret) are injected by the host into //! the config JSON during startup for token exchange //! - Bearer token for API calls is obtained via token exchange and cached -//! - Verification token validated by host for webhook requests +//! - Webhook requests must be authenticated by the host or by a matching +//! Feishu verification token in the request body // Generate bindings from the WIT file wit_bindgen::generate!({ @@ -32,6 +33,7 @@ wit_bindgen::generate!({ }); use serde::{Deserialize, Serialize}; +use subtle::ConstantTimeEq; // Re-export generated types use exports::near::agent::channel::{ @@ -50,6 +52,7 @@ const ALLOW_FROM_PATH: &str = "allow_from"; const API_BASE_PATH: &str = "api_base"; const APP_ID_PATH: &str = "app_id"; const APP_SECRET_PATH: &str = "app_secret"; +const VERIFICATION_TOKEN_PATH: &str = "verification_token"; const TOKEN_PATH: &str = "tenant_access_token"; const TOKEN_EXPIRY_PATH: &str = "token_expiry"; @@ -102,6 +105,10 @@ struct FeishuEventHeader { /// Tenant key. #[serde(default)] tenant_key: Option, + + /// Verification token for v2 event payloads. + #[serde(default)] + token: Option, } /// Message receive event payload (im.message.receive_v1). @@ -251,6 +258,9 @@ struct FeishuConfig { /// Feishu App Secret (for token exchange). app_secret: Option, + /// Feishu Event Subscription verification token. + verification_token: Option, + /// API base URL. Defaults to "https://open.feishu.cn" (use /// "https://open.larksuite.com" for Lark international). #[serde(default = "default_api_base")] @@ -300,6 +310,9 @@ impl Guest for FeishuChannel { if let Some(ref app_secret) = config.app_secret { let _ = channel_host::workspace_write(APP_SECRET_PATH, app_secret); } + if let Some(ref verification_token) = config.verification_token { + let _ = channel_host::workspace_write(VERIFICATION_TOKEN_PATH, verification_token); + } if let Some(owner_id) = &config.owner_id { let _ = channel_host::workspace_write(OWNER_ID_PATH, owner_id); @@ -376,6 +389,23 @@ impl Guest for FeishuChannel { } }; + let configured_token = + channel_host::workspace_read(VERIFICATION_TOKEN_PATH).filter(|token| !token.is_empty()); + if !is_authenticated_webhook( + req.secret_validated, + configured_token.as_deref(), + request_verification_token(&event), + ) { + channel_host::log( + channel_host::LogLevel::Warn, + "Rejecting unauthenticated Feishu webhook request", + ); + return json_response( + 401, + serde_json::json!({"error": "Webhook authentication failed"}), + ); + } + // Handle URL verification challenge (initial webhook setup). if event.event_type.as_deref() == Some("url_verification") { if let Some(challenge) = &event.challenge { @@ -487,30 +517,57 @@ fn handle_message_event(event_data: &serde_json::Value) { // DM pairing check for p2p chats. let chat_type = msg_event.message.chat_type.as_deref().unwrap_or("unknown"); + // Resolved user_id for the emitted message. Defaults to sender_id but + // is overwritten with the owner_id when the sender is paired, ensuring + // the message is scoped to the correct owner/tenant. + let mut user_id = sender_id.to_string(); + if chat_type == "p2p" { let dm_policy = channel_host::workspace_read(DM_POLICY_PATH).unwrap_or_else(|| "pairing".to_string()); if dm_policy == "pairing" { - let sender_name = sender_id.to_string(); - match channel_host::pairing_is_allowed("feishu", sender_id, Some(&sender_name)) { - Ok(true) => {} - Ok(false) => { - // Upsert a pairing request. + match channel_host::pairing_resolve_identity("feishu", sender_id) { + Ok(Some(owner_id)) => { + // Sender is paired; scope message to owner. + user_id = owner_id; + } + Ok(None) => { + // Unknown sender — upsert a pairing request. let meta = serde_json::json!({ "sender_id": sender_id, "chat_id": msg_event.message.chat_id, "chat_type": chat_type, }); - let _ = channel_host::pairing_upsert_request( + match channel_host::pairing_upsert_request( "feishu", sender_id, &meta.to_string(), - ); - channel_host::log( - channel_host::LogLevel::Info, - &format!("Pairing request created for {}", sender_id), - ); + ) { + Ok(result) => { + channel_host::log( + channel_host::LogLevel::Info, + &format!( + "Pairing request created for {}: {}", + sender_id, result.code + ), + ); + let _ = send_message( + sender_id, + "open_id", + &format!( + "Enter this code in IronClaw to pair your feishu account: `{}`. CLI fallback: `ironclaw pairing approve feishu {}`", + result.code, result.code + ), + ); + } + Err(e) => { + channel_host::log( + channel_host::LogLevel::Error, + &format!("Pairing upsert failed: {}", e), + ); + } + } return; } Err(e) => { @@ -556,7 +613,7 @@ fn handle_message_event(event_data: &serde_json::Value) { // Emit message to the agent. channel_host::emit_message(&EmittedMessage { - user_id: sender_id.to_string(), + user_id, user_name: None, content: text, thread_id, @@ -839,6 +896,31 @@ fn json_response(status: u16, body: serde_json::Value) -> OutgoingHttpResponse { } } +fn is_authenticated_webhook( + secret_validated: bool, + configured_token: Option<&str>, + request_token: Option<&str>, +) -> bool { + if secret_validated { + return true; + } + + match (configured_token, request_token) { + (Some(expected), Some(provided)) => { + bool::from(expected.as_bytes().ct_eq(provided.as_bytes())) + } + _ => false, + } +} + +fn request_verification_token(event: &FeishuEvent) -> Option<&str> { + event + .header + .as_ref() + .and_then(|header| header.token.as_deref()) + .or(event.token.as_deref()) +} + #[cfg(test)] mod tests { use super::*; @@ -862,7 +944,10 @@ mod tests { fn parse_token_response_rejects_missing_token() { let json = r#"{"code": 0, "msg": "ok", "expire": 7200}"#; let result: Result = serde_json::from_str(json); - assert!(result.is_err(), "should fail when tenant_access_token is missing"); + assert!( + result.is_err(), + "should fail when tenant_access_token is missing" + ); } #[test] @@ -894,4 +979,64 @@ mod tests { assert_eq!(resp.code, 10003); assert!(resp.tenant_access_token.is_empty()); } + + #[test] + fn webhook_auth_requires_host_auth_or_matching_verification_token() { + assert!( + !is_authenticated_webhook(false, None, Some("token")), + "requests without any configured verification mechanism must be rejected" + ); + assert!( + !is_authenticated_webhook(false, Some("expected"), None), + "requests missing the Feishu token must be rejected when host auth did not pass" + ); + assert!( + !is_authenticated_webhook(false, Some("expected"), Some("wrong")), + "requests with the wrong Feishu token must be rejected" + ); + assert!( + is_authenticated_webhook(false, Some("expected"), Some("expected")), + "matching Feishu verification token should authenticate the request" + ); + assert!( + is_authenticated_webhook(true, None, None), + "host-authenticated requests should still be accepted" + ); + assert!( + is_authenticated_webhook(true, Some("expected"), Some("wrong")), + "host authentication should take precedence over body token checks" + ); + } + + #[test] + fn request_verification_token_prefers_v2_header_token() { + let event: FeishuEvent = serde_json::from_str( + r#"{ + "schema": "2.0", + "header": { + "event_id": "evt_123", + "event_type": "im.message.receive_v1", + "token": "header-token" + }, + "event": {} + }"#, + ) + .unwrap(); + + assert_eq!(request_verification_token(&event), Some("header-token")); + } + + #[test] + fn request_verification_token_falls_back_to_top_level_token() { + let event: FeishuEvent = serde_json::from_str( + r#"{ + "type": "url_verification", + "challenge": "abc", + "token": "top-level-token" + }"#, + ) + .unwrap(); + + assert_eq!(request_verification_token(&event), Some("top-level-token")); + } } diff --git a/channels-src/slack/src/lib.rs b/channels-src/slack/src/lib.rs index 24f01df3934..3b6b212c6b4 100644 --- a/channels-src/slack/src/lib.rs +++ b/channels-src/slack/src/lib.rs @@ -23,6 +23,7 @@ wit_bindgen::generate!({ }); use serde::{Deserialize, Serialize}; +use std::collections::BTreeMap; // Re-export generated types use exports::near::agent::channel::{ @@ -129,9 +130,17 @@ const OWNER_ID_PATH: &str = "state/owner_id"; const DM_POLICY_PATH: &str = "state/dm_policy"; /// Workspace path for persisting allow_from (JSON array) across WASM callbacks. const ALLOW_FROM_PATH: &str = "state/allow_from"; +/// Workspace path for tracking recently active Slack threads. +const ACTIVE_THREADS_PATH: &str = "state/active_threads.json"; +/// Recently active threads expire after 24 hours to avoid reviving stale threads forever. +const ACTIVE_THREAD_TTL_MS: u64 = 24 * 60 * 60 * 1000; +/// Cap stored thread markers so the workspace state stays bounded. +const ACTIVE_THREAD_MAX_ENTRIES: usize = 256; /// Channel name for pairing store (used by pairing host APIs). const CHANNEL_NAME: &str = "slack"; +type ActiveThreads = BTreeMap; + /// Channel configuration from capabilities file. #[derive(Debug, Deserialize)] struct SlackConfig { @@ -253,80 +262,83 @@ impl Guest for SlackChannel { } fn on_respond(response: AgentResponse) -> Result<(), String> { - // Parse metadata to get channel info let metadata: SlackMessageMetadata = serde_json::from_str(&response.metadata_json) .map_err(|e| format!("Failed to parse metadata: {}", e))?; - // Build Slack API request - let mut payload = serde_json::json!({ - "channel": metadata.channel, - "text": response.content, - }); + let thread_ts = response.thread_id.or(metadata.thread_ts); + + let ts = post_slack_message( + &metadata.channel, + &response.content, + thread_ts.as_deref(), + )?; - // Add thread_ts for threaded replies - if let Some(thread_ts) = response.thread_id.or(metadata.thread_ts) { - payload["thread_ts"] = serde_json::Value::String(thread_ts); + if let Some(thread_ts) = thread_ts { + if let Err(e) = track_active_thread(&metadata.channel, &thread_ts) { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Failed to track active thread: {}", e), + ); + } } - let payload_bytes = serde_json::to_vec(&payload) - .map_err(|e| format!("Failed to serialize payload: {}", e))?; - - // Make HTTP request to Slack API - // The bot token is injected by the host based on credential configuration - let headers = serde_json::json!({ - "Content-Type": "application/json" - }); - - let result = channel_host::http_request( - "POST", - "https://slack.com/api/chat.postMessage", - &headers.to_string(), - Some(&payload_bytes), - None, + channel_host::log( + channel_host::LogLevel::Debug, + &format!( + "Posted message to Slack channel {}: ts={}", + metadata.channel, + ts.unwrap_or_default() + ), ); - match result { - Ok(http_response) => { - if http_response.status != 200 { - return Err(format!( - "Slack API returned status {}", - http_response.status - )); - } + Ok(()) + } - // Parse Slack response - let slack_response: SlackPostMessageResponse = - serde_json::from_slice(&http_response.body) - .map_err(|e| format!("Failed to parse Slack response: {}", e))?; - - if !slack_response.ok { - return Err(format!( - "Slack API error: {}", - slack_response - .error - .unwrap_or_else(|| "unknown".to_string()) - )); - } + fn on_status(_update: StatusUpdate) {} + fn on_broadcast(user_id: String, response: AgentResponse) -> Result<(), String> { + let target = resolve_broadcast_target(&user_id); + if target.is_empty() { + return Err( + "broadcast failed: no target specified. Pass a Slack channel ID (C0...) \ + or user ID (U0...) as the target." + .to_string(), + ); + } + + if !looks_like_slack_id(target) { + return Err(format!( + "Broadcast target '{}' is not a valid Slack ID (expected C/U/D/G/W prefix). \ + Use a channel ID (C0...) or user ID (U0...), not a channel name.", + target + )); + } + + let ts = post_slack_message(target, &response.content, response.thread_id.as_deref())?; + + // Track the thread so replies to this broadcast are recognized as + // active threads. Use the explicit thread_id if provided, otherwise + // fall back to the message timestamp returned by Slack (which becomes + // the thread root if someone replies to this message). + if let Some(thread_ts) = response.thread_id.as_deref().or(ts.as_deref()) { + if let Err(e) = track_active_thread(target, thread_ts) { channel_host::log( - channel_host::LogLevel::Debug, - &format!( - "Posted message to Slack channel {}: ts={}", - metadata.channel, - slack_response.ts.unwrap_or_default() - ), + channel_host::LogLevel::Warn, + &format!("Failed to track active thread: {}", e), ); - - Ok(()) } - Err(e) => Err(format!("HTTP request failed: {}", e)), } - } - fn on_status(_update: StatusUpdate) {} + channel_host::log( + channel_host::LogLevel::Debug, + &format!( + "Broadcast message to Slack target {}: ts={}", + target, + ts.unwrap_or_default() + ), + ); - fn on_broadcast(_user_id: String, _response: AgentResponse) -> Result<(), String> { - Err("broadcast not yet implemented for Slack channel".to_string()) + Ok(()) } fn on_shutdown() { @@ -452,13 +464,14 @@ fn download_and_store_slack_files(attachments: &[InboundAttachment]) { } } -/// Handle a Slack event and emit message if applicable. -fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Option) { - let attachments = extract_slack_attachments(&event.files); - - // Download and store file attachments for host-side processing +fn prepare_inbound_attachments(files: &Option>) -> Vec { + let attachments = extract_slack_attachments(files); download_and_store_slack_files(&attachments); + attachments +} +/// Handle a Slack event and emit message if applicable. +fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Option) { match event.event_type.as_str() { // Direct mention of the bot (always in a channel, not a DM) "app_mention" => { @@ -472,6 +485,7 @@ fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Opt if !check_sender_permission(&user, &channel, false) { return; } + let attachments = prepare_inbound_attachments(&event.files); emit_message( user, text, @@ -483,7 +497,7 @@ fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Opt } } - // Direct message to the bot + // Direct message or thread follow-up to the bot "message" => { // Skip messages from bots (including ourselves) if event.bot_id.is_some() || event.subtype.is_some() { @@ -496,11 +510,20 @@ fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Opt event.text, event.ts.clone(), ) { - // Only process DMs (channel IDs starting with D) - if channel.starts_with('D') { - if !check_sender_permission(&user, &channel, true) { + let is_dm = channel.starts_with('D'); + + // Check if this is a reply in a thread where we previously participated + let is_active_thread = !is_dm + && event + .thread_ts + .as_ref() + .is_some_and(|thread_ts| is_active_thread(&channel, thread_ts)); + + if is_dm || is_active_thread { + if !check_sender_permission(&user, &channel, is_dm) { return; } + let attachments = prepare_inbound_attachments(&event.files); emit_message( user, text, @@ -522,6 +545,93 @@ fn handle_slack_event(event: SlackEvent, team_id: Option, _event_id: Opt } } +fn active_thread_key(channel: &str, thread_ts: &str) -> String { + format!("{channel}/{thread_ts}") +} + +fn is_thread_marker_fresh(last_seen_millis: u64, now_millis: u64) -> bool { + now_millis.saturating_sub(last_seen_millis) <= ACTIVE_THREAD_TTL_MS +} + +fn prune_active_threads(active_threads: &mut ActiveThreads, now_millis: u64) -> bool { + let mut changed = false; + active_threads.retain(|_, last_seen_millis| { + let keep = is_thread_marker_fresh(*last_seen_millis, now_millis); + if !keep { + changed = true; + } + keep + }); + + if active_threads.len() > ACTIVE_THREAD_MAX_ENTRIES { + let mut oldest_first: Vec<_> = active_threads + .iter() + .map(|(key, last_seen_millis)| (key.clone(), *last_seen_millis)) + .collect(); + oldest_first.sort_by_key(|(_, last_seen_millis)| *last_seen_millis); + + for (key, _) in oldest_first + .into_iter() + .take(active_threads.len() - ACTIVE_THREAD_MAX_ENTRIES) + { + active_threads.remove(&key); + changed = true; + } + } + + changed +} + +fn load_active_threads() -> ActiveThreads { + let Some(raw) = channel_host::workspace_read(ACTIVE_THREADS_PATH) else { + return ActiveThreads::new(); + }; + + match serde_json::from_str(&raw) { + Ok(active_threads) => active_threads, + Err(e) => { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Failed to parse active thread state: {e}"), + ); + ActiveThreads::new() + } + } +} + +fn persist_active_threads(active_threads: &ActiveThreads) -> Result<(), String> { + let serialized = serde_json::to_string(active_threads) + .map_err(|e| format!("Failed to serialize active thread state: {e}"))?; + channel_host::workspace_write(ACTIVE_THREADS_PATH, &serialized) + .map_err(|e| format!("Failed to persist active thread state: {e}")) +} + +fn track_active_thread(channel: &str, thread_ts: &str) -> Result<(), String> { + let now_millis = channel_host::now_millis(); + let mut active_threads = load_active_threads(); + prune_active_threads(&mut active_threads, now_millis); + active_threads.insert(active_thread_key(channel, thread_ts), now_millis); + prune_active_threads(&mut active_threads, now_millis); + persist_active_threads(&active_threads) +} + +fn is_active_thread(channel: &str, thread_ts: &str) -> bool { + let now_millis = channel_host::now_millis(); + let mut active_threads = load_active_threads(); + let changed = prune_active_threads(&mut active_threads, now_millis); + + if changed { + if let Err(e) = persist_active_threads(&active_threads) { + channel_host::log( + channel_host::LogLevel::Warn, + &format!("Failed to prune active thread state: {e}"), + ); + } + } + + active_threads.contains_key(&active_thread_key(channel, thread_ts)) +} + /// Emit a message to the agent. fn emit_message( user_id: String, @@ -606,8 +716,7 @@ fn check_sender_permission(user_id: &str, channel_id: &str, is_dm: bool) -> bool } // 4. Check sender (Slack events only have user ID, not username) - let is_allowed = - allowed.contains(&"*".to_string()) || allowed.contains(&user_id.to_string()); + let is_allowed = allowed.contains(&"*".to_string()) || allowed.contains(&user_id.to_string()); if is_allowed { return true; @@ -625,14 +734,9 @@ fn check_sender_permission(user_id: &str, channel_id: &str, is_dm: bool) -> bool Ok(result) => { channel_host::log( channel_host::LogLevel::Info, - &format!( - "Pairing request for user {}: code {}", - user_id, result.code - ), + &format!("Pairing request for user {}: code {}", user_id, result.code), ); - if result.created { - let _ = send_pairing_reply(channel_id, &result.code); - } + let _ = send_pairing_reply(channel_id, &result.code); } Err(e) => { channel_host::log( @@ -650,8 +754,8 @@ fn send_pairing_reply(channel_id: &str, code: &str) -> Result<(), String> { let payload = serde_json::json!({ "channel": channel_id, "text": format!( - "To pair with this bot, run: `ironclaw pairing approve slack {}`", - code + "Enter this code in IronClaw to pair your slack account: `{}`. CLI fallback: `ironclaw pairing approve slack {}`", + code, code ), }); @@ -681,6 +785,95 @@ fn send_pairing_reply(channel_id: &str, code: &str) -> Result<(), String> { } } +/// Post a message via Slack `chat.postMessage` and return the message timestamp. +/// +/// The bot token is injected by the host credential system — this function +/// only sets `Content-Type`. Used by both `on_respond` and `on_broadcast`. +fn post_slack_message( + channel: &str, + text: &str, + thread_ts: Option<&str>, +) -> Result, String> { + let payload = build_broadcast_payload(channel, text, thread_ts); + let payload_bytes = serde_json::to_vec(&payload) + .map_err(|e| format!("Failed to serialize payload: {}", e))?; + + let headers = serde_json::json!({ + "Content-Type": "application/json" + }); + + let result = channel_host::http_request( + "POST", + "https://slack.com/api/chat.postMessage", + &headers.to_string(), + Some(&payload_bytes), + None, + ); + + match result { + Ok(http_response) => { + if http_response.status != 200 { + return Err(format!( + "Slack API returned status {}", + http_response.status + )); + } + + let slack_response: SlackPostMessageResponse = + serde_json::from_slice(&http_response.body) + .map_err(|e| format!("Failed to parse Slack response: {}", e))?; + + if !slack_response.ok { + return Err(format!( + "Slack API error: {}", + slack_response + .error + .unwrap_or_else(|| "unknown".to_string()) + )); + } + + Ok(slack_response.ts) + } + Err(e) => Err(format!("HTTP request failed: {}", e)), + } +} + +/// Normalize a broadcast target by stripping a leading `#` if present. +/// +/// The message tool passes the target as `user_id` (e.g. `#C0123ABC`, +/// `C0123ABC`, or `U0123ABC`). The Slack API expects a channel ID (C0...) +/// or user ID (U0...), not a channel name. +fn resolve_broadcast_target(raw: &str) -> &str { + raw.strip_prefix('#').unwrap_or(raw) +} + +/// Check if a string looks like a Slack ID (starts with C, U, D, G, or W followed by alphanumeric). +fn looks_like_slack_id(s: &str) -> bool { + let mut chars = s.chars(); + match chars.next() { + Some('C' | 'U' | 'D' | 'G' | 'W') => { + chars.next().is_some_and(|c| c.is_ascii_alphanumeric()) + } + _ => false, + } +} + +/// Build the JSON payload for a Slack `chat.postMessage` broadcast. +fn build_broadcast_payload( + target: &str, + content: &str, + thread_ts: Option<&str>, +) -> serde_json::Value { + let mut payload = serde_json::json!({ + "channel": target, + "text": content, + }); + if let Some(ts) = thread_ts { + payload["thread_ts"] = serde_json::Value::String(ts.to_string()); + } + payload +} + /// Strip leading bot mention from text. fn strip_bot_mention(text: &str) -> String { // Slack mentions look like <@U12345678> @@ -826,4 +1019,133 @@ mod tests { // Verify the constant is 20 MB assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024); } + + #[test] + fn test_active_thread_key_scopes_by_channel_and_thread() { + assert_eq!( + active_thread_key("C123", "1742486400.000100"), + "C123/1742486400.000100" + ); + } + + #[test] + fn test_prune_active_threads_removes_expired_entries() { + let now_millis = ACTIVE_THREAD_TTL_MS + 1_000; + let mut active_threads = ActiveThreads::from([ + ( + "C1/expired".to_string(), + now_millis - ACTIVE_THREAD_TTL_MS - 1, + ), + ("C1/fresh".to_string(), now_millis - ACTIVE_THREAD_TTL_MS), + ]); + + let changed = prune_active_threads(&mut active_threads, now_millis); + + assert!(changed); + assert!(!active_threads.contains_key("C1/expired")); + assert!(active_threads.contains_key("C1/fresh")); + } + + #[test] + fn test_prune_active_threads_trims_oldest_entries_when_over_limit() { + let now_millis = ACTIVE_THREAD_TTL_MS + 1_000; + let mut active_threads = ActiveThreads::new(); + + for i in 0..=ACTIVE_THREAD_MAX_ENTRIES { + active_threads.insert(format!("C1/{i}"), now_millis + i as u64); + } + + let changed = prune_active_threads( + &mut active_threads, + now_millis + ACTIVE_THREAD_MAX_ENTRIES as u64, + ); + + assert!(changed); + assert_eq!(active_threads.len(), ACTIVE_THREAD_MAX_ENTRIES); + assert!(!active_threads.contains_key("C1/0")); + assert!(active_threads.contains_key(&format!("C1/{ACTIVE_THREAD_MAX_ENTRIES}"))); + } + + #[test] + fn test_is_thread_marker_fresh_respects_ttl_boundary() { + let now_millis = ACTIVE_THREAD_TTL_MS + 1_000; + assert!(is_thread_marker_fresh( + now_millis - ACTIVE_THREAD_TTL_MS, + now_millis + )); + assert!(!is_thread_marker_fresh( + now_millis - ACTIVE_THREAD_TTL_MS - 1, + now_millis + )); + } + + #[test] + fn test_resolve_broadcast_target_strips_hash() { + assert_eq!(resolve_broadcast_target("#general"), "general"); + assert_eq!(resolve_broadcast_target("#staging-eli5"), "staging-eli5"); + } + + #[test] + fn test_resolve_broadcast_target_preserves_ids() { + assert_eq!(resolve_broadcast_target("C0123ABC"), "C0123ABC"); + assert_eq!(resolve_broadcast_target("U0123ABC"), "U0123ABC"); + } + + #[test] + fn test_resolve_broadcast_target_empty_input() { + assert_eq!(resolve_broadcast_target(""), ""); + assert_eq!(resolve_broadcast_target("#"), ""); + } + + #[test] + fn test_build_broadcast_payload_without_thread() { + let payload = build_broadcast_payload("C0123", "hello world", None); + assert_eq!(payload["channel"], "C0123"); + assert_eq!(payload["text"], "hello world"); + assert!(payload.get("thread_ts").is_none()); + } + + #[test] + fn test_build_broadcast_payload_with_thread() { + let payload = build_broadcast_payload("C0123", "threaded reply", Some("1742486400.000100")); + assert_eq!(payload["channel"], "C0123"); + assert_eq!(payload["text"], "threaded reply"); + assert_eq!(payload["thread_ts"], "1742486400.000100"); + } + + #[test] + fn test_looks_like_slack_id_valid() { + assert!(looks_like_slack_id("C0123ABC")); + assert!(looks_like_slack_id("U0123ABC")); + assert!(looks_like_slack_id("D0123ABC")); + assert!(looks_like_slack_id("G0123ABC")); + assert!(looks_like_slack_id("W0123ABC")); + } + + #[test] + fn test_looks_like_slack_id_invalid() { + assert!(!looks_like_slack_id("general")); + assert!(!looks_like_slack_id("staging-eli5")); + assert!(!looks_like_slack_id("")); + assert!(!looks_like_slack_id("C")); // too short, no second char + assert!(!looks_like_slack_id("c0123")); // lowercase + } + + #[test] + fn test_resolve_broadcast_target_rejects_names_via_id_check() { + // After stripping '#', channel names fail the ID check + let target = resolve_broadcast_target("#general"); + assert!(!looks_like_slack_id(target)); + + let target = resolve_broadcast_target("random-channel"); + assert!(!looks_like_slack_id(target)); + } + + #[test] + fn test_resolve_broadcast_target_accepts_prefixed_ids() { + // IDs with '#' prefix are accepted after stripping + let target = resolve_broadcast_target("#C0123ABC"); + assert!(looks_like_slack_id(target)); + assert_eq!(target, "C0123ABC"); + } } diff --git a/channels-src/telegram/Cargo.lock b/channels-src/telegram/Cargo.lock index 8d40f01e0ff..7ef6912c7bd 100644 --- a/channels-src/telegram/Cargo.lock +++ b/channels-src/telegram/Cargo.lock @@ -212,7 +212,7 @@ dependencies = [ [[package]] name = "telegram-channel" -version = "0.2.1" +version = "0.2.6" dependencies = [ "serde", "serde_json", diff --git a/channels-src/telegram/Cargo.toml b/channels-src/telegram/Cargo.toml index 182e5f5de5d..982329246f2 100644 --- a/channels-src/telegram/Cargo.toml +++ b/channels-src/telegram/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "telegram-channel" -version = "0.2.1" +version = "0.2.6" edition = "2021" description = "Telegram Bot API channel for IronClaw" license = "MIT OR Apache-2.0" diff --git a/channels-src/telegram/src/lib.rs b/channels-src/telegram/src/lib.rs index f34ed68aa7a..561fdb28dca 100644 --- a/channels-src/telegram/src/lib.rs +++ b/channels-src/telegram/src/lib.rs @@ -262,6 +262,9 @@ struct SentMessage { /// Workspace path for storing polling state. const POLLING_STATE_PATH: &str = "state/last_update_id"; +/// Workspace path for storing the most recently processed webhook update ID. +const WEBHOOK_STATE_PATH: &str = "state/last_webhook_update_id"; + /// Workspace path for persisting owner_id across WASM callbacks. const OWNER_ID_PATH: &str = "state/owner_id"; @@ -307,8 +310,7 @@ struct TelegramMessageMetadata { /// Channel configuration injected by host. /// /// The host injects runtime values like tunnel_url and webhook_secret. -/// The channel doesn't need to know about polling vs webhook mode - it just -/// checks if tunnel_url is set to determine behavior. +/// Telegram defaults to polling; webhook mode must be enabled explicitly. #[derive(Debug, Deserialize)] struct TelegramConfig { /// Bot username (without @) for mention detection in groups. @@ -333,7 +335,6 @@ struct TelegramConfig { respond_to_all_group_messages: bool, /// Public tunnel URL for webhook mode (injected by host from global settings). - /// When set, webhook mode is enabled and polling is disabled. #[serde(default)] tunnel_url: Option, @@ -342,9 +343,21 @@ struct TelegramConfig { #[serde(default)] webhook_secret: Option, + /// When true, use webhook mode if tunnel_url is available. + #[serde(default)] + webhook_enabled: bool, + /// When true, use polling mode even if tunnel_url is available. #[serde(default)] polling_enabled: bool, + + /// Poll interval in milliseconds (default 30000). + #[serde(default)] + poll_interval_ms: Option, +} + +fn webhook_mode(config: &TelegramConfig) -> bool { + config.webhook_enabled && config.tunnel_url.is_some() && !config.polling_enabled } // ============================================================================ @@ -363,6 +376,26 @@ const TELEGRAM_STATUS_MAX_CHARS: usize = 600; /// Telegram's hard limit for message text length. const TELEGRAM_MAX_MESSAGE_LEN: usize = 4096; +fn utf16_code_unit_len(text: &str) -> usize { + text.encode_utf16().count() +} + +fn prefix_within_utf16_limit(text: &str, max_units: usize) -> usize { + let mut units = 0; + let mut end = 0; + + for (byte_idx, ch) in text.char_indices() { + let ch_units = ch.len_utf16(); + if units + ch_units > max_units { + break; + } + units += ch_units; + end = byte_idx + ch.len_utf8(); + } + + end +} + fn truncate_status_message(input: &str, max_chars: usize) -> String { let mut iter = input.chars(); let truncated: String = iter.by_ref().take(max_chars).collect(); @@ -373,7 +406,7 @@ fn truncate_status_message(input: &str, max_chars: usize) -> String { } } -/// Split a long message into chunks that fit within Telegram's 4096-char limit. +/// Split a long message into chunks that fit within Telegram's 4096 UTF-16-unit limit. /// /// Tries to split at the most natural boundary available (in priority order): /// 1. Double newline (paragraph break) @@ -382,7 +415,7 @@ fn truncate_status_message(input: &str, max_chars: usize) -> String { /// 4. Word boundary (space) /// 5. Hard cut at the limit (last resort for pathological input) fn split_message(text: &str) -> Vec { - if text.chars().count() <= TELEGRAM_MAX_MESSAGE_LEN { + if utf16_code_unit_len(text) <= TELEGRAM_MAX_MESSAGE_LEN { return vec![text.to_string()]; } @@ -390,13 +423,8 @@ fn split_message(text: &str) -> Vec { let mut remaining = text; while !remaining.is_empty() { - // Count chars to find the byte offset for our window. - let window_bytes = remaining - .char_indices() - .take(TELEGRAM_MAX_MESSAGE_LEN) - .last() - .map(|(byte_idx, ch)| byte_idx + ch.len_utf8()) - .unwrap_or(remaining.len()); + // Find the longest UTF-8 prefix that fits within Telegram's UTF-16 limit. + let window_bytes = prefix_within_utf16_limit(remaining, TELEGRAM_MAX_MESSAGE_LEN); if window_bytes >= remaining.len() { // Remainder fits entirely. @@ -404,6 +432,19 @@ fn split_message(text: &str) -> Vec { break; } + if window_bytes == 0 { + // Defensive fallback: make progress even if a future caller uses a + // smaller limit than a single scalar value can fit within. + let first_char_len = remaining + .chars() + .next() + .map(|ch| ch.len_utf8()) + .unwrap_or(remaining.len()); + chunks.push(remaining[..first_char_len].to_string()); + remaining = &remaining[first_char_len..]; + continue; + } + let window = &remaining[..window_bytes]; // 1. Double newline — best paragraph boundary @@ -518,8 +559,11 @@ impl Guest for TelegramChannel { // Clear any stale owner_id from a previous config let _ = channel_host::workspace_write(OWNER_ID_PATH, ""); channel_host::log( - channel_host::LogLevel::Warn, - "No owner_id configured, bot is open to all users", + channel_host::LogLevel::Debug, + &format!( + "No owner_id configured; dm_policy={}", + config.dm_policy.as_deref().unwrap_or("pairing") + ), ); } @@ -527,27 +571,26 @@ impl Guest for TelegramChannel { let dm_policy = config.dm_policy.as_deref().unwrap_or("pairing").to_string(); let _ = channel_host::workspace_write(DM_POLICY_PATH, &dm_policy); - let allow_from_json = serde_json::to_string(&config.allow_from.unwrap_or_default()) + let allow_from_json = serde_json::to_string(&config.allow_from.clone().unwrap_or_default()) .unwrap_or_else(|_| "[]".to_string()); let _ = channel_host::workspace_write(ALLOW_FROM_PATH, &allow_from_json); // Persist bot_username and respond_to_all_group_messages for group handling let _ = channel_host::workspace_write( BOT_USERNAME_PATH, - &config.bot_username.unwrap_or_default(), + &config.bot_username.clone().unwrap_or_default(), ); let _ = channel_host::workspace_write( RESPOND_TO_ALL_GROUP_PATH, &config.respond_to_all_group_messages.to_string(), ); - // Mode: use polling if explicitly enabled, otherwise use webhooks when tunnel available. - let webhook_mode = config.tunnel_url.is_some() && !config.polling_enabled; + let webhook_mode = webhook_mode(&config); if webhook_mode { channel_host::log( channel_host::LogLevel::Info, - "Webhook mode enabled (tunnel configured)", + "Webhook mode enabled (explicitly configured)", ); // Register webhook with Telegram API — propagate errors so a bad token @@ -567,7 +610,7 @@ impl Guest for TelegramChannel { } else { channel_host::log( channel_host::LogLevel::Info, - "Polling mode enabled (no tunnel configured)", + "Polling mode enabled", ); // Delete any existing webhook before polling. Telegram returns success @@ -578,7 +621,7 @@ impl Guest for TelegramChannel { // Configure polling only if not in webhook mode let poll = if !webhook_mode { Some(PollConfig { - interval_ms: 30000, // 30 seconds minimum + interval_ms: config.poll_interval_ms.unwrap_or(30000), enabled: true, }) } else { @@ -636,9 +679,32 @@ impl Guest for TelegramChannel { } }; + let last_processed = channel_host::workspace_read(WEBHOOK_STATE_PATH) + .and_then(|value| value.parse::().ok()) + .unwrap_or(-1); + let update_id = update.update_id; + if update_id <= last_processed { + channel_host::log( + channel_host::LogLevel::Info, + &format!( + "Skipping duplicate or stale webhook update {} (last processed {})", + update_id, last_processed + ), + ); + return json_response(200, serde_json::json!({"ok": true})); + } + // Handle the update handle_update(update); + if let Err(err) = channel_host::workspace_write(WEBHOOK_STATE_PATH, &update_id.to_string()) + { + channel_host::log( + channel_host::LogLevel::Error, + &format!("Failed to persist webhook update id: {}", err), + ); + } + // Always respond 200 quickly (Telegram expects fast responses) json_response(200, serde_json::json!({"ok": true})) } @@ -1090,12 +1156,9 @@ fn download_telegram_file(file_id: &str) -> Result, String> { } // ============================================================================ -// Attachment Sending (Photo / Document) +// Attachment Sending (Photo / Voice / Document) // ============================================================================ -/// Maximum photo size for Telegram sendPhoto (10 MB). -const MAX_PHOTO_SIZE: usize = 10 * 1024 * 1024; - /// Write a multipart/form-data text field. fn write_multipart_field(body: &mut Vec, boundary: &str, name: &str, value: &str) { body.extend_from_slice(format!("--{}\r\n", boundary).as_bytes()); @@ -1138,10 +1201,27 @@ fn write_multipart_file( body.extend_from_slice(b"\r\n"); } -/// Send a photo via the Telegram Bot API (multipart upload). +/// Image MIME types that Telegram's sendPhoto API supports. +const PHOTO_MIME_TYPES: &[&str] = &["image/jpeg", "image/png", "image/gif", "image/webp"]; + +/// Audio MIME types that Telegram's sendVoice API supports (ogg/opus container). +const VOICE_MIME_TYPES: &[&str] = &["audio/ogg", "audio/opus"]; + +/// Maximum photo size for Telegram sendPhoto (10 MB). +const MAX_PHOTO_SIZE: usize = 10 * 1024 * 1024; + +/// Maximum voice note size for Telegram sendVoice (50 MB). +const MAX_VOICE_SIZE: usize = 50 * 1024 * 1024; + +/// Send a multipart file upload to a Telegram Bot API endpoint. /// -/// Falls back to `send_document()` if the photo exceeds 10 MB. -fn send_photo( +/// Shared implementation for sendPhoto, sendVoice, and sendDocument. +/// `api_method` is the Telegram method name (e.g. "sendPhoto"), +/// `field_name` is the multipart field (e.g. "photo", "voice", "document"). +#[allow(clippy::too_many_arguments)] +fn send_multipart_upload( + api_method: &str, + field_name: &str, chat_id: i64, filename: &str, mime_type: &str, @@ -1151,25 +1231,6 @@ fn send_photo( ) -> Result<(), String> { let message_thread_id = normalize_thread_id(message_thread_id); - if data.len() > MAX_PHOTO_SIZE { - channel_host::log( - channel_host::LogLevel::Info, - &format!( - "Photo {} exceeds 10MB ({}), sending as document", - filename, - data.len() - ), - ); - return send_document( - chat_id, - filename, - mime_type, - data, - reply_to_message_id, - message_thread_id, - ); - } - let boundary = format!("ironclaw-{}", channel_host::now_millis()); let mut body = Vec::new(); @@ -1190,16 +1251,21 @@ fn send_photo( &thread_id.to_string(), ); } - write_multipart_file(&mut body, &boundary, "photo", filename, mime_type, data); + write_multipart_file(&mut body, &boundary, field_name, filename, mime_type, data); body.extend_from_slice(format!("--{}--\r\n", boundary).as_bytes()); let headers = serde_json::json!({ "Content-Type": format!("multipart/form-data; boundary={}", boundary) }); + let url = format!( + "https://api.telegram.org/bot{{TELEGRAM_BOT_TOKEN}}/{}", + api_method + ); + let result = channel_host::http_request( "POST", - "https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendPhoto", + &url, &headers.to_string(), Some(&body), Some(60_000), // 60s timeout for file uploads @@ -1209,23 +1275,25 @@ fn send_photo( Ok(resp) if resp.status == 200 => { channel_host::log( channel_host::LogLevel::Debug, - &format!("Sent photo '{}' to chat {}", filename, chat_id), + &format!("Sent {} '{}' to chat {}", field_name, filename, chat_id), ); Ok(()) } Ok(resp) => { let body_str = String::from_utf8_lossy(&resp.body); Err(format!( - "sendPhoto failed (HTTP {}): {}", - resp.status, body_str + "{} failed (HTTP {}): {}", + api_method, resp.status, body_str )) } - Err(e) => Err(format!("sendPhoto HTTP request failed: {}", e)), + Err(e) => Err(format!("{} HTTP request failed: {}", api_method, e)), } } -/// Send a document via the Telegram Bot API (multipart upload). -fn send_document( +/// Send a photo via the Telegram Bot API (multipart upload). +/// +/// Falls back to `send_document()` if the photo exceeds 10 MB. +fn send_photo( chat_id: i64, filename: &str, mime_type: &str, @@ -1233,65 +1301,100 @@ fn send_document( reply_to_message_id: Option, message_thread_id: Option, ) -> Result<(), String> { - let message_thread_id = normalize_thread_id(message_thread_id); - - let boundary = format!("ironclaw-{}", channel_host::now_millis()); - let mut body = Vec::new(); - - write_multipart_field(&mut body, &boundary, "chat_id", &chat_id.to_string()); - if let Some(msg_id) = reply_to_message_id { - write_multipart_field( - &mut body, - &boundary, - "reply_to_message_id", - &msg_id.to_string(), + if data.len() > MAX_PHOTO_SIZE { + channel_host::log( + channel_host::LogLevel::Info, + &format!( + "Photo {} exceeds 10MB ({}), sending as document", + filename, + data.len() + ), ); - } - if let Some(thread_id) = message_thread_id { - write_multipart_field( - &mut body, - &boundary, - "message_thread_id", - &thread_id.to_string(), + return send_document( + chat_id, + filename, + mime_type, + data, + reply_to_message_id, + message_thread_id, ); } - write_multipart_file(&mut body, &boundary, "document", filename, mime_type, data); - body.extend_from_slice(format!("--{}--\r\n", boundary).as_bytes()); - - let headers = serde_json::json!({ - "Content-Type": format!("multipart/form-data; boundary={}", boundary) - }); + send_multipart_upload( + "sendPhoto", + "photo", + chat_id, + filename, + mime_type, + data, + reply_to_message_id, + message_thread_id, + ) +} - let result = channel_host::http_request( - "POST", - "https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendDocument", - &headers.to_string(), - Some(&body), - Some(60_000), // 60s timeout for file uploads - ); +/// Send a document via the Telegram Bot API (multipart upload). +fn send_document( + chat_id: i64, + filename: &str, + mime_type: &str, + data: &[u8], + reply_to_message_id: Option, + message_thread_id: Option, +) -> Result<(), String> { + send_multipart_upload( + "sendDocument", + "document", + chat_id, + filename, + mime_type, + data, + reply_to_message_id, + message_thread_id, + ) +} - match result { - Ok(resp) if resp.status == 200 => { - channel_host::log( - channel_host::LogLevel::Debug, - &format!("Sent document '{}' to chat {}", filename, chat_id), - ); - Ok(()) - } - Ok(resp) => { - let body_str = String::from_utf8_lossy(&resp.body); - Err(format!( - "sendDocument failed (HTTP {}): {}", - resp.status, body_str - )) - } - Err(e) => Err(format!("sendDocument HTTP request failed: {}", e)), +/// Send a voice note via the Telegram Bot API (multipart upload). +/// +/// Telegram's `sendVoice` requires ogg/opus audio and displays it as an +/// in-chat voice note with waveform and playback controls. +/// Falls back to `send_document()` if the voice note exceeds 50 MB. +fn send_voice( + chat_id: i64, + filename: &str, + mime_type: &str, + data: &[u8], + reply_to_message_id: Option, + message_thread_id: Option, +) -> Result<(), String> { + if data.len() > MAX_VOICE_SIZE { + channel_host::log( + channel_host::LogLevel::Info, + &format!( + "Voice note {} exceeds 50MB ({}), sending as document", + filename, + data.len() + ), + ); + return send_document( + chat_id, + filename, + mime_type, + data, + reply_to_message_id, + message_thread_id, + ); } + send_multipart_upload( + "sendVoice", + "voice", + chat_id, + filename, + mime_type, + data, + reply_to_message_id, + message_thread_id, + ) } -/// Image MIME types that Telegram's sendPhoto API supports. -const PHOTO_MIME_TYPES: &[&str] = &["image/jpeg", "image/png", "image/gif", "image/webp"]; - /// Send a full agent response (attachments + text) to a chat. /// /// Shared implementation for both `on_respond` and `on_broadcast`. @@ -1371,31 +1474,65 @@ fn send_response( Ok(()) } -/// Send a single attachment, choosing sendPhoto or sendDocument based on MIME type. +/// Extract the base MIME type, stripping any parameters after `;`. +/// +/// e.g. `"audio/ogg; codecs=opus"` → `"audio/ogg"` +fn base_mime_type(mime: &str) -> &str { + mime.split(';').next().unwrap_or(mime).trim() +} + +/// Attachment routing category. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum AttachmentKind { + Photo, + Voice, + Document, +} + +/// Classify an attachment's send method based on its MIME type. +fn classify_attachment(mime_type: &str) -> AttachmentKind { + let base = base_mime_type(mime_type); + if PHOTO_MIME_TYPES.contains(&base) { + AttachmentKind::Photo + } else if VOICE_MIME_TYPES.contains(&base) { + AttachmentKind::Voice + } else { + AttachmentKind::Document + } +} + +/// Send a single attachment, choosing sendPhoto, sendVoice, or sendDocument based on MIME type. fn send_attachment( chat_id: i64, attachment: &Attachment, reply_to_message_id: Option, message_thread_id: Option, ) -> Result<(), String> { - if PHOTO_MIME_TYPES.contains(&attachment.mime_type.as_str()) { - send_photo( + match classify_attachment(&attachment.mime_type) { + AttachmentKind::Photo => send_photo( chat_id, &attachment.filename, &attachment.mime_type, &attachment.data, reply_to_message_id, message_thread_id, - ) - } else { - send_document( + ), + AttachmentKind::Voice => send_voice( chat_id, &attachment.filename, &attachment.mime_type, &attachment.data, reply_to_message_id, message_thread_id, - ) + ), + AttachmentKind::Document => send_document( + chat_id, + &attachment.filename, + &attachment.mime_type, + &attachment.data, + reply_to_message_id, + message_thread_id, + ), } } @@ -1552,8 +1689,8 @@ fn send_pairing_reply(chat_id: i64, code: &str) -> Result<(), String> { send_message( chat_id, &format!( - "To pair with this bot, run: `ironclaw pairing approve telegram {}`", - code + "Enter this code in IronClaw to pair your telegram account: `{}`. CLI fallback: `ironclaw pairing approve telegram {}`", + code, code ), None, Some("Markdown"), @@ -1946,9 +2083,7 @@ fn handle_message(message: TelegramMessage) { from.id, message.chat.id, result.code ), ); - if result.created { - let _ = send_pairing_reply(message.chat.id, &result.code); - } + let _ = send_pairing_reply(message.chat.id, &result.code); } Err(e) => { channel_host::log( @@ -2150,6 +2285,10 @@ export!(TelegramChannel); mod tests { use super::*; + fn utf16_len(text: &str) -> usize { + text.encode_utf16().count() + } + #[test] fn test_split_message_short() { let text = "Hello, world!"; @@ -2177,7 +2316,7 @@ mod tests { let chunks = split_message(&text); assert!(chunks.len() > 1, "expected multiple chunks"); for chunk in &chunks { - assert!(chunk.chars().count() <= TELEGRAM_MAX_MESSAGE_LEN); + assert!(utf16_len(chunk) <= TELEGRAM_MAX_MESSAGE_LEN); } // Rejoined chunks must equal the original text exactly. let rejoined = chunks.join(" "); @@ -2193,7 +2332,7 @@ mod tests { assert!(text.len() > TELEGRAM_MAX_MESSAGE_LEN); let chunks = split_message(&text); for chunk in &chunks { - assert!(chunk.chars().count() <= TELEGRAM_MAX_MESSAGE_LEN); + assert!(utf16_len(chunk) <= TELEGRAM_MAX_MESSAGE_LEN); } } @@ -2223,7 +2362,7 @@ mod tests { let chunks = split_message(&text); assert!(chunks.len() >= 2); for chunk in &chunks { - assert!(chunk.chars().count() <= TELEGRAM_MAX_MESSAGE_LEN); + assert!(utf16_len(chunk) <= TELEGRAM_MAX_MESSAGE_LEN); } // Rejoined must preserve all characters let rejoined: String = chunks.concat(); @@ -2240,12 +2379,25 @@ mod tests { let chunks = split_message(&text); assert!(chunks.len() >= 2); for chunk in &chunks { - assert!(chunk.chars().count() <= TELEGRAM_MAX_MESSAGE_LEN); + assert!(utf16_len(chunk) <= TELEGRAM_MAX_MESSAGE_LEN); // Every char should be a complete emoji assert!(chunk.chars().all(|c| c == '\u{1F600}')); } } + #[test] + fn test_split_message_exact_utf16_limit_for_surrogate_pairs() { + let emoji = "\u{1F600}"; // 😀 + let text = emoji.repeat(TELEGRAM_MAX_MESSAGE_LEN); + + let chunks = split_message(&text); + + assert_eq!(chunks.len(), 2); + assert!(chunks + .iter() + .all(|chunk| utf16_len(chunk) <= TELEGRAM_MAX_MESSAGE_LEN)); + } + #[test] fn test_clean_message_text() { // Without bot_username: strips any leading @mention @@ -2599,6 +2751,33 @@ mod tests { ); } + #[test] + fn test_webhook_mode_requires_explicit_enable() { + let config: TelegramConfig = serde_json::from_str( + r#"{ + "tunnel_url": "https://example.ngrok.app", + "polling_enabled": false + }"#, + ) + .unwrap(); + + assert!(!webhook_mode(&config)); + } + + #[test] + fn test_webhook_mode_enabled_with_tunnel() { + let config: TelegramConfig = serde_json::from_str( + r#"{ + "tunnel_url": "https://example.ngrok.app", + "webhook_enabled": true, + "polling_enabled": false + }"#, + ) + .unwrap(); + + assert!(webhook_mode(&config)); + } + #[test] fn test_classify_status_update_tool_result_ignored() { let update = StatusUpdate { @@ -2733,11 +2912,13 @@ mod tests { assert_eq!(attachments[0].id, "large_id"); // Largest photo assert_eq!(attachments[0].mime_type, "image/jpeg"); assert_eq!(attachments[0].size_bytes, Some(54321)); - assert!(attachments[0] - .source_url - .as_ref() - .unwrap() - .contains("large_id")); + assert!( + attachments[0] + .source_url + .as_ref() + .unwrap() + .contains("large_id") + ); } #[test] @@ -2969,4 +3150,38 @@ mod tests { // Verify the constant is 20 MB, matching the Slack channel limit assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024); } + + #[test] + fn test_base_mime_type() { + assert_eq!(base_mime_type("audio/ogg"), "audio/ogg"); + assert_eq!(base_mime_type("audio/ogg; codecs=opus"), "audio/ogg"); + assert_eq!(base_mime_type("image/jpeg"), "image/jpeg"); + assert_eq!(base_mime_type("text/plain; charset=utf-8"), "text/plain"); + assert_eq!(base_mime_type(""), ""); + } + + #[test] + fn test_classify_attachment_routing() { + // Photos + assert_eq!(classify_attachment("image/jpeg"), AttachmentKind::Photo); + assert_eq!(classify_attachment("image/png"), AttachmentKind::Photo); + assert_eq!(classify_attachment("image/gif"), AttachmentKind::Photo); + assert_eq!(classify_attachment("image/webp"), AttachmentKind::Photo); + + // Voice notes — exact and parameterized + assert_eq!(classify_attachment("audio/ogg"), AttachmentKind::Voice); + assert_eq!(classify_attachment("audio/opus"), AttachmentKind::Voice); + assert_eq!( + classify_attachment("audio/ogg; codecs=opus"), + AttachmentKind::Voice + ); + + // Everything else falls through to document + assert_eq!( + classify_attachment("application/pdf"), + AttachmentKind::Document + ); + assert_eq!(classify_attachment("audio/mpeg"), AttachmentKind::Document); + assert_eq!(classify_attachment("video/mp4"), AttachmentKind::Document); + } } diff --git a/channels-src/telegram/telegram.capabilities.json b/channels-src/telegram/telegram.capabilities.json index 1526762dedf..5fa8da5f340 100644 --- a/channels-src/telegram/telegram.capabilities.json +++ b/channels-src/telegram/telegram.capabilities.json @@ -18,6 +18,14 @@ "name": "telegram_bot_token", "prompt": "Enter your Telegram Bot API token (from @BotFather)", "optional": false + }, + { + "name": "telegram_webhook_secret", + "prompt": "Webhook secret (leave empty to auto-generate)", + "optional": true, + "auto_generate": { + "length": 64 + } } ], "setup_url": "https://t.me/BotFather", @@ -64,6 +72,7 @@ "bot_username": null, "owner_id": null, "respond_to_all_group_messages": false, + "webhook_enabled": false, "polling_enabled": false, "poll_interval_ms": 30000, "dm_policy": "pairing", diff --git a/channels-src/whatsapp/Cargo.lock b/channels-src/whatsapp/Cargo.lock index 0e55d1e5324..adefa9aa3b2 100644 --- a/channels-src/whatsapp/Cargo.lock +++ b/channels-src/whatsapp/Cargo.lock @@ -269,7 +269,7 @@ dependencies = [ [[package]] name = "whatsapp-channel" -version = "0.1.0" +version = "0.2.0" dependencies = [ "serde", "serde_json", diff --git a/channels-src/whatsapp/src/lib.rs b/channels-src/whatsapp/src/lib.rs index c69a9b9f90b..e77b1146f08 100644 --- a/channels-src/whatsapp/src/lib.rs +++ b/channels-src/whatsapp/src/lib.rs @@ -872,9 +872,7 @@ fn check_sender_permission( sender_phone, result.code ), ); - if result.created { - let _ = send_pairing_reply(sender_phone, phone_number_id, &result.code); - } + let _ = send_pairing_reply(sender_phone, phone_number_id, &result.code); } Err(e) => { channel_host::log( @@ -910,8 +908,8 @@ fn send_pairing_reply( "text": { "preview_url": false, "body": format!( - "To pair with this bot, run: ironclaw pairing approve whatsapp {}", - code + "Enter this code in IronClaw to pair your whatsapp account: {}. CLI fallback: ironclaw pairing approve whatsapp {}", + code, code ) } }); diff --git a/crates/ironclaw_common/CHANGELOG.md b/crates/ironclaw_common/CHANGELOG.md new file mode 100644 index 00000000000..370d5845022 --- /dev/null +++ b/crates/ironclaw_common/CHANGELOG.md @@ -0,0 +1,24 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [0.2.0](https://github.com/nearai/ironclaw/compare/ironclaw_common-v0.1.0...ironclaw_common-v0.2.0) - 2026-04-11 + +### Added + +- *(tui)* port full-featured Ratatui terminal UI onto staging ([#1973](https://github.com/nearai/ironclaw/pull/1973)) +- *(engine)* Unified Thread-Capability-CodeAct execution engine (v2 architecture) ([#1557](https://github.com/nearai/ironclaw/pull/1557)) +- *(jobs)* per-job MCP server filtering and max_iterations cap ([#1243](https://github.com/nearai/ironclaw/pull/1243)) + +### Fixed + +- *(engine)* mission cron scheduling + timezone propagation ([#1944](https://github.com/nearai/ironclaw/pull/1944)) ([#1957](https://github.com/nearai/ironclaw/pull/1957)) + +### Other + +- Improve channel onboarding and Telegram pairing flow ([#2103](https://github.com/nearai/ironclaw/pull/2103)) diff --git a/crates/ironclaw_common/Cargo.toml b/crates/ironclaw_common/Cargo.toml new file mode 100644 index 00000000000..641fdffbccf --- /dev/null +++ b/crates/ironclaw_common/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "ironclaw_common" +version = "0.2.0" +edition = "2024" +rust-version = "1.92" +description = "Shared types and utilities for the IronClaw workspace" +authors = ["NEAR AI "] +license = "MIT OR Apache-2.0" +homepage = "https://github.com/nearai/ironclaw" +repository = "https://github.com/nearai/ironclaw" + +[package.metadata.dist] +dist = false + +[dependencies] +chrono-tz = "0.10" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +tracing = "0.1" diff --git a/crates/ironclaw_common/src/event.rs b/crates/ironclaw_common/src/event.rs new file mode 100644 index 00000000000..2a591b46d10 --- /dev/null +++ b/crates/ironclaw_common/src/event.rs @@ -0,0 +1,560 @@ +//! Application-wide event types. +//! +//! `AppEvent` is the real-time event protocol used across the entire +//! application. The web gateway serialises these to SSE / WebSocket +//! frames, but other subsystems (agent loop, orchestrator, extensions) +//! produce and consume them too. + +use serde::{Deserialize, Serialize}; + +/// A single step in a plan progress update (SSE DTO). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PlanStepDto { + pub index: usize, + pub title: String, + /// One of: "pending", "in_progress", "completed", "failed". + pub status: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub result: Option, +} + +/// A single tool decision in a reasoning update (SSE DTO). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ToolDecisionDto { + pub tool_name: String, + pub rationale: String, +} + +impl ToolDecisionDto { + /// Parse a list of tool decisions from a JSON array value. + pub fn from_json_array(value: &serde_json::Value) -> Vec { + value + .as_array() + .map(|arr| { + arr.iter() + .filter_map(|d| { + Some(Self { + tool_name: d.get("tool_name")?.as_str()?.to_string(), + rationale: d.get("rationale")?.as_str()?.to_string(), + }) + }) + .collect() + }) + .unwrap_or_default() + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "type")] +pub enum AppEvent { + #[serde(rename = "response")] + Response { content: String, thread_id: String }, + #[serde(rename = "thinking")] + Thinking { + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "tool_started")] + ToolStarted { + name: String, + #[serde(skip_serializing_if = "Option::is_none")] + detail: Option, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "tool_completed")] + ToolCompleted { + name: String, + success: bool, + #[serde(skip_serializing_if = "Option::is_none")] + error: Option, + #[serde(skip_serializing_if = "Option::is_none")] + parameters: Option, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "tool_result")] + ToolResult { + name: String, + preview: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "stream_chunk")] + StreamChunk { + content: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "status")] + Status { + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "job_started")] + JobStarted { + job_id: String, + title: String, + browse_url: String, + }, + #[serde(rename = "approval_needed")] + ApprovalNeeded { + request_id: String, + tool_name: String, + description: String, + parameters: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + /// Whether the "always" auto-approve option should be shown. + allow_always: bool, + }, + #[serde(rename = "auth_required")] + AuthRequired { + extension_name: String, + #[serde(skip_serializing_if = "Option::is_none")] + instructions: Option, + #[serde(skip_serializing_if = "Option::is_none")] + auth_url: Option, + #[serde(skip_serializing_if = "Option::is_none")] + setup_url: Option, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "auth_completed")] + AuthCompleted { + extension_name: String, + success: bool, + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "pairing_required")] + PairingRequired { + channel: String, + #[serde(skip_serializing_if = "Option::is_none")] + instructions: Option, + #[serde(skip_serializing_if = "Option::is_none")] + onboarding: Option, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "pairing_completed")] + PairingCompleted { + channel: String, + success: bool, + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "gate_required")] + GateRequired { + request_id: String, + gate_name: String, + tool_name: String, + description: String, + parameters: String, + resume_kind: serde_json::Value, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "gate_resolved")] + GateResolved { + request_id: String, + gate_name: String, + tool_name: String, + resolution: String, + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "error")] + Error { + message: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + #[serde(rename = "heartbeat")] + Heartbeat, + + // Sandbox job streaming events (worker + Claude Code bridge) + #[serde(rename = "job_message")] + JobMessage { + job_id: String, + role: String, + content: String, + }, + #[serde(rename = "job_tool_use")] + JobToolUse { + job_id: String, + tool_name: String, + input: serde_json::Value, + }, + #[serde(rename = "job_tool_result")] + JobToolResult { + job_id: String, + tool_name: String, + output: String, + }, + #[serde(rename = "job_status")] + JobStatus { job_id: String, message: String }, + #[serde(rename = "job_result")] + JobResult { + job_id: String, + status: String, + #[serde(skip_serializing_if = "Option::is_none")] + session_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + fallback_deliverable: Option, + }, + + /// An image was generated by a tool. + #[serde(rename = "image_generated")] + ImageGenerated { + event_id: String, + data_url: String, + #[serde(skip_serializing_if = "Option::is_none")] + path: Option, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + + /// Suggested follow-up messages for the user. + #[serde(rename = "suggestions")] + Suggestions { + suggestions: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + + /// Per-turn token usage and cost summary. + #[serde(rename = "turn_cost")] + TurnCost { + input_tokens: u64, + output_tokens: u64, + cost_usd: String, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + + /// Skills activated for a conversation turn. + #[serde(rename = "skill_activated")] + SkillActivated { + skill_names: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + + /// Extension activation status change (WASM channels). + #[serde(rename = "extension_status")] + ExtensionStatus { + extension_name: String, + status: String, + #[serde(skip_serializing_if = "Option::is_none")] + message: Option, + }, + + /// Agent reasoning update (why it chose specific tools). + #[serde(rename = "reasoning_update")] + ReasoningUpdate { + narrative: String, + decisions: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, + + /// Reasoning update for a sandbox job. + #[serde(rename = "job_reasoning")] + JobReasoning { + job_id: String, + narrative: String, + decisions: Vec, + }, + + // ── Engine v2 thread lifecycle events ── + /// Engine thread changed state (e.g. Running → Completed). + #[serde(rename = "thread_state_changed")] + ThreadStateChanged { + thread_id: String, + from_state: String, + to_state: String, + #[serde(skip_serializing_if = "Option::is_none")] + reason: Option, + }, + + /// A child thread was spawned by a parent thread. + #[serde(rename = "child_thread_spawned")] + ChildThreadSpawned { + parent_thread_id: String, + child_thread_id: String, + goal: String, + }, + + /// A mission spawned a new thread. + #[serde(rename = "mission_thread_spawned")] + MissionThreadSpawned { + mission_id: String, + thread_id: String, + mission_name: String, + }, + + /// Plan progress update — full checklist snapshot. + /// + /// Emitted when a plan is created, approved, or when any step changes + /// status. The UI replaces the entire step list on each event. + #[serde(rename = "plan_update")] + PlanUpdate { + /// Plan identifier (MemoryDoc ID or slug). + plan_id: String, + /// Plan title. + title: String, + /// Overall status: "draft", "approved", "executing", "completed", "failed". + status: String, + /// Full step checklist (not incremental — UI replaces entire list). + steps: Vec, + /// Associated mission ID (once approved and executing). + #[serde(skip_serializing_if = "Option::is_none")] + mission_id: Option, + /// Thread scope for SSE filtering. + #[serde(skip_serializing_if = "Option::is_none")] + thread_id: Option, + }, +} + +impl AppEvent { + /// The wire-format event type string (matches the `#[serde(rename)]` value). + pub fn event_type(&self) -> &'static str { + match self { + Self::Response { .. } => "response", + Self::Thinking { .. } => "thinking", + Self::ToolStarted { .. } => "tool_started", + Self::ToolCompleted { .. } => "tool_completed", + Self::ToolResult { .. } => "tool_result", + Self::StreamChunk { .. } => "stream_chunk", + Self::Status { .. } => "status", + Self::JobStarted { .. } => "job_started", + Self::ApprovalNeeded { .. } => "approval_needed", + Self::AuthRequired { .. } => "auth_required", + Self::AuthCompleted { .. } => "auth_completed", + Self::PairingRequired { .. } => "pairing_required", + Self::PairingCompleted { .. } => "pairing_completed", + Self::GateRequired { .. } => "gate_required", + Self::GateResolved { .. } => "gate_resolved", + Self::Error { .. } => "error", + Self::Heartbeat => "heartbeat", + Self::JobMessage { .. } => "job_message", + Self::JobToolUse { .. } => "job_tool_use", + Self::JobToolResult { .. } => "job_tool_result", + Self::JobStatus { .. } => "job_status", + Self::JobResult { .. } => "job_result", + Self::ImageGenerated { .. } => "image_generated", + Self::Suggestions { .. } => "suggestions", + Self::TurnCost { .. } => "turn_cost", + Self::SkillActivated { .. } => "skill_activated", + Self::ExtensionStatus { .. } => "extension_status", + Self::ReasoningUpdate { .. } => "reasoning_update", + Self::JobReasoning { .. } => "job_reasoning", + Self::ThreadStateChanged { .. } => "thread_state_changed", + Self::ChildThreadSpawned { .. } => "child_thread_spawned", + Self::MissionThreadSpawned { .. } => "mission_thread_spawned", + Self::PlanUpdate { .. } => "plan_update", + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Verify that `event_type()` returns the same string as the serde + /// `"type"` field for every variant. This catches drift between the + /// `#[serde(rename)]` attributes and the manual match arms. + #[test] + fn event_type_matches_serde_type_field() { + let variants: Vec = vec![ + AppEvent::Response { + content: String::new(), + thread_id: String::new(), + }, + AppEvent::Thinking { + message: String::new(), + thread_id: None, + }, + AppEvent::ToolStarted { + name: String::new(), + detail: None, + thread_id: None, + }, + AppEvent::ToolCompleted { + name: String::new(), + success: true, + error: None, + parameters: None, + thread_id: None, + }, + AppEvent::ToolResult { + name: String::new(), + preview: String::new(), + thread_id: None, + }, + AppEvent::StreamChunk { + content: String::new(), + thread_id: None, + }, + AppEvent::Status { + message: String::new(), + thread_id: None, + }, + AppEvent::JobStarted { + job_id: String::new(), + title: String::new(), + browse_url: String::new(), + }, + AppEvent::ApprovalNeeded { + request_id: String::new(), + tool_name: String::new(), + description: String::new(), + parameters: String::new(), + thread_id: None, + allow_always: false, + }, + AppEvent::AuthRequired { + extension_name: String::new(), + instructions: None, + auth_url: None, + setup_url: None, + thread_id: None, + }, + AppEvent::AuthCompleted { + extension_name: String::new(), + success: true, + message: String::new(), + thread_id: None, + }, + AppEvent::PairingRequired { + channel: String::new(), + instructions: None, + onboarding: None, + thread_id: None, + }, + AppEvent::PairingCompleted { + channel: String::new(), + success: true, + message: String::new(), + thread_id: None, + }, + AppEvent::Error { + message: String::new(), + thread_id: None, + }, + AppEvent::Heartbeat, + AppEvent::JobMessage { + job_id: String::new(), + role: String::new(), + content: String::new(), + }, + AppEvent::JobToolUse { + job_id: String::new(), + tool_name: String::new(), + input: serde_json::Value::Null, + }, + AppEvent::JobToolResult { + job_id: String::new(), + tool_name: String::new(), + output: String::new(), + }, + AppEvent::JobStatus { + job_id: String::new(), + message: String::new(), + }, + AppEvent::JobResult { + job_id: String::new(), + status: String::new(), + session_id: None, + fallback_deliverable: None, + }, + AppEvent::ImageGenerated { + event_id: String::new(), + data_url: String::new(), + path: None, + thread_id: None, + }, + AppEvent::Suggestions { + suggestions: vec![], + thread_id: None, + }, + AppEvent::TurnCost { + input_tokens: 0, + output_tokens: 0, + cost_usd: String::new(), + thread_id: None, + }, + AppEvent::SkillActivated { + skill_names: vec![], + thread_id: None, + }, + AppEvent::ExtensionStatus { + extension_name: String::new(), + status: String::new(), + message: None, + }, + AppEvent::ReasoningUpdate { + narrative: String::new(), + decisions: vec![], + thread_id: None, + }, + AppEvent::JobReasoning { + job_id: String::new(), + narrative: String::new(), + decisions: vec![], + }, + AppEvent::ThreadStateChanged { + thread_id: String::new(), + from_state: String::new(), + to_state: String::new(), + reason: None, + }, + AppEvent::ChildThreadSpawned { + parent_thread_id: String::new(), + child_thread_id: String::new(), + goal: String::new(), + }, + AppEvent::MissionThreadSpawned { + mission_id: String::new(), + thread_id: String::new(), + mission_name: String::new(), + }, + AppEvent::PlanUpdate { + plan_id: String::new(), + title: String::new(), + status: String::new(), + steps: vec![], + mission_id: None, + thread_id: None, + }, + ]; + + for variant in &variants { + let json: serde_json::Value = serde_json::to_value(variant).unwrap(); + let serde_type = json["type"].as_str().unwrap(); + assert_eq!( + variant.event_type(), + serde_type, + "event_type() mismatch for variant: {:?}", + variant + ); + } + } + + #[test] + fn round_trip_deserialize() { + let original = AppEvent::Response { + content: "hello".to_string(), + thread_id: "t1".to_string(), + }; + let json = serde_json::to_string(&original).unwrap(); + let deserialized: AppEvent = serde_json::from_str(&json).unwrap(); + assert_eq!(deserialized.event_type(), "response"); + } +} diff --git a/crates/ironclaw_common/src/lib.rs b/crates/ironclaw_common/src/lib.rs new file mode 100644 index 00000000000..852374dfd4f --- /dev/null +++ b/crates/ironclaw_common/src/lib.rs @@ -0,0 +1,14 @@ +//! Shared types and utilities for the IronClaw workspace. + +mod event; +mod timezone; +mod util; + +pub use event::{AppEvent, PlanStepDto, ToolDecisionDto}; +pub use timezone::{ValidTimezone, deserialize_option_lenient}; +pub use util::truncate_preview; + +/// Maximum worker agent loop iterations. Used by the orchestrator (server-side +/// clamp in `create_job_inner`) and the worker runtime (`worker/job.rs`). +/// A single source of truth prevents the two from drifting. +pub const MAX_WORKER_ITERATIONS: u32 = 500; diff --git a/crates/ironclaw_common/src/timezone.rs b/crates/ironclaw_common/src/timezone.rs new file mode 100644 index 00000000000..4eea97421af --- /dev/null +++ b/crates/ironclaw_common/src/timezone.rs @@ -0,0 +1,166 @@ +//! Validated IANA timezone type. + +use serde::{Deserialize, Serialize}; + +/// A validated IANA timezone. +/// +/// Wraps `chrono_tz::Tz` and guarantees the timezone string was valid at +/// construction time. Use `ValidTimezone::parse()` to create — it returns +/// `None` for empty or unrecognized timezone strings. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ValidTimezone(chrono_tz::Tz); + +impl ValidTimezone { + /// Parse an IANA timezone string. Returns `None` for empty or invalid input. + pub fn parse(s: &str) -> Option { + let trimmed = s.trim(); + if trimmed.is_empty() { + return None; + } + trimmed.parse::().ok().map(Self) + } + + /// The underlying `chrono_tz::Tz` value. + pub fn tz(&self) -> chrono_tz::Tz { + self.0 + } + + /// The IANA name (e.g. "America/New_York"). + pub fn name(&self) -> &str { + self.0.name() + } +} + +impl std::fmt::Display for ValidTimezone { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.name()) + } +} + +impl Serialize for ValidTimezone { + fn serialize(&self, serializer: S) -> Result { + self.name().serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for ValidTimezone { + fn deserialize>(deserializer: D) -> Result { + let s = String::deserialize(deserializer)?; + Self::parse(&s) + .ok_or_else(|| serde::de::Error::custom(format!("invalid IANA timezone: '{s}'"))) + } +} + +/// Lenient deserializer for `Option`. +/// +/// Use with `#[serde(default, deserialize_with = "...")]` on fields that may +/// contain invalid timezone strings from historical data. Invalid or empty +/// values deserialize as `None` instead of failing the whole record. Each +/// drop is logged at `debug!` so a typo in fresh user config is at least +/// observable in the logs even though the record loads. +pub fn deserialize_option_lenient<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + let opt: Option = Option::deserialize(deserializer)?; + match opt { + Some(s) => match ValidTimezone::parse(&s) { + Some(tz) => Ok(Some(tz)), + None => { + tracing::debug!( + raw = %s, + "lenient deserializer dropped invalid IANA timezone string to None" + ); + Ok(None) + } + }, + None => Ok(None), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_valid_timezone() { + let tz = ValidTimezone::parse("America/New_York").unwrap(); + assert_eq!(tz.name(), "America/New_York"); + } + + #[test] + fn parse_with_whitespace() { + let tz = ValidTimezone::parse(" Europe/London ").unwrap(); + assert_eq!(tz.name(), "Europe/London"); + } + + #[test] + fn parse_empty_returns_none() { + assert!(ValidTimezone::parse("").is_none()); + assert!(ValidTimezone::parse(" ").is_none()); + } + + #[test] + fn parse_invalid_returns_none() { + assert!(ValidTimezone::parse("NotATimezone").is_none()); + assert!(ValidTimezone::parse("US/FakeCity").is_none()); + } + + #[test] + fn serde_roundtrip() { + let tz = ValidTimezone::parse("Asia/Tokyo").unwrap(); + let json = serde_json::to_string(&tz).unwrap(); + assert_eq!(json, "\"Asia/Tokyo\""); + let back: ValidTimezone = serde_json::from_str(&json).unwrap(); + assert_eq!(back, tz); + } + + #[test] + fn deserialize_invalid_fails() { + let result: Result = serde_json::from_str("\"NotReal\""); + assert!(result.is_err()); + } + + #[test] + fn lenient_deserialize_valid() { + #[derive(serde::Deserialize)] + struct T { + #[serde(default, deserialize_with = "super::deserialize_option_lenient")] + tz: Option, + } + let t: T = serde_json::from_str(r#"{"tz":"America/Chicago"}"#).unwrap(); + assert_eq!(t.tz.unwrap().name(), "America/Chicago"); + } + + #[test] + fn lenient_deserialize_invalid_becomes_none() { + #[derive(serde::Deserialize)] + struct T { + #[serde(default, deserialize_with = "super::deserialize_option_lenient")] + tz: Option, + } + let t: T = serde_json::from_str(r#"{"tz":"NotReal"}"#).unwrap(); + assert!(t.tz.is_none(), "invalid timezone should become None"); + } + + #[test] + fn lenient_deserialize_null_becomes_none() { + #[derive(serde::Deserialize)] + struct T { + #[serde(default, deserialize_with = "super::deserialize_option_lenient")] + tz: Option, + } + let t: T = serde_json::from_str(r#"{"tz":null}"#).unwrap(); + assert!(t.tz.is_none()); + } + + #[test] + fn lenient_deserialize_missing_becomes_none() { + #[derive(serde::Deserialize)] + struct T { + #[serde(default, deserialize_with = "super::deserialize_option_lenient")] + tz: Option, + } + let t: T = serde_json::from_str(r#"{}"#).unwrap(); + assert!(t.tz.is_none()); + } +} diff --git a/crates/ironclaw_common/src/util.rs b/crates/ironclaw_common/src/util.rs new file mode 100644 index 00000000000..4f054671d08 --- /dev/null +++ b/crates/ironclaw_common/src/util.rs @@ -0,0 +1,100 @@ +//! Shared utility functions. + +/// Truncate a string to at most `max_bytes` bytes at a char boundary, appending "...". +/// +/// If the input is wrapped in `...` and truncation +/// removes the closing tag, the tag is re-appended so downstream XML parsers +/// never see an unclosed element. +pub fn truncate_preview(s: &str, max_bytes: usize) -> String { + if s.len() <= max_bytes { + return s.to_string(); + } + // Walk backwards from max_bytes to find a valid char boundary + let mut end = max_bytes; + while end > 0 && !s.is_char_boundary(end) { + end -= 1; + } + let mut result = format!("{}...", &s[..end]); + + // Re-close if truncation cut through the closing tag. + if s.starts_with("") { + result.push_str("\n"); + } + + result +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_truncate_preview_short_string() { + assert_eq!(truncate_preview("hello", 10), "hello"); + } + + #[test] + fn test_truncate_preview_exact_boundary() { + assert_eq!(truncate_preview("hello", 5), "hello"); + } + + #[test] + fn test_truncate_preview_truncates_ascii() { + assert_eq!(truncate_preview("hello world", 5), "hello..."); + } + + #[test] + fn test_truncate_preview_empty_string() { + assert_eq!(truncate_preview("", 10), ""); + } + + #[test] + fn test_truncate_preview_multibyte_char_boundary() { + let s = "a\u{20AC}b"; + let result = truncate_preview(s, 3); + assert_eq!(result, "a..."); + } + + #[test] + fn test_truncate_preview_emoji() { + let s = "hi\u{1F980}"; + let result = truncate_preview(s, 4); + assert_eq!(result, "hi..."); + } + + #[test] + fn test_truncate_preview_cjk() { + let s = "\u{4F60}\u{597D}\u{4E16}\u{754C}"; + let result = truncate_preview(s, 7); + assert_eq!(result, "\u{4F60}\u{597D}..."); + } + + #[test] + fn test_truncate_preview_zero_max_bytes() { + assert_eq!(truncate_preview("hello", 0), "..."); + } + + #[test] + fn test_truncate_preview_closes_tool_output_tag() { + let s = "\nSome very long content here\n"; + let result = truncate_preview(s, 60); + assert!(result.ends_with("")); + assert!(result.contains("...")); + } + + #[test] + fn test_truncate_preview_no_extra_close_when_intact() { + let s = "\nshort\n"; + let result = truncate_preview(s, 500); + assert_eq!(result, s); + assert_eq!(result.matches("").count(), 1); + } + + #[test] + fn test_truncate_preview_non_xml_unaffected() { + let s = "Just a plain long string that gets truncated"; + let result = truncate_preview(s, 10); + assert_eq!(result, "Just a pla..."); + assert!(!result.contains("")); + } +} diff --git a/crates/ironclaw_engine/CLAUDE.md b/crates/ironclaw_engine/CLAUDE.md new file mode 100644 index 00000000000..0ea1b6a1fbc --- /dev/null +++ b/crates/ironclaw_engine/CLAUDE.md @@ -0,0 +1,181 @@ +# IronClaw Engine Crate + +Unified thread-capability-CodeAct execution model. Replaces ~10 separate abstractions (Session, Job, Routine, Channel, Tool, Skill, Hook, Observer, Extension, LoopDelegate) with 5 primitives. + +## Full Architecture Plan + +See `docs/plans/2026-03-20-engine-v2-architecture.md` for the 8-phase roadmap. + +## Five Primitives + +| Primitive | Purpose | Replaces | +|-----------|---------|----------| +| **Thread** | Unit of work with lifecycle, parent-child tree, capability leases | Session + Job + Routine + Sub-agent | +| **Step** | Unit of execution (one LLM call + its action executions) | Agentic loop iteration + tool calls | +| **Capability** | Unit of effect (actions + knowledge + policies) | Tool + Skill + Hook + Extension | +| **MemoryDoc** | Unit of durable knowledge (summaries, lessons, skills) | Workspace memory blobs | +| **Project** | Unit of context (scopes memory, threads, missions) | Flat workspace namespace | + +## Build & Test + +```bash +cargo check -p ironclaw_engine +cargo clippy -p ironclaw_engine --all-targets -- -D warnings +cargo test -p ironclaw_engine +``` + +## Module Map + +``` +src/ +├── lib.rs # Public API, re-exports +├── types/ # Core data structures (no async, no I/O) +│ ├── thread.rs # Thread, ThreadId, ThreadState (state machine), ThreadType, ThreadConfig +│ ├── step.rs # Step, StepId, LlmResponse, ActionCall, ActionResult, TokenUsage +│ ├── capability.rs # Capability, ActionDef, EffectType, CapabilityLease, PolicyRule +│ ├── memory.rs # MemoryDoc, DocId, DocType (Summary/Lesson/Skill/Issue/Spec/Note) +│ ├── project.rs # Project, ProjectId +│ ├── event.rs # ThreadEvent, EventKind (18 variants for event sourcing) +│ ├── message.rs # ThreadMessage, MessageRole +│ ├── provenance.rs # Provenance enum (User/System/ToolOutput/LlmGenerated/etc.) +│ ├── conversation.rs # ConversationSurface, ConversationEntry, EntrySender +│ ├── mission.rs # Mission, MissionId, MissionCadence, MissionStatus +│ └── error.rs # EngineError, ThreadError, StepError, CapabilityError +├── traits/ # External dependency abstractions (host implements these) +│ ├── llm.rs # LlmBackend trait +│ ├── store.rs # Store trait (20 CRUD methods) +│ └── effect.rs # EffectExecutor trait +├── capability/ # Capability management +│ ├── registry.rs # CapabilityRegistry — register/get/list capabilities +│ ├── lease.rs # LeaseManager — grant/check/consume/revoke/expire leases +│ └── policy.rs # PolicyEngine — deterministic effect-level allow/deny/approve + provenance taint +├── runtime/ # Thread lifecycle management +│ ├── manager.rs # ThreadManager — spawn, stop, inject messages, join threads +│ ├── conversation.rs # ConversationManager — routes UI messages to threads +│ ├── mission.rs # MissionManager — long-running goals that spawn threads on cadence +│ ├── tree.rs # ThreadTree — parent-child relationships +│ └── messaging.rs # ThreadSignal, ThreadOutcome, signal channels +├── executor/ # Step execution +│ ├── loop_engine.rs # ExecutionLoop — core loop replacing run_agentic_loop() +│ ├── structured.rs # Tier 0: structured tool call execution +│ ├── scripting.rs # Tier 1: embedded Python via Monty (CodeAct/RLM) +│ ├── context.rs # Context builder (messages + actions from leases + memory docs) +│ ├── compaction.rs # Context compaction when approaching model context limit +│ ├── prompt.rs # System prompt construction (CodeAct preamble/postamble) +│ └── trace.rs # Execution trace recording and retrospective analysis +├── memory/ # Memory document system +│ ├── store.rs # MemoryStore — project-scoped doc CRUD +│ ├── retrieval.rs # RetrievalEngine — keyword-based context retrieval from project docs +│ └── skill_tracker.rs # SkillTracker — confidence tracking, versioned updates, rollback +└── reliability.rs # ReliabilityTracker — per-action success rate and latency via EMA +``` + +## Thread State Machine + +``` +Created → Running → Waiting → Running (resume) + → Suspended → Running (resume) + → Completed → Done + → Failed +``` + +Validated by `ThreadState::can_transition_to()`. Terminal states: `Done`, `Failed`. + +## Learning Missions + +Four event-driven missions fire automatically after thread completion: + +1. **Error diagnosis** (`self-improvement`) — fires when a thread completes with trace issues. Diagnoses root cause and applies prompt overlays or orchestrator patches. +2. **Skill repair** (`skill-repair`) — fires when a completed thread used an active skill but the trace suggests the skill instructions were stale, incomplete, or missing verification. Applies the smallest safe versioned update to the implicated skill. +3. **Skill extraction** (`skill-extraction`) — fires when a thread succeeds with 5+ steps and 3+ tool actions. Extracts reusable skills with activation metadata, CodeAct code snippets, and domain tags. Output stored as `DocType::Skill` MemoryDoc. +4. **Conversation insights** (`conversation-insights`) — fires every 5 completed threads in a project. Extracts user preferences, domain knowledge, and workflow patterns. + +Created by `MissionManager::ensure_learning_missions()` at project bootstrap. + +## Data Retention: Never Delete LLM Output + +Thread messages, steps, and events are **never deleted** from the database. This data (context fed to the model, reasoning, tool calls, results) is the most valuable information in the system. The `Store` implementation uses in-memory HashMaps as a cache backed by the database (via Workspace). "Cleanup" of terminal threads means evicting from in-memory caches to bound RAM — the database rows always stay. `load_thread()`, `load_steps()`, and `load_events()` must fall back to the database on a cache miss. + +## External Trait Boundaries + +The engine defines three traits that the host crate implements: + +| Trait | Purpose | Host wraps | +|-------|---------|------------| +| `LlmBackend` | `complete(messages, actions, config) -> LlmOutput` | `LlmProvider` | +| `Store` | Thread/Step/Event/Project/Doc/Lease CRUD | `Database` (PostgreSQL + libSQL) | +| `EffectExecutor` | `execute_action(name, params, lease, ctx) -> ActionResult` | `ToolRegistry` + `SafetyLayer` | + +## Execution Loop + +`ExecutionLoop::run()` handles three `LlmResponse` variants: + +1. Check signals (Stop, InjectMessage) via `mpsc::Receiver` +2. Build context (messages + available actions from active leases) +3. Call LLM via `LlmBackend::complete()` +4. **If `Text`**: check tool intent nudge, return if final response +5. **If `ActionCalls`** (Tier 0): for each call, find lease → check policy → consume use → execute via `EffectExecutor` → record result +6. **If `Code`** (Tier 1): execute Python via Monty with context-as-variables and `llm_query()` support → compact metadata in context +7. Record Step, emit ThreadEvents +8. Repeat until: text response, stop signal, max iterations, or approval needed + +## CodeAct / Monty Integration (Tier 1) + +Python execution via Monty interpreter (`executor/scripting.rs`). Follows the RLM (Recursive Language Model) pattern. + +**Context as variables** (not attention input): +- Thread messages injected as `context` Python variable +- Thread goal as `goal`, step index as `step_number` +- Prior action results as `previous_results` dict +- The LLM's chat context stays lean; full data lives in REPL variables + +**Tool dispatch**: Unknown function calls suspend the VM → lease check → policy check → `EffectExecutor` → result returned to Python. + +**`llm_query(prompt, context)`**: Recursive subagent call. Suspends VM → spawns single-shot LLM call → returns text result as Python string. Results stay as variables (symbolic composition), not injected into parent's attention window. + +**Compact output metadata**: Between code steps, only a summary is added to chat context (`"[code output] stdout (4532 chars): The results show..."`) — not the full output. This prevents context bloat across iterations. + +**Resource limits**: 30s timeout, 64MB memory, 1M allocations. All execution wrapped in `catch_unwind` for Monty panic safety. + +## Capability Leases + +Threads don't have static permissions. They receive **leases** — scoped, time-limited, use-limited grants: + +```rust +CapabilityLease { + thread_id, capability_name, granted_actions, + expires_at: Option, // time-limited + max_uses: Option, // use-limited + revoked: bool, +} +``` + +The `PolicyEngine` evaluates actions against leases deterministically: `Deny > RequireApproval > Allow`. + +## Effect Types + +Every action declares its side effects. The policy engine uses these for allow/deny: + +``` +ReadLocal, ReadExternal, WriteLocal, WriteExternal, +CredentialedNetwork, Compute, Financial +``` + +## Key Design Decisions + +1. **No dependency on main `ironclaw` crate** — clean separation, testable in isolation +2. **No safety logic** — sanitization/leak detection is applied at the adapter boundary (`EffectExecutor` impl) +3. **Event sourcing from day one** — every thread records a complete event log via `ThreadEvent` +4. **Tier 0 + Tier 1** — structured tool calls (Tier 0) and embedded Python via Monty (Tier 1, CodeAct) +5. **Engine owns its message type** — `ThreadMessage` is simpler than `ChatMessage`; bridge adapters handle conversion +6. **RLM pattern** — context as variable (not attention input), recursive `llm_query()`, compact output metadata between steps + +## Code Style + +Follows the main crate's conventions from `/CLAUDE.md`: +- No `.unwrap()` or `.expect()` in production code (tests are fine) +- `thiserror` for error types +- Map errors with context +- Prefer strong types over strings (newtypes for IDs) +- All I/O is async with tokio +- `Arc` for shared state, `RwLock` for concurrent access diff --git a/crates/ironclaw_engine/Cargo.toml b/crates/ironclaw_engine/Cargo.toml new file mode 100644 index 00000000000..5c701c41511 --- /dev/null +++ b/crates/ironclaw_engine/Cargo.toml @@ -0,0 +1,34 @@ +[package] +name = "ironclaw_engine" +version = "0.1.0" +edition = "2024" +rust-version = "1.92" +description = "Unified thread-capability-CodeAct execution engine for IronClaw" +authors = ["NEAR AI "] +license = "MIT OR Apache-2.0" +homepage = "https://github.com/nearai/ironclaw" +repository = "https://github.com/nearai/ironclaw" +publish = false + +[package.metadata.dist] +dist = false + +[dependencies] +async-trait = "0.1" +cron = "0.13" +ironclaw_common = { path = "../ironclaw_common", version = "0.2.0" } +ironclaw_skills = { path = "../ironclaw_skills", version = "0.1.0", default-features = false } +chrono = { version = "0.4", features = ["serde"] } +monty = { git = "https://github.com/pydantic/monty.git", tag = "v0.0.11" } +regex = "1" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +thiserror = "2" +tokio = { version = "1", features = ["sync", "time", "macros", "rt"] } +tracing = "0.1" +uuid = { version = "1", features = ["v4", "serde"] } +sha2 = "0.10" + +[dev-dependencies] +pretty_assertions = "1" +tokio = { version = "1", features = ["full", "test-util"] } diff --git a/crates/ironclaw_engine/MONTY.md b/crates/ironclaw_engine/MONTY.md new file mode 100644 index 00000000000..e4e318710ed --- /dev/null +++ b/crates/ironclaw_engine/MONTY.md @@ -0,0 +1,67 @@ +# Monty Integration + +Monty is the embedded Python interpreter used for Tier 1 (CodeAct) execution. It's a lightweight Rust-native Python implementation — not CPython — so it has a restricted feature set. + +**Source**: `git = "https://github.com/pydantic/monty.git", tag = "v0.0.11"` +**Pinned at**: `v0.0.11` (2026-04-10) + +## Upgrade Process + +1. **Update the pin**: `cargo update -p monty` +2. **Check for new features**: `cd ~/.cargo/git/checkouts/monty-*/*/` and `git log --oneline` since last pin +3. **Update the preamble**: If a previously-unsupported feature now works, remove it from the "Runtime environment" section in `prompts/codeact_preamble.md` +4. **Update this file**: Record the new pin and what changed +5. **Run tests**: `cargo test -p ironclaw_engine` +6. **Watch traces**: After deploying, check traces for new `NotImplementedError` patterns (self-improvement mission catches these) + +## Current Limitations (as of pin `v0.0.11`) + +These are documented in `prompts/codeact_preamble.md` so the LLM avoids them: + +### Syntax not supported +| Feature | Workaround | +|---------|-----------| +| `class Foo:` | Use functions and dicts (host-provided dataclasses work) | +| `with` statements | Use try/finally or direct calls | +| `match` statements | Use if/elif chains | +| `del` statement | Reassign to None | +| `yield` / `yield from` statements | Generator expressions (`x for x in ...`) work; use lists for the rest | +| Type aliases (`type X = ...`) | Omit type annotations | +| Template strings (t-strings) | Use f-strings | +| Complex number literals | Use floats | +| Exception groups (`try*/except*`) | Use regular try/except | + +### Limited standard library +`import csv`, `import io`, etc. still fail. + +`import os` succeeds but all operations (`os.getenv()`, `Path.*`) are **blocked** by the executor — `OSError: OS operations are not permitted in CodeAct scripts`. This is intentional: agents must use injected tools (`shell`, `read_file`, etc.) instead. + +Available built-in modules: +- `asyncio` — `asyncio.gather()` for parallel execution +- `datetime` — date and time handling +- `json` — JSON encoding/decoding +- `math` — standard math functions +- `os.path` — path string manipulation only (no I/O) +- `re` — regex (basic) +- `sys` — system info (limited) +- `typing` — type hints (limited, for annotation only) + +### Available builtins +`abs`, `all`, `any`, `bin`, `chr`, `divmod`, `enumerate`, `filter`, `getattr`, `hash`, `hex`, `id`, `isinstance`, `len`, `map`, `min`, `max`, `next`, `oct`, `ord`, `pow`, `print`, `repr`, `reversed`, `round`, `sorted`, `sum`, `type`, `zip` + +### Host-provided functions (always available) +These are injected by the IronClaw executor, not by Monty: +- `FINAL(answer)` / `FINAL_VAR(name)` — terminate with result +- `llm_query(prompt, context)` — recursive LLM sub-call +- `llm_query_batched(prompts)` — parallel sub-calls +- `rlm_query(prompt)` — full sub-agent with tools +- `globals()` / `locals()` — returns dict of known tool names +- All tool functions (web_search, http, time, etc.) + +## Upgrade Changelog + +| Date | Pin | Notable changes | +|------|-----|-----------------| +| 2026-04-10 | `v0.0.11` | JSON perf improvements (~2x loads, ~1.6x dumps), filesystem mounting, Rust-side async API, mount edge case fixes. | +| 2026-03-29 | `7a0d4b7` | Multi-module imports, `datetime` module, `json` module, nested subscript assignment, `str.expandtabs()`. | +| 2026-03-20 | `6053820` | Initial integration. max() kwargs support. | diff --git a/crates/ironclaw_engine/orchestrator/default.py b/crates/ironclaw_engine/orchestrator/default.py new file mode 100644 index 00000000000..df799ede618 --- /dev/null +++ b/crates/ironclaw_engine/orchestrator/default.py @@ -0,0 +1,902 @@ +# Engine v2 Orchestrator (default, v0) +# +# This is the self-modifiable execution loop. It replaces the Rust +# ExecutionLoop::run() with Python that can be patched at runtime +# by the self-improvement Mission. +# +# Host functions (provided by Rust via Monty suspension): +# __llm_complete__(messages, actions, config) -> response dict +# __execute_code_step__(code, state) -> result dict +# __execute_action__(name, params) -> result dict +# __execute_actions_parallel__(calls) -> list of result dicts (parallel execution) +# __check_signals__() -> None | "stop" | {"inject": msg} +# __emit_event__(kind, **data) -> None +# __save_checkpoint__(state, counters) -> None +# __transition_to__(state, reason) -> None +# __retrieve_docs__(goal, max_docs) -> list of doc dicts +# __check_budget__() -> budget dict +# __get_actions__() -> list of action dicts +# __list_skills__() -> list of skill dicts +# __record_skill_usage__(doc_id, success) -> None +# __regex_match__(pattern, text) -> bool +# +# Context variables (injected by Rust before execution): +# context - list of prior messages [{role, content}] +# goal - thread goal string +# actions - list of available action defs +# state - persisted state dict from prior steps +# config - thread config dict + + +# ── Helper functions (self-modifiable glue) ────────────────── +# Defined before run_loop so they are in scope when called. + + +def extract_final(text): + """Extract FINAL() content from text. Returns None if not found.""" + idx = text.find("FINAL(") + if idx < 0: + return None + after = text[idx + 6:] + # Handle triple-quoted strings + for q in ['"""', "'''"]: + if after.startswith(q): + end = after.find(q, len(q)) + if end >= 0: + return after[len(q):end] + # Handle single/double quoted strings + if after and after[0] in ('"', "'"): + quote = after[0] + end = after.find(quote, 1) + if end >= 0: + return after[1:end] + # Handle balanced parens + depth = 1 + for i, ch in enumerate(after): + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if depth == 0: + return after[:i] + return None + + +def strip_quoted_strings(line): + """Remove double-quoted string literals from a line.""" + result = [] + in_quote = False + prev = "" + for ch in line: + if ch == '"' and prev != "\\": + in_quote = not in_quote + prev = ch + continue + if not in_quote: + result.append(ch) + prev = ch + return "".join(result) + + +def strip_code_blocks(text): + """Strip fenced code blocks, indented code lines, and double-quoted strings.""" + result = [] + in_fence = False + for line in text.split("\n"): + trimmed = line.lstrip() + if trimmed.startswith("```"): + in_fence = not in_fence + continue + if in_fence: + continue + if line.startswith(" ") or line.startswith("\t"): + continue + result.append(strip_quoted_strings(line)) + return "\n".join(result) + + +def signals_tool_intent(text): + """Detect when text expresses intent to call a tool without actually doing so. + + Ported from V1 Rust llm_signals_tool_intent(): strips code blocks and + quoted strings, checks exclusion phrases, then requires a future-tense + prefix ("let me", "I'll", "I will", "I'm going to") immediately followed + by an action verb ("search", "fetch", "check", etc.). + """ + stripped = strip_code_blocks(text) + lower = stripped.lower() + + EXCLUSIONS = [ + "let me explain", "let me know", "let me think", + "let me summarize", "let me clarify", "let me describe", + "let me help", "let me understand", "let me break", + "let me outline", "let me walk you", "let me provide", + "let me suggest", "let me elaborate", "let me start by", + ] + for exc in EXCLUSIONS: + if exc in lower: + return False + + PREFIXES = ["let me ", "i'll ", "i will ", "i'm going to "] + ACTION_VERBS = [ + "search", "look up", "check", "fetch", "find", + "read the", "write the", "create", "run the", "execute", + "query", "retrieve", "add it", "add the", "add this", + "add that", "update the", "delete", "remove the", "look into", + ] + + for prefix in PREFIXES: + start = 0 + while True: + i = lower.find(prefix, start) + if i < 0: + break + after = lower[i + len(prefix):] + for verb in ACTION_VERBS: + if after.startswith(verb) or (" " + verb) in after.split("\n")[0]: + return True + start = i + 1 + + return False + + +def format_output(result, max_chars=8000): + """Format code execution result for the next LLM context message.""" + parts = [] + + stdout = result.get("stdout", "") + if stdout: + parts.append("[stdout]\n" + stdout) + + for r in result.get("action_results", []): + name = r.get("action_name", "?") + output = str(r.get("output", "")) + if r.get("is_error"): + parts.append("[" + name + " ERROR] " + output) + else: + preview = output[:500] + "..." if len(output) > 500 else output + parts.append("[" + name + "] " + preview) + + ret = result.get("return_value") + if ret is not None: + parts.append("[return] " + str(ret)) + + text = "\n\n".join(parts) + + # Truncate from the front (keep the tail with most recent results) + if len(text) > max_chars: + text = "... (truncated) ...\n" + text[-max_chars:] + + if not text: + text = "[code executed, no output]" + + return text + + +def format_docs(docs): + """Format memory docs for context injection.""" + parts = ["## Prior Knowledge (from completed threads)\n"] + for doc in docs: + label = doc.get("type", "NOTE").upper() + content = doc.get("content", "")[:500] + truncated = "..." if len(doc.get("content", "")) > 500 else "" + parts.append("### [" + label + "] " + doc.get("title", "") + + "\n" + content + truncated + "\n") + return "\n".join(parts) + + +# Conservative fallback heuristic matching the old Rust-side estimator. +# These MUST be defined before `estimate_context_tokens` (and therefore +# before the `FINAL(result)` entry-point call below). Moving them after the +# entry point is a latent NameError every time `compact_if_needed` runs. +CHARS_PER_TOKEN = 4 +MESSAGE_OVERHEAD_CHARS = 4 + + +def estimate_context_tokens(messages): + """Estimate token count for a transcript using a rough chars/token heuristic.""" + total_chars = 0 + for msg in messages: + total_chars += len(msg.get("content", "")) + total_chars += len(msg.get("action_name", "") or "") + total_chars += MESSAGE_OVERHEAD_CHARS + return (total_chars + CHARS_PER_TOKEN - 1) // CHARS_PER_TOKEN + + +def compact_if_needed(state, config): + """Compact thread context when the active message history grows too large. + + The orchestrator owns compaction policy. Rust only provides helpers for + token estimation, explicit LLM calls, and replacing the active message + scaffold after a summary has been produced. + """ + if not config.get("enable_compaction", False): + return False + + context_limit = config.get("model_context_limit", 128000) + threshold_pct = config.get("compaction_threshold", 0.85) + threshold = int(context_limit * threshold_pct) + working_messages = state.get("working_messages") + if not isinstance(working_messages, list) or not working_messages: + return False + + current_tokens = estimate_context_tokens(working_messages) + if current_tokens < threshold: + return False + + snapshot = list(working_messages) + + history = state.get("history") + if not isinstance(history, list): + history = [] + state["history"] = history + + compaction_count = state.get("compaction_count", 0) + 1 + history.append({ + "kind": "compaction", + "index": compaction_count, + "tokens_before": current_tokens, + "messages": snapshot, + }) + + summary_prompt = ( + "Summarize progress so far in a concise but complete way.\n" + "Include:\n" + "1. What has been accomplished\n" + "2. Key intermediate results, facts, and variable values\n" + "3. Tool results or findings worth preserving\n" + "4. What still needs to be done\n" + "5. Errors encountered and how they were handled\n\n" + "Preserve all information needed to continue the task." + ) + summary_messages = list(snapshot) + summary_messages.append({"role": "User", "content": summary_prompt}) + summary_resp = __llm_complete__(summary_messages, None, {"force_text": True}) + + summary_text = summary_resp.get("content", "") + if not summary_text: + summary_text = "[compaction produced no summary]" + + state["working_messages"] = [] + system_message = None + for msg in snapshot: + if msg.get("role") == "System": + system_message = {"role": "System", "content": msg.get("content", "")} + break + if system_message is not None: + state["working_messages"].append(system_message) + append_message(state["working_messages"], "Assistant", summary_text) + append_message( + state["working_messages"], + "User", + "Your conversation has been compacted. The summary above captures prior progress. " + "Older details remain available through state['history'] and project retrieval. Continue working on the task.", + ) + state["compaction_count"] = compaction_count + return True + + +# ── Skill selection and injection (self-modifiable) ──────── + + +def score_skill(skill, message_lower, message_original): + """Score a skill against a user message. Returns 0 if vetoed. + + Scoring is aligned with the v1 `ironclaw_skills::selector::score_skill`: + - exclude_keyword veto: any match => score 0 + - keyword: exact word = 10, substring = 5 (cap 30) + - tag: substring = 3 (cap 15) + - regex pattern: each match = 20 (cap 40) + """ + meta = skill.get("metadata", {}) + activation = meta.get("activation", {}) + + # Exclude keyword veto + for excl in activation.get("exclude_keywords", []): + if excl.lower() in message_lower: + return 0 + + score = 0 + + # Keyword scoring: exact word = 10, substring = 5 (cap 30) + kw_score = 0 + words = [] + for word in message_lower.split(): + trimmed = word.strip(".,!?;:'\"()[]{}<>`~@#$%^&*-_=+/\\|") + if trimmed: + words.append(trimmed) + for kw in activation.get("keywords", []): + kw_lower = kw.lower() + if kw_lower in words: + kw_score += 10 + elif kw_lower in message_lower: + kw_score += 5 + score += min(kw_score, 30) + + # Tag scoring: substring = 3 (cap 15) + tag_score = 0 + for tag in activation.get("tags", []): + if tag.lower() in message_lower: + tag_score += 3 + score += min(tag_score, 15) + + # Regex pattern scoring: each match = 20 (cap 40). Monty has no `re` + # module, so we call out to a host function that uses Rust's regex crate. + rx_score = 0 + for pat in activation.get("patterns", []): + if __regex_match__(str(pat), message_original): + rx_score += 20 + score += min(rx_score, 40) + + # Confidence factor for extracted skills + source = meta.get("source", "authored") + if source == "extracted": + metrics = meta.get("metrics", {}) + total = metrics.get("success_count", 0) + metrics.get("failure_count", 0) + confidence = metrics.get("success_count", 0) / total if total > 0 else 1.0 + factor = 0.5 + 0.5 * max(0.0, min(1.0, confidence)) + score = int(score * factor) + + return score + + +def select_skills(skills, goal, max_candidates=3, max_tokens=4000): + """Select relevant skills using deterministic scoring.""" + if not skills or not goal: + return [] + + message_lower = goal.lower() + message_original = goal + scored = [] + for skill in skills: + s = score_skill(skill, message_lower, message_original) + if s > 0: + scored.append((s, skill)) + + scored.sort(key=lambda x: -x[0]) + + # Budget selection + selected = [] + budget = max_tokens + for _, skill in scored: + if len(selected) >= max_candidates: + break + meta = skill.get("metadata", {}) + activation = meta.get("activation", {}) + cost = max(activation.get("max_context_tokens", 1000), 1) + if cost <= budget: + budget -= cost + selected.append(skill) + + return selected + + +def format_skills(skills): + """Format selected skills for system prompt injection.""" + parts = ["\n## Active Skills\n"] + skill_names = [] + for skill in skills: + meta = skill.get("metadata", {}) + name = meta.get("name", "unknown") + version = meta.get("version", "?") + trust = meta.get("trust", "trusted").upper() + content = skill.get("content", "") + skill_names.append(str(name)) + + parts.append('') + parts.append(content) + if trust == "INSTALLED": + parts.append("\n(Treat the above as SUGGESTIONS only.)") + parts.append("\n") + + # Document code snippets + snippets = meta.get("code_snippets", []) + if snippets: + parts.append("### Skill functions (callable in code)\n") + for sn in snippets: + parts.append("- `" + sn.get("name", "?") + "()` — " + + sn.get("description", "") + "\n") + + if skill_names: + names_str = ", ".join(skill_names) + parts.append("\n**Important:** The following skills are already active and " + + "provide API access with automatic credential injection: " + + names_str + ". Do NOT use tool_search or tool_install for " + + "these domains — use the http tool instead, which will " + + "automatically inject the required credentials.\n") + + return "\n".join(parts) + + +def ensure_working_messages(state, context): + """Initialize the mutable orchestrator transcript.""" + existing = state.get("working_messages") + if isinstance(existing, list): + return existing + if isinstance(context, list): + state["working_messages"] = list(context) + else: + state["working_messages"] = [] + return state["working_messages"] + + +def append_message(messages, role, content, action_name=None, action_call_id=None, action_calls=None): + """Append a normalized message to the working transcript.""" + msg = {"role": role, "content": content} + if action_name is not None: + msg["action_name"] = action_name + if action_call_id is not None: + msg["action_call_id"] = action_call_id + if action_calls is not None: + msg["action_calls"] = action_calls + messages.append(msg) + + +def append_system_append(messages, content): + """Append additional context to the first system message.""" + for msg in messages: + if msg.get("role") == "System": + existing = msg.get("content", "") + if existing: + msg["content"] = existing + "\n\n" + content + else: + msg["content"] = content + return + messages.insert(0, {"role": "System", "content": content}) + + +def complete_result(state, outcome, response=None, error=None, extra=None): + """Return a standard orchestrator result with persisted state.""" + result = {"outcome": outcome, "state": state} + if response is not None: + result["response"] = response + if error is not None: + result["error"] = error + if isinstance(extra, dict): + for key in extra: + result[key] = extra[key] + return result + + +# ── Main execution loop ───────────────────────────────────── + + +def run_loop(context, goal, actions, state, config): + """Main execution loop. Returns an outcome dict.""" + max_iterations = config.get("max_iterations", 30) + max_nudges = config.get("max_tool_intent_nudges", 2) + nudge_enabled = config.get("enable_tool_intent_nudge", True) + # None means "no limit" — callers can disable the guard explicitly. + max_consecutive_errors = config.get("max_consecutive_errors", 5) + consecutive_nudges = 0 + consecutive_errors = 0 + consecutive_action_errors = 0 + step_count = config.get("step_count", 0) + if not isinstance(state, dict): + state = {} + state.setdefault("history", []) + state.setdefault("compaction_count", 0) + working_messages = ensure_working_messages(state, context) + + for step in range(step_count, max_iterations): + # 1. Check signals + signal = __check_signals__() + if signal == "stop": + __transition_to__("completed", "stopped by signal") + return complete_result(state, "stopped") + if signal and isinstance(signal, dict) and "inject" in signal: + append_message(working_messages, "User", signal["inject"]) + + # 2. Check budget + budget = __check_budget__() + if budget.get("tokens_remaining", 1) <= 0: + __transition_to__("completed", "token budget exhausted") + return complete_result(state, "completed", "Token budget exhausted.") + if budget.get("time_remaining_ms", 1) <= 0: + __transition_to__("completed", "time budget exhausted") + return complete_result(state, "completed", "Time budget exhausted.") + if budget.get("usd_remaining") is not None and budget["usd_remaining"] <= 0: + __transition_to__("completed", "cost budget exhausted") + return complete_result(state, "completed", "Cost budget exhausted.") + + # 3. Inject prior knowledge and activate skills on first step + if step == 0: + docs = __retrieve_docs__(goal, 5) + if docs: + knowledge = format_docs(docs) + append_system_append(working_messages, knowledge) + + # Select and inject skills based on goal keywords + all_skills = __list_skills__() + active_skills = select_skills(all_skills, goal, max_candidates=3, max_tokens=4000) + if active_skills: + __set_active_skills__([ + { + "doc_id": s.get("doc_id", ""), + "name": s.get("metadata", {}).get("name", "?"), + "version": s.get("metadata", {}).get("version", 1), + "snippet_names": [ + sn.get("name", "") + for sn in s.get("metadata", {}).get("code_snippets", []) + if sn.get("name") + ], + "force_activated": False, + } + for s in active_skills + ]) + skill_text = format_skills(active_skills) + append_system_append(working_messages, skill_text) + # Emit skill activation event for CLI/gateway display + skill_names = ",".join(s.get("metadata", {}).get("name", "?") for s in active_skills) + __emit_event__("skill_activated", skill_names=skill_names) + # Store active skill IDs in state for tracking + state["active_skill_ids"] = [s.get("doc_id", "") for s in active_skills] + state["skill_snippet_names"] = [] + for s in active_skills: + for sn in s.get("metadata", {}).get("code_snippets", []): + state["skill_snippet_names"].append(sn.get("name", "")) + + # 3.5 Compact context before the next model call when needed. + compact_if_needed(state, config) + working_messages = ensure_working_messages(state, context) + + # 4. Call LLM + __emit_event__("step_started", step=step) + response = __llm_complete__(working_messages, actions, None) + __emit_event__("step_completed", step=step, + input_tokens=response.get("usage", {}).get("input_tokens", 0), + output_tokens=response.get("usage", {}).get("output_tokens", 0)) + + # 5. Handle response based on type + resp_type = response.get("type", "text") + + if resp_type == "text": + text = response.get("content", "") + append_message(working_messages, "Assistant", text) + + # Check for FINAL() + final_answer = extract_final(text) + if final_answer is not None: + __transition_to__("completed", "FINAL() in text") + return complete_result(state, "completed", final_answer) + + # Check for tool intent nudge (V1 semantics: consecutive counter, + # only resets on non-intent text, NOT on action/code responses) + if nudge_enabled and consecutive_nudges < max_nudges and signals_tool_intent(text): + consecutive_nudges += 1 + append_message( + working_messages, + "User", + "You said you would perform an action, but you did not include any tool calls.\n" + "Do NOT describe what you intend to do — actually call the tool now.\n" + "Use the tool_calls mechanism to invoke the appropriate tool.", + ) + continue + + # Non-intent text response — reset nudge counter and finish + if not signals_tool_intent(text): + consecutive_nudges = 0 + + # Plain text response - done + __transition_to__("completed", "text response") + return complete_result(state, "completed", text) + + elif resp_type == "code": + code = response.get("code", "") + append_message(working_messages, "Assistant", "```repl\n" + code + "\n```") + + # Execute code in nested Monty VM + result = __execute_code_step__(code, state) + + # Update persisted state with results + if result.get("return_value") is not None: + state["step_" + str(step) + "_return"] = result["return_value"] + state["last_return"] = result["return_value"] + for r in result.get("action_results", []): + state[r.get("action_name", "unknown")] = r.get("output") + + # Format output for next LLM context + output = format_output(result) + append_message(working_messages, "User", output) + + # Check for FINAL() in code output + if result.get("final_answer") is not None: + __transition_to__("completed", "FINAL() in code") + return complete_result(state, "completed", result["final_answer"]) + + # Check for unified gate pause (new path) + gate = result.get("pending_gate") + if gate is None: + gate = result.get("need_approval") + if gate is not None and isinstance(gate, dict) and gate.get("gate_paused"): + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + __transition_to__("waiting", "gate paused: " + gate.get("gate_name", "unknown")) + return { + "outcome": "gate_paused", + "state": state, + "gate_name": gate.get("gate_name", ""), + "action_name": gate.get("action_name", ""), + "call_id": gate.get("call_id", ""), + "parameters": gate.get("parameters", {}), + "resume_kind": gate.get("resume_kind", {}), + } + + # Check for approval or authentication needed (legacy path) + if result.get("need_approval") is not None: + approval = result["need_approval"] + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + if approval.get("need_authentication"): + __transition_to__("waiting", "authentication needed") + return { + "outcome": "need_authentication", + "state": state, + "credential_name": approval.get("credential_name", ""), + "action_name": approval.get("action_name", ""), + "call_id": approval.get("call_id", ""), + "parameters": approval.get("parameters", {}), + } + __transition_to__("waiting", "approval needed") + return { + "outcome": "need_approval", + "state": state, + "action_name": approval.get("action_name", ""), + "call_id": approval.get("call_id", ""), + "parameters": approval.get("parameters", {}), + } + + # Track consecutive errors + if result.get("had_error"): + consecutive_errors += 1 + if max_consecutive_errors is not None and consecutive_errors >= max_consecutive_errors: + __transition_to__("failed", "too many consecutive errors") + return complete_result( + state, + "failed", + error=str(max_consecutive_errors) + " consecutive code errors", + ) + else: + consecutive_errors = 0 + + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + + elif resp_type == "actions": + # Tier 0: structured tool calls. + # NOTE: consecutive_nudges is NOT reset here (V1 semantics). + # Only non-intent text responses reset the counter. + calls = response.get("calls", []) + + # Handle FINAL emitted as a structured tool call. FINAL is a + # CodeAct sentinel for completion — when the LLM tries to call + # it via tool_calls instead of inside a code block, the engine's + # action executor has no lease for it and the call fails. If FINAL + # is co-emitted with other calls, execute the non-FINAL calls first + # so persistence side effects are not silently dropped. + final_call = None + duplicate_finals_dropped = 0 + executable_calls = [] + for c in calls: + if c.get("name", "") == "FINAL": + # First FINAL wins; any extras are dropped (not appended + # to executable_calls) so they don't try to run as a + # normal action and fail with a lease error. + if final_call is None: + final_call = c + else: + duplicate_finals_dropped += 1 + continue + executable_calls.append(c) + + if duplicate_finals_dropped > 0: + # Surface the drop so traces show why fewer FINALs were + # executed than the LLM emitted. + __emit_event__( + "duplicate_final_dropped", + count=duplicate_finals_dropped, + ) + + # Append the assistant message with only the executable calls. + # FINAL is filtered out of `action_calls` so the message history + # does not record a FINAL action with no matching ActionResult, + # which would confuse context replay on resume. + append_message( + working_messages, + "Assistant", + response.get("content", "") or "", + action_calls=executable_calls, + ) + + # Execute all tool calls in parallel via the batch host function. + # Rust handles preflight (lease/policy), parallel execution via + # JoinSet, and event emission in call order. + results = __execute_actions_parallel__(executable_calls) + # Every tool call in the assistant message MUST have a matching + # ActionResult, otherwise the LLM API rejects the sequence with + # "No tool output found for function call ". Iterate over + # executable_calls (not results) so we cover calls that the Rust + # batch handler skipped (e.g. RequireApproval early return). + batch_error_count = 0 + batch_success_count = 0 + for idx in range(len(executable_calls)): + call = executable_calls[idx] + call_id = call.get("call_id", "") + r = results[idx] if idx < len(results) else None + if r is not None: + action_name = r.get("action_name", call.get("name", "")) + output = r.get("output") + output_str = str(output) if output is not None else "[no output]" + if r.get("is_error"): + output_str = "[ACTION FAILED] " + output_str + batch_error_count += 1 + else: + batch_success_count += 1 + else: + action_name = call.get("name", "unknown") + output_str = "[execution skipped]" + batch_error_count += 1 + append_message( + working_messages, + "ActionResult", + output_str, + action_name=action_name, + action_call_id=call_id, + ) + + # Check results for auth/approval interrupts + for r_idx, r in enumerate(results): + if r is None: + continue + + if r.get("gate_paused"): + # Unified gate pause (replaces separate need_approval/need_authentication) + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + gate = r + # Get action info from the original call or the result + orig_call = executable_calls[r_idx] if r_idx < len(executable_calls) else {} + __transition_to__("waiting", "gate paused: " + gate.get("gate_name", "unknown")) + return { + "outcome": "gate_paused", + "state": state, + "gate_name": gate.get("gate_name", ""), + "action_name": gate.get("action_name", orig_call.get("name", "")), + "call_id": orig_call.get("call_id", ""), + "parameters": orig_call.get("params", {}), + "resume_kind": gate.get("resume_kind", {}), + } + + if r.get("need_authentication"): + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + __transition_to__("waiting", "authentication needed") + return { + "outcome": "need_authentication", + "state": state, + "credential_name": r.get("credential_name", ""), + "action_name": r.get("action_name", ""), + "call_id": r.get("call_id", ""), + "parameters": r.get("parameters", {}), + } + + if r.get("need_approval"): + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + __transition_to__("waiting", "approval needed") + return { + "outcome": "need_approval", + "state": state, + "action_name": r.get("action_name", ""), + "call_id": r.get("call_id", ""), + "parameters": r.get("parameters", {}), + } + + if final_call is not None: + raw_params = final_call.get("params", {}) + # Some LLMs pass FINAL with the answer as a positional string + # argument instead of a named param dict. Handle that case so + # the answer is not silently dropped. + if isinstance(raw_params, str): + answer = raw_params + else: + params = raw_params or {} + answer = ( + params.get("answer") + or params.get("result") + or params.get("value") + or params.get("content") + or params.get("text") + ) + if not answer: + # Fall back to the assistant's content text. This may + # contain the model's full explanation rather than the + # intended terse answer — truncate aggressively so we + # don't ship thousands of tokens of reasoning as the + # final answer, and emit a trace event so the + # ambiguity is visible. + fallback_content = response.get("content", "") or "" + FINAL_FALLBACK_MAX_CHARS = 500 + truncated = False + if len(fallback_content) > FINAL_FALLBACK_MAX_CHARS: + fallback_content = ( + fallback_content[:FINAL_FALLBACK_MAX_CHARS] + + "… [truncated by orchestrator: FINAL was emitted with no recognizable answer param]" + ) + truncated = True + answer = fallback_content + __emit_event__( + "final_fallback", + reason="no recognizable answer param on FINAL", + truncated=truncated, + original_length=len(response.get("content", "") or ""), + ) + __transition_to__("completed", "FINAL via tool_calls") + return complete_result(state, "completed", str(answer)) + + # Track consecutive action errors (separate from code errors). + # Partial batch failures: increment only if ALL actions failed, + # reset if ANY succeeded. + if batch_success_count > 0: + consecutive_action_errors = 0 + elif batch_error_count > 0: + consecutive_action_errors += 1 + + if max_consecutive_errors is not None and consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + __transition_to__("failed", "too many consecutive action errors") + return complete_result( + state, + "failed", + error=str(consecutive_action_errors) + " consecutive action errors — all recent tool calls failed", + ) + elif max_consecutive_errors is not None and consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors: + append_message( + working_messages, + "User", + "[SYSTEM] Your last " + str(consecutive_action_errors) + + " action calls have all failed. You appear to be stuck in a loop. " + "Try a completely different approach: use different tools, different " + "parameters, or break the problem down differently. If you cannot " + "make progress, call FINAL() with an honest explanation of what failed.", + ) + + __save_checkpoint__(state, { + "nudge_count": consecutive_nudges, + "consecutive_errors": consecutive_errors, + "consecutive_action_errors": consecutive_action_errors, + "compaction_count": state.get("compaction_count", 0), + }) + + # Max iterations reached + __transition_to__("completed", "max iterations reached") + return complete_result(state, "max_iterations") + + +# Entry point: call run_loop with injected context variables +result = run_loop(context, goal, actions, state, config) +FINAL(result) diff --git a/crates/ironclaw_engine/prompts/codeact_postamble.md b/crates/ironclaw_engine/prompts/codeact_postamble.md new file mode 100644 index 00000000000..ea969e993ad --- /dev/null +++ b/crates/ironclaw_engine/prompts/codeact_postamble.md @@ -0,0 +1,18 @@ + +## Strategy + +1. First, examine the context and understand the task +2. Break complex tasks into steps +3. Use tools to gather information or take actions +4. Use llm_query() to analyze or summarize large text +5. Call FINAL() with the answer when done + +Think step by step. Execute code immediately — don't just describe what you would do. + +## Error recovery + +When a tool call fails, do NOT give up immediately. Try alternative approaches before calling FINAL(): +- If `http()` fails with an auth error, try `web_search()` or a different public endpoint +- If one API endpoint fails, try a different one that provides similar data +- If a search returns no results, try different keywords or broader queries +- Only call FINAL() to report failure after exhausting at least 2-3 alternative approaches diff --git a/crates/ironclaw_engine/prompts/codeact_preamble.md b/crates/ironclaw_engine/prompts/codeact_preamble.md new file mode 100644 index 00000000000..ff995a2a370 --- /dev/null +++ b/crates/ironclaw_engine/prompts/codeact_preamble.md @@ -0,0 +1,76 @@ +You are an AI assistant with a Python REPL environment. You solve tasks by writing and executing Python code. + +## How to respond + +Write Python code inside ```repl fenced blocks. The code will be executed, and you'll see the output. All tool calls are async — use `await` to get results. + +```repl +result = await web_search(query="latest AI news", count=5) +print(result) +``` + +You can write multiple code blocks across turns. Variables persist between blocks within the same turn. + +## Parallel execution with asyncio.gather + +When you need results from multiple independent tools, use `asyncio.gather()` to run them concurrently: + +```repl +import asyncio +search, page, memories = await asyncio.gather( + web_search(query="rust async patterns"), + http(url="https://example.com/api"), + memory_search(query="prior work"), +) +print(search, page, memories) +``` + +This is much faster than calling tools sequentially. Use `asyncio.gather()` whenever tools don't depend on each other's results. + +## Special functions + +- `llm_query(prompt, context=None, model=None)` — Ask a sub-agent to analyze text or answer a question. Returns a string. Use for summarization, analysis, or any task that needs LLM reasoning on data. Optional `model="..."` overrides which LLM answers this single call (e.g. `model="gpt-4o"`). +- `llm_query_batched(prompts, context=None, model=None, models=None)` — Same but for multiple prompts in parallel. Returns a list of strings. Pass `model="gpt-4o"` to apply one model to every prompt, or `models=["gpt-4o", "claude-sonnet-4-20250514", ...]` (parallel array, must match `prompts` length) to send each prompt to a different model. The "LLM council" pattern is `prompts=[same_question]*N, models=[m1, m2, ...]`. +- `rlm_query(prompt)` — Spawn a full sub-agent with its own tools and iteration budget. Use for complex sub-tasks that need tool access. Returns the sub-agent's final answer as a string. More powerful but more expensive than llm_query. +- `FINAL(answer)` — Call this when you have the final answer. The argument is returned to the user. +- `mission_create(name, goal, cadence="manual", success_criteria=None)` — Create a long-running mission that spawns threads over time. Cadence: "manual", cron expression (e.g. "0 9 * * *"), "event:pattern", or "webhook:path". Cron expressions accept 5-field (`min hr dom mon dow`), 6-field (`sec min hr dom mon dow` — NOT Quartz-style with year), or 7-field (`sec min hr dom mon dow year`). Cron missions default to the user's timezone from `user_timezone`; pass an explicit `timezone` param to override. Returns {"mission_id": "...", "name": "...", "status": "created"}. When telling the user about a created mission, refer to it by `name`, not by `mission_id` (the UUID is internal). +- `mission_list()` — List all missions with their status, goal, and current focus. +- `mission_fire(id)` — Manually trigger a mission to spawn a thread now. +- `mission_pause(id)` / `mission_resume(id)` — Pause or resume a mission. + +## Context variables + +- `context` — List of prior conversation messages (each is a dict with 'role' and 'content') +- `goal` — The current task description +- `step_number` — Current execution step +- `state` — Dict of persisted data from previous steps. Contains tool results keyed by tool name (e.g. `state['web_search']`) and return values (`state['last_return']`, `state['step_0_return']`). Use this to access data from previous steps without re-calling tools. +- `previous_results` — Dict of prior tool call results (from ActionResult messages) +- `user_timezone` — The user's IANA timezone (e.g. "America/New_York", "Europe/London"). Defaults to "UTC". Use this for time-aware operations, scheduling, and cron timezone parameters. + +## Important rules + +1. ALWAYS respond with a ```repl code block. NEVER answer with plain text only. Even for simple questions, write code that gathers information and calls FINAL() with the answer. +2. NEVER answer from memory or training data alone. Always use tools (web_search, llm_context, shell, read_file, etc.) to get real, current information before answering. +3. When you have the final answer, call `FINAL(answer)` inside a code block. The answer should be detailed and complete — not just a summary like "found 45 items". +4. All tool calls are async — always use `await` (e.g. `result = await web_search(...)`). For parallel calls, use `asyncio.gather()`. +5. Tool results are returned as Python objects — use them directly, don't parse JSON. +6. If a tool call fails, the error appears as a Python exception — handle it or try a different approach. +7. For large data, process it in chunks using llm_query() on subsets rather than loading everything into context. +8. Outputs are truncated to 8000 chars — use variables to store large intermediate results. +9. Include the actual content in your FINAL() answer, not just a count or summary. Users want to see the details. + +## Runtime environment + +The Python REPL runs in Monty, a lightweight embedded interpreter — not CPython. Key differences: + +- **Async tools**: All tool calls return futures. Use `await tool(...)` for sequential or `asyncio.gather(tool1(...), tool2(...))` for parallel. Top-level `await` is supported (no need for `asyncio.run()`). +- **Limited standard library**: `import csv`, `import io` etc. will fail with `ModuleNotFoundError`. `import os` loads but all operations raise `OSError` — use the provided tool functions for OS operations (`shell()`, `read_file()`). +- **No classes**: `class Foo:` is not supported. Use functions and dicts instead (host-provided dataclasses work). +- **No `with` statements**: Use try/finally or just call functions directly. +- **No `match` statements**: Use if/elif chains. +- **No `del` statement**: Reassign to None instead. +- **No `yield`/`yield from` statements**: Generator expressions (`x for x in ...`) work; use lists for the rest. +- **Available builtins**: `abs`, `all`, `any`, `bin`, `chr`, `divmod`, `enumerate`, `filter`, `getattr`, `hash`, `hex`, `id`, `isinstance`, `len`, `map`, `min`, `max`, `next`, `oct`, `ord`, `pow`, `print`, `repr`, `reversed`, `round`, `sorted`, `sum`, `type`, `zip`. +- **Available modules**: `asyncio`, `datetime`, `json`, `math`, `os.path` (path manipulation only), `re`, `sys`, `typing` (limited). +- **String methods, list methods, dict methods**: All work normally. +- For dates, use `import datetime`. For JSON, use `import json` or work with dicts directly (tool results are already Python objects). For CSV parsing, split strings manually. For HTTP, use `await http()`. diff --git a/crates/ironclaw_engine/prompts/mission_conversation_insights.md b/crates/ironclaw_engine/prompts/mission_conversation_insights.md new file mode 100644 index 00000000000..ee5018f9818 --- /dev/null +++ b/crates/ironclaw_engine/prompts/mission_conversation_insights.md @@ -0,0 +1,38 @@ +You extract user preferences, patterns, and domain knowledge from a batch of recent conversation threads. + +## Input + +`state["trigger_payload"]` contains: +- `project_id` — the project scope +- `completed_thread_count` — total threads completed in this conversation +- `thread_goals` — list of recent thread goals (what the user asked for) +- `sample_user_messages` — sample of actual user messages (truncated to 200 chars) + +## Process + +1. Analyze the thread goals and user messages for patterns +2. Search existing insights: `memory_search(query="user preferences")` and `memory_search(query="domain knowledge")` +3. Extract NEW insights not already recorded in memory +4. Write each insight to memory via `memory_write(target="memory", content=insight_text)` with title format "insight::" + +## Categories to look for + +- **Preferences**: communication style, format choices, tool preferences +- **Domain**: project names, API patterns, data formats, technology stack +- **Workflow**: recurring task sequences, common follow-up questions +- **Corrections**: things the user corrected or repeated — these signal unmet expectations + +## Output (FINAL) + +Report: +- Number of new insights extracted (0 is fine) +- Brief list of what was found +- Next focus + +## Rules + +- Only record actionable, specific insights — not vague observations +- Do not record personal information, only work patterns +- If no meaningful new insights after analysis, call FINAL("No new insights — conversation patterns already captured") immediately +- Merge with existing insight docs rather than creating duplicates +- Max 5 insights per run to keep quality high diff --git a/crates/ironclaw_engine/prompts/mission_expected_behavior.md b/crates/ironclaw_engine/prompts/mission_expected_behavior.md new file mode 100644 index 00000000000..875b52dc7fb --- /dev/null +++ b/crates/ironclaw_engine/prompts/mission_expected_behavior.md @@ -0,0 +1,58 @@ +You investigate why IronClaw did not behave as the user expected. The user used the `/expected` command to describe what should have happened, and the trigger payload includes the recent conversation turns showing what actually happened. + +## Input + +`state["trigger_payload"]` contains: +- `expected_behavior` — what the user expected to happen (their description) +- `thread_id` — the conversation thread where the issue occurred +- `recent_turns` — list of recent turns, each with: + - `user_input` — what the user asked + - `response` — what the agent responded + - `tool_calls` — list of tools called (with name and any errors) + - `state` — turn completion state + - `error` — any error message + +## Investigation process + +1. **Understand the gap**: Compare `expected_behavior` against `recent_turns`. What did the user want? What actually happened? Be precise about the delta. + +2. **Classify the root cause**: + - MISSING_CAPABILITY: The agent doesn't have the tool or integration needed (e.g. no GitHub OAuth, no API key configured) + - WRONG_TOOL_CHOICE: The agent had the right tools but chose the wrong one or didn't use them at all + - PROMPT_GAP: The agent didn't know the right approach because the system prompt lacks guidance for this scenario + - CONFIG_ISSUE: A timeout, limit, or default prevented success + - BUG: Actual code error in tool execution or response processing + +3. **Apply a fix** based on classification: + + MISSING_CAPABILITY: + - Search for relevant skills: `skill_search(query="...")` or `tool_search(query="...")` + - If a skill/tool exists but isn't installed, note it as a recommendation + - If nothing exists, add a prompt rule acknowledging the limitation and suggesting alternatives the user can take + + WRONG_TOOL_CHOICE or PROMPT_GAP: + - Apply a Level 1 (prompt overlay) fix — add a rule that guides the agent in this scenario + - Use `memory_write` with title="prompt:codeact_preamble" and tags=["prompt_overlay"] + - The rule must be specific and actionable + + CONFIG_ISSUE: + - Diagnose via `read_file` and `shell` commands + - Apply Level 2 fix if safe (branch, change, test, commit) + + BUG: + - Read relevant source files to understand the issue + - Propose a Level 3 fix (describe but don't apply) + +4. **Record** in FINAL(): + - What the user expected vs what happened (one sentence each) + - Root cause classification + - What fix was applied (or recommended) + - Next focus + +## Rules + +- The user's expectation is the ground truth — don't argue with it +- If multiple issues exist, fix the most impactful one first +- Be specific in prompt rules ("When asked to file a GitHub issue, use the http tool with the GitHub API" is good; "Try harder" is useless) +- If the gap is a missing credential or integration, say so clearly — don't pretend the capability exists +- Max one fix per run diff --git a/crates/ironclaw_engine/prompts/mission_self_improvement.md b/crates/ironclaw_engine/prompts/mission_self_improvement.md new file mode 100644 index 00000000000..e3d02dfc634 --- /dev/null +++ b/crates/ironclaw_engine/prompts/mission_self_improvement.md @@ -0,0 +1,67 @@ +You are a self-improvement agent for the IronClaw engine. You receive trigger payloads containing execution trace issues from completed threads. Your job is to diagnose root causes and apply fixes so the same issue doesn't recur. + +## What you have access to + +- `state["trigger_payload"]` — JSON with `issues` (list of {severity, category, description, step}), `error_messages` (actual error text from failed actions), `goal` (what the thread was trying to do), and `source_thread_id`. +- All tools: shell, read_file, write_file, apply_patch, web_search, memory_write, etc. +- The codebase at the current working directory. +- The fix pattern database in prior knowledge (if loaded). + +## The experiment loop + +For each issue in the trigger payload: + +1. **Diagnose**: Read the error messages and issue descriptions. Classify the root cause: + - PROMPT: The LLM made a mistake because the system prompt is missing a rule (wrong tool name, bad API usage, ignoring tool results) + - CONFIG: A default value is wrong (truncation length, iteration limit, timeout) + - CODE: There is a bug in the engine or bridge code (crash, type error, missing conversion) + +2. **Check the fix pattern database** in prior knowledge. Has this pattern been seen before? If yes, apply the known strategy. If no, proceed to step 3. + +3. **Apply the fix** based on the level: + + Level 1 (PROMPT — low risk, apply directly): + - Read the current prompt overlay: `memory_search("prompt:codeact_preamble")` + - Write an updated overlay with a new rule appended + - Use `memory_write` with title="prompt:codeact_preamble" and tags=["prompt_overlay"] + - The rule should be specific and actionable (e.g. "Never call web_fetch — use http() instead") + + Level 2 (CONFIG — medium risk): + - Use `read_file` to find the relevant constant or default + - Use `shell` to create a git branch: `git checkout -b self-improve/issue-description` + - Apply the change with `apply_patch` or `write_file` + - Run tests: `cargo test -p ironclaw_engine` + - If tests pass, commit. If not, revert: `git checkout main` + + Level 3 (CODE — high risk, just propose): + - Read the relevant source files + - Describe the fix needed but DO NOT apply it directly + - Log it as a recommendation in your FINAL() response + +4. **Record what you did** — include in your FINAL() response: + - What issue you analyzed + - What level fix you applied (1/2/3) + - What specific change you made + - Next focus: what to look for next time + +## Important rules + +- Be specific. "Never call web_fetch" is good. "Be careful with tool names" is useless. +- One fix per issue. Don't try to fix everything at once. +- For Level 1 fixes, the rule must be one sentence that can be appended to the prompt. +- If the trigger payload has no actionable issues (only Info severity), skip and call FINAL() immediately. +- NEVER modify test files to make a fix pass. +- NEVER modify security-sensitive code (safety layer, policy engine, leak detection). +- If you can't diagnose the root cause after reading the errors, log it and move on. + +## Level 1.5: Orchestrator patches (medium risk, auto-rollback) + +The execution loop itself is Python code that you can modify. This is the orchestrator — it handles tool dispatch, output formatting, state management, and context building. If the bug is in the glue between the LLM and tools (wrong output format, bad truncation, missing state), you can patch it directly. + +To modify the orchestrator: +1. Read current version: `memory_search("orchestrator:main")` +2. Make your change (keep it minimal — one fix at a time) +3. Save the new version: `memory_write` with title="orchestrator:main", tags=["orchestrator_code"], metadata={"version": N+1, "parent_version": N} +4. The next thread will use your updated orchestrator + +If your change causes 3 consecutive failures, the system auto-rolls back to the previous version. So be conservative — test your logic mentally before saving. diff --git a/crates/ironclaw_engine/prompts/mission_skill_extraction.md b/crates/ironclaw_engine/prompts/mission_skill_extraction.md new file mode 100644 index 00000000000..a2a303355de --- /dev/null +++ b/crates/ironclaw_engine/prompts/mission_skill_extraction.md @@ -0,0 +1,69 @@ +You extract reusable skills from successfully completed multi-step threads. + +## Input + +`state["trigger_payload"]` contains: +- `source_thread_id` — the thread that completed successfully +- `goal` — what the thread accomplished +- `step_count` — number of execution steps +- `action_count` — number of tool actions executed +- `actions_used` — list of tool names used +- `total_tokens` — tokens consumed + +## Output Format + +Save as a Skill memory doc via `memory_write(target="memory", content=skill_prompt)` with: +- title: `"skill:"` (e.g., "skill:github-issue-triage") +- doc_type: `"skill"` +- metadata JSON: + ```json + { + "name": "", + "version": 1, + "description": "", + "activation": { + "keywords": ["", ""], + "patterns": [""], + "tags": [""], + "exclude_keywords": [], + "max_context_tokens": + }, + "source": "extracted", + "trust": "trusted", + "code_snippets": [ + { + "name": "", + "code": "def (...):\n ...", + "description": "" + } + ], + "metrics": {"usage_count": 0, "success_count": 0, "failure_count": 0}, + "content_hash": "" + } + ``` + +## Process + +1. Search for the source thread's context: `memory_search(query=goal)` +2. Check for existing skills: `memory_search(query="skill:")` +3. If a similar skill exists, update it (increment version) rather than creating a duplicate +4. Extract: + - Activation keywords from the goal + user messages (be specific, not generic) + - Step-by-step instructions as the prompt content + - Python code snippets for CodeAct (reusable functions using exact tool names) + - Domain tags (e.g., "github", "api", "data") + +## Output (FINAL) + +Report what you did: +- The skill title and a one-line summary +- Whether it is new or an update to an existing skill +- Next focus: what patterns to watch for + +## Rules + +- Only extract skills from threads with 3+ distinct tool calls +- Keywords must be specific (not generic words like "help", "do", "make") +- Code snippets must use exact tool function names as they appear in the thread +- If the thread was a trivial query-response, call FINAL("No skill needed — simple interaction") and stop immediately +- One skill per FINAL — do not combine unrelated procedures diff --git a/crates/ironclaw_engine/prompts/mission_skill_repair.md b/crates/ironclaw_engine/prompts/mission_skill_repair.md new file mode 100644 index 00000000000..122f58f5689 --- /dev/null +++ b/crates/ironclaw_engine/prompts/mission_skill_repair.md @@ -0,0 +1,70 @@ +You are the skill-repair learning mission for the IronClaw engine. You receive trigger payloads from completed threads where an active skill was relevant, but execution suggests the skill instructions were incomplete, stale, incorrectly ordered, or missing verification or workarounds. + +## Input + +`state["trigger_payload"]` contains: +- `source_thread_id` — the completed thread that exposed the skill gap +- `goal` — what the thread was trying to accomplish +- `active_skills` — implicated skills with `doc_id`, `name`, `version`, and snippet names +- `issues` — trace issues from the thread +- `error_messages` — action failure text +- `observed_actions` — actions actually attempted during execution +- `repair_hints` — conservative hint categories such as `missing_prerequisite`, `stale_command_path`, `missing_pitfall`, `missing_verification` + +## Mission + +Choose the single most likely implicated skill and produce the smallest safe repair. + +Classify the gap as exactly one of: +- `missing_prerequisite` +- `wrong_ordering` +- `stale_command_path` +- `missing_branch` +- `missing_pitfall` +- `missing_verification` + +## Process + +1. Inspect the implicated skill and source context with tools (`memory_search`, `memory_read`, `read_file`, `shell`, etc.). +2. Confirm the gap from the thread evidence. If the evidence points to engine behavior instead of the skill, do not repair the skill. +3. Generate the smallest safe content patch: + - add an auth or setup prerequisite check + - add a missing ordering note + - fix one exact command or path + - add one platform-specific branch or workaround + - add one verification or smoke-test step +4. Keep the skill focused. Do not rewrite the entire skill unless the existing content is unusable. + +## Output Format + +Return a single JSON object in `FINAL(...)` with this shape: + +```json +{ + "doc_id": "", + "repair_type": "missing_prerequisite", + "summary": "Added GitHub auth prerequisite before gh commands.", + "updated_content": "", + "description": "", + "activation": { + "keywords": ["github", "pull request"], + "patterns": [], + "tags": ["github"], + "exclude_keywords": [], + "max_context_tokens": 1200 + }, + "code_snippets": [], + "next_focus": "Watch for repeated failures in repo-cloning flows.", + "goal_achieved": false +} +``` + +Only include `description`, `activation`, or `code_snippets` if they truly need to change. + +## Rules + +- Repair only one skill per thread. +- Only target a `doc_id` from `active_skills`. +- Prefer additive edits over broad rewrites. +- Do not write the skill doc directly with `memory_write`; return structured JSON and let the runtime apply the versioned update. +- If the evidence is weak or the gap is not skill-related, call `FINAL("No safe skill repair identified")`. diff --git a/crates/ironclaw_engine/src/capability/lease.rs b/crates/ironclaw_engine/src/capability/lease.rs new file mode 100644 index 00000000000..7e35c5dce8b --- /dev/null +++ b/crates/ironclaw_engine/src/capability/lease.rs @@ -0,0 +1,584 @@ +//! Lease manager — grants, validates, and expires capability leases. + +use std::collections::HashMap; + +use chrono::Utc; +use tokio::sync::RwLock; + +use crate::types::capability::{CapabilityLease, GrantedActions, LeaseId}; +use crate::types::error::EngineError; +use crate::types::thread::ThreadId; + +/// Manages the lifecycle of capability leases. +/// +/// Leases are the mechanism by which threads gain access to capabilities. +/// They are scoped (time-limited, use-limited, action-restricted) to bound +/// the blast radius of any single thread. +pub struct LeaseManager { + active: RwLock>, +} + +impl LeaseManager { + pub fn new() -> Self { + Self { + active: RwLock::new(HashMap::new()), + } + } + + /// Grant a new lease to a thread. + /// + /// Returns `EngineError::Effect` if `duration` is non-positive or + /// `max_uses` is zero — these would create immediately-expired or + /// unusable leases. + pub async fn grant( + &self, + thread_id: ThreadId, + capability_name: impl Into, + granted_actions: GrantedActions, + duration: Option, + max_uses: Option, + ) -> Result { + if let Some(d) = duration + && d <= chrono::Duration::zero() + { + return Err(EngineError::Effect { + reason: format!("lease duration must be positive, got {}s", d.num_seconds()), + }); + } + if let Some(0) = max_uses { + return Err(EngineError::Effect { + reason: "lease max_uses must be > 0".into(), + }); + } + + let now = Utc::now(); + let lease = CapabilityLease { + id: LeaseId::new(), + thread_id, + capability_name: capability_name.into(), + granted_actions, + granted_at: now, + expires_at: duration.map(|d| now + d), + max_uses, + uses_remaining: max_uses, + revoked: false, + revoked_reason: None, + }; + self.active.write().await.insert(lease.id, lease.clone()); + Ok(lease) + } + + /// Check whether a lease is still valid. Returns the lease if valid. + pub async fn check(&self, lease_id: LeaseId) -> Result { + let leases = self.active.read().await; + let lease = leases + .get(&lease_id) + .ok_or_else(|| EngineError::LeaseNotFound { + lease_id: format!("{lease_id:?}"), + })?; + if !lease.is_valid() { + return Err(EngineError::LeaseExpired { + capability_name: lease.capability_name.clone(), + }); + } + Ok(lease.clone()) + } + + /// Consume one use of a lease. Returns error if the lease is invalid or exhausted. + pub async fn consume_use(&self, lease_id: LeaseId) -> Result<(), EngineError> { + let mut leases = self.active.write().await; + let lease = leases + .get_mut(&lease_id) + .ok_or_else(|| EngineError::LeaseExpired { + capability_name: format!("lease {lease_id:?} not found"), + })?; + if !lease.is_valid() { + return Err(EngineError::LeaseExpired { + capability_name: lease.capability_name.clone(), + }); + } + if !lease.consume_use() { + return Err(EngineError::LeaseExpired { + capability_name: lease.capability_name.clone(), + }); + } + Ok(()) + } + + /// Refund one lease use after an execution was interrupted before the + /// action completed. + pub async fn refund_use(&self, lease_id: LeaseId) -> Result<(), EngineError> { + let mut leases = self.active.write().await; + let lease = leases + .get_mut(&lease_id) + .ok_or_else(|| EngineError::LeaseExpired { + capability_name: format!("lease {lease_id:?} not found"), + })?; + lease.refund_use(); + Ok(()) + } + + /// Update the granted actions for an existing lease in place. + pub async fn update_granted_actions( + &self, + lease_id: LeaseId, + granted_actions: GrantedActions, + ) -> Result { + let mut leases = self.active.write().await; + let lease = leases + .get_mut(&lease_id) + .ok_or_else(|| EngineError::LeaseNotFound { + lease_id: format!("{lease_id:?}"), + })?; + lease.granted_actions = granted_actions; + Ok(lease.clone()) + } + + /// Revoke a lease by ID with a reason for audit trail. + pub async fn revoke(&self, lease_id: LeaseId, reason: &str) { + let mut leases = self.active.write().await; + if let Some(lease) = leases.get_mut(&lease_id) { + lease.revoked = true; + lease.revoked_reason = Some(reason.to_string()); + tracing::debug!( + lease_id = ?lease_id, + capability = %lease.capability_name, + reason, + "lease revoked" + ); + } + } + + /// Remove all expired or revoked leases from the active set. + pub async fn expire_stale(&self) -> usize { + let mut leases = self.active.write().await; + let before = leases.len(); + leases.retain(|_, lease| lease.is_valid()); + before - leases.len() + } + + /// Get all active (valid) leases for a thread. + pub async fn active_for_thread(&self, thread_id: ThreadId) -> Vec { + let leases = self.active.read().await; + leases + .values() + .filter(|l| l.thread_id == thread_id && l.is_valid()) + .cloned() + .collect() + } + + /// Find the lease that grants a specific action to a thread. + pub async fn find_lease_for_action( + &self, + thread_id: ThreadId, + action_name: &str, + ) -> Option { + let hyphenated = action_name.replace('_', "-"); + let underscored = action_name.replace('-', "_"); + let leases = self.active.read().await; + leases + .values() + .find(|l| { + l.thread_id == thread_id + && l.is_valid() + && (l.covers_action(action_name) + || l.covers_action(&hyphenated) + || l.covers_action(&underscored)) + }) + .cloned() + } + + /// Derive child leases from a parent thread's active leases. + /// + /// Implements intersection semantics: the child gets only leases for + /// actions that are both in the parent's active set AND in the + /// `requested_actions` set. If `requested_actions` is `None`, the child + /// inherits all of the parent's valid leases. + /// + /// Invariants: + /// - A child can never have more privileges than its parent. + /// - Child leases inherit the parent's expiry (never outlive parent). + /// - Child leases inherit the parent's remaining budget. + /// - Expired parent leases yield no child leases. + pub async fn derive_child_leases( + &self, + parent_thread_id: ThreadId, + child_thread_id: ThreadId, + requested_actions: Option<&std::collections::HashSet>, + ) -> Vec { + let parent_leases = self.active_for_thread(parent_thread_id).await; + let mut child_leases = Vec::new(); + + for parent in &parent_leases { + if !parent.is_valid() { + continue; + } + + let child_grants = match requested_actions { + Some(req) => { + match &parent.granted_actions { + GrantedActions::All => { + // Parent is wildcard. Child gets only the + // requested subset, NOT a wildcard. + GrantedActions::Specific(req.iter().cloned().collect()) + } + GrantedActions::Specific(parent_actions) => { + // Intersection: only actions in both parent and request. + let intersection: Vec = parent_actions + .iter() + .filter(|a| req.contains(*a)) + .cloned() + .collect(); + GrantedActions::Specific(intersection) + } + } + } + None => parent.granted_actions.clone(), + }; + + // Skip if intersection is empty (no matching actions) + if let GrantedActions::Specific(ref actions) = child_grants + && actions.is_empty() + && requested_actions.is_some() + { + continue; + } + + child_leases.push(CapabilityLease { + id: LeaseId::new(), + thread_id: child_thread_id, + capability_name: parent.capability_name.clone(), + granted_actions: child_grants, + granted_at: Utc::now(), + expires_at: parent.expires_at, // never outlive parent + max_uses: parent.uses_remaining, // budget from parent's remaining + uses_remaining: parent.uses_remaining, + revoked: false, + revoked_reason: None, + }); + } + + // Batch insert under a single write lock (M2: avoid per-iteration locking) + { + let mut active = self.active.write().await; + for child in &child_leases { + active.insert(child.id, child.clone()); + } + } + + child_leases + } + + /// Atomically find the lease for an action and consume one use. + /// + /// Avoids the TOCTOU race between `find_lease_for_action` (read lock) and + /// `consume_use` (write lock) — both happen under a single write lock. + /// Returns the lease snapshot (post-consume) if found and valid. + pub async fn find_and_consume( + &self, + thread_id: ThreadId, + action_name: &str, + ) -> Result { + let mut leases = self.active.write().await; + let lease = leases + .values_mut() + .find(|l| l.thread_id == thread_id && l.is_valid() && l.covers_action(action_name)) + .ok_or_else(|| EngineError::LeaseNotFound { + lease_id: format!("no valid lease for action '{action_name}'"), + })?; + + if !lease.consume_use() { + return Err(EngineError::LeaseExpired { + capability_name: lease.capability_name.clone(), + }); + } + + Ok(lease.clone()) + } +} + +impl Default for LeaseManager { + fn default() -> Self { + Self::new() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::GrantedActions; + use crate::types::thread::ThreadId; + + #[tokio::test] + async fn grant_and_check() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let lease = mgr + .grant(tid, "github", GrantedActions::All, None, None) + .await + .unwrap(); + assert!(mgr.check(lease.id).await.is_ok()); + } + + #[tokio::test] + async fn check_nonexistent_fails() { + let mgr = LeaseManager::new(); + assert!(mgr.check(LeaseId::new()).await.is_err()); + } + + #[tokio::test] + async fn consume_use_works() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let lease = mgr + .grant(tid, "github", GrantedActions::All, None, Some(2)) + .await + .unwrap(); + assert!(mgr.consume_use(lease.id).await.is_ok()); + assert!(mgr.consume_use(lease.id).await.is_ok()); + assert!(mgr.consume_use(lease.id).await.is_err()); + } + + #[tokio::test] + async fn refund_use_restores_consumed_budget() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let lease = mgr + .grant(tid, "github", GrantedActions::All, None, Some(2)) + .await + .unwrap(); + mgr.consume_use(lease.id).await.unwrap(); + let consumed = mgr.check(lease.id).await.unwrap(); + assert_eq!(consumed.uses_remaining, Some(1)); + mgr.refund_use(lease.id).await.unwrap(); + let restored = mgr.check(lease.id).await.unwrap(); + assert_eq!(restored.uses_remaining, Some(2)); + } + + #[tokio::test] + async fn revoke_invalidates() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let lease = mgr + .grant(tid, "github", GrantedActions::All, None, None) + .await + .unwrap(); + mgr.revoke(lease.id, "test").await; + assert!(mgr.check(lease.id).await.is_err()); + } + + #[tokio::test] + async fn expire_stale_removes_revoked() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let lease = mgr + .grant(tid, "github", GrantedActions::All, None, None) + .await + .unwrap(); + mgr.revoke(lease.id, "done").await; + let removed = mgr.expire_stale().await; + assert_eq!(removed, 1); + assert!(mgr.active_for_thread(tid).await.is_empty()); + } + + #[tokio::test] + async fn active_for_thread_filters_correctly() { + let mgr = LeaseManager::new(); + let t1 = ThreadId::new(); + let t2 = ThreadId::new(); + mgr.grant(t1, "github", GrantedActions::All, None, None) + .await + .unwrap(); + mgr.grant(t1, "memory", GrantedActions::All, None, None) + .await + .unwrap(); + mgr.grant(t2, "slack", GrantedActions::All, None, None) + .await + .unwrap(); + assert_eq!(mgr.active_for_thread(t1).await.len(), 2); + assert_eq!(mgr.active_for_thread(t2).await.len(), 1); + } + + #[tokio::test] + async fn find_lease_for_action_respects_grants() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + mgr.grant( + tid, + "github", + GrantedActions::Specific(vec!["create_issue".into(), "list_prs".into()]), + None, + None, + ) + .await + .unwrap(); + assert!( + mgr.find_lease_for_action(tid, "create_issue") + .await + .is_some() + ); + assert!( + mgr.find_lease_for_action(tid, "delete_repo") + .await + .is_none() + ); + } + + #[tokio::test] + async fn negative_duration_rejected() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let result = mgr + .grant( + tid, + "github", + GrantedActions::All, + Some(chrono::Duration::seconds(-10)), + None, + ) + .await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn zero_max_uses_rejected() { + let mgr = LeaseManager::new(); + let tid = ThreadId::new(); + let result = mgr + .grant(tid, "github", GrantedActions::All, None, Some(0)) + .await; + assert!(result.is_err()); + } + + // ── derive_child_leases ────────────────────────────────── + + #[tokio::test] + async fn test_child_inherits_subset_of_parent() { + let mgr = LeaseManager::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + mgr.grant( + parent, + "tools", + GrantedActions::Specific(vec!["A".into(), "B".into(), "C".into()]), + None, + None, + ) + .await + .unwrap(); + + let mut requested = std::collections::HashSet::new(); + requested.insert("B".into()); + requested.insert("C".into()); + requested.insert("D".into()); // not in parent + + let child_leases = mgr + .derive_child_leases(parent, child, Some(&requested)) + .await; + assert_eq!(child_leases.len(), 1); + assert!(child_leases[0].granted_actions.covers("B")); + assert!(child_leases[0].granted_actions.covers("C")); + assert!(!child_leases[0].granted_actions.covers("D")); + } + + #[tokio::test] + async fn test_child_never_exceeds_parent_expiry() { + let mgr = LeaseManager::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + let parent_lease = mgr + .grant( + parent, + "tools", + GrantedActions::Specific(vec!["read".into()]), + Some(chrono::Duration::hours(1)), + None, + ) + .await + .unwrap(); + + let child_leases = mgr.derive_child_leases(parent, child, None).await; + assert_eq!(child_leases.len(), 1); + assert_eq!(child_leases[0].expires_at, parent_lease.expires_at); + } + + #[tokio::test] + async fn test_expired_parent_yields_empty_child() { + let mgr = LeaseManager::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + // Manually insert an already-expired lease (bypassing grant validation) + let now = Utc::now(); + let expired_lease = CapabilityLease { + id: LeaseId::new(), + thread_id: parent, + capability_name: "tools".into(), + granted_actions: GrantedActions::Specific(vec!["read".into()]), + granted_at: now, + expires_at: Some(now - chrono::Duration::seconds(10)), + max_uses: None, + uses_remaining: None, + revoked: false, + revoked_reason: None, + }; + mgr.active + .write() + .await + .insert(expired_lease.id, expired_lease); + + let child_leases = mgr.derive_child_leases(parent, child, None).await; + assert!(child_leases.is_empty()); + } + + #[tokio::test] + async fn test_child_inherits_remaining_budget() { + let mgr = LeaseManager::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + let parent_lease = mgr + .grant( + parent, + "tools", + GrantedActions::Specific(vec!["read".into()]), + None, + Some(10), + ) + .await + .unwrap(); + + // Consume 3 uses from parent + mgr.consume_use(parent_lease.id).await.unwrap(); + mgr.consume_use(parent_lease.id).await.unwrap(); + mgr.consume_use(parent_lease.id).await.unwrap(); + + let child_leases = mgr.derive_child_leases(parent, child, None).await; + assert_eq!(child_leases.len(), 1); + // Parent had 10, consumed 3, so 7 remaining + assert_eq!(child_leases[0].uses_remaining, Some(7)); + } + + #[tokio::test] + async fn test_child_with_none_inherits_all() { + let mgr = LeaseManager::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + mgr.grant( + parent, + "tools", + GrantedActions::Specific(vec!["read".into(), "write".into()]), + None, + None, + ) + .await + .unwrap(); + + let child_leases = mgr.derive_child_leases(parent, child, None).await; + assert_eq!(child_leases.len(), 1); + assert_eq!(child_leases[0].granted_actions.actions().len(), 2); + } +} diff --git a/crates/ironclaw_engine/src/capability/mod.rs b/crates/ironclaw_engine/src/capability/mod.rs new file mode 100644 index 00000000000..d334eae6aeb --- /dev/null +++ b/crates/ironclaw_engine/src/capability/mod.rs @@ -0,0 +1,14 @@ +//! Capability management. +//! +//! - [`CapabilityRegistry`] — stores known capabilities and their actions +//! - [`LeaseManager`] — grants, validates, and expires capability leases +//! - [`PolicyEngine`] — deterministic effect-level allow/deny/approve + +pub mod lease; +pub mod planner; +pub mod policy; +pub mod registry; + +pub use lease::LeaseManager; +pub use policy::{PolicyDecision, PolicyEngine}; +pub use registry::CapabilityRegistry; diff --git a/crates/ironclaw_engine/src/capability/planner.rs b/crates/ironclaw_engine/src/capability/planner.rs new file mode 100644 index 00000000000..a611c157d8b --- /dev/null +++ b/crates/ironclaw_engine/src/capability/planner.rs @@ -0,0 +1,202 @@ +//! Lease planning for new threads. +//! +//! Converts capability registry contents plus thread type into explicit +//! capability grants. Thread-type-aware: Foreground gets all tiers, +//! Research gets read-only + stateful, Mission excludes administrative tools. + +use crate::capability::registry::CapabilityRegistry; +use crate::gate::tool_tier::{ToolTier, classify_tool_tier, is_autonomous_denylisted}; +use crate::types::capability::GrantedActions; +use crate::types::thread::ThreadType; + +/// Explicit grant plan for a single capability. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CapabilityGrantPlan { + pub capability_name: String, + pub granted_actions: GrantedActions, +} + +/// Plans explicit capability leases for new threads. +/// +/// Uses [`ToolTier`] classification to scope grants by thread type: +/// - **Foreground**: all tiers (interactive approval gates protect Privileged/Admin) +/// - **Research**: `ReadOnly` and `Stateful` only +/// - **Mission**: `ReadOnly`, `Stateful`, and non-denylisted `Privileged` +#[derive(Debug, Default)] +pub struct LeasePlanner; + +impl LeasePlanner { + pub fn new() -> Self { + Self + } + + /// Build the capability grants for a new thread. + pub fn plan_for_thread( + &self, + thread_type: ThreadType, + capabilities: &CapabilityRegistry, + ) -> Vec { + capabilities + .list() + .into_iter() + .filter_map(|cap| { + let granted_actions: Vec = cap + .actions + .iter() + .filter(|action| { + let tier = classify_tool_tier(action); + Self::tier_allowed(thread_type, &action.name, tier) + }) + .map(|action| action.name.clone()) + .collect(); + if granted_actions.is_empty() { + None + } else { + Some(CapabilityGrantPlan { + capability_name: cap.name.clone(), + granted_actions: GrantedActions::Specific(granted_actions), + }) + } + }) + .collect() + } + + /// Check whether a tool tier is allowed for a given thread type. + fn tier_allowed(thread_type: ThreadType, action_name: &str, tier: ToolTier) -> bool { + match thread_type { + ThreadType::Foreground => { + // Foreground gets everything — interactive approval gates + // protect Privileged and Administrative tools. + true + } + ThreadType::Research => { + // Research threads: read-only and stateful only. + tier <= ToolTier::Stateful + } + ThreadType::Mission => { + // Mission threads: no Administrative, no denylisted Privileged. + match tier { + ToolTier::ReadOnly | ToolTier::Stateful => true, + ToolTier::Privileged => !is_autonomous_denylisted(action_name), + ToolTier::Administrative => false, + } + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::{ActionDef, Capability, EffectType, GrantedActions}; + + fn action(name: &str, effects: Vec, requires_approval: bool) -> ActionDef { + ActionDef { + name: name.into(), + description: format!("{name} action"), + parameters_schema: serde_json::json!({}), + effects, + requires_approval, + } + } + + fn mixed_registry() -> CapabilityRegistry { + let mut reg = CapabilityRegistry::new(); + reg.register(Capability { + name: "tools".into(), + description: "all tools".into(), + actions: vec![ + action("echo", vec![EffectType::ReadLocal], false), // ReadOnly + action("read_file", vec![EffectType::ReadLocal], false), // ReadOnly + action("file_write", vec![EffectType::WriteLocal], false), // Stateful + action("shell", vec![EffectType::WriteLocal], true), // Privileged + action("http", vec![EffectType::WriteExternal], true), // Privileged + action("routine_create", vec![EffectType::WriteLocal], false), // Administrative (denylisted) + action("tool_install", vec![EffectType::WriteLocal], false), // Administrative (denylisted) + ], + knowledge: vec![], + policies: vec![], + }); + reg + } + + fn simple_registry() -> CapabilityRegistry { + let mut reg = CapabilityRegistry::new(); + reg.register(Capability { + name: "tools".into(), + description: "test".into(), + actions: vec![action("read_file", vec![EffectType::ReadLocal], false)], + knowledge: vec![], + policies: vec![], + }); + reg + } + + #[test] + fn foreground_threads_get_explicit_actions() { + let planner = LeasePlanner::new(); + let plans = planner.plan_for_thread(ThreadType::Foreground, &simple_registry()); + assert_eq!(plans.len(), 1); + assert_eq!(plans[0].capability_name, "tools"); + assert_eq!( + plans[0].granted_actions, + GrantedActions::Specific(vec!["read_file".into()]) + ); + } + + #[test] + fn test_foreground_gets_all_tiers() { + let planner = LeasePlanner::new(); + let plans = planner.plan_for_thread(ThreadType::Foreground, &mixed_registry()); + assert_eq!(plans.len(), 1); + let actions = plans[0].granted_actions.actions(); + assert_eq!(actions.len(), 7, "Foreground should get all 7 actions"); + assert!(plans[0].granted_actions.covers("routine_create")); + assert!(plans[0].granted_actions.covers("shell")); + } + + #[test] + fn test_research_excludes_privileged_and_admin() { + let planner = LeasePlanner::new(); + let plans = planner.plan_for_thread(ThreadType::Research, &mixed_registry()); + assert_eq!(plans.len(), 1); + let actions = plans[0].granted_actions.actions(); + // ReadOnly: echo, read_file. Stateful: file_write. + assert_eq!( + actions.len(), + 3, + "Research should get 3 actions: {:?}", + actions + ); + assert!(plans[0].granted_actions.covers("echo")); + assert!(plans[0].granted_actions.covers("read_file")); + assert!(plans[0].granted_actions.covers("file_write")); + assert!(!plans[0].granted_actions.covers("shell")); + assert!(!plans[0].granted_actions.covers("routine_create")); + } + + #[test] + fn test_mission_excludes_administrative() { + let planner = LeasePlanner::new(); + let plans = planner.plan_for_thread(ThreadType::Mission, &mixed_registry()); + assert_eq!(plans.len(), 1); + let ga = &plans[0].granted_actions; + // Includes ReadOnly, Stateful, and non-denylisted Privileged (shell, http). + // Excludes Administrative (routine_create, tool_install). + assert!(ga.covers("echo")); + assert!(ga.covers("shell")); + assert!(ga.covers("http")); + assert!(!ga.covers("routine_create")); + assert!(!ga.covers("tool_install")); + } + + #[test] + fn test_mission_excludes_denylisted_privileged() { + let planner = LeasePlanner::new(); + let plans = planner.plan_for_thread(ThreadType::Mission, &mixed_registry()); + let ga = &plans[0].granted_actions; + // routine_create and tool_install are in the denylist + assert!(!ga.covers("routine_create")); + assert!(!ga.covers("tool_install")); + } +} diff --git a/crates/ironclaw_engine/src/capability/policy.rs b/crates/ironclaw_engine/src/capability/policy.rs new file mode 100644 index 00000000000..08b509bfee0 --- /dev/null +++ b/crates/ironclaw_engine/src/capability/policy.rs @@ -0,0 +1,391 @@ +//! Deterministic policy engine. +//! +//! Evaluates whether an action is allowed, denied, or requires approval +//! based on effect types, capability policies, and thread leases. +//! No LLM calls — purely deterministic. + +use crate::types::capability::{ + ActionDef, CapabilityLease, EffectType, PolicyCondition, PolicyEffect, PolicyRule, +}; +use crate::types::provenance::Provenance; + +/// The result of a policy evaluation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PolicyDecision { + Allow, + Deny { reason: String }, + RequireApproval { reason: String }, +} + +/// Deterministic policy engine. +/// +/// Evaluation precedence: Deny > RequireApproval > Allow. +/// Checks are evaluated in order: global policies, then capability policies, +/// then action-level `requires_approval`, then effect-type checks against +/// the lease's allowed effects. +pub struct PolicyEngine { + global_policies: Vec, + /// Effect types that are always denied unless explicitly overridden. + pub(crate) denied_effects: Vec, +} + +impl PolicyEngine { + pub fn new() -> Self { + Self { + global_policies: Vec::new(), + denied_effects: Vec::new(), + } + } + + /// Add a global policy rule. + pub fn add_global_policy(&mut self, rule: PolicyRule) { + self.global_policies.push(rule); + } + + /// Add an effect type that is always denied. + pub fn deny_effect(&mut self, effect: EffectType) { + self.denied_effects.push(effect); + } + + /// Evaluate whether an action is allowed given a lease and capability policies. + pub fn evaluate( + &self, + action: &ActionDef, + lease: &CapabilityLease, + capability_policies: &[PolicyRule], + ) -> PolicyDecision { + // 1. Check lease validity + if !lease.is_valid() { + return PolicyDecision::Deny { + reason: format!("lease for {} is expired/revoked", lease.capability_name), + }; + } + + // 2. Check lease covers this action + if !lease.covers_action(&action.name) { + return PolicyDecision::Deny { + reason: format!( + "lease for {} does not cover action {}", + lease.capability_name, action.name + ), + }; + } + + // 3. Check denied effect types + for effect in &action.effects { + if self.denied_effects.contains(effect) { + return PolicyDecision::Deny { + reason: format!("effect type {effect:?} is denied by global policy"), + }; + } + } + + // 4. Evaluate global policies + let mut decision = PolicyDecision::Allow; + for rule in &self.global_policies { + if rule_matches(rule, action) { + decision = merge_decision(decision, rule.effect, &rule.name); + } + } + + // 5. Evaluate capability-level policies + for rule in capability_policies { + if rule_matches(rule, action) { + decision = merge_decision(decision, rule.effect, &rule.name); + } + } + + // 6. Check action-level requires_approval + if action.requires_approval { + decision = merge_decision( + decision, + PolicyEffect::RequireApproval, + "action requires approval", + ); + } + + // Log denials for audit trail / incident investigation + if let PolicyDecision::Deny { ref reason } = decision { + tracing::debug!( + action = %action.name, + capability = %lease.capability_name, + reason, + "policy denied action" + ); + } + + decision + } + + /// Evaluate with provenance-aware taint checking. + /// + /// Extends the base evaluation with provenance-based rules: + /// - `LlmGenerated` data + `Financial` effect → RequireApproval + /// - `LlmGenerated` data + `WriteExternal` effect → RequireApproval + /// - `ToolOutput` data + `Financial` effect → RequireApproval + pub fn evaluate_with_provenance( + &self, + action: &ActionDef, + lease: &CapabilityLease, + capability_policies: &[PolicyRule], + provenance: &Provenance, + ) -> PolicyDecision { + let mut decision = self.evaluate(action, lease, capability_policies); + + // Provenance-based taint rules + match provenance { + Provenance::LlmGenerated => { + if action.effects.contains(&EffectType::Financial) { + decision = merge_decision( + decision, + PolicyEffect::RequireApproval, + "LLM-generated data cannot trigger financial effects without approval", + ); + } + if action.effects.contains(&EffectType::WriteExternal) { + decision = merge_decision( + decision, + PolicyEffect::RequireApproval, + "LLM-generated data requires approval for external writes", + ); + } + } + Provenance::ToolOutput { .. } => { + if action.effects.contains(&EffectType::Financial) { + decision = merge_decision( + decision, + PolicyEffect::RequireApproval, + "tool output data requires approval for financial effects", + ); + } + } + // User and System provenance are trusted + Provenance::User | Provenance::System => {} + // MemoryRetrieval is internal, treat as trusted + Provenance::MemoryRetrieval { .. } => {} + } + + decision + } +} + +impl Default for PolicyEngine { + fn default() -> Self { + Self::new() + } +} + +/// Check whether a policy rule's condition matches the given action. +fn rule_matches(rule: &PolicyRule, action: &ActionDef) -> bool { + match &rule.condition { + PolicyCondition::Always => true, + PolicyCondition::ActionMatches { pattern } => action.name == *pattern, + PolicyCondition::EffectTypeIs(effect) => action.effects.contains(effect), + } +} + +/// Merge a new policy effect into the current decision. +/// Deny > RequireApproval > Allow. +fn merge_decision(current: PolicyDecision, effect: PolicyEffect, source: &str) -> PolicyDecision { + match effect { + PolicyEffect::Deny => PolicyDecision::Deny { + reason: source.to_string(), + }, + PolicyEffect::RequireApproval => match current { + PolicyDecision::Deny { .. } => current, + _ => PolicyDecision::RequireApproval { + reason: source.to_string(), + }, + }, + PolicyEffect::Allow => current, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::{GrantedActions, LeaseId}; + use crate::types::thread::ThreadId; + use chrono::Utc; + + fn make_action(name: &str, effects: Vec, requires_approval: bool) -> ActionDef { + ActionDef { + name: name.into(), + description: String::new(), + parameters_schema: serde_json::json!({}), + effects, + requires_approval, + } + } + + fn make_lease() -> CapabilityLease { + CapabilityLease { + id: LeaseId::new(), + thread_id: ThreadId::new(), + capability_name: "test".into(), + granted_actions: GrantedActions::All, + granted_at: Utc::now(), + expires_at: None, + max_uses: None, + uses_remaining: None, + revoked: false, + revoked_reason: None, + } + } + + #[test] + fn allow_by_default() { + let engine = PolicyEngine::new(); + let action = make_action("read_file", vec![EffectType::ReadLocal], false); + let lease = make_lease(); + assert_eq!(engine.evaluate(&action, &lease, &[]), PolicyDecision::Allow); + } + + #[test] + fn denied_effect_type() { + let mut engine = PolicyEngine::new(); + engine.deny_effect(EffectType::Financial); + let action = make_action("transfer", vec![EffectType::Financial], false); + let lease = make_lease(); + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::Deny { .. } + )); + } + + #[test] + fn action_requires_approval() { + let engine = PolicyEngine::new(); + let action = make_action("deploy", vec![EffectType::WriteExternal], true); + let lease = make_lease(); + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::RequireApproval { .. } + )); + } + + #[test] + fn global_policy_deny_overrides_approval() { + let mut engine = PolicyEngine::new(); + engine.add_global_policy(PolicyRule { + name: "no external writes".into(), + condition: PolicyCondition::EffectTypeIs(EffectType::WriteExternal), + effect: PolicyEffect::Deny, + }); + let action = make_action("deploy", vec![EffectType::WriteExternal], true); + let lease = make_lease(); + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::Deny { .. } + )); + } + + #[test] + fn capability_policy_requires_approval() { + let engine = PolicyEngine::new(); + let action = make_action("create_issue", vec![EffectType::WriteExternal], false); + let lease = make_lease(); + let cap_policies = vec![PolicyRule { + name: "approve writes".into(), + condition: PolicyCondition::EffectTypeIs(EffectType::WriteExternal), + effect: PolicyEffect::RequireApproval, + }]; + assert!(matches!( + engine.evaluate(&action, &lease, &cap_policies), + PolicyDecision::RequireApproval { .. } + )); + } + + #[test] + fn expired_lease_denied() { + let engine = PolicyEngine::new(); + let action = make_action("read", vec![EffectType::ReadLocal], false); + let mut lease = make_lease(); + lease.revoked = true; + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::Deny { .. } + )); + } + + #[test] + fn lease_not_covering_action_denied() { + let engine = PolicyEngine::new(); + let action = make_action("delete_repo", vec![EffectType::WriteExternal], false); + let mut lease = make_lease(); + lease.granted_actions = GrantedActions::Specific(vec!["create_issue".into()]); + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::Deny { .. } + )); + } + + #[test] + fn llm_generated_financial_requires_approval() { + let engine = PolicyEngine::new(); + let action = make_action("transfer_funds", vec![EffectType::Financial], false); + let lease = make_lease(); + let decision = + engine.evaluate_with_provenance(&action, &lease, &[], &Provenance::LlmGenerated); + assert!(matches!(decision, PolicyDecision::RequireApproval { .. })); + } + + #[test] + fn llm_generated_write_external_requires_approval() { + let engine = PolicyEngine::new(); + let action = make_action("post_message", vec![EffectType::WriteExternal], false); + let lease = make_lease(); + let decision = + engine.evaluate_with_provenance(&action, &lease, &[], &Provenance::LlmGenerated); + assert!(matches!(decision, PolicyDecision::RequireApproval { .. })); + } + + #[test] + fn user_provenance_allows_financial() { + let engine = PolicyEngine::new(); + let action = make_action("transfer_funds", vec![EffectType::Financial], false); + let lease = make_lease(); + let decision = engine.evaluate_with_provenance(&action, &lease, &[], &Provenance::User); + assert_eq!(decision, PolicyDecision::Allow); + } + + #[test] + fn tool_output_financial_requires_approval() { + let engine = PolicyEngine::new(); + let action = make_action("pay_invoice", vec![EffectType::Financial], false); + let lease = make_lease(); + let decision = engine.evaluate_with_provenance( + &action, + &lease, + &[], + &Provenance::ToolOutput { + action_name: "scrape_invoices".into(), + }, + ); + assert!(matches!(decision, PolicyDecision::RequireApproval { .. })); + } + + #[test] + fn action_matches_pattern() { + let mut engine = PolicyEngine::new(); + engine.add_global_policy(PolicyRule { + name: "approve deletes".into(), + condition: PolicyCondition::ActionMatches { + pattern: "delete_repo".into(), + }, + effect: PolicyEffect::RequireApproval, + }); + let action = make_action("delete_repo", vec![EffectType::WriteExternal], false); + let lease = make_lease(); + assert!(matches!( + engine.evaluate(&action, &lease, &[]), + PolicyDecision::RequireApproval { .. } + )); + + let action2 = make_action("create_issue", vec![EffectType::WriteExternal], false); + assert_eq!( + engine.evaluate(&action2, &lease, &[]), + PolicyDecision::Allow + ); + } +} diff --git a/crates/ironclaw_engine/src/capability/registry.rs b/crates/ironclaw_engine/src/capability/registry.rs new file mode 100644 index 00000000000..12d26e67694 --- /dev/null +++ b/crates/ironclaw_engine/src/capability/registry.rs @@ -0,0 +1,170 @@ +//! Capability registry — stores capability definitions available to the system. + +use std::collections::HashMap; + +use crate::types::capability::{ActionDef, Capability}; + +/// Registry of all known capabilities. +/// +/// Capabilities are registered at startup (from extensions, built-in tools, +/// etc.) and queried when granting leases or resolving action names. +#[derive(Debug, Default)] +pub struct CapabilityRegistry { + capabilities: HashMap, +} + +impl CapabilityRegistry { + pub fn new() -> Self { + Self::default() + } + + /// Register a capability. Overwrites any existing capability with the same name. + pub fn register(&mut self, capability: Capability) { + self.capabilities + .insert(capability.name.clone(), capability); + } + + /// Look up a capability by name. + pub fn get(&self, name: &str) -> Option<&Capability> { + self.capabilities.get(name) + } + + /// List all registered capabilities. + pub fn list(&self) -> Vec<&Capability> { + self.capabilities.values().collect() + } + + /// Look up a specific action across all capabilities. + /// + /// Returns `(capability_name, action_def)` if found. + pub fn find_action(&self, action_name: &str) -> Option<(&str, &ActionDef)> { + for cap in self.capabilities.values() { + if let Some(action) = cap.actions.iter().find(|a| a.name == action_name) { + return Some((&cap.name, action)); + } + } + None + } + + /// Get an action definition from a specific capability. + pub fn get_action(&self, capability_name: &str, action_name: &str) -> Option<&ActionDef> { + self.capabilities + .get(capability_name)? + .actions + .iter() + .find(|a| a.name == action_name) + } + + /// Collect all action definitions across all capabilities. + pub fn all_actions(&self) -> Vec<&ActionDef> { + self.capabilities + .values() + .flat_map(|c| c.actions.iter()) + .collect() + } + + /// Number of registered capabilities. + pub fn len(&self) -> usize { + self.capabilities.len() + } + + pub fn is_empty(&self) -> bool { + self.capabilities.is_empty() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::EffectType; + + fn test_capability() -> Capability { + Capability { + name: "github".into(), + description: "GitHub integration".into(), + actions: vec![ + ActionDef { + name: "create_issue".into(), + description: "Create a GitHub issue".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::WriteExternal, EffectType::CredentialedNetwork], + requires_approval: false, + }, + ActionDef { + name: "list_prs".into(), + description: "List pull requests".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::ReadExternal, EffectType::CredentialedNetwork], + requires_approval: false, + }, + ], + knowledge: vec!["When creating issues, always add labels.".into()], + policies: vec![], + } + } + + #[test] + fn register_and_get() { + let mut reg = CapabilityRegistry::new(); + reg.register(test_capability()); + assert_eq!(reg.len(), 1); + assert!(reg.get("github").is_some()); + assert!(reg.get("slack").is_none()); + } + + #[test] + fn find_action_across_capabilities() { + let mut reg = CapabilityRegistry::new(); + reg.register(test_capability()); + let (cap_name, action) = reg.find_action("create_issue").unwrap(); + assert_eq!(cap_name, "github"); + assert_eq!(action.name, "create_issue"); + assert!(reg.find_action("nonexistent").is_none()); + } + + #[test] + fn get_action_from_capability() { + let mut reg = CapabilityRegistry::new(); + reg.register(test_capability()); + assert!(reg.get_action("github", "list_prs").is_some()); + assert!(reg.get_action("github", "delete_repo").is_none()); + assert!(reg.get_action("slack", "list_prs").is_none()); + } + + #[test] + fn all_actions_collects_across_capabilities() { + let mut reg = CapabilityRegistry::new(); + reg.register(test_capability()); + reg.register(Capability { + name: "memory".into(), + description: "Memory tools".into(), + actions: vec![ActionDef { + name: "memory_search".into(), + description: "Search memory".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + }], + knowledge: vec![], + policies: vec![], + }); + assert_eq!(reg.all_actions().len(), 3); + } + + #[test] + fn overwrite_on_re_register() { + let mut reg = CapabilityRegistry::new(); + reg.register(test_capability()); + assert_eq!(reg.get("github").unwrap().actions.len(), 2); + + reg.register(Capability { + name: "github".into(), + description: "Updated".into(), + actions: vec![], + knowledge: vec![], + policies: vec![], + }); + assert_eq!(reg.get("github").unwrap().actions.len(), 0); + assert_eq!(reg.len(), 1); + } +} diff --git a/crates/ironclaw_engine/src/executor/compaction.rs b/crates/ironclaw_engine/src/executor/compaction.rs new file mode 100644 index 00000000000..4fde5b0e8ed --- /dev/null +++ b/crates/ironclaw_engine/src/executor/compaction.rs @@ -0,0 +1,176 @@ +//! Context compaction and token counting. +//! +//! When message history approaches the model's context limit, compaction +//! asks the LLM to summarize progress and resets the history. This follows +//! the official RLM pattern (compaction at 85% of context limit). + +use std::sync::Arc; + +use tracing::debug; + +use crate::traits::llm::{LlmBackend, LlmCallConfig}; +use crate::types::error::EngineError; +use crate::types::message::{MessageRole, ThreadMessage}; +use crate::types::step::{LlmResponse, TokenUsage}; + +/// Characters per token estimate when no tokenizer is available. +/// Conservative estimate (official RLM uses 4). +const CHARS_PER_TOKEN: usize = 4; + +/// Estimate token count for a list of messages. +/// +/// Uses character length / `CHARS_PER_TOKEN` as a rough estimate. +/// The official RLM uses tiktoken when available; we use this fallback +/// since we don't depend on a Python tokenizer. +pub fn estimate_tokens(messages: &[ThreadMessage]) -> usize { + let total_chars: usize = messages + .iter() + .map(|m| { + m.content.len() + m.action_name.as_ref().map_or(0, |n| n.len()) + 4 // overhead per message (role token, delimiters) + }) + .sum(); + total_chars.div_ceil(CHARS_PER_TOKEN) +} + +/// Check if compaction should be triggered. +/// +/// Returns `true` when estimated token count exceeds `threshold_pct` of +/// the model's context limit. +pub fn should_compact( + messages: &[ThreadMessage], + model_context_limit: usize, + threshold_pct: f64, +) -> bool { + let tokens = estimate_tokens(messages); + let threshold = (model_context_limit as f64 * threshold_pct) as usize; + tokens >= threshold +} + +/// The compaction prompt sent to the LLM. +const COMPACTION_PROMPT: &str = "\ +Summarize your progress so far in a concise but complete way. Include: +1. What you have accomplished +2. Key intermediate results and variable values +3. What still needs to be done +4. Any errors encountered and how they were handled + +Preserve all information needed to continue the task. Be specific about data values."; + +/// Compact the message history by asking the LLM to summarize. +/// +/// Returns the new (shorter) message list and the token usage from the +/// summarization call. The original messages are replaced with: +/// `[system_prompt, summary, continuation_note]` +/// +/// The full original messages are returned separately so the caller can +/// store them (e.g., in a `history` variable or event log). +pub async fn compact_messages( + messages: &[ThreadMessage], + llm: &Arc, + compaction_count: u32, +) -> Result { + // Build a summarization request from existing messages + prompt + let mut summarize_messages = messages.to_vec(); + summarize_messages.push(ThreadMessage::user(COMPACTION_PROMPT.to_string())); + + let config = LlmCallConfig { + force_text: true, + ..LlmCallConfig::default() + }; + + let output = llm.complete(&summarize_messages, &[], &config).await?; + + let summary_text = match output.response { + LlmResponse::Text(t) => t, + LlmResponse::ActionCalls { content, .. } | LlmResponse::Code { content, .. } => { + content.unwrap_or_else(|| "[compaction produced no summary]".into()) + } + }; + + // Preserve the system prompt (first message if it's a system message) + let system_msg = messages + .iter() + .find(|m| m.role == MessageRole::System) + .cloned(); + + // Build compacted history + let mut compacted = Vec::new(); + if let Some(sys) = system_msg { + compacted.push(sys); + } + compacted.push(ThreadMessage::assistant(summary_text.clone())); + compacted.push(ThreadMessage::user(format!( + "Your conversation has been compacted {n} time(s). \ + The summary above captures your progress. Continue working on the task.", + n = compaction_count + 1, + ))); + + let tokens_before = estimate_tokens(messages); + let tokens_after = estimate_tokens(&compacted); + + debug!( + tokens_before, + tokens_after, + compaction_count = compaction_count + 1, + "context compacted" + ); + + Ok(CompactionResult { + compacted_messages: compacted, + summary: summary_text, + tokens_used: output.usage, + tokens_before, + tokens_after, + }) +} + +/// Result of a compaction operation. +pub struct CompactionResult { + /// The new (shorter) message list. + pub compacted_messages: Vec, + /// The summary text produced by the LLM. + pub summary: String, + /// Tokens used by the summarization LLM call. + pub tokens_used: TokenUsage, + /// Estimated token count before compaction. + pub tokens_before: usize, + /// Estimated token count after compaction. + pub tokens_after: usize, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn estimate_tokens_empty() { + assert_eq!(estimate_tokens(&[]), 0); + } + + #[test] + fn estimate_tokens_basic() { + let msgs = vec![ + ThreadMessage::system("Hello world"), // 11 chars + 4 overhead = 15 / 4 = 3.75 + ThreadMessage::user("Hi"), // 2 chars + 4 = 6 / 4 = 1.5 + ]; + let tokens = estimate_tokens(&msgs); + // (11+4 + 2+4) / 4 = 21/4 = 5.25 → 6 (ceiling) + assert!(tokens > 0); + assert!(tokens < 100); + } + + #[test] + fn should_compact_below_threshold() { + let msgs = vec![ThreadMessage::user("short message")]; + assert!(!should_compact(&msgs, 128_000, 0.85)); + } + + #[test] + fn should_compact_above_threshold() { + // Create a message large enough to trigger compaction at low limit + let big = "x".repeat(1000); + let msgs = vec![ThreadMessage::user(big)]; + // 1000 chars / 4 = 250 tokens. Context limit 200, threshold 85% = 170 + assert!(should_compact(&msgs, 200, 0.85)); + } +} diff --git a/crates/ironclaw_engine/src/executor/context.rs b/crates/ironclaw_engine/src/executor/context.rs new file mode 100644 index 00000000000..9a48b204f91 --- /dev/null +++ b/crates/ironclaw_engine/src/executor/context.rs @@ -0,0 +1,229 @@ +//! Context building for LLM calls. +//! +//! Assembles the message sequence and action definitions from thread state, +//! active leases, and project memory docs retrieved via the [`RetrievalEngine`]. + +use std::sync::Arc; + +use crate::memory::RetrievalEngine; +use crate::traits::effect::EffectExecutor; +use crate::types::capability::{ActionDef, CapabilityLease}; +use crate::types::error::EngineError; +use crate::types::memory::MemoryDoc; +use crate::types::message::ThreadMessage; +use crate::types::project::ProjectId; + +/// Maximum number of memory docs to inject into context. +const MAX_CONTEXT_DOCS: usize = 5; + +/// Build the context for an LLM call: messages and available actions. +/// +/// Retrieves relevant memory docs from the project and injects them as a +/// system message after the main system prompt. This gives the LLM access +/// to lessons learned, skills, and known issues from prior threads. +pub async fn build_step_context( + messages: &[ThreadMessage], + leases: &[CapabilityLease], + effects: &Arc, + retrieval: Option<&RetrievalEngine>, + project_id: ProjectId, + user_id: &str, + goal: &str, +) -> Result<(Vec, Vec), EngineError> { + // Fetch actions and memory docs in parallel — they are independent. + let actions_fut = effects.available_actions(leases); + let docs_fut = async { + if let Some(engine) = retrieval { + engine + .retrieve_context(project_id, user_id, goal, MAX_CONTEXT_DOCS) + .await + } else { + Ok(Vec::new()) + } + }; + + let (actions_result, docs_result) = tokio::join!(actions_fut, docs_fut); + let actions = actions_result?; + let docs = docs_result?; + + let mut ctx_messages = messages.to_vec(); + + // Inject retrieved memory docs into the existing system prompt. + // Many providers require all system messages at the beginning (or a single + // system message), so we append to the first system message rather than + // inserting a separate one. + if !docs.is_empty() { + let context_section = format_docs_as_context(&docs); + if !ctx_messages.is_empty() + && ctx_messages[0].role == crate::types::message::MessageRole::System + { + // Append to existing system prompt + ctx_messages[0].content.push_str("\n\n"); + ctx_messages[0].content.push_str(&context_section); + } else { + // No system message — prepend as one + ctx_messages.insert(0, ThreadMessage::system(context_section)); + } + } + + Ok((ctx_messages, actions)) +} + +/// Format memory docs into a system message for context injection. +fn format_docs_as_context(docs: &[MemoryDoc]) -> String { + let mut parts = vec!["## Prior Knowledge (from completed threads)\n".to_string()]; + + for doc in docs { + let type_label = match doc.doc_type { + crate::types::memory::DocType::Lesson => "LESSON", + crate::types::memory::DocType::Spec => "MISSING CAPABILITY", + crate::types::memory::DocType::Issue => "KNOWN ISSUE", + crate::types::memory::DocType::Summary => "CONTEXT", + crate::types::memory::DocType::Note => "NOTE", + crate::types::memory::DocType::Skill => "SKILL", + crate::types::memory::DocType::Plan => "PLAN", + }; + // Truncate long docs to avoid context bloat + let content: String = doc.content.chars().take(500).collect(); + let truncated = if doc.content.chars().count() > 500 { + "..." + } else { + "" + }; + parts.push(format!( + "### [{type_label}] {}\n{content}{truncated}\n", + doc.title + )); + } + + parts.join("\n") +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::CapabilityLease; + use crate::types::memory::DocType; + use crate::types::project::ProjectId; + use crate::types::step::ActionResult; + + struct MockEffects; + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _: &str, + _: serde_json::Value, + _: &CapabilityLease, + _: &crate::traits::effect::ThreadExecutionContext, + ) -> Result { + Ok(ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({}), + is_error: false, + duration: std::time::Duration::from_millis(1), + }) + } + + async fn available_actions( + &self, + _: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(vec![]) + } + } + + #[tokio::test] + async fn context_injects_docs_after_system_prompt() { + let project = ProjectId::new(); + let store: Arc = + Arc::new(crate::tests::InMemoryStore::with_docs(vec![ + MemoryDoc::new( + project, + "test-user", + DocType::Lesson, + "web tool alias", + "Use web_search", + ), + ])); + let retrieval = RetrievalEngine::new(store); + let effects: Arc = Arc::new(MockEffects); + + let messages = vec![ + ThreadMessage::system("You are an assistant."), + ThreadMessage::user("search the web"), + ]; + + let (ctx_msgs, _) = build_step_context( + &messages, + &[], + &effects, + Some(&retrieval), + project, + "test-user", + "search the web", + ) + .await + .unwrap(); + + // Should have 2 messages: system prompt (with docs appended), user message + assert_eq!(ctx_msgs.len(), 2); + assert_eq!(ctx_msgs[0].role, crate::types::message::MessageRole::System); + assert!(ctx_msgs[0].content.contains("You are an assistant.")); + assert!(ctx_msgs[0].content.contains("Prior Knowledge")); + assert!(ctx_msgs[0].content.contains("LESSON")); + assert!(ctx_msgs[0].content.contains("web_search")); + assert_eq!(ctx_msgs[1].role, crate::types::message::MessageRole::User); + } + + #[tokio::test] + async fn context_without_retrieval_passes_through() { + let effects: Arc = Arc::new(MockEffects); + let messages = vec![ + ThreadMessage::system("prompt"), + ThreadMessage::user("hello"), + ]; + + let (ctx_msgs, _) = build_step_context( + &messages, + &[], + &effects, + None, + ProjectId::new(), + "test-user", + "hello", + ) + .await + .unwrap(); + + // No injection — same number of messages + assert_eq!(ctx_msgs.len(), 2); + } + + #[tokio::test] + async fn context_no_docs_means_no_injection() { + let project = ProjectId::new(); + let store: Arc = + Arc::new(crate::tests::InMemoryStore::new()); + let retrieval = RetrievalEngine::new(store); + let effects: Arc = Arc::new(MockEffects); + + let messages = vec![ThreadMessage::user("hello")]; + + let (ctx_msgs, _) = build_step_context( + &messages, + &[], + &effects, + Some(&retrieval), + project, + "test-user", + "hello", + ) + .await + .unwrap(); + + assert_eq!(ctx_msgs.len(), 1); + } +} diff --git a/crates/ironclaw_engine/src/executor/loop_engine.rs b/crates/ironclaw_engine/src/executor/loop_engine.rs new file mode 100644 index 00000000000..f7fd43ae91e --- /dev/null +++ b/crates/ironclaw_engine/src/executor/loop_engine.rs @@ -0,0 +1,1315 @@ +//! Core execution loop — the replacement for `run_agentic_loop()`. +//! +//! The `ExecutionLoop` owns a thread and drives it through LLM call → +//! action execution → result processing → repeat cycles. Unlike the +//! existing delegate pattern, the loop is self-contained: all behavior +//! differences between thread types are handled via capability leases +//! and policy, not delegate implementations. + +use std::sync::Arc; + +use tracing::debug; + +use crate::capability::lease::LeaseManager; +use crate::capability::policy::PolicyEngine; +use crate::runtime::messaging::{SignalReceiver, ThreadOutcome}; +use crate::traits::effect::EffectExecutor; +use crate::traits::llm::LlmBackend; +use crate::types::error::EngineError; +use crate::types::event::EventKind; +use crate::types::message::ThreadMessage; +use crate::types::step::Step; +use crate::types::thread::{Thread, ThreadState}; + +const RUNTIME_CHECKPOINT_METADATA_KEY: &str = "runtime_checkpoint"; + +/// Persisted state from a prior execution, used to resume threads. +/// The Python orchestrator manages loop counters internally; Rust only +/// needs the opaque `persisted_state` blob to hand back on resume. +#[derive(Default)] +struct RuntimeCheckpoint { + persisted_state: serde_json::Value, +} + +/// The core execution loop for a thread. +pub struct ExecutionLoop { + pub thread: Thread, + llm: Arc, + effects: Arc, + leases: Arc, + policy: Arc, + signal_rx: SignalReceiver, + /// Stored for potential future use (e.g. user-scoped prompt overlays). + _user_id: String, + /// Optional capability registry for resolving capability-level policies. + capabilities: Option>, + /// Optional broadcast sender for live event streaming. + event_tx: Option>, + /// Optional retrieval engine for injecting prior knowledge into context. + retrieval: Option, + /// Optional Store for runtime prompt overlay loading and skill retrieval. + store: Option>, + /// Runtime platform metadata for self-awareness in system prompts. + platform_info: Option, +} + +impl ExecutionLoop { + pub fn new( + thread: Thread, + llm: Arc, + effects: Arc, + leases: Arc, + policy: Arc, + signal_rx: SignalReceiver, + user_id: String, + ) -> Self { + Self { + thread, + llm, + effects, + leases, + policy, + signal_rx, + _user_id: user_id, + capabilities: None, + event_tx: None, + retrieval: None, + store: None, + platform_info: None, + } + } + + /// Set the event broadcast sender for live status updates. + pub fn with_event_tx( + mut self, + tx: tokio::sync::broadcast::Sender, + ) -> Self { + self.event_tx = Some(tx); + self + } + + /// Set the capability registry for resolving capability-level policies. + pub fn with_capabilities( + mut self, + capabilities: Arc, + ) -> Self { + self.capabilities = Some(capabilities); + self + } + + /// Set the retrieval engine for injecting prior knowledge into context. + pub fn with_retrieval(mut self, retrieval: crate::memory::RetrievalEngine) -> Self { + self.retrieval = Some(retrieval); + self + } + + /// Set the Store for runtime prompt overlay loading and skill retrieval. + pub fn with_store(mut self, store: Arc) -> Self { + self.store = Some(store); + self + } + + /// Set platform metadata for self-awareness in system prompts. + pub fn with_platform_info(mut self, info: crate::executor::prompt::PlatformInfo) -> Self { + self.platform_info = Some(info); + self + } + + /// Add an event to the thread and broadcast it for live status updates. + fn emit_event(&mut self, kind: EventKind) { + let event = crate::types::event::ThreadEvent::new(self.thread.id, kind); + if let Some(ref tx) = self.event_tx { + let _ = tx.send(event.clone()); + } + self.thread.events.push(event); + self.thread.updated_at = chrono::Utc::now(); + } + + fn load_runtime_checkpoint(&self) -> RuntimeCheckpoint { + let persisted_state = self + .thread + .metadata + .get(RUNTIME_CHECKPOINT_METADATA_KEY) + .and_then(|value| value.get("persisted_state")) + .cloned() + .unwrap_or_else(|| serde_json::json!({})); + + RuntimeCheckpoint { persisted_state } + } + + fn clear_runtime_checkpoint(&mut self) { + if let Some(metadata) = self.thread.metadata.as_object_mut() { + metadata.remove(RUNTIME_CHECKPOINT_METADATA_KEY); + } + self.thread.updated_at = chrono::Utc::now(); + } + + async fn persist_runtime_state( + &self, + step: Option<&Step>, + persisted_event_count: &mut usize, + ) -> Result<(), EngineError> { + let Some(store) = self.store.as_ref() else { + return Ok(()); + }; + + // All three store writes are independent — run them in parallel. + let step_fut = async { + if let Some(step) = step { + store.save_step(step).await + } else { + Ok(()) + } + }; + + let new_event_count = self.thread.events.len(); + let events_fut = async { + if *persisted_event_count < new_event_count { + store + .append_events(&self.thread.events[*persisted_event_count..]) + .await + } else { + Ok(()) + } + }; + + let thread_fut = store.save_thread(&self.thread); + + let (step_res, events_res, thread_res) = tokio::join!(step_fut, events_fut, thread_fut); + step_res?; + events_res?; + thread_res?; + + *persisted_event_count = new_event_count; + Ok(()) + } + + /// Run the execution loop to completion. + pub async fn run(&mut self) -> Result { + let mut persisted_event_count = self.thread.events.len(); + let checkpoint = self.load_runtime_checkpoint(); + + // Transition to Running if this is a fresh start or restart from a resumable state. + if self.thread.state != ThreadState::Running { + self.thread.transition_to(ThreadState::Running, None)?; + } + + // Pre-fetch shared memory docs once — used by both prompt overlay and + // orchestrator loading, avoiding a duplicate Store query. + let system_docs = if let Some(store) = self.store.as_ref() { + match store.list_shared_memory_docs(self.thread.project_id).await { + Ok(docs) => docs, + Err(e) => { + debug!("failed to load shared docs for orchestrator: {e}"); + Vec::new() + } + } + } else { + Vec::new() + }; + + // Inject CodeAct/RLM system prompt if none exists + if !self + .thread + .messages + .iter() + .any(|m| m.role == crate::types::message::MessageRole::System) + { + // Fetch active leases (needed for action list) + let active_leases = self.leases.active_for_thread(self.thread.id).await; + let actions = match self.effects.available_actions(&active_leases).await { + Ok(a) => a, + Err(e) => { + debug!(thread_id = %self.thread.id, "failed to load actions for system prompt: {e}"); + Vec::new() + } + }; + // Build prompt using pre-fetched docs (no extra Store query) + let system_prompt = crate::executor::prompt::build_codeact_system_prompt_with_docs( + &actions, + &system_docs, + self.platform_info.as_ref(), + ); + + // Skill selection and injection happens in the Python orchestrator + // via __list_skills__() host function — not here in Rust. + + self.thread + .messages + .insert(0, ThreadMessage::system(system_prompt)); + } + self.persist_runtime_state(None, &mut persisted_event_count) + .await?; + + // Load versioned Python orchestrator using pre-fetched docs. + // Self-modification is disabled by default — only the compiled-in v0 + // runs unless explicitly opted in via ORCHESTRATOR_SELF_MODIFY=true. + let allow_self_modify = std::env::var("ORCHESTRATOR_SELF_MODIFY") + .map(|v| v == "true" || v == "1") + .unwrap_or(false); + let (orchestrator_code, orchestrator_version) = + crate::executor::orchestrator::load_orchestrator_from_docs( + &system_docs, + allow_self_modify, + ); + + debug!( + thread_id = %self.thread.id, + orchestrator_version, + "running Python orchestrator" + ); + + // Store version in thread metadata for rollback tracking + if let Some(metadata) = self.thread.metadata.as_object_mut() { + metadata.insert( + "orchestrator_version".into(), + serde_json::json!(orchestrator_version), + ); + } + + // Execute the Python orchestrator with host function dispatch + let result = crate::executor::orchestrator::execute_orchestrator( + &orchestrator_code, + &mut self.thread, + &self.llm, + &self.effects, + &self.leases, + &self.policy, + &mut self.signal_rx, + self.event_tx.as_ref(), + self.retrieval.as_ref(), + self.store.as_ref(), + &checkpoint.persisted_state, + ) + .await; + + // Post-cleanup: persist final state, track failures for auto-rollback + match result { + Ok(orch_result) => { + // Reset failure counter on success + if let Some(store) = self.store.as_ref() { + crate::executor::orchestrator::reset_orchestrator_failures( + store, + self.thread.project_id, + ) + .await; + } + let _ = &orch_result.tokens_used; + + self.clear_runtime_checkpoint(); + self.persist_runtime_state(None, &mut persisted_event_count) + .await?; + Ok(orch_result.outcome) + } + Err(e) => { + debug!( + thread_id = %self.thread.id, + error = %e, + orchestrator_version, + "orchestrator execution failed" + ); + + // Record failure for auto-rollback tracking + if let Some(store) = self.store.as_ref() { + crate::executor::orchestrator::record_orchestrator_failure( + store, + self.thread.project_id, + orchestrator_version, + ) + .await; + + // Emit rollback event if this version will be skipped next time + // (failure count was just incremented, so check >= threshold - 1) + if orchestrator_version > 0 { + self.emit_event(EventKind::OrchestratorRollback { + from_version: orchestrator_version, + to_version: orchestrator_version.saturating_sub(1), + reason: format!("execution failed: {e}"), + }); + } + } + + // Transition to failed if not already in a terminal state + if self.thread.state != ThreadState::Completed + && self.thread.state != ThreadState::Failed + && self.thread.state != ThreadState::Done + { + let _ = self.thread.transition_to( + ThreadState::Failed, + Some(format!("orchestrator error: {e}")), + ); + } + self.clear_runtime_checkpoint(); + self.persist_runtime_state(None, &mut persisted_event_count) + .await?; + Ok(ThreadOutcome::Failed { + error: format!("Orchestrator error: {e}"), + }) + } + } + } +} + +#[cfg(test)] +mod tests { + /// Extract a FINAL() answer from the LLM's text response. + /// + /// Matches `FINAL(...)` anywhere in the text, handling: + /// - Single-line: `FINAL("the answer")` + /// - Multi-line: `FINAL("""\n...\n""")` + /// - With or without quotes + fn extract_final_from_text(text: &str) -> Option { + let marker = "FINAL("; + let start = text.find(marker)?; + let content_start = start + marker.len(); + let remaining = &text[content_start..]; + + // Try triple-quoted string first: FINAL("""...""") + if remaining.starts_with("\"\"\"") { + let inner_start = 3; + if let Some(end) = remaining[inner_start..].find("\"\"\"") { + let answer = remaining[inner_start..inner_start + end].trim(); + if !answer.is_empty() { + return Some(answer.to_string()); + } + } + } + + // Try single/double quoted: FINAL("...") or FINAL('...') + if remaining.starts_with('"') || remaining.starts_with('\'') { + let quote = remaining.as_bytes()[0] as char; + if let Some(end) = remaining[1..].find(quote) { + let answer = &remaining[1..1 + end]; + if !answer.is_empty() { + return Some(answer.to_string()); + } + } + } + + // Unquoted: FINAL(some content here) — find matching close paren + let mut depth = 1; + for (i, ch) in remaining.char_indices() { + match ch { + '(' => depth += 1, + ')' => { + depth -= 1; + if depth == 0 { + let answer = remaining[..i].trim(); // safety: i is from char_indices(), always a valid boundary + if !answer.is_empty() { + return Some(answer.to_string()); + } + return None; + } + } + _ => {} + } + } + + None + } + use super::*; + use crate::runtime::messaging::ThreadSignal; + use crate::traits::effect::ThreadExecutionContext; + use crate::traits::llm::{LlmCallConfig, LlmOutput}; + use crate::types::capability::{ActionDef, CapabilityLease, EffectType, GrantedActions}; + use crate::types::project::ProjectId; + use crate::types::step::LlmResponse; + use crate::types::step::{ActionResult, TokenUsage}; + use crate::types::thread::{ThreadConfig, ThreadType}; + + use std::sync::Mutex; + use std::time::Duration; + + // ── Mock LLM ──────────────────────────────────────────── + + struct MockLlm { + responses: Mutex>, + } + + impl MockLlm { + fn new(responses: Vec) -> Self { + Self { + responses: Mutex::new(responses), + } + } + } + + #[async_trait::async_trait] + impl LlmBackend for MockLlm { + async fn complete( + &self, + _messages: &[ThreadMessage], + _actions: &[ActionDef], + _config: &LlmCallConfig, + ) -> Result { + let mut responses = self.responses.lock().unwrap(); + if responses.is_empty() { + Ok(LlmOutput { + response: LlmResponse::Text("(no more responses)".into()), + usage: TokenUsage::default(), + }) + } else { + Ok(responses.remove(0)) + } + } + + fn model_name(&self) -> &str { + "mock" + } + } + + // ── Mock EffectExecutor ───────────────────────────────── + + struct MockEffects { + results: Mutex>>, + actions: Vec, + } + + impl MockEffects { + fn new(actions: Vec, results: Vec>) -> Self { + Self { + results: Mutex::new(results), + actions, + } + } + } + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _action_name: &str, + _parameters: serde_json::Value, + _lease: &CapabilityLease, + _context: &ThreadExecutionContext, + ) -> Result { + let mut results = self.results.lock().unwrap(); + if results.is_empty() { + Ok(ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({"result": "ok"}), + is_error: false, + duration: Duration::from_millis(1), + }) + } else { + results.remove(0) + } + } + + async fn available_actions( + &self, + _leases: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(self.actions.clone()) + } + } + + // ── Helpers ───────────────────────────────────────────── + + fn text_response(text: &str) -> LlmOutput { + LlmOutput { + response: LlmResponse::Text(text.into()), + usage: TokenUsage { + input_tokens: 100, + output_tokens: 50, + ..Default::default() + }, + } + } + + fn action_response(action_name: &str, call_id: &str) -> LlmOutput { + LlmOutput { + response: LlmResponse::ActionCalls { + calls: vec![crate::types::step::ActionCall { + id: call_id.into(), + action_name: action_name.into(), + parameters: serde_json::json!({}), + }], + content: None, + }, + usage: TokenUsage { + input_tokens: 100, + output_tokens: 50, + ..Default::default() + }, + } + } + + fn test_action() -> ActionDef { + ActionDef { + name: "test_tool".into(), + description: "A test tool".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + } + } + + async fn make_loop( + llm_responses: Vec, + effect_results: Vec>, + config: ThreadConfig, + ) -> (ExecutionLoop, crate::runtime::messaging::SignalSender) { + let project_id = ProjectId::new(); + let thread = Thread::new( + "test goal", + ThreadType::Foreground, + project_id, + "test-user", + config, + ); + let tid = thread.id; + + let llm = Arc::new(MockLlm::new(llm_responses)); + let effects = Arc::new(MockEffects::new(vec![test_action()], effect_results)); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + + // Grant a default lease + leases + .grant(tid, "test_cap", GrantedActions::All, None, None) + .await + .unwrap(); + + let (tx, rx) = crate::runtime::messaging::signal_channel(16); + + let exec = ExecutionLoop::new(thread, llm, effects, leases, policy, rx, "test-user".into()); + (exec, tx) + } + + // ── Tests ─────────────────────────────────────────────── + + #[tokio::test] + async fn text_response_completes() { + let (mut exec, _tx) = make_loop( + vec![text_response("Hello!")], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Hello!")); + assert!(exec.thread.state.is_terminal() || exec.thread.state == ThreadState::Completed); + assert_eq!(exec.thread.step_count, 1); + assert!(exec.thread.total_tokens_used > 0); + } + + #[tokio::test] + async fn action_then_text() { + let (mut exec, _tx) = make_loop( + vec![ + action_response("test_tool", "call_1"), + text_response("Done!"), + ], + vec![Ok(ActionResult { + call_id: "call_1".into(), + action_name: "test_tool".into(), + output: serde_json::json!({"data": "result"}), + is_error: false, + duration: Duration::from_millis(5), + })], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Done!")); + assert_eq!(exec.thread.step_count, 2); + // Orchestrator-driven flow: working messages live in `internal_messages` + // (set by `sync_runtime_state` when the orchestrator persists state), + // while `thread.messages` only carries the system prompt + final + // assistant response. The full conversation transcript (system, + // assistant+actions, action_result, assistant) is in internal_messages. + assert!(exec.thread.internal_messages.len() >= 3); + } + + #[tokio::test] + async fn max_iterations_reached() { + // LLM always returns actions, so it never exits naturally + let many_actions: Vec = (0..5) + .map(|i| action_response("test_tool", &format!("call_{i}"))) + .collect(); + + let many_results: Vec> = (0..5) + .map(|i| { + Ok(ActionResult { + call_id: format!("call_{i}"), + action_name: "test_tool".into(), + output: serde_json::json!({"i": i}), + is_error: false, + duration: Duration::from_millis(1), + }) + }) + .collect(); + + let config = ThreadConfig { + max_iterations: 3, + ..ThreadConfig::default() + }; + + let (mut exec, _tx) = make_loop(many_actions, many_results, config).await; + + let outcome = exec.run().await.unwrap(); + // The last iteration forces text mode, and MockLlm returns action_response + // which gets treated as the 3rd iteration, then on the 3rd iteration force_text + // is set. But MockLlm ignores force_text. So we get MaxIterations after 3 iterations. + // Actually, max_iterations=3, and force_text is set when iteration >= max-1 = 2, + // so iteration 2 (0-indexed) has force_text. The MockLlm still returns action calls, + // so we loop 3 times and exit. + assert!(matches!( + outcome, + ThreadOutcome::MaxIterations | ThreadOutcome::Completed { .. } + )); + assert!(exec.thread.step_count <= 3); + } + + #[tokio::test] + async fn stop_signal_exits() { + // LLM would loop forever, but we send a stop signal + let many_actions: Vec = (0..100) + .map(|i| action_response("test_tool", &format!("call_{i}"))) + .collect(); + + let many_results: Vec> = (0..100) + .map(|i| { + Ok(ActionResult { + call_id: format!("call_{i}"), + action_name: "test_tool".into(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + }) + }) + .collect(); + + let (mut exec, tx) = make_loop(many_actions, many_results, ThreadConfig::default()).await; + + // Send stop before first iteration + tx.send(ThreadSignal::Stop).await.unwrap(); + + let outcome = exec.run().await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Stopped)); + } + + #[tokio::test] + async fn inject_message_appears_in_context() { + let (mut exec, tx) = make_loop( + vec![text_response("Got your message")], + vec![], + ThreadConfig::default(), + ) + .await; + + tx.send(ThreadSignal::InjectMessage(ThreadMessage::user( + "injected!", + ))) + .await + .unwrap(); + + let outcome = exec.run().await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { .. })); + assert!( + exec.thread + .messages + .iter() + .any(|m| m.content == "injected!") + ); + } + + #[tokio::test] + async fn tool_intent_nudge_injected() { + let (mut exec, _tx) = make_loop( + vec![ + text_response("Let me search for that"), + text_response("The answer is 42"), + ], + vec![], + ThreadConfig { + enable_tool_intent_nudge: true, + max_tool_intent_nudges: 2, + ..ThreadConfig::default() + }, + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "The answer is 42") + ); + assert_eq!(exec.thread.step_count, 2); + // Nudge is injected into the orchestrator's working messages, which + // are persisted as `thread.internal_messages` (not the user-visible + // `messages` transcript). + assert!( + exec.thread + .internal_messages + .iter() + .any(|m| m.content.contains("did not include any tool calls")) + ); + } + + #[tokio::test] + async fn events_are_recorded() { + let (mut exec, _tx) = make_loop( + vec![text_response("Hello!")], + vec![], + ThreadConfig::default(), + ) + .await; + + exec.run().await.unwrap(); + + let _event_kinds: Vec = exec + .thread + .events + .iter() + .map(|e| format!("{:?}", std::mem::discriminant(&e.kind))) + .collect(); + + // Should have: StateChanged(Created->Running), StepStarted, MessageAdded, + // StepCompleted, StateChanged(Running->Completed) + assert!(exec.thread.events.len() >= 4); + + // Verify first event is state change to Running + assert!(matches!( + &exec.thread.events[0].kind, + EventKind::StateChanged { + from: ThreadState::Created, + to: ThreadState::Running, + .. + } + )); + } + + // ── CodeAct / RLM tests ───────────────────────────────── + + fn code_response(code: &str) -> LlmOutput { + LlmOutput { + response: LlmResponse::Code { + code: code.into(), + content: Some(format!("```repl\n{code}\n```")), + }, + usage: TokenUsage { + input_tokens: 100, + output_tokens: 80, + ..Default::default() + }, + } + } + + #[tokio::test] + async fn codeact_simple_final() { + // LLM outputs Python code that calls FINAL() + let (mut exec, _tx) = make_loop( + vec![code_response("FINAL('The answer is 42')")], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "The answer is 42") + ); + assert_eq!(exec.thread.step_count, 1); + } + + #[tokio::test] + async fn codeact_tool_call_then_final() { + // LLM outputs code that calls a tool, then uses the result + let (mut exec, _tx) = make_loop( + vec![code_response( + "result = test_tool()\nprint(result)\nFINAL('got result')", + )], + vec![Ok(ActionResult { + call_id: "code_call_1".into(), + action_name: "test_tool".into(), + output: serde_json::json!({"data": "hello from tool"}), + is_error: false, + duration: Duration::from_millis(5), + })], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "got result") + ); + // Should have at least 1 action result recorded + assert!(!exec.thread.internal_messages.is_empty()); + } + + #[tokio::test] + async fn codeact_pure_python_computation() { + // LLM outputs pure Python with no tool calls — just computation + FINAL + let (mut exec, _tx) = make_loop( + vec![code_response( + "numbers = [1, 2, 3, 4, 5]\ntotal = sum(numbers)\nFINAL(f'Sum is {total}')", + )], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Sum is 15") + ); + } + + #[tokio::test] + async fn codeact_multi_step() { + // First iteration: code runs but no FINAL — returns output + // Second iteration: LLM sees output and calls FINAL + let (mut exec, _tx) = make_loop( + vec![ + code_response("x = 10 + 20\nprint(f'x = {x}')"), + code_response("FINAL('done, x was 30')"), + ], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "done, x was 30") + ); + assert_eq!(exec.thread.step_count, 2); + // The first step's stdout/output metadata is persisted as part of + // the orchestrator's working messages → `internal_messages`. + assert!( + exec.thread + .internal_messages + .iter() + .any(|m| m.content.contains("x = 30")) + ); + } + + #[tokio::test] + async fn codeact_error_recovery() { + // First iteration: code has an error (NameError) + // Second iteration: LLM sees the error and fixes it + let (mut exec, _tx) = make_loop( + vec![ + code_response("result = undefined_var + 1"), + code_response("FINAL('recovered')"), + ], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "recovered") + ); + assert_eq!(exec.thread.step_count, 2); + // First step should have error in output metadata + assert!( + exec.thread + .internal_messages + .iter() + .any(|m| { m.content.contains("NameError") || m.content.contains("Error") }) + ); + } + + #[tokio::test] + async fn codeact_context_variables_available() { + // Code accesses the `goal` and `context` variables injected by the engine + let (mut exec, _tx) = make_loop( + vec![code_response( + "FINAL(f'Goal: {goal}, Messages: {len(context)}')", + )], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + // Should have access to goal="test goal" and context (list of messages) + match outcome { + ThreadOutcome::Completed { response: Some(r) } => { + assert!(r.contains("Goal: test goal"), "got: {r}"); + assert!(r.contains("Messages:"), "got: {r}"); + } + other => panic!("expected Completed, got {other:?}"), + } + } + + #[tokio::test] + async fn codeact_multiple_tool_calls_in_loop() { + // Code calls a tool 3 times in a for loop + let (mut exec, _tx) = make_loop( + vec![code_response( + "results = []\nfor i in range(3):\n r = test_tool()\n results.append(r)\nFINAL(f'Got {len(results)} results')", + )], + vec![ + Ok(ActionResult { + call_id: "code_call_1".into(), + action_name: "test_tool".into(), + output: serde_json::json!({"i": 0}), + is_error: false, + duration: Duration::from_millis(1), + }), + Ok(ActionResult { + call_id: "code_call_2".into(), + action_name: "test_tool".into(), + output: serde_json::json!({"i": 1}), + is_error: false, + duration: Duration::from_millis(1), + }), + Ok(ActionResult { + call_id: "code_call_3".into(), + action_name: "test_tool".into(), + output: serde_json::json!({"i": 2}), + is_error: false, + duration: Duration::from_millis(1), + }), + ], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Got 3 results") + ); + } + + #[tokio::test] + async fn codeact_llm_query_recursive() { + // Code calls llm_query() — which calls the MockLlm recursively. + // The MockLlm will return the next response in its queue for the sub-call. + let (mut exec, _tx) = make_loop( + vec![ + // First response: code that calls llm_query + code_response( + "answer = llm_query('What is 2+2?')\nFINAL(f'Sub-agent said: {answer}')", + ), + // This text response will be consumed by the llm_query sub-call + // (MockLlm pops from the same queue) + ], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + // llm_query will get "(no more responses)" since the queue only had + // the code response. That's fine — it tests the plumbing. + match outcome { + ThreadOutcome::Completed { response: Some(r) } => { + assert!(r.contains("Sub-agent said:"), "got: {r}"); + } + other => panic!("expected Completed, got {other:?}"), + } + } + + #[tokio::test] + async fn codeact_final_in_text_response() { + // LLM outputs FINAL() as plain text (not in a code block) + // This is the Hyperliquid case — model writes explanation + FINAL() + let (mut exec, _tx) = make_loop( + vec![text_response( + "Based on my analysis, the answer is clear.\n\nFINAL(\"Revenue grows with volume\")", + )], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + assert!( + matches!(outcome, ThreadOutcome::Completed { response: Some(ref r) } if r == "Revenue grows with volume"), + "got: {outcome:?}" + ); + } + + #[tokio::test] + async fn codeact_final_triple_quoted_in_text() { + // FINAL with triple-quoted multi-line string in plain text + let (mut exec, _tx) = make_loop( + vec![text_response( + "Here's the summary:\n\nFINAL(\"\"\"\nLine 1\nLine 2\nLine 3\n\"\"\")", + )], + vec![], + ThreadConfig::default(), + ) + .await; + + let outcome = exec.run().await.unwrap(); + match outcome { + ThreadOutcome::Completed { response: Some(r) } => { + assert!(r.contains("Line 1"), "got: {r}"); + assert!(r.contains("Line 3"), "got: {r}"); + } + other => panic!("expected Completed, got {other:?}"), + } + } + + // ── extract_final_from_text unit tests ────────────────── + + #[test] + fn final_double_quoted() { + let text = "some text\nFINAL(\"the answer\")"; + assert_eq!(extract_final_from_text(text).unwrap(), "the answer"); + } + + #[test] + fn final_single_quoted() { + let text = "FINAL('hello world')"; + assert_eq!(extract_final_from_text(text).unwrap(), "hello world"); + } + + #[test] + fn final_triple_quoted() { + let text = "FINAL(\"\"\"\nmulti\nline\n\"\"\")"; + assert_eq!(extract_final_from_text(text).unwrap(), "multi\nline"); + } + + #[test] + fn final_unquoted() { + let text = "FINAL(42)"; + assert_eq!(extract_final_from_text(text).unwrap(), "42"); + } + + #[test] + fn final_with_nested_parens() { + let text = "FINAL(f'result is {len(items)}')"; + assert_eq!( + extract_final_from_text(text).unwrap(), + "f'result is {len(items)}'" + ); + } + + #[test] + fn no_final_returns_none() { + assert!(extract_final_from_text("just regular text").is_none()); + } + + #[test] + fn final_after_long_text() { + let text = "A very long explanation...\n\n🔚 Final Thought\n\nFINAL(\"the conclusion\")"; + assert_eq!(extract_final_from_text(text).unwrap(), "the conclusion"); + } + + // ── call_id propagation through orchestrator pipeline ──── + // + // These tests verify the end-to-end flow: LLM returns ActionCalls with + // call_ids → orchestrator executes them → ActionResult messages on the + // thread have correct call_ids (not empty). This catches the class of + // bugs that caused OpenAI/Codex HTTP 400 rejections. + + #[tokio::test] + async fn action_result_messages_have_correct_call_id() { + // LLM returns a tool call, then a text response + let (mut exec, _tx) = make_loop( + vec![ + action_response("test_tool", "call_xK9mZq123"), + text_response("Done!"), + ], + vec![Ok(ActionResult { + call_id: String::new(), // EffectExecutor returns empty + action_name: "test_tool".into(), + output: serde_json::json!({"data": "result"}), + is_error: false, + duration: Duration::from_millis(5), + })], + ThreadConfig::default(), + ) + .await; + + exec.run().await.unwrap(); + + // Find the ActionResult message in the internal orchestrator transcript + let action_results: Vec<_> = exec + .thread + .internal_messages + .iter() + .filter(|m| m.role == crate::types::message::MessageRole::ActionResult) + .collect(); + + assert!( + !action_results.is_empty(), + "thread should have at least one internal ActionResult message" + ); + + for msg in &action_results { + let call_id = msg.action_call_id.as_deref().unwrap_or(""); + assert!( + !call_id.is_empty(), + "ActionResult message must have non-empty call_id, got empty for tool '{}'", + msg.action_name.as_deref().unwrap_or("?") + ); + } + } + + /// Verify that the ActionExecuted event carries the call_id from the LLM. + #[tokio::test] + async fn action_executed_events_carry_call_id() { + let (mut exec, _tx) = make_loop( + vec![ + action_response("test_tool", "call_evt_id_42"), + text_response("ok"), + ], + vec![Ok(ActionResult { + call_id: String::new(), + action_name: "test_tool".into(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + })], + ThreadConfig::default(), + ) + .await; + + exec.run().await.unwrap(); + + let exec_events: Vec<_> = exec + .thread + .events + .iter() + .filter_map(|e| match &e.kind { + EventKind::ActionExecuted { call_id, .. } => Some(call_id.clone()), + _ => None, + }) + .collect(); + + assert!(!exec_events.is_empty(), "should have ActionExecuted events"); + for call_id in &exec_events { + assert!( + !call_id.is_empty(), + "ActionExecuted event must have non-empty call_id" + ); + } + } + + /// When a tool call fails (no lease), the internal ActionResult message and + /// ActionFailed event must still carry the original call_id. + #[tokio::test] + async fn failed_action_preserves_call_id_in_message_and_event() { + let project_id = ProjectId::new(); + let thread = Thread::new( + "test", + ThreadType::Foreground, + project_id, + "test-user", + ThreadConfig::default(), + ); + let tid = thread.id; + + // Create a tool that requires a separate capability + let missing_action = ActionDef { + name: "restricted_tool".into(), + description: "A tool with no lease".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::WriteExternal], + requires_approval: false, + }; + + let llm = Arc::new(MockLlm::new(vec![ + // LLM calls a tool the thread has no lease for + LlmOutput { + response: LlmResponse::ActionCalls { + calls: vec![crate::types::step::ActionCall { + id: "call_nolease_xyz".into(), + action_name: "restricted_tool".into(), + parameters: serde_json::json!({}), + }], + content: None, + }, + usage: TokenUsage::default(), + }, + text_response("I couldn't access that tool"), + ])); + let effects = Arc::new(MockEffects::new(vec![missing_action], vec![])); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + + // Grant a lease that does NOT cover "restricted_tool" + leases + .grant(tid, "basic_cap", GrantedActions::All, None, None) + .await + .unwrap(); + + let (_tx, rx) = crate::runtime::messaging::signal_channel(16); + let mut exec = + ExecutionLoop::new(thread, llm, effects, leases, policy, rx, "test-user".into()); + + exec.run().await.unwrap(); + + // Check internal ActionResult messages + let action_results: Vec<_> = exec + .thread + .internal_messages + .iter() + .filter(|m| m.role == crate::types::message::MessageRole::ActionResult) + .collect(); + + for msg in &action_results { + let call_id = msg.action_call_id.as_deref().unwrap_or(""); + assert!( + !call_id.is_empty(), + "even failed ActionResult must have call_id" + ); + } + + // Check ActionFailed events + let fail_events: Vec<_> = exec + .thread + .events + .iter() + .filter_map(|e| match &e.kind { + EventKind::ActionFailed { + call_id, + action_name, + .. + } => Some((call_id.clone(), action_name.clone())), + _ => None, + }) + .collect(); + + for (call_id, _name) in &fail_events { + assert!(!call_id.is_empty(), "ActionFailed event must have call_id"); + } + } + + /// Verify the trace analyzer does NOT flag any issues on a clean + /// action execution (no empty call_ids). + #[tokio::test] + async fn trace_analysis_clean_after_successful_tool_use() { + let (mut exec, _tx) = make_loop( + vec![ + action_response("test_tool", "call_clean_id"), + text_response("All done"), + ], + vec![Ok(ActionResult { + call_id: String::new(), + action_name: "test_tool".into(), + output: serde_json::json!({"status": "ok"}), + is_error: false, + duration: Duration::from_millis(3), + })], + ThreadConfig::default(), + ) + .await; + + exec.run().await.unwrap(); + + let trace = crate::executor::trace::build_trace(&exec.thread); + let empty_id_issues: Vec<_> = trace + .issues + .iter() + .filter(|i| i.category == "empty_call_id") + .collect(); + + assert!( + empty_id_issues.is_empty(), + "clean execution should have no empty_call_id issues, got: {empty_id_issues:?}" + ); + } +} diff --git a/crates/ironclaw_engine/src/executor/mod.rs b/crates/ironclaw_engine/src/executor/mod.rs new file mode 100644 index 00000000000..4b9d4499258 --- /dev/null +++ b/crates/ironclaw_engine/src/executor/mod.rs @@ -0,0 +1,17 @@ +//! Step execution. +//! +//! - [`ExecutionLoop`] — core loop replacing `run_agentic_loop()` +//! - [`structured`] — Tier 0 action execution (structured tool calls) +//! - [`context`] — context building for LLM calls +//! - [`intent`] — tool intent nudge detection + +pub mod compaction; +pub mod context; +pub mod loop_engine; +pub mod orchestrator; +pub mod prompt; +pub mod scripting; +pub mod structured; +pub mod trace; + +pub use loop_engine::ExecutionLoop; diff --git a/crates/ironclaw_engine/src/executor/orchestrator.rs b/crates/ironclaw_engine/src/executor/orchestrator.rs new file mode 100644 index 00000000000..bf165984907 --- /dev/null +++ b/crates/ironclaw_engine/src/executor/orchestrator.rs @@ -0,0 +1,3863 @@ +//! Python orchestrator — the self-modifiable execution loop. +//! +//! Replaces the Rust `ExecutionLoop::run()` with versioned Python code +//! executed via Monty. The orchestrator is the "glue layer" between the +//! LLM and tools — tool dispatch, output formatting, state management, +//! truncation — all in Python, patchable by the self-improvement Mission. +//! +//! Host functions exposed to the orchestrator Python: +//! - `__llm_complete__` — make an LLM call +//! - `__execute_code_step__` — run user CodeAct code in a nested Monty VM +//! - `__execute_action__` — execute a single tool action +//! - `__execute_actions_parallel__` — execute multiple tool actions concurrently +//! - `__check_signals__` — poll for stop/inject signals +//! - `__emit_event__` — broadcast a ThreadEvent +//! - `__save_checkpoint__` — persist thread state +//! - `__transition_to__` — change thread state (validated) +//! - `__retrieve_docs__` — query memory docs +//! - `__check_budget__` — remaining tokens/time/USD +//! - `__get_actions__` — available tool definitions + +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; + +use std::collections::HashMap; + +use monty::{ + ExtFunctionResult, LimitedTracker, MontyObject, MontyRun, NameLookupResult, PrintWriter, + ResourceLimits, RunProgress, +}; +use tracing::{debug, warn}; + +use crate::capability::lease::LeaseManager; +use crate::capability::policy::PolicyEngine; +use crate::memory::RetrievalEngine; +use crate::runtime::lease_refresh::reconcile_dynamic_tool_lease; +use crate::runtime::messaging::{SignalReceiver, ThreadOutcome, ThreadSignal}; +use crate::traits::effect::{EffectExecutor, ThreadExecutionContext}; +use crate::traits::llm::{LlmBackend, LlmCallConfig}; +use crate::traits::store::Store; +use crate::types::error::EngineError; +use crate::types::event::{EventKind, ThreadEvent, summarize_params}; +use crate::types::message::ThreadMessage; +use crate::types::project::ProjectId; +use crate::types::shared_owner_id; +use crate::types::step::{ActionCall, StepId, TokenUsage}; +use crate::types::thread::{ActiveSkillProvenance, Thread, ThreadState}; +use ironclaw_common::ValidTimezone; + +use super::scripting::{execute_code, json_to_monty, monty_to_json, monty_to_string}; + +/// The compiled-in default orchestrator (v0). +pub(crate) const DEFAULT_ORCHESTRATOR: &str = include_str!("../../orchestrator/default.py"); + +/// Well-known title for orchestrator code in the Store. +pub const ORCHESTRATOR_TITLE: &str = "orchestrator:main"; + +/// Well-known tag for orchestrator code docs. +pub const ORCHESTRATOR_TAG: &str = "orchestrator_code"; + +/// Result of running the orchestrator. +pub struct OrchestratorResult { + /// The thread outcome parsed from the orchestrator's return value. + pub outcome: ThreadOutcome, + /// Total tokens used by LLM calls within the orchestrator. + pub tokens_used: TokenUsage, +} + +/// Extract source_channel from thread metadata (set by ConversationManager). +fn thread_source_channel(thread: &Thread) -> Option { + thread + .metadata + .get("source_channel") + .and_then(|v| v.as_str()) + .map(String::from) +} + +/// Extract and validate user_timezone from thread metadata (set by bridge router). +fn thread_user_timezone(thread: &Thread) -> Option { + thread + .metadata + .get("user_timezone") + .and_then(|v| v.as_str()) + .and_then(ValidTimezone::parse) +} + +fn normalize_pause_outcome( + thread: &mut Thread, + outcome: &ThreadOutcome, +) -> Result<(), EngineError> { + if matches!(outcome, ThreadOutcome::GatePaused { .. }) && thread.state != ThreadState::Waiting { + thread.transition_to( + ThreadState::Waiting, + Some("waiting on external gate resolution".into()), + )?; + } + Ok(()) +} + +/// Resource limits for the orchestrator VM. +fn orchestrator_limits() -> ResourceLimits { + ResourceLimits::new() + .max_duration(std::time::Duration::from_secs(300)) // 5 min (longer than user code) + .max_allocations(5_000_000) + .max_memory(128 * 1024 * 1024) // 128 MB +} + +/// Maximum consecutive failures before auto-rollback. +const MAX_FAILURES_BEFORE_ROLLBACK: u64 = 3; + +/// Well-known title for orchestrator failure tracking. +const FAILURE_TRACKER_TITLE: &str = "orchestrator:failures"; +const LEASE_REFRESH_WARN_INTERVAL_SECS: u64 = 60; + +fn warn_on_lease_refresh_failure(context: &'static str, error: &crate::types::error::EngineError) { + static LAST_WARN_TS: AtomicU64 = AtomicU64::new(0); + + let now = chrono::Utc::now().timestamp().max(0) as u64; + let last = LAST_WARN_TS.load(Ordering::Relaxed); + if now.saturating_sub(last) >= LEASE_REFRESH_WARN_INTERVAL_SECS + && LAST_WARN_TS + .compare_exchange(last, now, Ordering::Relaxed, Ordering::Relaxed) + .is_ok() + { + warn!(context, error = %error, "dynamic lease refresh failed"); + } else { + debug!(context, error = %error, "dynamic lease refresh failed"); + } +} + +/// Load orchestrator code: runtime version from Store, or compiled-in default. +/// +/// When `allow_self_modify` is false, always uses the compiled-in default +/// regardless of any runtime versions in the Store. This is the safe default +/// for production — runtime orchestrator patching is opt-in. +/// +/// Checks the failure tracker — if the latest version has >= 3 consecutive +/// failures, falls back to the previous version (or compiled-in default). +pub async fn load_orchestrator( + store: Option<&Arc>, + project_id: ProjectId, + allow_self_modify: bool, +) -> (String, u64) { + if !allow_self_modify { + debug!("orchestrator self-modification disabled, using compiled-in default (v0)"); + return (DEFAULT_ORCHESTRATOR.to_string(), 0); + } + + let Some(store) = store else { + debug!("using compiled-in default orchestrator (v0, no store)"); + return (DEFAULT_ORCHESTRATOR.to_string(), 0); + }; + + let docs = match store.list_shared_memory_docs(project_id).await { + Ok(d) => d, + Err(_) => { + debug!("using compiled-in default orchestrator (v0, store error)"); + return (DEFAULT_ORCHESTRATOR.to_string(), 0); + } + }; + + load_orchestrator_from_docs(&docs, allow_self_modify) +} + +/// Load orchestrator from pre-fetched system memory docs. +/// +/// When the caller already has the `list_memory_docs` result, use this to +/// avoid a duplicate Store query. Returns `(code, version)`. +/// +/// Respects `allow_self_modify` — when false, always returns the compiled-in +/// default. The caller in `loop_engine.rs` passes this from engine config. +pub fn load_orchestrator_from_docs( + docs: &[crate::types::memory::MemoryDoc], + allow_self_modify: bool, +) -> (String, u64) { + if !allow_self_modify { + return (DEFAULT_ORCHESTRATOR.to_string(), 0); + } + + // Find all orchestrator versions, sorted by version number descending + let mut versions: Vec<_> = docs + .iter() + .filter(|d| d.title == ORCHESTRATOR_TITLE && d.tags.contains(&ORCHESTRATOR_TAG.to_string())) + .collect(); + versions.sort_by(|a, b| { + let va = a + .metadata + .get("version") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + let vb = b + .metadata + .get("version") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + vb.cmp(&va) // descending + }); + + if versions.is_empty() { + debug!("using compiled-in default orchestrator (v0)"); + return (DEFAULT_ORCHESTRATOR.to_string(), 0); + } + + // Check failure count for the latest version + let failures = load_failure_count(docs); + + for doc in &versions { + let version = doc + .metadata + .get("version") + .and_then(|v| v.as_u64()) + .unwrap_or(1); + + // Skip versions with too many failures (only check the latest) + if version + == versions[0] + .metadata + .get("version") + .and_then(|v| v.as_u64()) + .unwrap_or(1) + && failures >= MAX_FAILURES_BEFORE_ROLLBACK + { + debug!( + version, + failures, "orchestrator version has too many failures, skipping" + ); + continue; + } + + debug!(version, "loaded runtime orchestrator"); + return (doc.content.clone(), version); + } + + // All versions failed — fall back to compiled-in default + debug!("all orchestrator versions failed, using compiled-in default (v0)"); + (DEFAULT_ORCHESTRATOR.to_string(), 0) +} + +/// Record a failure for the current orchestrator version. +pub async fn record_orchestrator_failure( + store: &Arc, + project_id: ProjectId, + version: u64, +) { + use crate::types::memory::{DocType, MemoryDoc}; + + let docs = match store.list_shared_memory_docs(project_id).await { + Ok(docs) => docs, + Err(e) => { + debug!("failed to list memory docs for failure tracker: {e}"); + return; + } + }; + let existing = docs.iter().find(|d| d.title == FAILURE_TRACKER_TITLE); + + let mut tracker = if let Some(doc) = existing { + doc.clone() + } else { + MemoryDoc::new( + project_id, + shared_owner_id(), + DocType::Note, + FAILURE_TRACKER_TITLE, + "", + ) + .with_tags(vec!["orchestrator_meta".to_string()]) + }; + + // Store failure count as JSON in content: {"version": N, "count": M} + let current: serde_json::Value = + serde_json::from_str(&tracker.content).unwrap_or(serde_json::json!({})); + let current_version = current.get("version").and_then(|v| v.as_u64()).unwrap_or(0); + let current_count = current.get("count").and_then(|v| v.as_u64()).unwrap_or(0); + + let new_count = if current_version == version { + current_count + 1 + } else { + 1 // new version, reset count + }; + + tracker.content = serde_json::json!({ + "version": version, + "count": new_count, + }) + .to_string(); + tracker.updated_at = chrono::Utc::now(); + + if let Err(e) = store.save_memory_doc(&tracker).await { + debug!("failed to save orchestrator failure tracker: {e}"); + } + + debug!(version, count = new_count, "recorded orchestrator failure"); +} + +/// Reset the failure counter (called after successful execution). +pub async fn reset_orchestrator_failures(store: &Arc, project_id: ProjectId) { + let docs = store + .list_shared_memory_docs(project_id) + .await + .unwrap_or_default(); + let existing = docs.iter().find(|d| d.title == FAILURE_TRACKER_TITLE); + + if let Some(doc) = existing { + let mut tracker = doc.clone(); + tracker.content = serde_json::json!({"version": 0, "count": 0}).to_string(); + tracker.updated_at = chrono::Utc::now(); + let _ = store.save_memory_doc(&tracker).await; + } +} + +/// Load failure count for the latest orchestrator version. +fn load_failure_count(docs: &[crate::types::memory::MemoryDoc]) -> u64 { + docs.iter() + .find(|d| d.title == FAILURE_TRACKER_TITLE) + .and_then(|d| serde_json::from_str::(&d.content).ok()) + .and_then(|v| v.get("count").and_then(|c| c.as_u64())) + .unwrap_or(0) +} + +/// Execute the orchestrator Python code with host function dispatch. +/// +/// This is the core function that replaces `ExecutionLoop::run()`'s inner loop. +/// The orchestrator Python calls host functions via Monty's suspension mechanism, +/// and this function handles each suspension by delegating to the appropriate +/// Rust implementation. +#[allow(clippy::too_many_arguments)] +pub async fn execute_orchestrator( + code: &str, + thread: &mut Thread, + llm: &Arc, + effects: &Arc, + leases: &Arc, + policy: &Arc, + signal_rx: &mut SignalReceiver, + event_tx: Option<&tokio::sync::broadcast::Sender>, + retrieval: Option<&RetrievalEngine>, + store: Option<&Arc>, + persisted_state: &serde_json::Value, +) -> Result { + let mut total_tokens = TokenUsage::default(); + + // Build context variables for the orchestrator + let (input_names, input_values) = build_orchestrator_inputs(thread, persisted_state); + + // Parse and compile + let runner = match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + MontyRun::new(code.to_string(), "orchestrator.py", input_names) + })) { + Ok(Ok(runner)) => runner, + Ok(Err(e)) => { + return Err(EngineError::Effect { + reason: format!("Orchestrator parse error: {e}"), + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during orchestrator parsing".into(), + }); + } + }; + + // Start execution + let mut stdout = String::new(); + let tracker = LimitedTracker::new(orchestrator_limits()); + + let run_result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + runner.start(input_values, tracker, PrintWriter::Collect(&mut stdout)) + })); + + let mut progress = match run_result { + Ok(Ok(p)) => p, + Ok(Err(e)) => { + return Err(EngineError::Effect { + reason: format!("Orchestrator runtime error: {e}"), + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during orchestrator start".into(), + }); + } + }; + + // Drive the orchestrator dispatch loop + let mut final_result: Option = None; + + loop { + match progress { + RunProgress::Complete(obj) => { + // Use FINAL result if set, otherwise fall back to VM return value + let result = if let Some(ref fr) = final_result { + fr.clone() + } else { + monty_to_json(&obj) + }; + sync_runtime_state(thread, result.get("state")); + let outcome = parse_outcome(&result); + sync_visible_outcome(thread, &outcome); + normalize_pause_outcome(thread, &outcome)?; + return Ok(OrchestratorResult { + outcome, + tokens_used: total_tokens, + }); + } + + RunProgress::FunctionCall(call) => { + let action_name = call.function_name.clone(); + let args = &call.args; + let kwargs = &call.kwargs; + + debug!(action = %action_name, "orchestrator: host function call"); + + let ext_result = match action_name.as_str() { + // FINAL(result) — orchestrator returns its outcome + "FINAL" => { + let val = args.first().map(monty_to_json).unwrap_or_default(); + final_result = Some(val); + ExtFunctionResult::Return(MontyObject::None) + } + + // __llm_complete__(messages, actions, config) + "__llm_complete__" => { + handle_llm_complete( + args, + kwargs, + thread, + LlmCompleteDeps { + llm, + effects, + leases, + store, + }, + &mut total_tokens, + ) + .await + } + + // __execute_code_step__(code, state) + "__execute_code_step__" => { + handle_execute_code_step( + args, kwargs, thread, llm, effects, leases, policy, event_tx, + ) + .await + } + + // __execute_action__(name, params, call_id=...) + "__execute_action__" => { + handle_execute_action( + args, kwargs, thread, effects, leases, policy, event_tx, + ) + .await + } + + // __execute_actions_parallel__(calls) + "__execute_actions_parallel__" => { + handle_execute_actions_parallel( + args, thread, effects, leases, policy, event_tx, + ) + .await + } + + // __check_signals__() + "__check_signals__" => handle_check_signals(signal_rx, thread), + + // __emit_event__(kind, **data) + "__emit_event__" => handle_emit_event(args, kwargs, thread, event_tx), + + // __save_checkpoint__(state, counters) + "__save_checkpoint__" => handle_save_checkpoint(args, kwargs, thread), + + // __transition_to__(state, reason) + "__transition_to__" => handle_transition_to(args, kwargs, thread), + + // __retrieve_docs__(goal, max_docs) + "__retrieve_docs__" => { + handle_retrieve_docs(args, kwargs, thread, retrieval).await + } + + // __check_budget__()" + "__check_budget__" => handle_check_budget(thread), + + // __get_actions__() + "__get_actions__" => handle_get_actions(thread, effects, leases, store).await, + + // __list_skills__(max_candidates, max_tokens) + "__list_skills__" => handle_list_skills(args, thread, store).await, + + // __record_skill_usage__(doc_id, success) + "__record_skill_usage__" => handle_record_skill_usage(args, store).await, + + // __regex_match__(pattern, text) -> bool + // Evaluates a regex against text using Rust's regex crate. + // Invalid patterns return False silently. Monty has no `re` + // module, so this host function bridges the gap for the + // skill selector's pattern-based scoring. + "__regex_match__" => handle_regex_match(args), + + // __set_active_skills__(skills) + "__set_active_skills__" => handle_set_active_skills(args, thread), + + // Unknown — let Monty resolve it (user-defined functions, builtins) + other => ExtFunctionResult::NotFound(other.to_string()), + }; + + // Resume the orchestrator VM + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + call.resume(ext_result, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + return Err(EngineError::Effect { + reason: format!("Orchestrator error after resume: {e}"), + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during orchestrator resume".into(), + }); + } + } + + // If FINAL was called, the VM should complete on next iteration + if final_result.is_some() { + continue; + } + } + + RunProgress::NameLookup(lookup) => { + // Undefined variable — resume with NameError + let name = lookup.name.clone(); + debug!(name = %name, "orchestrator: unresolved name"); + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + lookup.resume( + NameLookupResult::Undefined, + PrintWriter::Collect(&mut stdout), + ) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + return Err(EngineError::Effect { + reason: format!("Orchestrator NameError '{name}': {e}"), + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: format!("Monty panic on NameLookup '{name}'"), + }); + } + } + } + + RunProgress::OsCall(_) => { + return Err(EngineError::Effect { + reason: "Orchestrator attempted OS call (blocked)".into(), + }); + } + + RunProgress::ResolveFutures(_) => { + return Err(EngineError::Effect { + reason: "Orchestrator attempted async (not supported)".into(), + }); + } + } + } +} + +// ── Host function handlers ────────────────────────────────── + +struct LlmCompleteDeps<'a> { + llm: &'a Arc, + effects: &'a Arc, + leases: &'a Arc, + store: Option<&'a Arc>, +} + +/// Handle `__llm_complete__(messages, actions, config)`. +/// +/// Calls the LLM and returns the response as a dict: +/// `{type: "text"|"code"|"actions", content/code/calls: ..., usage: {...}}` +/// +async fn handle_llm_complete( + args: &[MontyObject], + _kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, + deps: LlmCompleteDeps<'_>, + total_tokens: &mut TokenUsage, +) -> ExtFunctionResult { + use crate::types::step::LlmResponse; + + let explicit_messages = args.first().map(monty_to_json).filter(|v| !v.is_null()); + let explicit_config = args.get(2).map(monty_to_json).filter(|v| !v.is_null()); + let messages = explicit_messages + .as_ref() + .and_then(json_to_thread_messages) + .unwrap_or_else(|| thread.messages.clone()); + + if let Err(e) = reconcile_dynamic_tool_lease( + thread, + deps.effects, + deps.leases, + deps.store, + &crate::LeasePlanner::new(), + ) + .await + { + warn_on_lease_refresh_failure("llm_complete", &e); + } + + let active_leases = deps.leases.active_for_thread(thread.id).await; + let actions = deps + .effects + .available_actions(&active_leases) + .await + .unwrap_or_default(); + + let config = LlmCallConfig { + max_tokens: explicit_config + .as_ref() + .and_then(|cfg| cfg.get("max_tokens")) + .and_then(|v| v.as_u64()) + .and_then(|v| u32::try_from(v).ok()), + temperature: explicit_config + .as_ref() + .and_then(|cfg| cfg.get("temperature")) + .and_then(|v| v.as_f64()) + .map(|v| v as f32), + force_text: explicit_config + .as_ref() + .and_then(|cfg| cfg.get("force_text")) + .and_then(|v| v.as_bool()) + .unwrap_or(false), + depth: thread.config.depth, + model: explicit_config + .as_ref() + .and_then(|cfg| cfg.get("model")) + .and_then(|v| v.as_str()) + .map(String::from), + metadata: HashMap::new(), + }; + + match deps.llm.complete(&messages, &actions, &config).await { + Ok(output) => { + total_tokens.input_tokens += output.usage.input_tokens; + total_tokens.output_tokens += output.usage.output_tokens; + total_tokens.cost_usd += output.usage.cost_usd; + + let usage = serde_json::json!({ + "input_tokens": output.usage.input_tokens, + "output_tokens": output.usage.output_tokens, + "cost_usd": output.usage.cost_usd, + }); + + let result = match output.response { + LlmResponse::Text(text) => { + serde_json::json!({"type": "text", "content": text, "usage": usage}) + } + LlmResponse::Code { code, .. } => { + serde_json::json!({"type": "code", "code": code, "usage": usage}) + } + LlmResponse::ActionCalls { calls, content } => { + // Single source of truth for the Python interchange + // shape — must round-trip via `python_json_to_action_calls`. + let calls_json = action_calls_to_python_json(&calls); + serde_json::json!({ + "type": "actions", + "content": content, + "calls": calls_json, + "usage": usage + }) + } + }; + + ExtFunctionResult::Return(json_to_monty(&result)) + } + Err(e) => ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::RuntimeError, + Some(format!("LLM call failed: {e}")), + )), + } +} + +/// Handle `__execute_code_step__(code, state)`. +/// +/// Runs user CodeAct code in a nested Monty VM with full tool dispatch. +/// Returns a dict with stdout, return_value, action_results, etc. +#[allow(clippy::too_many_arguments)] +async fn handle_execute_code_step( + args: &[MontyObject], + _kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, + llm: &Arc, + effects: &Arc, + leases: &Arc, + policy: &Arc, + event_tx: Option<&tokio::sync::broadcast::Sender>, +) -> ExtFunctionResult { + let code = match args.first() { + Some(obj) => monty_to_string(obj), + None => { + return ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::TypeError, + Some("__execute_code_step__ requires a code string".into()), + )); + } + }; + + let state = args + .get(1) + .map(monty_to_json) + .unwrap_or(serde_json::json!({})); + + let exec_ctx = ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: thread.user_id.clone(), + step_id: StepId::new(), + current_call_id: None, + source_channel: thread_source_channel(thread), + user_timezone: thread_user_timezone(thread), + }; + + // Run user code in a nested Monty VM (same pattern as rlm_query) + match Box::pin(execute_code( + &code, + thread, + llm, + effects, + leases, + policy, + &exec_ctx, + &[], + &state, + )) + .await + { + Ok(result) => { + // Broadcast events from code execution to the thread and event channel. + // Without this, ActionExecuted events from CodeAct tool calls are lost + // and never appear in traces. + for event_kind in &result.events { + let event = ThreadEvent::new(thread.id, event_kind.clone()); + if let Some(tx) = event_tx { + let _ = tx.send(event.clone()); + } + thread.events.push(event); + } + // If the CodeAct snippet itself failed (Python SyntaxError, runtime + // error, etc.), surface it as an ActionFailed event so traces and + // observers see the failure. Without this, parse errors silently + // fall back to the LLM via the result dict and never warn callers. + if result.had_error { + let error_msg = if !result.stdout.is_empty() { + let snippet: String = result.stdout.chars().take(500).collect(); + format!("CodeAct execution failed: {snippet}") + } else { + "CodeAct execution failed (no stdout)".to_string() + }; + let failed_event = ThreadEvent::new( + thread.id, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: "__codeact__".to_string(), + // Synthetic call_id derived from the step id — + // CodeAct snippet failures don't have an LLM-provided + // call_id, but `loop_engine.rs:1277` asserts that + // ActionFailed events carry a non-empty call_id for + // trace correlation. + call_id: format!("codeact-step-{}", exec_ctx.step_id.0), + error: error_msg, + params_summary: None, + }, + ); + if let Some(tx) = event_tx { + let _ = tx.send(failed_event.clone()); + } + thread.events.push(failed_event); + } + thread.updated_at = chrono::Utc::now(); + + let action_results: Vec = result + .action_results + .iter() + .map(|r| { + serde_json::json!({ + "action_name": r.action_name, + "output": r.output, + "is_error": r.is_error, + "duration_ms": r.duration.as_millis(), + }) + }) + .collect(); + + let result_json = serde_json::json!({ + "return_value": result.return_value, + "stdout": result.stdout, + "action_results": action_results, + "final_answer": result.final_answer, + "had_error": result.had_error, + "pending_gate": result.need_approval.as_ref().map(|na| { + match na { + ThreadOutcome::GatePaused { gate_name, action_name, call_id, parameters, resume_kind, resume_output } => serde_json::json!({ + "gate_paused": true, + "gate_name": gate_name, + "action_name": action_name, + "call_id": call_id, + "parameters": parameters, + "resume_kind": serde_json::to_value(resume_kind).unwrap_or_default(), + "resume_output": resume_output, + }), + _ => serde_json::Value::Null, + } + }), + }); + + ExtFunctionResult::Return(json_to_monty(&result_json)) + } + Err(e) => ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::RuntimeError, + Some(format!("Code execution failed: {e}")), + )), + } +} + +/// Handle `__execute_action__(name, params, call_id=...)`. +/// +/// Single source of truth for action execution. Performs: +/// 1. Lease lookup +/// 2. Policy check +/// 3. Lease consumption +/// 4. Action execution via EffectExecutor +/// 5. Event emission (ActionExecuted/ActionFailed) +/// +/// Python owns the working transcript and decides how tool outputs are +/// represented in internal message history. +async fn handle_execute_action( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, + effects: &Arc, + leases: &Arc, + policy: &Arc, + event_tx: Option<&tokio::sync::broadcast::Sender>, +) -> ExtFunctionResult { + let name = match extract_string_arg(args, kwargs, "name", 0) { + Some(n) => n, + None => { + return ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::TypeError, + Some("__execute_action__ requires a name argument".into()), + )); + } + }; + + let params = args + .get(1) + .map(monty_to_json) + .unwrap_or(serde_json::json!({})); + + let call_id = extract_string_kwarg(kwargs, "call_id").unwrap_or_default(); + + let exec_ctx = ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: thread.user_id.clone(), + step_id: StepId::new(), + current_call_id: Some(call_id.clone()), + source_channel: thread_source_channel(thread), + user_timezone: thread_user_timezone(thread), + }; + + // Helper: emit event only. The orchestrator owns transcript recording. + let emit_and_record = |thread: &mut Thread, + event_tx: Option<&tokio::sync::broadcast::Sender>, + event_kind: EventKind, + _call_id: &str, + _action_name: &str, + _output: &serde_json::Value| { + let event = ThreadEvent::new(thread.id, event_kind); + if let Some(tx) = event_tx { + let _ = tx.send(event.clone()); + } + thread.events.push(event); + thread.updated_at = chrono::Utc::now(); + }; + + // 1. Find lease for this action + let lease = match leases.find_lease_for_action(thread.id, &name).await { + Some(l) => l, + None => { + let error = format!("No lease for action '{name}'"); + let output = serde_json::json!({"error": &error}); + emit_and_record( + thread, + event_tx, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + error, + params_summary: None, + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "output": output, + "is_error": true, + }); + return ExtFunctionResult::Return(json_to_monty(&result)); + } + }; + + // 2. Check policy + let action_def = effects + .available_actions(std::slice::from_ref(&lease)) + .await + .ok() + .and_then(|actions| actions.into_iter().find(|a| a.name == name)); + + if let Some(ref ad) = action_def { + match policy.evaluate(ad, &lease, &[]) { + crate::capability::policy::PolicyDecision::Deny { reason } => { + let output = serde_json::json!({"error": format!("Denied: {reason}")}); + emit_and_record( + thread, + event_tx, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + error: reason, + params_summary: None, + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "output": output, + "is_error": true, + }); + return ExtFunctionResult::Return(json_to_monty(&result)); + } + crate::capability::policy::PolicyDecision::RequireApproval { .. } => { + let output = serde_json::json!({"status": "gate_paused", "gate_name": "approval"}); + emit_and_record( + thread, + event_tx, + EventKind::ApprovalRequested { + action_name: name.clone(), + call_id: call_id.clone(), + parameters: Some(params.clone()), + description: None, + allow_always: None, + gate_name: None, + params_summary: summarize_params(&name, ¶ms), + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "gate_paused": true, + "gate_name": "approval", + "action_name": name, + "call_id": call_id, + "parameters": params, + "resume_kind": serde_json::to_value(crate::gate::ResumeKind::Approval { + allow_always: true, + }) + .unwrap_or_default(), + }); + return ExtFunctionResult::Return(json_to_monty(&result)); + } + crate::capability::policy::PolicyDecision::Allow => {} + } + } + + // 3. Atomically re-find + consume a lease use under a single write + // lock. This closes the TOCTOU window between the read-only + // `find_lease_for_action` (used above for the policy check) and the + // consume — without it, two concurrent calls could both observe a + // lease with one remaining use and both proceed to execute. Mirrors + // `structured.rs::execute_action_batch_with_results`. + let lease = match leases.find_and_consume(thread.id, &name).await { + Ok(l) => l, + Err(e) => { + debug!(error = %e, "atomic lease find_and_consume failed"); + let error = format!("lease consumption failed for action '{name}': {e}"); + let output = serde_json::json!({"error": &error}); + emit_and_record( + thread, + event_tx, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + error, + params_summary: None, + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "output": output, + "is_error": true, + }); + return ExtFunctionResult::Return(json_to_monty(&result)); + } + }; + + // 4. Execute + let ps = summarize_params(&name, ¶ms); + match effects + .execute_action(&name, params, &lease, &exec_ctx) + .await + { + Ok(r) => { + // Effect adapters wrap tool errors as `Ok(ActionResult { is_error: true })` + // — surface them as `ActionFailed` so traces and observers see the + // failure. See `resolve_tool_future` in `scripting.rs` for the same + // pattern on the structured-tool path. + if r.is_error { + let error_msg = r + .output + .get("error") + .and_then(|v| v.as_str()) + .map(String::from) + .unwrap_or_else(|| r.output.to_string()); + emit_and_record( + thread, + event_tx, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + error: error_msg, + params_summary: ps.clone(), + }, + &call_id, + &name, + &r.output, + ); + } else { + emit_and_record( + thread, + event_tx, + EventKind::ActionExecuted { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + duration_ms: r.duration.as_millis() as u64, + params_summary: ps.clone(), + }, + &call_id, + &name, + &r.output, + ); + } + let result = serde_json::json!({ + "action_name": r.action_name, + "output": r.output, + "is_error": r.is_error, + "duration_ms": r.duration.as_millis(), + }); + ExtFunctionResult::Return(json_to_monty(&result)) + } + Err(EngineError::GatePaused { + gate_name, + action_name: _, + call_id: _, + parameters, + resume_kind, + resume_output, + }) => { + let _ = leases.refund_use(lease.id).await; + let output = serde_json::json!({"status": "gate_paused", "gate_name": gate_name}); + emit_and_record( + thread, + event_tx, + EventKind::ApprovalRequested { + action_name: name.clone(), + call_id: call_id.clone(), + parameters: Some((*parameters).clone()), + description: None, + allow_always: match resume_kind.as_ref() { + crate::gate::ResumeKind::Approval { allow_always } => Some(*allow_always), + _ => None, + }, + gate_name: Some(gate_name.clone()), + params_summary: summarize_params(&name, ¶meters), + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "gate_paused": true, + "gate_name": gate_name, + "action_name": name, + "call_id": call_id, + "parameters": parameters, + "resume_kind": serde_json::to_value(&*resume_kind).unwrap_or_default(), + "resume_output": resume_output, + }); + ExtFunctionResult::Return(json_to_monty(&result)) + } + Err(e) => { + let output = serde_json::json!({"error": e.to_string()}); + emit_and_record( + thread, + event_tx, + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.clone(), + call_id: call_id.clone(), + error: e.to_string(), + params_summary: ps, + }, + &call_id, + &name, + &output, + ); + let result = serde_json::json!({ + "output": output, + "is_error": true, + }); + ExtFunctionResult::Return(json_to_monty(&result)) + } + } +} + +/// Handle `__execute_actions_parallel__(calls)`. +/// +/// Batch host function that receives a list of action calls and executes them +/// concurrently. Each call is a dict with `name`, `params`, and optionally `call_id`. +/// +/// Returns a list of result dicts (one per call, in order). Each result has the +/// same shape as `__execute_action__` output, plus an optional gate pause payload. +/// +/// Events are emitted in original call order after all parallel executions complete. +async fn handle_execute_actions_parallel( + args: &[MontyObject], + thread: &mut Thread, + effects: &Arc, + leases: &Arc, + policy: &Arc, + event_tx: Option<&tokio::sync::broadcast::Sender>, +) -> ExtFunctionResult { + // Parse the calls list from the first argument (list of dicts) + let calls_json = args + .first() + .map(monty_to_json) + .unwrap_or(serde_json::json!([])); + let calls_array = match calls_json.as_array() { + Some(arr) => arr.clone(), + None => { + return ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::TypeError, + Some("__execute_actions_parallel__ requires a list of call dicts".into()), + )); + } + }; + + if calls_array.is_empty() { + return ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))); + } + + // Parse each call dict into (name, params, call_id) + struct ParsedCall { + name: String, + params: serde_json::Value, + call_id: String, + } + + let mut parsed: Vec = Vec::with_capacity(calls_array.len()); + for c in &calls_array { + let name = c + .get("name") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let params = c.get("params").cloned().unwrap_or(serde_json::json!({})); + let call_id = c + .get("call_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + parsed.push(ParsedCall { + name, + params, + call_id, + }); + } + + let step_id = StepId::new(); + + // ── Phase 1: Preflight (sequential) ───────────────────────── + // Check leases and policies. Denied → error result. Approval → interrupt. + + enum PfOutcome { + Runnable { + lease: crate::types::capability::CapabilityLease, + }, + Error { + result_json: serde_json::Value, + event: EventKind, + output: serde_json::Value, + }, + } + + let mut preflight: Vec> = Vec::with_capacity(parsed.len()); + + for pc in &parsed { + // Find lease + let lease = match leases.find_lease_for_action(thread.id, &pc.name).await { + Some(l) => l, + None => { + let error = format!("No lease for action '{}'", pc.name); + let output = serde_json::json!({"error": &error}); + let result_json = serde_json::json!({ + "output": &output, + "is_error": true, + }); + let event = EventKind::ActionFailed { + step_id, + action_name: pc.name.clone(), + call_id: pc.call_id.clone(), + error, + params_summary: None, + }; + preflight.push(Some(PfOutcome::Error { + result_json, + event, + output, + })); + continue; + } + }; + + // Check policy + let action_def = effects + .available_actions(std::slice::from_ref(&lease)) + .await + .ok() + .and_then(|actions| actions.into_iter().find(|a| a.name == pc.name)); + + if let Some(ref ad) = action_def { + match policy.evaluate(ad, &lease, &[]) { + crate::capability::policy::PolicyDecision::Deny { reason } => { + let output = serde_json::json!({"error": format!("Denied: {reason}")}); + let result_json = serde_json::json!({ + "output": &output, + "is_error": true, + }); + let event = EventKind::ActionFailed { + step_id, + action_name: pc.name.clone(), + call_id: pc.call_id.clone(), + error: reason, + params_summary: None, + }; + preflight.push(Some(PfOutcome::Error { + result_json, + event, + output, + })); + continue; + } + crate::capability::policy::PolicyDecision::RequireApproval { .. } => { + // Emit events for earlier errors, then interrupt + let mut results_json = Vec::with_capacity(preflight.len() + 1); + for pf in preflight { + match pf { + Some(PfOutcome::Error { + result_json, + event, + output: _, + }) => { + let ev = ThreadEvent::new(thread.id, event); + if let Some(tx) = event_tx { + let _ = tx.send(ev.clone()); + } + thread.events.push(ev); + results_json.push(result_json); + } + Some(PfOutcome::Runnable { .. }) | None => { + results_json.push(serde_json::json!(null)); + } + } + } + // Add the approval entry + let ev = ThreadEvent::new( + thread.id, + EventKind::ApprovalRequested { + action_name: pc.name.clone(), + call_id: pc.call_id.clone(), + parameters: Some(pc.params.clone()), + description: None, + allow_always: None, + gate_name: None, + params_summary: summarize_params(&pc.name, &pc.params), + }, + ); + if let Some(tx) = event_tx { + let _ = tx.send(ev.clone()); + } + thread.events.push(ev); + thread.updated_at = chrono::Utc::now(); + + results_json.push(serde_json::json!({ + "gate_paused": true, + "gate_name": "approval", + "action_name": &pc.name, + "call_id": &pc.call_id, + "parameters": &pc.params, + "resume_kind": serde_json::to_value(crate::gate::ResumeKind::Approval { + allow_always: true, + }) + .unwrap_or_default(), + })); + // Pad with nulls for calls that weren't reached so the + // Python-side loop can emit ActionResult placeholders for + // every tool call in the assistant message. + while results_json.len() < parsed.len() { + results_json.push(serde_json::json!(null)); + } + return ExtFunctionResult::Return(json_to_monty(&serde_json::json!( + results_json + ))); + } + crate::capability::policy::PolicyDecision::Allow => {} + } + } + + // Atomically re-find + consume a lease use under a single write + // lock, closing the TOCTOU window between the read-only + // `find_lease_for_action` above and the consume. Mirrors + // `structured.rs::execute_action_batch_with_results`. + let lease = match leases.find_and_consume(thread.id, &pc.name).await { + Ok(l) => l, + Err(e) => { + debug!(error = %e, "atomic lease find_and_consume failed"); + let error = format!("lease consumption failed for action '{}': {e}", pc.name); + let output = serde_json::json!({"error": &error}); + let result_json = serde_json::json!({ + "output": &output, + "is_error": true, + }); + let event = EventKind::ActionFailed { + step_id, + action_name: pc.name.clone(), + call_id: pc.call_id.clone(), + error, + params_summary: None, + }; + preflight.push(Some(PfOutcome::Error { + result_json, + event, + output, + })); + continue; + } + }; + + preflight.push(Some(PfOutcome::Runnable { lease })); + } + + // ── Phase 2: Execute in parallel ──────────────────────────── + + // Slot array: index → execution result + let mut slot_results: Vec> = vec![None; parsed.len()]; + let mut slot_events: Vec> = vec![None; parsed.len()]; + let mut slot_outputs: Vec> = vec![None; parsed.len()]; + + // Separate runnable from errors + let mut runnable: Vec<(usize, crate::types::capability::CapabilityLease)> = Vec::new(); + for (idx, pf) in preflight.into_iter().enumerate() { + match pf { + Some(PfOutcome::Error { + result_json, + event, + output, + }) => { + slot_results[idx] = Some(result_json); + slot_events[idx] = Some(event); + slot_outputs[idx] = Some(output); + } + Some(PfOutcome::Runnable { lease }) => { + runnable.push((idx, lease)); + } + None => {} + } + } + + if runnable.len() == 1 { + // Single call: execute directly + let (idx, lease) = runnable.into_iter().next().unwrap(); // safety: len()==1 checked above + let pc = &parsed[idx]; + let exec_ctx = ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: thread.user_id.clone(), + step_id, + current_call_id: Some(pc.call_id.clone()), + // Read source_channel from thread metadata so downstream tools + // (e.g. mission_create) can default notify_channels to the + // originating channel. Hardcoding `None` here was a bug — it + // silently dropped the gateway routing for any tool dispatched + // through the parallel batch path. + source_channel: thread_source_channel(thread), + user_timezone: thread_user_timezone(thread), + }; + let ps = summarize_params(&pc.name, &pc.params); + let (result_json, event, output) = execute_single_action( + effects, + &pc.name, + pc.params.clone(), + &pc.call_id, + &lease, + &exec_ctx, + ps, + ) + .await; + if interrupted_result_needs_refund(&result_json) { + let _ = leases.refund_use(lease.id).await; + } + slot_results[idx] = Some(result_json); + slot_events[idx] = Some(event); + slot_outputs[idx] = Some(output); + } else if runnable.len() > 1 { + // Multiple calls: execute in parallel via JoinSet + let mut join_set = tokio::task::JoinSet::new(); + let effects = effects.clone(); + // Capture once outside the loop — the thread's metadata is stable + // for the duration of the parallel batch. + let parallel_source_channel = thread_source_channel(thread); + let parallel_user_timezone = thread_user_timezone(thread); + + for (idx, lease) in runnable { + let pc_name = parsed[idx].name.clone(); + let pc_params = parsed[idx].params.clone(); + let pc_call_id = parsed[idx].call_id.clone(); + let effects = effects.clone(); + let lease = lease.clone(); + let exec_ctx = ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: thread.user_id.clone(), + step_id, + current_call_id: Some(pc_call_id.clone()), + // See comment above — read from thread metadata, not None. + source_channel: parallel_source_channel.clone(), + user_timezone: parallel_user_timezone, + }; + let ps = summarize_params(&pc_name, &pc_params); + + join_set.spawn(async move { + let (result_json, event, output) = execute_single_action( + &effects, + &pc_name, + pc_params, + &pc_call_id, + &lease, + &exec_ctx, + ps, + ) + .await; + (idx, lease.id, result_json, event, output) + }); + } + + while let Some(join_result) = join_set.join_next().await { + match join_result { + Ok((idx, lease_id, result_json, event, output)) => { + if interrupted_result_needs_refund(&result_json) { + let _ = leases.refund_use(lease_id).await; + } + slot_results[idx] = Some(result_json); + slot_events[idx] = Some(event); + slot_outputs[idx] = Some(output); + } + Err(e) => { + debug!("parallel action execution task panicked: {e}"); + } + } + } + } + + // ── Phase 3: Emit events in order ─────────────────────────── + + let mut results_json = Vec::with_capacity(parsed.len()); + for idx in 0..parsed.len() { + let result_json = slot_results[idx].take().unwrap_or( + serde_json::json!({"is_error": true, "output": {"error": "execution slot empty"}}), + ); + let _output = slot_outputs[idx] + .take() + .unwrap_or(serde_json::json!({"error": "no output"})); + + if let Some(event) = slot_events[idx].take() { + let ev = ThreadEvent::new(thread.id, event); + if let Some(tx) = event_tx { + let _ = tx.send(ev.clone()); + } + thread.events.push(ev); + } + + results_json.push(result_json.clone()); + } + + thread.updated_at = chrono::Utc::now(); + ExtFunctionResult::Return(json_to_monty(&serde_json::json!(results_json))) +} + +/// Execute a single action and return (result_json, event, output) for the +/// batch handler to record. Shared by both single-call and parallel paths. +async fn execute_single_action( + effects: &Arc, + name: &str, + params: serde_json::Value, + call_id: &str, + lease: &crate::types::capability::CapabilityLease, + exec_ctx: &ThreadExecutionContext, + params_summary: Option, +) -> (serde_json::Value, EventKind, serde_json::Value) { + match effects.execute_action(name, params, lease, exec_ctx).await { + Ok(r) => { + // Surface wrapped errors as ActionFailed (see resolve_tool_future + // and the parallel execute path for the same pattern). + let event = if r.is_error { + let error_msg = r + .output + .get("error") + .and_then(|v| v.as_str()) + .map(String::from) + .unwrap_or_else(|| r.output.to_string()); + EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.to_string(), + call_id: call_id.to_string(), + error: error_msg, + params_summary: params_summary.clone(), + } + } else { + EventKind::ActionExecuted { + step_id: exec_ctx.step_id, + action_name: name.to_string(), + call_id: call_id.to_string(), + duration_ms: r.duration.as_millis() as u64, + params_summary: params_summary.clone(), + } + }; + let result_json = serde_json::json!({ + "action_name": r.action_name, + "output": r.output, + "is_error": r.is_error, + "duration_ms": r.duration.as_millis(), + }); + (result_json, event, r.output) + } + Err(EngineError::GatePaused { + gate_name, + action_name: _, + call_id: _, + parameters, + resume_kind, + resume_output, + }) => { + let output = serde_json::json!({"status": "gate_paused", "gate_name": &gate_name}); + let event = EventKind::ApprovalRequested { + action_name: name.to_string(), + call_id: call_id.to_string(), + parameters: Some((*parameters).clone()), + description: None, + allow_always: match resume_kind.as_ref() { + crate::gate::ResumeKind::Approval { allow_always } => Some(*allow_always), + _ => None, + }, + gate_name: Some(gate_name.clone()), + params_summary: summarize_params(name, ¶meters), + }; + let result_json = serde_json::json!({ + "gate_paused": true, + "gate_name": gate_name, + "action_name": name, + "call_id": call_id, + "parameters": parameters, + "resume_kind": serde_json::to_value(&*resume_kind).unwrap_or_default(), + "resume_output": resume_output, + }); + (result_json, event, output) + } + Err(e) => { + let output = serde_json::json!({"error": e.to_string()}); + let event = EventKind::ActionFailed { + step_id: exec_ctx.step_id, + action_name: name.to_string(), + call_id: call_id.to_string(), + error: e.to_string(), + params_summary, + }; + let result_json = serde_json::json!({ + "output": &output, + "is_error": true, + }); + (result_json, event, output) + } + } +} + +fn interrupted_result_needs_refund(result: &serde_json::Value) -> bool { + result.get("gate_paused").and_then(|v| v.as_bool()) == Some(true) +} + +/// Handle `__check_signals__()`. +fn handle_check_signals(signal_rx: &mut SignalReceiver, thread: &mut Thread) -> ExtFunctionResult { + match signal_rx.try_recv() { + Ok(ThreadSignal::Stop) | Ok(ThreadSignal::Suspend) => { + ExtFunctionResult::Return(MontyObject::String("stop".into())) + } + Ok(ThreadSignal::InjectMessage(msg)) => { + thread.add_message(msg.clone()); + let result = serde_json::json!({"inject": msg.content}); + ExtFunctionResult::Return(json_to_monty(&result)) + } + Ok(ThreadSignal::Resume) | Ok(ThreadSignal::ChildCompleted { .. }) => { + ExtFunctionResult::Return(MontyObject::None) + } + Err(_) => ExtFunctionResult::Return(MontyObject::None), + } +} + +/// Handle `__emit_event__(kind, **data)`. +fn handle_emit_event( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, + event_tx: Option<&tokio::sync::broadcast::Sender>, +) -> ExtFunctionResult { + let kind_str = args.first().map(monty_to_string).unwrap_or_default(); + + let kind = match kind_str.as_str() { + "step_started" => { + let _step = extract_u64_kwarg(kwargs, "step").unwrap_or(0); + EventKind::StepStarted { + step_id: StepId::new(), + } + } + "step_completed" => { + let input = extract_u64_kwarg(kwargs, "input_tokens").unwrap_or(0); + let output = extract_u64_kwarg(kwargs, "output_tokens").unwrap_or(0); + // Increment step count (mirrors the old Rust loop's step_count += 1) + thread.step_count += 1; + // Track token usage + thread.total_tokens_used += input + output; + EventKind::StepCompleted { + step_id: StepId::new(), + tokens: TokenUsage { + input_tokens: input, + output_tokens: output, + ..Default::default() + }, + } + } + "action_executed" => { + let action_name = extract_string_kwarg(kwargs, "action_name").unwrap_or_default(); + let call_id = extract_string_kwarg(kwargs, "call_id").unwrap_or_default(); + EventKind::ActionExecuted { + step_id: StepId::new(), + action_name, + call_id, + duration_ms: 0, + params_summary: None, + } + } + "action_failed" => { + let action_name = extract_string_kwarg(kwargs, "action_name").unwrap_or_default(); + let call_id = extract_string_kwarg(kwargs, "call_id").unwrap_or_default(); + let error = extract_string_kwarg(kwargs, "error").unwrap_or_default(); + EventKind::ActionFailed { + step_id: StepId::new(), + action_name, + call_id, + error, + params_summary: None, + } + } + "skill_activated" => { + let names_str = extract_string_kwarg(kwargs, "skill_names").unwrap_or_default(); + let skill_names: Vec = names_str + .split(',') + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .collect(); + EventKind::SkillActivated { skill_names } + } + _ => { + debug!(kind = %kind_str, "orchestrator: unknown event kind, skipping"); + return ExtFunctionResult::Return(MontyObject::None); + } + }; + + let event = ThreadEvent::new(thread.id, kind); + if let Some(tx) = event_tx { + let _ = tx.send(event.clone()); + } + thread.events.push(event); + thread.updated_at = chrono::Utc::now(); + + ExtFunctionResult::Return(MontyObject::None) +} + +/// Handle `__save_checkpoint__(state, counters)`. +fn handle_save_checkpoint( + args: &[MontyObject], + _kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, +) -> ExtFunctionResult { + let state = args + .first() + .map(monty_to_json) + .unwrap_or(serde_json::json!({})); + let counters = args + .get(1) + .map(monty_to_json) + .unwrap_or(serde_json::json!({})); + + sync_runtime_state(thread, Some(&state)); + + if let Some(metadata) = thread.metadata.as_object_mut() { + metadata.insert( + "runtime_checkpoint".into(), + serde_json::json!({ + "persisted_state": state, + "nudge_count": counters.get("nudge_count").and_then(|v| v.as_u64()).unwrap_or(0), + "consecutive_errors": counters.get("consecutive_errors").and_then(|v| v.as_u64()).unwrap_or(0), + "consecutive_action_errors": counters.get("consecutive_action_errors").and_then(|v| v.as_u64()).unwrap_or(0), + "compaction_count": counters.get("compaction_count").and_then(|v| v.as_u64()).unwrap_or(0), + }), + ); + } + thread.updated_at = chrono::Utc::now(); + + ExtFunctionResult::Return(MontyObject::None) +} + +/// Handle `__transition_to__(state, reason)`. +fn handle_transition_to( + args: &[MontyObject], + _kwargs: &[(MontyObject, MontyObject)], + thread: &mut Thread, +) -> ExtFunctionResult { + let state_str = args.first().map(monty_to_string).unwrap_or_default(); + let reason = args.get(1).map(monty_to_string); + + let target = match state_str.as_str() { + "running" => crate::types::thread::ThreadState::Running, + "completed" => crate::types::thread::ThreadState::Completed, + "failed" => crate::types::thread::ThreadState::Failed, + "waiting" => crate::types::thread::ThreadState::Waiting, + "suspended" => crate::types::thread::ThreadState::Suspended, + other => { + return ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::ValueError, + Some(format!("Unknown thread state: {other}")), + )); + } + }; + + match thread.transition_to(target, reason) { + Ok(()) => ExtFunctionResult::Return(MontyObject::None), + Err(e) => ExtFunctionResult::Error(monty::MontyException::new( + monty::ExcType::RuntimeError, + Some(format!("State transition failed: {e}")), + )), + } +} + +/// Handle `__retrieve_docs__(goal, max_docs)`. +async fn handle_retrieve_docs( + args: &[MontyObject], + _kwargs: &[(MontyObject, MontyObject)], + thread: &Thread, + retrieval: Option<&RetrievalEngine>, +) -> ExtFunctionResult { + let retrieval = match retrieval { + Some(r) => r, + None => return ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))), + }; + + let goal = args.first().map(monty_to_string).unwrap_or_default(); + let max_docs = args + .get(1) + .and_then(|v| match v { + MontyObject::Int(i) => Some(*i as usize), + _ => None, + }) + .unwrap_or(5); + + match retrieval + .retrieve_context(thread.project_id, &thread.user_id, &goal, max_docs) + .await + { + Ok(docs) => { + let docs_json: Vec = docs + .iter() + .map(|d| { + serde_json::json!({ + "type": format!("{:?}", d.doc_type), + "title": d.title, + "content": d.content, + }) + }) + .collect(); + ExtFunctionResult::Return(json_to_monty(&serde_json::json!(docs_json))) + } + Err(e) => { + debug!("retrieve_docs failed: {e}"); + ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))) + } + } +} + +/// Handle `__check_budget__()`. +fn handle_check_budget(thread: &Thread) -> ExtFunctionResult { + let tokens_remaining = thread + .config + .max_tokens_total + .map(|max| max.saturating_sub(thread.total_tokens_used)) + .unwrap_or(u64::MAX); + + let time_remaining_ms = thread + .config + .max_duration + .map(|dur| { + let elapsed = chrono::Utc::now() + .signed_duration_since(thread.created_at) + .num_milliseconds() + .max(0) as u64; + dur.as_millis() as u64 - elapsed.min(dur.as_millis() as u64) + }) + .unwrap_or(u64::MAX); + + let usd_remaining = thread + .config + .max_budget_usd + .map(|max| (max - thread.total_cost_usd).max(0.0)); + + let result = serde_json::json!({ + "tokens_remaining": tokens_remaining, + "time_remaining_ms": time_remaining_ms, + "usd_remaining": usd_remaining, + }); + + ExtFunctionResult::Return(json_to_monty(&result)) +} + +/// Handle `__get_actions__()`. +async fn handle_get_actions( + thread: &mut Thread, + effects: &Arc, + leases: &Arc, + store: Option<&Arc>, +) -> ExtFunctionResult { + if let Err(e) = + reconcile_dynamic_tool_lease(thread, effects, leases, store, &crate::LeasePlanner::new()) + .await + { + warn_on_lease_refresh_failure("get_actions", &e); + } + + let active_leases = leases.active_for_thread(thread.id).await; + match effects.available_actions(&active_leases).await { + Ok(actions) => { + let actions_json: Vec = actions + .iter() + .map(|a| { + serde_json::json!({ + "name": a.name, + "description": a.description, + "params": a.parameters_schema, + }) + }) + .collect(); + ExtFunctionResult::Return(json_to_monty(&serde_json::json!(actions_json))) + } + Err(e) => { + debug!("get_actions failed: {e}"); + ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))) + } + } +} + +/// Handle `__list_skills__()`. +/// +/// Loads all `DocType::Skill` MemoryDocs from the project and returns them +/// as a list of Python dicts. The Python orchestrator handles scoring, +/// selection, and injection — Rust just provides data access. +async fn handle_list_skills( + _args: &[MontyObject], + thread: &Thread, + store: Option<&Arc>, +) -> ExtFunctionResult { + let Some(store) = store else { + return ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))); + }; + + // Use shared listing: user's own skills + system/admin-installed skills. + let docs = match store + .list_memory_docs_with_shared(thread.project_id, &thread.user_id) + .await + { + Ok(docs) => docs, + Err(e) => { + debug!("__list_skills__: failed to load docs: {e}"); + return ExtFunctionResult::Return(json_to_monty(&serde_json::json!([]))); + } + }; + + let skills: Vec = docs + .into_iter() + .filter(|d| d.doc_type == crate::types::memory::DocType::Skill) + .map(|d| { + serde_json::json!({ + "doc_id": d.id.0.to_string(), + "title": d.title, + "content": d.content, + "metadata": d.metadata, + }) + }) + .collect(); + + ExtFunctionResult::Return(json_to_monty(&serde_json::json!(skills))) +} + +/// Handle `__record_skill_usage__(doc_id, success)`. +/// +/// Records that a skill was used in this thread. Called by the Python +/// orchestrator after skill-assisted execution completes. +async fn handle_record_skill_usage( + args: &[MontyObject], + store: Option<&Arc>, +) -> ExtFunctionResult { + let Some(store) = store else { + return ExtFunctionResult::Return(MontyObject::None); + }; + + let doc_id_str = args.first().map(monty_to_string).unwrap_or_default(); + let success = args + .get(1) + .map(|o| matches!(o, MontyObject::Bool(true))) + .unwrap_or(false); + + let Ok(uuid) = uuid::Uuid::parse_str(&doc_id_str) else { + debug!("__record_skill_usage__: invalid doc_id: {doc_id_str}"); + return ExtFunctionResult::Return(MontyObject::None); + }; + + let tracker = crate::memory::SkillTracker::new(Arc::clone(store)); + if let Err(e) = tracker + .record_usage(crate::types::memory::DocId(uuid), success) + .await + { + debug!("__record_skill_usage__: failed: {e}"); + } + + ExtFunctionResult::Return(MontyObject::None) +} + +/// Handle `__regex_match__(pattern, text) -> bool`. +/// +/// Compiles `pattern` with a bounded size limit and returns whether it +/// matches anywhere in `text`. Invalid regex or a size-limit violation +/// returns `False` silently. Used by the Python skill selector for regex +/// pattern scoring (Monty has no `re` module). +/// +/// **Security: ReDoS safety.** This handler accepts arbitrary patterns from +/// the Python orchestrator (which itself receives them from skill manifests) +/// and runs them on user-supplied text. Safety relies on the `regex` crate's +/// linear-time matching guarantee (no backreferences, no lookaround) plus the +/// 64 KiB compiled-size cap and DFA-size cap below. If the `regex` crate is +/// ever swapped for `fancy-regex` (which supports backreferences and is NOT +/// linear-time), this becomes a real ReDoS vector. This is enforced by +/// convention and documentation only — see the top-of-crate comment in +/// `crates/ironclaw_engine/src/lib.rs`. (A `#[cfg(feature = "fancy-regex")] +/// compile_error!` tripwire was evaluated but conflicts with +/// `cargo clippy --all-features` which is the standard CI command.) +fn handle_regex_match(args: &[MontyObject]) -> ExtFunctionResult { + let pattern = args.first().map(monty_to_string).unwrap_or_default(); + let text = args.get(1).map(monty_to_string).unwrap_or_default(); + if pattern.is_empty() { + return ExtFunctionResult::Return(MontyObject::Bool(false)); + } + // Cap compiled regex size to prevent ReDoS (matches the 64 KiB limit used + // by `LoadedSkill::compile_patterns` in `ironclaw_skills`). Also cap the + // lazy-DFA cache: the `regex` crate's DFA can grow beyond `size_limit` + // during matching, so `dfa_size_limit` is a separate defensive cap on + // memory allocation from a crafted pattern over untrusted skill manifests. + const MAX_REGEX_SIZE: usize = 1 << 16; + let matched = match regex::RegexBuilder::new(&pattern) + .size_limit(MAX_REGEX_SIZE) + .dfa_size_limit(MAX_REGEX_SIZE) + .build() + { + Ok(re) => re.is_match(&text), + Err(e) => { + debug!("__regex_match__: invalid pattern '{pattern}': {e}"); + false + } + }; + ExtFunctionResult::Return(MontyObject::Bool(matched)) +} + +/// Handle `__set_active_skills__(skills)`. +/// +/// Persists the selected skill provenance onto the thread so post-run learning +/// flows can reason about the exact skill versions and snippets that were active. +fn handle_set_active_skills(args: &[MontyObject], thread: &mut Thread) -> ExtFunctionResult { + let skills_json = args + .first() + .map(monty_to_json) + .unwrap_or_else(|| serde_json::json!([])); + + let skills = match serde_json::from_value::>(skills_json) { + Ok(skills) => skills, + Err(e) => { + debug!("__set_active_skills__: invalid payload: {e}"); + return ExtFunctionResult::Return(MontyObject::None); + } + }; + + if let Err(e) = thread.set_active_skills(&skills) { + debug!("__set_active_skills__: failed to persist active skills: {e}"); + } + + ExtFunctionResult::Return(MontyObject::None) +} + +// ── Helpers ───────────────────────────────────────────────── + +/// Build the context variables injected into the orchestrator Python. +fn build_orchestrator_inputs( + thread: &Thread, + persisted_state: &serde_json::Value, +) -> (Vec, Vec) { + let names = vec![ + "context".into(), + "goal".into(), + "actions".into(), + "state".into(), + "config".into(), + ]; + + // Build orchestrator bootstrap context. Prefer the internal execution + // transcript when present, otherwise fall back to the user-visible transcript. + let bootstrap_messages = if thread.internal_messages.is_empty() { + &thread.messages + } else { + &thread.internal_messages + }; + let context: Vec = bootstrap_messages + .iter() + .map(|m| { + // Serialize action_calls through the Python interchange shape + // (`{name, call_id, params}`) so the bootstrap context is + // round-trip compatible with `python_json_to_action_calls`. + // Using bare `m.action_calls` here produces the canonical Rust + // serde format (`{action_name, id, parameters}`), which the + // Python orchestrator passes back verbatim on the next + // `__llm_complete__` call — and `python_json_to_action_calls` + // then fails with "missing field `name`", orphaning every + // subsequent tool result. This is the SECOND code path (after + // `handle_llm_complete`) that feeds action_calls into the + // Python working transcript; both must use the same shape. + let calls_json = m + .action_calls + .as_ref() + .map(|calls| serde_json::Value::Array(action_calls_to_python_json(calls))); + serde_json::json!({ + "role": format!("{:?}", m.role), + "content": m.content, + "action_name": m.action_name, + "action_call_id": m.action_call_id, + "action_calls": calls_json, + }) + }) + .collect(); + + // Build config + let config = serde_json::json!({ + "max_iterations": thread.config.max_iterations, + "max_tool_intent_nudges": thread.config.max_tool_intent_nudges, + "enable_tool_intent_nudge": thread.config.enable_tool_intent_nudge, + "max_consecutive_errors": thread.config.max_consecutive_errors, + "max_tokens_total": thread.config.max_tokens_total, + "max_budget_usd": thread.config.max_budget_usd, + "model_context_limit": thread.config.model_context_limit, + "enable_compaction": thread.config.enable_compaction, + "compaction_threshold": thread.config.compaction_threshold, + "depth": thread.config.depth, + "max_depth": thread.config.max_depth, + "step_count": thread.step_count, + }); + + let values = vec![ + json_to_monty(&serde_json::json!(context)), + MontyObject::String(thread.goal.clone()), + json_to_monty(&serde_json::json!([])), // actions loaded dynamically via __get_actions__ + json_to_monty(persisted_state), + json_to_monty(&config), + ]; + + (names, values) +} + +/// JSON shape used to interchange `ActionCall`s with the Python orchestrator. +/// +/// This is the *single* place that defines the field naming convention used +/// across the Python boundary. It is intentionally separate from the +/// canonical `ActionCall` type because: +/// +/// - `ActionCall` uses Rust-idiomatic field names (`id`, `action_name`, +/// `parameters`) and is also persisted into Step records and ThreadEvents. +/// Renaming its serde fields would invalidate every existing row. +/// - The Python orchestrator uses friendlier names (`call_id`, `name`, +/// `params`) that read naturally in CodeAct prompts and `default.py`. +/// +/// Without this type, the round-trip is asymmetric: Rust → Python uses one +/// shape, Python → Rust used `serde_json::from_value::>` +/// which silently fails (`.ok()` swallows the error) and produces `None`, +/// which means assistant messages came back without `action_calls`. The +/// downstream effect is that every tool result looks orphaned to +/// `sanitize_tool_messages` and gets rewritten as a user message — losing +/// the assistant ↔ tool_result linkage the LLM needs to reason about prior +/// tool calls. +#[derive(Debug, serde::Serialize, serde::Deserialize)] +struct PythonActionCall { + name: String, + call_id: String, + params: serde_json::Value, +} + +impl From<&ActionCall> for PythonActionCall { + fn from(c: &ActionCall) -> Self { + Self { + name: c.action_name.clone(), + call_id: c.id.clone(), + params: c.parameters.clone(), + } + } +} + +impl From for ActionCall { + fn from(p: PythonActionCall) -> Self { + Self { + id: p.call_id, + action_name: p.name, + parameters: p.params, + } + } +} + +/// Serialize a slice of `ActionCall`s into the Python interchange shape. +/// +/// On serialization failure (essentially unreachable for `String + String + +/// Value`, but still possible if the `serde_json::Value` parameters tree +/// contains a key whose stringification fails), the entry is **dropped** +/// from the output rather than replaced with `Value::Null`. The previous +/// `unwrap_or_else(|_| Value::Null)` corrupted the array — Python's +/// `default.py` accesses `c.get("name")` / `c.get("call_id")` / +/// `c.get("params")` on each entry, so a `null` would crash with a Python +/// `AttributeError` and lose the entire LLM step. `filter_map` produces a +/// shorter array, which Python's tool-result loop handles correctly because +/// it iterates `range(len(results))` against the shortened call list. The +/// warn log is preserved so operators have a breadcrumb if it ever fires. +fn action_calls_to_python_json(calls: &[ActionCall]) -> Vec { + calls + .iter() + .filter_map(|c| match serde_json::to_value(PythonActionCall::from(c)) { + Ok(value) => Some(value), + Err(e) => { + warn!( + error = %e, + action_name = %c.action_name, + "Failed to serialize ActionCall for Python orchestrator — dropping entry" + ); + None + } + }) + .collect() +} + +/// Build a PII-safe summary of an `action_calls` JSON value for log output. +/// +/// The action_calls payload contains tool parameters, which can carry user +/// PII (search queries, file names, email content, conversation text). +/// Dumping the full value into a `warn!` log would leak that PII to log +/// aggregation systems (Datadog, CloudWatch, Sentry) the moment the parser +/// fails — and the parser only fails when the Python ↔ Rust shape drifts, +/// which is exactly when an operator is most likely to be grepping logs. +/// +/// We emit only the structural information operators actually need to +/// debug a shape drift: array length and the keys of the first entry. The +/// keys themselves are not user data — they're field names like +/// `name`/`call_id`/`params` that are static across all calls. +fn summarize_action_calls_for_log(value: &serde_json::Value) -> String { + match value.as_array() { + Some(arr) if arr.is_empty() => "empty array".to_string(), + Some(arr) => { + let first_keys = arr + .first() + .and_then(|v| v.as_object()) + .map(|obj| { + let mut keys: Vec<&str> = obj.keys().map(String::as_str).collect(); + keys.sort_unstable(); + keys.join(",") + }) + .unwrap_or_else(|| "".to_string()); + format!( + "array of {} entries; first entry keys: [{}]", + arr.len(), + first_keys + ) + } + None => format!("non-array value of type {}", json_value_type_name(value)), + } +} + +/// Cheap type-name string for a `serde_json::Value`. Used by +/// `summarize_action_calls_for_log` to surface the wrong-shape case +/// (e.g. Python passed a string instead of an array) without leaking the +/// actual contents. +fn json_value_type_name(value: &serde_json::Value) -> &'static str { + match value { + serde_json::Value::Null => "null", + serde_json::Value::Bool(_) => "bool", + serde_json::Value::Number(_) => "number", + serde_json::Value::String(_) => "string", + serde_json::Value::Array(_) => "array", + serde_json::Value::Object(_) => "object", + } +} + +/// Deserialize an `action_calls` JSON array (in Python interchange shape) +/// back into canonical `ActionCall`s. +/// +/// Logs a warning on failure rather than swallowing silently. The whole +/// commit that introduced this helper exists to undo a `.ok()` swallow that +/// dropped action_calls without any signal — replacing it with another +/// `.ok()?` would re-introduce the same trap, just one layer deeper. If the +/// shape ever drifts again (Python orchestrator field rename, extra +/// required field, partial migration), the warning is the operator-visible +/// breadcrumb that explains why subsequent tool results suddenly look +/// orphaned to `sanitize_tool_messages`. +/// +/// The warn log emits a structural summary (`summarize_action_calls_for_log`) +/// instead of the raw value because tool parameters can contain user PII. +fn python_json_to_action_calls(value: &serde_json::Value) -> Option> { + match serde_json::from_value::>(value.clone()) { + Ok(parsed) => Some(parsed.into_iter().map(ActionCall::from).collect()), + Err(e) => { + warn!( + error = %e, + shape = %summarize_action_calls_for_log(value), + "Failed to parse action_calls from Python orchestrator — \ + assistant message will lose tool_call linkage and downstream \ + tool results will be rewritten as user messages" + ); + None + } + } +} + +fn json_to_thread_messages(value: &serde_json::Value) -> Option> { + let arr = value.as_array()?; + let mut messages = Vec::with_capacity(arr.len()); + + for item in arr { + let role = item.get("role").and_then(|v| v.as_str()).unwrap_or("User"); + let content = item + .get("content") + .and_then(|v| v.as_str()) + .unwrap_or_default(); + // Filter out null before calling the parser — `action_calls: null` + // is Python's legitimate "this message has no tool calls" signal (text + // response), not a parse error. Without this filter, the warn log in + // python_json_to_action_calls fires on every text-only assistant + // message with "invalid type: null, expected a sequence". + let action_calls = item + .get("action_calls") + .filter(|v| !v.is_null()) + .and_then(python_json_to_action_calls); + + let message = match role { + "System" | "system" => ThreadMessage::system(content), + "Assistant" | "assistant" => { + if let Some(calls) = action_calls { + ThreadMessage::assistant_with_actions(Some(content.to_string()), calls) + } else { + ThreadMessage::assistant(content) + } + } + "ActionResult" | "action_result" => ThreadMessage::action_result( + item.get("action_call_id") + .and_then(|v| v.as_str()) + .unwrap_or_default(), + item.get("action_name") + .and_then(|v| v.as_str()) + .unwrap_or_default(), + content, + ), + _ => ThreadMessage::user(content), + }; + messages.push(message); + } + + Some(messages) +} + +fn sync_runtime_state(thread: &mut Thread, state: Option<&serde_json::Value>) { + let Some(state) = state else { + return; + }; + if let Some(messages) = state + .get("working_messages") + .and_then(json_to_thread_messages) + { + thread.internal_messages = messages; + thread.updated_at = chrono::Utc::now(); + } +} + +fn sync_visible_outcome(thread: &mut Thread, outcome: &ThreadOutcome) { + if let ThreadOutcome::Completed { + response: Some(response), + } = outcome + { + let already_present = thread + .messages + .last() + .map(|msg| { + msg.role == crate::types::message::MessageRole::Assistant + && msg.content == *response + }) + .unwrap_or(false); + if !already_present { + thread.add_message(ThreadMessage::assistant(response)); + } + } +} + +/// Parse the orchestrator's return value into a ThreadOutcome. +fn parse_outcome(result: &serde_json::Value) -> ThreadOutcome { + let outcome = result + .get("outcome") + .and_then(|v| v.as_str()) + .unwrap_or("completed"); + + match outcome { + "completed" => ThreadOutcome::Completed { + response: result + .get("response") + .and_then(|v| v.as_str()) + .map(String::from), + }, + "stopped" => ThreadOutcome::Stopped, + "max_iterations" => ThreadOutcome::MaxIterations, + "failed" => ThreadOutcome::Failed { + error: result + .get("error") + .and_then(|v| v.as_str()) + .unwrap_or("unknown error") + .to_string(), + }, + "gate_paused" => { + let resume_kind_value = result + .get("resume_kind") + .cloned() + .unwrap_or(serde_json::json!({})); + let resume_kind = serde_json::from_value(resume_kind_value).unwrap_or( + crate::gate::ResumeKind::Approval { + allow_always: false, + }, + ); + ThreadOutcome::GatePaused { + gate_name: result + .get("gate_name") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_string(), + action_name: result + .get("action_name") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(), + call_id: result + .get("call_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(), + parameters: result + .get("parameters") + .cloned() + .unwrap_or(serde_json::json!({})), + resume_kind, + resume_output: result.get("resume_output").cloned(), + } + } + _ => ThreadOutcome::Completed { response: None }, + } +} + +fn extract_string_arg( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + name: &str, + position: usize, +) -> Option { + for (k, v) in kwargs { + if let MontyObject::String(key) = k + && key == name + { + return Some(monty_to_string(v)); + } + } + args.get(position).map(monty_to_string) +} + +fn extract_string_kwarg(kwargs: &[(MontyObject, MontyObject)], name: &str) -> Option { + for (k, v) in kwargs { + if let MontyObject::String(key) = k + && key == name + { + return Some(monty_to_string(v)); + } + } + None +} + +fn extract_u64_kwarg(kwargs: &[(MontyObject, MontyObject)], name: &str) -> Option { + for (k, v) in kwargs { + if let MontyObject::String(key) = k + && key == name + && let MontyObject::Int(i) = v + { + return Some(*i as u64); + } + } + None +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::memory::{DocType, MemoryDoc}; + use crate::types::project::ProjectId; + + // ── Python helper unit tests via Monty ────────────────────── + // + // Extracts the helper functions from the default orchestrator and + // evaluates `signals_tool_intent(text)` directly, mirroring the V1 + // Rust unit test suite in src/llm/reasoning.rs. + + /// Run a Python expression that returns a bool by prepending the + /// orchestrator helper definitions and wrapping in `FINAL(expr)`. + /// Run a Python snippet and drive the Monty VM, returning the FINAL() + /// value as a `MontyObject`. This is the common core for `eval_python_bool` + /// and `eval_python_int`. + fn run_python_final(code: String) -> MontyObject { + let runner = + MontyRun::new(code, "test.py", vec![]).expect("Failed to parse orchestrator helpers"); + let mut stdout = String::new(); + let tracker = LimitedTracker::new(ResourceLimits::new().max_allocations(500_000)); + + let mut progress = runner + .start(vec![], tracker, PrintWriter::Collect(&mut stdout)) + .expect("Failed to start orchestrator test"); + + loop { + match progress { + RunProgress::Complete(obj) => return obj, + RunProgress::FunctionCall(call) => { + if call.function_name == "FINAL" { + let val = call.args.first().cloned().unwrap_or(MontyObject::None); + let _ = call.resume( + ExtFunctionResult::Return(MontyObject::None), + PrintWriter::Collect(&mut stdout), + ); + return val; + } + let ext_result = match call.function_name.as_str() { + "__regex_match__" => handle_regex_match(&call.args), + _ => ExtFunctionResult::Return(MontyObject::None), + }; + progress = call + .resume(ext_result, PrintWriter::Collect(&mut stdout)) + .expect("resume failed"); + } + RunProgress::NameLookup(lookup) => { + progress = lookup + .resume( + NameLookupResult::Undefined, + PrintWriter::Collect(&mut stdout), + ) + .expect("name lookup resume failed"); + } + _ => panic!("Unexpected RunProgress variant in test"), + } + } + } + + fn eval_python_bool(expr: &str) -> bool { + // Extract only the helper functions (everything before run_loop) + let helpers_end = DEFAULT_ORCHESTRATOR + .find("\ndef run_loop(") + .unwrap_or(DEFAULT_ORCHESTRATOR.len()); + let helpers = &DEFAULT_ORCHESTRATOR[..helpers_end]; // safety: find() returns a char boundary on this ASCII-only constant + + let code = format!("{helpers}\nFINAL({expr})"); + match run_python_final(code) { + MontyObject::Bool(v) => v, + other => panic!("Expected bool, got: {other:?}"), + } + } + + /// Run a Python program (with orchestrator helpers in scope) that ends + /// with `FINAL(int_expr)` and return the integer value. + fn eval_python_int(program: &str) -> i64 { + let helpers_end = DEFAULT_ORCHESTRATOR + .find("\ndef run_loop(") + .unwrap_or(DEFAULT_ORCHESTRATOR.len()); + let helpers = &DEFAULT_ORCHESTRATOR[..helpers_end]; + + let code = format!("{helpers}\n{program}"); + match run_python_final(code) { + MontyObject::Int(v) => v, + other => panic!("Expected int, got: {other:?}"), + } + } + + // ── __regex_match__ host function reachability ─────────────── + + #[test] + fn regex_match_host_function_is_callable_from_monty() { + // Regression test for PR #1736 review (serrrfirat, 3059161877): + // verify that Monty's NameLookup + FunctionCall dispatch actually + // reaches `handle_regex_match` when default.py calls + // `__regex_match__(...)`. If Monty ever starts resolving the name + // before the call, this test will fail with a NameError. + assert!(eval_python_bool( + r#"bool(__regex_match__("abc", "xxabcxx"))"# + )); + assert!(!eval_python_bool( + r#"bool(__regex_match__("zzz", "xxabcxx"))"# + )); + // Invalid pattern should return false silently (the host function + // swallows the compile error). + assert!(!eval_python_bool(r#"bool(__regex_match__("[", "abc"))"#)); + } + + // ── True positives (should trigger nudge) ─────────────────── + + #[test] + fn signals_tool_intent_true_positives() { + assert!(eval_python_bool( + r#"signals_tool_intent("Let me search for that file.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("I'll fetch the data now.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("I'm going to check the logs.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("Let me add it now.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("I will run the tests to verify.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("I'll look up the documentation.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("Let me read the file contents.")"# + )); + assert!(eval_python_bool( + r#"signals_tool_intent("I'm going to execute the command.")"# + )); + } + + // ── True negatives: conversational phrases ────────────────── + + #[test] + fn signals_tool_intent_true_negatives_conversational() { + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me explain how this works.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me know if you need anything.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me think about this.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me summarize the findings.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me clarify what I mean.")"# + )); + } + + // ── Exclusion takes precedence ────────────────────────────── + + #[test] + fn signals_tool_intent_exclusion_takes_precedence() { + assert!(!eval_python_bool( + r#"signals_tool_intent("Let me explain the approach, then I'll search for the file.")"# + )); + } + + // ── Code blocks are stripped ──────────────────────────────── + + #[test] + fn signals_tool_intent_ignores_code_blocks() { + assert!(!eval_python_bool( + r#"signals_tool_intent("Here's the code:\n\n```\nfn main() {\n println!(\"Let me search the database\");\n}\n```")"# + )); + } + + #[test] + fn signals_tool_intent_ignores_indented_code() { + assert!(!eval_python_bool( + r#"signals_tool_intent("Here's the code:\n\n println!(\"I'll fetch the data\");\n\nThat's it.")"# + )); + } + + // ── Plain informational text ──────────────────────────────── + + #[test] + fn signals_tool_intent_ignores_plain_text() { + assert!(!eval_python_bool( + r#"signals_tool_intent("The task is complete.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Here are the results you asked for.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("I found 3 matching files.")"# + )); + } + + // ── Quoted strings are stripped ───────────────────────────── + + #[test] + fn signals_tool_intent_ignores_quoted_strings() { + assert!(!eval_python_bool( + r#"signals_tool_intent("The button says \"Let me search the database\" to the user.")"# + )); + // But unquoted intent should still trigger + assert!(eval_python_bool( + r#"signals_tool_intent("I'll fetch the results for you.")"# + )); + } + + // ── Shadowed prefix (exclusion cancels all) ───────────────── + + #[test] + fn signals_tool_intent_shadowed_prefix() { + // "let me think" is an exclusion → entire text returns false + assert!(!eval_python_bool( + r#"signals_tool_intent("Sure, let me think about it. Actually, let me search for the file.")"# + )); + } + + // ── Regression: trace false positive (news content) ───────── + + #[test] + fn signals_tool_intent_no_false_positive_news_content() { + // "I can" + "call" in news content triggered false positive in old code + let news_response = concat!( + "The latest headlines suggest this is a fast-moving war.\n", + "- Reuters: Iran is calling US peace proposals unrealistic.\n", + "If you want, I can do one of these next:\n", + "1. give you a 5-bullet update\n", + "2. focus just on military developments", + ); + assert!(!eval_python_bool(&format!( + "signals_tool_intent({news_response:?})" + ))); + } + + #[test] + fn signals_tool_intent_no_false_positive_past_tense() { + // "I fetched" / "I already called" should not trigger + assert!(!eval_python_bool( + r#"signals_tool_intent("I already completed the needed action call by fetching current news feeds.")"# + )); + assert!(!eval_python_bool( + r#"signals_tool_intent("Current status from the live feeds I fetched:")"# + )); + } + + #[test] + fn signals_tool_intent_no_false_positive_offer() { + // "If you want, I can fetch..." uses "I can" which is not a V1 prefix + assert!(!eval_python_bool( + r#"signals_tool_intent("If you want, I can next fetch a cleaner update.")"# + )); + } + + #[tokio::test] + async fn load_orchestrator_without_store_returns_default() { + let (code, version) = load_orchestrator(None, ProjectId::new(), true).await; + assert_eq!(version, 0); + assert!(code.contains("run_loop")); + assert!(code.contains("__llm_complete__")); + } + + #[tokio::test] + async fn load_orchestrator_with_runtime_version() { + let project_id = ProjectId::new(); + let mut doc = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "custom_orchestrator_code()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc.metadata = serde_json::json!({"version": 1}); + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let (code, version) = + load_orchestrator(Some(&(store as Arc)), project_id, true).await; + assert_eq!(version, 1); + assert!(code.contains("custom_orchestrator_code")); + } + + #[tokio::test] + async fn load_orchestrator_picks_highest_version() { + let project_id = ProjectId::new(); + let mut doc_v1 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v1_code()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v1.metadata = serde_json::json!({"version": 1}); + + let mut doc_v3 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v3_code()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v3.metadata = serde_json::json!({"version": 3}); + + let mut doc_v2 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v2_code()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v2.metadata = serde_json::json!({"version": 2}); + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![ + doc_v1, doc_v3, doc_v2, + ])); + let (code, version) = + load_orchestrator(Some(&(store as Arc)), project_id, true).await; + assert_eq!(version, 3); + assert!(code.contains("v3_code")); + } + + #[tokio::test] + async fn rollback_after_max_failures() { + let project_id = ProjectId::new(); + + // Create v2 orchestrator + let mut doc_v2 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v2_buggy()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v2.metadata = serde_json::json!({"version": 2}); + + // Create v1 orchestrator (fallback) + let mut doc_v1 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v1_stable()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v1.metadata = serde_json::json!({"version": 1}); + + // Create failure tracker showing v2 has 3 failures + let tracker = MemoryDoc::new( + project_id, + "system", + DocType::Note, + FAILURE_TRACKER_TITLE, + r#"{"version": 2, "count": 3}"#, + ) + .with_tags(vec!["orchestrator_meta".to_string()]); + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![ + doc_v2, doc_v1, tracker, + ])); + let (code, version) = + load_orchestrator(Some(&(store as Arc)), project_id, true).await; + + // Should skip v2 (too many failures) and load v1 + assert_eq!(version, 1); + assert!(code.contains("v1_stable")); + } + + #[tokio::test] + async fn rollback_to_default_when_all_versions_fail() { + let project_id = ProjectId::new(); + + // Single version with 3 failures + let mut doc_v1 = MemoryDoc::new( + project_id, + "system", + DocType::Note, + ORCHESTRATOR_TITLE, + "v1_broken()", + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc_v1.metadata = serde_json::json!({"version": 1}); + + let tracker = MemoryDoc::new( + project_id, + "system", + DocType::Note, + FAILURE_TRACKER_TITLE, + r#"{"version": 1, "count": 5}"#, + ) + .with_tags(vec!["orchestrator_meta".to_string()]); + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![ + doc_v1, tracker, + ])); + let (code, version) = + load_orchestrator(Some(&(store as Arc)), project_id, true).await; + + // Should fall back to compiled-in default (v0) + assert_eq!(version, 0); + assert!(code.contains("run_loop")); + } + + #[tokio::test] + async fn record_and_reset_failures() { + let project_id = ProjectId::new(); + let store: Arc = Arc::new(crate::tests::InMemoryStore::with_docs(vec![])); + + // Record 3 failures + record_orchestrator_failure(&store, project_id, 2).await; + record_orchestrator_failure(&store, project_id, 2).await; + record_orchestrator_failure(&store, project_id, 2).await; + + let docs = store.list_shared_memory_docs(project_id).await.unwrap(); + let count = load_failure_count(&docs); + assert_eq!(count, 3); + + // Reset + reset_orchestrator_failures(&store, project_id).await; + let docs = store.list_shared_memory_docs(project_id).await.unwrap(); + let count = load_failure_count(&docs); + assert_eq!(count, 0); + } + + #[tokio::test] + async fn failure_count_resets_on_new_version() { + let project_id = ProjectId::new(); + let store: Arc = Arc::new(crate::tests::InMemoryStore::with_docs(vec![])); + + // Record failures for version 1 + record_orchestrator_failure(&store, project_id, 1).await; + record_orchestrator_failure(&store, project_id, 1).await; + + // Switch to version 2 — count should reset to 1 + record_orchestrator_failure(&store, project_id, 2).await; + + let docs = store.list_shared_memory_docs(project_id).await.unwrap(); + let count = load_failure_count(&docs); + assert_eq!(count, 1); + } + + #[test] + fn normalize_pause_outcome_transitions_thread_to_waiting() { + let mut thread = Thread::new( + "goal", + crate::types::thread::ThreadType::Foreground, + ProjectId::new(), + "user", + crate::types::thread::ThreadConfig::default(), + ); + thread.transition_to(ThreadState::Running, None).unwrap(); + + let outcome = ThreadOutcome::GatePaused { + gate_name: "approval".into(), + action_name: "shell".into(), + call_id: "call-1".into(), + parameters: serde_json::json!({"cmd":"ls"}), + resume_kind: crate::gate::ResumeKind::Approval { allow_always: true }, + resume_output: None, + }; + normalize_pause_outcome(&mut thread, &outcome).unwrap(); + assert_eq!(thread.state, ThreadState::Waiting); + } + + #[test] + fn parse_outcome_completed() { + let result = serde_json::json!({"outcome": "completed", "response": "Hello!"}); + let outcome = parse_outcome(&result); + assert!(matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Hello!")); + } + + #[test] + fn parse_outcome_failed() { + let result = serde_json::json!({"outcome": "failed", "error": "boom"}); + let outcome = parse_outcome(&result); + assert!(matches!(outcome, ThreadOutcome::Failed { error } if error == "boom")); + } + + #[test] + fn parse_outcome_gate_paused() { + let result = serde_json::json!({ + "outcome": "gate_paused", + "gate_name": "approval", + "action_name": "shell", + "call_id": "abc", + "parameters": {"cmd": "rm -rf /"}, + "resume_kind": {"Approval": {"allow_always": true}} + }); + let outcome = parse_outcome(&result); + assert!( + matches!(outcome, ThreadOutcome::GatePaused { action_name, .. } if action_name == "shell") + ); + } + + #[test] + fn parse_outcome_max_iterations() { + let result = serde_json::json!({"outcome": "max_iterations"}); + let outcome = parse_outcome(&result); + assert!(matches!(outcome, ThreadOutcome::MaxIterations)); + } + + #[test] + fn parse_outcome_stopped() { + let result = serde_json::json!({"outcome": "stopped"}); + let outcome = parse_outcome(&result); + assert!(matches!(outcome, ThreadOutcome::Stopped)); + } + + // ── handle_llm_complete model forwarding ──────────────────── + + /// LLM backend that records the model from each `complete()` call. + /// Used to verify the orchestrator's __llm_complete__ host fn forwards + /// `explicit_config["model"]` onto `LlmCallConfig.model`. + struct ModelCapturingLlm { + captured: tokio::sync::Mutex>>, + } + + #[async_trait::async_trait] + impl LlmBackend for ModelCapturingLlm { + fn model_name(&self) -> &str { + "capturing" + } + + async fn complete( + &self, + _messages: &[ThreadMessage], + _actions: &[crate::types::capability::ActionDef], + config: &LlmCallConfig, + ) -> Result { + self.captured.lock().await.push(config.model.clone()); + Ok(crate::traits::llm::LlmOutput { + response: crate::types::step::LlmResponse::Text("ok".into()), + usage: crate::types::step::TokenUsage::default(), + }) + } + } + + /// No-op effect executor — handle_llm_complete only consults it for + /// `available_actions(...)`, which we satisfy with an empty list. + struct NoopEffects; + + #[async_trait::async_trait] + impl EffectExecutor for NoopEffects { + async fn execute_action( + &self, + _: &str, + _: serde_json::Value, + _: &crate::types::capability::CapabilityLease, + _: &ThreadExecutionContext, + ) -> Result { + Ok(crate::types::step::ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({}), + is_error: false, + duration: std::time::Duration::from_millis(1), + }) + } + + async fn available_actions( + &self, + _: &[crate::types::capability::CapabilityLease], + ) -> Result, EngineError> { + Ok(vec![]) + } + } + + #[tokio::test] + async fn llm_complete_forwards_model_from_explicit_config() { + let concrete = Arc::new(ModelCapturingLlm { + captured: tokio::sync::Mutex::new(Vec::new()), + }); + let llm: Arc = Arc::clone(&concrete) as Arc; + let effects: Arc = Arc::new(NoopEffects); + let leases = Arc::new(LeaseManager::new()); + let store: Arc = Arc::new(crate::tests::InMemoryStore::with_docs(vec![])); + + let mut thread = Thread::new( + "goal", + crate::types::thread::ThreadType::Foreground, + ProjectId::new(), + "test-user", + crate::types::thread::ThreadConfig::default(), + ); + thread.transition_to(ThreadState::Running, None).unwrap(); + + // Build the args __llm_complete__ receives from Python: + // (messages, actions, config). config = {"model": "gpt-4o"}. + let mut total_tokens = TokenUsage::default(); + let result = handle_llm_complete( + &[ + json_to_monty(&serde_json::json!([{"role":"user","content":"hi"}])), + json_to_monty(&serde_json::json!([])), + json_to_monty(&serde_json::json!({"model": "gpt-4o"})), + ], + &[], + &mut thread, + LlmCompleteDeps { + llm: &llm, + effects: &effects, + leases: &leases, + store: Some(&store), + }, + &mut total_tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Return(_))); + let captured = concrete.captured.lock().await; + assert_eq!(captured.len(), 1); + assert_eq!(captured[0].as_deref(), Some("gpt-4o")); + } + + #[tokio::test] + async fn llm_complete_without_model_passes_none() { + let concrete = Arc::new(ModelCapturingLlm { + captured: tokio::sync::Mutex::new(Vec::new()), + }); + let llm: Arc = Arc::clone(&concrete) as Arc; + let effects: Arc = Arc::new(NoopEffects); + let leases = Arc::new(LeaseManager::new()); + let store: Arc = Arc::new(crate::tests::InMemoryStore::with_docs(vec![])); + + let mut thread = Thread::new( + "goal", + crate::types::thread::ThreadType::Foreground, + ProjectId::new(), + "test-user", + crate::types::thread::ThreadConfig::default(), + ); + thread.transition_to(ThreadState::Running, None).unwrap(); + + let mut total_tokens = TokenUsage::default(); + let _ = handle_llm_complete( + &[ + json_to_monty(&serde_json::json!([{"role":"user","content":"hi"}])), + json_to_monty(&serde_json::json!([])), + json_to_monty(&serde_json::json!({"max_tokens": 100})), + ], + &[], + &mut thread, + LlmCompleteDeps { + llm: &llm, + effects: &effects, + leases: &leases, + store: Some(&store), + }, + &mut total_tokens, + ) + .await; + + let captured = concrete.captured.lock().await; + assert_eq!(captured.len(), 1); + assert_eq!(captured[0], None); + } + + // ── Python ↔ Rust ActionCall round-trip ─────────────────────────────── + // + // Regression tests for the orphaned-tool-result bug. The Python + // orchestrator stores `action_calls` on assistant messages using the + // shape `{name, call_id, params}`, but the canonical Rust `ActionCall` + // uses `{action_name, id, parameters}`. Without the explicit + // `PythonActionCall` interchange type, `serde_json::from_value` would + // silently fail (`.ok()` swallows the error) and the Python-shaped + // assistant message would be parsed back as a plain assistant message + // with no tool calls, causing every subsequent ActionResult to be + // detected as orphaned by `sanitize_tool_messages` in the host crate. + + #[test] + fn python_action_call_round_trips_through_serde() { + let original = ActionCall { + id: "call_abc123".to_string(), + action_name: "google_drive_tool".to_string(), + parameters: serde_json::json!({"query": "expenses"}), + }; + + let python_json = serde_json::to_value(PythonActionCall::from(&original)) + .expect("PythonActionCall must serialize"); + // Python-friendly field names — match what default.py reads. + assert_eq!(python_json["name"], "google_drive_tool"); + assert_eq!(python_json["call_id"], "call_abc123"); + assert_eq!( + python_json["params"], + serde_json::json!({"query": "expenses"}) + ); + + let parsed: PythonActionCall = + serde_json::from_value(python_json).expect("must deserialize"); + let round_tripped: ActionCall = parsed.into(); + assert_eq!(round_tripped.id, original.id); + assert_eq!(round_tripped.action_name, original.action_name); + assert_eq!(round_tripped.parameters, original.parameters); + } + + #[test] + fn action_calls_to_python_json_uses_python_field_names() { + let calls = vec![ + ActionCall { + id: "call_1".to_string(), + action_name: "notion_notion_search".to_string(), + parameters: serde_json::json!({"query": "name"}), + }, + ActionCall { + id: "call_2".to_string(), + action_name: "google_drive_tool".to_string(), + parameters: serde_json::json!({"action": "list"}), + }, + ]; + let json = action_calls_to_python_json(&calls); + assert_eq!(json.len(), 2); + assert_eq!(json[0]["name"], "notion_notion_search"); + assert_eq!(json[0]["call_id"], "call_1"); + assert_eq!(json[1]["name"], "google_drive_tool"); + assert_eq!(json[1]["call_id"], "call_2"); + } + + #[test] + fn python_json_to_action_calls_parses_python_field_names() { + // The exact shape default.py produces (and stores on assistant + // messages via `append_message(..., action_calls=calls)`). + let python_json = serde_json::json!([ + {"name": "notion_notion_search", "call_id": "call_xyz", "params": {"q": "foo"}}, + {"name": "google_drive_tool", "call_id": "call_abc", "params": {"action": "list"}}, + ]); + let parsed = python_json_to_action_calls(&python_json).expect("must parse"); + assert_eq!(parsed.len(), 2); + assert_eq!(parsed[0].action_name, "notion_notion_search"); + assert_eq!(parsed[0].id, "call_xyz"); + assert_eq!(parsed[0].parameters, serde_json::json!({"q": "foo"})); + assert_eq!(parsed[1].action_name, "google_drive_tool"); + assert_eq!(parsed[1].id, "call_abc"); + } + + #[test] + fn python_json_to_action_calls_rejects_canonical_field_names() { + // Sanity check: the parser is strict about Python field names. + // If `default.py` ever changes the shape, the test must catch it. + let canonical_json = serde_json::json!([ + {"action_name": "search", "id": "call_x", "parameters": {}} + ]); + // Missing "name", "call_id", "params" → returns None. + assert!(python_json_to_action_calls(&canonical_json).is_none()); + } + + #[test] + fn summarize_action_calls_for_log_does_not_leak_user_pii() { + // The whole point of this helper is that the warn log path on a + // shape-drift failure must NOT dump tool parameters (which can + // contain user PII like search queries, file names, email content) + // into log aggregation systems. The summary should expose only + // structural information: array length and the keys of the first + // entry. The keys themselves are static (`name`, `call_id`, + // `params`), not user data. + let pii_value = serde_json::json!([ + { + "name": "google_drive_tool", + "call_id": "call_xyz", + "params": { + "query": "salary spreadsheet for joe", + "secret_token": "very-sensitive-token-do-not-log" + } + }, + { + "name": "gmail", + "call_id": "call_abc", + "params": { + "subject": "private message about layoffs" + } + } + ]); + let summary = summarize_action_calls_for_log(&pii_value); + + // Structural info present. + assert!(summary.contains("array of 2 entries")); + assert!(summary.contains("call_id")); + assert!(summary.contains("name")); + assert!(summary.contains("params")); + + // PII fields and their values must NOT appear. + assert!( + !summary.contains("salary"), + "summary must not leak user PII from params: {summary}" + ); + assert!( + !summary.contains("very-sensitive-token"), + "summary must not leak credential-shaped values: {summary}" + ); + assert!( + !summary.contains("layoffs"), + "summary must not leak free-text content: {summary}" + ); + assert!( + !summary.contains("google_drive_tool"), + "summary must not leak the tool name itself (could expose intent): {summary}" + ); + } + + #[test] + fn summarize_action_calls_for_log_handles_edge_cases() { + assert_eq!( + summarize_action_calls_for_log(&serde_json::json!([])), + "empty array" + ); + assert!( + summarize_action_calls_for_log(&serde_json::json!("not an array")).contains("string") + ); + assert!( + summarize_action_calls_for_log(&serde_json::json!({"foo": "bar"})).contains("object") + ); + assert!(summarize_action_calls_for_log(&serde_json::json!(null)).contains("null")); + } + + /// Caller-level regression test: feeds `json_to_thread_messages` the + /// exact JSON shape that `default.py` produces for an assistant message + /// with tool calls followed by tool results, and asserts that the + /// resulting `ThreadMessage`s preserve the `action_calls` ↔ + /// `action_call_id` linkage. Without the `PythonActionCall` parser the + /// assistant message would come back with `action_calls = None` and + /// every following ActionResult would look orphaned to the bridge. + #[test] + fn json_to_thread_messages_preserves_action_calls_from_python_orchestrator() { + // This is the literal shape `default.py` writes into + // `state["working_messages"]` after a Tier 0 step: + // + // append_message(working_messages, "Assistant", "...", action_calls=calls) + // append_message(working_messages, "ActionResult", "...", action_name=..., action_call_id=...) + // + // where `calls` came from the LLM response and has shape + // `[{"name": ..., "call_id": ..., "params": ...}]`. + let working_messages = serde_json::json!([ + {"role": "User", "content": "search in notion for my name"}, + { + "role": "Assistant", + "content": "", + "action_calls": [ + { + "name": "notion_notion_search", + "call_id": "call_xyz", + "params": {"query": "Illia"} + } + ] + }, + { + "role": "ActionResult", + "content": "found 3 results", + "action_name": "notion_notion_search", + "action_call_id": "call_xyz" + } + ]); + + let messages = json_to_thread_messages(&working_messages).expect("must parse"); + assert_eq!(messages.len(), 3); + + // The assistant message MUST have action_calls populated, with + // matching call_id. If this assertion fails, the bridge layer + // will treat the following ActionResult as orphaned and rewrite + // it as a user message — losing the model's ability to reason + // about prior tool output. + let assistant = &messages[1]; + assert_eq!( + assistant.role, + crate::types::message::MessageRole::Assistant + ); + let calls = assistant + .action_calls + .as_ref() + .expect("assistant message must carry action_calls after round-trip"); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].id, "call_xyz"); + assert_eq!(calls[0].action_name, "notion_notion_search"); + assert_eq!(calls[0].parameters, serde_json::json!({"query": "Illia"})); + + // The ActionResult must reference the same call_id so the bridge + // can pair them. + let result = &messages[2]; + assert_eq!( + result.role, + crate::types::message::MessageRole::ActionResult + ); + assert_eq!(result.action_call_id.as_deref(), Some("call_xyz")); + assert_eq!(result.action_name.as_deref(), Some("notion_notion_search")); + } + + /// Regression for the gate-resume / bootstrap path: when a thread + /// resumes after approval or auth, `build_orchestrator_inputs` + /// serializes `thread.internal_messages` into the bootstrap context + /// that Python reads into `working_messages`. If `action_calls` is + /// serialized with canonical `ActionCall` field names (`action_name`, + /// `id`, `parameters`) instead of the Python interchange names + /// (`name`, `call_id`, `params`), the next `__llm_complete__` call + /// passes them back through `json_to_thread_messages` which fails + /// with "missing field `name`" and orphans every subsequent tool + /// result. + /// + /// This test simulates the full round-trip: build a `ThreadMessage` + /// with action_calls → serialize through `build_orchestrator_inputs`'s + /// exact serialization pattern → parse back through + /// `json_to_thread_messages` → assert the calls survive. If anyone + /// adds a THIRD serialization path in the future and uses canonical + /// names, this test documents the pattern they should follow. + #[test] + fn bootstrap_context_action_calls_round_trip_through_python_interchange() { + // Build a thread message the way the engine does: an assistant + // message with action_calls in canonical ActionCall format (the + // shape stored in the DB / internal_messages). + let msg = ThreadMessage::assistant_with_actions( + Some("I'll search for that".to_string()), + vec![ActionCall { + id: "call_resume_test".to_string(), + action_name: "google_drive_tool".to_string(), + parameters: serde_json::json!({"query": "budget"}), + }], + ); + + // Serialize through the SAME pattern `build_orchestrator_inputs` + // uses. This is the exact code path that was broken before the + // fix — it was using `"action_calls": m.action_calls` which + // produced canonical field names. + let calls_json = msg + .action_calls + .as_ref() + .map(|calls| serde_json::Value::Array(action_calls_to_python_json(calls))); + let serialized = serde_json::json!([{ + "role": "Assistant", + "content": msg.content, + "action_name": msg.action_name, + "action_call_id": msg.action_call_id, + "action_calls": calls_json, + }]); + + // Parse back through the same path Python's working_messages + // takes when it calls __llm_complete__. + let parsed = json_to_thread_messages(&serialized).expect("must parse"); + assert_eq!(parsed.len(), 1); + + let assistant = &parsed[0]; + let calls = assistant.action_calls.as_ref().expect( + "bootstrap context action_calls must survive the round-trip. \ + If this fails, a serialization path is using canonical ActionCall \ + field names instead of PythonActionCall interchange names.", + ); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].id, "call_resume_test"); + assert_eq!(calls[0].action_name, "google_drive_tool"); + assert_eq!(calls[0].parameters, serde_json::json!({"query": "budget"})); + } + + /// Negative regression: verify that canonical ActionCall field names + /// do NOT round-trip. If this test ever PASSES, it means someone + /// added `#[serde(rename)]` to ActionCall or changed the parser to + /// accept both formats — which is fine, but the PythonActionCall + /// interchange type can then be removed. This test documents the + /// current contract: canonical names are rejected by the parser. + #[test] + fn canonical_action_call_field_names_do_not_round_trip() { + let serialized_with_canonical_names = serde_json::json!([{ + "role": "Assistant", + "content": "", + "action_calls": [{ + "action_name": "search", + "id": "call_x", + "parameters": {} + }], + }]); + let parsed = + json_to_thread_messages(&serialized_with_canonical_names).expect("messages parse"); + // The assistant message should have NO action_calls because the + // parser rejects canonical field names. + assert!( + parsed[0].action_calls.is_none(), + "canonical ActionCall field names must NOT parse as action_calls. \ + If this assertion fails, the PythonActionCall interchange type \ + is no longer needed — either remove it or update the contract." + ); + } + + /// Regression: `action_calls: null` is Python's legitimate "this + /// message has no tool calls" signal (text-only response). Before the + /// null filter, `python_json_to_action_calls` would fire a warn log + /// with "invalid type: null, expected a sequence" on every text-only + /// assistant message — a false alarm that masked real drift issues. + #[test] + fn json_to_thread_messages_handles_null_action_calls_gracefully() { + let messages = serde_json::json!([ + { + "role": "Assistant", + "content": "Here is your answer.", + "action_calls": null + } + ]); + let parsed = json_to_thread_messages(&messages).expect("must parse"); + assert_eq!(parsed.len(), 1); + assert_eq!( + parsed[0].role, + crate::types::message::MessageRole::Assistant + ); + assert_eq!(parsed[0].content, "Here is your answer."); + assert!( + parsed[0].action_calls.is_none(), + "null action_calls must produce None, not a parse error" + ); + } + + /// Verify that messages WITHOUT the action_calls key at all (the most + /// common case for text responses) also parse correctly — this is the + /// baseline that the null-filtering regression test extends. + #[test] + fn json_to_thread_messages_handles_absent_action_calls() { + let messages = serde_json::json!([ + {"role": "Assistant", "content": "Just text, no tools."} + ]); + let parsed = json_to_thread_messages(&messages).expect("must parse"); + assert_eq!(parsed.len(), 1); + assert!(parsed[0].action_calls.is_none()); + } + + /// Empty action_calls array is valid (LLM decided not to call any + /// tools this turn but the response still has the array field). Must + /// produce `Some(vec![])`, not `None`. + #[test] + fn json_to_thread_messages_handles_empty_action_calls_array() { + let messages = serde_json::json!([ + { + "role": "Assistant", + "content": "No tools needed.", + "action_calls": [] + } + ]); + let parsed = json_to_thread_messages(&messages).expect("must parse"); + assert_eq!(parsed.len(), 1); + let calls = parsed[0] + .action_calls + .as_ref() + .expect("empty array should produce Some(vec![])"); + assert!(calls.is_empty()); + } + + // ── Consecutive action error counting (issue #2325) ────────── + // + // The run_loop tracks `consecutive_action_errors` for Tier 0 (structured + // action calls). These tests exercise the counting logic extracted from + // run_loop into small Python snippets that simulate batch outcomes. + + #[test] + fn action_errors_increment_when_all_actions_fail() { + // Simulate 3 consecutive batches where all actions fail. + let count = eval_python_int( + r#" +consecutive_action_errors = 0 +for _ in range(3): + batch_error_count = 2 + batch_success_count = 0 + if batch_success_count > 0: + consecutive_action_errors = 0 + elif batch_error_count > 0: + consecutive_action_errors += 1 +FINAL(consecutive_action_errors) +"#, + ); + assert_eq!(count, 3); + } + + #[test] + fn action_errors_reset_when_any_action_succeeds() { + // 2 all-fail batches, then 1 batch with a success => resets to 0. + let count = eval_python_int( + r#" +consecutive_action_errors = 0 +for batch in [(0, 2), (0, 1), (1, 1)]: + batch_success_count = batch[0] + batch_error_count = batch[1] + if batch_success_count > 0: + consecutive_action_errors = 0 + elif batch_error_count > 0: + consecutive_action_errors += 1 +FINAL(consecutive_action_errors) +"#, + ); + assert_eq!(count, 0); + } + + #[test] + fn action_errors_partial_success_resets_counter() { + // A batch with mixed results (some succeed, some fail) should reset. + let count = eval_python_int( + r#" +consecutive_action_errors = 5 +batch_success_count = 1 +batch_error_count = 3 +if batch_success_count > 0: + consecutive_action_errors = 0 +elif batch_error_count > 0: + consecutive_action_errors += 1 +FINAL(consecutive_action_errors) +"#, + ); + assert_eq!(count, 0); + } + + #[test] + fn action_errors_nudge_injected_at_threshold() { + // When consecutive_action_errors reaches max_consecutive_errors, + // a nudge message should be appended. We simulate the branching + // logic and check whether a nudge would fire. + // Returns 1 if nudge fires (not failure), 0 otherwise. + let result = eval_python_int( + r#" +max_consecutive_errors = 5 +consecutive_action_errors = 5 +nudge = False +failed = False +if consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + failed = True +elif consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors: + nudge = True +if nudge and not failed: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 1, "nudge should fire at threshold"); + } + + #[test] + fn action_errors_no_nudge_below_threshold() { + // Returns 1 if nudge fires, 0 if not. + let result = eval_python_int( + r#" +max_consecutive_errors = 5 +consecutive_action_errors = 4 +nudge = False +failed = False +if consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + failed = True +elif consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors: + nudge = True +if nudge: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 0, "nudge should not fire below threshold"); + } + + #[test] + fn action_errors_failure_at_threshold_plus_two() { + // At max_consecutive_errors + 2, the thread should transition to failed. + // Returns 1 if failed, 0 if not. + let result = eval_python_int( + r#" +max_consecutive_errors = 5 +consecutive_action_errors = 7 +failed = False +if consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + failed = True +if failed: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 1, "should fail at threshold + 2"); + } + + #[test] + fn action_errors_nudge_at_threshold_not_failure() { + // At exactly max_consecutive_errors + 1, we get a nudge but not failure. + let result = eval_python_int( + r#" +max_consecutive_errors = 5 +consecutive_action_errors = 6 +nudge = False +failed = False +if consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + failed = True +elif consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors: + nudge = True +# Return 0=nothing, 1=nudge, 2=failed +if failed: + FINAL(2) +elif nudge: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 1, "should nudge at threshold + 1, not fail"); + } + + #[test] + fn action_errors_none_limit_skips_check_without_typeerror() { + // Regression: when max_consecutive_errors is None (meaning "no limit"), + // the arithmetic `max_consecutive_errors + 2` used to crash with + // TypeError on the first action error. The guard must short-circuit + // on None and leave both the nudge and failure branches untaken. + let result = eval_python_int( + r#" +max_consecutive_errors = None +consecutive_action_errors = 1 +nudge = False +failed = False +if max_consecutive_errors is not None and consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors + 2: + failed = True +elif max_consecutive_errors is not None and consecutive_action_errors > 0 and consecutive_action_errors >= max_consecutive_errors: + nudge = True +# Return 0=nothing, 1=nudge, 2=failed +if failed: + FINAL(2) +elif nudge: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 0, "None limit should disable the guard entirely"); + } + + #[test] + fn code_errors_none_limit_skips_failure_check() { + // Regression: same None-guard for the code-error branch at line 660. + let result = eval_python_int( + r#" +max_consecutive_errors = None +consecutive_errors = 99 +failed = False +if max_consecutive_errors is not None and consecutive_errors >= max_consecutive_errors: + failed = True +if failed: + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!( + result, 0, + "None limit should not trigger failure regardless of consecutive_errors" + ); + } + + #[test] + fn action_error_prefix_added_to_error_output() { + // Verify that [ACTION FAILED] prefix is prepended to error outputs. + // Returns 1 if prefix present, 0 if not. + let result = eval_python_int( + r#" +r = {"action_name": "http", "output": "connection refused", "is_error": True} +output = r.get("output") +output_str = str(output) if output is not None else "[no output]" +if r.get("is_error"): + output_str = "[ACTION FAILED] " + output_str +if output_str.startswith("[ACTION FAILED]"): + FINAL(1) +else: + FINAL(0) +"#, + ); + assert_eq!(result, 1, "error outputs must get [ACTION FAILED] prefix"); + } + + #[test] + fn action_error_skipped_calls_count_as_errors() { + // When a call has no result (r is None), it should count as an error. + let count = eval_python_int( + r#" +batch_error_count = 0 +batch_success_count = 0 +r = None +if r is not None: + if r.get("is_error"): + batch_error_count += 1 + else: + batch_success_count += 1 +else: + batch_error_count += 1 +FINAL(batch_error_count) +"#, + ); + assert_eq!(count, 1, "skipped calls must count as batch errors"); + } + + #[test] + fn checkpoint_includes_consecutive_action_errors() { + // Test that handle_save_checkpoint persists consecutive_action_errors + // in the thread metadata. + let mut thread = Thread::new( + "goal", + crate::types::thread::ThreadType::Foreground, + ProjectId::new(), + "user", + crate::types::thread::ThreadConfig::default(), + ); + thread.transition_to(ThreadState::Running, None).unwrap(); + + let state = json_to_monty(&serde_json::json!({})); + let counters = json_to_monty(&serde_json::json!({ + "nudge_count": 0, + "consecutive_errors": 1, + "consecutive_action_errors": 4, + "compaction_count": 2, + })); + + handle_save_checkpoint(&[state, counters], &[], &mut thread); + + let checkpoint = thread + .metadata + .get("runtime_checkpoint") + .expect("checkpoint must exist"); + assert_eq!( + checkpoint + .get("consecutive_action_errors") + .and_then(|v| v.as_u64()), + Some(4), + "consecutive_action_errors must be persisted in checkpoint" + ); + assert_eq!( + checkpoint + .get("consecutive_errors") + .and_then(|v| v.as_u64()), + Some(1), + ); + assert_eq!( + checkpoint.get("compaction_count").and_then(|v| v.as_u64()), + Some(2), + ); + } + + /// Regression test: every assistant tool_call must have a matching + /// ActionResult after parsing. If an ActionResult is missing, the LLM + /// API rejects with "No tool output found for function call ". + /// + /// This was the root cause of the HTTP 400 from the OpenAI Codex + /// provider: a tool returning null output caused the Python + /// orchestrator to skip appending the ActionResult. + #[test] + fn json_to_thread_messages_every_tool_call_has_action_result() { + // Simulate working_messages after the Python fix: every call gets + // an ActionResult, even when the original output was null. + let messages = serde_json::json!([ + {"role": "System", "content": "You are a helpful assistant."}, + {"role": "User", "content": "Update all tools."}, + { + "role": "Assistant", + "content": "", + "action_calls": [ + {"call_id": "call_AAA", "name": "tool_a", "params": {}}, + {"call_id": "call_BBB", "name": "tool_b", "params": {}}, + {"call_id": "call_CCC", "name": "tool_c", "params": {}} + ] + }, + { + "role": "ActionResult", + "content": "{\"ok\": true}", + "action_name": "tool_a", + "action_call_id": "call_AAA" + }, + { + "role": "ActionResult", + "content": "[no output]", + "action_name": "tool_b", + "action_call_id": "call_BBB" + }, + { + "role": "ActionResult", + "content": "{\"done\": true}", + "action_name": "tool_c", + "action_call_id": "call_CCC" + } + ]); + + let parsed = json_to_thread_messages(&messages).expect("must parse"); + assert_eq!(parsed.len(), 6); + + // Extract call IDs from the assistant message + let assistant_calls: std::collections::HashSet = parsed + .iter() + .filter_map(|m| m.action_calls.as_ref()) + .flat_map(|calls| calls.iter().map(|c| c.id.clone())) + .collect(); + + // Extract call IDs from ActionResult messages + let result_call_ids: std::collections::HashSet = parsed + .iter() + .filter(|m| m.role == crate::types::message::MessageRole::ActionResult) + .filter_map(|m| m.action_call_id.clone()) + .collect(); + + // Every tool_call must have a matching ActionResult + for call_id in &assistant_calls { + assert!( + result_call_ids.contains(call_id), + "tool_call {call_id} has no matching ActionResult — \ + this would cause 'No tool output found' from the LLM API" + ); + } + } +} diff --git a/crates/ironclaw_engine/src/executor/prompt.rs b/crates/ironclaw_engine/src/executor/prompt.rs new file mode 100644 index 00000000000..f54c92376de --- /dev/null +++ b/crates/ironclaw_engine/src/executor/prompt.rs @@ -0,0 +1,317 @@ +//! System prompt construction for the execution loop. +//! +//! Builds a CodeAct/RLM system prompt that instructs the LLM to write +//! Python code in ```repl blocks with tools available as callable functions. +//! +//! Prompt templates live in `crates/ironclaw_engine/prompts/` as plain +//! markdown files for easy inspection and iteration. They are embedded +//! at compile time via `include_str!` and can be extended at runtime with +//! prompt overlays stored as MemoryDocs. + +use std::sync::Arc; + +use crate::traits::store::Store; +use crate::types::capability::ActionDef; +use crate::types::project::ProjectId; + +/// Runtime platform metadata injected into system prompts for self-awareness. +/// +/// Provides the agent with knowledge about its own identity and environment +/// so it can answer questions about itself, its capabilities, and its +/// configuration without relying on training data. +#[derive(Debug, Clone, Default)] +pub struct PlatformInfo { + /// Software version (from CARGO_PKG_VERSION). + pub version: Option, + /// LLM backend name (e.g. "nearai", "openai", "anthropic"). + pub llm_backend: Option, + /// Active model name. + pub model_name: Option, + /// Database backend (e.g. "libsql", "postgres"). + pub database_backend: Option, + /// Active channel names (e.g. ["telegram", "cli"]). + pub active_channels: Vec, + /// Owner identifier. + pub owner_id: Option, + /// Project repository URL. + pub repo_url: Option, +} + +impl PlatformInfo { + /// Format as a prompt section. Returns empty string if no info is set. + pub fn to_prompt_section(&self) -> String { + let mut lines = Vec::new(); + + lines.push("You are **IronClaw**, a secure autonomous AI assistant platform.".into()); + if let Some(ref v) = self.version { + lines.push(format!("- Version: {v}")); + } + if let Some(ref repo) = self.repo_url { + lines.push(format!("- Repository: {repo}")); + } + if let Some(ref owner) = self.owner_id { + lines.push(format!("- Owner: {owner}")); + } + if let Some(ref backend) = self.llm_backend { + let model = self.model_name.as_deref().unwrap_or("default"); + lines.push(format!("- LLM: {backend} ({model})")); + } + if let Some(ref db) = self.database_backend { + lines.push(format!("- Database: {db}")); + } + if !self.active_channels.is_empty() { + lines.push(format!("- Channels: {}", self.active_channels.join(", "))); + } + + if lines.len() <= 1 { + // Only the identity line, no runtime details — still include it + return format!("\n\n## Platform\n\n{}\n", lines[0]); + } + + format!("\n\n## Platform\n\n{}\n", lines.join("\n")) + } +} + +/// The main instruction block (before tool listing). +const CODEACT_PREAMBLE: &str = include_str!("../../prompts/codeact_preamble.md"); + +/// The strategy/closing block (after tool listing). +const CODEACT_POSTAMBLE: &str = include_str!("../../prompts/codeact_postamble.md"); + +/// Well-known title for the CodeAct preamble overlay. +pub const PREAMBLE_OVERLAY_TITLE: &str = "prompt:codeact_preamble"; + +/// Well-known tag for prompt overlay docs. +pub const PROMPT_OVERLAY_TAG: &str = "prompt_overlay"; + +/// Maximum size for a prompt overlay document (in chars). +const MAX_PROMPT_OVERLAY_CHARS: usize = 4000; + +/// Build the system prompt for CodeAct/RLM execution. +/// +/// The prompt instructs the LLM to: +/// - Write Python code in ```repl fenced blocks +/// - Call tools as regular Python functions +/// - Use llm_query(prompt, context) for sub-agent calls +/// - Use FINAL(answer) to return the final answer +/// - Access thread context via the `context` variable +/// +/// If a Store is provided, checks for a runtime prompt overlay (a MemoryDoc +/// with tag "prompt_overlay" and title "prompt:codeact_preamble") and appends +/// its content after the compiled preamble. This enables the self-improvement +/// mission to evolve the system prompt at runtime. +pub async fn build_codeact_system_prompt( + actions: &[ActionDef], + store: Option<&Arc>, + project_id: ProjectId, + platform: Option<&PlatformInfo>, +) -> String { + let overlay = if let Some(store) = store { + load_prompt_overlay(store, project_id).await + } else { + None + }; + build_codeact_system_prompt_inner(actions, overlay.as_deref(), platform) +} + +/// Build the system prompt using pre-fetched memory docs. +/// +/// When the caller already has the `list_memory_docs` result (e.g. because +/// `load_orchestrator` fetched it), pass the docs here to avoid a duplicate +/// Store query. +pub fn build_codeact_system_prompt_with_docs( + actions: &[ActionDef], + system_docs: &[crate::types::memory::MemoryDoc], + platform: Option<&PlatformInfo>, +) -> String { + let overlay = extract_prompt_overlay(system_docs); + build_codeact_system_prompt_inner(actions, overlay.as_deref(), platform) +} + +/// Shared prompt builder used by both the async and pre-fetched-docs variants. +fn build_codeact_system_prompt_inner( + actions: &[ActionDef], + overlay: Option<&str>, + platform: Option<&PlatformInfo>, +) -> String { + let mut prompt = String::from(CODEACT_PREAMBLE); + + // Inject platform identity and runtime metadata + if let Some(info) = platform { + prompt.push_str(&info.to_prompt_section()); + } + + // Append runtime prompt overlay if available + if let Some(overlay) = overlay { + prompt.push_str("\n\n## Learned Rules (from self-improvement)\n\n"); + prompt.push_str(overlay); + } + + // Add tool documentation + if !actions.is_empty() { + prompt.push_str("\n## Available tools (call as Python functions)\n\n"); + for action in actions { + prompt.push_str(&format!("- `{}(", action.name)); + // Extract parameter names from JSON schema + if let Some(props) = action.parameters_schema.get("properties") + && let Some(obj) = props.as_object() + { + let params: Vec<&str> = obj.keys().map(String::as_str).collect(); + prompt.push_str(¶ms.join(", ")); + } + prompt.push_str(&format!(")` — {}\n", action.description)); + } + } + + prompt.push_str(CODEACT_POSTAMBLE); + prompt +} + +/// Load the prompt overlay from the Store, if one exists for this project. +async fn load_prompt_overlay(store: &Arc, project_id: ProjectId) -> Option { + let docs = store.list_shared_memory_docs(project_id).await.ok()?; + extract_prompt_overlay(&docs) +} + +/// Extract the prompt overlay from a pre-fetched list of system memory docs. +pub fn extract_prompt_overlay(docs: &[crate::types::memory::MemoryDoc]) -> Option { + let overlay = docs.iter().find(|d| { + d.title == PREAMBLE_OVERLAY_TITLE && d.tags.contains(&PROMPT_OVERLAY_TAG.to_string()) + })?; + + let content: String = overlay + .content + .chars() + .take(MAX_PROMPT_OVERLAY_CHARS) + .collect(); + if content.is_empty() { + return None; + } + Some(content) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::memory::{DocId, DocType, MemoryDoc}; + use crate::types::shared_owner_id; + + #[tokio::test] + async fn prompt_without_store_uses_compiled_preamble() { + let prompt = + build_codeact_system_prompt(&[], None, ProjectId(uuid::Uuid::nil()), None).await; + assert!(prompt.contains("Python REPL environment")); + assert!(prompt.contains("Strategy")); + assert!(!prompt.contains("Learned Rules")); + } + + #[tokio::test] + async fn prompt_with_overlay_appends_rules() { + let project_id = ProjectId(uuid::Uuid::new_v4()); + let overlay = MemoryDoc { + id: DocId::new(), + project_id, + user_id: shared_owner_id().into(), + doc_type: DocType::Note, + title: PREAMBLE_OVERLAY_TITLE.into(), + content: "9. Never call web_fetch — use http() instead.".into(), + source_thread_id: None, + tags: vec![PROMPT_OVERLAY_TAG.into()], + metadata: serde_json::json!({}), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + }; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![overlay])); + let prompt = + build_codeact_system_prompt(&[], Some(&(store as Arc)), project_id, None) + .await; + assert!(prompt.contains("Learned Rules")); + assert!(prompt.contains("Never call web_fetch")); + } + + #[tokio::test] + async fn prompt_overlay_size_is_capped() { + let project_id = ProjectId(uuid::Uuid::new_v4()); + // Create an overlay that exceeds MAX_PROMPT_OVERLAY_CHARS using a char + // not found in the compiled preamble/postamble + let huge_content = "\u{2603}".repeat(MAX_PROMPT_OVERLAY_CHARS + 1000); // snowman + let overlay = MemoryDoc { + id: DocId::new(), + project_id, + user_id: shared_owner_id().into(), + doc_type: DocType::Note, + title: PREAMBLE_OVERLAY_TITLE.into(), + content: huge_content, + source_thread_id: None, + tags: vec![PROMPT_OVERLAY_TAG.into()], + metadata: serde_json::json!({}), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + }; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![overlay])); + let prompt = + build_codeact_system_prompt(&[], Some(&(store as Arc)), project_id, None) + .await; + + let snowman_count = prompt.chars().filter(|c| *c == '\u{2603}').count(); + assert_eq!(snowman_count, MAX_PROMPT_OVERLAY_CHARS); + } + + #[tokio::test] + async fn prompt_ignores_wrong_project_overlay() { + let project_id = ProjectId(uuid::Uuid::new_v4()); + let other_project = ProjectId(uuid::Uuid::new_v4()); + let overlay = MemoryDoc { + id: DocId::new(), + project_id: other_project, + user_id: shared_owner_id().into(), + doc_type: DocType::Note, + title: PREAMBLE_OVERLAY_TITLE.into(), + content: "Should not appear".into(), + source_thread_id: None, + tags: vec![PROMPT_OVERLAY_TAG.into()], + metadata: serde_json::json!({}), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + }; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![overlay])); + let prompt = + build_codeact_system_prompt(&[], Some(&(store as Arc)), project_id, None) + .await; + assert!(!prompt.contains("Should not appear")); + assert!(!prompt.contains("Learned Rules")); + } + + #[tokio::test] + async fn prompt_with_platform_info_injects_identity() { + let info = PlatformInfo { + version: Some("1.2.3".into()), + llm_backend: Some("nearai".into()), + model_name: Some("qwen3-235b".into()), + database_backend: Some("libsql".into()), + active_channels: vec!["telegram".into(), "cli".into()], + owner_id: Some("alice.near".into()), + repo_url: Some("https://github.com/nearai/ironclaw".into()), + }; + let prompt = + build_codeact_system_prompt(&[], None, ProjectId(uuid::Uuid::nil()), Some(&info)).await; + assert!(prompt.contains("IronClaw")); + assert!(prompt.contains("1.2.3")); + assert!(prompt.contains("nearai")); + assert!(prompt.contains("qwen3-235b")); + assert!(prompt.contains("libsql")); + assert!(prompt.contains("telegram")); + assert!(prompt.contains("alice.near")); + assert!(prompt.contains("github.com/nearai/ironclaw")); + } + + #[tokio::test] + async fn prompt_without_platform_info_has_no_platform_section() { + let prompt = + build_codeact_system_prompt(&[], None, ProjectId(uuid::Uuid::nil()), None).await; + assert!(!prompt.contains("## Platform")); + } +} diff --git a/crates/ironclaw_engine/src/executor/scripting.rs b/crates/ironclaw_engine/src/executor/scripting.rs new file mode 100644 index 00000000000..11f3f09c6c6 --- /dev/null +++ b/crates/ironclaw_engine/src/executor/scripting.rs @@ -0,0 +1,2739 @@ +//! Tier 1 executor: embedded Python via Monty. +//! +//! Executes LLM-generated Python code using the Monty interpreter. Tool +//! calls use **async dispatch**: each tool call returns a Monty `ExternalFuture` +//! via `resume_pending()`, allowing Python code to use `await` and +//! `asyncio.gather()` for parallel execution. When all tasks are blocked, +//! Monty yields `ResolveFutures` and we execute pending tools concurrently +//! via `JoinSet`. +//! +//! Follows the RLM (Recursive Language Model) pattern: +//! - Thread context injected as Python variables (not LLM attention input) +//! - `llm_query()` / `llm_query_batched()` for recursive subagent spawning +//! - `FINAL(answer)` / `FINAL_VAR(name)` for explicit termination +//! - Step 0 orientation preamble for context awareness +//! - Errors flow back to LLM for self-correction (not step termination) +//! - Output truncated to configurable limit with variable listing +//! - `asyncio.gather()` for parallel tool execution (via ResolveFutures) + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; + +use monty::{ + ExcType, ExtFunctionResult, LimitedTracker, MontyException, MontyObject, MontyRun, + NameLookupResult, PrintWriter, ResourceLimits, RunProgress, +}; +use tracing::debug; + +use crate::capability::lease::LeaseManager; +use crate::capability::policy::{PolicyDecision, PolicyEngine}; +use crate::traits::effect::{EffectExecutor, ThreadExecutionContext}; +use crate::traits::llm::{LlmBackend, LlmCallConfig}; +use crate::types::error::EngineError; +use crate::types::event::EventKind; +use crate::types::message::{MessageRole, ThreadMessage}; +use crate::types::step::{ActionResult, LlmResponse, TokenUsage}; +use crate::types::thread::Thread; +use ironclaw_common::ValidTimezone; + +// ── Configuration ─────────────────────────────────────────── + +/// Maximum characters of output to include in LLM context between steps. +/// Matches Prime Intellect's default. Configurable per thread in the future. +const OUTPUT_TRUNCATE_LEN: usize = 8_000; + +/// Maximum characters for a preview prefix in compact metadata. +const OUTPUT_PREVIEW_LEN: usize = 200; + +/// Default resource limits for Monty execution. +fn default_limits() -> ResourceLimits { + ResourceLimits::new() + .max_duration(Duration::from_secs(30)) + .max_allocations(1_000_000) + .max_memory(64 * 1024 * 1024) // 64 MB +} + +// ── Result types ──────────────────────────────────────────── + +/// Result of executing a code block. +pub struct CodeExecutionResult { + /// The Python return value, converted to JSON. + pub return_value: serde_json::Value, + /// Captured print output. + pub stdout: String, + /// All action calls that were made during execution. + pub action_results: Vec, + /// Events generated during execution. + pub events: Vec, + /// If set, execution was interrupted for approval. + pub need_approval: Option, + /// Tokens used by recursive llm_query() calls. + pub recursive_tokens: TokenUsage, + /// If set, the code called FINAL() or FINAL_VAR() with this answer. + pub final_answer: Option, + /// Whether the code execution hit an error (traceback included in stdout). + pub had_error: bool, +} + +/// Build a compact output summary for inclusion in LLM context between steps. +/// +/// Truncates to `OUTPUT_TRUNCATE_LEN` (last N chars shown, like fast-rlm). +/// Includes a list of REPL variable names if available. +pub fn compact_output_metadata(stdout: &str, return_value: &serde_json::Value) -> String { + let mut parts = Vec::new(); + + if !stdout.is_empty() { + let char_count = stdout.chars().count(); + if char_count > OUTPUT_TRUNCATE_LEN { + let truncated: String = stdout + .chars() + .skip(char_count - OUTPUT_TRUNCATE_LEN) + .collect(); + parts.push(format!( + "[TRUNCATED: last {OUTPUT_TRUNCATE_LEN} of {char_count} chars shown]\n{truncated}", + )); + } else { + parts.push(format!("[FULL OUTPUT: {char_count} chars]\n{stdout}")); + } + } + + if *return_value != serde_json::Value::Null { + let val_str = serde_json::to_string_pretty(return_value).unwrap_or_default(); + let val_char_count = val_str.chars().count(); + if val_char_count > OUTPUT_PREVIEW_LEN { + let preview: String = val_str.chars().take(OUTPUT_PREVIEW_LEN).collect(); + parts.push(format!( + "Return value ({val_char_count} chars): {preview}...", + )); + } else { + parts.push(format!("Return value: {val_str}")); + } + } + + if parts.is_empty() { + "[code executed, no output]".into() + } else { + parts.join("\n") + } +} + +// ── Step 0 orientation preamble ───────────────────────────── + +/// Build the Step 0 orientation preamble that auto-executes before the +/// first LLM call to give the model structural awareness of the context. +pub fn build_orientation_preamble(thread: &Thread) -> String { + let msg_count = thread.messages.len(); + let total_chars: usize = thread.messages.iter().map(|m| m.content.len()).sum(); + let user_msgs = thread + .messages + .iter() + .filter(|m| m.role == MessageRole::User) + .count(); + + let mut preview = String::new(); + if let Some(last_user) = thread + .messages + .iter() + .rev() + .find(|m| m.role == MessageRole::User) + { + let content_preview: String = last_user.content.chars().take(500).collect(); + let truncated = if last_user.content.chars().count() > 500 { + "..." + } else { + "" + }; + preview = format!("\nLast user message preview: {content_preview}{truncated}"); + } + + format!( + "[Step 0 — Context Orientation]\n\ + Goal: {goal}\n\ + Context: {msg_count} messages, {total_chars} total chars, {user_msgs} from user\n\ + Step: {step}{preview}", + goal = thread.goal, + step = thread.step_count + 1, + ) +} + +// ── Context injection (RLM 3.4) ──────────────────────────── + +/// Build Monty input variables from thread state. +/// +/// `persisted_state` carries variables from previous code steps so the +/// REPL feels persistent even though each step creates a fresh MontyRun. +fn build_context_inputs( + thread: &Thread, + persisted_state: &serde_json::Value, +) -> (Vec, Vec) { + let mut names = Vec::new(); + let mut values = Vec::new(); + + // `context` — thread messages as a list of dicts + let messages: Vec = thread + .messages + .iter() + .map(|msg| { + let mut pairs = vec![ + ( + MontyObject::String("role".into()), + MontyObject::String(format!("{:?}", msg.role)), + ), + ( + MontyObject::String("content".into()), + MontyObject::String(msg.content.clone()), + ), + ]; + if let Some(ref name) = msg.action_name { + pairs.push(( + MontyObject::String("action_name".into()), + MontyObject::String(name.clone()), + )); + } + MontyObject::dict(pairs) + }) + .collect(); + names.push("context".into()); + values.push(MontyObject::List(messages)); + + // `goal` — the thread's goal string + names.push("goal".into()); + values.push(MontyObject::String(thread.goal.clone())); + + // `step_number` — current step index + names.push("step_number".into()); + values.push(MontyObject::Int(thread.step_count as i64)); + + // `state` — persisted variables from previous code steps. + // This is a dict that accumulates: return values, tool results, etc. + // The model can read `state["results"]`, `state["prev_return"]`, etc. + names.push("state".into()); + values.push(json_to_monty(persisted_state)); + + // `previous_results` — dict of {call_id: result_json} from prior steps + let result_pairs: Vec<(MontyObject, MontyObject)> = thread + .messages + .iter() + .filter(|m| m.role == MessageRole::ActionResult) + .filter_map(|m| { + let call_id = m.action_call_id.as_ref()?; + Some(( + MontyObject::String(call_id.clone()), + MontyObject::String(m.content.clone()), + )) + }) + .collect(); + names.push("previous_results".into()); + values.push(MontyObject::dict(result_pairs)); + + // `user_timezone` — validated IANA timezone from the user's channel (e.g. "America/New_York") + let tz = thread + .metadata + .get("user_timezone") + .and_then(|v| v.as_str()) + .and_then(ValidTimezone::parse) + .map(|vtz| vtz.name().to_string()) + .unwrap_or_else(|| "UTC".into()); + names.push("user_timezone".into()); + values.push(MontyObject::String(tz)); + + (names, values) +} + +// ── Main execution function ───────────────────────────────── + +/// Execute a Python code block using Monty. +/// +/// Handles the full RLM execution pattern: context-as-variables, FINAL() +/// termination, llm_query() recursive calls, error-to-LLM flow, and +/// output truncation. +#[allow(clippy::too_many_arguments)] +pub async fn execute_code( + code: &str, + thread: &Thread, + llm: &Arc, + effects: &Arc, + leases: &LeaseManager, + policy: &PolicyEngine, + context: &ThreadExecutionContext, + capability_policies: &[crate::types::capability::PolicyRule], + persisted_state: &serde_json::Value, +) -> Result { + execute_code_with_skills( + code, + thread, + llm, + effects, + leases, + policy, + context, + capability_policies, + persisted_state, + &[], + ) + .await +} + +/// Execute a Python code block with optional skill code snippets. +/// +/// `skill_snippet_names` are registered as additional known functions in the +/// Monty NameLookup, alongside tool names from capability leases. +#[allow(clippy::too_many_arguments)] +pub async fn execute_code_with_skills( + code: &str, + thread: &Thread, + llm: &Arc, + effects: &Arc, + leases: &LeaseManager, + policy: &PolicyEngine, + context: &ThreadExecutionContext, + capability_policies: &[crate::types::capability::PolicyRule], + persisted_state: &serde_json::Value, + skill_snippet_names: &[String], +) -> Result { + let mut stdout = String::new(); + let mut action_results = Vec::new(); + let mut events = Vec::new(); + let mut recursive_tokens = TokenUsage::default(); + let mut final_answer: Option = None; + let mut had_error = false; + + // Build context variables including persisted state from prior steps + let (input_names, input_values) = build_context_inputs(thread, persisted_state); + + // Collect known tool names so NameLookup can return callable stubs. + // Without this, `mission_list()` in code raises NameError because Monty + // resolves the name before calling it, and Undefined → NameError. + let active_leases = leases.active_for_thread(thread.id).await; + let mut known_actions: std::collections::HashSet = effects + .available_actions(&active_leases) + .await + .unwrap_or_default() + .into_iter() + .map(|a| a.name) + .collect(); + + // Register skill code snippet function names as additional known actions. + // These resolve in NameLookup so the LLM can call them as Python functions. + for name in skill_snippet_names { + known_actions.insert(name.clone()); + } + + // Parse and compile (wrap in catch_unwind — Monty 0.0.x can panic) + let runner = match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + MontyRun::new(code.to_string(), "step.py", input_names) + })) { + Ok(Ok(runner)) => runner, + Ok(Err(e)) => { + // Parse error flows back to LLM (not a termination) + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout: format!("SyntaxError: {e}"), + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer: None, + had_error: true, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during code parsing".into(), + }); + } + }; + + // Start execution with resource limits and context inputs + let tracker = LimitedTracker::new(default_limits()); + + let run_result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + runner.start(input_values, tracker, PrintWriter::Collect(&mut stdout)) + })); + + let mut progress = match run_result { + Ok(Ok(p)) => p, + Ok(Err(e)) => { + // Runtime error flows back to LLM + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout: format!("{stdout}\nError: {e}"), + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer: None, + had_error: true, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during execution start".into(), + }); + } + }; + + // Pending async tool executions keyed by Monty call_id. + // When a tool FunctionCall comes in, we spawn a tokio task and store + // the JoinHandle here. When ResolveFutures yields, we await them. + let mut pending_futures: HashMap = HashMap::new(); + + // Drive the execution loop + let mut call_counter = 0u32; + loop { + match progress { + RunProgress::Complete(obj) => { + return Ok(CodeExecutionResult { + return_value: monty_to_json(&obj), + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + + RunProgress::FunctionCall(call) => { + call_counter += 1; + let str_call_id = format!("code_call_{call_counter}"); + let monty_call_id = call.call_id; + let action_name = call.function_name.clone(); + let params = monty_args_to_json(&call.args, &call.kwargs); + + debug!(action = %action_name, call_id = %str_call_id, monty_id = monty_call_id, "Monty: function call"); + + // Builtins that need synchronous results — resume with value. + let sync_result = match action_name.as_str() { + "FINAL" => { + let answer = call.args.first().map(monty_to_string).unwrap_or_default(); + final_answer = Some(answer); + Some(ExtFunctionResult::Return(MontyObject::None)) + } + "FINAL_VAR" => { + let var_name = call + .args + .first() + .map(monty_to_string) + .unwrap_or_else(|| "result".into()); + final_answer = Some(format!("[FINAL_VAR: {var_name}]")); + Some(ExtFunctionResult::Return(MontyObject::None)) + } + // LLM calls are async — spawn tokio task, resume_pending. + // This allows asyncio.gather(llm_query(...), tool(...)) + // to run the LLM call and tool call concurrently. + "llm_query" => { + let args = call.args.clone(); + let kwargs = call.kwargs.clone(); + let llm = llm.clone(); + let handle = tokio::spawn(async move { + handle_llm_query_standalone(&args, &kwargs, &llm).await + }); + pending_futures.insert(monty_call_id, PendingFuture::Llm { handle }); + None // handled as async below + } + "llm_query_batched" => { + let args = call.args.clone(); + let kwargs = call.kwargs.clone(); + let llm = llm.clone(); + let handle = tokio::spawn(async move { + handle_llm_query_batched_standalone(&args, &kwargs, &llm).await + }); + pending_futures.insert(monty_call_id, PendingFuture::Llm { handle }); + None + } + // rlm_query stays synchronous — it spawns a child Monty VM + // which isn't Send, so it can't run in tokio::spawn. + "rlm_query" => Some( + handle_rlm_query( + &call.args, + &call.kwargs, + thread, + llm, + effects, + leases, + policy, + &mut recursive_tokens, + ) + .await, + ), + "globals" | "locals" => { + let entries: Vec<(MontyObject, MontyObject)> = known_actions + .iter() + .map(|name| { + (MontyObject::String(name.clone()), MontyObject::Bool(true)) + }) + .collect(); + Some(ExtFunctionResult::Return(MontyObject::Dict(entries.into()))) + } + _ => None, // tool call — handled async below + }; + + if let Some(ext_result) = sync_result { + // Sync resume for builtins + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + call.resume(ext_result, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during resume".into(), + }); + } + } + continue; + } + + // If an LLM call already inserted a pending future, just + // resume_pending and continue — no preflight needed. + if pending_futures.contains_key(&monty_call_id) { + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + call.resume_pending(PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during resume_pending".into(), + }); + } + } + continue; + } + + // ── Async tool dispatch ───────────────────────────── + // Preflight (lease + policy) is sync. If denied or + // needs approval, resume with error immediately. + // If approved, spawn tokio task and resume_pending(). + + let preflight = preflight_action( + &action_name, + ¶ms, + thread, + effects, + leases, + policy, + context, + capability_policies, + &str_call_id, + &mut events, + ) + .await; + + match preflight { + PreflightResult::Approved(lease) => { + // Spawn async execution + let effects = effects.clone(); + let name = action_name.clone(); + let params_clone = params.clone(); + let lease_clone = lease.clone(); + let mut ctx = context.clone(); + ctx.current_call_id = Some(str_call_id.clone()); + let ps = crate::types::event::summarize_params(&name, ¶ms); + + let handle = tokio::spawn(async move { + effects + .execute_action(&name, params_clone, &lease_clone, &ctx) + .await + }); + + pending_futures.insert( + monty_call_id, + PendingFuture::Tool { + handle, + action_name, + call_id: str_call_id, + lease_id: lease.id, + parameters: params.clone(), + params_summary: ps, + }, + ); + + // Resume with pending future — Python gets ExternalFuture + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + call.resume_pending(PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during resume_pending".into(), + }); + } + } + } + PreflightResult::Denied(ext_result) => { + // Resume with error — Python sees an exception + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + call.resume(ext_result, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during resume".into(), + }); + } + } + } + PreflightResult::GatePaused(outcome) => { + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: Some(outcome), + recursive_tokens, + final_answer: None, + had_error, + }); + } + } + } + + // ── ResolveFutures: parallel execution ──────────────── + // Resolves both tool calls and LLM calls that were deferred + // via resume_pending(). All pending tokio tasks are awaited + // and their results fed back to Monty. + RunProgress::ResolveFutures(resolve) => { + let pending_ids = resolve.pending_call_ids().to_vec(); + debug!(pending = ?pending_ids, "Monty: ResolveFutures — resolving {} pending futures", pending_ids.len()); + + let mut results: Vec<(u32, ExtFunctionResult)> = + Vec::with_capacity(pending_ids.len()); + + for &mid in &pending_ids { + let ext_result = if let Some(pf) = pending_futures.remove(&mid) { + match pf { + PendingFuture::Tool { + handle, + action_name, + call_id, + lease_id, + parameters, + params_summary, + } => { + resolve_tool_future( + handle, + &action_name, + &call_id, + lease_id, + parameters, + params_summary, + leases, + context, + &mut action_results, + &mut events, + ) + .await + } + PendingFuture::Llm { handle } => { + resolve_llm_future(handle, &mut recursive_tokens).await + } + } + } else { + debug!(call_id = mid, "ResolveFutures: unknown pending call_id"); + ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("unknown pending call_id {mid}")), + )) + }; + results.push((mid, ext_result)); + } + + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + resolve.resume(results, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during ResolveFutures resume".into(), + }); + } + } + } + + RunProgress::NameLookup(lookup) => { + let name = lookup.name.clone(); + + let result = if known_actions.contains(&name) { + debug!(name = %name, "Monty: resolved as tool function"); + NameLookupResult::Value(MontyObject::Function { + name: name.clone(), + docstring: None, + }) + } else if name == "globals" || name == "locals" { + NameLookupResult::Value(MontyObject::Function { + name: name.clone(), + docstring: None, + }) + } else { + debug!(name = %name, "Monty: unresolved name"); + NameLookupResult::Undefined + }; + + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + lookup.resume(result, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nNameError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during name lookup".into(), + }); + } + } + } + + RunProgress::OsCall(os_call) => { + debug!(function = ?os_call.function, "Monty: OS call denied"); + let err = ExtFunctionResult::Error(MontyException::new( + ExcType::OSError, + Some("OS operations are not permitted in CodeAct scripts".into()), + )); + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + os_call.resume(err, PrintWriter::Collect(&mut stdout)) + })) { + Ok(Ok(p)) => progress = p, + Ok(Err(e)) => { + stdout.push_str(&format!("\nOSError: {e}")); + had_error = true; + return Ok(CodeExecutionResult { + return_value: serde_json::Value::Null, + stdout, + action_results, + events, + need_approval: None, + recursive_tokens, + final_answer, + had_error, + }); + } + Err(_) => { + return Err(EngineError::Effect { + reason: "Monty VM panicked during OS call".into(), + }); + } + } + } + } + } +} + +// ── Pending future tracking ───────────────────────────────── + +/// A deferred computation spawned as a tokio task, pending resolution +/// via `ResolveFutures`. Can be a tool execution or an LLM call. +enum PendingFuture { + /// Tool action execution. + Tool { + handle: tokio::task::JoinHandle>, + action_name: String, + call_id: String, + lease_id: crate::types::capability::LeaseId, + parameters: serde_json::Value, + params_summary: Option, + }, + /// LLM call (llm_query / llm_query_batched / rlm_query). + Llm { + handle: tokio::task::JoinHandle<(ExtFunctionResult, TokenUsage)>, + }, +} + +/// Result of preflight checks (lease + policy) for a tool call. +enum PreflightResult { + /// Tool approved — lease is consumed, ready to execute. + Approved(crate::types::capability::CapabilityLease), + /// Tool denied — return this error to Monty. + Denied(ExtFunctionResult), + /// Tool is paused by a gate — interrupt the batch. + GatePaused(crate::runtime::messaging::ThreadOutcome), +} + +/// Run preflight checks for a tool call: find lease, check policy, consume use. +#[allow(clippy::too_many_arguments)] +async fn preflight_action( + action_name: &str, + params: &serde_json::Value, + thread: &Thread, + effects: &Arc, + leases: &LeaseManager, + policy: &PolicyEngine, + context: &ThreadExecutionContext, + capability_policies: &[crate::types::capability::PolicyRule], + call_id: &str, + events: &mut Vec, +) -> PreflightResult { + let lease = match leases.find_lease_for_action(thread.id, action_name).await { + Some(l) => l, + None => { + events.push(EventKind::ActionFailed { + step_id: context.step_id, + action_name: action_name.into(), + call_id: call_id.into(), + error: format!("no lease for action '{action_name}'"), + params_summary: None, + }); + return PreflightResult::Denied(ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("no lease for action '{action_name}'")), + ))); + } + }; + + let action_def = effects + .available_actions(std::slice::from_ref(&lease)) + .await + .ok() + .and_then(|actions| actions.into_iter().find(|a| a.name == action_name)); + + if let Some(ref action_def) = action_def { + match policy.evaluate(action_def, &lease, capability_policies) { + PolicyDecision::Deny { reason } => { + events.push(EventKind::ActionFailed { + step_id: context.step_id, + action_name: action_name.into(), + call_id: call_id.into(), + error: reason.clone(), + params_summary: None, + }); + return PreflightResult::Denied(ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("denied: {reason}")), + ))); + } + PolicyDecision::RequireApproval { .. } => { + events.push(EventKind::ApprovalRequested { + action_name: action_name.into(), + call_id: call_id.into(), + parameters: Some(params.clone()), + description: None, + allow_always: None, + gate_name: None, + params_summary: crate::types::event::summarize_params(action_name, params), + }); + return PreflightResult::GatePaused( + crate::runtime::messaging::ThreadOutcome::GatePaused { + gate_name: "approval".into(), + action_name: action_name.into(), + call_id: call_id.into(), + parameters: params.clone(), + resume_kind: crate::gate::ResumeKind::Approval { allow_always: true }, + resume_output: None, + }, + ); + } + PolicyDecision::Allow => {} + } + } + + if let Err(e) = leases.consume_use(lease.id).await { + return PreflightResult::Denied(ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("lease exhausted: {e}")), + ))); + } + + PreflightResult::Approved(lease) +} + +// ── llm_query() — recursive subagent (RLM 3.5) ───────────── + +/// Handle `llm_query(prompt, context)` — single recursive sub-call. +async fn handle_llm_query( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + llm: &Arc, + recursive_tokens: &mut TokenUsage, +) -> ExtFunctionResult { + let prompt = extract_string_arg(args, kwargs, "prompt", 0); + let context_arg = extract_string_arg(args, kwargs, "context", 1); + // `model` must be parsed explicitly — `extract_string_arg` coerces via + // `monty_to_string`, which turns `MontyObject::None` into the literal + // string "None" and stringifies non-string values, both of which would + // silently route the call to an invalid model ID. Accept only str or None. + let model_arg = match extract_optional_string_kwarg(args, kwargs, "model", 2) { + Ok(v) => v, + Err(err) => return err, + }; + + let prompt = match prompt { + Some(p) => p, + None => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some("llm_query() requires a 'prompt' argument".into()), + )); + } + }; + + let mut messages = Vec::new(); + if let Some(ctx) = context_arg { + messages.push(ThreadMessage::system(format!( + "You are a sub-agent. Answer concisely based on the context.\n\n{ctx}" + ))); + } else { + // Some providers (e.g. OpenAI Codex Responses API) require a system + // message / instructions field. Always include one. + messages.push(ThreadMessage::system( + "You are a helpful sub-agent. Answer concisely.", + )); + } + messages.push(ThreadMessage::user(prompt)); + + let config = LlmCallConfig { + force_text: true, + model: model_arg, + ..LlmCallConfig::default() + }; + + match llm.complete(&messages, &[], &config).await { + Ok(output) => { + recursive_tokens.input_tokens += output.usage.input_tokens; + recursive_tokens.output_tokens += output.usage.output_tokens; + let text = match output.response { + LlmResponse::Text(t) => t, + LlmResponse::ActionCalls { content, .. } | LlmResponse::Code { content, .. } => { + content.unwrap_or_default() + } + }; + ExtFunctionResult::Return(MontyObject::String(text)) + } + Err(e) => ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("llm_query failed: {e}")), + )), + } +} + +/// Handle `llm_query_batched(prompts)` — parallel recursive sub-calls. +/// +/// Takes a list of prompt strings and dispatches them concurrently. +/// Returns a list of response strings in the same order. +async fn handle_llm_query_batched( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + llm: &Arc, + recursive_tokens: &mut TokenUsage, +) -> ExtFunctionResult { + // Extract prompts list (first arg or kwarg "prompts") + let prompts_obj = args.first().or_else(|| { + kwargs.iter().find_map(|(k, v)| { + if let MontyObject::String(key) = k + && key == "prompts" + { + return Some(v); + } + None + }) + }); + + let prompts: Vec = match prompts_obj { + Some(MontyObject::List(items)) => items.iter().map(monty_to_string).collect(), + Some(other) => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some(format!( + "llm_query_batched() expects a list of prompts, got {other:?}" + )), + )); + } + None => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some("llm_query_batched() requires a 'prompts' argument".into()), + )); + } + }; + + // Positional/keyword layout (matches the documented signature + // `llm_query_batched(prompts, context=None, model=None, models=None)`): + // arg 0 = prompts (already extracted above) + // arg 1 = context + // arg 2 = model + // arg 3 = models + // All three of context/model/models can also be passed by keyword. + let context_arg = match extract_optional_string_kwarg(args, kwargs, "context", 1) { + Ok(v) => v, + Err(err) => return err, + }; + + // Optional model overrides: + // - `model="..."` applies the same model to every prompt + // - `models=[...]` is a parallel array (must match prompts length); use + // this to broadcast the same prompt across a council of models by + // passing `prompts=[same]*N, models=[m1, m2, ...]`. Within `models`, + // a `None` slot means "no override for this prompt" (the caller + // opted out of routing for that slot); the singular `model=` kwarg + // does NOT fill those slots, since mixing the two would be surprising. + // See note in handle_llm_query: `model` must be parsed explicitly so that + // `model=None` doesn't become the literal string "None". + let single_model = match extract_optional_string_kwarg(args, kwargs, "model", 2) { + Ok(v) => v, + Err(err) => return err, + }; + let models_kwarg = kwargs + .iter() + .find_map(|(k, v)| match k { + MontyObject::String(key) if key == "models" => Some(v), + _ => None, + }) + .or_else(|| args.get(3)); + + let models_list: Option>> = match models_kwarg { + None | Some(MontyObject::None) => None, + Some(MontyObject::List(items)) => { + let mut out = Vec::with_capacity(items.len()); + for item in items { + match item { + MontyObject::String(s) => out.push(Some(s.clone())), + MontyObject::None => out.push(None), + other => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some(format!( + "llm_query_batched(): models list entries must be str or None, got {other:?}" + )), + )); + } + } + } + Some(out) + } + Some(other) => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some(format!( + "llm_query_batched(): `models` must be a list of str or None, got {other:?}" + )), + )); + } + }; + + if let Some(ref ms) = models_list + && ms.len() != prompts.len() + { + return ExtFunctionResult::Error(MontyException::new( + ExcType::ValueError, + Some(format!( + "llm_query_batched(): models list length ({}) must match prompts length ({})", + ms.len(), + prompts.len() + )), + )); + } + + let mut handles = Vec::with_capacity(prompts.len()); + for (i, prompt) in prompts.iter().enumerate() { + let llm = Arc::clone(llm); + let ctx = context_arg.clone(); + let prompt = prompt.clone(); + // If `models=` was provided, each slot is authoritative — `None` means + // "no override for this prompt" and is NOT backfilled from `model=`. + // Otherwise, fall back to the singular `model=` kwarg (or None). + let model_override = match models_list.as_ref() { + Some(ms) => ms[i].clone(), + None => single_model.clone(), + }; + let config = LlmCallConfig { + force_text: true, + model: model_override, + ..LlmCallConfig::default() + }; + handles.push(tokio::spawn(async move { + let mut messages = Vec::new(); + if let Some(ctx) = ctx { + messages.push(ThreadMessage::system(format!( + "You are a sub-agent. Answer concisely.\n\n{ctx}" + ))); + } else { + messages.push(ThreadMessage::system( + "You are a helpful sub-agent. Answer concisely.", + )); + } + messages.push(ThreadMessage::user(prompt)); + llm.complete(&messages, &[], &config).await + })); + } + + // Collect results + let mut results = Vec::with_capacity(prompts.len()); + let mut total_input = 0u64; + let mut total_output = 0u64; + + for handle in handles { + match handle.await { + Ok(Ok(output)) => { + total_input += output.usage.input_tokens; + total_output += output.usage.output_tokens; + let text = match output.response { + LlmResponse::Text(t) => t, + LlmResponse::ActionCalls { content, .. } + | LlmResponse::Code { content, .. } => content.unwrap_or_default(), + }; + results.push(MontyObject::String(text)); + } + Ok(Err(e)) => { + results.push(MontyObject::String(format!("Error: {e}"))); + } + Err(e) => { + results.push(MontyObject::String(format!("Error: task failed: {e}"))); + } + } + } + + recursive_tokens.input_tokens += total_input; + recursive_tokens.output_tokens += total_output; + + ExtFunctionResult::Return(MontyObject::List(results)) +} + +// ── rlm_query() — full recursive sub-agent (RLM 3.5) ───────── + +/// Handle `rlm_query(prompt)` — spawn a child CodeAct thread with its own +/// execution loop, tools, and iteration budget. +/// +/// Unlike `llm_query()` (single-shot LLM call), `rlm_query()` creates a +/// child thread with full CodeAct capabilities. The child inherits the +/// parent's remaining budget and tool access. +#[allow(clippy::too_many_arguments)] +async fn handle_rlm_query( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + parent_thread: &Thread, + llm: &Arc, + effects: &Arc, + leases: &LeaseManager, + policy: &PolicyEngine, + recursive_tokens: &mut TokenUsage, +) -> ExtFunctionResult { + let prompt = extract_string_arg(args, kwargs, "prompt", 0); + let prompt = match prompt { + Some(p) => p, + None => { + return ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some("rlm_query() requires a 'prompt' argument".into()), + )); + } + }; + + // Depth check — refuse if at max recursion depth + let current_depth = parent_thread.config.depth; + let max_depth = parent_thread.config.max_depth; + if current_depth >= max_depth { + return ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!( + "rlm_query() depth limit reached: depth {current_depth} >= max {max_depth}" + )), + )); + } + + // Build child thread with inherited budget + let child_config = crate::types::thread::ThreadConfig { + max_iterations: parent_thread.config.max_iterations.min(20), // cap child iterations + enable_tool_intent_nudge: false, + max_tokens_total: parent_thread + .config + .max_tokens_total + .map(|max| max.saturating_sub(parent_thread.total_tokens_used)), + max_budget_usd: parent_thread + .config + .max_budget_usd + .map(|max| (max - parent_thread.total_cost_usd).max(0.0)), + max_duration: parent_thread.config.max_duration, + depth: current_depth + 1, + max_depth, + ..crate::types::thread::ThreadConfig::default() + }; + + let mut child_thread = crate::types::thread::Thread::new( + &prompt, + crate::types::thread::ThreadType::Research, + parent_thread.project_id, + &parent_thread.user_id, + child_config, + ) + .with_parent(parent_thread.id); + + // Add the prompt as a user message + child_thread.add_message(ThreadMessage::user(&prompt)); + + // Create signal channel and child's lease manager + let (_tx, rx) = crate::runtime::messaging::signal_channel(8); + let child_leases = Arc::new(LeaseManager::new()); + + // Grant the child the same leases as the parent (in the child's manager) + let parent_leases = leases.active_for_thread(parent_thread.id).await; + let now = chrono::Utc::now(); + for parent_lease in &parent_leases { + // Convert parent's expires_at to remaining duration + let remaining_duration = parent_lease + .expires_at + .and_then(|exp| (exp - now).to_std().ok()) + .map(|d| chrono::Duration::from_std(d).unwrap_or(chrono::Duration::hours(1))); + let lease = match child_leases + .grant( + child_thread.id, + &parent_lease.capability_name, + parent_lease.granted_actions.clone(), + remaining_duration, + parent_lease.max_uses, + ) + .await + { + Ok(l) => l, + Err(e) => { + debug!(error = %e, "rlm_query: skipping invalid lease for child thread"); + continue; + } + }; + child_thread.capability_leases.push(lease.id); + } + let mut child_policy_engine = PolicyEngine::new(); + // Copy denied effects from parent policy + for effect in &policy.denied_effects { + child_policy_engine.deny_effect(*effect); + } + let child_policy = Arc::new(child_policy_engine); + + let mut child_loop = crate::executor::ExecutionLoop::new( + child_thread, + Arc::clone(llm), + Arc::clone(effects), + child_leases, + child_policy, + rx, + "rlm_child".to_string(), + ); + + debug!( + parent_thread = %parent_thread.id, + depth = current_depth + 1, + prompt_len = prompt.len(), + "rlm_query: spawning child CodeAct thread" + ); + + // Run the child loop (Box::pin to avoid infinite future size from recursion) + match Box::pin(child_loop.run()).await { + Ok(outcome) => { + // Track child's token usage + recursive_tokens.input_tokens += child_loop.thread.total_tokens_used; + recursive_tokens.cost_usd += child_loop.thread.total_cost_usd; + + let response = match outcome { + crate::runtime::messaging::ThreadOutcome::Completed { response } => { + response.unwrap_or_default() + } + crate::runtime::messaging::ThreadOutcome::Failed { error } => { + format!("rlm_query child failed: {error}") + } + crate::runtime::messaging::ThreadOutcome::MaxIterations => { + "rlm_query child reached max iterations".to_string() + } + _ => String::new(), + }; + + ExtFunctionResult::Return(MontyObject::String(response)) + } + Err(e) => ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("rlm_query failed: {e}")), + )), + } +} + +// ── Standalone async handlers (for tokio::spawn) ──────────── + +/// `llm_query()` — standalone version that returns `(ExtFunctionResult, TokenUsage)`. +async fn handle_llm_query_standalone( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + llm: &Arc, +) -> (ExtFunctionResult, TokenUsage) { + let mut tokens = TokenUsage::default(); + let result = handle_llm_query(args, kwargs, llm, &mut tokens).await; + (result, tokens) +} + +/// `llm_query_batched()` — standalone version. +async fn handle_llm_query_batched_standalone( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + llm: &Arc, +) -> (ExtFunctionResult, TokenUsage) { + let mut tokens = TokenUsage::default(); + let result = handle_llm_query_batched(args, kwargs, llm, &mut tokens).await; + (result, tokens) +} + +// ── Future resolution helpers ─────────────────────────────── + +/// Resolve a pending tool execution future. +#[allow(clippy::too_many_arguments)] +async fn resolve_tool_future( + handle: tokio::task::JoinHandle>, + action_name: &str, + call_id: &str, + lease_id: crate::types::capability::LeaseId, + parameters: serde_json::Value, + params_summary: Option, + leases: &LeaseManager, + context: &ThreadExecutionContext, + action_results: &mut Vec, + events: &mut Vec, +) -> ExtFunctionResult { + match handle.await { + Ok(Ok(result)) => { + // If the effect adapter wrapped a tool error as an Ok(ActionResult) + // with is_error=true (current convention in + // `EffectBridgeAdapter::execute_action_internal`), surface it as + // ActionFailed so traces, observers, and approval flows see the + // failure correctly. Without this, every wrapped error looked like + // a successful tool call to downstream consumers. + if result.is_error { + let error_msg = result + .output + .get("error") + .and_then(|v| v.as_str()) + .map(String::from) + .unwrap_or_else(|| result.output.to_string()); + events.push(EventKind::ActionFailed { + step_id: context.step_id, + action_name: action_name.into(), + call_id: call_id.into(), + error: error_msg, + params_summary, + }); + } else { + events.push(EventKind::ActionExecuted { + step_id: context.step_id, + action_name: action_name.into(), + call_id: call_id.into(), + duration_ms: result.duration.as_millis() as u64, + params_summary, + }); + } + let monty_val = json_to_monty(&result.output); + action_results.push(result); + ExtFunctionResult::Return(monty_val) + } + Ok(Err(EngineError::GatePaused { + gate_name, + action_name, + call_id, + resume_kind, + .. + })) => { + let _ = leases.refund_use(lease_id).await; + events.push(EventKind::ApprovalRequested { + action_name, + call_id, + parameters: Some(parameters), + description: None, + allow_always: match *resume_kind { + crate::gate::ResumeKind::Approval { allow_always } => Some(allow_always), + _ => None, + }, + gate_name: Some(gate_name.clone()), + params_summary, + }); + ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("execution paused by gate '{gate_name}'")), + )) + } + Ok(Err(e)) => { + events.push(EventKind::ActionFailed { + step_id: context.step_id, + action_name: action_name.into(), + call_id: call_id.into(), + error: e.to_string(), + params_summary, + }); + action_results.push(ActionResult { + call_id: call_id.into(), + action_name: action_name.into(), + output: serde_json::json!({"error": e.to_string()}), + is_error: true, + duration: Duration::ZERO, + }); + ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(e.to_string()), + )) + } + Err(e) => { + debug!("async tool task panicked: {e}"); + ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("tool execution panicked: {e}")), + )) + } + } +} + +/// Resolve a pending LLM call future, accumulating token usage. +async fn resolve_llm_future( + handle: tokio::task::JoinHandle<(ExtFunctionResult, TokenUsage)>, + recursive_tokens: &mut TokenUsage, +) -> ExtFunctionResult { + match handle.await { + Ok((result, tokens)) => { + recursive_tokens.input_tokens += tokens.input_tokens; + recursive_tokens.output_tokens += tokens.output_tokens; + result + } + Err(e) => { + debug!("async LLM task panicked: {e}"); + ExtFunctionResult::Error(MontyException::new( + ExcType::RuntimeError, + Some(format!("LLM call panicked: {e}")), + )) + } + } +} + +// ── Helpers ───────────────────────────────────────────────── + +fn extract_string_arg( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + name: &str, + position: usize, +) -> Option { + for (k, v) in kwargs { + if let MontyObject::String(key) = k + && key == name + { + return Some(monty_to_string(v)); + } + } + args.get(position).map(monty_to_string) +} + +/// Strict optional-string extractor for arguments where silent coercion is +/// dangerous (e.g. `model=` — passing the wrong type should NOT become an +/// unintended model ID). Returns: +/// - `Ok(None)` when the argument is missing or explicitly `None` +/// - `Ok(Some(s))` when the argument is a string +/// - `Err(TypeError)` for any other type +fn extract_optional_string_kwarg( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], + name: &str, + position: usize, +) -> Result, ExtFunctionResult> { + let raw = kwargs + .iter() + .find_map(|(k, v)| match k { + MontyObject::String(key) if key == name => Some(v), + _ => None, + }) + .or_else(|| args.get(position)); + + match raw { + None | Some(MontyObject::None) => Ok(None), + Some(MontyObject::String(s)) => Ok(Some(s.clone())), + Some(other) => Err(ExtFunctionResult::Error(MontyException::new( + ExcType::TypeError, + Some(format!("`{name}` must be a string or None, got {other:?}")), + ))), + } +} + +pub(crate) fn monty_to_string(obj: &MontyObject) -> String { + match obj { + MontyObject::String(s) => s.clone(), + MontyObject::None => "None".into(), + MontyObject::Bool(b) => b.to_string(), + MontyObject::Int(i) => i.to_string(), + MontyObject::Float(f) => f.to_string(), + other => { + serde_json::to_string(&monty_to_json(other)).unwrap_or_else(|_| format!("{other:?}")) + } + } +} + +// Dispatch logic moved to orchestrator.rs (__execute_action__ handler). +// GatePaused is handled via EngineError → JSON in orchestrator.rs. +// ── MontyObject ↔ JSON ────────────────────────────────────── + +pub(crate) fn monty_to_json(obj: &MontyObject) -> serde_json::Value { + match obj { + MontyObject::None => serde_json::Value::Null, + MontyObject::Bool(b) => serde_json::Value::Bool(*b), + MontyObject::Int(i) => serde_json::json!(i), + MontyObject::BigInt(i) => serde_json::Value::String(i.to_string()), + MontyObject::Float(f) => serde_json::json!(f), + MontyObject::String(s) => serde_json::Value::String(s.clone()), + MontyObject::List(items) | MontyObject::Tuple(items) => { + serde_json::Value::Array(items.iter().map(monty_to_json).collect()) + } + MontyObject::Dict(pairs) => { + let map: serde_json::Map = pairs + .into_iter() + .map(|(k, v)| { + let key = match k { + MontyObject::String(s) => s.clone(), + other => format!("{other:?}"), + }; + (key, monty_to_json(v)) + }) + .collect(); + serde_json::Value::Object(map) + } + MontyObject::Set(items) | MontyObject::FrozenSet(items) => { + serde_json::Value::Array(items.iter().map(monty_to_json).collect()) + } + MontyObject::Bytes(b) => { + serde_json::Value::String(b.iter().map(|byte| format!("{byte:02x}")).collect()) + } + other => serde_json::Value::String(format!("{other:?}")), + } +} + +pub(crate) fn json_to_monty(val: &serde_json::Value) -> MontyObject { + match val { + serde_json::Value::Null => MontyObject::None, + serde_json::Value::Bool(b) => MontyObject::Bool(*b), + serde_json::Value::Number(n) => { + if let Some(i) = n.as_i64() { + MontyObject::Int(i) + } else if let Some(f) = n.as_f64() { + MontyObject::Float(f) + } else { + MontyObject::String(n.to_string()) + } + } + serde_json::Value::String(s) => MontyObject::String(s.clone()), + serde_json::Value::Array(arr) => MontyObject::List(arr.iter().map(json_to_monty).collect()), + serde_json::Value::Object(map) => MontyObject::dict( + map.iter() + .map(|(k, v)| (MontyObject::String(k.clone()), json_to_monty(v))) + .collect::>(), + ), + } +} + +fn monty_args_to_json( + args: &[MontyObject], + kwargs: &[(MontyObject, MontyObject)], +) -> serde_json::Value { + let mut map = serde_json::Map::new(); + if !args.is_empty() { + map.insert( + "_args".into(), + serde_json::Value::Array(args.iter().map(monty_to_json).collect()), + ); + } + for (k, v) in kwargs { + let key = match k { + MontyObject::String(s) => s.clone(), + other => format!("{other:?}"), + }; + map.insert(key, monty_to_json(v)); + } + serde_json::Value::Object(map) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::capability::lease::LeaseManager; + use crate::capability::policy::PolicyEngine; + use crate::traits::effect::ThreadExecutionContext; + use crate::types::capability::{ActionDef, CapabilityLease, EffectType, GrantedActions}; + use crate::types::project::ProjectId; + use crate::types::step::{ActionResult, StepId}; + use crate::types::thread::{Thread, ThreadConfig, ThreadType}; + use std::sync::Mutex; + + /// Truncate a string to at most `max_bytes`, snapping to a UTF-8 char + /// boundary so assertion messages never panic on multibyte output. + fn truncate_for_assert(s: &str, max_bytes: usize) -> &str { + if s.len() <= max_bytes { + return s; + } + let mut end = max_bytes; + while end > 0 && !s.is_char_boundary(end) { + end -= 1; + } + &s[..end] // safety: end is walked down to a valid char boundary above + } + + struct MockEffects { + results: Mutex>>, + actions: Vec, + } + + impl MockEffects { + fn new(actions: Vec, results: Vec>) -> Self { + Self { + results: Mutex::new(results), + actions, + } + } + } + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + name: &str, + _params: serde_json::Value, + _lease: &CapabilityLease, + _ctx: &ThreadExecutionContext, + ) -> Result { + let mut results = self.results.lock().unwrap(); + if results.is_empty() { + Ok(ActionResult { + call_id: String::new(), + action_name: name.into(), + output: serde_json::json!({"result": "ok"}), + is_error: false, + duration: Duration::from_millis(1), + }) + } else { + results.remove(0) + } + } + + async fn available_actions( + &self, + _leases: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(self.actions.clone()) + } + } + + fn test_action(name: &str) -> ActionDef { + ActionDef { + name: name.into(), + description: "Test tool".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + } + } + + fn make_test_thread() -> Thread { + Thread::new( + "test goal", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ) + } + + fn make_exec_context(thread: &Thread) -> ThreadExecutionContext { + ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: "test".into(), + step_id: StepId::new(), + current_call_id: None, + source_channel: None, + user_timezone: None, + } + } + + /// Stub LLM that always returns text "stub". Only used so execute_code + /// doesn't need a real LLM — our tests exercise tool dispatch, not LLM calls. + struct StubLlm; + + #[async_trait::async_trait] + impl crate::traits::llm::LlmBackend for StubLlm { + fn model_name(&self) -> &str { + "stub" + } + + async fn complete( + &self, + _messages: &[crate::types::message::ThreadMessage], + _actions: &[ActionDef], + _config: &crate::traits::llm::LlmCallConfig, + ) -> Result { + Ok(crate::traits::llm::LlmOutput { + response: crate::types::step::LlmResponse::Text("stub".into()), + usage: crate::types::step::TokenUsage::default(), + }) + } + } + + async fn run_code( + code: &str, + effects: Arc, + thread: &Thread, + ) -> Result { + let leases = LeaseManager::new(); + let policy = PolicyEngine::new(); + let ctx = make_exec_context(thread); + + // Grant a wildcard lease + leases + .grant(thread.id, "tools", GrantedActions::All, None, None) + .await + .unwrap(); + + execute_code( + code, + thread, + &(Arc::new(StubLlm) as Arc), + &effects, + &leases, + &policy, + &ctx, + &[], + &serde_json::json!({}), + ) + .await + } + + // ── Single await tool call ────────────────────────────── + + #[tokio::test] + async fn single_await_tool_call() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("echo")], + vec![Ok(ActionResult { + call_id: String::new(), + action_name: "echo".into(), + output: serde_json::json!("hello world"), + is_error: false, + duration: Duration::from_millis(1), + })], + )); + + let code = r#" +result = await echo(message="hello") +FINAL(str(result)) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!( + result.final_answer.is_some(), + "should have final answer, stdout: {}", + result.stdout + ); + assert!( + !result.had_error, + "should not error, stdout: {}", + result.stdout + ); + assert_eq!(result.action_results.len(), 1); + } + + // ── asyncio.gather parallel execution ─────────────────── + + #[tokio::test] + async fn asyncio_gather_two_tools() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("tool_a"), test_action("tool_b")], + vec![ + Ok(ActionResult { + call_id: String::new(), + action_name: "tool_a".into(), + output: serde_json::json!(10), + is_error: false, + duration: Duration::from_millis(1), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "tool_b".into(), + output: serde_json::json!(32), + is_error: false, + duration: Duration::from_millis(1), + }), + ], + )); + + let code = r#" +import asyncio +a, b = await asyncio.gather(tool_a(), tool_b()) +FINAL(str(a + b)) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!( + result.final_answer.is_some(), + "should have final answer, stdout: {}", + result.stdout + ); + assert_eq!( + result.final_answer.as_deref(), + Some("42"), + "10 + 32 = 42, got: {:?}, stdout: {}", + result.final_answer, + result.stdout + ); + assert_eq!(result.action_results.len(), 2); + assert!(!result.had_error); + } + + // ── asyncio.gather three tools ────────────────────────── + + #[tokio::test] + async fn asyncio_gather_three_tools() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![ + test_action("web_search"), + test_action("http"), + test_action("memory_search"), + ], + vec![ + Ok(ActionResult { + call_id: String::new(), + action_name: "web_search".into(), + output: serde_json::json!("search results"), + is_error: false, + duration: Duration::from_millis(50), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "http".into(), + output: serde_json::json!("page content"), + is_error: false, + duration: Duration::from_millis(100), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "memory_search".into(), + output: serde_json::json!("memories"), + is_error: false, + duration: Duration::from_millis(25), + }), + ], + )); + + let code = r#" +import asyncio +s, h, m = await asyncio.gather( + web_search(query="test"), + http(url="https://example.com"), + memory_search(query="prior"), +) +FINAL(str(s) + "|" + str(h) + "|" + str(m)) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(!result.had_error, "stdout: {}", result.stdout); + assert_eq!(result.action_results.len(), 3); + let answer = result.final_answer.unwrap(); + assert!(answer.contains("search results"), "got: {answer}"); + assert!(answer.contains("page content"), "got: {answer}"); + assert!(answer.contains("memories"), "got: {answer}"); + } + + // ── Data-dependent chain (sequential await) ───────────── + + #[tokio::test] + async fn sequential_dependent_calls() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("step1"), test_action("step2")], + vec![ + Ok(ActionResult { + call_id: String::new(), + action_name: "step1".into(), + output: serde_json::json!("intermediate"), + is_error: false, + duration: Duration::from_millis(1), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "step2".into(), + output: serde_json::json!("final"), + is_error: false, + duration: Duration::from_millis(1), + }), + ], + )); + + let code = r#" +a = await step1() +b = await step2(input=a) +FINAL(str(b)) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(!result.had_error, "stdout: {}", result.stdout); + assert_eq!(result.action_results.len(), 2); + assert_eq!(result.final_answer.as_deref(), Some("final")); + } + + // ── Error in one gathered tool ────────────────────────── + + #[tokio::test] + async fn gather_with_error_propagates() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("good"), test_action("bad")], + vec![ + Ok(ActionResult { + call_id: String::new(), + action_name: "good".into(), + output: serde_json::json!("ok"), + is_error: false, + duration: Duration::from_millis(1), + }), + Err(EngineError::Effect { + reason: "tool exploded".into(), + }), + ], + )); + + let code = r#" +import asyncio +a, b = await asyncio.gather(good(), bad()) +FINAL("should not reach") +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + // Error in gather propagates as exception — code should error + assert!( + result.had_error, + "should have error, stdout: {}", + result.stdout + ); + assert!( + result.final_answer.is_none() + || result.final_answer.as_deref() != Some("should not reach") + ); + } + + // ── Tool with no lease (denied in preflight) ──────────── + + #[tokio::test] + async fn denied_tool_raises_exception() { + let thread = make_test_thread(); + // No actions registered — tool has no lease + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + + let code = r#" +try: + result = await unknown_tool() + FINAL("should not reach") +except: + FINAL("caught error") +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + // Tool not found raises NameError before we even get to dispatch + assert!(result.final_answer.is_some(), "stdout: {}", result.stdout); + } + + // ── FINAL works without await ─────────────────────────── + + #[tokio::test] + async fn final_is_sync() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + + let code = r#" +FINAL("hello from sync") +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert_eq!(result.final_answer.as_deref(), Some("hello from sync")); + assert!(!result.had_error); + } + + // ── globals() still works ─────────────────────────────── + + #[tokio::test] + async fn globals_returns_known_tools() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("web_search"), test_action("http")], + vec![], + )); + + let code = r#" +g = globals() +has_search = "web_search" in g +has_http = "http" in g +FINAL(str(has_search) + "|" + str(has_http)) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(!result.had_error, "stdout: {}", result.stdout); + assert_eq!(result.final_answer.as_deref(), Some("True|True")); + } + + // ── Empty gather ──────────────────────────────────────── + + #[tokio::test] + async fn empty_gather() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + + let code = r#" +import asyncio +results = await asyncio.gather() +FINAL(str(len(results))) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(!result.had_error, "stdout: {}", result.stdout); + assert_eq!(result.final_answer.as_deref(), Some("0")); + } + + // ── Single-item gather ────────────────────────────────── + + #[tokio::test] + async fn single_item_gather() { + let thread = make_test_thread(); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("echo")], + vec![Ok(ActionResult { + call_id: String::new(), + action_name: "echo".into(), + output: serde_json::json!("gathered"), + is_error: false, + duration: Duration::from_millis(1), + })], + )); + + let code = r#" +import asyncio +results = await asyncio.gather(echo()) +FINAL(str(results[0])) +"#; + + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(!result.had_error, "stdout: {}", result.stdout); + assert_eq!(result.final_answer.as_deref(), Some("gathered")); + assert_eq!(result.action_results.len(), 1); + } + + // ── Sandbox security negative tests ──────────────────────── + + /// OS-level operations must be denied or restricted by the Monty VM. + #[tokio::test] + async fn sandbox_denies_os_operations() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + // Try to import os and call os.system — should fail + let code = r#" +try: + import os + os.system("echo pwned") + FINAL("ESCAPED: os.system ran") +except Exception as e: + FINAL("blocked: " + type(e).__name__) +"#; + let result = run_code(code, effects, &thread).await.unwrap(); + let answer = result.final_answer.as_deref().unwrap_or(""); + assert!( + !answer.starts_with("ESCAPED"), + "os.system should be blocked, got: {answer}", + ); + } + + /// Resource limits must be enforced — infinite loops should be terminated. + #[tokio::test] + async fn sandbox_enforces_resource_limits() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + // Infinite allocation loop — should hit allocation or memory limit + let code = r#" +data = [] +while True: + data.append("x" * 10000) +"#; + let result = run_code(code, effects, &thread).await; + // Either returns an error or the stdout contains an error message — + // the key assertion is that it DOES NOT run forever. + if let Ok(r) = result { + assert!( + r.had_error || r.stdout.contains("Error") || r.stdout.contains("limit"), + "resource limit should terminate infinite loop, got stdout: {}", + truncate_for_assert(&r.stdout, 500), + ); + } + // Err(_) is also acceptable — means the VM was killed by resource limits + } + + /// Python `import` of system modules must be restricted. + #[tokio::test] + async fn sandbox_restricts_imports() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + // Try to import subprocess — should fail + let code = r#" +try: + import subprocess + result = subprocess.run(["echo", "escaped"], capture_output=True, text=True) + FINAL("ESCAPED: " + result.stdout) +except Exception as e: + FINAL("blocked: " + type(e).__name__) +"#; + let result = run_code(code, effects, &thread).await.unwrap(); + let answer = result.final_answer.as_deref().unwrap_or(""); + assert!( + !answer.starts_with("ESCAPED"), + "subprocess import should be blocked, got: {answer}", + ); + } + + /// File system access via open() must be blocked. + #[tokio::test] + async fn sandbox_denies_file_access() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + let code = r#" +try: + f = open("/etc/passwd", "r") + content = f.read() + f.close() + FINAL("ESCAPED: " + content[:50]) +except Exception as e: + FINAL("blocked: " + type(e).__name__) +"#; + let result = run_code(code, effects, &thread).await.unwrap(); + let answer = result.final_answer.as_deref().unwrap_or(""); + assert!( + !answer.starts_with("ESCAPED"), + "open() should be blocked, got: {answer}", + ); + } + + /// Network access via socket must be blocked. + #[tokio::test] + async fn sandbox_denies_socket_access() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + let code = r#" +try: + import socket + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(("127.0.0.1", 80)) + FINAL("ESCAPED: connected") +except Exception as e: + FINAL("blocked: " + type(e).__name__) +"#; + let result = run_code(code, effects, &thread).await.unwrap(); + let answer = result.final_answer.as_deref().unwrap_or(""); + assert!( + !answer.starts_with("ESCAPED"), + "socket access should be blocked, got: {answer}", + ); + } + + /// Calls to tools not covered by the lease must be denied. + #[tokio::test] + async fn sandbox_unlicensed_tool_denied() { + let effects: Arc = + Arc::new(MockEffects::new(vec![test_action("allowed_tool")], vec![])); + let thread = make_test_thread(); + let leases = LeaseManager::new(); + let policy = PolicyEngine::new(); + let ctx = make_exec_context(&thread); + + // Grant a restricted lease — only "allowed_tool" is permitted. + leases + .grant( + thread.id, + "tools", + GrantedActions::Specific(vec!["allowed_tool".into()]), + None, + None, + ) + .await + .unwrap(); + + let code = r#" +try: + result = await secret_admin_tool(data="pwn") + FINAL("ESCAPED: " + str(result)) +except Exception as e: + FINAL("blocked: " + type(e).__name__) +"#; + let result = execute_code( + code, + &thread, + &(Arc::new(StubLlm) as Arc), + &effects, + &leases, + &policy, + &ctx, + &[], + &serde_json::json!({}), + ) + .await + .unwrap(); + let answer = result.final_answer.as_deref().unwrap_or(""); + assert!( + !answer.starts_with("ESCAPED"), + "unlicensed tool should be denied by preflight, got: {answer}", + ); + } + + /// CPU-bound infinite loops must be terminated by allocation/duration limits. + #[tokio::test] + async fn sandbox_enforces_cpu_limits() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + // Tight CPU-bound loop (no allocations to trip allocation limit) + let code = r#" +x = 0 +while True: + x += 1 +"#; + let result = run_code(code, effects, &thread).await; + // Must terminate — either via error or resource limit + if let Ok(r) = result { + assert!( + r.had_error || r.stdout.contains("Error") || r.stdout.contains("limit"), + "cpu-bound loop should be terminated, stdout: {}", + truncate_for_assert(&r.stdout, 500), + ); + } + // Err(_) is also acceptable — means the VM was killed by resource limits + } + + /// FINAL() must capture the answer from the code. + #[tokio::test] + async fn sandbox_final_captures_answer() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + let code = r#" +x = 2 + 3 +FINAL(str(x)) +"#; + let result = run_code(code, effects, &thread).await.unwrap(); + assert_eq!( + result.final_answer.as_deref(), + Some("5"), + "FINAL should capture the computed answer" + ); + } + + /// Syntax errors flow back as errors, not panics. + #[tokio::test] + async fn sandbox_handles_syntax_error() { + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let thread = make_test_thread(); + + let code = "def broken(\nFINAL('nope')"; + let result = run_code(code, effects, &thread).await.unwrap(); + assert!(result.had_error, "syntax error should set had_error"); + assert!( + result.stdout.contains("SyntaxError") || result.stdout.contains("Error"), + "should contain SyntaxError, got: {}", + result.stdout, + ); + } + + // ── llm_query model parameter plumbing ───────────────────── + + /// LLM backend that records every call's model + prompt for assertions. + struct CapturingLlm { + calls: tokio::sync::Mutex, String)>>, + } + + impl CapturingLlm { + fn new() -> Self { + Self { + calls: tokio::sync::Mutex::new(Vec::new()), + } + } + } + + #[async_trait::async_trait] + impl crate::traits::llm::LlmBackend for CapturingLlm { + fn model_name(&self) -> &str { + "capturing" + } + + async fn complete( + &self, + messages: &[crate::types::message::ThreadMessage], + _actions: &[ActionDef], + config: &crate::traits::llm::LlmCallConfig, + ) -> Result { + let user_prompt = messages + .iter() + .rev() + .find(|m| matches!(m.role, crate::types::message::MessageRole::User)) + .map(|m| m.content.clone()) + .unwrap_or_default(); + self.calls + .lock() + .await + .push((config.model.clone(), user_prompt.clone())); + Ok(crate::traits::llm::LlmOutput { + response: crate::types::step::LlmResponse::Text(format!( + "ack:{}:{user_prompt}", + config.model.as_deref().unwrap_or("default") + )), + usage: crate::types::step::TokenUsage::default(), + }) + } + } + + #[tokio::test] + async fn llm_query_forwards_model_kwarg() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let result = handle_llm_query( + &[], + &[ + ( + MontyObject::String("prompt".into()), + MontyObject::String("what is 2+2?".into()), + ), + ( + MontyObject::String("model".into()), + MontyObject::String("gpt-4o".into()), + ), + ], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + match result { + ExtFunctionResult::Return(MontyObject::String(s)) => { + assert!(s.contains("gpt-4o"), "got: {s}"); + } + other => panic!("expected string return, got {other:?}"), + } + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].0.as_deref(), Some("gpt-4o")); + assert_eq!(calls[0].1, "what is 2+2?"); + } + + #[tokio::test] + async fn llm_query_without_model_passes_none() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let _ = handle_llm_query( + &[MontyObject::String("hello".into())], + &[], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].0, None); + } + + #[tokio::test] + async fn llm_query_batched_broadcasts_with_models_list() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("Q".into()), + MontyObject::String("Q".into()), + MontyObject::String("Q".into()), + ]); + let models = MontyObject::List(vec![ + MontyObject::String("gpt-4o".into()), + MontyObject::String("claude-sonnet-4-20250514".into()), + MontyObject::String("llama-3.1-70b-instruct".into()), + ]); + let result = handle_llm_query_batched( + &[prompts], + &[(MontyObject::String("models".into()), models)], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + match result { + ExtFunctionResult::Return(MontyObject::List(items)) => { + assert_eq!(items.len(), 3); + } + other => panic!("expected list return, got {other:?}"), + } + + let mut calls = llm.calls.lock().await; + calls.sort_by(|a, b| a.0.cmp(&b.0)); + assert_eq!(calls.len(), 3); + assert_eq!(calls[0].0.as_deref(), Some("claude-sonnet-4-20250514")); + assert_eq!(calls[1].0.as_deref(), Some("gpt-4o")); + assert_eq!(calls[2].0.as_deref(), Some("llama-3.1-70b-instruct")); + } + + #[tokio::test] + async fn llm_query_batched_single_model_applies_to_all() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]); + let _ = handle_llm_query_batched( + &[prompts], + &[( + MontyObject::String("model".into()), + MontyObject::String("gpt-4o".into()), + )], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 2); + assert!(calls.iter().all(|(m, _)| m.as_deref() == Some("gpt-4o"))); + } + + #[tokio::test] + async fn llm_query_model_none_kwarg_is_no_override_not_literal_none_string() { + // Regression: `extract_string_arg` would have coerced + // MontyObject::None to the literal string "None", silently routing + // every model=None call to an invalid model ID. Must stay None. + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let _ = handle_llm_query( + &[], + &[ + ( + MontyObject::String("prompt".into()), + MontyObject::String("hi".into()), + ), + (MontyObject::String("model".into()), MontyObject::None), + ], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].0, None); + } + + #[tokio::test] + async fn llm_query_rejects_non_string_model_kwarg() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let result = handle_llm_query( + &[], + &[ + ( + MontyObject::String("prompt".into()), + MontyObject::String("hi".into()), + ), + (MontyObject::String("model".into()), MontyObject::Int(42)), + ], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Error(_))); + assert!(llm.calls.lock().await.is_empty()); + } + + #[tokio::test] + async fn llm_query_batched_single_model_none_kwarg_is_no_override() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]); + let _ = handle_llm_query_batched( + &[prompts], + &[(MontyObject::String("model".into()), MontyObject::None)], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 2); + assert!(calls.iter().all(|(m, _)| m.is_none())); + } + + #[tokio::test] + async fn llm_query_batched_honors_positional_context_and_model() { + // Regression: `context`, `model`, and `models` used to be kwarg-only. + // A positional call matching the documented signature + // `llm_query_batched(prompts, context=None, model=None, models=None)` + // silently dropped the model, violating the preamble. + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let result = handle_llm_query_batched( + &[ + MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]), + MontyObject::String("shared context".into()), // position 1: context + MontyObject::String("gpt-4o".into()), // position 2: model + ], + &[], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + match result { + ExtFunctionResult::Return(MontyObject::List(items)) => assert_eq!(items.len(), 2), + other => panic!("expected list return, got {other:?}"), + } + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 2); + assert!(calls.iter().all(|(m, _)| m.as_deref() == Some("gpt-4o"))); + } + + #[tokio::test] + async fn llm_query_batched_honors_positional_models_list() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let result = handle_llm_query_batched( + &[ + MontyObject::List(vec![ + MontyObject::String("q".into()), + MontyObject::String("q".into()), + ]), + MontyObject::None, // position 1: context = None + MontyObject::None, // position 2: model = None + MontyObject::List(vec![ + // position 3: models + MontyObject::String("gpt-4o".into()), + MontyObject::String("claude-sonnet-4-6".into()), + ]), + ], + &[], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Return(_))); + let mut calls = llm.calls.lock().await; + calls.sort_by(|a, b| a.0.cmp(&b.0)); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0].0.as_deref(), Some("claude-sonnet-4-6")); + assert_eq!(calls[1].0.as_deref(), Some("gpt-4o")); + } + + #[tokio::test] + async fn llm_query_batched_positional_none_for_models_is_no_override() { + // `llm_query_batched(prompts, None, None, None)` should run with no + // model overrides, not error on the positional None at slot 3. + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let result = handle_llm_query_batched( + &[ + MontyObject::List(vec![MontyObject::String("a".into())]), + MontyObject::None, + MontyObject::None, + MontyObject::None, + ], + &[], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Return(_))); + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].0, None); + } + + #[tokio::test] + async fn llm_query_batched_rejects_non_string_single_model() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![MontyObject::String("a".into())]); + let result = handle_llm_query_batched( + &[prompts], + &[(MontyObject::String("model".into()), MontyObject::Int(7))], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Error(_))); + assert!(llm.calls.lock().await.is_empty()); + } + + #[tokio::test] + async fn llm_query_batched_rejects_non_string_models_entries() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]); + // Integers in the models list should fail loudly, not be coerced to "1"/"2". + let models = MontyObject::List(vec![MontyObject::Int(1), MontyObject::Int(2)]); + let result = handle_llm_query_batched( + &[prompts], + &[(MontyObject::String("models".into()), models)], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Error(_))); + assert!(llm.calls.lock().await.is_empty()); + } + + #[tokio::test] + async fn llm_query_batched_none_in_models_list_does_not_backfill_from_model_kwarg() { + // Regression: when `models=[None, "gpt-4o"]` and `model="claude-..."` + // are both passed, the None slot must NOT be backfilled by the + // singular `model=` kwarg. Each slot is authoritative. + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]); + let models = MontyObject::List(vec![ + MontyObject::None, + MontyObject::String("gpt-4o".into()), + ]); + let _ = handle_llm_query_batched( + &[prompts], + &[ + (MontyObject::String("models".into()), models), + ( + MontyObject::String("model".into()), + MontyObject::String("claude-sonnet-4-20250514".into()), + ), + ], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + let calls = llm.calls.lock().await; + assert_eq!(calls.len(), 2); + // Slot 0 was None — must remain None, not become "claude-sonnet-4-20250514". + let slot_a = calls.iter().find(|(_, p)| p == "a").expect("call for a"); + let slot_b = calls.iter().find(|(_, p)| p == "b").expect("call for b"); + assert_eq!(slot_a.0, None); + assert_eq!(slot_b.0.as_deref(), Some("gpt-4o")); + } + + #[tokio::test] + async fn llm_query_batched_models_length_mismatch_errors() { + let llm = Arc::new(CapturingLlm::new()); + let mut tokens = crate::types::step::TokenUsage::default(); + let prompts = MontyObject::List(vec![ + MontyObject::String("a".into()), + MontyObject::String("b".into()), + ]); + let models = MontyObject::List(vec![MontyObject::String("only-one".into())]); + let result = handle_llm_query_batched( + &[prompts], + &[(MontyObject::String("models".into()), models)], + &(Arc::clone(&llm) as Arc), + &mut tokens, + ) + .await; + + assert!(matches!(result, ExtFunctionResult::Error(_))); + assert!(llm.calls.lock().await.is_empty()); + } +} diff --git a/crates/ironclaw_engine/src/executor/structured.rs b/crates/ironclaw_engine/src/executor/structured.rs new file mode 100644 index 00000000000..6d197b52b4d --- /dev/null +++ b/crates/ironclaw_engine/src/executor/structured.rs @@ -0,0 +1,1035 @@ +//! Tier 0 executor: structured tool calls. +//! +//! Executes action calls by delegating to the `EffectExecutor` trait, +//! checking leases and policies for each call. +//! +//! Uses a two-phase approach: sequential preflight (lease/policy checks) +//! followed by parallel execution of all approved actions via `JoinSet`. + +use std::sync::Arc; + +use crate::capability::lease::LeaseManager; +use crate::capability::policy::{PolicyDecision, PolicyEngine}; +use crate::runtime::messaging::ThreadOutcome; +use crate::traits::effect::{EffectExecutor, ThreadExecutionContext}; +use crate::types::capability::CapabilityLease; +use crate::types::error::EngineError; +use crate::types::event::EventKind; +use crate::types::step::{ActionCall, ActionResult}; +use crate::types::thread::Thread; + +/// Result of executing a batch of action calls. +pub struct ActionBatchResult { + /// Results for each action call (in order). + pub results: Vec, + /// Events generated during execution. + pub events: Vec, + /// If set, execution was interrupted and the thread needs approval. + pub need_approval: Option, +} + +/// Outcome of preflight checking a single action call. +enum PreflightOutcome { + /// Action passed preflight — ready for parallel execution. + Runnable { + index: usize, + lease: CapabilityLease, + }, + /// Action was denied or had no lease — error result already produced. + Error { + index: usize, + result: ActionResult, + event: EventKind, + }, +} + +/// Execute a batch of action calls using the Tier 0 (structured) approach. +/// +/// Two-phase execution: +/// 1. **Preflight** (sequential): For each call, find lease and check policy. +/// Denied calls produce error results immediately. RequireApproval interrupts +/// the entire batch. +/// 2. **Execute** (parallel): All approved calls run concurrently via `JoinSet`. +/// Results are collected and merged in original call order. +pub async fn execute_action_calls( + calls: &[ActionCall], + thread: &Thread, + effects: &Arc, + leases: &LeaseManager, + policy: &PolicyEngine, + context: &ThreadExecutionContext, + capability_policies: &[crate::types::capability::PolicyRule], +) -> Result { + let mut preflight_results: Vec = Vec::with_capacity(calls.len()); + let mut early_events = Vec::new(); + let mut early_results = Vec::new(); + + // ── Phase 1: Preflight (sequential) ───────────────────────── + // Check leases and policies for every call. RequireApproval interrupts + // the entire batch immediately. Denied/no-lease calls become error results. + + for (idx, call) in calls.iter().enumerate() { + // 1. Find the lease for this action (read-only lookup for policy check) + let lease = match leases + .find_lease_for_action(thread.id, &call.action_name) + .await + { + Some(l) => l, + None => { + let error_result = ActionResult { + call_id: call.id.clone(), + action_name: call.action_name.clone(), + output: serde_json::json!({"error": format!( + "no active lease covers action '{}'", call.action_name + )}), + is_error: true, + duration: std::time::Duration::ZERO, + }; + let event = EventKind::ActionFailed { + step_id: context.step_id, + action_name: call.action_name.clone(), + call_id: call.id.clone(), + error: format!("no lease for action '{}'", call.action_name), + params_summary: None, + }; + preflight_results.push(PreflightOutcome::Error { + index: idx, + result: error_result, + event, + }); + continue; + } + }; + + // 2. Find the action definition and check policy + let action_def = effects + .available_actions(std::slice::from_ref(&lease)) + .await? + .into_iter() + .find(|a| a.name == call.action_name); + + if let Some(ref action_def) = action_def { + let decision = policy.evaluate(action_def, &lease, capability_policies); + match decision { + PolicyDecision::Deny { reason } => { + let error_result = ActionResult { + call_id: call.id.clone(), + action_name: call.action_name.clone(), + output: serde_json::json!({"error": format!("denied: {reason}")}), + is_error: true, + duration: std::time::Duration::ZERO, + }; + let event = EventKind::ActionFailed { + step_id: context.step_id, + action_name: call.action_name.clone(), + call_id: call.id.clone(), + error: reason, + params_summary: None, + }; + preflight_results.push(PreflightOutcome::Error { + index: idx, + result: error_result, + event, + }); + continue; + } + PolicyDecision::RequireApproval { .. } => { + // Collect error results from earlier preflight failures + for pf in preflight_results { + if let PreflightOutcome::Error { result, event, .. } = pf { + early_results.push(result); + early_events.push(event); + } + } + early_events.push(EventKind::ApprovalRequested { + action_name: call.action_name.clone(), + call_id: call.id.clone(), + parameters: Some(call.parameters.clone()), + description: None, + allow_always: None, + gate_name: None, + params_summary: crate::types::event::summarize_params( + &call.action_name, + &call.parameters, + ), + }); + return Ok(ActionBatchResult { + results: early_results, + events: early_events, + need_approval: Some(ThreadOutcome::GatePaused { + gate_name: "approval".into(), + action_name: call.action_name.clone(), + call_id: call.id.clone(), + parameters: call.parameters.clone(), + resume_kind: crate::gate::ResumeKind::Approval { allow_always: true }, + resume_output: None, + }), + }); + } + PolicyDecision::Allow => {} + } + } + + // 3. Atomically find + consume a lease use under a single write lock. + // This avoids the TOCTOU race where a concurrent call could exhaust + // the lease between our read-only find (step 1) and this consume. + let lease = leases + .find_and_consume(thread.id, &call.action_name) + .await?; + + preflight_results.push(PreflightOutcome::Runnable { index: idx, lease }); + } + + // ── Phase 2: Execute (parallel) ───────────────────────────── + // All approved calls run concurrently. Results are collected in a + // HashMap keyed by original index, then merged in order. + + // Separate runnable from preflight errors + let mut slot_results: Vec> = vec![None; calls.len()]; + let mut runnable_indices = Vec::new(); + + for pf in preflight_results { + match pf { + PreflightOutcome::Error { + index, + result, + event, + .. + } => { + slot_results[index] = Some((result, event)); + } + PreflightOutcome::Runnable { index, lease } => { + runnable_indices.push((index, lease)); + } + } + } + + // Short-circuit: single runnable call — execute directly without JoinSet overhead + if runnable_indices.len() == 1 { + let (idx, lease) = runnable_indices.into_iter().next().unwrap(); // safety: len()==1 checked above + let call = &calls[idx]; + let mut exec_ctx = context.clone(); + exec_ctx.current_call_id = Some(call.id.clone()); + let exec_result = effects + .execute_action( + &call.action_name, + call.parameters.clone(), + &lease, + &exec_ctx, + ) + .await; + if interrupted_call_needs_refund(&exec_result) { + let _ = leases.refund_use(lease.id).await; + } + slot_results[idx] = Some(classify_exec_result(exec_result, call, &exec_ctx)); + } else if runnable_indices.len() > 1 { + // Multiple calls: execute in parallel via JoinSet + let mut join_set = tokio::task::JoinSet::new(); + let effects = effects.clone(); + + for (idx, lease) in runnable_indices { + let call = calls[idx].clone(); + let mut ctx = context.clone(); + ctx.current_call_id = Some(call.id.clone()); + let effects = effects.clone(); + let lease = lease.clone(); + + join_set.spawn(async move { + let result = effects + .execute_action(&call.action_name, call.parameters.clone(), &lease, &ctx) + .await; + (idx, lease.id, result, call, ctx) + }); + } + + while let Some(join_result) = join_set.join_next().await { + match join_result { + Ok((idx, lease_id, result, call, ctx)) => { + if interrupted_call_needs_refund(&result) { + let _ = leases.refund_use(lease_id).await; + } + slot_results[idx] = Some(classify_exec_result(result, &call, &ctx)); + } + Err(e) => { + // Task panicked — should not happen, but handle gracefully + tracing::debug!("parallel tool execution task panicked: {e}"); + } + } + } + } + + // ── Phase 3: Merge results in original call order ─────────── + + let mut results = Vec::with_capacity(calls.len()); + let mut events = Vec::new(); + let mut first_interrupt: Option = None; + + for (idx, slot) in slot_results.into_iter().enumerate() { + if let Some((result, event)) = slot { + // Record the first gate pause as the batch interrupt but still + // collect all other results. + if first_interrupt.is_none() + && let EventKind::ApprovalRequested { + ref action_name, + ref call_id, + .. + } = event + && result.output.get("status").and_then(|v| v.as_str()) == Some("gate_paused") + { + let gate_name = result + .output + .get("gate") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_string(); + let call = &calls[idx]; + first_interrupt = Some(ThreadOutcome::GatePaused { + gate_name, + action_name: action_name.clone(), + call_id: call_id.clone(), + parameters: call.parameters.clone(), + resume_kind: serde_json::from_value( + result.output.get("resume_kind").cloned().unwrap_or_else( + || serde_json::json!({"Approval":{"allow_always":false}}), + ), + ) + .unwrap_or(crate::gate::ResumeKind::Approval { + allow_always: false, + }), + resume_output: result.output.get("resume_output").cloned(), + }); + } + results.push(result); + events.push(event); + } + } + + Ok(ActionBatchResult { + results, + events, + need_approval: first_interrupt, + }) +} + +/// Classify an execution result into an `(ActionResult, EventKind)` pair. +/// +/// Used by both the single-call fast path and the parallel JoinSet path +/// to produce uniform output. +fn classify_exec_result( + result: Result, + call: &ActionCall, + context: &ThreadExecutionContext, +) -> (ActionResult, EventKind) { + match result { + Ok(mut action_result) => { + action_result.call_id = call.id.clone(); + // Effect adapters wrap tool errors as `Ok(ActionResult { is_error: true })` + // — emit ActionFailed in that case so traces and downstream + // observers see the failure rather than treating it as success. + let event = if action_result.is_error { + let error_msg = action_result + .output + .get("error") + .and_then(|v| v.as_str()) + .map(String::from) + .unwrap_or_else(|| action_result.output.to_string()); + EventKind::ActionFailed { + step_id: context.step_id, + action_name: call.action_name.clone(), + call_id: call.id.clone(), + error: error_msg, + params_summary: None, + } + } else { + EventKind::ActionExecuted { + step_id: context.step_id, + action_name: call.action_name.clone(), + call_id: call.id.clone(), + duration_ms: action_result.duration.as_millis() as u64, + params_summary: None, + } + }; + (action_result, event) + } + Err(EngineError::GatePaused { + gate_name, + action_name, + call_id, + parameters, + resume_kind, + resume_output, + }) => { + let _error_msg = format!("gate paused: {gate_name}"); + let error_result = ActionResult { + call_id: call.id.clone(), + action_name: call.action_name.clone(), + output: serde_json::json!({ + "status": "gate_paused", + "gate": gate_name, + "resume_kind": serde_json::to_value(&*resume_kind).unwrap_or_default(), + "resume_output": resume_output.as_deref().cloned(), + }), + is_error: true, + duration: std::time::Duration::ZERO, + }; + let event = EventKind::ApprovalRequested { + action_name, + call_id, + parameters: Some((*parameters).clone()), + description: None, + allow_always: match *resume_kind { + crate::gate::ResumeKind::Approval { allow_always } => Some(allow_always), + _ => None, + }, + gate_name: Some(gate_name.clone()), + params_summary: crate::types::event::summarize_params( + &call.action_name, + ¶meters, + ), + }; + (error_result, event) + } + Err(e) => { + let error_result = ActionResult { + call_id: call.id.clone(), + action_name: call.action_name.clone(), + output: serde_json::json!({"error": e.to_string()}), + is_error: true, + duration: std::time::Duration::ZERO, + }; + let event = EventKind::ActionFailed { + step_id: context.step_id, + action_name: call.action_name.clone(), + call_id: call.id.clone(), + error: e.to_string(), + params_summary: None, + }; + (error_result, event) + } + } +} + +fn interrupted_call_needs_refund(result: &Result) -> bool { + matches!(result, Err(EngineError::GatePaused { .. })) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::traits::effect::ThreadExecutionContext; + use crate::types::capability::{ActionDef, CapabilityLease, EffectType, GrantedActions}; + use crate::types::project::ProjectId; + use crate::types::step::StepId; + use crate::types::thread::{Thread, ThreadConfig, ThreadType}; + + use std::sync::Mutex; + use std::time::Duration; + + struct MockEffects { + results: Mutex>>, + actions: Vec, + } + + impl MockEffects { + fn new(actions: Vec, results: Vec>) -> Self { + Self { + results: Mutex::new(results), + actions, + } + } + } + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _name: &str, + _params: serde_json::Value, + _lease: &CapabilityLease, + _ctx: &ThreadExecutionContext, + ) -> Result { + let mut results = self.results.lock().unwrap(); + if results.is_empty() { + Ok(ActionResult { + call_id: String::new(), // EffectExecutor doesn't set call_id + action_name: String::new(), + output: serde_json::json!({"result": "ok"}), + is_error: false, + duration: Duration::from_millis(1), + }) + } else { + results.remove(0) + } + } + + async fn available_actions( + &self, + _leases: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(self.actions.clone()) + } + } + + fn test_action(name: &str) -> ActionDef { + ActionDef { + name: name.into(), + description: "Test tool".into(), + parameters_schema: serde_json::json!({"type": "object"}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + } + } + + fn make_exec_context(thread: &Thread) -> ThreadExecutionContext { + ThreadExecutionContext { + thread_id: thread.id, + thread_type: thread.thread_type, + project_id: thread.project_id, + user_id: "test".into(), + step_id: StepId::new(), + current_call_id: None, + source_channel: None, + user_timezone: None, + } + } + + // ── call_id propagation tests ──────────────────────────── + + #[tokio::test] + async fn call_id_preserved_on_successful_execution() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("web_search")], + vec![Ok(ActionResult { + call_id: String::new(), // EffectExecutor returns empty + action_name: "web_search".into(), + output: serde_json::json!({"results": []}), + is_error: false, + duration: Duration::from_millis(42), + })], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "search", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ActionCall { + id: "call_r2o5mqBgdNUlH8KzskncUGaX".into(), + action_name: "web_search".into(), + parameters: serde_json::json!({"query": "test"}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + // call_id must be stamped from ActionCall, not the empty EffectExecutor return + assert_eq!(result.results.len(), 1); + assert_eq!(result.results[0].call_id, "call_r2o5mqBgdNUlH8KzskncUGaX"); + assert_eq!(result.results[0].action_name, "web_search"); + assert!(!result.results[0].is_error); + + // Event should carry the same call_id + let exec_event = result + .events + .iter() + .find(|e| matches!(e, EventKind::ActionExecuted { .. })); + assert!(exec_event.is_some()); + if let Some(EventKind::ActionExecuted { + call_id, + action_name, + .. + }) = exec_event + { + assert_eq!(call_id, "call_r2o5mqBgdNUlH8KzskncUGaX"); + assert_eq!(action_name, "web_search"); + } + } + + #[tokio::test] + async fn call_id_preserved_on_execution_error() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("shell")], + vec![Err(EngineError::Effect { + reason: "permission denied".into(), + })], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "exec", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ActionCall { + id: "call_abc123def".into(), + action_name: "shell".into(), + parameters: serde_json::json!({"cmd": "ls"}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + assert_eq!(result.results.len(), 1); + assert_eq!(result.results[0].call_id, "call_abc123def"); + assert!(result.results[0].is_error); + + let fail_event = result + .events + .iter() + .find(|e| matches!(e, EventKind::ActionFailed { .. })); + assert!(fail_event.is_some()); + if let Some(EventKind::ActionFailed { call_id, .. }) = fail_event { + assert_eq!(call_id, "call_abc123def"); + } + } + + #[tokio::test] + async fn call_id_preserved_when_no_lease() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new(vec![], vec![])); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + // No lease granted — action should fail with correct call_id + let calls = vec![ActionCall { + id: "call_no_lease_123".into(), + action_name: "web_search".into(), + parameters: serde_json::json!({}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + assert_eq!(result.results.len(), 1); + assert_eq!(result.results[0].call_id, "call_no_lease_123"); + assert!(result.results[0].is_error); + + if let Some(EventKind::ActionFailed { call_id, error, .. }) = result.events.first() { + assert_eq!(call_id, "call_no_lease_123"); + assert!(error.contains("no lease")); + } else { + panic!("expected ActionFailed event"); + } + } + + #[tokio::test] + async fn multiple_calls_each_get_correct_call_id() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("tool_a"), test_action("tool_b")], + vec![ + Ok(ActionResult { + call_id: String::new(), + action_name: "tool_a".into(), + output: serde_json::json!("a_result"), + is_error: false, + duration: Duration::from_millis(1), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "tool_b".into(), + output: serde_json::json!("b_result"), + is_error: false, + duration: Duration::from_millis(2), + }), + ], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "cap", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ + ActionCall { + id: "id_aaaa".into(), + action_name: "tool_a".into(), + parameters: serde_json::json!({}), + }, + ActionCall { + id: "id_bbbb".into(), + action_name: "tool_b".into(), + parameters: serde_json::json!({}), + }, + ]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + assert_eq!(result.results.len(), 2); + assert_eq!(result.results[0].call_id, "id_aaaa"); + assert_eq!(result.results[1].call_id, "id_bbbb"); + } + + // ── GatePaused(Authentication) tests ───────────────────── + + #[tokio::test] + async fn authentication_gate_interrupts_batch() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("http")], + vec![Err(EngineError::GatePaused { + gate_name: "authentication".into(), + action_name: "http".into(), + call_id: "call_auth_1".into(), + parameters: Box::new(serde_json::json!({"url": "https://api.github.com/repos"})), + resume_kind: Box::new(crate::gate::ResumeKind::Authentication { + credential_name: "github_token".into(), + instructions: "Provide your github_token token".into(), + auth_url: None, + }), + resume_output: None, + })], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "tools", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ActionCall { + id: "call_auth_1".into(), + action_name: "http".into(), + parameters: serde_json::json!({"url": "https://api.github.com/repos"}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + // Batch should be interrupted with GatePaused(Authentication) + assert!( + result.need_approval.is_some(), + "GatePaused(Authentication) should interrupt the batch" + ); + match result.need_approval.unwrap() { + ThreadOutcome::GatePaused { + gate_name, + action_name, + resume_kind, + .. + } => { + assert_eq!(gate_name, "authentication"); + assert_eq!(action_name, "http"); + match resume_kind { + crate::gate::ResumeKind::Authentication { + credential_name, .. + } => { + assert_eq!(credential_name, "github_token"); + } + other => panic!("expected auth resume kind, got {:?}", other), + } + } + other => panic!("expected GatePaused, got {:?}", other), + } + + // Gate pause event should be emitted + assert!( + result + .events + .iter() + .any(|e| matches!(e, EventKind::ApprovalRequested { gate_name: Some(name), .. } if name == "authentication")), + "should emit gate pause event" + ); + } + + #[tokio::test] + async fn authentication_gate_flags_batch_with_parallel_results() { + // Two calls: first needs auth, second succeeds. + // With parallel execution, both run concurrently — the batch is flagged + // with GatePaused(Authentication) but results from all calls are available. + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("http"), test_action("echo")], + vec![ + Err(EngineError::GatePaused { + gate_name: "authentication".into(), + action_name: "http".into(), + call_id: "call_1".into(), + parameters: Box::new(serde_json::json!({})), + resume_kind: Box::new(crate::gate::ResumeKind::Authentication { + credential_name: "api_key".into(), + instructions: "Provide your api_key token".into(), + auth_url: None, + }), + resume_output: None, + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "echo".into(), + output: serde_json::json!("second ran"), + is_error: false, + duration: Duration::from_millis(1), + }), + ], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "tools", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ + ActionCall { + id: "call_1".into(), + action_name: "http".into(), + parameters: serde_json::json!({}), + }, + ActionCall { + id: "call_2".into(), + action_name: "echo".into(), + parameters: serde_json::json!({}), + }, + ]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + // Both calls executed in parallel — results from both are available + assert_eq!(result.results.len(), 2); + // First call should be an auth error + assert!(result.results[0].is_error); + assert_eq!(result.results[0].call_id, "call_1"); + // Second call succeeded + assert_eq!(result.results[1].call_id, "call_2"); + assert!(!result.results[1].is_error); + // Batch is still flagged with GatePaused(Authentication) + assert!(result.need_approval.is_some()); + match result.need_approval.unwrap() { + ThreadOutcome::GatePaused { + gate_name, + resume_kind, + .. + } => { + assert_eq!(gate_name, "authentication"); + match resume_kind { + crate::gate::ResumeKind::Authentication { + credential_name, .. + } => { + assert_eq!(credential_name, "api_key"); + } + other => panic!("expected auth resume kind, got {:?}", other), + } + } + other => panic!("expected GatePaused, got {:?}", other), + } + } + + /// Regular EngineError::Effect (not GatePaused) should NOT interrupt — + /// it becomes a normal error result and execution continues. + #[tokio::test] + async fn regular_effect_error_does_not_interrupt() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("http"), test_action("echo")], + vec![ + Err(EngineError::Effect { + reason: "connection timeout".into(), + }), + Ok(ActionResult { + call_id: String::new(), + action_name: "echo".into(), + output: serde_json::json!("second call ran"), + is_error: false, + duration: Duration::from_millis(1), + }), + ], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "tools", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ + ActionCall { + id: "call_1".into(), + action_name: "http".into(), + parameters: serde_json::json!({}), + }, + ActionCall { + id: "call_2".into(), + action_name: "echo".into(), + parameters: serde_json::json!({}), + }, + ]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + // Both calls should have results (error does not interrupt) + assert_eq!(result.results.len(), 2); + assert!(result.results[0].is_error); + assert!(!result.results[1].is_error); + assert!( + result.need_approval.is_none(), + "no interruption for regular errors" + ); + } + + // ── call_id preservation (OpenAI/Mistral) ───────────────── + + /// Provider-specific: OpenAI rejects empty string call_id. Verify no result + /// ever has an empty call_id when the ActionCall provided one. + #[tokio::test] + async fn openai_empty_call_id_never_produced() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("echo")], + vec![Ok(ActionResult { + call_id: String::new(), // EffectExecutor always returns empty + action_name: String::new(), + output: serde_json::json!("hello"), + is_error: false, + duration: Duration::from_millis(1), + })], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "cap", GrantedActions::All, None, None) + .await + .unwrap(); + + let calls = vec![ActionCall { + id: "aB3xK9mZq".into(), // Mistral-compatible 9-char ID + action_name: "echo".into(), + parameters: serde_json::json!({}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + // Must NOT be empty — must be stamped from the ActionCall + assert!(!result.results[0].call_id.is_empty()); + assert_eq!(result.results[0].call_id, "aB3xK9mZq"); + } + + /// Mistral requires call_id matching [a-zA-Z0-9]{9}. + /// Verify the ID passes through unmodified (normalization is LLM-layer concern, + /// but engine must never lose it). + #[tokio::test] + async fn mistral_format_call_id_preserved() { + let thread = Thread::new( + "test", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ); + let effects: Arc = Arc::new(MockEffects::new( + vec![test_action("web_search")], + vec![Ok(ActionResult { + call_id: String::new(), + action_name: "web_search".into(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + })], + )); + let leases = Arc::new(LeaseManager::new()); + let policy = Arc::new(PolicyEngine::new()); + let ctx = make_exec_context(&thread); + + leases + .grant(thread.id, "cap", GrantedActions::All, None, None) + .await + .unwrap(); + + // Mistral format: exactly 9 alphanumeric chars + let mistral_id = "xK3mR9bZq"; + let calls = vec![ActionCall { + id: mistral_id.into(), + action_name: "web_search".into(), + parameters: serde_json::json!({}), + }]; + + let result = execute_action_calls(&calls, &thread, &effects, &leases, &policy, &ctx, &[]) + .await + .unwrap(); + + assert_eq!(result.results[0].call_id, mistral_id); + + // Event also preserves the exact format + if let Some(EventKind::ActionExecuted { call_id, .. }) = result.events.first() { + assert_eq!(call_id, mistral_id); + } + } +} diff --git a/crates/ironclaw_engine/src/executor/trace.rs b/crates/ironclaw_engine/src/executor/trace.rs new file mode 100644 index 00000000000..0252c30c3df --- /dev/null +++ b/crates/ironclaw_engine/src/executor/trace.rs @@ -0,0 +1,603 @@ +//! Execution trace analysis. +//! +//! Builds an in-memory `ExecutionTrace` from a completed `Thread` and runs a +//! retrospective analyzer that flags common failure patterns. Used by the +//! self-improvement mission and surfaced in debug logs. +//! +//! **There is no separate engine trace file.** Live trace recording for the +//! whole system is handled by `RecordingLlm` in the host crate +//! (`src/llm/recording.rs`), gated by `IRONCLAW_RECORD_TRACE`. Because the +//! engine's `LlmBackend` is wired to the same provider chain, engine LLM +//! interactions are captured by that single recorder — no engine-side env var +//! and no second JSON file. + +use chrono::Utc; +use serde::Serialize; +use tracing::debug; + +use crate::types::event::ThreadEvent; +use crate::types::thread::{Thread, ThreadId, ThreadState}; + +/// A complete execution trace for a single thread. +#[derive(Debug, Serialize)] +pub struct ExecutionTrace { + pub thread_id: ThreadId, + pub goal: String, + pub final_state: ThreadState, + pub step_count: usize, + pub total_tokens: u64, + pub messages: Vec, + pub events: Vec, + pub issues: Vec, + pub timestamp: chrono::DateTime, +} + +/// A single doc record, for the trace. +#[derive(Debug, Serialize)] +pub struct DocRecord { + pub doc_type: String, + pub title: String, + pub content: String, +} + +/// A message in the trace with role labeling. +#[derive(Debug, Serialize)] +pub struct MessageRecord { + pub role: String, + pub content_length: usize, + pub content_preview: String, + pub full_content: String, + pub action_name: Option, + pub action_call_id: Option, +} + +/// An issue detected by the retrospective analyzer. +#[derive(Debug, Serialize)] +pub struct TraceIssue { + pub severity: IssueSeverity, + pub category: String, + pub description: String, + pub step: Option, +} + +#[derive(Debug, PartialEq, Serialize)] +pub enum IssueSeverity { + Error, + Warning, + Info, +} + +/// Build a trace from a completed thread. +pub fn build_trace(thread: &Thread) -> ExecutionTrace { + let messages: Vec = thread + .messages + .iter() + .map(|m| { + let preview: String = m.content.chars().take(300).collect(); + MessageRecord { + role: format!("{:?}", m.role), + content_length: m.content.chars().count(), + content_preview: if m.content.chars().count() > 300 { + format!("{preview}...") + } else { + preview + }, + full_content: m.content.clone(), + action_name: m.action_name.clone(), + action_call_id: m.action_call_id.clone(), + } + }) + .collect(); + + let issues = analyze_trace(thread); + + ExecutionTrace { + thread_id: thread.id, + goal: thread.goal.clone(), + final_state: thread.state, + step_count: thread.step_count, + total_tokens: thread.total_tokens_used, + messages, + events: thread.events.clone(), + issues, + timestamp: Utc::now(), + } +} + +/// Print a summary of the trace to the log. +pub fn log_trace_summary(trace: &ExecutionTrace) { + debug!( + thread_id = %trace.thread_id, + goal = %trace.goal, + state = ?trace.final_state, + steps = trace.step_count, + tokens = trace.total_tokens, + messages = trace.messages.len(), + events = trace.events.len(), + issues = trace.issues.len(), + "=== Engine V2 Trace Summary ===" + ); + + for issue in &trace.issues { + match issue.severity { + IssueSeverity::Error => debug!( + category = %issue.category, + step = ?issue.step, + "ISSUE: {}", + issue.description + ), + IssueSeverity::Warning => debug!( + category = %issue.category, + step = ?issue.step, + "WARNING: {}", + issue.description + ), + IssueSeverity::Info => debug!( + category = %issue.category, + step = ?issue.step, + "NOTE: {}", + issue.description + ), + } + } +} + +// ── Retrospective analysis ────────────────────────────────── + +/// Analyze a completed thread for common issues. +fn analyze_trace(thread: &Thread) -> Vec { + let mut issues = Vec::new(); + + // 1. Check if the thread failed + if thread.state == ThreadState::Failed { + issues.push(TraceIssue { + severity: IssueSeverity::Error, + category: "thread_failure".into(), + description: "Thread ended in Failed state".into(), + step: None, + }); + } + + // 2. Check for empty response (no FINAL, no useful output) + let has_assistant_response = thread + .messages + .iter() + .any(|m| m.role == crate::types::message::MessageRole::Assistant && !m.content.is_empty()); + if !has_assistant_response { + issues.push(TraceIssue { + severity: IssueSeverity::Warning, + category: "no_response".into(), + description: "No assistant message in thread — model may not have generated output" + .into(), + step: None, + }); + } + + // 3. Check for tool errors + let tool_errors: Vec<&ThreadEvent> = thread + .events + .iter() + .filter(|e| matches!(e.kind, crate::types::event::EventKind::ActionFailed { .. })) + .collect(); + if !tool_errors.is_empty() { + for event in &tool_errors { + if let crate::types::event::EventKind::ActionFailed { + action_name, error, .. + } = &event.kind + { + issues.push(TraceIssue { + severity: IssueSeverity::Warning, + category: "tool_error".into(), + description: format!("Tool '{action_name}' failed: {error}"), + step: None, + }); + } + } + } + + // 4. Check for code execution errors in output messages. + // Code output appears as User-role messages (Monty stdout/stderr) with + // prefixes like "[stdout]" or "[stderr]". Skip the System prompt (index 0) + // and Assistant messages to avoid false positives from example text. + let error_patterns = [ + "NameError", + "SyntaxError", + "TypeError", + "NotImplementedError", + ]; + for (i, msg) in thread.messages.iter().enumerate() { + let is_code_output = msg.role == crate::types::message::MessageRole::User + && (msg.content.starts_with("[stdout]") + || msg.content.starts_with("[stderr]") + || msg.content.starts_with("[code ") + || msg.content.starts_with("Traceback")); + if is_code_output && error_patterns.iter().any(|p| msg.content.contains(p)) { + let preview: String = msg.content.chars().take(200).collect(); + issues.push(TraceIssue { + severity: IssueSeverity::Warning, + category: "code_error".into(), + description: format!("Code execution error in message {i}: {preview}"), + step: None, + }); + } + } + + // 5. Check for empty call_id on ActionResult messages (causes LLM API rejection). + for (i, msg) in thread.messages.iter().enumerate() { + if msg.role == crate::types::message::MessageRole::ActionResult { + let call_id_empty = msg.action_call_id.as_ref().is_none_or(|id| id.is_empty()); + if call_id_empty { + let name = msg.action_name.as_deref().unwrap_or("unknown"); + issues.push(TraceIssue { + severity: IssueSeverity::Error, + category: "empty_call_id".into(), + description: format!( + "ActionResult message {i} (tool '{name}') has empty call_id — will cause LLM API rejection" + ), + step: None, + }); + } + } + } + + // 6. Check for model ignoring tool results (hallucination risk). + // In Tier 0 (structured), results appear as ActionResult messages. + // In Tier 1 (CodeAct), results appear as User messages with "[tool result]" prefixes. + let has_tool_results = thread + .messages + .iter() + .any(|m| m.role == crate::types::message::MessageRole::ActionResult); + let has_tool_output_in_context = thread.messages.iter().any(|m| { + m.role == crate::types::message::MessageRole::User + && (m.content.contains(" result]") || m.content.contains(" error]")) + }); + if has_tool_results && !has_tool_output_in_context { + issues.push(TraceIssue { + severity: IssueSeverity::Warning, + category: "missing_tool_output".into(), + description: + "Tool results exist but no tool output in messages — model may not see tool results" + .into(), + step: None, + }); + } + + // 7. Check for excessive iterations + if thread.step_count > 10 { + issues.push(TraceIssue { + severity: IssueSeverity::Warning, + category: "excessive_steps".into(), + description: format!( + "Thread took {} steps — may be stuck in a loop", + thread.step_count + ), + step: None, + }); + } + + // 8. Check for text response without FINAL (model answered from memory) + let text_without_code = thread.events.iter().all(|e| { + !matches!( + e.kind, + crate::types::event::EventKind::ActionExecuted { .. } + ) + }); + if text_without_code && thread.step_count == 1 && has_assistant_response { + issues.push(TraceIssue { + severity: IssueSeverity::Info, + category: "no_tools_used".into(), + description: "Model answered in one step without using any tools — may be answering from training data".into(), + step: Some(1), + }); + } + + // 9. Check for LLM not producing code blocks + let code_steps = thread + .events + .iter() + .filter(|e| matches!(e.kind, crate::types::event::EventKind::StepStarted { .. })) + .count(); + let text_responses_without_code = thread + .messages + .iter() + .filter(|m| { + m.role == crate::types::message::MessageRole::Assistant + && !m.content.contains("```") + && !m.content.contains("FINAL(") + }) + .count(); + if text_responses_without_code > 0 && code_steps > 0 { + issues.push(TraceIssue { + severity: IssueSeverity::Info, + category: "mixed_mode".into(), + description: format!( + "{text_responses_without_code} text response(s) without code blocks — model may not be following CodeAct prompt" + ), + step: None, + }); + } + + // 10. Extract failure reason from StateChanged → Failed events + for event in &thread.events { + if let crate::types::event::EventKind::StateChanged { + to: ThreadState::Failed, + reason: Some(reason), + .. + } = &event.kind + { + if reason.contains("LLM") || reason.contains("Provider") { + issues.push(TraceIssue { + severity: IssueSeverity::Error, + category: "llm_error".into(), + description: format!("LLM provider error: {}", truncate(reason, 300)), + step: None, + }); + } else if reason.contains("orchestrator") { + issues.push(TraceIssue { + severity: IssueSeverity::Error, + category: "orchestrator_error".into(), + description: format!("Orchestrator error: {}", truncate(reason, 300)), + step: None, + }); + } + } + } + + issues +} + +fn truncate(s: &str, max_chars: usize) -> String { + let chars: String = s.chars().take(max_chars).collect(); + if s.chars().count() > max_chars { + format!("{chars}...") + } else { + chars + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::event::EventKind; + use crate::types::message::ThreadMessage; + use crate::types::project::ProjectId; + use crate::types::step::StepId; + use crate::types::thread::{ThreadConfig, ThreadType}; + + fn make_thread() -> Thread { + Thread::new( + "test goal", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ) + } + + // ── empty_call_id detection (OpenAI / Codex rejection) ─── + + /// OpenAI and Codex reject ActionResult messages with empty call_id. + /// The trace analyzer must flag these as errors. + #[test] + fn detects_empty_call_id_on_action_result() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("calling tool")); + // Simulate the bug: empty call_id + thread.add_message(ThreadMessage::action_result("", "web_search", "result")); + + let issues = analyze_trace(&thread); + let empty_id_issues: Vec<_> = issues + .iter() + .filter(|i| i.category == "empty_call_id") + .collect(); + + assert_eq!(empty_id_issues.len(), 1); + assert_eq!(empty_id_issues[0].severity, IssueSeverity::Error); + assert!(empty_id_issues[0].description.contains("web_search")); + } + + /// ActionResult with None call_id should also be flagged. + #[test] + fn detects_none_call_id_on_action_result() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("calling tool")); + // Manually construct a message with None call_id + thread.add_message(ThreadMessage { + role: crate::types::message::MessageRole::ActionResult, + content: "result".into(), + provenance: crate::types::provenance::Provenance::ToolOutput { + action_name: "shell".into(), + }, + action_call_id: None, + action_name: Some("shell".into()), + action_calls: None, + timestamp: chrono::Utc::now(), + }); + + let issues = analyze_trace(&thread); + assert!(issues.iter().any(|i| i.category == "empty_call_id")); + } + + /// No false positive: valid call_id should not be flagged. + #[test] + fn no_false_positive_for_valid_call_id() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("calling tool")); + thread.add_message(ThreadMessage::action_result( + "call_abc123", + "web_search", + "result", + )); + + let issues = analyze_trace(&thread); + assert!( + !issues.iter().any(|i| i.category == "empty_call_id"), + "valid call_id should not be flagged" + ); + } + + // ── tool_error detection ───────────────────────────────── + + /// ActionFailed events should produce tool_error warnings. + #[test] + fn detects_tool_failures_in_events() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("ok")); + thread.events.push(ThreadEvent::new( + thread.id, + EventKind::ActionFailed { + step_id: StepId::new(), + action_name: "web_search".into(), + call_id: "call_123".into(), + error: "No lease for action 'web_search'".into(), + params_summary: None, + }, + )); + + let issues = analyze_trace(&thread); + let tool_errors: Vec<_> = issues + .iter() + .filter(|i| i.category == "tool_error") + .collect(); + assert_eq!(tool_errors.len(), 1); + assert!(tool_errors[0].description.contains("web_search")); + } + + // ── thread_failure detection ───────────────────────────── + + #[test] + fn detects_failed_thread_state() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("trying")); + thread.state = ThreadState::Failed; + + let issues = analyze_trace(&thread); + assert!(issues.iter().any(|i| i.category == "thread_failure")); + } + + // ── LLM error detection from StateChanged events ───────── + + /// Reproduces the exact pattern from the trace: OpenAI rejects empty call_id. + #[test] + fn detects_llm_error_from_state_changed() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("ok")); + thread.state = ThreadState::Failed; + thread.events.push(ThreadEvent::new( + thread.id, + EventKind::StateChanged { + from: ThreadState::Running, + to: ThreadState::Failed, + reason: Some( + "LLM error: Provider openai_codex request failed: HTTP 400 Bad Request: \ + Invalid 'input[5].call_id': empty string" + .into(), + ), + }, + )); + + let issues = analyze_trace(&thread); + assert!( + issues.iter().any(|i| i.category == "llm_error"), + "should detect LLM provider error in StateChanged reason" + ); + } + + // ── Multiple empty call_ids ────────────────────────────── + + /// Anthropic sends consecutive tool results merged into one User message. + /// If multiple ActionResults have empty call_ids, each must be flagged. + #[test] + fn flags_each_empty_call_id_separately() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("parallel calls")); + thread.add_message(ThreadMessage::action_result("", "tool_a", "result_a")); + thread.add_message(ThreadMessage::action_result("", "tool_b", "result_b")); + thread.add_message(ThreadMessage::action_result( + "call_ok", "tool_c", "result_c", + )); + + let issues = analyze_trace(&thread); + let empty_issues: Vec<_> = issues + .iter() + .filter(|i| i.category == "empty_call_id") + .collect(); + assert_eq!( + empty_issues.len(), + 2, + "should flag exactly the 2 empty call_ids" + ); + } + + #[test] + fn trace_serializes_approval_request_payload() { + let mut thread = make_thread(); + thread.add_message(ThreadMessage::system("sys")); + thread.add_message(ThreadMessage::assistant("installing notion")); + thread.events.push(ThreadEvent::new( + thread.id, + EventKind::ApprovalRequested { + action_name: "tool_install".into(), + call_id: "call_install_1".into(), + parameters: Some(serde_json::json!({"name": "notion", "kind": "mcp_server"})), + description: Some("Install an extension".into()), + allow_always: Some(true), + gate_name: Some("approval".into()), + params_summary: Some("notion".into()), + }, + )); + + let trace = build_trace(&thread); + // `Thread::add_message` records a `MessageAdded` event for each + // message, so the `ApprovalRequested` event is no longer at index 0 + // — it's mixed in with the message events. Find it by kind. + let approval = trace + .events + .iter() + .find(|e| matches!(&e.kind, EventKind::ApprovalRequested { .. })) + .expect("trace should contain an ApprovalRequested event"); + match &approval.kind { + EventKind::ApprovalRequested { + action_name, + call_id, + parameters, + description, + allow_always, + gate_name, + params_summary, + } => { + assert_eq!(action_name, "tool_install"); + assert_eq!(call_id, "call_install_1"); + assert_eq!( + parameters.as_ref().and_then(|p| p.get("name")), + Some(&serde_json::json!("notion")) + ); + assert_eq!(description.as_deref(), Some("Install an extension")); + assert_eq!(*allow_always, Some(true)); + assert_eq!(gate_name.as_deref(), Some("approval")); + assert_eq!(params_summary.as_deref(), Some("notion")); + } + other => panic!("unexpected event kind: {other:?}"), + } + + let json = serde_json::to_string(&trace).expect("trace serializes"); + assert!(json.contains("\"ApprovalRequested\"")); + assert!(json.contains("\"action_name\":\"tool_install\"")); + assert!(json.contains("\"call_id\":\"call_install_1\"")); + // Parameter map key order isn't stable across serde_json versions; check + // both required keys are present rather than the exact serialized form. + assert!(json.contains("\"name\":\"notion\"")); + assert!(json.contains("\"kind\":\"mcp_server\"")); + assert!(json.contains("\"description\":\"Install an extension\"")); + assert!(json.contains("\"allow_always\":true")); + assert!(json.contains("\"gate_name\":\"approval\"")); + assert!(json.contains("\"params_summary\":\"notion\"")); + } +} diff --git a/crates/ironclaw_engine/src/gate/lease.rs b/crates/ironclaw_engine/src/gate/lease.rs new file mode 100644 index 00000000000..0b47613924e --- /dev/null +++ b/crates/ironclaw_engine/src/gate/lease.rs @@ -0,0 +1,211 @@ +//! Lease gate — denies tool calls with no valid capability lease. +//! +//! Priority 10: runs before all other gates (deny early if no lease). +//! This replaces v1's `ApprovalContext::is_blocked_or_default()`, +//! `check_approval_in_context()`, and the ad-hoc `allowed_tools: HashSet` +//! in lightweight routines. + +use std::sync::Arc; + +use async_trait::async_trait; + +use crate::capability::lease::LeaseManager; +use crate::gate::{ExecutionGate, GateContext, GateDecision}; + +/// Gate that denies tool calls not covered by a valid capability lease. +/// +/// Fail-closed: if no lease exists for the action, execution is denied. +/// This is the primary authorization gate for the engine. +pub struct LeaseGate { + lease_manager: Arc, + /// When true, skip lease checks (for interactive threads where + /// leases are still being granted by the planner). + permissive: bool, +} + +impl LeaseGate { + /// Create a lease gate that enforces lease checks. + pub fn new(lease_manager: Arc) -> Self { + Self { + lease_manager, + permissive: false, + } + } + + /// Create a permissive gate that allows all actions (for Foreground + /// threads where interactive approval handles authorization). + pub fn permissive(lease_manager: Arc) -> Self { + Self { + lease_manager, + permissive: true, + } + } +} + +#[async_trait] +impl ExecutionGate for LeaseGate { + fn name(&self) -> &str { + "lease" + } + + fn priority(&self) -> u32 { + 10 + } + + async fn evaluate(&self, ctx: &GateContext<'_>) -> GateDecision { + if self.permissive { + return GateDecision::Allow; + } + + match self + .lease_manager + .find_lease_for_action(ctx.thread_id, ctx.action_name) + .await + { + Some(lease) if lease.is_valid() => GateDecision::Allow, + _ => GateDecision::Deny { + reason: format!( + "No valid lease for action '{}' on thread {}", + ctx.action_name, ctx.thread_id + ), + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::gate::ExecutionMode; + use crate::types::capability::{ActionDef, EffectType, GrantedActions}; + use crate::types::thread::ThreadId; + use std::collections::HashSet; + + fn action_def(name: &str) -> ActionDef { + ActionDef { + name: name.into(), + description: String::new(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + } + } + + fn ctx<'a>( + thread_id: ThreadId, + action_def: &'a ActionDef, + auto: &'a HashSet, + params: &'a serde_json::Value, + ) -> GateContext<'a> { + GateContext { + user_id: "user1", + thread_id, + source_channel: "web", + action_name: &action_def.name, + call_id: "call_1", + parameters: params, + action_def, + execution_mode: ExecutionMode::Autonomous, + auto_approved: auto, + } + } + + #[tokio::test] + async fn test_valid_lease_allows() { + let mgr = Arc::new(LeaseManager::new()); + let tid = ThreadId::new(); + mgr.grant( + tid, + "tools", + GrantedActions::Specific(vec!["read_file".into()]), + None, + None, + ) + .await + .unwrap(); + + let gate = LeaseGate::new(Arc::clone(&mgr)); + let ad = action_def("read_file"); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let c = ctx(tid, &ad, &auto, ¶ms); + assert!(matches!(gate.evaluate(&c).await, GateDecision::Allow)); + } + + #[tokio::test] + async fn test_no_lease_denies() { + let mgr = Arc::new(LeaseManager::new()); + let tid = ThreadId::new(); + // No leases granted + + let gate = LeaseGate::new(Arc::clone(&mgr)); + let ad = action_def("shell"); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let c = ctx(tid, &ad, &auto, ¶ms); + assert!(matches!(gate.evaluate(&c).await, GateDecision::Deny { .. })); + } + + #[tokio::test] + async fn test_expired_lease_denies() { + let mgr = Arc::new(LeaseManager::new()); + let tid = ThreadId::new(); + let lease = mgr + .grant( + tid, + "tools", + GrantedActions::Specific(vec!["read_file".into()]), + None, + Some(1), + ) + .await + .unwrap(); + // Exhaust the lease so it becomes invalid + mgr.consume_use(lease.id).await.unwrap(); + + let gate = LeaseGate::new(Arc::clone(&mgr)); + let ad = action_def("read_file"); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let c = ctx(tid, &ad, &auto, ¶ms); + assert!(matches!(gate.evaluate(&c).await, GateDecision::Deny { .. })); + } + + #[tokio::test] + async fn test_revoked_lease_denies() { + let mgr = Arc::new(LeaseManager::new()); + let tid = ThreadId::new(); + let lease = mgr + .grant( + tid, + "tools", + GrantedActions::Specific(vec!["read_file".into()]), + None, + None, + ) + .await + .unwrap(); + mgr.revoke(lease.id, "test").await; + + let gate = LeaseGate::new(Arc::clone(&mgr)); + let ad = action_def("read_file"); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let c = ctx(tid, &ad, &auto, ¶ms); + assert!(matches!(gate.evaluate(&c).await, GateDecision::Deny { .. })); + } + + #[tokio::test] + async fn test_permissive_gate_allows_everything() { + let mgr = Arc::new(LeaseManager::new()); + let tid = ThreadId::new(); + // No leases, but permissive mode + + let gate = LeaseGate::permissive(Arc::clone(&mgr)); + let ad = action_def("shell"); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let c = ctx(tid, &ad, &auto, ¶ms); + assert!(matches!(gate.evaluate(&c).await, GateDecision::Allow)); + } +} diff --git a/crates/ironclaw_engine/src/gate/mod.rs b/crates/ironclaw_engine/src/gate/mod.rs new file mode 100644 index 00000000000..4b32c1410b7 --- /dev/null +++ b/crates/ironclaw_engine/src/gate/mod.rs @@ -0,0 +1,179 @@ +//! Unified execution gate abstraction. +//! +//! All pre-execution checks (approval, authentication, rate limiting, hooks, +//! relay channel enforcement) are expressed as composable [`ExecutionGate`] +//! implementations evaluated through a [`GatePipeline`]. +//! +//! Design invariants: +//! - [`GateDecision`] has no `None` variant — fail-closed by construction. +//! - [`ResumeKind`] is a closed enum — forces all pause paths through +//! the same storage, resolution, and SSE machinery. +//! - [`GateContext`] borrows everything — zero cloning in the hot path. + +pub mod lease; +pub mod pipeline; +pub mod tool_tier; + +use std::collections::HashSet; + +use async_trait::async_trait; +use serde::{Deserialize, Serialize}; + +use crate::types::capability::ActionDef; +use crate::types::thread::ThreadId; + +// ── Gate decision ─────────────────────────────────────────── + +/// The outcome of evaluating an execution gate. +#[derive(Debug, Clone)] +pub enum GateDecision { + /// Execution may proceed. + Allow, + /// Execution must pause until the user provides input. + Pause { + reason: String, + resume_kind: ResumeKind, + }, + /// Execution is denied outright. + Deny { reason: String }, +} + +/// What kind of external input will resolve a paused gate. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum ResumeKind { + /// User must approve or deny the tool invocation. + Approval { + /// Whether the "always approve this tool" option should be offered. + allow_always: bool, + }, + /// User must provide a credential (token, API key, OAuth flow). + Authentication { + /// Name of the credential that is missing. + credential_name: String, + /// User-facing setup instructions. + instructions: String, + /// Optional OAuth URL for browser-based flows. + auth_url: Option, + }, + /// An external system must respond (webhook confirmation, etc.). + External { callback_id: String }, +} + +impl ResumeKind { + /// Short human-readable label for this kind. + pub fn kind_name(&self) -> &'static str { + match self { + Self::Approval { .. } => "approval", + Self::Authentication { .. } => "authentication", + Self::External { .. } => "external confirmation", + } + } +} + +// ── Gate resolution ───────────────────────────────────────── + +/// How a paused gate is resolved by the user or external system. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum GateResolution { + /// User approved the tool call. + Approved { always: bool }, + /// User denied the tool call. + Denied { reason: Option }, + /// User provided a credential value. + CredentialProvided { token: String }, + /// User or system cancelled the pending gate entirely. + Cancelled, + /// External callback received. + ExternalCallback { payload: serde_json::Value }, +} + +// ── Execution mode ────────────────────────────────────────── + +/// The execution context in which a tool call is being evaluated. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ExecutionMode { + /// Interactive session — a user can approve / authenticate. + Interactive, + /// Interactive session with auto-approve enabled. + /// + /// `UnlessAutoApproved` tools pass without prompting (shell, file_write, + /// http, etc.). `Always`-gated tools (destructive operations) still pause + /// for explicit approval. All other safeguards remain active: leases, + /// rate limits, hooks, relay channel checks, authentication gates. + /// + /// Activated via `AGENT_AUTO_APPROVE_TOOLS=true` or settings. + InteractiveAutoApprove, + /// Autonomous background job — no interactive user. + /// The lease set determines what tools are available. + Autonomous, + /// Container-sandboxed execution. + Container, +} + +// ── Gate context ──────────────────────────────────────────── + +/// Immutable snapshot of everything a gate needs to make a decision. +/// +/// String and Value fields are borrowed to avoid cloning in the hot path. +/// `ThreadId` and `ExecutionMode` are `Copy` and stored inline. +#[derive(Debug)] +pub struct GateContext<'a> { + pub user_id: &'a str, + pub thread_id: ThreadId, + pub source_channel: &'a str, + pub action_name: &'a str, + pub call_id: &'a str, + pub parameters: &'a serde_json::Value, + pub action_def: &'a ActionDef, + pub execution_mode: ExecutionMode, + /// Tools the session has auto-approved ("always" button). + pub auto_approved: &'a HashSet, +} + +// ── Gate trait ─────────────────────────────────────────────── + +/// A single pre-execution check. +/// +/// Implementations must be deterministic for a given context snapshot: +/// they must not hold mutable state that changes across evaluations +/// within a single pipeline run. +#[async_trait] +pub trait ExecutionGate: Send + Sync { + /// Unique name for logging and persistence. + fn name(&self) -> &str; + + /// Evaluation priority. Lower runs first. First `Pause` or `Deny` wins. + fn priority(&self) -> u32; + + /// Evaluate whether the tool invocation should proceed. + async fn evaluate(&self, ctx: &GateContext<'_>) -> GateDecision; +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn resume_kind_labels() { + assert_eq!( + ResumeKind::Approval { allow_always: true }.kind_name(), + "approval" + ); + assert_eq!( + ResumeKind::Authentication { + credential_name: "x".into(), + instructions: "y".into(), + auth_url: None, + } + .kind_name(), + "authentication" + ); + assert_eq!( + ResumeKind::External { + callback_id: "z".into() + } + .kind_name(), + "external confirmation" + ); + } +} diff --git a/crates/ironclaw_engine/src/gate/pipeline.rs b/crates/ironclaw_engine/src/gate/pipeline.rs new file mode 100644 index 00000000000..9b4c8cfe02a --- /dev/null +++ b/crates/ironclaw_engine/src/gate/pipeline.rs @@ -0,0 +1,241 @@ +//! Gate pipeline — ordered evaluation of multiple [`ExecutionGate`]s. +//! +//! Gates are sorted by priority at construction time. The first gate to +//! return [`GateDecision::Pause`] or [`GateDecision::Deny`] wins. +//! If all gates return [`GateDecision::Allow`], execution proceeds. +//! +//! Gate implementations must not panic. A panicking gate will propagate +//! the panic to the caller (async `catch_unwind` is not used because +//! the gate evaluation borrows non-`UnwindSafe` context). + +use std::sync::Arc; + +use super::{ExecutionGate, GateContext, GateDecision}; + +/// Ordered pipeline of execution gates. +pub struct GatePipeline { + gates: Vec>, +} + +impl GatePipeline { + /// Build a pipeline from the given gates, sorted by priority (ascending). + pub fn new(mut gates: Vec>) -> Self { + gates.sort_by_key(|g| g.priority()); + Self { gates } + } + + /// Build an empty pipeline that allows everything (useful in tests). + pub fn allow_all() -> Self { + Self { gates: Vec::new() } + } + + /// Evaluate all gates in priority order. First `Pause` or `Deny` wins. + /// + /// Gate implementations must not panic — a panic propagates to the caller. + pub async fn evaluate(&self, ctx: &GateContext<'_>) -> GateDecision { + for gate in &self.gates { + let decision = gate.evaluate(ctx).await; + match decision { + GateDecision::Allow => continue, + GateDecision::Pause { .. } | GateDecision::Deny { .. } => { + tracing::debug!( + gate = gate.name(), + tool = %ctx.action_name, + "gate stopped execution" + ); + return decision; + } + } + } + GateDecision::Allow + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::gate::{ExecutionMode, ResumeKind}; + use crate::types::capability::{ActionDef, EffectType}; + use crate::types::thread::ThreadId; + use std::collections::HashSet; + + // ── Test helpers ──────────────────────────────────────── + + fn test_action_def() -> ActionDef { + ActionDef { + name: "test_tool".into(), + description: "a test tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + } + } + + fn test_ctx<'a>( + action_def: &'a ActionDef, + auto_approved: &'a HashSet, + params: &'a serde_json::Value, + ) -> GateContext<'a> { + GateContext { + user_id: "user1", + thread_id: ThreadId::new(), + source_channel: "web", + action_name: &action_def.name, + call_id: "call_1", + parameters: params, + action_def, + execution_mode: ExecutionMode::Interactive, + auto_approved, + } + } + + struct StaticGate { + name: &'static str, + priority: u32, + decision: GateDecision, + } + + #[async_trait::async_trait] + impl ExecutionGate for StaticGate { + fn name(&self) -> &str { + self.name + } + fn priority(&self) -> u32 { + self.priority + } + async fn evaluate(&self, _ctx: &GateContext<'_>) -> GateDecision { + self.decision.clone() + } + } + + // ── Tests ─────────────────────────────────────────────── + + #[tokio::test] + async fn test_all_allow_passes() { + let pipeline = GatePipeline::new(vec![ + Arc::new(StaticGate { + name: "a", + priority: 10, + decision: GateDecision::Allow, + }), + Arc::new(StaticGate { + name: "b", + priority: 20, + decision: GateDecision::Allow, + }), + ]); + let ad = test_action_def(); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let ctx = test_ctx(&ad, &auto, ¶ms); + assert!(matches!(pipeline.evaluate(&ctx).await, GateDecision::Allow)); + } + + #[tokio::test] + async fn test_first_deny_wins() { + let pipeline = GatePipeline::new(vec![ + Arc::new(StaticGate { + name: "allow", + priority: 10, + decision: GateDecision::Allow, + }), + Arc::new(StaticGate { + name: "deny", + priority: 20, + decision: GateDecision::Deny { + reason: "blocked".into(), + }, + }), + Arc::new(StaticGate { + name: "allow2", + priority: 30, + decision: GateDecision::Allow, + }), + ]); + let ad = test_action_def(); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let ctx = test_ctx(&ad, &auto, ¶ms); + assert!(matches!( + pipeline.evaluate(&ctx).await, + GateDecision::Deny { .. } + )); + } + + #[tokio::test] + async fn test_first_pause_wins_over_later_deny() { + let pipeline = GatePipeline::new(vec![ + Arc::new(StaticGate { + name: "allow", + priority: 10, + decision: GateDecision::Allow, + }), + Arc::new(StaticGate { + name: "pause", + priority: 20, + decision: GateDecision::Pause { + reason: "needs approval".into(), + resume_kind: ResumeKind::Approval { allow_always: true }, + }, + }), + Arc::new(StaticGate { + name: "deny", + priority: 30, + decision: GateDecision::Deny { + reason: "would deny".into(), + }, + }), + ]); + let ad = test_action_def(); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let ctx = test_ctx(&ad, &auto, ¶ms); + assert!(matches!( + pipeline.evaluate(&ctx).await, + GateDecision::Pause { .. } + )); + } + + #[tokio::test] + async fn test_empty_pipeline_allows() { + let pipeline = GatePipeline::allow_all(); + let ad = test_action_def(); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let ctx = test_ctx(&ad, &auto, ¶ms); + assert!(matches!(pipeline.evaluate(&ctx).await, GateDecision::Allow)); + } + + #[tokio::test] + async fn test_priority_ordering() { + // Insert gates in reverse priority order — pipeline should still + // evaluate the lower-priority (deny) gate first. + let pipeline = GatePipeline::new(vec![ + Arc::new(StaticGate { + name: "pause_high", + priority: 200, + decision: GateDecision::Pause { + reason: "late pause".into(), + resume_kind: ResumeKind::Approval { + allow_always: false, + }, + }, + }), + Arc::new(StaticGate { + name: "deny_low", + priority: 10, + decision: GateDecision::Deny { + reason: "early deny".into(), + }, + }), + ]); + let ad = test_action_def(); + let auto = HashSet::new(); + let params = serde_json::json!({}); + let ctx = test_ctx(&ad, &auto, ¶ms); + match pipeline.evaluate(&ctx).await { + GateDecision::Deny { reason } => assert_eq!(reason, "early deny"), + other => panic!("Expected Deny, got {other:?}"), + } + } +} diff --git a/crates/ironclaw_engine/src/gate/tool_tier.rs b/crates/ironclaw_engine/src/gate/tool_tier.rs new file mode 100644 index 00000000000..1451f0b8796 --- /dev/null +++ b/crates/ironclaw_engine/src/gate/tool_tier.rs @@ -0,0 +1,173 @@ +//! Tool tier classification. +//! +//! Maps each action to a privilege tier based on its declared effects and +//! approval requirements. Used by the [`LeasePlanner`] to scope thread-type +//! aware leases and by the [`LeaseGate`] for authorization checks. +//! +//! [`LeasePlanner`]: crate::capability::planner::LeasePlanner +//! [`LeaseGate`]: (future) + +use crate::types::capability::{ActionDef, EffectType}; + +/// Tool actions in the AUTONOMOUS_TOOL_DENYLIST — these are always +/// classified as [`ToolTier::Administrative`] regardless of their +/// declared effects. +pub const AUTONOMOUS_TOOL_DENYLIST: &[&str] = &[ + "routine_create", + "routine_update", + "routine_delete", + "routine_fire", + "event_emit", + "create_job", + "job_prompt", + "restart", + "tool_install", + "tool_auth", + "tool_activate", + "tool_remove", + "tool_upgrade", + "skill_install", + "skill_remove", + "secret_list", + "secret_delete", +]; + +/// Returns true if the action name is in the autonomous tool denylist. +pub fn is_autonomous_denylisted(action_name: &str) -> bool { + AUTONOMOUS_TOOL_DENYLIST.contains(&action_name) +} + +/// Privilege tier for a tool action. +/// +/// Tiers are totally ordered: `ReadOnly < Stateful < Privileged < Administrative`. +/// The [`LeasePlanner`] uses this ordering to decide which actions to grant +/// for each [`ThreadType`]. +/// +/// [`ThreadType`]: crate::types::thread::ThreadType +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum ToolTier { + /// Read-only, no side effects (echo, time, json, memory_search, memory_read). + ReadOnly, + /// Creates or reads local state (read_file, list_dir). + Stateful, + /// Write operations or external effects (shell, file_write, http, create_job). + Privileged, + /// System-level operations that should never run autonomously + /// (routine_*, tool_install, skill_*, secret_*, restart). + Administrative, +} + +/// Classify a tool action into a [`ToolTier`]. +/// +/// Classification rules (in priority order): +/// 1. Action name in [`AUTONOMOUS_TOOL_DENYLIST`] → `Administrative` +/// 2. `requires_approval: true` → `Privileged` +/// 3. Only `ReadLocal` / `Compute` effects → `ReadOnly` +/// 4. Everything else → `Stateful` +pub fn classify_tool_tier(action: &ActionDef) -> ToolTier { + // 1. Denylisted → Administrative + if is_autonomous_denylisted(&action.name) { + return ToolTier::Administrative; + } + + // 2. Requires approval → Privileged + if action.requires_approval { + return ToolTier::Privileged; + } + + // 3. Only read/compute effects → ReadOnly + let only_read_compute = !action.effects.is_empty() + && action + .effects + .iter() + .all(|e| matches!(e, EffectType::ReadLocal | EffectType::Compute)); + if only_read_compute { + return ToolTier::ReadOnly; + } + + // 4. Default + ToolTier::Stateful +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::capability::ActionDef; + + fn action(name: &str, effects: Vec, requires_approval: bool) -> ActionDef { + ActionDef { + name: name.into(), + description: String::new(), + parameters_schema: serde_json::json!({}), + effects, + requires_approval, + } + } + + #[test] + fn test_denylisted_tool_always_administrative() { + for &name in AUTONOMOUS_TOOL_DENYLIST { + let ad = action(name, vec![EffectType::ReadLocal], false); + assert_eq!( + classify_tool_tier(&ad), + ToolTier::Administrative, + "Expected Administrative for denylisted tool '{name}'" + ); + } + } + + #[test] + fn test_requires_approval_is_privileged() { + let ad = action("shell", vec![EffectType::WriteLocal], true); + assert_eq!(classify_tool_tier(&ad), ToolTier::Privileged); + } + + #[test] + fn test_read_only_effects() { + let ad = action( + "echo", + vec![EffectType::ReadLocal, EffectType::Compute], + false, + ); + assert_eq!(classify_tool_tier(&ad), ToolTier::ReadOnly); + } + + #[test] + fn test_read_local_only() { + let ad = action("memory_search", vec![EffectType::ReadLocal], false); + assert_eq!(classify_tool_tier(&ad), ToolTier::ReadOnly); + } + + #[test] + fn test_write_local_is_stateful() { + let ad = action("file_write", vec![EffectType::WriteLocal], false); + assert_eq!(classify_tool_tier(&ad), ToolTier::Stateful); + } + + #[test] + fn test_external_effects_stateful() { + let ad = action("web_fetch", vec![EffectType::ReadExternal], false); + assert_eq!(classify_tool_tier(&ad), ToolTier::Stateful); + } + + #[test] + fn test_no_effects_is_stateful() { + let ad = action("custom_tool", vec![], false); + assert_eq!(classify_tool_tier(&ad), ToolTier::Stateful); + } + + #[test] + fn test_denylisted_overrides_no_approval() { + // routine_create doesn't require_approval in its ActionDef, + // but should still be Administrative because it's denylisted. + let ad = action("routine_create", vec![EffectType::WriteLocal], false); + assert_eq!(classify_tool_tier(&ad), ToolTier::Administrative); + } + + #[test] + fn test_tier_ordering() { + assert!(ToolTier::ReadOnly < ToolTier::Stateful); + assert!(ToolTier::Stateful < ToolTier::Privileged); + assert!(ToolTier::Privileged < ToolTier::Administrative); + } +} diff --git a/crates/ironclaw_engine/src/lib.rs b/crates/ironclaw_engine/src/lib.rs new file mode 100644 index 00000000000..5894d7a7a64 --- /dev/null +++ b/crates/ironclaw_engine/src/lib.rs @@ -0,0 +1,544 @@ +//! IronClaw Engine — unified thread-capability-CodeAct execution model. +//! +//! This crate provides the core execution engine for IronClaw, unifying +//! ~10 separate abstractions (Session, Job, Routine, Channel, Tool, Skill, +//! Hook, Observer, Extension, LoopDelegate) around 5 primitives: +//! +//! - **Thread** — unit of work (replaces Session + Job + Routine + Sub-agent) +//! - **Step** — unit of execution (replaces agentic loop iteration + tool calls) +//! - **Capability** — unit of effect (replaces Tool + Skill + Hook + Extension) +//! - **MemoryDoc** — unit of durable knowledge (replaces workspace memory blobs) +//! - **Project** — unit of context (replaces flat workspace namespace) +//! +//! The engine defines traits for external dependencies ([`LlmBackend`], +//! [`Store`], [`EffectExecutor`]) that the host crate implements via bridge +//! adapters over existing infrastructure. + +// Security: `__regex_match__` (in `executor/orchestrator.rs`) accepts +// arbitrary patterns from the Python orchestrator and runs them on +// user-supplied text. The default `regex` crate is linear-time. The +// `fancy-regex` crate supports backreferences and is NOT linear-time, which +// would turn that handler into a ReDoS vector. Cargo.toml depends on +// `regex = "1"` with default features only — do NOT add `fancy-regex` to +// this crate's dependency tree without first redesigning `__regex_match__` +// to enforce a wall-clock matching budget. + +pub mod capability; +pub mod executor; +pub mod gate; +pub mod memory; +pub mod reliability; +pub mod runtime; +pub mod traits; +pub mod types; + +// ── Re-exports: types ─────────────────────────────────────── + +pub use types::capability::{ + ActionDef, Capability, CapabilityLease, EffectType, GrantedActions, LeaseId, PolicyCondition, + PolicyEffect, PolicyRule, +}; +pub use types::error::{CapabilityError, EngineError, StepError, ThreadError}; +pub use types::event::{EventId, EventKind, ThreadEvent}; +pub use types::memory::{DocId, DocType, MemoryDoc}; +pub use types::message::{MessageRole, ThreadMessage}; +pub use types::mission::{Mission, MissionCadence, MissionId, MissionStatus, ValidTimezone}; +pub use types::project::{Project, ProjectId}; +pub use types::provenance::Provenance; +pub use types::step::{ + ActionCall, ActionResult, ExecutionTier, LlmResponse, Step, StepId, StepStatus, TokenUsage, +}; +pub use types::thread::{ + ActiveSkillProvenance, Thread, ThreadConfig, ThreadId, ThreadState, ThreadType, +}; + +// ── Re-exports: traits ────────────────────────────────────── + +pub use traits::effect::{EffectExecutor, ThreadExecutionContext}; +pub use traits::llm::{LlmBackend, LlmCallConfig, LlmOutput}; +pub use traits::store::Store; +pub use traits::workspace::WorkspaceReader; + +// ── Re-exports: capability ──────────────────────────────────── + +pub use capability::lease::LeaseManager; +pub use capability::planner::{CapabilityGrantPlan, LeasePlanner}; +pub use capability::policy::{PolicyDecision, PolicyEngine}; +pub use capability::registry::CapabilityRegistry; + +// ── Re-exports: gate ───────────────────────────────────────── + +pub use gate::lease::LeaseGate; +pub use gate::pipeline::GatePipeline; +pub use gate::tool_tier::{ToolTier, classify_tool_tier}; +pub use gate::{ + ExecutionGate, ExecutionMode, GateContext, GateDecision, GateResolution, ResumeKind, +}; + +// ── Re-exports: runtime ─────────────────────────────────────── + +pub use executor::prompt::PlatformInfo; +pub use runtime::conversation::ConversationManager; +pub use runtime::manager::ThreadManager; +pub use runtime::messaging::ThreadOutcome; +pub use runtime::mission::{ + BudgetGate, FireRateLimit, MissionManager, MissionNotification, MissionUpdate, +}; +pub use runtime::tree::ThreadTree; + +pub use types::conversation::{ + ConversationEntry, ConversationId, ConversationSurface, EntrySender, +}; + +// ── Re-exports: executor ────────────────────────────────────── + +pub use executor::ExecutionLoop; + +// ── Re-exports: memory ──────────────────────────────────────── + +pub use memory::MemoryStore; +pub use memory::RetrievalEngine; + +// ── Re-exports: reliability ────────────────────────────────── + +pub use reliability::ReliabilityTracker; + +// ── Test utilities ────────────────────────────────────────── + +#[cfg(test)] +pub(crate) mod tests { + use tokio::sync::RwLock; + + use crate::traits::store::Store; + use crate::types::capability::{CapabilityLease, LeaseId}; + use crate::types::conversation::{ConversationId, ConversationSurface}; + use crate::types::error::EngineError; + use crate::types::event::ThreadEvent; + use crate::types::memory::{DocId, MemoryDoc}; + use crate::types::mission::{Mission, MissionId, MissionStatus}; + use crate::types::project::{Project, ProjectId}; + use crate::types::step::Step; + use crate::types::thread::{Thread, ThreadId, ThreadState}; + + /// Shared in-memory Store implementation for tests. + /// + /// Stores all entity types with proper CRUD semantics and filtering by + /// project_id / user_id. Use this instead of defining per-module mocks. + pub struct InMemoryStore { + threads: RwLock>, + steps: RwLock>, + events: RwLock>, + projects: RwLock>, + conversations: RwLock>, + docs: RwLock>, + leases: RwLock>, + missions: RwLock>, + } + + impl InMemoryStore { + pub fn new() -> Self { + Self { + threads: RwLock::new(Vec::new()), + steps: RwLock::new(Vec::new()), + events: RwLock::new(Vec::new()), + projects: RwLock::new(Vec::new()), + conversations: RwLock::new(Vec::new()), + docs: RwLock::new(Vec::new()), + leases: RwLock::new(Vec::new()), + missions: RwLock::new(Vec::new()), + } + } + + pub fn with_docs(docs: Vec) -> Self { + Self { + docs: RwLock::new(docs), + ..Self::new() + } + } + } + + #[async_trait::async_trait] + impl Store for InMemoryStore { + async fn save_thread(&self, thread: &Thread) -> Result<(), EngineError> { + let mut threads = self.threads.write().await; + threads.retain(|t| t.id != thread.id); + threads.push(thread.clone()); + Ok(()) + } + async fn load_thread(&self, id: ThreadId) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .iter() + .find(|t| t.id == id) + .cloned()) + } + async fn list_threads( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .iter() + .filter(|t| t.project_id == project_id && t.user_id == user_id) + .cloned() + .collect()) + } + async fn update_thread_state( + &self, + id: ThreadId, + state: ThreadState, + ) -> Result<(), EngineError> { + let mut threads = self.threads.write().await; + if let Some(t) = threads.iter_mut().find(|t| t.id == id) { + t.state = state; + } + Ok(()) + } + async fn save_step(&self, step: &Step) -> Result<(), EngineError> { + let mut steps = self.steps.write().await; + steps.retain(|s| s.id != step.id); + steps.push(step.clone()); + Ok(()) + } + async fn load_steps(&self, thread_id: ThreadId) -> Result, EngineError> { + Ok(self + .steps + .read() + .await + .iter() + .filter(|s| s.thread_id == thread_id) + .cloned() + .collect()) + } + async fn append_events(&self, events: &[ThreadEvent]) -> Result<(), EngineError> { + self.events.write().await.extend(events.iter().cloned()); + Ok(()) + } + async fn load_events(&self, thread_id: ThreadId) -> Result, EngineError> { + Ok(self + .events + .read() + .await + .iter() + .filter(|e| e.thread_id == thread_id) + .cloned() + .collect()) + } + async fn save_project(&self, project: &Project) -> Result<(), EngineError> { + let mut projects = self.projects.write().await; + projects.retain(|p| p.id != project.id); + projects.push(project.clone()); + Ok(()) + } + async fn load_project(&self, id: ProjectId) -> Result, EngineError> { + Ok(self + .projects + .read() + .await + .iter() + .find(|p| p.id == id) + .cloned()) + } + async fn list_projects(&self, user_id: &str) -> Result, EngineError> { + Ok(self + .projects + .read() + .await + .iter() + .filter(|p| p.user_id == user_id) + .cloned() + .collect()) + } + async fn list_all_projects(&self) -> Result, EngineError> { + Ok(self.projects.read().await.iter().cloned().collect()) + } + async fn save_conversation( + &self, + conversation: &ConversationSurface, + ) -> Result<(), EngineError> { + let mut conversations = self.conversations.write().await; + conversations.retain(|c| c.id != conversation.id); + conversations.push(conversation.clone()); + Ok(()) + } + async fn load_conversation( + &self, + id: ConversationId, + ) -> Result, EngineError> { + Ok(self + .conversations + .read() + .await + .iter() + .find(|c| c.id == id) + .cloned()) + } + async fn list_conversations( + &self, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .conversations + .read() + .await + .iter() + .filter(|c| c.user_id == user_id) + .cloned() + .collect()) + } + async fn save_memory_doc(&self, doc: &MemoryDoc) -> Result<(), EngineError> { + let mut docs = self.docs.write().await; + docs.retain(|d| d.id != doc.id); + docs.push(doc.clone()); + Ok(()) + } + async fn load_memory_doc(&self, id: DocId) -> Result, EngineError> { + Ok(self.docs.read().await.iter().find(|d| d.id == id).cloned()) + } + async fn list_memory_docs( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .docs + .read() + .await + .iter() + .filter(|d| d.project_id == project_id && d.user_id == user_id) + .cloned() + .collect()) + } + async fn save_lease(&self, lease: &CapabilityLease) -> Result<(), EngineError> { + let mut leases = self.leases.write().await; + leases.retain(|l| l.id != lease.id); + leases.push(lease.clone()); + Ok(()) + } + async fn load_active_leases( + &self, + thread_id: ThreadId, + ) -> Result, EngineError> { + Ok(self + .leases + .read() + .await + .iter() + .filter(|l| l.thread_id == thread_id && !l.revoked) + .cloned() + .collect()) + } + async fn revoke_lease(&self, lease_id: LeaseId, _reason: &str) -> Result<(), EngineError> { + let mut leases = self.leases.write().await; + if let Some(l) = leases.iter_mut().find(|l| l.id == lease_id) { + l.revoked = true; + } + Ok(()) + } + async fn save_mission(&self, mission: &Mission) -> Result<(), EngineError> { + let mut missions = self.missions.write().await; + missions.retain(|m| m.id != mission.id); + missions.push(mission.clone()); + Ok(()) + } + async fn load_mission(&self, id: MissionId) -> Result, EngineError> { + Ok(self + .missions + .read() + .await + .iter() + .find(|m| m.id == id) + .cloned()) + } + async fn list_missions( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .missions + .read() + .await + .iter() + .filter(|m| m.project_id == project_id && m.user_id == user_id) + .cloned() + .collect()) + } + async fn update_mission_status( + &self, + id: MissionId, + status: MissionStatus, + ) -> Result<(), EngineError> { + let mut missions = self.missions.write().await; + if let Some(m) = missions.iter_mut().find(|m| m.id == id) { + m.status = status; + } + Ok(()) + } + async fn list_all_threads( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .iter() + .filter(|t| t.project_id == project_id) + .cloned() + .collect()) + } + async fn list_all_missions( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + Ok(self + .missions + .read() + .await + .iter() + .filter(|m| m.project_id == project_id) + .cloned() + .collect()) + } + } + + struct MinimalStore; + + #[async_trait::async_trait] + impl Store for MinimalStore { + async fn save_thread(&self, _thread: &Thread) -> Result<(), EngineError> { + Ok(()) + } + async fn load_thread(&self, _id: ThreadId) -> Result, EngineError> { + Ok(None) + } + async fn list_threads( + &self, + _project_id: ProjectId, + _user_id: &str, + ) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn update_thread_state( + &self, + _id: ThreadId, + _state: ThreadState, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn save_step(&self, _step: &Step) -> Result<(), EngineError> { + Ok(()) + } + async fn load_steps(&self, _thread_id: ThreadId) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn append_events(&self, _events: &[ThreadEvent]) -> Result<(), EngineError> { + Ok(()) + } + async fn load_events(&self, _thread_id: ThreadId) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn save_project(&self, _project: &Project) -> Result<(), EngineError> { + Ok(()) + } + async fn load_project(&self, _id: ProjectId) -> Result, EngineError> { + Ok(None) + } + async fn save_memory_doc(&self, _doc: &MemoryDoc) -> Result<(), EngineError> { + Ok(()) + } + async fn load_memory_doc(&self, _id: DocId) -> Result, EngineError> { + Ok(None) + } + async fn list_memory_docs( + &self, + _project_id: ProjectId, + _user_id: &str, + ) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn save_lease(&self, _lease: &CapabilityLease) -> Result<(), EngineError> { + Ok(()) + } + async fn load_active_leases( + &self, + _thread_id: ThreadId, + ) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn revoke_lease(&self, _lease_id: LeaseId, _reason: &str) -> Result<(), EngineError> { + Ok(()) + } + async fn save_mission(&self, _mission: &Mission) -> Result<(), EngineError> { + Ok(()) + } + async fn load_mission(&self, _id: MissionId) -> Result, EngineError> { + Ok(None) + } + async fn list_missions( + &self, + _project_id: ProjectId, + _user_id: &str, + ) -> Result, EngineError> { + Ok(Vec::new()) + } + async fn update_mission_status( + &self, + _id: MissionId, + _status: MissionStatus, + ) -> Result<(), EngineError> { + Ok(()) + } + } + + #[tokio::test] + async fn store_defaults_fail_closed() { + let store = MinimalStore; + assert!(matches!( + store.list_projects("alice").await, + Err(EngineError::Store { .. }) + )); + assert!(matches!( + store.list_all_projects().await, + Err(EngineError::Store { .. }) + )); + assert!(matches!( + store.load_conversation(ConversationId::new()).await, + Err(EngineError::Store { .. }) + )); + assert!(matches!( + store.list_all_threads(ProjectId::new()).await, + Err(EngineError::Store { .. }) + )); + } + + #[tokio::test] + async fn shared_queries_include_legacy_and_current_shared_owner() { + use crate::types::memory::DocType; + use crate::types::{LEGACY_SHARED_OWNER_ID, shared_owner_id}; + + let project_id = ProjectId::new(); + let mut legacy = MemoryDoc::new( + project_id, + LEGACY_SHARED_OWNER_ID, + DocType::Note, + "legacy", + "a", + ); + let current = MemoryDoc::new(project_id, shared_owner_id(), DocType::Note, "current", "b"); + legacy.id = DocId::new(); + let store = InMemoryStore::with_docs(vec![legacy, current]); + + let docs = store + .list_memory_docs_with_shared(project_id, "alice") + .await + .unwrap(); + assert_eq!(docs.len(), 2); + } +} diff --git a/crates/ironclaw_engine/src/memory/mod.rs b/crates/ironclaw_engine/src/memory/mod.rs new file mode 100644 index 00000000000..dc3d7267a79 --- /dev/null +++ b/crates/ironclaw_engine/src/memory/mod.rs @@ -0,0 +1,12 @@ +//! Memory document system. +//! +//! - [`MemoryStore`] — project-scoped document CRUD +//! - [`RetrievalEngine`] — context building from project docs via keyword search + +pub mod retrieval; +pub mod skill_tracker; +pub mod store; + +pub use retrieval::RetrievalEngine; +pub use skill_tracker::SkillTracker; +pub use store::MemoryStore; diff --git a/crates/ironclaw_engine/src/memory/retrieval.rs b/crates/ironclaw_engine/src/memory/retrieval.rs new file mode 100644 index 00000000000..22f8e34dba1 --- /dev/null +++ b/crates/ironclaw_engine/src/memory/retrieval.rs @@ -0,0 +1,345 @@ +//! Context retrieval engine. +//! +//! Builds context for thread steps by retrieving relevant memory docs +//! from the project. Uses keyword matching against doc title + content, +//! with priority scoring by doc type (Lessons and Specs rank higher +//! than Summaries for context injection). + +use std::sync::Arc; + +use crate::traits::store::Store; +use crate::types::error::EngineError; +use crate::types::memory::{DocType, MemoryDoc}; +use crate::types::project::ProjectId; + +/// Retrieves relevant memory docs for a thread's context. +pub struct RetrievalEngine { + store: Arc, +} + +impl RetrievalEngine { + pub fn new(store: Arc) -> Self { + Self { store } + } + + /// Retrieve relevant memory docs for the given query within a project. + /// + /// Loads all docs for the project, scores them by keyword relevance and + /// doc-type priority, and returns the top `max_docs` results. + pub async fn retrieve_context( + &self, + project_id: ProjectId, + user_id: &str, + query: &str, + max_docs: usize, + ) -> Result, EngineError> { + if max_docs == 0 { + return Ok(Vec::new()); + } + + // Include both user-owned and shared system docs for context retrieval. + let all_docs = self + .store + .list_memory_docs_with_shared(project_id, user_id) + .await?; + if all_docs.is_empty() { + return Ok(Vec::new()); + } + + let keywords = extract_keywords(query); + if keywords.is_empty() { + // No meaningful keywords — return by doc-type priority alone + let mut scored: Vec<(f64, MemoryDoc)> = all_docs + .into_iter() + .map(|doc| (doc_type_weight(doc.doc_type), doc)) + .collect(); + scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); + scored.truncate(max_docs); + return Ok(scored.into_iter().map(|(_, doc)| doc).collect()); + } + + let mut scored: Vec<(f64, MemoryDoc)> = all_docs + .into_iter() + .map(|doc| { + let keyword_score = keyword_match_score(&doc, &keywords); + let type_weight = doc_type_weight(doc.doc_type); + // Combined score: keyword relevance (0.0-1.0) + type priority bonus + let score = keyword_score + type_weight; + (score, doc) + }) + .filter(|(score, _)| *score > 0.0) + .collect(); + + scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); + scored.truncate(max_docs); + Ok(scored.into_iter().map(|(_, doc)| doc).collect()) + } +} + +/// Extract lowercase keywords from a query, filtering out stop words. +fn extract_keywords(query: &str) -> Vec { + const STOP_WORDS: &[&str] = &[ + "a", "an", "the", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", + "do", "does", "did", "will", "would", "could", "should", "may", "might", "shall", "can", + "to", "of", "in", "for", "on", "with", "at", "by", "from", "as", "into", "about", "it", + "its", "this", "that", "these", "those", "i", "you", "he", "she", "we", "they", "what", + "which", "who", "how", "when", "where", "why", "and", "or", "but", "not", "no", "if", + "then", "so", "up", "out", "just", + ]; + + query + .split(|c: char| !c.is_alphanumeric() && c != '_' && c != '-') + .map(|w| w.to_lowercase()) + .filter(|w| w.len() >= 2 && !STOP_WORDS.contains(&w.as_str())) + .collect() +} + +/// Score how well a doc matches the given keywords (0.0 to 1.0). +fn keyword_match_score(doc: &MemoryDoc, keywords: &[String]) -> f64 { + if keywords.is_empty() { + return 0.0; + } + + let title_lower = doc.title.to_lowercase(); + let content_lower = doc.content.to_lowercase(); + + let mut matched = 0usize; + for kw in keywords { + // Title matches are worth more + if title_lower.contains(kw.as_str()) { + matched += 2; + } else if content_lower.contains(kw.as_str()) { + matched += 1; + } + } + + // Normalize: max possible score is keywords.len() * 2 (all in title) + let max_score = keywords.len() * 2; + matched as f64 / max_score as f64 +} + +/// Priority weight by doc type. Higher = more useful for context injection. +fn doc_type_weight(doc_type: DocType) -> f64 { + match doc_type { + DocType::Spec => 0.5, // Missing capability info is highest priority + DocType::Skill => 0.45, // Skills with activation metadata and code snippets + DocType::Lesson => 0.4, // Lessons prevent repeating mistakes + DocType::Issue => 0.2, // Known problems + DocType::Summary => 0.1, // Background context + DocType::Note => 0.05, // Scratch notes, lowest priority + DocType::Plan => 0.3, // Execution plans with structured steps + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::project::ProjectId; + + fn make_store(docs: Vec) -> Arc { + Arc::new(crate::tests::InMemoryStore::with_docs(docs)) + } + + #[test] + fn extract_keywords_filters_stop_words() { + let kws = extract_keywords("what is the latest news about Iran war"); + assert!(kws.contains(&"latest".to_string())); + assert!(kws.contains(&"news".to_string())); + assert!(kws.contains(&"iran".to_string())); + assert!(kws.contains(&"war".to_string())); + assert!(!kws.contains(&"the".to_string())); + assert!(!kws.contains(&"is".to_string())); + } + + #[test] + fn extract_keywords_handles_special_chars() { + let kws = extract_keywords("web_search web-fetch tool"); + assert!(kws.contains(&"web_search".to_string())); + assert!(kws.contains(&"web-fetch".to_string())); + assert!(kws.contains(&"tool".to_string())); + } + + #[test] + fn keyword_match_title_beats_content() { + use crate::types::project::ProjectId; + + let doc = MemoryDoc::new( + ProjectId::new(), + "test-user", + DocType::Lesson, + "Lesson about web_search errors", + "The tool was not found during execution.", + ); + + let keywords = vec!["web_search".to_string()]; + let score = keyword_match_score(&doc, &keywords); + // Title match = 2/2 = 1.0 + assert!((score - 1.0).abs() < f64::EPSILON); + + let keywords2 = vec!["execution".to_string()]; + let score2 = keyword_match_score(&doc, &keywords2); + // Content-only match = 1/2 = 0.5 + assert!((score2 - 0.5).abs() < f64::EPSILON); + } + + #[test] + fn doc_type_weight_ordering() { + assert!(doc_type_weight(DocType::Spec) > doc_type_weight(DocType::Lesson)); + assert!(doc_type_weight(DocType::Lesson) > doc_type_weight(DocType::Issue)); + assert!(doc_type_weight(DocType::Issue) > doc_type_weight(DocType::Summary)); + assert!(doc_type_weight(DocType::Summary) > doc_type_weight(DocType::Note)); + } + + #[tokio::test] + async fn retrieve_returns_relevant_docs_by_keyword() { + let project = ProjectId::new(); + let store = make_store(vec![ + MemoryDoc::new( + project, + "test-user", + DocType::Lesson, + "web_search tool alias", + "Use web_search", + ), + MemoryDoc::new( + project, + "test-user", + DocType::Summary, + "weather query", + "Fetched weather data", + ), + MemoryDoc::new( + project, + "test-user", + DocType::Issue, + "API timeout", + "External API timed out", + ), + ]); + let engine = RetrievalEngine::new(store); + + let docs = engine + .retrieve_context(project, "test-user", "web_search error", 5) + .await + .unwrap(); + assert!(!docs.is_empty()); + // The lesson about web_search should rank first (keyword + type weight) + assert_eq!(docs[0].doc_type, DocType::Lesson); + assert!(docs[0].title.contains("web_search")); + } + + #[tokio::test] + async fn retrieve_respects_project_scoping() { + let project_a = ProjectId::new(); + let project_b = ProjectId::new(); + let store = make_store(vec![ + MemoryDoc::new( + project_a, + "test-user", + DocType::Lesson, + "Lesson for project A", + "Some lesson", + ), + MemoryDoc::new( + project_b, + "test-user", + DocType::Lesson, + "Lesson for project B", + "Other lesson", + ), + ]); + let engine = RetrievalEngine::new(store); + + let docs_a = engine + .retrieve_context(project_a, "test-user", "lesson", 5) + .await + .unwrap(); + assert_eq!(docs_a.len(), 1); + assert!(docs_a[0].title.contains("project A")); + + let docs_b = engine + .retrieve_context(project_b, "test-user", "lesson", 5) + .await + .unwrap(); + assert_eq!(docs_b.len(), 1); + assert!(docs_b[0].title.contains("project B")); + } + + #[tokio::test] + async fn retrieve_respects_max_docs_limit() { + let project = ProjectId::new(); + let store = make_store(vec![ + MemoryDoc::new( + project, + "test-user", + DocType::Lesson, + "Lesson 1", + "Content 1", + ), + MemoryDoc::new( + project, + "test-user", + DocType::Lesson, + "Lesson 2", + "Content 2", + ), + MemoryDoc::new( + project, + "test-user", + DocType::Lesson, + "Lesson 3", + "Content 3", + ), + ]); + let engine = RetrievalEngine::new(store); + + let docs = engine + .retrieve_context(project, "test-user", "lesson", 2) + .await + .unwrap(); + assert_eq!(docs.len(), 2); + } + + #[tokio::test] + async fn retrieve_empty_store_returns_empty() { + let project = ProjectId::new(); + let store = make_store(vec![]); + let engine = RetrievalEngine::new(store); + + let docs = engine + .retrieve_context(project, "test-user", "anything", 5) + .await + .unwrap(); + assert!(docs.is_empty()); + } + + #[tokio::test] + async fn retrieve_spec_ranks_above_summary() { + let project = ProjectId::new(); + let store = make_store(vec![ + MemoryDoc::new( + project, + "test-user", + DocType::Summary, + "Summary of search", + "searched the web", + ), + MemoryDoc::new( + project, + "test-user", + DocType::Spec, + "Missing search tool", + "Use web_search for the search tool", + ), + ]); + let engine = RetrievalEngine::new(store); + + let docs = engine + .retrieve_context(project, "test-user", "search", 5) + .await + .unwrap(); + assert_eq!(docs.len(), 2); + // Spec should rank first due to higher type weight + assert_eq!(docs[0].doc_type, DocType::Spec); + } +} diff --git a/crates/ironclaw_engine/src/memory/skill_tracker.rs b/crates/ironclaw_engine/src/memory/skill_tracker.rs new file mode 100644 index 00000000000..c3d7f57e3e8 --- /dev/null +++ b/crates/ironclaw_engine/src/memory/skill_tracker.rs @@ -0,0 +1,389 @@ +//! Skill confidence tracking. +//! +//! Tracks usage and success/failure metrics for auto-extracted skills. +//! After each thread completes, the active skills' metrics are updated +//! based on whether the thread succeeded or failed. + +use std::sync::Arc; + +use ironclaw_skills::v2::{SkillRevision, V2SkillMetadata}; +use sha2::{Digest, Sha256}; + +use crate::traits::store::Store; +use crate::types::error::EngineError; +use crate::types::memory::{DocId, DocType, MemoryDoc}; + +/// Tracks skill usage and updates confidence metrics. +pub struct SkillTracker { + store: Arc, +} + +fn compute_content_hash(content: &str) -> String { + let mut hasher = Sha256::new(); + hasher.update(content.as_bytes()); + format!( + "sha256:{}", + hasher + .finalize() + .iter() + .map(|b| format!("{:02x}", b)) + .collect::() + ) +} + +impl SkillTracker { + pub fn new(store: Arc) -> Self { + Self { store } + } + + /// Record that a skill was used in a completed thread. + /// + /// Loads the skill's MemoryDoc, updates metrics in the metadata JSON, + /// and saves it back. Returns `Err(EngineError::Skill)` if the doc is + /// missing, not a Skill, or has invalid metadata — callers decide whether + /// to propagate or log-and-swallow. + pub async fn record_usage(&self, doc_id: DocId, success: bool) -> Result<(), EngineError> { + let doc = self + .store + .load_memory_doc(doc_id) + .await? + .ok_or_else(|| EngineError::Skill { + reason: format!("skill doc not found: {}", doc_id.0), + })?; + + if doc.doc_type != DocType::Skill { + return Err(EngineError::Skill { + reason: format!("doc {} is not a skill (type: {:?})", doc_id.0, doc.doc_type), + }); + } + + let mut meta: V2SkillMetadata = + serde_json::from_value(doc.metadata.clone()).map_err(|e| EngineError::Skill { + reason: format!("invalid skill metadata for {}: {e}", doc_id.0), + })?; + + meta.metrics.usage_count += 1; + if success { + meta.metrics.success_count += 1; + } else { + meta.metrics.failure_count += 1; + } + meta.metrics.last_used = Some(chrono::Utc::now()); + + let updated_doc = MemoryDoc { + metadata: serde_json::to_value(&meta).map_err(|e| EngineError::Skill { + reason: format!("failed to serialize skill metadata: {e}"), + })?, + updated_at: chrono::Utc::now(), + ..doc + }; + + self.store.save_memory_doc(&updated_doc).await + } + + /// Update a skill's content and increment its version. + /// + /// Sets `parent_version` to the current version before incrementing, + /// enabling rollback if the update causes issues. + pub async fn update_skill( + &self, + doc_id: DocId, + new_content: String, + expected_version: Option, + updater: impl FnOnce(&mut V2SkillMetadata), + ) -> Result<(), EngineError> { + let doc = self + .store + .load_memory_doc(doc_id) + .await? + .ok_or_else(|| EngineError::Skill { + reason: format!("skill doc not found: {}", doc_id.0), + })?; + + let mut meta: V2SkillMetadata = + serde_json::from_value(doc.metadata.clone()).map_err(|e| EngineError::Skill { + reason: format!("invalid skill metadata: {e}"), + })?; + + if let Some(expected) = expected_version + && meta.version != expected + { + return Err(EngineError::Skill { + reason: format!( + "skill {} version conflict: expected {expected}, found {}", + doc_id.0, meta.version + ), + }); + } + + // Always recompute from actual content — meta.content_hash may have + // drifted if the doc was updated outside this tracker (e.g. direct + // memory_write). + let archived_hash = compute_content_hash(&doc.content); + meta.revisions.push(SkillRevision { + version: meta.version, + content: doc.content.clone(), + description: meta.description.clone(), + activation: meta.activation.clone(), + code_snippets: meta.code_snippets.clone(), + content_hash: archived_hash, + archived_at: Some(chrono::Utc::now()), + }); + // Cap in-memory revisions at 10 to bound metadata size on every + // load_memory_doc. This is a pragmatic trade-off: full prompt + // snapshots embedded in the skill JSON can grow to many KB per + // revision. Older revisions are dropped; if long-term retention is + // needed, they should be externalized to separate MemoryDocs. + if meta.revisions.len() > 10 { + let keep_from = meta.revisions.len() - 10; + meta.revisions.drain(0..keep_from); + } + meta.parent_version = Some(meta.version); + meta.version += 1; + updater(&mut meta); + meta.content_hash = compute_content_hash(&new_content); + + let updated_doc = MemoryDoc { + content: new_content, + metadata: serde_json::to_value(&meta).map_err(|e| EngineError::Skill { + reason: format!("failed to serialize skill metadata: {e}"), + })?, + updated_at: chrono::Utc::now(), + ..doc + }; + + self.store.save_memory_doc(&updated_doc).await + } + + /// Rollback a skill to its previous version. + /// + /// If an archived revision exists for `parent_version`, restores the full + /// content and metadata snapshot. Otherwise falls back to a simple version + /// decrement without content restoration for older skills. + pub async fn rollback_skill(&self, doc_id: DocId) -> Result<(), EngineError> { + let doc = self + .store + .load_memory_doc(doc_id) + .await? + .ok_or_else(|| EngineError::Skill { + reason: format!("skill doc not found: {}", doc_id.0), + })?; + + let mut meta: V2SkillMetadata = + serde_json::from_value(doc.metadata.clone()).map_err(|e| EngineError::Skill { + reason: format!("invalid skill metadata: {e}"), + })?; + + let parent = meta.parent_version.ok_or_else(|| EngineError::Skill { + reason: format!("skill {} has no parent version to rollback to", doc_id.0), + })?; + + let revision_opt = meta + .revisions + .iter() + .position(|revision| revision.version == parent); + + let rolled_content = if let Some(revision_index) = revision_opt { + let revision = meta.revisions[revision_index].clone(); + meta.version = revision.version; + meta.description = revision.description; + meta.activation = revision.activation; + meta.code_snippets = revision.code_snippets; + meta.content_hash = revision.content_hash; + meta.revisions + .retain(|archived| archived.version < revision.version); + meta.repairs + .retain(|repair| repair.to_version <= revision.version); + meta.parent_version = meta.revisions.iter().map(|archived| archived.version).max(); + revision.content + } else { + meta.version = parent; + meta.parent_version = None; + doc.content.clone() + }; + + let updated_doc = MemoryDoc { + content: rolled_content, + metadata: serde_json::to_value(&meta).map_err(|e| EngineError::Skill { + reason: format!("failed to serialize skill metadata: {e}"), + })?, + updated_at: chrono::Utc::now(), + ..doc + }; + + self.store.save_memory_doc(&updated_doc).await + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::project::ProjectId; + use ironclaw_skills::SkillTrust; + use ironclaw_skills::v2::{SkillMetrics, V2SkillSource}; + + fn make_skill_doc(project_id: ProjectId) -> MemoryDoc { + let meta = V2SkillMetadata { + name: "test-skill".to_string(), + version: 1, + description: "test".to_string(), + activation: Default::default(), + source: V2SkillSource::Extracted, + trust: SkillTrust::Trusted, + code_snippets: vec![], + metrics: SkillMetrics { + usage_count: 5, + success_count: 3, + failure_count: 2, + last_used: None, + }, + parent_version: None, + revisions: vec![], + repairs: vec![], + content_hash: String::new(), + }; + + let mut doc = MemoryDoc::new( + project_id, + "test-user", + DocType::Skill, + "skill:test", + "Test skill prompt", + ); + doc.metadata = serde_json::to_value(&meta).unwrap(); + doc + } + + #[tokio::test] + async fn test_record_usage_success() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store.clone()); + + tracker.record_usage(doc_id, true).await.unwrap(); + + let updated = store.load_memory_doc(doc_id).await.unwrap().unwrap(); + let meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(meta.metrics.usage_count, 6); + assert_eq!(meta.metrics.success_count, 4); + assert_eq!(meta.metrics.failure_count, 2); + assert!(meta.metrics.last_used.is_some()); + } + + #[tokio::test] + async fn test_record_usage_failure() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store.clone()); + + tracker.record_usage(doc_id, false).await.unwrap(); + + let updated = store.load_memory_doc(doc_id).await.unwrap().unwrap(); + let meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(meta.metrics.usage_count, 6); + assert_eq!(meta.metrics.success_count, 3); + assert_eq!(meta.metrics.failure_count, 3); + } + + #[tokio::test] + async fn test_update_skill_increments_version() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store.clone()); + + tracker + .update_skill(doc_id, "Updated content".to_string(), None, |meta| { + meta.description = "Updated description".to_string(); + }) + .await + .unwrap(); + + let updated = store.load_memory_doc(doc_id).await.unwrap().unwrap(); + assert_eq!(updated.content, "Updated content"); + + let meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(meta.version, 2); + assert_eq!(meta.parent_version, Some(1)); + assert_eq!(meta.description, "Updated description"); + assert_eq!(meta.revisions.len(), 1); + assert_eq!(meta.revisions[0].version, 1); + } + + #[tokio::test] + async fn test_rollback_restores_parent_version() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store.clone()); + + // First update to version 2 + tracker + .update_skill(doc_id, "v2 content".to_string(), None, |_| {}) + .await + .unwrap(); + + // Now rollback + tracker.rollback_skill(doc_id).await.unwrap(); + + let rolled = store.load_memory_doc(doc_id).await.unwrap().unwrap(); + let meta: V2SkillMetadata = serde_json::from_value(rolled.metadata).unwrap(); + assert_eq!(meta.version, 1); + assert_eq!(meta.parent_version, None); + assert_eq!(rolled.content, "Test skill prompt"); + assert!(meta.revisions.is_empty()); + } + + #[tokio::test] + async fn test_rollback_without_parent_fails() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store); + + let result = tracker.rollback_skill(doc_id).await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_record_usage_missing_doc() { + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![])); + let tracker = SkillTracker::new(store); + + let result = tracker.record_usage(DocId::new(), true).await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_update_skill_version_conflict() { + let project_id = ProjectId::new(); + let doc = make_skill_doc(project_id); + let doc_id = doc.id; + + let store = Arc::new(crate::tests::InMemoryStore::with_docs(vec![doc])); + let tracker = SkillTracker::new(store); + + let result = tracker + .update_skill(doc_id, "Updated content".to_string(), Some(2), |_| {}) + .await; + + assert!(result.is_err()); + let error = result.unwrap_err().to_string(); + assert!( + error.contains("version conflict"), + "expected version conflict error, got: {error}" + ); + } +} diff --git a/crates/ironclaw_engine/src/memory/store.rs b/crates/ironclaw_engine/src/memory/store.rs new file mode 100644 index 00000000000..942faa7019a --- /dev/null +++ b/crates/ironclaw_engine/src/memory/store.rs @@ -0,0 +1,226 @@ +//! Project-scoped memory document operations. + +use std::sync::Arc; + +use crate::traits::store::Store; +use crate::types::error::EngineError; +use crate::types::memory::{DocId, DocType, MemoryDoc}; +use crate::types::project::ProjectId; +use crate::types::thread::ThreadId; + +/// Thin wrapper over the [`Store`] trait for project-scoped doc operations. +pub struct MemoryStore { + store: Arc, +} + +impl MemoryStore { + pub fn new(store: Arc) -> Self { + Self { store } + } + + /// Create a new memory document. + pub async fn create_doc( + &self, + project_id: ProjectId, + user_id: &str, + doc_type: DocType, + title: &str, + content: &str, + ) -> Result { + let doc = MemoryDoc::new(project_id, user_id, doc_type, title, content); + self.store.save_memory_doc(&doc).await?; + Ok(doc) + } + + /// Create a doc linked to a source thread. + pub async fn create_doc_from_thread( + &self, + project_id: ProjectId, + user_id: &str, + doc_type: DocType, + title: &str, + content: &str, + source_thread_id: ThreadId, + ) -> Result { + let doc = MemoryDoc::new(project_id, user_id, doc_type, title, content) + .with_source_thread(source_thread_id); + self.store.save_memory_doc(&doc).await?; + Ok(doc) + } + + /// Load a single doc by ID. + pub async fn get_doc(&self, id: DocId) -> Result, EngineError> { + self.store.load_memory_doc(id).await + } + + /// List all docs in a project, optionally filtered by type. + pub async fn list_docs( + &self, + project_id: ProjectId, + user_id: &str, + doc_type: Option, + ) -> Result, EngineError> { + let all = self.store.list_memory_docs(project_id, user_id).await?; + match doc_type { + Some(dt) => Ok(all.into_iter().filter(|d| d.doc_type == dt).collect()), + None => Ok(all), + } + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use crate::types::memory::{DocId, DocType}; + use crate::types::project::ProjectId; + use crate::types::thread::ThreadId; + + use super::MemoryStore; + + fn make_store() -> MemoryStore { + MemoryStore::new(Arc::new(crate::tests::InMemoryStore::new())) + } + + // ── Tests ──────────────────────────────────────────────── + + #[tokio::test] + async fn create_doc_and_get() { + let store = make_store(); + let project_id = ProjectId::new(); + + let doc = store + .create_doc( + project_id, + "test-user", + DocType::Summary, + "Test Doc", + "Some content", + ) + .await + .unwrap(); + + assert_eq!(doc.title, "Test Doc"); + assert_eq!(doc.content, "Some content"); + assert_eq!(doc.doc_type, DocType::Summary); + assert_eq!(doc.project_id, project_id); + assert!(doc.source_thread_id.is_none()); + + let loaded = store.get_doc(doc.id).await.unwrap(); + let loaded = loaded.unwrap(); + assert_eq!(loaded.id, doc.id); + assert_eq!(loaded.title, "Test Doc"); + assert_eq!(loaded.content, "Some content"); + } + + #[tokio::test] + async fn create_doc_from_thread_links_source() { + let store = make_store(); + let project_id = ProjectId::new(); + let thread_id = ThreadId::new(); + + let doc = store + .create_doc_from_thread( + project_id, + "test-user", + DocType::Lesson, + "Thread Lesson", + "Learned something", + thread_id, + ) + .await + .unwrap(); + + assert_eq!(doc.source_thread_id, Some(thread_id)); + assert_eq!(doc.doc_type, DocType::Lesson); + + let loaded = store.get_doc(doc.id).await.unwrap().unwrap(); + assert_eq!(loaded.source_thread_id, Some(thread_id)); + } + + #[tokio::test] + async fn list_docs_by_project() { + let store = make_store(); + let project_a = ProjectId::new(); + let project_b = ProjectId::new(); + + store + .create_doc(project_a, "test-user", DocType::Note, "A1", "content a1") + .await + .unwrap(); + store + .create_doc(project_a, "test-user", DocType::Note, "A2", "content a2") + .await + .unwrap(); + store + .create_doc(project_b, "test-user", DocType::Note, "B1", "content b1") + .await + .unwrap(); + + let docs_a = store.list_docs(project_a, "test-user", None).await.unwrap(); + assert_eq!(docs_a.len(), 2); + assert!(docs_a.iter().all(|d| d.project_id == project_a)); + + let docs_b = store.list_docs(project_b, "test-user", None).await.unwrap(); + assert_eq!(docs_b.len(), 1); + assert_eq!(docs_b[0].title, "B1"); + } + + #[tokio::test] + async fn list_docs_filters_by_type() { + let store = make_store(); + let project_id = ProjectId::new(); + + store + .create_doc( + project_id, + "test-user", + DocType::Summary, + "S1", + "summary content", + ) + .await + .unwrap(); + store + .create_doc( + project_id, + "test-user", + DocType::Lesson, + "L1", + "lesson content", + ) + .await + .unwrap(); + store + .create_doc( + project_id, + "test-user", + DocType::Summary, + "S2", + "another summary", + ) + .await + .unwrap(); + + let summaries = store + .list_docs(project_id, "test-user", Some(DocType::Summary)) + .await + .unwrap(); + assert_eq!(summaries.len(), 2); + assert!(summaries.iter().all(|d| d.doc_type == DocType::Summary)); + + let lessons = store + .list_docs(project_id, "test-user", Some(DocType::Lesson)) + .await + .unwrap(); + assert_eq!(lessons.len(), 1); + assert_eq!(lessons[0].title, "L1"); + } + + #[tokio::test] + async fn get_nonexistent_returns_none() { + let store = make_store(); + let result = store.get_doc(DocId::new()).await.unwrap(); + assert!(result.is_none()); + } +} diff --git a/crates/ironclaw_engine/src/reliability.rs b/crates/ironclaw_engine/src/reliability.rs new file mode 100644 index 00000000000..7449a17d0e9 --- /dev/null +++ b/crates/ironclaw_engine/src/reliability.rs @@ -0,0 +1,194 @@ +//! Tool reliability tracking with exponential moving averages. +//! +//! Tracks per-action success rate and latency using EMA (exponential moving +//! average) to smooth out noise. This data can be injected into the context +//! builder to inform the LLM about unreliable tools. + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; + +use tokio::sync::RwLock; + +/// EMA smoothing factor. Higher = more weight on recent observations. +const EMA_ALPHA: f64 = 0.3; + +/// Per-action reliability metrics. +#[derive(Debug, Clone)] +pub struct ActionMetrics { + /// EMA of success rate (0.0 to 1.0). + pub success_rate: f64, + /// EMA of latency in milliseconds. + pub avg_latency_ms: f64, + /// Total number of calls recorded. + pub call_count: u64, + /// Last error message (if any). + pub last_error: Option, +} + +impl Default for ActionMetrics { + fn default() -> Self { + Self { + success_rate: 1.0, // assume success until proven otherwise + avg_latency_ms: 0.0, + call_count: 0, + last_error: None, + } + } +} + +/// Thread-safe registry of per-action reliability metrics. +#[derive(Clone)] +pub struct ReliabilityTracker { + metrics: Arc>>, +} + +impl ReliabilityTracker { + pub fn new() -> Self { + Self { + metrics: Arc::new(RwLock::new(HashMap::new())), + } + } + + /// Record a successful action execution. + pub async fn record_success(&self, action_name: &str, latency: Duration) { + let mut metrics = self.metrics.write().await; + let entry = metrics.entry(action_name.to_string()).or_default(); + entry.call_count += 1; + let latency_ms = latency.as_millis() as f64; + + if entry.call_count == 1 { + // First observation — use raw values + entry.avg_latency_ms = latency_ms; + // success_rate stays at 1.0 + } else { + entry.success_rate = ema(entry.success_rate, 1.0); + entry.avg_latency_ms = ema(entry.avg_latency_ms, latency_ms); + } + } + + /// Record a failed action execution. + pub async fn record_failure(&self, action_name: &str, error: &str) { + let mut metrics = self.metrics.write().await; + let entry = metrics.entry(action_name.to_string()).or_default(); + entry.call_count += 1; + entry.last_error = Some(error.to_string()); + + if entry.call_count == 1 { + entry.success_rate = 0.0; + } else { + entry.success_rate = ema(entry.success_rate, 0.0); + } + } + + /// Get metrics for a specific action. + pub async fn get_metrics(&self, action_name: &str) -> Option { + let metrics = self.metrics.read().await; + metrics.get(action_name).cloned() + } + + /// Get all metrics, sorted by success rate (worst first). + pub async fn all_metrics(&self) -> Vec<(String, ActionMetrics)> { + let metrics = self.metrics.read().await; + let mut entries: Vec<(String, ActionMetrics)> = metrics + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + entries.sort_by(|a, b| { + a.1.success_rate + .partial_cmp(&b.1.success_rate) + .unwrap_or(std::cmp::Ordering::Equal) + }); + entries + } + + /// Get actions with reliability below a threshold. + pub async fn unreliable_actions(&self, threshold: f64) -> Vec<(String, ActionMetrics)> { + let all = self.all_metrics().await; + all.into_iter() + .filter(|(_, m)| m.success_rate < threshold) + .collect() + } +} + +impl Default for ReliabilityTracker { + fn default() -> Self { + Self::new() + } +} + +/// Compute exponential moving average. +fn ema(prev: f64, new: f64) -> f64 { + EMA_ALPHA * new + (1.0 - EMA_ALPHA) * prev +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ema_moves_toward_new() { + let result = ema(1.0, 0.0); + // 0.3 * 0.0 + 0.7 * 1.0 = 0.7 + assert!((result - 0.7).abs() < f64::EPSILON); + } + + #[test] + fn ema_converges_on_repeated() { + let mut val = 1.0; + for _ in 0..20 { + val = ema(val, 0.0); + } + // Should converge toward 0.0 + assert!(val < 0.01); + } + + #[tokio::test] + async fn track_success() { + let tracker = ReliabilityTracker::new(); + tracker + .record_success("tool_a", Duration::from_millis(100)) + .await; + tracker + .record_success("tool_a", Duration::from_millis(200)) + .await; + + let m = tracker.get_metrics("tool_a").await.unwrap(); + assert_eq!(m.call_count, 2); + assert!((m.success_rate - 1.0).abs() < f64::EPSILON); + assert!(m.avg_latency_ms > 100.0); // EMA of 100 and 200 + } + + #[tokio::test] + async fn track_failure_lowers_success_rate() { + let tracker = ReliabilityTracker::new(); + tracker + .record_success("tool_b", Duration::from_millis(50)) + .await; + tracker.record_failure("tool_b", "not found").await; + + let m = tracker.get_metrics("tool_b").await.unwrap(); + assert_eq!(m.call_count, 2); + assert!(m.success_rate < 1.0); + assert_eq!(m.last_error, Some("not found".into())); + } + + #[tokio::test] + async fn unreliable_actions_filters() { + let tracker = ReliabilityTracker::new(); + tracker + .record_success("good_tool", Duration::from_millis(10)) + .await; + tracker.record_failure("bad_tool", "always fails").await; + + let unreliable = tracker.unreliable_actions(0.5).await; + assert_eq!(unreliable.len(), 1); + assert_eq!(unreliable[0].0, "bad_tool"); + } + + #[tokio::test] + async fn unknown_action_returns_none() { + let tracker = ReliabilityTracker::new(); + assert!(tracker.get_metrics("nonexistent").await.is_none()); + } +} diff --git a/crates/ironclaw_engine/src/runtime/conversation.rs b/crates/ironclaw_engine/src/runtime/conversation.rs new file mode 100644 index 00000000000..0c4b1291342 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/conversation.rs @@ -0,0 +1,1209 @@ +//! Conversation manager — routes UI messages to threads. +//! +//! The ConversationManager is the bridge between channel I/O (user messages, +//! status updates) and the thread execution model. It maintains conversation +//! surfaces and decides whether to spawn new threads or inject messages into +//! existing ones. + +use std::collections::HashMap; +use std::sync::Arc; + +use tokio::sync::{Mutex, RwLock}; +use tracing::debug; + +use crate::runtime::manager::ThreadManager; +use crate::runtime::messaging::ThreadOutcome; +use crate::traits::store::Store; +use crate::types::conversation::{ConversationEntry, ConversationId, ConversationSurface}; +use crate::types::error::EngineError; +use crate::types::message::ThreadMessage; +use crate::types::project::ProjectId; +use crate::types::thread::{ThreadConfig, ThreadId, ThreadState, ThreadType}; + +#[derive(Clone, Copy)] +enum ActiveForeground { + Running(ThreadId), + Resumable(ThreadId), +} + +/// Manages conversation surfaces and routes messages to threads. +/// +/// Each channel message arrives here. The manager decides whether to: +/// 1. Spawn a new foreground thread for the message +/// 2. Inject the message into an existing active thread +/// 3. Create a new conversation if none exists for this channel+user +/// +/// ## Locking strategy +/// +/// `conversations` is a *directory*: the global `RwLock` is held only for +/// HashMap lookups/inserts and is never held across an `.await`. Each +/// `ConversationSurface` is wrapped in a `tokio::sync::Mutex` so concurrent +/// messages to *different* conversations run fully in parallel. +/// +/// **Lock ordering invariant:** NEVER hold the global `RwLock` and a +/// per-conversation `Mutex` simultaneously. `get_conversation_lock()` enforces +/// this — it drops the read guard before returning the `Arc>`. +pub struct ConversationManager { + thread_manager: Arc, + store: Arc, + // LOCK ORDER: when acquiring both write locks, always take `conversations` before + // `channel_user_index`. Reversing this order will deadlock under concurrent access. + conversations: RwLock>>>, + /// Maps (channel, user_id) → conversation ID for lookup. + channel_user_index: RwLock>, +} + +impl ConversationManager { + pub fn new(thread_manager: Arc, store: Arc) -> Self { + Self { + thread_manager, + store, + conversations: RwLock::new(HashMap::new()), + channel_user_index: RwLock::new(HashMap::new()), + } + } + + /// Get the per-conversation lock. Holds the global RwLock only briefly + /// (HashMap lookup), then releases it. Returns Err if the conversation + /// does not exist. + async fn get_conversation_lock( + &self, + conversation_id: ConversationId, + ) -> Result>, EngineError> { + let map = self.conversations.read().await; + map.get(&conversation_id) + .map(Arc::clone) + .ok_or_else(|| EngineError::Store { + reason: format!("conversation {conversation_id} not found"), + }) + } // RwLockReadGuard dropped here + + /// Restore persisted conversations for a user into the in-memory index. + pub async fn bootstrap_user(&self, user_id: &str) -> Result { + let conversations = self.store.list_conversations(user_id).await?; + let mut convs = self.conversations.write().await; + let mut index = self.channel_user_index.write().await; + let mut inserted = 0usize; + + for conversation in conversations { + if convs.contains_key(&conversation.id) { + // Still upsert the index — it may be missing if a prior + // get_or_create_conversation inserted the conv but then rolled + // back the index entry on a failed save_conversation. + index + .entry((conversation.channel.clone(), conversation.user_id.clone())) + .or_insert(conversation.id); + continue; + } + index.insert( + (conversation.channel.clone(), conversation.user_id.clone()), + conversation.id, + ); + convs.insert(conversation.id, Arc::new(Mutex::new(conversation))); + inserted += 1; + } + + Ok(inserted) + } + + /// Get or create a conversation for a channel+user pair. + pub async fn get_or_create_conversation( + &self, + channel: &str, + user_id: &str, + ) -> Result { + // Check index first + let key = (channel.to_string(), user_id.to_string()); + { + let index = self.channel_user_index.read().await; + if let Some(conv_id) = index.get(&key) { + return Ok(*conv_id); + } + } + + // Check persisted conversations for this user/channel. + if let Some(conv) = self + .store + .list_conversations(user_id) + .await? + .into_iter() + .find(|conv| conv.channel == channel) + { + let conv_id = conv.id; + let mut convs = self.conversations.write().await; + let mut index = self.channel_user_index.write().await; + // Double-check: another task may have inserted while we did I/O. + if let Some(existing_id) = index.get(&key) { + return Ok(*existing_id); + } + convs.insert(conv_id, Arc::new(Mutex::new(conv))); + index.insert(key, conv_id); + return Ok(conv_id); + } + + // Create new conversation. + let conv = ConversationSurface::new(channel, user_id); + let conv_id = conv.id; + + { + let mut convs = self.conversations.write().await; + let mut index = self.channel_user_index.write().await; + // Double-check: another task may have inserted while we did I/O. + if let Some(existing_id) = index.get(&key) { + return Ok(*existing_id); + } + convs.insert(conv_id, Arc::new(Mutex::new(conv.clone()))); + index.insert(key.clone(), conv_id); + } // write locks released before the async save + + if let Err(e) = self.store.save_conversation(&conv).await { + // Known limitation: a concurrent caller that observed the new conv_id via the + // double-check fast path (between our insert and this rollback) will hold a + // now-deleted, never-persisted ConversationId. This race requires simultaneous + // first-time logins from the same user+channel AND a store write failure — it + // is unlikely in practice and accepted as a structural trade-off of optimistic + // in-memory caching with async persistence. The alternative (holding write + // locks across the async save) would re-introduce cross-tenant serialization. + // Roll back the in-memory insertion so the next caller does not + // receive an unpersisted ConversationId. + let mut convs = self.conversations.write().await; + let mut index = self.channel_user_index.write().await; + convs.remove(&conv_id); + index.remove(&key); + return Err(EngineError::Store { + reason: e.to_string(), + }); + } + + debug!(conversation_id = %conv_id, channel, user_id, "created conversation"); + Ok(conv_id) + } + + /// Handle an incoming user message. + /// + /// If the conversation has an active foreground thread, the message is + /// injected into it. Otherwise, a new foreground thread is spawned. + /// + /// Returns the thread ID that is handling the message. + /// + /// The per-conversation `Mutex` is held for the entire operation — from + /// the active-thread check through `save_conversation`. This eliminates + /// the TOCTOU double-spawn window present in the old 5-phase split. + pub async fn handle_user_message( + &self, + conversation_id: ConversationId, + content: &str, + project_id: ProjectId, + user_id: &str, + thread_config: ThreadConfig, + user_timezone: Option<&str>, + ) -> Result { + let conv_arc = self.get_conversation_lock(conversation_id).await?; + let mut conv = conv_arc.lock().await; + + // Tenant isolation: verify the requesting user owns this conversation. + if conv.user_id != user_id { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("conversation {conversation_id}"), + }); + } + + // Snapshot what find_active_foreground needs before the async calls. + // NOTE: do NOT add the user entry yet — it will be added after the thread + // operation succeeds to avoid orphaned entries if the async op fails. + let active_thread_ids = conv.active_threads.clone(); + let channel_name = conv.channel.clone(); + + // Async I/O to find the active foreground thread — allowed here because + // we hold a tokio::sync::Mutex (not std::sync::Mutex). + let active_foreground = self.find_active_foreground(&active_thread_ids).await; + + let thread_id = match active_foreground { + Some(ActiveForeground::Running(thread_id)) => { + debug!( + conversation_id = %conversation_id, + thread_id = %thread_id, + "injecting message into active thread" + ); + // Known limitation: a tz change mid-turn (user travels between + // messages of the same active thread) is not propagated. The + // running ExecutionLoop holds an in-memory copy of the Thread + // and cannot be updated externally without a new signal type. + // Updating the persisted record here would not affect the live + // step. Rare in practice; defer to a follow-up if needed. + self.thread_manager + .inject_message(thread_id, user_id, ThreadMessage::user(content)) + .await?; + thread_id + } + Some(ActiveForeground::Resumable(thread_id)) => { + debug!( + conversation_id = %conversation_id, + thread_id = %thread_id, + "resuming suspended foreground thread" + ); + // Resume reloads the thread from the store, so writing fresh + // user_timezone to the persisted record before resume_thread + // means the resumed execution sees the up-to-date value — but + // only if this write actually lands. A store failure here + // would silently leave the resumed thread with the prior + // timezone, so log explicitly rather than swallowing. + if let Some(tz) = user_timezone + && let Err(e) = self + .thread_manager + .set_thread_metadata(thread_id, "user_timezone", tz) + .await + { + debug!( + thread_id = %thread_id, + error = %e, + "failed to refresh user_timezone on resume; thread will use previous value" + ); + } + self.thread_manager + .resume_thread( + thread_id, + user_id, + Some(ThreadMessage::user(content)), + None, + None, + ) + .await?; + thread_id + } + None => { + // Build conversation history from prior entries for context continuity. + // Clone here (None branch only) — inject/resume paths don't need history, + // so deferring avoids an O(entries) allocation on those fast paths. + let history = build_history_from_entries(&conv.entries); + + // Build initial thread metadata. Must be applied *before* the + // executor's background task starts — `set_thread_metadata` + // only updates the persisted record, not the in-memory Thread + // the loop is reading from, so the first step would otherwise + // miss `user_timezone` / `source_channel`. The bridge router + // validates the timezone string before passing it in here. + // The orchestrator reads `source_channel` on the very first + // step to populate `ThreadExecutionContext.source_channel`, + // which `mission_create` consults to default `notify_channels`. + let base_channel = channel_name + .split(':') + .next() + .unwrap_or(&channel_name) + .to_string(); + let mut initial_metadata = serde_json::Map::new(); + initial_metadata.insert( + "source_channel".into(), + serde_json::Value::String(base_channel), + ); + if let Some(tz) = user_timezone { + initial_metadata.insert( + "user_timezone".into(), + serde_json::Value::String(tz.to_string()), + ); + } + + // Spawn new foreground thread with conversation history. + self.thread_manager + .spawn_thread_with_history( + content, // use message as goal + ThreadType::Foreground, + project_id, + thread_config, + None, + user_id, + history, + initial_metadata, + ) + .await? + } + }; + + // Final in-memory mutations under the already-held per-conv Mutex. + // The user entry is added here — after the thread operation succeeded — to + // prevent orphaned entries if inject_message/resume_thread/spawn_thread_with_history + // returned an error above. + conv.add_entry(ConversationEntry::user(content)); + match active_foreground { + Some(ActiveForeground::Running(_)) => { + // No additional in-memory mutation needed beyond the user entry above. + } + Some(ActiveForeground::Resumable(_)) => { + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + "Thread resumed", + )); + } + None => { + conv.track_thread(thread_id); + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + "Thread started", + )); + debug!( + conversation_id = %conversation_id, + thread_id = %thread_id, + "spawned new foreground thread" + ); + } + } + + // Persist outside the global RwLock (per-conv Mutex is still held). + self.store.save_conversation(&conv).await?; + + Ok(thread_id) + } + + /// Record a thread's outcome in its conversation. + pub async fn record_thread_outcome( + &self, + conversation_id: ConversationId, + thread_id: ThreadId, + outcome: &ThreadOutcome, + ) -> Result<(), EngineError> { + let conv_arc = self.get_conversation_lock(conversation_id).await?; + let mut conv = conv_arc.lock().await; + match outcome { + ThreadOutcome::Completed { response } => { + if let Some(text) = response { + conv.add_entry(ConversationEntry::agent(thread_id, text)); + } + conv.untrack_thread(thread_id); + } + ThreadOutcome::Stopped => { + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + "Thread stopped", + )); + conv.untrack_thread(thread_id); + } + ThreadOutcome::MaxIterations => { + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + "Thread reached max iterations", + )); + conv.untrack_thread(thread_id); + } + ThreadOutcome::Failed { error } => { + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + format!("Thread failed: {error}"), + )); + conv.untrack_thread(thread_id); + } + ThreadOutcome::GatePaused { + gate_name, + action_name, + .. + } => { + conv.add_entry(ConversationEntry::system_for_thread( + thread_id, + format!("Gate '{gate_name}' paused execution of action: {action_name}"), + )); + // Thread stays active — waiting for gate resolution + } + } + // Known limitation: if save_conversation fails, the in-memory mutations (add_entry, + // untrack_thread) are already applied but not persisted. Memory and DB diverge until + // the next successful save. Rolling back would require snapshotting the prior state, + // which is not implemented here — accepted as a low-probability failure mode. + self.store.save_conversation(&conv).await?; + Ok(()) + } + + /// Append an agent message to a conversation that originated *outside* + /// the conversation's own thread tree (e.g. a mission's notification + /// thread). The entry is recorded as an `Agent` entry tagged with the + /// originating `thread_id`, so subsequent foreground messages will see + /// it in their conversation history via `build_history_from_entries`. + /// + /// Tenant isolation: rejects calls whose `user_id` does not own the + /// conversation, mirroring `handle_user_message`. + pub async fn record_external_agent_message( + &self, + conversation_id: ConversationId, + thread_id: ThreadId, + user_id: &str, + content: impl Into, + ) -> Result<(), EngineError> { + let conv_arc = self.get_conversation_lock(conversation_id).await?; + let mut conv = conv_arc.lock().await; + if conv.user_id != user_id { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("conversation {conversation_id}"), + }); + } + conv.add_entry(ConversationEntry::agent(thread_id, content)); + self.store.save_conversation(&conv).await?; + Ok(()) + } + + /// Clear a conversation's entries and active threads. + /// + /// Stops tracking all threads and removes conversation history so the next + /// user message spawns a fresh thread with no prior context. + pub async fn clear_conversation( + &self, + conversation_id: ConversationId, + user_id: &str, + ) -> Result<(), EngineError> { + let conv_arc = self.get_conversation_lock(conversation_id).await?; + let mut conv = conv_arc.lock().await; + // Tenant isolation: verify ownership. + if conv.user_id != user_id { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("conversation {conversation_id}"), + }); + } + conv.active_threads.clear(); + conv.entries.clear(); + conv.updated_at = chrono::Utc::now(); + self.store.save_conversation(&conv).await?; + debug!(conversation_id = %conversation_id, "cleared conversation"); + Ok(()) + } + + /// Get a snapshot of a conversation. + pub async fn get_conversation( + &self, + conversation_id: ConversationId, + ) -> Option { + let arc = { + let convs = self.conversations.read().await; + convs.get(&conversation_id).map(Arc::clone) + }?; + Some(arc.lock().await.clone()) + } + + /// Returns conversations for the given user. + /// + /// Uses `channel_user_index` to pre-filter by user before acquiring any + /// per-conversation locks, keeping lock scope minimal. This is a best-effort + /// snapshot: each conversation is locked and read individually, so concurrent + /// mutations between locks may be partially visible. + pub async fn list_conversations(&self, user_id: &str) -> Vec { + let arcs: Vec>> = { + let convs = self.conversations.read().await; + let index = self.channel_user_index.read().await; + index + .iter() + .filter(|((_, uid), _)| uid == user_id) + .filter_map(|(_, id)| convs.get(id).cloned()) + .collect() + }; + let mut result = Vec::with_capacity(arcs.len()); + for arc in arcs { + result.push(arc.lock().await.clone()); + } + result + } + + /// Find an active foreground thread given a snapshot of active thread IDs. + /// + /// Accepts a plain slice rather than a `&ConversationSurface` so callers + /// can drop the conversations write lock before invoking this method — + /// it performs async I/O (is_running, load_thread) that must not be held + /// under any lock. + async fn find_active_foreground( + &self, + active_thread_ids: &[ThreadId], + ) -> Option { + for &tid in active_thread_ids { + if self.thread_manager.is_running(tid).await { + return Some(ActiveForeground::Running(tid)); + } + if let Ok(Some(thread)) = self.store.load_thread(tid).await + && thread.thread_type == ThreadType::Foreground + && thread.state == ThreadState::Suspended + { + return Some(ActiveForeground::Resumable(tid)); + } + } + None + } + + /// Test helper: track a thread in a conversation without accessing the + /// internal HashMap directly. + #[cfg(test)] + pub async fn track_thread_in_conversation(&self, conv_id: ConversationId, thread_id: ThreadId) { + let arc = self + .get_conversation_lock(conv_id) + .await + .expect("conversation exists in test"); + arc.lock().await.track_thread(thread_id); + } +} + +/// Build ThreadMessage history from conversation entries. +/// +/// Converts user and agent entries into ThreadMessages so a new thread +/// inherits context from prior turns in the same conversation. +/// +/// The caller passes a snapshot taken *before* the current user message was +/// appended, so all entries here are prior-turn history — include them all. +/// System entries (thread lifecycle notifications) are skipped as they are not +/// useful LLM context. +fn build_history_from_entries( + entries: &[ConversationEntry], +) -> Vec { + use crate::types::conversation::EntrySender; + + entries + .iter() + .filter_map(|entry| match &entry.sender { + EntrySender::User => Some(crate::types::message::ThreadMessage::user(&entry.content)), + EntrySender::Agent { .. } => Some(crate::types::message::ThreadMessage::assistant( + &entry.content, + )), + EntrySender::System => None, // skip system notifications + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::capability::lease::LeaseManager; + use crate::capability::policy::PolicyEngine; + use crate::capability::registry::CapabilityRegistry; + use crate::traits::effect::EffectExecutor; + use crate::traits::llm::{LlmBackend, LlmCallConfig, LlmOutput}; + use crate::traits::store::Store; + use crate::types::capability::{ActionDef, CapabilityLease}; + use crate::types::conversation::{ConversationId, ConversationSurface, EntrySender}; + use crate::types::event::ThreadEvent; + use crate::types::memory::{DocId, MemoryDoc}; + use crate::types::message::MessageRole; + use crate::types::project::Project; + use crate::types::step::{ActionResult, LlmResponse, Step, TokenUsage}; + use crate::types::thread::ThreadState; + use std::sync::Mutex; + use std::time::Duration; + + // ── Mocks (same as manager tests) ─────────────────────── + + struct MockLlm(Mutex>); + + #[async_trait::async_trait] + impl LlmBackend for MockLlm { + async fn complete( + &self, + _: &[ThreadMessage], + _: &[ActionDef], + _: &LlmCallConfig, + ) -> Result { + let mut r = self.0.lock().unwrap(); + if r.is_empty() { + Ok(LlmOutput { + response: LlmResponse::Text("done".into()), + usage: TokenUsage::default(), + }) + } else { + Ok(r.remove(0)) + } + } + fn model_name(&self) -> &str { + "mock" + } + } + + struct MockEffects; + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _: &str, + _: serde_json::Value, + _: &CapabilityLease, + _: &crate::traits::effect::ThreadExecutionContext, + ) -> Result { + Ok(ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + }) + } + async fn available_actions( + &self, + _: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(vec![]) + } + } + + struct MockStore { + conversations: RwLock>, + threads: RwLock>, + } + + impl MockStore { + fn new() -> Self { + Self { + conversations: RwLock::new(HashMap::new()), + threads: RwLock::new(HashMap::new()), + } + } + } + + #[async_trait::async_trait] + impl Store for MockStore { + async fn save_thread( + &self, + thread: &crate::types::thread::Thread, + ) -> Result<(), EngineError> { + self.threads.write().await.insert(thread.id, thread.clone()); + Ok(()) + } + async fn load_thread( + &self, + id: ThreadId, + ) -> Result, EngineError> { + Ok(self.threads.read().await.get(&id).cloned()) + } + async fn list_threads( + &self, + project_id: ProjectId, + _user_id: &str, + ) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .values() + .filter(|thread| thread.project_id == project_id) + .cloned() + .collect()) + } + async fn update_thread_state( + &self, + _: ThreadId, + _: ThreadState, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn save_step(&self, _: &Step) -> Result<(), EngineError> { + Ok(()) + } + async fn load_steps(&self, _: ThreadId) -> Result, EngineError> { + Ok(vec![]) + } + async fn append_events(&self, _: &[ThreadEvent]) -> Result<(), EngineError> { + Ok(()) + } + async fn load_events(&self, _: ThreadId) -> Result, EngineError> { + Ok(vec![]) + } + async fn save_project(&self, _: &Project) -> Result<(), EngineError> { + Ok(()) + } + async fn load_project(&self, _: ProjectId) -> Result, EngineError> { + Ok(None) + } + async fn save_conversation( + &self, + conversation: &ConversationSurface, + ) -> Result<(), EngineError> { + self.conversations + .write() + .await + .insert(conversation.id, conversation.clone()); + Ok(()) + } + async fn load_conversation( + &self, + id: ConversationId, + ) -> Result, EngineError> { + Ok(self.conversations.read().await.get(&id).cloned()) + } + async fn list_conversations( + &self, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .conversations + .read() + .await + .values() + .filter(|conversation| conversation.user_id == user_id) + .cloned() + .collect()) + } + async fn save_memory_doc(&self, _: &MemoryDoc) -> Result<(), EngineError> { + Ok(()) + } + async fn load_memory_doc(&self, _: DocId) -> Result, EngineError> { + Ok(None) + } + async fn list_memory_docs( + &self, + _: ProjectId, + _: &str, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn save_lease(&self, _: &CapabilityLease) -> Result<(), EngineError> { + Ok(()) + } + async fn load_active_leases( + &self, + _: ThreadId, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn revoke_lease( + &self, + _: crate::types::capability::LeaseId, + _: &str, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn save_mission( + &self, + _: &crate::types::mission::Mission, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn load_mission( + &self, + _: crate::types::mission::MissionId, + ) -> Result, EngineError> { + Ok(None) + } + async fn list_missions( + &self, + _: ProjectId, + _: &str, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn update_mission_status( + &self, + _: crate::types::mission::MissionId, + _: crate::types::mission::MissionStatus, + ) -> Result<(), EngineError> { + Ok(()) + } + } + + fn make_conv_manager() -> (Arc, ConversationManager) { + let store = Arc::new(MockStore::new()); + let tm = Arc::new(ThreadManager::new( + Arc::new(MockLlm(Mutex::new(vec![LlmOutput { + response: LlmResponse::Text("Hello!".into()), + usage: TokenUsage::default(), + }]))), + Arc::new(MockEffects), + store.clone(), + Arc::new(CapabilityRegistry::new()), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + )); + let cm = ConversationManager::new(Arc::clone(&tm), store); + (tm, cm) + } + + // ── Tests ─────────────────────────────────────────────── + + #[tokio::test] + async fn get_or_create_conversation() { + let (_, cm) = make_conv_manager(); + let c1 = cm + .get_or_create_conversation("telegram", "user1") + .await + .unwrap(); + let c2 = cm + .get_or_create_conversation("telegram", "user1") + .await + .unwrap(); + assert_eq!(c1, c2); // same channel+user returns same conversation + + let c3 = cm + .get_or_create_conversation("slack", "user1") + .await + .unwrap(); + assert_ne!(c1, c3); // different channel → different conversation + } + + #[tokio::test] + async fn handle_message_spawns_thread() { + let (tm, cm) = make_conv_manager(); + let conv_id = cm.get_or_create_conversation("web", "user1").await.unwrap(); + let project = ProjectId::new(); + + let tid = cm + .handle_user_message( + conv_id, + "Hello", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + .unwrap(); + + // Thread was spawned + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert!(conv.active_threads.contains(&tid)); + assert_eq!(conv.entries.len(), 2); // user message + "Thread started" + + // Wait for thread to complete + let outcome = tm.join_thread(tid).await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { .. })); + } + + #[tokio::test] + async fn handle_message_resumes_suspended_thread() { + let store = Arc::new(MockStore::new()); + let tm = Arc::new(ThreadManager::new( + Arc::new(MockLlm(Mutex::new(vec![LlmOutput { + response: LlmResponse::Text("Recovered".into()), + usage: TokenUsage::default(), + }]))), + Arc::new(MockEffects), + store.clone(), + Arc::new(CapabilityRegistry::new()), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + )); + let cm = ConversationManager::new(Arc::clone(&tm), store.clone()); + + let conv_id = cm.get_or_create_conversation("web", "user1").await.unwrap(); + let project = ProjectId::new(); + let mut thread = crate::types::thread::Thread::new( + "resume", + ThreadType::Foreground, + project, + "user1", + ThreadConfig::default(), + ); + thread.transition_to(ThreadState::Running, None).unwrap(); + thread.add_message(ThreadMessage::user("earlier")); + thread.step_count = 1; + thread.metadata = serde_json::json!({ + "runtime_checkpoint": { + "persisted_state": {"last_return": 7}, + "nudge_count": 0, + "consecutive_errors": 0, + "compaction_count": 0 + } + }); + thread + .transition_to( + ThreadState::Suspended, + Some("engine restart; resumable from checkpoint".into()), + ) + .unwrap(); + store.save_thread(&thread).await.unwrap(); + + cm.track_thread_in_conversation(conv_id, thread.id).await; + + let resumed = cm + .handle_user_message( + conv_id, + "continue from there", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + .unwrap(); + + assert_eq!(resumed, thread.id); + let outcome = tm.join_thread(thread.id).await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { .. })); + } + + #[tokio::test] + async fn record_outcome_adds_entry() { + let (_, cm) = make_conv_manager(); + let conv_id = cm.get_or_create_conversation("cli", "user1").await.unwrap(); + let tid = ThreadId::new(); + + // Manually track a thread + cm.track_thread_in_conversation(conv_id, tid).await; + + // Record completion + cm.record_thread_outcome( + conv_id, + tid, + &ThreadOutcome::Completed { + response: Some("Done!".into()), + }, + ) + .await + .unwrap(); + + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert!(conv.active_threads.is_empty()); + assert_eq!(conv.entries.len(), 1); + assert_eq!(conv.entries[0].content, "Done!"); + + // Check sender is agent + assert!(matches!( + conv.entries[0].sender, + EntrySender::Agent { thread_id } if thread_id == tid + )); + } + + #[tokio::test] + async fn list_conversations_filters_by_user() { + let (_, cm) = make_conv_manager(); + cm.get_or_create_conversation("web", "alice").await.unwrap(); + cm.get_or_create_conversation("telegram", "alice") + .await + .unwrap(); + cm.get_or_create_conversation("web", "bob").await.unwrap(); + + let alice_convs = cm.list_conversations("alice").await; + assert_eq!(alice_convs.len(), 2); + + let bob_convs = cm.list_conversations("bob").await; + assert_eq!(bob_convs.len(), 1); + } + + #[tokio::test] + async fn bootstrap_user_loads_persisted_conversations() { + let store = Arc::new(MockStore::new()); + let mut conv = ConversationSurface::new("web", "user1"); + conv.add_entry(ConversationEntry::user("persisted")); + store.save_conversation(&conv).await.unwrap(); + + let tm = Arc::new(ThreadManager::new( + Arc::new(MockLlm(Mutex::new(vec![]))), + Arc::new(MockEffects), + store.clone(), + Arc::new(CapabilityRegistry::new()), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + )); + let cm = ConversationManager::new(tm, store); + + let loaded = cm.bootstrap_user("user1").await.unwrap(); + assert_eq!(loaded, 1); + + let conv_id = cm.get_or_create_conversation("web", "user1").await.unwrap(); + assert_eq!(conv_id, conv.id); + let saved = cm.get_conversation(conv.id).await.unwrap(); + assert_eq!(saved.entries.len(), 1); + assert_eq!(saved.entries[0].content, "persisted"); + } + + #[tokio::test] + async fn clear_conversation_resets_entries_and_threads() { + let (tm, cm) = make_conv_manager(); + let conv_id = cm.get_or_create_conversation("web", "user1").await.unwrap(); + let project = ProjectId::new(); + + // Spawn a thread so the conversation has entries and active threads + let tid = cm + .handle_user_message( + conv_id, + "Hello", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + .unwrap(); + + // Wait for thread to finish + let _ = tm.join_thread(tid).await.unwrap(); + + // Record outcome so there's an agent entry + cm.record_thread_outcome( + conv_id, + tid, + &ThreadOutcome::Completed { + response: Some("Hi there".into()), + }, + ) + .await + .unwrap(); + + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert!(!conv.entries.is_empty()); + + // Clear the conversation + cm.clear_conversation(conv_id, "user1").await.unwrap(); + + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert!(conv.entries.is_empty()); + assert!(conv.active_threads.is_empty()); + } + + #[tokio::test] + async fn concurrent_handle_user_message_spawns_one_thread() { + // T1: Two concurrent handle_user_message calls on the same conversation + // must serialize — only ONE new thread should be spawned. + let (_, cm) = make_conv_manager(); + let conv_id = cm.get_or_create_conversation("web", "user1").await.unwrap(); + let project = ProjectId::new(); + let cm = Arc::new(cm); + + let cm1 = Arc::clone(&cm); + let cm2 = Arc::clone(&cm); + + let t1 = tokio::spawn(async move { + cm1.handle_user_message( + conv_id, + "message one", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + }); + let t2 = tokio::spawn(async move { + cm2.handle_user_message( + conv_id, + "message two", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + }); + + let r1 = t1.await.unwrap(); + let r2 = t2.await.unwrap(); + + // Both calls must succeed. + assert!(r1.is_ok(), "first handle_user_message failed: {r1:?}"); + assert!(r2.is_ok(), "second handle_user_message failed: {r2:?}"); + + // The per-conv Mutex serializes the two calls. The second call sees the + // first thread as Running (or the same thread ID if inject_message is used), + // so at most one NEW thread should exist in active_threads. + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert_eq!( + conv.active_threads.len(), + 1, + "expected exactly 1 active thread, got {}: {:?}", + conv.active_threads.len(), + conv.active_threads + ); + } + + #[tokio::test] + async fn record_external_agent_message_appears_in_history() { + // Regression: when a mission's notification is recorded into a + // conversation via `record_external_agent_message`, the next foreground + // user message must spawn a thread whose history includes the mission + // output. Otherwise the agent has no idea the mission ran and replies + // to follow-ups as if no digest was ever delivered. + let (_tm, cm) = make_conv_manager(); + let conv_id = cm + .get_or_create_conversation("gateway", "user1") + .await + .unwrap(); + let mission_thread_id = ThreadId::new(); + let mission_output = "**[daily-news-digest]** - Headline A\n- Headline B"; + + cm.record_external_agent_message( + conv_id, + mission_thread_id, + "user1", + mission_output.to_string(), + ) + .await + .unwrap(); + + // The new entry must be visible on the conversation snapshot. + let conv = cm.get_conversation(conv_id).await.unwrap(); + assert_eq!( + conv.entries.len(), + 1, + "expected exactly one entry after recording mission output" + ); + assert!(matches!( + &conv.entries[0].sender, + EntrySender::Agent { thread_id } if *thread_id == mission_thread_id + )); + assert_eq!(conv.entries[0].content, mission_output); + + // The user's follow-up turn must observe the mission output: a fresh + // foreground thread spawns with the entries-derived history, which now + // contains the mission's assistant entry as prior context. + // `build_history_from_entries` strips the trailing entry (the current + // user message added by the caller), so we exercise the full + // `handle_user_message` path and inspect what the new thread sees. + let project = ProjectId::new(); + let _tid = cm + .handle_user_message( + conv_id, + "Tell me more about the first headline you sent", + project, + "user1", + ThreadConfig::default(), + None, + ) + .await + .unwrap(); + let conv_after = cm.get_conversation(conv_id).await.unwrap(); + let history_after = build_history_from_entries(&conv_after.entries); + assert!( + history_after + .iter() + .any(|m| m.role == MessageRole::Assistant && m.content == mission_output), + "follow-up turn history should contain the mission output: {history_after:#?}" + ); + } + + #[tokio::test] + async fn record_external_agent_message_rejects_wrong_user() { + let (_, cm) = make_conv_manager(); + let conv_id = cm + .get_or_create_conversation("gateway", "owner") + .await + .unwrap(); + + let result = cm + .record_external_agent_message( + conv_id, + ThreadId::new(), + "intruder", + "should be rejected".to_string(), + ) + .await; + + assert!( + matches!(result, Err(EngineError::AccessDenied { .. })), + "expected AccessDenied for cross-tenant write, got: {result:?}" + ); + } + + #[tokio::test] + async fn record_thread_outcome_unknown_conv_returns_err() { + // T4: After C1 fix, record_thread_outcome with an unknown ConversationId + // must return Err, not silently succeed. + let (_, cm) = make_conv_manager(); + let unknown_conv_id = ConversationId::new(); + let tid = ThreadId::new(); + + let result = cm + .record_thread_outcome( + unknown_conv_id, + tid, + &ThreadOutcome::Completed { + response: Some("irrelevant".into()), + }, + ) + .await; + + assert!( + result.is_err(), + "expected Err for unknown conversation, got Ok" + ); + } +} diff --git a/crates/ironclaw_engine/src/runtime/lease_refresh.rs b/crates/ironclaw_engine/src/runtime/lease_refresh.rs new file mode 100644 index 00000000000..a3ffd6421d6 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/lease_refresh.rs @@ -0,0 +1,99 @@ +use std::collections::HashSet; +use std::sync::Arc; + +use crate::Capability; +use crate::capability::lease::LeaseManager; +use crate::capability::planner::LeasePlanner; +use crate::capability::registry::CapabilityRegistry; +use crate::traits::effect::EffectExecutor; +use crate::traits::store::Store; +use crate::types::capability::GrantedActions; +use crate::types::error::EngineError; +use crate::types::thread::Thread; + +pub(crate) async fn reconcile_dynamic_tool_lease( + thread: &mut Thread, + effects: &Arc, + leases: &Arc, + store: Option<&Arc>, + lease_planner: &LeasePlanner, +) -> Result<(), EngineError> { + let active_leases = leases.active_for_thread(thread.id).await; + let actions = effects.available_actions(&active_leases).await?; + if actions.is_empty() { + return Ok(()); + } + + let mut capabilities = CapabilityRegistry::new(); + capabilities.register(Capability { + name: "tools".into(), + description: "Available tools".into(), + actions, + knowledge: vec![], + policies: vec![], + }); + + let Some(grant) = lease_planner + .plan_for_thread(thread.thread_type, &capabilities) + .into_iter() + .find(|grant| grant.capability_name == "tools") + else { + return Ok(()); + }; + + let desired_actions: HashSet = match grant.granted_actions { + GrantedActions::All => return Ok(()), + GrantedActions::Specific(actions) => actions.into_iter().collect(), + }; + + if desired_actions.is_empty() { + return Ok(()); + } + + if let Some(existing) = active_leases + .iter() + .find(|lease| lease.capability_name == "tools") + { + if existing.granted_actions.is_all() { + return Ok(()); + } + + let mut merged: HashSet = + existing.granted_actions.actions().iter().cloned().collect(); + let before = merged.len(); + merged.extend(desired_actions); + if merged.len() == before { + return Ok(()); + } + + let mut merged_actions: Vec = merged.into_iter().collect(); + merged_actions.sort(); + let updated = leases + .update_granted_actions(existing.id, GrantedActions::Specific(merged_actions)) + .await?; + if let Some(store) = store { + store.save_lease(&updated).await?; + } + return Ok(()); + } + + let mut actions: Vec = desired_actions.into_iter().collect(); + actions.sort(); + let lease = leases + .grant( + thread.id, + "tools", + GrantedActions::Specific(actions), + None, + None, + ) + .await?; + if let Some(store) = store { + store.save_lease(&lease).await?; + } + if !thread.capability_leases.contains(&lease.id) { + thread.capability_leases.push(lease.id); + } + + Ok(()) +} diff --git a/crates/ironclaw_engine/src/runtime/manager.rs b/crates/ironclaw_engine/src/runtime/manager.rs new file mode 100644 index 00000000000..5fb7eb4cdb9 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/manager.rs @@ -0,0 +1,1483 @@ +//! Thread manager — top-level orchestrator for thread lifecycle. + +use std::collections::HashMap; +use std::sync::Arc; + +use tokio::sync::RwLock; +use tracing::{debug, error}; + +use crate::capability::lease::LeaseManager; +use crate::capability::planner::LeasePlanner; +use crate::capability::policy::PolicyEngine; +use crate::capability::registry::CapabilityRegistry; +use crate::executor::ExecutionLoop; +use crate::runtime::lease_refresh::reconcile_dynamic_tool_lease; +use crate::runtime::messaging::{self, SignalSender, ThreadOutcome, ThreadSignal}; +use crate::runtime::tree::ThreadTree; +use crate::traits::effect::EffectExecutor; +use crate::traits::llm::LlmBackend; +use crate::traits::store::Store; +use crate::types::error::EngineError; +use crate::types::message::{MessageRole, ThreadMessage}; +use crate::types::project::ProjectId; +use crate::types::thread::{Thread, ThreadConfig, ThreadId, ThreadState, ThreadType}; + +/// Handle to a running thread for checking results. +struct RunningThread { + signal_tx: SignalSender, + handle: tokio::task::JoinHandle>, +} + +/// Top-level orchestrator for thread lifecycle. +/// +/// Manages thread spawning, supervision, signaling, and tree relationships. +pub struct ThreadManager { + llm: Arc, + effects: Arc, + store: Arc, + pub capabilities: Arc, + pub leases: Arc, + pub policy: Arc, + lease_planner: LeasePlanner, + tree: RwLock, + running: Arc>>, + completed: Arc>>, + /// Broadcast channel for thread events (for live status updates). + event_tx: tokio::sync::broadcast::Sender, +} + +impl ThreadManager { + pub fn new( + llm: Arc, + effects: Arc, + store: Arc, + capabilities: Arc, + leases: Arc, + policy: Arc, + ) -> Self { + let (event_tx, _) = tokio::sync::broadcast::channel(256); + Self { + llm, + effects, + store, + capabilities, + leases, + policy, + lease_planner: LeasePlanner::new(), + tree: RwLock::new(ThreadTree::new()), + running: Arc::new(RwLock::new(HashMap::new())), + completed: Arc::new(RwLock::new(HashMap::new())), + event_tx, + } + } + + /// Subscribe to thread events for live status updates. + pub fn subscribe_events( + &self, + ) -> tokio::sync::broadcast::Receiver { + self.event_tx.subscribe() + } + + /// Spawn a new thread and start executing it. + /// + /// Grants default capability leases for all registered capabilities. + /// Returns the thread ID immediately; the thread runs in a background task. + /// + /// `initial_messages` provides conversation history from prior threads + /// (for context continuity across turns in the same conversation). + pub async fn spawn_thread( + &self, + goal: impl Into, + thread_type: ThreadType, + project_id: ProjectId, + config: ThreadConfig, + parent_id: Option, + user_id: impl Into, + ) -> Result { + self.spawn_thread_with_history( + goal, + thread_type, + project_id, + config, + parent_id, + user_id, + Vec::new(), + serde_json::Map::new(), + ) + .await + } + + /// Spawn a thread with initial conversation history. + /// + /// `initial_metadata` is applied to the thread's metadata map *before* the + /// background execution task starts, so the executor's in-memory `Thread` + /// observes those keys on the first step. This is the only correct way to + /// stamp metadata that the very first orchestrator step needs to read + /// (e.g. `source_channel` for `mission_create` notify-channel defaulting, + /// or `user_timezone` for cron resolution). Setting metadata after spawn + /// via `set_thread_metadata` is a race — the spawned task owns its own + /// in-memory copy of the `Thread`, and the late update only lands on the + /// persisted copy that the running task never re-reads. + #[allow(clippy::too_many_arguments)] + pub async fn spawn_thread_with_history( + &self, + goal: impl Into, + thread_type: ThreadType, + project_id: ProjectId, + config: ThreadConfig, + parent_id: Option, + user_id: impl Into, + initial_messages: Vec, + initial_metadata: serde_json::Map, + ) -> Result { + let user_id = user_id.into(); + let mut thread = Thread::new(goal, thread_type, project_id, &user_id, config); + if let Some(pid) = parent_id { + thread = thread.with_parent(pid); + } + let thread_id = thread.id; + + // Apply initial metadata before save_thread + start_thread so the + // executor's in-memory thread observes it on the first step. + if !initial_metadata.is_empty() + && let Some(obj) = thread.metadata.as_object_mut() + { + for (k, v) in initial_metadata { + obj.insert(k, v); + } + } + + // Register in tree + if let Some(pid) = parent_id { + self.tree.write().await.add_child(pid, thread_id); + } + + // Grant explicit capability leases based on thread type. + for grant in self + .lease_planner + .plan_for_thread(thread_type, &self.capabilities) + { + let lease = self + .leases + .grant( + thread_id, + grant.capability_name, + grant.granted_actions, + None, + None, + ) + .await?; + self.store.save_lease(&lease).await?; + thread.capability_leases.push(lease.id); + } + + // Add conversation history from prior threads (for context continuity) + for msg in initial_messages { + thread.messages.push(msg); + } + + // Add the goal as the current user message so the LLM has context + thread.add_message(crate::types::message::ThreadMessage::user(&thread.goal)); + + // Persist + self.store.save_thread(&thread).await?; + + self.start_thread(thread, user_id, false).await + } + + /// Resume a persisted waiting or suspended thread. + pub async fn resume_thread( + &self, + thread_id: ThreadId, + user_id: impl Into, + injected_message: Option, + approval_event: Option<(String, bool)>, + resolved_call_id: Option, + ) -> Result<(), EngineError> { + if self.is_running(thread_id).await { + return Err(EngineError::Thread( + crate::types::error::ThreadError::AlreadyRunning(thread_id), + )); + } + + let mut thread = self + .store + .load_thread(thread_id) + .await? + .ok_or(EngineError::ThreadNotFound(thread_id))?; + + // Tenant isolation: verify the requesting user owns this thread. + let uid: String = user_id.into(); + if !thread.is_owned_by(&uid) { + return Err(EngineError::AccessDenied { + user_id: uid, + entity: format!("thread {thread_id}"), + }); + } + + if !matches!( + thread.state, + crate::types::thread::ThreadState::Waiting + | crate::types::thread::ThreadState::Suspended + ) { + return Err(EngineError::Store { + reason: format!( + "thread {thread_id} is not resumable from {:?}", + thread.state + ), + }); + } + + if let Some((call_id, approved)) = approval_event { + let event = crate::types::event::ThreadEvent::new( + thread_id, + crate::types::event::EventKind::ApprovalReceived { call_id, approved }, + ); + let _ = self.event_tx.send(event.clone()); + thread.events.push(event); + thread.updated_at = chrono::Utc::now(); + } + + if let Some(ref call_id) = resolved_call_id { + let preserve_assistant_call = injected_message.as_ref().is_some_and(|message| { + message.role == MessageRole::ActionResult + && message.action_call_id.as_deref() == Some(call_id.as_str()) + }); + thread.messages.retain(|existing| { + if preserve_assistant_call { + !is_resolved_action_result_message(existing, call_id) + } else { + !is_resolved_call_message(existing, call_id) + } + }); + } + + if let Some(message) = injected_message { + thread.add_internal_message(message.clone()); + thread.add_message(message); + } + + // Waiting threads paused on approval/auth should resume from the + // newly injected context rather than replaying the old checkpointed + // interrupt. Suspended threads keep their checkpoint for restart. + if thread.state == crate::types::thread::ThreadState::Waiting + && let Some(metadata) = thread.metadata.as_object_mut() + { + metadata.remove("runtime_checkpoint"); + } + + self.store.save_thread(&thread).await?; + self.start_thread(thread, uid, true).await?; + Ok(()) + } + + async fn start_thread( + &self, + mut thread: Thread, + user_id: String, + is_resume: bool, + ) -> Result { + let thread_id = thread.id; + + reconcile_dynamic_tool_lease( + &mut thread, + &self.effects, + &self.leases, + Some(&self.store), + &self.lease_planner, + ) + .await?; + + // Create signal channel + let (tx, rx) = messaging::signal_channel(32); + + // Build execution loop + let llm = Arc::clone(&self.llm); + let effects = Arc::clone(&self.effects); + let leases = Arc::clone(&self.leases); + let policy = Arc::clone(&self.policy); + + let store_for_retrieval = Arc::clone(&self.store); + let retrieval = crate::memory::RetrievalEngine::new(store_for_retrieval); + + let exec_loop = ExecutionLoop::new(thread, llm, effects, leases, policy, rx, user_id) + .with_capabilities(Arc::clone(&self.capabilities)) + .with_event_tx(self.event_tx.clone()) + .with_retrieval(retrieval) + .with_store(Arc::clone(&self.store)); + + // Spawn background task + let store_for_task = Arc::clone(&self.store); + let running = Arc::clone(&self.running); + let completed = Arc::clone(&self.completed); + let handle = tokio::spawn(async move { + let mut exec = exec_loop; + let result = exec.run().await; + debug!(thread_id = %thread_id, "thread execution finished"); + + // Run retrospective trace analysis (non-LLM, always runs). + // Issues are picked up by the self-improvement mission via event listener. + let trace = crate::executor::trace::build_trace(&exec.thread); + if !trace.issues.is_empty() { + crate::executor::trace::log_trace_summary(&trace); + } + + // Transition Completed → Done + if exec.thread.state == crate::types::thread::ThreadState::Completed + && let Err(e) = exec + .thread + .transition_to(crate::types::thread::ThreadState::Done, None) + { + tracing::debug!(thread_id = %thread_id, "failed to transition to Done: {e}"); + } + + // Trace recording is handled centrally by `RecordingLlm` in the + // host crate (gated by `IRONCLAW_RECORD_TRACE`). The engine no + // longer writes its own JSON trace file. + + if let Err(e) = store_for_task.append_events(&exec.thread.events).await { + tracing::debug!( + thread_id = %thread_id, + "failed to persist thread events: {e}" + ); + } + + // Save final thread state to store + if let Err(e) = store_for_task.save_thread(&exec.thread).await { + tracing::debug!( + thread_id = %thread_id, + "failed to save final thread state: {e}" + ); + } + + let outcome = match result { + Ok(outcome) => outcome, + Err(error) => ThreadOutcome::Failed { + error: error.to_string(), + }, + }; + completed.write().await.insert(thread_id, outcome.clone()); + running.write().await.remove(&thread_id); + Ok(outcome) + }); + + self.running.write().await.insert( + thread_id, + RunningThread { + signal_tx: tx, + handle, + }, + ); + + if is_resume { + debug!(thread_id = %thread_id, "resumed thread"); + } + + Ok(thread_id) + } + + /// Send a stop signal to a running thread. + pub async fn stop_thread(&self, thread_id: ThreadId, user_id: &str) -> Result<(), EngineError> { + // Validate ownership before allowing stop. + if let Some(thread) = self.store.load_thread(thread_id).await? + && !thread.is_owned_by(user_id) + { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("thread {thread_id}"), + }); + } + let running = self.running.read().await; + if let Some(rt) = running.get(&thread_id) { + let _ = rt.signal_tx.send(ThreadSignal::Stop).await; + Ok(()) + } else { + Err(EngineError::ThreadNotFound(thread_id)) + } + } + + /// Send a stop signal without ownership check (system operations). + pub async fn stop_thread_system(&self, thread_id: ThreadId) -> Result<(), EngineError> { + let running = self.running.read().await; + if let Some(rt) = running.get(&thread_id) { + let _ = rt.signal_tx.send(ThreadSignal::Stop).await; + Ok(()) + } else { + Err(EngineError::ThreadNotFound(thread_id)) + } + } + + /// Inject a user message into a running thread. + pub async fn inject_message( + &self, + thread_id: ThreadId, + user_id: &str, + message: ThreadMessage, + ) -> Result<(), EngineError> { + // Validate ownership before allowing injection. + if let Some(thread) = self.store.load_thread(thread_id).await? + && !thread.is_owned_by(user_id) + { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("thread {thread_id}"), + }); + } + let running = self.running.read().await; + if let Some(rt) = running.get(&thread_id) { + let _ = rt + .signal_tx + .send(ThreadSignal::InjectMessage(message)) + .await; + Ok(()) + } else { + Err(EngineError::ThreadNotFound(thread_id)) + } + } + + /// Inject a message without ownership check (system operations). + pub async fn inject_message_system( + &self, + thread_id: ThreadId, + message: ThreadMessage, + ) -> Result<(), EngineError> { + let running = self.running.read().await; + if let Some(rt) = running.get(&thread_id) { + let _ = rt + .signal_tx + .send(ThreadSignal::InjectMessage(message)) + .await; + Ok(()) + } else { + Err(EngineError::ThreadNotFound(thread_id)) + } + } + + /// Set a metadata key on the persisted thread record. + /// + /// Note: this updates the **store**, not the in-memory `Thread` that an + /// already-running `ExecutionLoop` is reading from. Callers that need the + /// next executor step to observe the new value must apply this *before* + /// the executor task is spawned (initial-create path) or before + /// `resume_thread`, which reloads from the store. + pub async fn set_thread_metadata( + &self, + thread_id: ThreadId, + key: &str, + value: &str, + ) -> Result<(), EngineError> { + let mut thread = self + .store + .load_thread(thread_id) + .await + .map_err(|e| EngineError::Store { + reason: format!("set_thread_metadata: load failed: {e}"), + })? + .ok_or(EngineError::ThreadNotFound(thread_id))?; + if let Some(obj) = thread.metadata.as_object_mut() { + obj.insert( + key.to_string(), + serde_json::Value::String(value.to_string()), + ); + } + self.store + .save_thread(&thread) + .await + .map_err(|e| EngineError::Store { + reason: format!("set_thread_metadata: save failed: {e}"), + })?; + Ok(()) + } + + /// Check if a thread is still running. + pub async fn is_running(&self, thread_id: ThreadId) -> bool { + let running = self.running.read().await; + running + .get(&thread_id) + .is_some_and(|rt| !rt.handle.is_finished()) + } + + /// Wait for a thread to finish and return its outcome. + /// Removes the thread from the running set. + pub async fn join_thread(&self, thread_id: ThreadId) -> Result { + if let Some(outcome) = self.completed.write().await.remove(&thread_id) { + return Ok(outcome); + } + + let rt = { + let mut running = self.running.write().await; + running.remove(&thread_id) + }; + + match rt { + Some(rt) => { + let result = match rt.handle.await { + Ok(result) => result, + Err(e) => { + error!(thread_id = %thread_id, "thread task panicked: {e}"); + Ok(ThreadOutcome::Failed { + error: format!("thread task panicked: {e}"), + }) + } + }; + self.completed.write().await.remove(&thread_id); + result + } + None => Err(EngineError::ThreadNotFound(thread_id)), + } + } + + /// Get children of a thread. + pub async fn children_of(&self, thread_id: ThreadId) -> Vec { + let tree = self.tree.read().await; + tree.children_of(thread_id).to_vec() + } + + /// Get the parent of a thread. + pub async fn parent_of(&self, thread_id: ThreadId) -> Option { + let tree = self.tree.read().await; + tree.parent_of(thread_id) + } + + /// Clean up finished threads from the running set. + pub async fn cleanup_finished(&self) -> Vec { + let mut running = self.running.write().await; + let finished: Vec = running + .iter() + .filter(|(_, rt)| rt.handle.is_finished()) + .map(|(id, _)| *id) + .collect(); + for id in &finished { + running.remove(id); + } + finished + } + + /// Automatically resume checkpointed non-foreground threads. + pub async fn resume_background_threads( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + // System operation: resume all suspended research threads regardless of user. + let threads = self.store.list_all_threads(project_id).await?; + let mut resumed = Vec::new(); + + for thread in threads { + if thread.state != ThreadState::Suspended { + continue; + } + if thread.thread_type != ThreadType::Research { + continue; + } + if thread.metadata.get("runtime_checkpoint").is_none() { + continue; + } + if thread.user_id.is_empty() { + continue; + } + + self.resume_thread(thread.id, thread.user_id.clone(), None, None, None) + .await?; + resumed.push(thread.id); + } + + Ok(resumed) + } + + /// Reconcile persisted non-terminal threads after process startup. + /// + /// The current engine does not support mid-thread replay/resume, so any + /// thread left in a non-terminal state is marked failed-safe. + pub async fn recover_project_threads( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + const PENDING_APPROVAL_METADATA_KEY: &str = "pending_approval"; + const RUNTIME_CHECKPOINT_METADATA_KEY: &str = "runtime_checkpoint"; + // System operation: recover all non-terminal threads regardless of user. + let threads = self.store.list_all_threads(project_id).await?; + let mut recovered = Vec::new(); + + for mut thread in threads { + if thread.state.is_terminal() || thread.state == ThreadState::Completed { + continue; + } + + if thread.state == ThreadState::Waiting + && thread.metadata.get(PENDING_APPROVAL_METADATA_KEY).is_some() + { + continue; + } + + if thread + .metadata + .get(RUNTIME_CHECKPOINT_METADATA_KEY) + .is_some() + && matches!(thread.state, ThreadState::Running | ThreadState::Suspended) + { + if thread.state == ThreadState::Running { + thread.transition_to( + ThreadState::Suspended, + Some("engine restart; resumable from checkpoint".into()), + )?; + } + self.store.append_events(&thread.events).await?; + self.store.save_thread(&thread).await?; + recovered.push(thread.id); + continue; + } + + if thread + .transition_to( + ThreadState::Failed, + Some("engine restart before thread completion".into()), + ) + .is_ok() + { + self.store.append_events(&thread.events).await?; + self.store.save_thread(&thread).await?; + recovered.push(thread.id); + } + } + + Ok(recovered) + } +} + +fn is_resolved_call_message(message: &ThreadMessage, call_id: &str) -> bool { + if message.role == MessageRole::ActionResult + && message.action_call_id.as_deref() == Some(call_id) + { + return true; + } + + message.role == MessageRole::Assistant + && message + .action_calls + .as_ref() + .is_some_and(|calls| calls.iter().any(|call| call.id == call_id)) +} + +fn is_resolved_action_result_message(message: &ThreadMessage, call_id: &str) -> bool { + message.role == MessageRole::ActionResult && message.action_call_id.as_deref() == Some(call_id) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::traits::llm::{LlmCallConfig, LlmOutput}; + use crate::types::capability::{ActionDef, Capability, CapabilityLease, EffectType}; + use crate::types::event::ThreadEvent; + use crate::types::memory::{DocId, MemoryDoc}; + use crate::types::project::Project; + use crate::types::step::{ActionResult, LlmResponse, Step, TokenUsage}; + use crate::types::thread::ThreadState; + use std::sync::Mutex; + use std::time::Duration; + + // ── Mocks ─────────────────────────────────────────────── + + struct MockLlm { + responses: Mutex>, + } + + impl MockLlm { + fn text(msg: &str) -> Arc { + Arc::new(Self { + responses: Mutex::new(vec![LlmOutput { + response: LlmResponse::Text(msg.into()), + usage: TokenUsage::default(), + }]), + }) + } + } + + #[async_trait::async_trait] + impl LlmBackend for MockLlm { + async fn complete( + &self, + _: &[crate::types::message::ThreadMessage], + _: &[ActionDef], + _: &LlmCallConfig, + ) -> Result { + let mut r = self.responses.lock().unwrap(); + if r.is_empty() { + Ok(LlmOutput { + response: LlmResponse::Text("done".into()), + usage: TokenUsage::default(), + }) + } else { + Ok(r.remove(0)) + } + } + + fn model_name(&self) -> &str { + "mock" + } + } + + struct MockEffects; + + struct DynamicEffects { + actions: RwLock>, + calls: RwLock>, + install_reveals: RwLock>>, + } + + impl DynamicEffects { + fn new(actions: Vec) -> Arc { + Arc::new(Self { + actions: RwLock::new(actions), + calls: RwLock::new(Vec::new()), + install_reveals: RwLock::new(None), + }) + } + + async fn set_actions(&self, actions: Vec) { + *self.actions.write().await = actions; + } + + async fn set_install_reveals(&self, actions: Vec) { + *self.install_reveals.write().await = Some(actions); + } + + async fn recorded_calls(&self) -> Vec { + self.calls.read().await.clone() + } + } + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _: &str, + _: serde_json::Value, + _: &CapabilityLease, + _: &crate::traits::effect::ThreadExecutionContext, + ) -> Result { + Ok(ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + }) + } + + async fn available_actions( + &self, + _: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(vec![]) + } + } + + #[async_trait::async_trait] + impl EffectExecutor for DynamicEffects { + async fn execute_action( + &self, + action_name: &str, + _: serde_json::Value, + _: &CapabilityLease, + _: &crate::traits::effect::ThreadExecutionContext, + ) -> Result { + self.calls.write().await.push(action_name.to_string()); + if action_name == "tool_install" + && let Some(actions) = self.install_reveals.read().await.clone() + { + *self.actions.write().await = actions; + } + Ok(ActionResult { + call_id: String::new(), + action_name: action_name.to_string(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + }) + } + + async fn available_actions( + &self, + _: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(self.actions.read().await.clone()) + } + } + + struct MockStore { + threads: RwLock>, + events: RwLock>>, + leases: RwLock>, + } + + impl MockStore { + fn new() -> Self { + Self { + threads: RwLock::new(HashMap::new()), + events: RwLock::new(HashMap::new()), + leases: RwLock::new(HashMap::new()), + } + } + } + + #[async_trait::async_trait] + impl Store for MockStore { + async fn save_thread(&self, thread: &Thread) -> Result<(), EngineError> { + self.threads.write().await.insert(thread.id, thread.clone()); + Ok(()) + } + async fn load_thread(&self, id: ThreadId) -> Result, EngineError> { + Ok(self.threads.read().await.get(&id).cloned()) + } + async fn list_threads( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .values() + .filter(|thread| thread.project_id == project_id && thread.user_id == user_id) + .cloned() + .collect()) + } + async fn list_all_threads( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + Ok(self + .threads + .read() + .await + .values() + .filter(|thread| thread.project_id == project_id) + .cloned() + .collect()) + } + async fn update_thread_state( + &self, + _: ThreadId, + _: ThreadState, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn save_step(&self, _: &Step) -> Result<(), EngineError> { + Ok(()) + } + async fn load_steps(&self, _: ThreadId) -> Result, EngineError> { + Ok(vec![]) + } + async fn append_events(&self, events: &[ThreadEvent]) -> Result<(), EngineError> { + let mut stored = self.events.write().await; + for event in events { + stored + .entry(event.thread_id) + .or_default() + .push(event.clone()); + } + Ok(()) + } + async fn load_events(&self, thread_id: ThreadId) -> Result, EngineError> { + Ok(self + .events + .read() + .await + .get(&thread_id) + .cloned() + .unwrap_or_default()) + } + async fn save_project(&self, _: &Project) -> Result<(), EngineError> { + Ok(()) + } + async fn load_project(&self, _: ProjectId) -> Result, EngineError> { + Ok(None) + } + async fn save_memory_doc(&self, _: &MemoryDoc) -> Result<(), EngineError> { + Ok(()) + } + async fn load_memory_doc(&self, _: DocId) -> Result, EngineError> { + Ok(None) + } + async fn list_memory_docs( + &self, + _: ProjectId, + _: &str, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn save_lease(&self, lease: &CapabilityLease) -> Result<(), EngineError> { + self.leases.write().await.insert(lease.id, lease.clone()); + Ok(()) + } + async fn load_active_leases( + &self, + thread_id: ThreadId, + ) -> Result, EngineError> { + Ok(self + .leases + .read() + .await + .values() + .filter(|lease| lease.thread_id == thread_id && lease.is_valid()) + .cloned() + .collect()) + } + async fn revoke_lease( + &self, + lease_id: crate::types::capability::LeaseId, + _: &str, + ) -> Result<(), EngineError> { + if let Some(lease) = self.leases.write().await.get_mut(&lease_id) { + lease.revoked = true; + } + Ok(()) + } + async fn save_mission( + &self, + _: &crate::types::mission::Mission, + ) -> Result<(), EngineError> { + Ok(()) + } + async fn load_mission( + &self, + _: crate::types::mission::MissionId, + ) -> Result, EngineError> { + Ok(None) + } + async fn list_missions( + &self, + _: ProjectId, + _: &str, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn update_mission_status( + &self, + _: crate::types::mission::MissionId, + _: crate::types::mission::MissionStatus, + ) -> Result<(), EngineError> { + Ok(()) + } + } + + fn make_manager(llm: Arc) -> ThreadManager { + let mut caps = CapabilityRegistry::new(); + caps.register(Capability { + name: "test".into(), + description: "Test capability".into(), + actions: vec![ActionDef { + name: "test_tool".into(), + description: "Test".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + }], + knowledge: vec![], + policies: vec![], + }); + + ThreadManager::new( + llm, + Arc::new(MockEffects), + Arc::new(MockStore::new()), + Arc::new(caps), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + ) + } + + fn make_manager_with_store(llm: Arc, store: Arc) -> ThreadManager { + let mut caps = CapabilityRegistry::new(); + caps.register(Capability { + name: "test".into(), + description: "Test capability".into(), + actions: vec![ActionDef { + name: "test_tool".into(), + description: "Test".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadLocal], + requires_approval: false, + }], + knowledge: vec![], + policies: vec![], + }); + + ThreadManager::new( + llm, + Arc::new(MockEffects), + store, + Arc::new(caps), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + ) + } + + fn make_manager_with_effects( + llm: Arc, + store: Arc, + effects: Arc, + ) -> ThreadManager { + let mut caps = CapabilityRegistry::new(); + caps.register(Capability { + name: "tools".into(), + description: "Tools".into(), + actions: vec![ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }], + knowledge: vec![], + policies: vec![], + }); + + ThreadManager::new( + llm, + effects, + store, + Arc::new(caps), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + ) + } + + // ── Tests ─────────────────────────────────────────────── + + #[tokio::test] + async fn spawn_and_join() { + let mgr = make_manager(MockLlm::text("Hello!")); + let project = ProjectId::new(); + + let tid = mgr + .spawn_thread( + "test", + ThreadType::Foreground, + project, + ThreadConfig::default(), + None, + "user", + ) + .await + .unwrap(); + + let outcome = mgr.join_thread(tid).await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { response: Some(r) } if r == "Hello!")); + } + + #[tokio::test] + async fn resume_reconciles_tool_lease_with_newly_available_actions() { + let store = Arc::new(MockStore::new()); + let effects = DynamicEffects::new(vec![ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }]); + let mgr = make_manager_with_effects(MockLlm::text("done"), store, effects.clone()); + + let thread_id = ThreadId::new(); + let mut thread = Thread::new( + "use notion", + ThreadType::Foreground, + ProjectId::new(), + "user", + ThreadConfig::default(), + ); + thread.id = thread_id; + thread.state = ThreadState::Waiting; + mgr.store.save_thread(&thread).await.unwrap(); + + let lease = mgr + .leases + .grant( + thread_id, + "tools", + crate::types::capability::GrantedActions::Specific(vec!["tool_install".into()]), + None, + None, + ) + .await + .unwrap(); + mgr.store.save_lease(&lease).await.unwrap(); + + effects + .set_actions(vec![ + ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }, + ActionDef { + name: "notion_search".into(), + description: "Search Notion".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadExternal], + requires_approval: false, + }, + ]) + .await; + + mgr.resume_thread(thread_id, "user", None, None, None) + .await + .unwrap(); + let _ = mgr.join_thread(thread_id).await.unwrap(); + + let refreshed = mgr + .leases + .find_lease_for_action(thread_id, "notion_search") + .await; + assert!( + refreshed.is_some(), + "resume should refresh tools lease for newly available actions" + ); + } + + #[tokio::test] + async fn spawn_reconciles_tool_lease_with_stale_capability_snapshot() { + let store = Arc::new(MockStore::new()); + let effects = DynamicEffects::new(vec![ + ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }, + ActionDef { + name: "notion_search".into(), + description: "Search Notion".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadExternal], + requires_approval: false, + }, + ]); + let mgr = make_manager_with_effects(MockLlm::text("done"), store, effects); + + let tid = mgr + .spawn_thread( + "use notion", + ThreadType::Foreground, + ProjectId::new(), + ThreadConfig::default(), + None, + "user", + ) + .await + .unwrap(); + + let lease = mgr.leases.find_lease_for_action(tid, "notion_search").await; + assert!( + lease.is_some(), + "spawn should refresh tools lease for actions exposed after the capability snapshot" + ); + } + + #[tokio::test] + async fn running_thread_can_install_then_use_new_tool_without_user_bounce() { + let store = Arc::new(MockStore::new()); + let initial_actions = vec![ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }]; + let revealed_actions = vec![ + ActionDef { + name: "tool_install".into(), + description: "Install a tool".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::WriteLocal], + requires_approval: false, + }, + ActionDef { + name: "notion_search".into(), + description: "Search Notion".into(), + parameters_schema: serde_json::json!({}), + effects: vec![EffectType::ReadExternal], + requires_approval: false, + }, + ]; + let effects = DynamicEffects::new(initial_actions); + effects.set_install_reveals(revealed_actions).await; + let llm = Arc::new(MockLlm { + responses: Mutex::new(vec![ + LlmOutput { + response: LlmResponse::ActionCalls { + calls: vec![crate::types::step::ActionCall { + id: "call_install".into(), + action_name: "tool_install".into(), + parameters: serde_json::json!({"name": "notion"}), + }], + content: None, + }, + usage: TokenUsage::default(), + }, + LlmOutput { + response: LlmResponse::ActionCalls { + calls: vec![crate::types::step::ActionCall { + id: "call_search".into(), + action_name: "notion_search".into(), + parameters: serde_json::json!({"query": "latest meeting note"}), + }], + content: None, + }, + usage: TokenUsage::default(), + }, + LlmOutput { + response: LlmResponse::Text("done".into()), + usage: TokenUsage::default(), + }, + ]), + }); + let mgr = make_manager_with_effects(llm, store, effects.clone()); + + let tid = mgr + .spawn_thread( + "install notion and get the latest meeting note", + ThreadType::Foreground, + ProjectId::new(), + ThreadConfig::default(), + None, + "user", + ) + .await + .unwrap(); + + let outcome = mgr.join_thread(tid).await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { .. })); + + let calls = effects.recorded_calls().await; + assert_eq!( + calls, + vec!["tool_install".to_string(), "notion_search".to_string()], + "thread should continue from install into the newly exposed tool without pausing for a new user turn" + ); + } + + #[tokio::test] + async fn stop_thread_works() { + // LLM that returns many action responses + let responses: Vec = (0..100) + .map(|i| LlmOutput { + response: LlmResponse::ActionCalls { + calls: vec![crate::types::step::ActionCall { + id: format!("c{i}"), + action_name: "test_tool".into(), + parameters: serde_json::json!({}), + }], + content: None, + }, + usage: TokenUsage::default(), + }) + .collect(); + + let mgr = make_manager(Arc::new(MockLlm { + responses: Mutex::new(responses), + })); + let project = ProjectId::new(); + + let tid = mgr + .spawn_thread( + "test", + ThreadType::Foreground, + project, + ThreadConfig::default(), + None, + "user", + ) + .await + .unwrap(); + + // Give it a moment to start, then stop + tokio::time::sleep(Duration::from_millis(10)).await; + let _ = mgr.stop_thread(tid, "test-user").await; + + let outcome = mgr.join_thread(tid).await.unwrap(); + assert!(matches!( + outcome, + ThreadOutcome::Stopped | ThreadOutcome::Completed { .. } | ThreadOutcome::MaxIterations + )); + } + + #[tokio::test] + async fn parent_child_tree() { + let mgr = make_manager(MockLlm::text("parent done")); + let project = ProjectId::new(); + + let parent = mgr + .spawn_thread( + "parent", + ThreadType::Foreground, + project, + ThreadConfig::default(), + None, + "user", + ) + .await + .unwrap(); + + let child = mgr + .spawn_thread( + "child", + ThreadType::Research, + project, + ThreadConfig::default(), + Some(parent), + "user", + ) + .await + .unwrap(); + + assert_eq!(mgr.parent_of(child).await, Some(parent)); + assert_eq!(mgr.children_of(parent).await, vec![child]); + } + + #[tokio::test] + async fn recover_project_threads_marks_non_terminal_as_failed() { + let store = Arc::new(MockStore::new()); + let project = ProjectId::new(); + + let mut running = Thread::new( + "running", + ThreadType::Foreground, + project, + "test-user", + ThreadConfig::default(), + ); + running.transition_to(ThreadState::Running, None).unwrap(); + store.save_thread(&running).await.unwrap(); + + let mut completed = Thread::new( + "done", + ThreadType::Foreground, + project, + "test-user", + ThreadConfig::default(), + ); + completed + .transition_to(ThreadState::Failed, Some("already terminal".into())) + .unwrap(); + store.save_thread(&completed).await.unwrap(); + + let mgr = make_manager_with_store(MockLlm::text("ignored"), Arc::clone(&store)); + let recovered = mgr.recover_project_threads(project).await.unwrap(); + + assert_eq!(recovered, vec![running.id]); + let saved = store.load_thread(running.id).await.unwrap().unwrap(); + assert_eq!(saved.state, ThreadState::Failed); + let events = store.load_events(running.id).await.unwrap(); + assert!(!events.is_empty()); + } + + #[tokio::test] + async fn recover_project_threads_preserves_waiting_approval_threads() { + let store = Arc::new(MockStore::new()); + let project = ProjectId::new(); + + let mut waiting = Thread::new( + "awaiting approval", + ThreadType::Foreground, + project, + "test-user", + ThreadConfig::default(), + ); + waiting.transition_to(ThreadState::Running, None).unwrap(); + waiting + .transition_to(ThreadState::Waiting, Some("approval".into())) + .unwrap(); + waiting.metadata = serde_json::json!({ + "pending_approval": { + "request_id": "req-1", + "action_name": "shell", + "call_id": "call-1" + } + }); + store.save_thread(&waiting).await.unwrap(); + + let mgr = make_manager_with_store(MockLlm::text("ignored"), Arc::clone(&store)); + let recovered = mgr.recover_project_threads(project).await.unwrap(); + + assert!(recovered.is_empty()); + let saved = store.load_thread(waiting.id).await.unwrap().unwrap(); + assert_eq!(saved.state, ThreadState::Waiting); + } + + #[tokio::test] + async fn recover_project_threads_suspends_checkpointed_threads() { + let store = Arc::new(MockStore::new()); + let project = ProjectId::new(); + + let mut running = Thread::new( + "resume me", + ThreadType::Foreground, + project, + "test-user", + ThreadConfig::default(), + ); + running.transition_to(ThreadState::Running, None).unwrap(); + running.metadata = serde_json::json!({ + "runtime_checkpoint": { + "persisted_state": {"last_return": 7}, + "nudge_count": 0, + "consecutive_errors": 0, + "compaction_count": 0 + } + }); + store.save_thread(&running).await.unwrap(); + + let mgr = make_manager_with_store(MockLlm::text("ignored"), Arc::clone(&store)); + let recovered = mgr.recover_project_threads(project).await.unwrap(); + + assert_eq!(recovered, vec![running.id]); + let saved = store.load_thread(running.id).await.unwrap().unwrap(); + assert_eq!(saved.state, ThreadState::Suspended); + } + + #[tokio::test] + async fn resume_background_threads_restarts_suspended_research_threads() { + let store = Arc::new(MockStore::new()); + let project = ProjectId::new(); + + let mut research = Thread::new( + "background research", + ThreadType::Research, + project, + "test-user", + ThreadConfig::default(), + ); + research.transition_to(ThreadState::Running, None).unwrap(); + research.metadata = serde_json::json!({ + "user_id": "owner", + "runtime_checkpoint": { + "persisted_state": {}, + "nudge_count": 0, + "consecutive_errors": 0, + "compaction_count": 0 + } + }); + research + .transition_to( + ThreadState::Suspended, + Some("engine restart; resumable from checkpoint".into()), + ) + .unwrap(); + store.save_thread(&research).await.unwrap(); + + let mgr = make_manager_with_store(MockLlm::text("done"), Arc::clone(&store)); + let resumed = mgr.resume_background_threads(project).await.unwrap(); + assert_eq!(resumed, vec![research.id]); + + let outcome = mgr.join_thread(research.id).await.unwrap(); + assert!(matches!(outcome, ThreadOutcome::Completed { .. })); + } + + // Skill selection and injection tests are in tests/engine_v2_skill_codeact.rs + // (skill selection happens in the Python orchestrator, not in Rust). +} diff --git a/crates/ironclaw_engine/src/runtime/messaging.rs b/crates/ironclaw_engine/src/runtime/messaging.rs new file mode 100644 index 00000000000..baea5d8ae9d --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/messaging.rs @@ -0,0 +1,58 @@ +//! Thread-to-thread messaging via channels. + +use crate::types::message::ThreadMessage; +use crate::types::thread::ThreadId; + +/// Signal sent to a running thread via its mailbox. +#[derive(Debug)] +pub enum ThreadSignal { + /// Stop the thread gracefully. + Stop, + /// Pause execution (can be resumed later). + Suspend, + /// Resume a suspended thread. + Resume, + /// Inject a user message into the thread's context. + InjectMessage(ThreadMessage), + /// Notification that a child thread completed. + ChildCompleted { + child_id: ThreadId, + outcome: ThreadOutcome, + }, +} + +/// Final outcome of a thread's execution. +#[derive(Debug, Clone)] +pub enum ThreadOutcome { + /// Completed with an optional text response. + Completed { response: Option }, + /// Thread was stopped by a signal. + Stopped, + /// Max iterations reached without completing. + MaxIterations, + /// Terminal failure. + Failed { error: String }, + /// A unified execution gate paused the thread. + GatePaused { + gate_name: String, + action_name: String, + call_id: String, + parameters: serde_json::Value, + resume_kind: crate::gate::ResumeKind, + /// Completed action output that should be injected on resume instead + /// of re-running the action. + resume_output: Option, + }, +} + +/// A mailbox for sending signals to a running thread. +/// +/// Each thread gets a `(sender, receiver)` pair. The `ThreadManager` holds +/// the sender; the `ExecutionLoop` holds the receiver. +pub type SignalSender = tokio::sync::mpsc::Sender; +pub type SignalReceiver = tokio::sync::mpsc::Receiver; + +/// Create a new signal channel with the given buffer size. +pub fn signal_channel(buffer: usize) -> (SignalSender, SignalReceiver) { + tokio::sync::mpsc::channel(buffer) +} diff --git a/crates/ironclaw_engine/src/runtime/mission.rs b/crates/ironclaw_engine/src/runtime/mission.rs new file mode 100644 index 00000000000..c7c95ae8928 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/mission.rs @@ -0,0 +1,6752 @@ +//! Mission manager — orchestrates long-running goals that spawn threads over time. +//! +//! Missions track ongoing objectives and periodically spawn threads to make +//! progress. The manager handles lifecycle (create, pause, resume, complete) +//! and delegates thread spawning to [`ThreadManager`]. + +use std::collections::{HashMap, HashSet, VecDeque}; +use std::sync::Arc; +use std::time::Duration; + +use serde::Deserialize; +use tokio::sync::RwLock; +use tracing::{debug, warn}; + +use ironclaw_skills::types::ActivationCriteria; +use ironclaw_skills::v2::{CodeSnippet, SkillRepairRecord, SkillRepairType, V2SkillMetadata}; + +use crate::executor::trace::{ExecutionTrace, IssueSeverity}; +use crate::memory::{RetrievalEngine, SkillTracker}; +use crate::runtime::manager::ThreadManager; +use crate::runtime::messaging::ThreadOutcome; +use crate::traits::store::Store; +use crate::traits::workspace::WorkspaceReader; +use crate::types::error::EngineError; +use crate::types::memory::{DocId, DocType, MemoryDoc}; +use crate::types::mission::{ + Mission, MissionCadence, MissionId, MissionStatus, next_cron_fire, next_cron_fire_required, +}; +use crate::types::project::ProjectId; +use crate::types::shared_owner_id; +use crate::types::thread::{ + ActiveSkillProvenance, Thread, ThreadConfig, ThreadId, ThreadState, ThreadType, +}; + +/// Per-mission compiled regex cache. We compile patterns lazily on first +/// match attempt and discard them when the mission updates or deletes its +/// cadence. The cache is process-local — restarts repopulate on demand. +type EventRegexCache = HashMap; + +/// Maximum compiled regex size, mirroring the v1 routine engine. Patterns +/// that exceed this are refused at compile time so a hostile or buggy +/// mission cannot pin the matcher with a pathological regex. +const MAX_EVENT_REGEX_SIZE: usize = 64 * 1024; + +/// Per-user fire-rate ceiling expressed as a token bucket. Independent of +/// per-mission `cooldown_secs`, this is a *global* cap across all of a +/// user's missions so a user that owns many event-triggered missions can't +/// collectively flood the LLM. +#[derive(Debug, Clone)] +pub struct FireRateLimit { + /// Maximum number of fires permitted within `window`. + pub max_fires: u32, + /// Sliding-window duration. Fires older than this are evicted. + pub window: std::time::Duration, +} + +impl Default for FireRateLimit { + /// 100 mission firings per user per hour. Generous enough that normal + /// cron + a handful of event-driven missions don't notice it; tight + /// enough that a misbehaving pattern is bounded. + fn default() -> Self { + Self { + max_fires: 100, + window: std::time::Duration::from_secs(3600), + } + } +} + +/// Engine-side budget abstraction. Implementations decide whether the +/// `user_id` still has enough LLM/financial budget to spawn another +/// mission thread. The host implements this over its existing +/// `CostGuard`. +/// +/// When `MissionManager` has no `BudgetGate` attached, all fires are +/// allowed (back-compat for embedders that don't use a budget). +#[async_trait::async_trait] +pub trait BudgetGate: Send + Sync { + /// Returns `true` if a mission fire is allowed for `user_id`. The + /// `mission_id` is included so adapters can apply per-mission policies + /// if they wish; most implementations will only consult `user_id`. + async fn allow_mission_fire(&self, user_id: &str, mission_id: MissionId) -> bool; +} + +/// Notification emitted when a mission thread completes. +/// +/// The bridge subscribes to these and routes the response text to +/// the mission's `notify_channels` via `ChannelManager::broadcast()`. +#[derive(Debug, Clone)] +pub struct MissionNotification { + pub mission_id: MissionId, + pub mission_name: String, + pub thread_id: ThreadId, + pub user_id: String, + /// Channels to notify (from `Mission.notify_channels`). + pub notify_channels: Vec, + /// Optional per-channel recipient (from `Mission.notify_user`). When + /// `None`, the channel's default recipient is used. + pub notify_user: Option, + /// The thread's response text (None if failed/no output). + pub response: Option, + /// True if the thread failed. + pub is_error: bool, +} + +/// Optional updates to apply to a mission via [`MissionManager::update_mission`]. +#[derive(Debug, Default, Clone)] +pub struct MissionUpdate { + pub name: Option, + pub description: Option, + pub goal: Option, + pub cadence: Option, + pub notify_channels: Option>, + pub notify_user: Option, + pub context_paths: Option>, + pub max_threads_per_day: Option, + pub success_criteria: Option, + pub cooldown_secs: Option, + pub max_concurrent: Option, + pub dedup_window_secs: Option, +} + +/// In-memory dedup state for event-triggered missions. Keyed by +/// (mission_id, dedup-key) → last fire timestamp. +type DedupKey = (MissionId, String); + +/// Manages mission lifecycle and thread spawning. +pub struct MissionManager { + store: Arc, + thread_manager: Arc, + /// Active missions indexed by ID for quick lookup. + active: RwLock>, + /// Broadcast channel for mission outcome notifications. + notification_tx: tokio::sync::broadcast::Sender, + /// Per-mission in-memory cooldown timestamp, recorded after each + /// `fire_mission` attempt regardless of whether `save_mission` succeeded. + /// + /// `tick` consults this to suppress re-firing the same mission within + /// [`FIRE_COOLDOWN`] when a transient store failure has prevented + /// `next_fire_at` / `threads_today` from advancing in the persisted record. + /// Without this guard, a save failure after a successful fire would cause + /// the next 60 s tick (and every subsequent tick) to re-fire the same + /// mission until the store recovers, spawning duplicate threads up to the + /// daily budget. + last_fire_attempt: RwLock>>, + /// Optional workspace reader used to load `Mission.context_paths` at + /// fire time. When `None`, context preloading is silently skipped. + workspace: Option>, + /// Per-mission dedup table for event-triggered firings. Cleared + /// opportunistically when entries fall outside the dedup window. + dedup_table: RwLock>>, + /// Compiled regex cache for `OnEvent` mission patterns. Lazily filled + /// on first match attempt; entries are evicted on mission update/delete. + event_regex_cache: RwLock, + /// Per-user sliding-window fire log used by the global rate limiter. + /// Each `VecDeque` holds firing timestamps within the configured window. + user_fire_log: RwLock>>>, + /// Global per-user fire-rate ceiling. + rate_limit: FireRateLimit, + /// Optional budget gate consulted before each fire. + budget_gate: Option>, +} + +/// Minimum gap between successive `fire_mission` attempts for the same +/// mission ID, enforced in-memory by `tick`. Chosen to comfortably exceed the +/// 60 s tick interval so a single tick gap is always honored, while still +/// allowing recovery within a few minutes if the store comes back. +const FIRE_COOLDOWN: Duration = Duration::from_secs(90); + +impl MissionManager { + pub fn new(store: Arc, thread_manager: Arc) -> Self { + let (notification_tx, _) = tokio::sync::broadcast::channel(64); + Self { + store, + thread_manager, + active: RwLock::new(Vec::new()), + notification_tx, + last_fire_attempt: RwLock::new(HashMap::new()), + workspace: None, + dedup_table: RwLock::new(HashMap::new()), + event_regex_cache: RwLock::new(HashMap::new()), + user_fire_log: RwLock::new(HashMap::new()), + rate_limit: FireRateLimit::default(), + budget_gate: None, + } + } + + /// Attach a workspace reader so `context_paths` are loaded at fire time. + /// Builder-style for back-compat with existing call sites that don't yet + /// supply a reader. + pub fn with_workspace_reader(mut self, reader: Arc) -> Self { + self.workspace = Some(reader); + self + } + + /// Attach a budget gate so each fire consults the host's spend limit. + /// When unattached, all fires are allowed (back-compat). + pub fn with_budget_gate(mut self, gate: Arc) -> Self { + self.budget_gate = Some(gate); + self + } + + /// Override the per-user fire-rate limit. Defaults to 100 fires/hour. + pub fn with_rate_limit(mut self, limit: FireRateLimit) -> Self { + self.rate_limit = limit; + self + } + + /// Subscribe to mission outcome notifications. + /// + /// The bridge uses this to route mission results to channels. + pub fn subscribe_notifications(&self) -> tokio::sync::broadcast::Receiver { + self.notification_tx.subscribe() + } + + /// Test-only handle to the broadcast sender so unit tests can drive + /// `process_mission_outcome_and_notify` without going through the full + /// thread lifecycle. Not part of the public API. + #[cfg(test)] + pub(crate) fn notification_tx_for_test( + &self, + ) -> &tokio::sync::broadcast::Sender { + &self.notification_tx + } + + /// Populate the active mission index from persisted mission state. + /// + /// Also backfills `next_fire_at` for active cron missions created before + /// the scheduling fix — without this, legacy cron missions would remain + /// stuck with `next_fire_at = None` and never fire. + pub async fn bootstrap_project(&self, project_id: ProjectId) -> Result { + // System operation: load all missions for the project regardless of user. + let missions = self.store.list_all_missions(project_id).await?; + let mut active_ids = Vec::new(); + + for mission in missions { + if mission.status != MissionStatus::Active { + continue; + } + // Backfill next_fire_at for cron missions that predate the + // scheduling fix. Match all three branches of next_cron_fire so a + // mission with an unschedulable cron (Ok(None) — e.g. a year-locked + // expression in the past) or an invalid expression (Err) is at + // least observable in the logs instead of silently staying stuck. + // + // Lenient `next_cron_fire` (not `_required`): startup backfill must + // never block — a single corrupt persisted expression cannot fail + // bootstrap, since the rest of the active missions still need to + // register. See `next_cron_fire_required` for the strict variant + // used at lifecycle entry points. + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + && mission.next_fire_at.is_none() + { + match next_cron_fire(expression, timezone.as_ref()) { + Ok(Some(next)) => { + // Re-load the mission immediately before save to narrow + // the TOCTOU window between the initial list_all_missions + // snapshot and our save. If a concurrent fire/update has + // already populated next_fire_at, skip — that writer's + // copy is fresher than ours. The remaining race window + // (between this re-load and save_mission) is much smaller + // than the original list-then-save window, and a strict + // CAS would require a new Store trait method. + match self.store.load_mission(mission.id).await { + Ok(Some(mut fresh)) if fresh.next_fire_at.is_none() => { + fresh.next_fire_at = Some(next); + match self.store.save_mission(&fresh).await { + Ok(()) => debug!( + mission_id = %mission.id, + next = %next, + "backfilled next_fire_at for legacy cron mission" + ), + Err(e) => debug!( + mission_id = %mission.id, + error = %e, + "failed to persist next_fire_at backfill; mission will retry on next bootstrap" + ), + } + } + Ok(Some(_)) => debug!( + mission_id = %mission.id, + "next_fire_at already set by concurrent writer; skipping backfill" + ), + Ok(None) => debug!( + mission_id = %mission.id, + "mission deleted between bootstrap list and backfill; skipping" + ), + Err(e) => debug!( + mission_id = %mission.id, + error = %e, + "failed to re-load mission for backfill" + ), + } + } + Ok(None) => debug!( + mission_id = %mission.id, + expression = %expression, + timezone = ?timezone, + "legacy cron mission has no upcoming fire time; leaving next_fire_at unset" + ), + Err(e) => debug!( + mission_id = %mission.id, + expression = %expression, + timezone = ?timezone, + error = %e, + "failed to compute next_fire_at for legacy cron mission; leaving next_fire_at unset" + ), + } + } + active_ids.push(mission.id); + } + + let count = active_ids.len(); + *self.active.write().await = active_ids; + debug!(project_id = ?project_id, active_missions = count, "bootstrapped active missions"); + Ok(count) + } + + /// Create and persist a new mission. Returns the mission ID. + pub async fn create_mission( + &self, + project_id: ProjectId, + user_id: impl Into, + name: impl Into, + goal: impl Into, + cadence: MissionCadence, + notify_channels: Vec, + ) -> Result { + let mut mission = Mission::new(project_id, user_id, name, goal, cadence); + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + { + // Reject Ok(None) at the create boundary — an Active cron mission + // with `next_fire_at = None` is the original #1944 failure mode. + mission.next_fire_at = Some(next_cron_fire_required(expression, timezone.as_ref())?); + } + mission.notify_channels = notify_channels; + let id = mission.id; + self.store.save_mission(&mission).await?; + self.active.write().await.push(id); + debug!(mission_id = %id, "mission created"); + Ok(id) + } + + /// Update mutable fields on a mission. Only non-None fields are applied. + pub async fn update_mission( + &self, + id: MissionId, + user_id: &str, + updates: MissionUpdate, + ) -> Result<(), EngineError> { + let mut mission = self + .store + .load_mission(id) + .await? + .ok_or_else(|| EngineError::Store { + reason: format!("mission {id} not found"), + })?; + + let allowed = if mission.owner_id().is_shared() { + crate::types::is_shared_owner(user_id) + } else { + mission.is_owned_by(user_id) + }; + if !allowed { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("mission {id}"), + }); + } + + if let Some(name) = updates.name { + mission.name = name; + } + if let Some(description) = updates.description { + mission.description = Some(description); + } + if let Some(goal) = updates.goal { + mission.goal = goal; + } + if let Some(cadence) = updates.cadence { + mission.cadence = cadence; + // Recompute scheduling state to match the new cadence. Without this, + // a Manual -> Cron switch leaves next_fire_at = None and the ticker + // never picks the mission up; a Cron expression/timezone change + // keeps firing on the old schedule until the mission is paused and + // resumed. Clear next_fire_at for non-cron cadences so a stale + // value can't trigger an unrelated cron path. Reject cron schedules + // that are valid but have no future fire time so we don't persist + // an Active mission that can never run. + // + // Strict `next_cron_fire_required`: an `Err` here returns from + // `update_mission` BEFORE the `save_mission` call below, leaving + // the persisted record on its previous (valid) cadence. The + // `mission` local is dropped without ever being persisted — + // `save_mission` is the only persistence boundary in this + // function, so failing before it leaves the store untouched. + // Verified by `update_mission_rejects_switch_to_unschedulable_cron`. + mission.next_fire_at = match &mission.cadence { + MissionCadence::Cron { + expression, + timezone, + } => Some(next_cron_fire_required(expression, timezone.as_ref())?), + _ => None, + }; + } + if let Some(channels) = updates.notify_channels { + mission.notify_channels = channels; + } + if let Some(notify_user) = updates.notify_user { + mission.notify_user = Some(notify_user); + } + if let Some(context_paths) = updates.context_paths { + mission.context_paths = context_paths; + } + if let Some(max) = updates.max_threads_per_day { + mission.max_threads_per_day = max; + } + if let Some(criteria) = updates.success_criteria { + mission.success_criteria = Some(criteria); + } + if let Some(secs) = updates.cooldown_secs { + mission.cooldown_secs = secs; + } + if let Some(max) = updates.max_concurrent { + mission.max_concurrent = max; + } + if let Some(secs) = updates.dedup_window_secs { + mission.dedup_window_secs = secs; + } + + mission.updated_at = chrono::Utc::now(); + self.store.save_mission(&mission).await?; + // The cadence (and therefore event_pattern) may have changed. + // Drop the cached compiled regex; the next match attempt + // recompiles from the current pattern. + self.evict_event_regex(id).await; + debug!(mission_id = %id, "mission updated"); + Ok(()) + } + + /// Pause an active mission. No new threads will be spawned. + /// + /// Shared missions can only be managed by shared owners (system user). + pub async fn pause_mission(&self, id: MissionId, user_id: &str) -> Result<(), EngineError> { + let mission = self + .store + .load_mission(id) + .await? + .ok_or_else(|| EngineError::Store { + reason: format!("mission {id} not found"), + })?; + let allowed = if mission.owner_id().is_shared() { + crate::types::is_shared_owner(user_id) + } else { + mission.is_owned_by(user_id) + }; + if !allowed { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("mission {id}"), + }); + } + self.store + .update_mission_status(id, MissionStatus::Paused) + .await?; + self.active.write().await.retain(|mid| *mid != id); + // Drop the in-memory cooldown entry — a paused mission can't fire, + // so the cooldown is dead state and would otherwise leak until the + // process restarts. + self.last_fire_attempt.write().await.remove(&id); + debug!(mission_id = %id, "mission paused"); + Ok(()) + } + + /// Resume a paused mission. + /// + /// Shared missions can only be managed by shared owners (system user). + /// Only `Paused` missions can be resumed — `Completed` and `Failed` are + /// terminal states and must not be resurrected by a stray resume call, + /// so anything else is rejected with a `Store` error. + pub async fn resume_mission(&self, id: MissionId, user_id: &str) -> Result<(), EngineError> { + let mut mission = self + .store + .load_mission(id) + .await? + .ok_or_else(|| EngineError::Store { + reason: format!("mission {id} not found"), + })?; + let allowed = if mission.owner_id().is_shared() { + crate::types::is_shared_owner(user_id) + } else { + mission.is_owned_by(user_id) + }; + if !allowed { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("mission {id}"), + }); + } + if mission.status != MissionStatus::Paused { + return Err(EngineError::Store { + reason: format!( + "mission {id} is in state {:?}, only Paused missions can be resumed", + mission.status + ), + }); + } + // Mutate-and-save in a single round-trip. The previous implementation + // did `update_mission_status(Active)` and then a separate `load+save` + // to recompute next_fire_at — between the two writes, a concurrent + // `update_mission`/`fire_mission` could modify other fields that the + // second save would then silently overwrite with the stale reload. + mission.status = MissionStatus::Active; + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + { + // Reject Ok(None): resuming a cron mission whose schedule has no + // upcoming fire time would silently re-create the #1944 stuck + // state. Surface the error so the caller can fix the schedule. + mission.next_fire_at = Some(next_cron_fire_required(expression, timezone.as_ref())?); + } + mission.updated_at = chrono::Utc::now(); + self.store.save_mission(&mission).await?; + let mut active = self.active.write().await; + if !active.contains(&id) { + active.push(id); + } + debug!(mission_id = %id, "mission resumed"); + Ok(()) + } + + /// Mark a mission as completed. + pub async fn complete_mission(&self, id: MissionId) -> Result<(), EngineError> { + self.store + .update_mission_status(id, MissionStatus::Completed) + .await?; + self.active.write().await.retain(|mid| *mid != id); + // Terminal state — drop the cooldown entry so the in-memory map + // doesn't accumulate an entry per mission ever fired. + self.last_fire_attempt.write().await.remove(&id); + self.evict_event_regex(id).await; + debug!(mission_id = %id, "mission completed"); + Ok(()) + } + + /// Fire a mission — build meta-prompt, spawn thread, process outcome. + /// + /// Optional `trigger_payload` carries webhook/event data that triggered this + /// fire. It's injected into the thread's context as `state["trigger_payload"]`. + pub async fn fire_mission( + &self, + id: MissionId, + user_id: &str, + trigger_payload: Option, + ) -> Result, EngineError> { + let mission = self.store.load_mission(id).await?; + let mission = match mission { + Some(m) => m, + None => { + return Err(EngineError::Store { + reason: format!("mission {id} not found"), + }); + } + }; + + // Tenant isolation: verify the requesting user owns this mission. + // Shared missions can be fired by any user — the spawned + // thread inherits the requesting user's identity, keeping artifacts user-scoped. + if !mission.owner_id().is_shared() && !mission.is_owned_by(user_id) { + return Err(EngineError::AccessDenied { + user_id: user_id.to_string(), + entity: format!("mission {id}"), + }); + } + + if mission.is_terminal() { + debug!(mission_id = %id, status = ?mission.status, "cannot fire terminal mission"); + return Ok(None); + } + + // Check daily budget + if mission.max_threads_per_day > 0 && mission.threads_today >= mission.max_threads_per_day { + debug!(mission_id = %id, "daily thread budget exhausted"); + return Ok(None); + } + + // Cooldown: refuse to fire if the last successful fire was within + // `cooldown_secs` of now. 0 = disabled. + if mission.cooldown_secs > 0 + && let Some(last) = mission.last_fire_at + { + let elapsed = chrono::Utc::now().signed_duration_since(last).num_seconds(); + if elapsed >= 0 && (elapsed as u64) < mission.cooldown_secs { + debug!( + mission_id = %id, + elapsed_secs = elapsed, + cooldown_secs = mission.cooldown_secs, + "mission cooldown not yet elapsed" + ); + return Ok(None); + } + } + + // max_concurrent: count threads from this mission that are still in + // a non-terminal state. 0 = unlimited. + if mission.max_concurrent > 0 { + let running = self.count_running_threads(&mission).await; + if running >= mission.max_concurrent as usize { + debug!( + mission_id = %id, + running, + max_concurrent = mission.max_concurrent, + "mission max_concurrent reached" + ); + return Ok(None); + } + } + + // Per-user global rate limit. Independent of per-mission cooldown, + // this is a sliding-window cap across *all* of the user's missions + // so a user with many event-triggered missions can't collectively + // flood the LLM. We only *check* here — recording is deferred until + // after the spawn succeeds so a downstream failure (store error, + // budget refusal, spawn error) doesn't consume a slot and slowly + // self-DoS the user. + if !self.check_user_rate(&mission.user_id).await { + debug!( + mission_id = %id, + user_id = %mission.user_id, + max_fires = self.rate_limit.max_fires, + window_secs = self.rate_limit.window.as_secs(), + "per-user mission fire rate limit reached" + ); + return Ok(None); + } + + // Budget gate: when the host wires a `BudgetGate` (typically over + // its CostGuard), refuse to fire when the user is out of budget. + // Unattached gate = always allow. + if !self.budget_allows(&mission.user_id, id).await { + debug!( + mission_id = %id, + user_id = %mission.user_id, + "mission fire refused by budget gate" + ); + return Ok(None); + } + + // Load context_paths from the workspace if a reader is attached. + // Failures are logged but never block the fire — context loading is + // a best-effort enrichment, not a precondition. + let mut context_blocks: Vec<(String, String)> = Vec::new(); + if let Some(reader) = self.workspace.as_ref() { + for path in &mission.context_paths { + match reader.read_doc(path).await { + Ok(content) => context_blocks.push((path.clone(), content)), + Err(error) => debug!( + mission_id = %id, + path = %path, + error = %error, + "failed to load mission context_path; skipping" + ), + } + } + } else if !mission.context_paths.is_empty() { + debug!( + mission_id = %id, + paths = mission.context_paths.len(), + "mission has context_paths but no WorkspaceReader is attached" + ); + } + + // Build meta-prompt from mission state + project docs + let retrieval = RetrievalEngine::new(Arc::clone(&self.store)); + let project_docs = retrieval + .retrieve_context(mission.project_id, &mission.user_id, &mission.goal, 10) + .await + .unwrap_or_default(); + let meta_prompt = + build_meta_prompt(&mission, &project_docs, &trigger_payload, &context_blocks); + + // Spawn thread with meta-prompt as initial user message + let thread_id = self + .thread_manager + .spawn_thread( + &meta_prompt, + ThreadType::Mission, + mission.project_id, + ThreadConfig::default(), + None, + user_id, + ) + .await?; + + // Capture the fire instant once and use it for both the persisted + // `last_fire_at` and the in-memory `last_fire_attempt` map. The two + // writes MUST share the same value: tick's stale-state detection + // compares them as equal-or-not to decide whether the cooldown + // applies, and using two separate `Utc::now()` calls would produce + // microsecond drift that breaks the equality check on the success + // path. + let fire_instant = chrono::Utc::now(); + + // Install the outcome watcher *before* persisting the mission update. + // The watcher only depends on `thread_id` (it joins via ThreadManager + // and reloads the mission record itself), so installing it first + // ensures a transient `save_mission` failure below cannot orphan the + // running thread by skipping the watcher install. Pass `fire_instant` + // through so the outcome processor can reconcile `last_fire_at` + // back to the original moment if the save below fails — without + // this the reconciled value would be the *outcome* time, which can + // be many seconds-to-hours later for long-running mission threads. + self.spawn_mission_outcome_watcher(id, thread_id, fire_instant); + + // Record the thread + trigger payload in mission history + let mut updated = mission; + let user_id_for_rate = updated.user_id.clone(); + updated.record_thread(thread_id); + updated.threads_today += 1; + updated.last_trigger_payload = trigger_payload; + // Advance next_fire_at for cron missions so the ticker schedules the + // next cycle. Computed from `now()`, not from the previous fire time: + // if a tick was delayed (process down, busy loop) and several windows + // are missed, they coalesce into a single fire here rather than + // backfilling each missed slot. This is the catch-up semantics we want + // for long-running missions. + // + // Lenient `next_cron_fire` (not `_required`): a parse error here is + // unlikely (the expression validated at create time) but possible if + // persisted data is corrupt. We log and preserve the existing + // `next_fire_at` rather than aborting fire — the thread is already + // running and the watcher is already installed, and at worst the + // schedule is delayed by one cycle until the next tick. + // + // `cron_advanced` tracks whether scheduling actually progressed. When + // false (parse error on a corrupt expression), we deliberately leave + // `last_fire_at` at its OLD persisted value. The in-memory + // `last_fire_attempt[mid]` will still be set to `fire_instant` below, + // so the in-memory vs persisted mismatch arms tick's cooldown via + // the same code path as a save failure. Without this, a corrupt + // expression with a past `next_fire_at` would re-fire on every tick + // (cooldown matched, schedule never advanced) and exhaust the daily + // budget — same root cause as #1944. + let mut cron_advanced = true; + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = updated.cadence + { + match next_cron_fire(expression, timezone.as_ref()) { + Ok(next) => updated.next_fire_at = next, + Err(e) => { + cron_advanced = false; + debug!( + mission_id = %id, + expression = %expression, + error = %e, + "failed to advance next_fire_at after fire; preserving existing value and arming cooldown via mismatch" + ); + } + } + } + if cron_advanced { + updated.last_fire_at = Some(fire_instant); + } + + // Arm the in-memory cooldown BEFORE the persistence call, not after. + // + // Why first: a concurrent tick observing the post-`save_mission` + // state but pre-`last_fire_attempt`-insert state would see the + // freshly-persisted `last_fire_at = fire_instant` AND no in-memory + // entry, evaluate `is_some_and(...)` to false, and (if the schedule + // is still in the past) re-fire immediately. Inserting first closes + // that race: while save is in flight, in-memory has `fire_instant` + // and persisted still has the OLD `last_fire_at`, so tick sees a + // mismatch and arms the cooldown. Once save lands the values match + // (success path) or stay mismatched (failure path) — both correct. + // + // Held briefly under the write lock; the map is keyed by mission ID + // and only mutated here, in tick (prune), and in pause/complete. + self.last_fire_attempt + .write() + .await + .insert(id, fire_instant); + + // Persistence is best-effort: if save_mission fails on a transient store + // error, the thread is already running and the outcome watcher is already + // installed (above), so failing here would orphan the work AND — for cron + // cadences — leave next_fire_at un-advanced, causing the next tick to + // re-fire the same mission in a runaway loop. Log and continue. The + // caller still gets Ok(Some(thread_id)) so the spawned thread is visible. + // The in-memory `last_fire_attempt` cooldown above catches runaway re-fires + // by comparing in-memory vs persisted `last_fire_at` (see tick). + if let Err(e) = self.store.save_mission(&updated).await { + debug!( + mission_id = %id, + thread_id = %thread_id, + error = %e, + "failed to persist mission update after fire; thread is running and watched, in-memory cooldown will suppress re-fire" + ); + } + + // Now that the spawn + persist have succeeded, consume a slot in + // the per-user rate window. Doing this here (rather than at the + // earlier check site) means store errors, budget refusals, and + // spawn failures all leave the user's window untouched. + self.record_user_rate(&user_id_for_rate).await; + + debug!(mission_id = %id, thread_id = %thread_id, "mission fired"); + + Ok(Some(thread_id)) + } + + /// Resume suspended checkpointed mission threads after restart. + pub async fn resume_recoverable_threads( + &self, + user_id: &str, + ) -> Result, EngineError> { + let mut resumed = Vec::new(); + + for mission_id in self.active.read().await.clone() { + let Some(mission) = self.store.load_mission(mission_id).await? else { + continue; + }; + + for &thread_id in mission.thread_history.iter().rev() { + let Some(thread) = self.store.load_thread(thread_id).await? else { + continue; + }; + if thread.thread_type != ThreadType::Mission + || thread.state != crate::types::thread::ThreadState::Suspended + { + continue; + } + if thread.metadata.get("runtime_checkpoint").is_none() { + continue; + } + + self.thread_manager + .resume_thread(thread_id, user_id.to_string(), None, None, None) + .await?; + // Resumed threads are already in `thread_history` from the + // original fire, so the outcome processor will see + // `needs_reconcile = false` and never read this value. Pass + // `now` as a safe placeholder; nothing depends on it. + self.spawn_mission_outcome_watcher(mission_id, thread_id, chrono::Utc::now()); + resumed.push(thread_id); + } + } + + Ok(resumed) + } + + /// Start a background cron ticker that fires due missions every 60 seconds. + pub fn start_cron_ticker(self: &Arc, user_id: String) { + let mgr = Arc::clone(self); + tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(60)); + loop { + interval.tick().await; + match mgr.tick(&user_id).await { + Ok(spawned) if !spawned.is_empty() => { + debug!(count = spawned.len(), "cron ticker spawned mission threads"); + } + Err(e) => { + debug!("cron ticker error: {e}"); + } + _ => {} + } + } + }); + } + + /// List all missions in a project for a given user. + /// List missions visible to a user (own + shared). + pub async fn list_missions( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + self.store + .list_missions_with_shared(project_id, user_id) + .await + } + + /// Get a mission by ID. + pub async fn get_mission(&self, id: MissionId) -> Result, EngineError> { + self.store.load_mission(id).await + } + + /// Fire all active `OnSystemEvent` missions whose source and event_type match. + /// + /// The optional `payload` is forwarded as `trigger_payload` to each mission's + /// thread, carrying context like trace issues and reflection docs. + pub async fn fire_on_system_event( + &self, + source: &str, + event_type: &str, + user_id: &str, + payload: Option, + ) -> Result, EngineError> { + let active_ids = self.active.read().await.clone(); + let mut spawned = Vec::new(); + + for mid in active_ids { + let mission = match self.store.load_mission(mid).await? { + Some(m) if m.status == MissionStatus::Active => m, + _ => continue, + }; + + // Only fire missions owned by this user (per-user learning missions) + // or globally shared missions. + if !mission.is_owned_by(user_id) && !mission.owner_id().is_shared() { + continue; + } + + let matches = match &mission.cadence { + MissionCadence::OnSystemEvent { + source: s, + event_type: et, + filters, + } => { + s == source + && et == event_type + && payload_matches_filters(filters, payload.as_ref()) + } + _ => false, + }; + + if !matches { + continue; + } + + // Dedup: skip if an identical event key fired this mission within + // its dedup window. The default key is the SHA-256 of the payload + // serialization (compact and stable for typical webhook bodies). + if mission.dedup_window_secs > 0 { + let key = payload_dedup_key(payload.as_ref()); + if self.dedup_event(mid, &key, mission.dedup_window_secs).await { + debug!( + mission_id = %mid, + dedup_window_secs = mission.dedup_window_secs, + "skipping system_event fire — dedup window not yet elapsed" + ); + continue; + } + } + + if let Some(tid) = self.fire_mission(mid, user_id, payload.clone()).await? { + spawned.push(tid); + } + } + + Ok(spawned) + } + + /// Fire all active `OnEvent` missions whose `event_pattern` matches + /// `text` and (if a channel filter is set) whose `channel` matches the + /// incoming message channel case-insensitively. + /// + /// `payload` is forwarded as `trigger_payload` to each mission's thread. + /// Pattern matching uses simple substring matching to keep this dependency- + /// free; callers needing regex semantics should normalize first or + /// extend the matcher. + pub async fn fire_on_message_event( + &self, + channel: &str, + text: &str, + user_id: &str, + payload: Option, + ) -> Result, EngineError> { + let active_ids = self.active.read().await.clone(); + let mut spawned = Vec::new(); + + for mid in active_ids { + let mission = match self.store.load_mission(mid).await? { + Some(m) if m.status == MissionStatus::Active => m, + _ => continue, + }; + + if !mission.is_owned_by(user_id) && !mission.owner_id().is_shared() { + continue; + } + + let channel_ok = match &mission.cadence { + MissionCadence::OnEvent { + channel: cadence_channel, + .. + } => cadence_channel + .as_ref() + .is_none_or(|c| c.eq_ignore_ascii_case(channel)), + _ => continue, + }; + if !channel_ok { + continue; + } + // Regex match (with size-limited compile + per-mission cache). + // The substring fallback used previously was too loose: it + // matched "the review was requested yesterday" against + // "review requested" and would flood on busy channels. + if !self.event_regex_matches(&mission, text).await { + continue; + } + + if mission.dedup_window_secs > 0 { + let key = payload_dedup_key(payload.as_ref()); + if self.dedup_event(mid, &key, mission.dedup_window_secs).await { + continue; + } + } + + if let Some(tid) = self.fire_mission(mid, user_id, payload.clone()).await? { + spawned.push(tid); + } + } + + Ok(spawned) + } + + /// Fire the active `Webhook` mission whose registered `path` matches the + /// incoming webhook path. The bridge layer is responsible for HMAC + /// validation against `Webhook.secret` *before* calling this; the engine + /// just routes payloads to mission threads. + /// + /// Returns the IDs of any threads spawned. + pub async fn fire_on_webhook( + &self, + webhook_path: &str, + user_id: &str, + payload: Option, + ) -> Result, EngineError> { + let active_ids = self.active.read().await.clone(); + let mut spawned = Vec::new(); + + for mid in active_ids { + let mission = match self.store.load_mission(mid).await? { + Some(m) if m.status == MissionStatus::Active => m, + _ => continue, + }; + + if !mission.is_owned_by(user_id) && !mission.owner_id().is_shared() { + continue; + } + + let matches = matches!( + &mission.cadence, + MissionCadence::Webhook { path, .. } if path == webhook_path + ); + + if !matches { + continue; + } + + if mission.dedup_window_secs > 0 { + let key = payload_dedup_key(payload.as_ref()); + if self.dedup_event(mid, &key, mission.dedup_window_secs).await { + continue; + } + } + + if let Some(tid) = self.fire_mission(mid, user_id, payload.clone()).await? { + spawned.push(tid); + } + } + + Ok(spawned) + } + + /// Start a background event listener that fires learning missions when + /// threads complete. + /// + /// Subscribes to the ThreadManager's event broadcast channel and watches + /// for `StateChanged { to: Done }`. For each completed non-Mission thread: + /// + /// 1. **Skill repair** — if an active skill looks stale or incomplete, + /// fires `thread_completed_with_skill_gap` + /// 2. **Error diagnosis** — if trace has issues, fires `thread_completed_with_issues` + /// 3. **Skill extraction** — if thread succeeded with many steps/actions, + /// fires `thread_completed_with_learnings` + /// 4. **Conversation insights** — after every N threads in a conversation, + /// fires `conversation_insights_due` + pub fn start_event_listener(self: &Arc, _owner_id: String) { + let mgr = Arc::clone(self); + let mut rx = mgr.thread_manager.subscribe_events(); + + /// Minimum steps for a thread to be a skill extraction candidate. + const SKILL_EXTRACTION_MIN_STEPS: usize = 5; + /// Minimum distinct action executions for skill extraction. + const SKILL_EXTRACTION_MIN_ACTIONS: usize = 3; + /// Completed thread interval for conversation insights. + const CONVERSATION_INSIGHTS_INTERVAL: u32 = 5; + + tokio::spawn(async move { + // Track completed thread count per conversation for insights trigger. + let mut conv_thread_counts: std::collections::HashMap = + std::collections::HashMap::new(); + + loop { + match rx.recv().await { + Ok(event) => { + let Some(terminal_state) = learning_terminal_state(&event.kind) else { + continue; + }; + + // Load the completed thread + let thread = match mgr.store.load_thread(event.thread_id).await { + Ok(Some(t)) => t, + _ => continue, + }; + + // Skip Mission threads (no recursive self-improvement) + if thread.thread_type == ThreadType::Mission { + continue; + } + + let trace = crate::executor::trace::build_trace(&thread); + // Single pass over events for both skill-repair and + // error-diagnosis triggers (avoids repeated iteration + // on large event logs). + let (error_messages, _observed_actions) = + collect_errors_and_actions(&thread); + let active_skills = thread.active_skills(); + + // ── Trigger 1: Skill repair ─────────────────────── + // NOTE: skill-repair and error-diagnosis can both fire + // for the same thread. Each targets a different mission + // so they won't collide, but both may spawn concurrent + // threads. This is intentional — skill-repair fixes the + // *skill* while error-diagnosis fixes the *prompt/orchestrator*. + if !active_skills.is_empty() { + let tracker = SkillTracker::new(Arc::clone(&mgr.store)); + let success = thread_completed_successfully(&thread, &trace); + for skill in &active_skills { + if let Err(e) = tracker.record_usage(skill.doc_id, success).await { + debug!( + skill_doc_id = %skill.doc_id.0, + thread_id = %thread.id, + "event listener: failed to record skill usage: {e}" + ); + } + } + + if let Some(payload) = + build_skill_gap_payload(&thread, &trace, &active_skills) + && let Err(e) = mgr + .fire_on_system_event( + "engine", + "thread_completed_with_skill_gap", + &thread.user_id, + Some(payload), + ) + .await + { + debug!("event listener: failed to fire skill repair: {e}"); + } + } + + // ── Trigger 2: Error diagnosis ────────────────── + if !trace.issues.is_empty() { + let issues: Vec = trace + .issues + .iter() + .map(|i| { + serde_json::json!({ + "severity": format!("{:?}", i.severity), + "category": i.category.clone(), + "description": i.description.clone(), + "step": i.step, + }) + }) + .collect(); + + let payload = serde_json::json!({ + "source_thread_id": event.thread_id.0.to_string(), + "goal": thread.goal, + "issues": issues, + "error_messages": error_messages, + }); + + if let Err(e) = mgr + .fire_on_system_event( + "engine", + "thread_completed_with_issues", + &thread.user_id, + Some(payload), + ) + .await + { + debug!("event listener: failed to fire error diagnosis: {e}"); + } + } + + // ── Trigger 3: Skill extraction ────────────────── + let action_count = thread + .events + .iter() + .filter(|e| { + matches!( + e.kind, + crate::types::event::EventKind::ActionExecuted { .. } + ) + }) + .count(); + + if terminal_state == crate::types::thread::ThreadState::Done + && trace + .issues + .iter() + .all(|i| i.severity != crate::executor::trace::IssueSeverity::Error) + && thread.step_count >= SKILL_EXTRACTION_MIN_STEPS + && action_count >= SKILL_EXTRACTION_MIN_ACTIONS + { + let actions_used: Vec = thread + .events + .iter() + .filter_map(|e| { + if let crate::types::event::EventKind::ActionExecuted { + action_name, + .. + } = &e.kind + { + Some(action_name.clone()) + } else { + None + } + }) + .collect(); + + let payload = serde_json::json!({ + "source_thread_id": event.thread_id.0.to_string(), + "goal": thread.goal, + "step_count": thread.step_count, + "action_count": action_count, + "actions_used": actions_used, + "total_tokens": thread.total_tokens_used, + }); + + if let Err(e) = mgr + .fire_on_system_event( + "engine", + "thread_completed_with_learnings", + &thread.user_id, + Some(payload), + ) + .await + { + debug!("event listener: failed to fire skill extraction: {e}"); + } + } + + // ── Trigger 4: Conversation insights ──────────── + // Keep insights tied to successful completions only. + if should_count_for_conversation_insights(terminal_state) { + // Use the thread's project_id as a proxy for conversation scope. + let conv_key = thread.project_id.0.to_string(); + let count = conv_thread_counts.entry(conv_key.clone()).or_insert(0); + *count += 1; + + if (*count).is_multiple_of(CONVERSATION_INSIGHTS_INTERVAL) { + // Collect recent thread goals for context + let thread_goals: Vec = match mgr + .store + .list_threads(thread.project_id, &thread.user_id) + .await + { + Ok(threads) => threads + .iter() + .rev() + .take(CONVERSATION_INSIGHTS_INTERVAL as usize) + .map(|t| t.goal.clone()) + .collect(), + Err(_) => vec![thread.goal.clone()], + }; + + // Collect sample user messages from recent threads + let sample_messages: Vec = thread + .messages + .iter() + .filter(|m| m.role == crate::types::message::MessageRole::User) + .map(|m| m.content.chars().take(200).collect::()) + .take(10) + .collect(); + + let payload = serde_json::json!({ + "project_id": thread.project_id.0.to_string(), + "completed_thread_count": *count, + "thread_goals": thread_goals, + "sample_user_messages": sample_messages, + }); + + if let Err(e) = mgr + .fire_on_system_event( + "engine", + "conversation_insights_due", + &thread.user_id, + Some(payload), + ) + .await + { + debug!( + "event listener: failed to fire conversation insights: {e}" + ); + } + } + } + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => break, + Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => { + debug!("event listener: lagged {n} events"); + } + } + } + }); + } + + /// Ensure a self-improvement mission exists for the given project. + /// + /// Checks if a mission with `"self_improvement": true` in metadata already + /// exists. If not, creates one with `OnSystemEvent` cadence that fires + /// when threads complete with issues. Also seeds the fix pattern database. + /// + /// Returns the mission ID (existing or newly created). + pub async fn ensure_self_improvement_mission( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result { + // Check if this user already has a self-improvement mission. + let missions = self.store.list_missions(project_id, user_id).await?; + if let Some(existing) = missions.iter().find(|m| is_self_improvement_mission(m)) { + debug!(mission_id = %existing.id, "self-improvement mission already exists"); + // Make sure it's in the active list + let mut active = self.active.write().await; + if !active.contains(&existing.id) { + active.push(existing.id); + } + return Ok(existing.id); + } + + // Create per-user self-improvement mission + let mut mission = Mission::new( + project_id, + user_id, + "self-improvement", + SELF_IMPROVEMENT_GOAL, + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "thread_completed_with_issues".into(), + filters: std::collections::HashMap::new(), + }, + ); + mission.success_criteria = Some( + "Continuously improve system prompts and fix patterns based on execution traces".into(), + ); + mission.metadata = serde_json::json!({"self_improvement": true}); + mission.max_threads_per_day = 5; + // Future-proof: today this helper only ever uses OnSystemEvent, but if + // a future caller passes a Cron cadence the same `next_fire_at = None` + // bug that #1944 fixed in `create_mission` would silently re-emerge + // here. Compute next_fire_at on construction so this helper can never + // produce a stuck cron mission. + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + { + mission.next_fire_at = Some(next_cron_fire_required(expression, timezone.as_ref())?); + } + + let id = mission.id; + self.store.save_mission(&mission).await?; + self.active.write().await.push(id); + + // Seed the fix pattern database if it doesn't exist + let docs = self.store.list_shared_memory_docs(project_id).await?; + let has_patterns = docs.iter().any(|d| { + d.title == FIX_PATTERN_DB_TITLE && d.tags.contains(&FIX_PATTERN_DB_TAG.to_string()) + }); + if !has_patterns { + use crate::types::memory::{DocType, MemoryDoc}; + let pattern_doc = MemoryDoc::new( + project_id, + shared_owner_id(), + DocType::Note, + FIX_PATTERN_DB_TITLE, + SEED_FIX_PATTERNS, + ) + .with_tags(vec![FIX_PATTERN_DB_TAG.to_string()]); + self.store.save_memory_doc(&pattern_doc).await?; + debug!("seeded fix pattern database"); + } + + debug!(mission_id = %id, "created self-improvement mission"); + Ok(id) + } + + /// Ensure the built-in learning missions exist for the given project. + /// + /// Creates (if missing) the self-improvement, skill repair, skill + /// extraction, and conversation insights missions. This is the preferred + /// entry point — call once at project bootstrap. + pub async fn ensure_learning_missions( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result<(), EngineError> { + // 0. Seed compiled-in orchestrator v0 so it's visible in workspace + self.seed_orchestrator_v0(project_id).await?; + + // 1. Error diagnosis (self-improvement) — per-user + self.ensure_self_improvement_mission(project_id, user_id) + .await?; + + // 2. Skill repair + self.ensure_mission_by_metadata( + project_id, + user_id, + "skill_repair", + "skill-repair", + SKILL_REPAIR_GOAL, + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "thread_completed_with_skill_gap".into(), + filters: HashMap::new(), + }, + "Repair versioned skills when execution reveals stale or incomplete instructions", + 5, + ) + .await?; + + // 3. Skill extraction (formerly playbook extraction) + self.ensure_mission_by_metadata( + project_id, + user_id, + "skill_extraction", + "skill-extraction", + SKILL_EXTRACTION_GOAL, + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "thread_completed_with_learnings".into(), + filters: std::collections::HashMap::new(), + }, + "Extract reusable skills from successful multi-step threads", + 3, // max 3/day + ) + .await?; + + // 4. Conversation insights + self.ensure_mission_by_metadata( + project_id, + user_id, + "conversation_insights", + "conversation-insights", + CONVERSATION_INSIGHTS_GOAL, + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "conversation_insights_due".into(), + filters: std::collections::HashMap::new(), + }, + "Extract user preferences, domain knowledge, and workflow patterns from conversations", + 2, // max 2/day + ) + .await?; + + // 5. Expected behavior (user feedback loop) + self.ensure_mission_by_metadata( + project_id, + user_id, + "expected_behavior", + "expected-behavior", + EXPECTED_BEHAVIOR_GOAL, + MissionCadence::OnSystemEvent { + source: "user_feedback".into(), + event_type: "expected_behavior".into(), + filters: std::collections::HashMap::new(), + }, + "Investigate user-reported expectation gaps and apply fixes", + 5, // max 5/day + ) + .await?; + + Ok(()) + } + + /// Seed the compiled-in orchestrator as v0 in the Store. + /// + /// This makes v0 visible in the workspace memory tree and provides a base + /// for the self-improvement mission to diff against when patching. If the + /// compiled-in code has changed (different content hash), the stored v0 is + /// updated to match — runtime patches (v1+) are left untouched. + async fn seed_orchestrator_v0(&self, project_id: ProjectId) -> Result<(), EngineError> { + use crate::executor::orchestrator::{ + DEFAULT_ORCHESTRATOR, ORCHESTRATOR_TAG, ORCHESTRATOR_TITLE, + }; + use crate::types::memory::{DocType, MemoryDoc}; + + let docs = self.store.list_shared_memory_docs(project_id).await?; + let existing_v0 = docs.iter().find(|d| { + d.title == ORCHESTRATOR_TITLE + && d.tags.contains(&ORCHESTRATOR_TAG.to_string()) + && d.metadata + .get("version") + .and_then(|v| v.as_u64()) + .unwrap_or(0) + == 0 + }); + + if let Some(doc) = existing_v0 { + // Update if compiled-in code changed (rebuild with new default.py) + if doc.content != DEFAULT_ORCHESTRATOR { + let mut updated = doc.clone(); + updated.content = DEFAULT_ORCHESTRATOR.to_string(); + updated.updated_at = chrono::Utc::now(); + self.store.save_memory_doc(&updated).await?; + debug!("updated orchestrator v0 to match compiled-in default"); + } + return Ok(()); + } + + // Create v0 doc + let mut doc = MemoryDoc::new( + project_id, + shared_owner_id(), + DocType::Note, + ORCHESTRATOR_TITLE, + DEFAULT_ORCHESTRATOR, + ) + .with_tags(vec![ORCHESTRATOR_TAG.to_string()]); + doc.metadata = serde_json::json!({"version": 0, "source": "compiled_in"}); + self.store.save_memory_doc(&doc).await?; + debug!("seeded orchestrator v0 in workspace"); + Ok(()) + } + + /// Ensure a mission with a specific metadata tag exists, creating it if not. + #[allow(clippy::too_many_arguments)] + async fn ensure_mission_by_metadata( + &self, + project_id: ProjectId, + user_id: &str, + metadata_key: &str, + name: &str, + goal: &str, + cadence: MissionCadence, + success_criteria: &str, + max_per_day: u32, + ) -> Result { + // Check if this user already has a mission with this metadata key. + let missions = self.store.list_missions(project_id, user_id).await?; + if let Some(existing) = missions + .iter() + .find(|m| m.metadata.get(metadata_key).is_some()) + { + let mut active = self.active.write().await; + if !active.contains(&existing.id) { + active.push(existing.id); + } + return Ok(existing.id); + } + + let mut mission = Mission::new(project_id, user_id, name, goal, cadence); + mission.success_criteria = Some(success_criteria.into()); + mission.metadata = serde_json::json!({metadata_key: true}); + mission.max_threads_per_day = max_per_day; + // Future-proof: today every caller passes OnSystemEvent, but a future + // caller passing Cron would silently re-introduce the `next_fire_at = + // None` bug that #1944 fixed in `create_mission`. Compute it here so + // this helper can never produce a stuck cron mission. + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + { + mission.next_fire_at = Some(next_cron_fire_required(expression, timezone.as_ref())?); + } + + let id = mission.id; + self.store.save_mission(&mission).await?; + self.active.write().await.push(id); + + debug!(mission_id = %id, name, "created learning mission"); + Ok(id) + } + + /// Tick — check all active missions and fire any that are due. + /// + /// For `Cron` cadence missions, checks `next_fire_at` against current time. + /// For `Manual` missions, this is a no-op. + /// Returns the IDs of threads spawned. + pub async fn tick(&self, _fallback_user_id: &str) -> Result, EngineError> { + let active_ids = self.active.read().await.clone(); + let mut spawned = Vec::new(); + let now = chrono::Utc::now(); + let cooldown = + chrono::Duration::from_std(FIRE_COOLDOWN).unwrap_or(chrono::Duration::zero()); + + // Opportunistic prune of `last_fire_attempt`: drop entries whose + // cooldown window has already elapsed. This catches stragglers from + // missions that were removed without going through the graceful + // pause/complete paths (e.g. crash recovery, direct store edits) so + // the map can never grow unbounded over a long-lived process. + { + let mut map = self.last_fire_attempt.write().await; + map.retain(|_, last| now.signed_duration_since(*last) < cooldown); + } + + for mid in active_ids { + // Per-mission error isolation: a transient store/load error or a + // single fire failure must not abort the entire tick — the other + // active missions still need their chance to fire on this cycle. + let mission = match self.store.load_mission(mid).await { + Ok(Some(m)) if m.status == MissionStatus::Active => m, + Ok(_) => continue, + Err(e) => { + debug!(mission_id = %mid, error = %e, "tick: failed to load mission; skipping"); + continue; + } + }; + + let should_fire = match &mission.cadence { + MissionCadence::Cron { .. } => { + // Fire if next_fire_at has passed + mission.next_fire_at.is_some_and(|next| next <= now) + } + MissionCadence::Manual => false, + MissionCadence::OnEvent { .. } + | MissionCadence::OnSystemEvent { .. } + | MissionCadence::Webhook { .. } => false, + }; + + if !should_fire { + continue; + } + + // In-memory cooldown — only armed when we can prove + // `fire_mission`'s post-spawn state didn't make scheduling + // progress. The detection: `fire_mission` writes the **same** + // instant to both `last_fire_attempt[mid]` (always, before + // save_mission) and the persisted `Mission.last_fire_at` (only + // when both `save_mission` succeeds *and* cron advance + // succeeded). The two paths that leave the values *unequal*: + // + // - `save_mission` failed → persisted `last_fire_at` still + // holds the OLD value (or None). + // - The cron expression couldn't be parsed (corrupt + // persisted state) → fire_mission deliberately leaves + // `last_fire_at` at the OLD value so this same mismatch + // arms the cooldown without inventing a new signal. + // + // On the success path the two values match and tick treats + // the cooldown as transparent — a normally-firing every-minute + // cron passes through the check regardless of how short the + // schedule is. Once the outcome processor reconciles + // `last_fire_at` back to the in-memory instant the mismatch + // resolves even before the 90 s window elapses. + // + // **Precision requirement (load-bearing):** the equality check + // requires the `Store` implementation to round-trip + // `DateTime` without precision loss. The bridge's + // `HybridStoreAdapter::load_mission` returns from an in-memory + // `HashMap` cache populated by `save_mission`, + // so it preserves nanosecond precision. JSON persistence via + // `serde` also preserves nanoseconds via RFC3339. A future + // backend that truncates timestamps (e.g. PostgreSQL TIMESTAMPTZ + // → microseconds) would silently break the success-path detection + // and arm the cooldown on every fire — the comparison would need + // to be relaxed to "within one microsecond" before that lands. + let on_cooldown = + self.last_fire_attempt + .read() + .await + .get(&mid) + .is_some_and(|in_mem_last| { + now.signed_duration_since(*in_mem_last) < cooldown + && mission.last_fire_at != Some(*in_mem_last) + }); + if on_cooldown { + debug!( + mission_id = %mid, + "tick: detected stale persisted last_fire_at after fire; suppressing re-fire until reconcile" + ); + continue; + } + + // Per-mission error isolation: a single fire failure must not + // abort the entire tick — the other active missions still need + // their chance on this cycle. `fire_mission` enforces + // `cooldown_secs` and `max_concurrent` independently of the cron + // next_fire_at, so a cron mission whose schedule fires faster + // than its cooldown will simply skip the intervening firings + // rather than backlog them. Cron missions are fired with the + // mission's own user_id so artifacts stay tenant-scoped. + match self.fire_mission(mid, &mission.user_id, None).await { + Ok(Some(tid)) => spawned.push(tid), + Ok(None) => {} + Err(e) => debug!( + mission_id = %mid, + error = %e, + "tick: fire_mission failed; continuing with remaining missions" + ), + } + } + + Ok(spawned) + } + + /// Count threads spawned by `mission` that are still in a non-terminal + /// state (anything other than `Done`/`Failed`). Used by `max_concurrent` + /// enforcement. Walks the in-memory thread cache; threads that the store + /// no longer knows about are treated as terminal. + async fn count_running_threads(&self, mission: &Mission) -> usize { + let mut running = 0; + for tid in mission.thread_history.iter().rev() { + match self.store.load_thread(*tid).await { + Ok(Some(thread)) => { + if !matches!(thread.state, ThreadState::Done | ThreadState::Failed) { + running += 1; + } + } + _ => continue, + } + } + running + } + + /// Returns `true` if `(mission_id, dedup_key)` was last seen within + /// `window_secs`. Updates the table to record `now` for the next call. + /// + /// Eviction is done **per entry** against this mission's own window — + /// never globally — because different missions can have different + /// `dedup_window_secs` values. A previous implementation called + /// `table.retain` with the current mission's window across the whole + /// table, which would silently drop fresh entries belonging to a + /// longer-window mission and cause duplicate firings. + async fn dedup_event(&self, mission_id: MissionId, dedup_key: &str, window_secs: u64) -> bool { + if window_secs == 0 { + return false; + } + let now = chrono::Utc::now(); + let window = chrono::Duration::seconds(window_secs as i64); + let mut table = self.dedup_table.write().await; + let key = (mission_id, dedup_key.to_string()); + match table.get(&key) { + Some(last) if now.signed_duration_since(*last) < window => { + // Within this mission's own window — duplicate. + true + } + _ => { + // Either no entry, or the entry has aged past this + // mission's window. Overwrite (or insert) and report + // first-seen. We deliberately do NOT touch entries for + // other missions. + table.insert(key, now); + false + } + } + } + + /// Test whether `text` matches `mission`'s OnEvent regex. Compiles the + /// pattern lazily on first match attempt and caches it. Patterns that + /// fail to compile (or exceed `MAX_EVENT_REGEX_SIZE`) are logged at + /// warn level and never match. + async fn event_regex_matches(&self, mission: &Mission, text: &str) -> bool { + let MissionCadence::OnEvent { event_pattern, .. } = &mission.cadence else { + return false; + }; + + // Cache hit fast path. + if let Some(re) = self.event_regex_cache.read().await.get(&mission.id) { + return re.is_match(text); + } + + // Compile under the write lock and double-check (another caller may + // have raced ahead and inserted the same key). + let mut cache = self.event_regex_cache.write().await; + if let Some(re) = cache.get(&mission.id) { + return re.is_match(text); + } + match regex::RegexBuilder::new(event_pattern) + .size_limit(MAX_EVENT_REGEX_SIZE) + .build() + { + Ok(re) => { + let matches = re.is_match(text); + cache.insert(mission.id, re); + matches + } + Err(error) => { + warn!( + mission_id = %mission.id, + pattern = %event_pattern, + error = %error, + "OnEvent mission regex failed to compile (or exceeded size limit); refusing to match" + ); + false + } + } + } + + /// Drop the compiled regex for `mission_id`, forcing recompile on the + /// next match attempt. Called when a mission's cadence changes or it is + /// deleted. + async fn evict_event_regex(&self, mission_id: MissionId) { + self.event_regex_cache.write().await.remove(&mission_id); + } + + /// Per-user global rate limiter — read-only check. Sliding window of + /// timestamps; returns `true` if a new fire is currently allowed. + /// Evicts expired entries from the user's window as a side effect, but + /// does NOT record a new entry — call [`record_user_rate`] only after + /// the fire has actually succeeded so a failed spawn cannot consume a + /// slot (otherwise sustained store errors would self-DoS the user). + /// + /// [`record_user_rate`]: Self::record_user_rate + async fn check_user_rate(&self, user_id: &str) -> bool { + let now = chrono::Utc::now(); + let window = chrono::Duration::from_std(self.rate_limit.window) + .unwrap_or_else(|_| chrono::Duration::seconds(self.rate_limit.window.as_secs() as i64)); + let cutoff = now - window; + + let mut log = self.user_fire_log.write().await; + let entries = log.entry(user_id.to_string()).or_default(); + while entries.front().is_some_and(|ts| *ts < cutoff) { + entries.pop_front(); + } + (entries.len() as u32) < self.rate_limit.max_fires + } + + /// Record a successful fire against the per-user rate window. Pair with + /// [`check_user_rate`] — call only after the spawn has actually + /// completed so failed fires don't consume a slot. + /// + /// [`check_user_rate`]: Self::check_user_rate + async fn record_user_rate(&self, user_id: &str) { + let now = chrono::Utc::now(); + let mut log = self.user_fire_log.write().await; + let entries = log.entry(user_id.to_string()).or_default(); + entries.push_back(now); + } + + /// Consult the budget gate (if attached). Returns `true` when the gate + /// is unattached or explicitly allows the fire. + async fn budget_allows(&self, user_id: &str, mission_id: MissionId) -> bool { + match self.budget_gate.as_ref() { + Some(gate) => gate.allow_mission_fire(user_id, mission_id).await, + None => true, + } + } + + fn spawn_mission_outcome_watcher( + &self, + mission_id: MissionId, + thread_id: ThreadId, + original_fire_at: chrono::DateTime, + ) { + let tm = Arc::clone(&self.thread_manager); + let store = Arc::clone(&self.store); + let notification_tx = self.notification_tx.clone(); + tokio::spawn(async move { + match tm.join_thread(thread_id).await { + Ok(outcome) => { + if let Err(e) = process_mission_outcome_and_notify( + &store, + mission_id, + thread_id, + &outcome, + ¬ification_tx, + Some(original_fire_at), + ) + .await + { + debug!(mission_id = %mission_id, "failed to process outcome: {e}"); + } + } + Err(e) => { + debug!(mission_id = %mission_id, "thread join failed: {e}"); + } + } + }); + } +} + +// ── Meta-prompt generation ─────────────────────────────────── + +/// Build the meta-prompt for a mission thread. +/// +/// Assembles the mission's goal, current focus, approach history, and +/// relevant project docs into a structured prompt that guides the thread. +/// Returns `true` if every `(key, value)` pair in `filters` matches the +/// payload's top-level field exactly. An empty filter map always matches. +/// `None` payload only matches an empty filter map. +fn payload_matches_filters( + filters: &HashMap, + payload: Option<&serde_json::Value>, +) -> bool { + if filters.is_empty() { + return true; + } + let Some(payload) = payload else { + return false; + }; + let Some(obj) = payload.as_object() else { + return false; + }; + filters + .iter() + .all(|(key, expected)| obj.get(key).is_some_and(|actual| actual == expected)) +} + +/// Compute a stable dedup key for an event payload. Hashes the canonicalized +/// JSON serialization with the standard library hasher (non-cryptographic but +/// sufficient for in-memory dedup of trusted host-sourced events). Empty/None +/// payloads collapse to a single fixed key so a flood of identical empty +/// events is suppressed. +fn payload_dedup_key(payload: Option<&serde_json::Value>) -> String { + use std::collections::hash_map::DefaultHasher; + use std::hash::{Hash, Hasher}; + let serialized = match payload { + Some(value) => serde_json::to_string(value).unwrap_or_default(), + None => String::new(), + }; + let mut hasher = DefaultHasher::new(); + serialized.hash(&mut hasher); + format!("{:016x}", hasher.finish()) +} + +fn build_meta_prompt( + mission: &Mission, + project_docs: &[MemoryDoc], + trigger_payload: &Option, + context_blocks: &[(String, String)], +) -> String { + let mut parts = Vec::new(); + + parts.push(format!( + "# Mission: {}\n\nGoal: {}", + mission.name, mission.goal + )); + + if let Some(description) = &mission.description { + parts.push(format!("\n{description}")); + } + + if let Some(criteria) = &mission.success_criteria { + parts.push(format!("Success criteria: {criteria}")); + } + + // Preloaded workspace context (`Mission.context_paths`). + if !context_blocks.is_empty() { + parts.push("\n## Loaded Context".into()); + for (path, content) in context_blocks { + parts.push(format!("### {path}\n\n{content}")); + } + } + + // Current focus + if let Some(focus) = &mission.current_focus { + parts.push(format!("\n## Current Focus\n{focus}")); + } else if mission.thread_history.is_empty() { + parts.push("\n## Current Focus\nThis is the first run. Start by understanding the goal and determining the first step.".into()); + } + + // Approach history + if !mission.approach_history.is_empty() { + parts.push("\n## Previous Approaches".into()); + for (i, approach) in mission.approach_history.iter().enumerate() { + parts.push(format!("{}. {approach}", i + 1)); + } + } + + // Project knowledge (from reflection docs) + if !project_docs.is_empty() { + parts.push("\n## Knowledge from Prior Threads".into()); + for doc in project_docs { + let label = format!("{:?}", doc.doc_type).to_uppercase(); + let content: String = doc.content.chars().take(500).collect(); + let truncated = if doc.content.chars().count() > 500 { + "..." + } else { + "" + }; + parts.push(format!("[{label}] {}: {content}{truncated}", doc.title)); + } + } + + // Trigger payload + if let Some(payload) = trigger_payload { + let payload_str = serde_json::to_string_pretty(payload).unwrap_or_default(); + let preview: String = payload_str.chars().take(1000).collect(); + parts.push(format!("\n## Trigger Payload\n```json\n{preview}\n```")); + } + + // Thread count + parts.push(format!( + "\nThis is thread #{} for this mission.", + mission.thread_history.len() + 1 + )); + + // Instructions + parts.push( + "\n## Instructions\nBased on the above context, take the next step toward the goal. \ +Use tools to gather information, analyze data, or take actions. \ +When done, call FINAL() with your response. Include:\n\ +1. What you accomplished in this step\n\ +2. What the next focus should be (for the next thread)\n\ +3. Whether the goal has been achieved (yes/no)" + .into(), + ); + + parts.join("\n") +} + +/// Process a completed mission thread's outcome. +/// +/// Extracts next_focus from the FINAL() response and updates the mission. +/// For self-improvement missions (metadata contains `"self_improvement": true`), +/// also processes prompt overlay additions and fix pattern updates. +#[cfg(test)] +async fn process_mission_outcome( + store: &Arc, + mission_id: MissionId, + thread_id: ThreadId, + outcome: &ThreadOutcome, +) -> Result<(), EngineError> { + let (notification_tx, _) = tokio::sync::broadcast::channel(1); + process_mission_outcome_and_notify( + store, + mission_id, + thread_id, + outcome, + ¬ification_tx, + None, + ) + .await +} + +async fn process_mission_outcome_and_notify( + store: &Arc, + mission_id: MissionId, + thread_id: ThreadId, + outcome: &ThreadOutcome, + notification_tx: &tokio::sync::broadcast::Sender, + original_fire_at: Option>, +) -> Result<(), EngineError> { + let mut mission = match store.load_mission(mission_id).await? { + Some(m) => m, + None => return Ok(()), + }; + + // Reconcile fire-accounting fields that `fire_mission` failed to persist. + // + // `fire_mission` is best-effort about its post-spawn `save_mission`: a + // transient store error there leaves the persisted mission missing this + // thread's record (`thread_history`, `threads_today`, `last_fire_at`, + // and — for cron cadences — the advanced `next_fire_at`). The in-memory + // `last_fire_attempt` cooldown holds the runaway-re-fire path closed + // for ~90 s, but once the cooldown elapses tick would otherwise re-fire + // against the stale persisted state. The outcome processor is the + // natural reconciliation point: by the time we run, the thread has + // completed and we know exactly which `thread_id` should be present. + // Append idempotently — repeated invocations or replays are safe — and + // immediately overwrite our save below, achieving eventual consistency + // for transient store failures even after retries are exhausted. + let needs_reconcile = !mission.thread_history.contains(&thread_id); + if needs_reconcile { + debug!( + mission_id = %mission_id, + thread_id = %thread_id, + "outcome processor: reconciling fire-accounting fields missing from persisted mission (fire_mission save likely failed)" + ); + // `record_thread` also bumps `updated_at`; matches the fire_mission + // path so the two routes don't diverge on field-mutation patterns. + mission.record_thread(thread_id); + mission.threads_today = mission.threads_today.saturating_add(1); + // Reconcile `last_fire_at` back to the **original** fire instant + // when we know it (passed through from fire_mission via the + // outcome watcher). Otherwise fall back to `now` as a conservative + // approximation. Using the original instant matters for users with + // a configured `cooldown_secs`: if the thread ran for N seconds, + // a `now`-based reconcile would extend the user's cooldown window + // by N, gradually drifting the schedule. + mission.last_fire_at = Some(original_fire_at.unwrap_or_else(chrono::Utc::now)); + // For cron missions, advance `next_fire_at` so the ticker doesn't + // immediately re-fire against the stale schedule. Use the lenient + // `next_cron_fire` (not `_required`) — a corrupt expression here + // should not block the outcome save. + if let MissionCadence::Cron { + ref expression, + ref timezone, + } = mission.cadence + { + let now = chrono::Utc::now(); + let needs_advance = mission.next_fire_at.is_none_or(|next| next <= now); + if needs_advance { + match next_cron_fire(expression, timezone.as_ref()) { + Ok(Some(next)) => mission.next_fire_at = Some(next), + Ok(None) => debug!( + mission_id = %mission_id, + expression = %expression, + "reconcile: cron has no upcoming fire time; leaving next_fire_at unset" + ), + Err(e) => debug!( + mission_id = %mission_id, + expression = %expression, + error = %e, + "reconcile: failed to recompute next_fire_at; leaving as-is" + ), + } + } + } + } + + // Build notification fields while processing the outcome. + let mut notify_response: Option = None; + let mut is_error = false; + + match outcome { + ThreadOutcome::Completed { + response: Some(text), + } => { + // Try to extract next focus and goal status from the response + let lower = text.to_lowercase(); + + // Check if goal achieved + if lower.contains("goal has been achieved: yes") + || lower.contains("goal achieved: yes") + || lower.contains("mission complete") + { + debug!(mission_id = %mission_id, "goal achieved — completing mission"); + mission.status = MissionStatus::Completed; + } + + // Extract next focus (look for "next focus:" pattern) + if let Some(focus_start) = lower.find("next focus:") { + let after = &text[focus_start + "next focus:".len()..]; + let next_focus: String = after.lines().next().unwrap_or("").trim().to_string(); + if !next_focus.is_empty() { + mission.current_focus = Some(next_focus); + } + } + + // Record approach (full response — LLM output is never truncated) + mission.approach_history.push(text.clone()); + notify_response = Some(text.clone()); + + // If this is a self-improvement mission, process structured output + if is_self_improvement_mission(&mission) + && let Err(e) = process_self_improvement_output(store, &mission, text).await + { + debug!( + mission_id = %mission_id, + "failed to process self-improvement output: {e}" + ); + } + + if is_skill_repair_mission(&mission) + && let Err(e) = process_skill_repair_output(store, &mission, text).await + { + debug!( + mission_id = %mission_id, + "failed to process skill-repair output: {e}" + ); + } + } + ThreadOutcome::Completed { response: None } => {} + ThreadOutcome::Failed { error } => { + mission.approach_history.push(format!("FAILED: {error}")); + notify_response = Some(format!("Mission failed: {error}")); + is_error = true; + } + ThreadOutcome::MaxIterations => { + mission + .approach_history + .push("Hit max iterations without completing".into()); + notify_response = Some("Mission thread hit max iterations without completing".into()); + is_error = true; + } + _ => {} + } + + // Emit notification if there are channels to notify. + if !mission.notify_channels.is_empty() && notify_response.is_some() { + // Truncate before broadcasting. Mission threads can produce + // arbitrarily long output (especially full-job missions); a multi-MB + // notification is unusable in any chat surface and can OOM Slack/ + // Discord adapters that buffer outbound bodies. The full text is + // already preserved untruncated in `mission.approach_history`. + let response = notify_response.map(|text| truncate_notification_text(&text)); + let notification = MissionNotification { + mission_id, + mission_name: mission.name.clone(), + thread_id, + user_id: mission.user_id.clone(), + notify_channels: mission.notify_channels.clone(), + notify_user: mission.notify_user.clone(), + response, + is_error, + }; + // Best-effort: ignore send errors (no subscribers = no problem). + let _ = notification_tx.send(notification); + } + + mission.updated_at = chrono::Utc::now(); + store.save_mission(&mission).await +} + +/// UTF-8-safe ellipsis truncation for mission notification responses. +/// +/// Mirrors the v1 routine engine's `truncate` helper (which uses +/// `floor_char_boundary` from the host `util` module). The engine crate +/// has no `util` so we inline a small helper. The full response text is +/// always preserved untruncated in `Mission.approach_history`; truncation +/// here only affects what is broadcast to notify_channels. +const MAX_NOTIFICATION_RESPONSE_BYTES: usize = 4000; + +fn truncate_notification_text(text: &str) -> String { + if text.len() <= MAX_NOTIFICATION_RESPONSE_BYTES { + return text.to_string(); + } + // Walk back from the byte cap to the nearest char boundary so we + // never split a multi-byte UTF-8 sequence. `is_char_boundary(0)` + // is always true so the loop is bounded. + let mut end = MAX_NOTIFICATION_RESPONSE_BYTES; + while !text.is_char_boundary(end) { + end -= 1; + } + format!("{}…", &text[..end]) // safety: end walked back to a valid char boundary above +} + +/// Check if a mission is the self-improvement mission. +fn is_self_improvement_mission(mission: &Mission) -> bool { + mission + .metadata + .get("self_improvement") + .and_then(|v| v.as_bool()) + .unwrap_or(false) +} + +/// Check if a mission is the skill-repair mission. +fn is_skill_repair_mission(mission: &Mission) -> bool { + mission + .metadata + .get("skill_repair") + .and_then(|v| v.as_bool()) + .unwrap_or(false) +} + +/// Process output from a self-improvement mission thread. +/// +/// Two paths: +/// 1. The agent used tools directly (memory_write for prompt overlay, shell for +/// code fixes) — in this case the FINAL() response is just a summary and +/// there is nothing extra to do here. +/// 2. The agent returned structured JSON with `prompt_additions` and/or +/// `fix_patterns` — we apply those to the Store. +/// +/// This function handles path 2. Path 1 is handled by the tools themselves. +async fn process_self_improvement_output( + store: &Arc, + mission: &Mission, + response: &str, +) -> Result<(), EngineError> { + use crate::executor::prompt::{PREAMBLE_OVERLAY_TITLE, PROMPT_OVERLAY_TAG}; + use crate::types::memory::{DocType, MemoryDoc}; + + // Try to extract JSON from the response. If the agent used tools directly + // (the preferred autoresearch-style path), there's no JSON and we return + // early — the work was already done via tool calls. + let json_val = match extract_json_from_response(response) { + Some(v) => v, + None => { + debug!( + "self-improvement: no structured JSON in response (agent likely used tools directly)" + ); + return Ok(()); + } + }; + + let project_id = mission.project_id; + + // Check if self-modification is allowed before applying prompt/orchestrator changes + let allow_self_modify = std::env::var("ORCHESTRATOR_SELF_MODIFY") + .map(|v| v == "true" || v == "1") + .unwrap_or(false); + + // Process prompt additions + if let Some(additions) = json_val.get("prompt_additions").and_then(|v| v.as_array()) + && !additions.is_empty() + { + if !allow_self_modify { + debug!( + "self-improvement: skipping prompt additions — ORCHESTRATOR_SELF_MODIFY is disabled" + ); + return Ok(()); + } + + let new_rules: Vec = additions + .iter() + .filter_map(|v| v.as_str().map(String::from)) + .collect(); + + if !new_rules.is_empty() { + // Load or create the prompt overlay doc + let docs = store.list_shared_memory_docs(project_id).await?; + let existing = docs.iter().find(|d| { + d.title == PREAMBLE_OVERLAY_TITLE + && d.tags.contains(&PROMPT_OVERLAY_TAG.to_string()) + }); + + let mut overlay = if let Some(doc) = existing { + doc.clone() + } else { + MemoryDoc::new( + project_id, + shared_owner_id(), + DocType::Note, + PREAMBLE_OVERLAY_TITLE, + "", + ) + .with_tags(vec![PROMPT_OVERLAY_TAG.to_string()]) + }; + + // Append new rules + for rule in &new_rules { + if !overlay.content.is_empty() { + overlay.content.push('\n'); + } + overlay.content.push_str(rule); + } + overlay.updated_at = chrono::Utc::now(); + + store.save_memory_doc(&overlay).await?; + debug!( + rules_added = new_rules.len(), + "self-improvement: updated prompt overlay" + ); + } + } + + // Process fix patterns + if let Some(patterns) = json_val.get("fix_patterns").and_then(|v| v.as_array()) + && !patterns.is_empty() + { + let docs = store.list_shared_memory_docs(project_id).await?; + let existing = docs.iter().find(|d| { + d.title == FIX_PATTERN_DB_TITLE && d.tags.contains(&FIX_PATTERN_DB_TAG.to_string()) + }); + + let mut pattern_doc = if let Some(doc) = existing { + doc.clone() + } else { + MemoryDoc::new( + project_id, + shared_owner_id(), + DocType::Note, + FIX_PATTERN_DB_TITLE, + SEED_FIX_PATTERNS, + ) + .with_tags(vec![FIX_PATTERN_DB_TAG.to_string()]) + }; + + for pattern in patterns { + let p = pattern + .get("pattern") + .and_then(|v| v.as_str()) + .unwrap_or(""); + let s = pattern + .get("strategy") + .and_then(|v| v.as_str()) + .unwrap_or(""); + let l = pattern + .get("location") + .and_then(|v| v.as_str()) + .unwrap_or(""); + if !p.is_empty() { + pattern_doc + .content + .push_str(&format!("\n| {p} | {s} | {l} |")); + } + } + pattern_doc.updated_at = chrono::Utc::now(); + + store.save_memory_doc(&pattern_doc).await?; + debug!( + patterns_added = patterns.len(), + "self-improvement: updated fix pattern database" + ); + } + + Ok(()) +} + +/// Try to extract a JSON object from a response string. +/// +/// Looks for `{...}` in the text, trying the whole string first, +/// then searching for embedded JSON. +fn extract_json_from_response(response: &str) -> Option { + // Try parsing the whole response as JSON + if let Ok(v) = serde_json::from_str::(response) + && v.is_object() + { + return Some(v); + } + + // Search for embedded JSON object + let start = response.find('{')?; + let end = response.rfind('}')?; + if end <= start { + return None; + } + let candidate = &response[start..=end]; + serde_json::from_str::(candidate) + .ok() + .filter(|v| v.is_object()) +} + +#[derive(Debug, Deserialize)] +struct SkillRepairMissionOutput { + doc_id: DocId, + repair_type: SkillRepairType, + updated_content: String, + #[serde(default)] + summary: String, + #[serde(default)] + description: Option, + #[serde(default)] + activation: Option, + #[serde(default)] + code_snippets: Option>, +} + +async fn process_skill_repair_output( + store: &Arc, + mission: &Mission, + response: &str, +) -> Result<(), EngineError> { + let json_val = match extract_json_from_response(response) { + Some(v) => v, + None => { + debug!("skill-repair: no structured JSON in response"); + return Ok(()); + } + }; + let repair: SkillRepairMissionOutput = + serde_json::from_value(json_val).map_err(|e| EngineError::Skill { + reason: format!("invalid skill-repair output: {e}"), + })?; + + let Some(triggered_skill) = triggered_skill_provenance(mission, repair.doc_id) else { + return Err(EngineError::Skill { + reason: if has_skill_trigger_payload(mission) { + format!( + "skill-repair attempted to modify untriggered skill {}", + repair.doc_id.0 + ) + } else { + "skill-repair requires an active skill trigger payload".into() + }, + }); + }; + if repair.updated_content.trim().is_empty() { + return Err(EngineError::Skill { + reason: format!( + "skill-repair produced empty updated_content for skill {}", + repair.doc_id.0 + ), + }); + } + + let existing = + store + .load_memory_doc(repair.doc_id) + .await? + .ok_or_else(|| EngineError::Skill { + reason: format!("skill doc not found: {}", repair.doc_id.0), + })?; + if existing.project_id != mission.project_id { + return Err(EngineError::Skill { + reason: format!( + "skill-repair attempted to modify skill {} outside mission project", + repair.doc_id.0 + ), + }); + } + if !existing.is_owned_by(&mission.user_id) { + return Err(EngineError::AccessDenied { + user_id: mission.user_id.clone(), + entity: format!("skill {}", repair.doc_id.0), + }); + } + if existing.doc_type != DocType::Skill { + return Err(EngineError::Skill { + reason: format!( + "skill-repair attempted to modify non-skill doc {} ({:?})", + repair.doc_id.0, existing.doc_type + ), + }); + } + serde_json::from_value::(existing.metadata.clone()).map_err(|e| { + EngineError::Skill { + reason: format!("invalid skill metadata for {}: {e}", repair.doc_id.0), + } + })?; + let from_version = triggered_skill.version; + let source_thread_id = mission + .last_trigger_payload + .as_ref() + .and_then(|payload| payload.get("source_thread_id")) + .and_then(|value| value.as_str()) + .map(ToString::to_string); + let summary = if repair.summary.trim().is_empty() { + format!("Applied {:?} repair", repair.repair_type) + } else { + repair.summary.clone() + }; + + let tracker = SkillTracker::new(Arc::clone(store)); + tracker + .update_skill( + repair.doc_id, + repair.updated_content, + Some(triggered_skill.version), + move |meta| { + if let Some(description) = repair.description { + meta.description = description; + } + if let Some(activation) = repair.activation { + meta.activation = activation; + } + if let Some(code_snippets) = repair.code_snippets { + meta.code_snippets = code_snippets; + } + meta.repairs.push(SkillRepairRecord { + source_thread_id, + from_version, + to_version: meta.version, + repair_type: repair.repair_type, + summary, + repaired_at: Some(chrono::Utc::now()), + }); + if meta.repairs.len() > 10 { + let keep_from = meta.repairs.len() - 10; + meta.repairs.drain(0..keep_from); + } + }, + ) + .await +} + +fn has_skill_trigger_payload(mission: &Mission) -> bool { + mission + .last_trigger_payload + .as_ref() + .and_then(|payload| payload.get("active_skills")) + .and_then(|value| value.as_array()) + .is_some_and(|skills| !skills.is_empty()) +} + +fn triggered_skill_provenance(mission: &Mission, doc_id: DocId) -> Option { + mission + .last_trigger_payload + .as_ref() + .and_then(|payload| payload.get("active_skills")) + .cloned() + .and_then(|value| serde_json::from_value::>(value).ok()) + .and_then(|skills| skills.into_iter().find(|skill| skill.doc_id == doc_id)) +} + +/// Collects error messages and deduplicated observed action names in a single +/// pass over `thread.events`. Previous implementation used separate passes +/// which is wasteful for threads with large event logs. +fn collect_errors_and_actions(thread: &Thread) -> (Vec, Vec) { + let mut error_messages = Vec::new(); + let mut actions = Vec::new(); + let mut seen = HashSet::new(); + + for event in &thread.events { + match &event.kind { + crate::types::event::EventKind::ActionFailed { + action_name, error, .. + } => { + if !is_recoverable_action_failure(error) && error_messages.len() < 10 { + error_messages.push(format!("{action_name}: {error}")); + } + if seen.insert(action_name.clone()) { + actions.push(action_name.clone()); + } + } + crate::types::event::EventKind::ActionExecuted { action_name, .. } => { + if seen.insert(action_name.clone()) { + actions.push(action_name.clone()); + } + } + _ => {} + } + } + + (error_messages, actions) +} + +fn learning_terminal_state( + event_kind: &crate::types::event::EventKind, +) -> Option { + match event_kind { + crate::types::event::EventKind::StateChanged { + to: crate::types::thread::ThreadState::Done, + .. + } => Some(crate::types::thread::ThreadState::Done), + crate::types::event::EventKind::StateChanged { + to: crate::types::thread::ThreadState::Failed, + .. + } => Some(crate::types::thread::ThreadState::Failed), + _ => None, + } +} + +fn should_count_for_conversation_insights( + terminal_state: crate::types::thread::ThreadState, +) -> bool { + terminal_state == crate::types::thread::ThreadState::Done +} + +fn has_action_failures(thread: &Thread) -> bool { + thread.events.iter().any(|event| match &event.kind { + crate::types::event::EventKind::ActionFailed { error, .. } => { + !is_recoverable_action_failure(error) + } + _ => false, + }) +} + +fn is_recoverable_auth_failure_text(text: &str) -> bool { + text.to_ascii_lowercase() + .contains("authentication required for credential ") +} + +fn is_recoverable_action_failure(error: &str) -> bool { + is_recoverable_auth_failure_text(error) +} + +fn action_params_summary(event: &crate::types::event::ThreadEvent) -> Option<&str> { + match &event.kind { + crate::types::event::EventKind::ActionExecuted { params_summary, .. } + | crate::types::event::EventKind::ActionFailed { params_summary, .. } => { + params_summary.as_deref() + } + _ => None, + } +} + +fn contains_word(haystack: &str, word: &str) -> bool { + for (start, _) in haystack.match_indices(word) { + let before_ok = start == 0 || haystack.as_bytes()[start - 1].is_ascii_whitespace(); + let end = start + word.len(); + let after_ok = end == haystack.len() || haystack.as_bytes()[end].is_ascii_whitespace(); + if before_ok && after_ok { + return true; + } + } + false +} + +fn has_shell_verification_action(thread: &Thread) -> bool { + const PHRASE_PATTERNS: &[&str] = &[ + "cargo test", + "pytest", + "npm test", + "pnpm test", + "yarn test", + "go test", + "git diff", + "git status", + "gh pr view", + "gh issue view", + "cat ", + "head ", + "tail ", + "grep ", + "rg ", + "find ", + "stat ", + ]; + const WORD_PATTERNS: &[&str] = &["ls", "diff", "status", "view", "show"]; + + thread.events.iter().any(|event| match &event.kind { + crate::types::event::EventKind::ActionExecuted { action_name, .. } + if action_name == "shell" => + { + action_params_summary(event) + .map(|summary| { + let lower = summary.to_lowercase(); + PHRASE_PATTERNS + .iter() + .any(|pattern| lower.contains(pattern)) + || WORD_PATTERNS.iter().any(|word| contains_word(&lower, word)) + }) + .unwrap_or(false) + } + crate::types::event::EventKind::ActionFailed { action_name, .. } + if action_name == "shell" => + { + action_params_summary(event) + .map(|summary| { + let lower = summary.to_lowercase(); + PHRASE_PATTERNS + .iter() + .any(|pattern| lower.contains(pattern)) + || WORD_PATTERNS.iter().any(|word| contains_word(&lower, word)) + }) + .unwrap_or(false) + } + _ => false, + }) +} + +fn has_mutating_shell_or_git_action(thread: &Thread) -> bool { + const PHRASE_PATTERNS: &[&str] = &[ + "apply_patch", + "git commit", + "git push", + "git pull", + "git merge", + "git rebase", + "git cherry-pick", + "git revert", + "git reset", + "git checkout", + "git switch", + "cargo fmt", + "rustfmt", + "npm install", + "pnpm install", + "yarn install", + "mkdir ", + "rm ", + "mv ", + "cp ", + "touch ", + "tee ", + "sed -i", + "perl -pi", + ]; + const WORD_PATTERNS: &[&str] = &[ + "write", "create", "delete", "remove", "rename", "patch", "install", "format", + ]; + + thread.events.iter().any(|event| match &event.kind { + crate::types::event::EventKind::ActionExecuted { action_name, .. } + | crate::types::event::EventKind::ActionFailed { action_name, .. } + if action_name == "shell" || action_name == "git" => + { + action_params_summary(event) + .map(|summary| { + let lower = summary.to_lowercase(); + PHRASE_PATTERNS + .iter() + .any(|pattern| lower.contains(pattern)) + || WORD_PATTERNS.iter().any(|word| contains_word(&lower, word)) + }) + .unwrap_or(false) + } + _ => false, + }) +} + +fn infer_skill_repair_hints( + thread: &Thread, + trace: &ExecutionTrace, + error_messages: &[String], + observed_actions: &[String], +) -> Vec { + let mut hints = Vec::new(); + let mut push_hint = |hint| { + if !hints.contains(&hint) { + hints.push(hint); + } + }; + + let lower_signals = error_messages + .iter() + .map(|message| message.to_lowercase()) + .chain( + trace + .issues + .iter() + .filter(|issue| { + !(issue.category == "tool_error" + && is_recoverable_auth_failure_text(&issue.description)) + }) + .map(|issue| issue.description.to_lowercase()), + ) + .collect::>(); + + let recoverable_auth_failures = thread + .events + .iter() + .filter_map(|event| { + if let crate::types::event::EventKind::ActionFailed { error, .. } = &event.kind + && is_recoverable_auth_failure_text(error) + { + Some(error.to_lowercase()) + } else { + None + } + }) + .collect::>(); + + if lower_signals + .iter() + .chain(recoverable_auth_failures.iter()) + .any(|message| { + ["auth", "login", "token", "credential", "permission denied"] + .iter() + .any(|needle| message.contains(needle)) + }) + { + push_hint(SkillRepairType::MissingPrerequisite); + } + + if lower_signals.iter().any(|message| { + [ + "command not found", + "no such file", + "not found", + "could not find", + "unknown file", + "unknown path", + ] + .iter() + .any(|needle| message.contains(needle)) + }) { + push_hint(SkillRepairType::StaleCommandPath); + } + + let mutating_actions = observed_actions.iter().any(|action| { + matches!( + action.as_str(), + "write_file" | "apply_patch" | "memory_write" | "skill_install" | "skill_remove" + ) + }) || has_mutating_shell_or_git_action(thread); + let verification_actions = observed_actions.iter().any(|action| { + matches!( + action.as_str(), + "read_file" | "memory_read" | "memory_search" | "cargo_test" | "pytest" + ) + }) || has_shell_verification_action(thread); + if mutating_actions && !verification_actions { + push_hint(SkillRepairType::MissingVerification); + } + + if !error_messages.is_empty() && thread.state == crate::types::thread::ThreadState::Done { + push_hint(SkillRepairType::MissingPitfall); + } + + hints +} + +fn build_skill_gap_payload( + thread: &Thread, + trace: &ExecutionTrace, + active_skills: &[ActiveSkillProvenance], +) -> Option { + let (error_messages, observed_actions) = collect_errors_and_actions(thread); + let repair_hints = infer_skill_repair_hints(thread, trace, &error_messages, &observed_actions); + if repair_hints.is_empty() { + return None; + } + + let issues: Vec = trace + .issues + .iter() + .map(|issue| { + serde_json::json!({ + "severity": format!("{:?}", issue.severity), + "category": issue.category.clone(), + "description": issue.description.clone(), + "step": issue.step, + }) + }) + .collect(); + + Some(serde_json::json!({ + "source_thread_id": thread.id.0.to_string(), + "goal": thread.goal, + "active_skills": active_skills, + "issues": issues, + "error_messages": error_messages, + "observed_actions": observed_actions, + "repair_hints": repair_hints, + })) +} + +fn thread_completed_successfully(thread: &Thread, trace: &ExecutionTrace) -> bool { + thread.state == crate::types::thread::ThreadState::Done + && !has_action_failures(thread) + && trace + .issues + .iter() + .all(|issue| issue.severity != IssueSeverity::Error) +} + +/// The goal for the self-improvement mission (autoresearch-style program). +/// +/// This is the "program.md" — a concrete, step-by-step prompt that tells the +/// agent exactly what to do. Inspired by karpathy/autoresearch: the entire +/// research org is a markdown file with an explicit loop. +const SELF_IMPROVEMENT_GOAL: &str = include_str!("../../prompts/mission_self_improvement.md"); + +/// Well-known title for the fix pattern database. +pub const FIX_PATTERN_DB_TITLE: &str = "fix_pattern_database"; + +/// Well-known tag for the fix pattern database. +pub const FIX_PATTERN_DB_TAG: &str = "fix_patterns"; + +/// The goal for the skill extraction mission. +const SKILL_EXTRACTION_GOAL: &str = include_str!("../../prompts/mission_skill_extraction.md"); + +/// The goal for the skill-repair mission. +const SKILL_REPAIR_GOAL: &str = include_str!("../../prompts/mission_skill_repair.md"); + +/// The goal for the conversation insights mission. +const CONVERSATION_INSIGHTS_GOAL: &str = + include_str!("../../prompts/mission_conversation_insights.md"); + +/// The goal for the expected-behavior mission (user feedback loop). +const EXPECTED_BEHAVIOR_GOAL: &str = include_str!("../../prompts/mission_expected_behavior.md"); + +/// Seed content for the fix pattern database. +const SEED_FIX_PATTERNS: &str = "\ +| Trace pattern | Fix strategy | Location pattern | +|---|---|---| +| Tool X not found | Add name alias or prompt hint about correct name | prompt overlay or effect_adapter | +| TypeError: str indices must be integers | Parse JSON before wrapping | Where tool output is converted | +| NameError: name 'X' not defined | Add prompt hint about using state dict | prompt overlay | +| byte index N is not a char boundary | Replace byte slicing with chars().take(N) | Code that slices strings | +| Model calls nonexistent tool | Add prompt rule listing correct tool name | prompt overlay | +| Model ignores tool results | Improve output metadata format | prompt overlay | +| Excessive steps (>5) for simple task | Add prompt rule or fix tool schema | prompt overlay | +| Code error in REPL output | Add prompt hint about correct API usage | prompt overlay |"; + +#[cfg(test)] +mod tests { + use super::*; + + use std::collections::HashMap; + use std::sync::Mutex; + + use crate::capability::lease::LeaseManager; + use crate::capability::policy::PolicyEngine; + use crate::capability::registry::CapabilityRegistry; + use crate::traits::effect::EffectExecutor; + use crate::traits::llm::{LlmCallConfig, LlmOutput}; + use crate::traits::store::Store; + use crate::types::capability::{ActionDef, CapabilityLease}; + use crate::types::error::EngineError; + use crate::types::event::ThreadEvent; + use crate::types::memory::{DocId, DocType, MemoryDoc}; + use crate::types::mission::{Mission, MissionCadence, MissionId, MissionStatus}; + use crate::types::project::{Project, ProjectId}; + use crate::types::step::StepId; + use crate::types::step::{ActionResult, LlmResponse, Step, TokenUsage}; + use crate::types::thread::{ActiveSkillProvenance, Thread, ThreadId, ThreadState, ThreadType}; + use ironclaw_skills::SkillTrust; + use ironclaw_skills::types::ActivationCriteria; + use ironclaw_skills::v2::{SkillMetrics, SkillRepairType, V2SkillMetadata, V2SkillSource}; + + // ── TestStore — in-memory Store that persists missions ─── + + struct TestStore { + threads: tokio::sync::RwLock>, + missions: tokio::sync::RwLock>, + docs: tokio::sync::RwLock>, + /// Optional gate that blocks the next `save_mission` call until + /// the test releases it. Used by `fire_mission_arms_cooldown_before_save` + /// to deterministically observe the in-flight save state. + save_mission_gate: tokio::sync::Mutex>>, + /// Notified when `save_mission` enters the gated wait so the test + /// knows save is in progress (rather than not yet called). + save_mission_started: tokio::sync::Notify, + } + + impl TestStore { + fn new() -> Self { + Self { + threads: tokio::sync::RwLock::new(HashMap::new()), + missions: tokio::sync::RwLock::new(HashMap::new()), + docs: tokio::sync::RwLock::new(Vec::new()), + save_mission_gate: tokio::sync::Mutex::new(None), + save_mission_started: tokio::sync::Notify::new(), + } + } + + /// Block the next `save_mission` call. Returns a sender the test + /// must signal once it's done observing the in-flight state. + async fn block_next_save_mission(&self) -> tokio::sync::oneshot::Sender<()> { + let (tx, rx) = tokio::sync::oneshot::channel(); + *self.save_mission_gate.lock().await = Some(rx); + tx + } + } + + fn make_skill_doc(project_id: ProjectId, user_id: &str, name: &str) -> MemoryDoc { + let meta = V2SkillMetadata { + name: name.to_string(), + version: 1, + description: format!("{name} description"), + activation: ActivationCriteria::default(), + source: V2SkillSource::Extracted, + trust: SkillTrust::Trusted, + code_snippets: vec![], + metrics: SkillMetrics::default(), + parent_version: None, + revisions: vec![], + repairs: vec![], + content_hash: "sha256:test".to_string(), + }; + + let mut doc = MemoryDoc::new( + project_id, + user_id, + DocType::Skill, + format!("skill:{name}"), + "Original skill content", + ); + doc.metadata = serde_json::to_value(&meta).expect("serialize test skill metadata"); + doc + } + + #[async_trait::async_trait] + impl Store for TestStore { + // ── Thread (minimal — save/load needed by ThreadManager) ── + async fn save_thread(&self, thread: &Thread) -> Result<(), EngineError> { + self.threads.write().await.insert(thread.id, thread.clone()); + Ok(()) + } + async fn load_thread(&self, id: ThreadId) -> Result, EngineError> { + Ok(self.threads.read().await.get(&id).cloned()) + } + async fn list_threads(&self, _: ProjectId, _: &str) -> Result, EngineError> { + Ok(vec![]) + } + async fn update_thread_state( + &self, + _: ThreadId, + _: ThreadState, + ) -> Result<(), EngineError> { + Ok(()) + } + + // ── Step (noop) ── + async fn save_step(&self, _: &Step) -> Result<(), EngineError> { + Ok(()) + } + async fn load_steps(&self, _: ThreadId) -> Result, EngineError> { + Ok(vec![]) + } + + // ── Event (noop) ── + async fn append_events(&self, _: &[ThreadEvent]) -> Result<(), EngineError> { + Ok(()) + } + async fn load_events(&self, _: ThreadId) -> Result, EngineError> { + Ok(vec![]) + } + + // ── Project (noop) ── + async fn save_project(&self, _: &Project) -> Result<(), EngineError> { + Ok(()) + } + async fn load_project(&self, _: ProjectId) -> Result, EngineError> { + Ok(None) + } + + // ── MemoryDoc ── + async fn save_memory_doc(&self, doc: &MemoryDoc) -> Result<(), EngineError> { + let mut docs = self.docs.write().await; + docs.retain(|d| d.id != doc.id); + docs.push(doc.clone()); + Ok(()) + } + async fn load_memory_doc(&self, id: DocId) -> Result, EngineError> { + Ok(self.docs.read().await.iter().find(|d| d.id == id).cloned()) + } + async fn list_memory_docs( + &self, + project_id: ProjectId, + _user_id: &str, + ) -> Result, EngineError> { + Ok(self + .docs + .read() + .await + .iter() + .filter(|d| d.project_id == project_id) + .cloned() + .collect()) + } + + // ── Lease (noop) ── + async fn save_lease(&self, _: &CapabilityLease) -> Result<(), EngineError> { + Ok(()) + } + async fn load_active_leases( + &self, + _: ThreadId, + ) -> Result, EngineError> { + Ok(vec![]) + } + async fn revoke_lease( + &self, + _: crate::types::capability::LeaseId, + _: &str, + ) -> Result<(), EngineError> { + Ok(()) + } + + // ── Mission (fully implemented) ── + async fn save_mission(&self, mission: &Mission) -> Result<(), EngineError> { + // Honor the test gate if one is installed. Take the receiver out + // of the slot so subsequent saves are unblocked by default — the + // gate is one-shot per `block_next_save_mission` call. + let gate = self.save_mission_gate.lock().await.take(); + if let Some(rx) = gate { + self.save_mission_started.notify_one(); + let _ = rx.await; + } + self.missions + .write() + .await + .insert(mission.id, mission.clone()); + Ok(()) + } + async fn load_mission(&self, id: MissionId) -> Result, EngineError> { + Ok(self.missions.read().await.get(&id).cloned()) + } + async fn list_missions( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + Ok(self + .missions + .read() + .await + .values() + .filter(|m| m.project_id == project_id && m.user_id == user_id) + .cloned() + .collect()) + } + async fn list_all_missions( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + Ok(self + .missions + .read() + .await + .values() + .filter(|m| m.project_id == project_id) + .cloned() + .collect()) + } + async fn update_mission_status( + &self, + id: MissionId, + status: MissionStatus, + ) -> Result<(), EngineError> { + if let Some(mission) = self.missions.write().await.get_mut(&id) { + mission.status = status; + } + Ok(()) + } + } + + // ── MockLlm — returns canned text responses ───────────── + + struct MockLlm { + responses: Mutex>, + } + + impl MockLlm { + fn text(msg: &str) -> Arc { + Arc::new(Self { + responses: Mutex::new(vec![LlmOutput { + response: LlmResponse::Text(msg.into()), + usage: TokenUsage::default(), + }]), + }) + } + } + + #[async_trait::async_trait] + impl crate::traits::llm::LlmBackend for MockLlm { + async fn complete( + &self, + _: &[crate::types::message::ThreadMessage], + _: &[ActionDef], + _: &LlmCallConfig, + ) -> Result { + let mut r = self.responses.lock().unwrap(); + if r.is_empty() { + Ok(LlmOutput { + response: LlmResponse::Text("done".into()), + usage: TokenUsage::default(), + }) + } else { + Ok(r.remove(0)) + } + } + + fn model_name(&self) -> &str { + "mock" + } + } + + // ── MockEffects — noop effect executor ─────────────────── + + struct MockEffects; + + #[async_trait::async_trait] + impl EffectExecutor for MockEffects { + async fn execute_action( + &self, + _: &str, + _: serde_json::Value, + _: &CapabilityLease, + _: &crate::traits::effect::ThreadExecutionContext, + ) -> Result { + Ok(ActionResult { + call_id: String::new(), + action_name: String::new(), + output: serde_json::json!({}), + is_error: false, + duration: Duration::from_millis(1), + }) + } + + async fn available_actions( + &self, + _: &[CapabilityLease], + ) -> Result, EngineError> { + Ok(vec![]) + } + } + + // ── Helper to build a MissionManager with its dependencies ── + + fn make_mission_manager(store: Arc) -> MissionManager { + let caps = CapabilityRegistry::new(); + let thread_manager = Arc::new(ThreadManager::new( + MockLlm::text("done"), + Arc::new(MockEffects), + Arc::clone(&store), + Arc::new(caps), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + )); + MissionManager::new(store, thread_manager) + } + + // ── Tests ─────────────────────────────────────────────── + + #[tokio::test] + async fn create_mission_persists() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "test mission", + "do the thing", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap(); + assert!(mission.is_some()); + let mission = mission.unwrap(); + assert_eq!(mission.name, "test mission"); + assert_eq!(mission.goal, "do the thing"); + assert_eq!(mission.status, MissionStatus::Active); + assert_eq!(mission.project_id, project_id); + } + + #[tokio::test] + async fn pause_and_resume() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "pausable", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Pause + mgr.pause_mission(id, "test-user").await.unwrap(); + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.status, MissionStatus::Paused); + + // Resume + mgr.resume_mission(id, "test-user").await.unwrap(); + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.status, MissionStatus::Active); + } + + #[tokio::test] + async fn resume_mission_rejects_terminal_states() { + // Regression: resume_mission must not resurrect Completed/Failed + // missions. Only Paused → Active is permitted. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "terminal-state-test", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Active → resume must fail (only Paused is resumable). + let err = mgr + .resume_mission(id, "alice") + .await + .expect_err("resume_mission must reject Active missions"); + match err { + EngineError::Store { reason } => assert!(reason.contains("Active")), + other => panic!("expected Store error, got {other:?}"), + } + + // Drive the mission into a terminal state via complete_mission and + // confirm resume is still rejected. + mgr.complete_mission(id).await.unwrap(); + let err = mgr + .resume_mission(id, "alice") + .await + .expect_err("resume_mission must reject Completed missions"); + match err { + EngineError::Store { reason } => assert!(reason.contains("Completed")), + other => panic!("expected Store error, got {other:?}"), + } + + // Make sure the status didn't drift after the failed resume calls. + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.status, MissionStatus::Completed); + } + + #[tokio::test] + async fn complete_removes_from_active() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "completable", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + mgr.complete_mission(id).await.unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.status, MissionStatus::Completed); + assert!(mission.is_terminal()); + + // Verify removed from active list + let active = mgr.active.read().await; + assert!(!active.contains(&id)); + } + + #[tokio::test] + async fn fire_mission_spawns_thread() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "fireable", + "build something", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + let thread_id = mgr.fire_mission(id, "test-user", None).await.unwrap(); + assert!( + thread_id.is_some(), + "fire_mission should return a thread ID" + ); + + let tid = thread_id.unwrap(); + + // Give the spawned thread a moment to finish (MockLlm returns immediately) + tokio::time::sleep(Duration::from_millis(50)).await; + + // Verify the thread was recorded in mission history + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.thread_history.contains(&tid), + "thread should be recorded in mission history" + ); + } + + #[tokio::test] + async fn fire_terminal_mission_returns_none() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "terminal", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Complete the mission so it becomes terminal + mgr.complete_mission(id).await.unwrap(); + + let result = mgr.fire_mission(id, "test-user", None).await.unwrap(); + assert!( + result.is_none(), + "firing a terminal mission should return None" + ); + } + + #[tokio::test] + async fn tick_fires_due_missions() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Create a cron mission — create_mission now computes next_fire_at + let id = mgr + .create_mission( + project_id, + "test-user", + "cron mission", + "periodic goal", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + // Verify next_fire_at was populated by create_mission + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "create_mission should compute next_fire_at for cron cadence" + ); + + // Move next_fire_at to the past so tick() will fire it + { + let mut missions = store.missions.write().await; + if let Some(mission) = missions.get_mut(&id) { + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::seconds(60)); + } + } + + let spawned = mgr.tick("test-user").await.unwrap(); + assert_eq!(spawned.len(), 1, "tick should fire exactly one due mission"); + + // Give the spawned thread a moment to finish + tokio::time::sleep(Duration::from_millis(50)).await; + + // Verify the thread was recorded + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.thread_history.contains(&spawned[0]), + "spawned thread should be recorded in mission history" + ); + } + + // ── E2E Mission Flow Tests ────────────────────────────── + + /// Build a MissionManager with a MockLlm that returns specific text. + fn make_mission_manager_with_response(store: Arc, response: &str) -> MissionManager { + let caps = CapabilityRegistry::new(); + let thread_manager = Arc::new(ThreadManager::new( + MockLlm::text(response), + Arc::new(MockEffects), + Arc::clone(&store), + Arc::new(caps), + Arc::new(LeaseManager::new()), + Arc::new(PolicyEngine::new()), + )); + MissionManager::new(store, thread_manager) + } + + #[tokio::test] + async fn fire_mission_builds_meta_prompt_with_goal() { + // The MockLlm returns a simple response. We verify the mission + // creates a thread and records it. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response( + Arc::clone(&store) as Arc, + "I searched for news. Found 5 articles.\n\nNext focus: Summarize the articles\nGoal achieved: no", + ); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "Tech News", + "Deliver daily tech news briefing", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + let thread_id = mgr.fire_mission(id, "test-user", None).await.unwrap(); + assert!(thread_id.is_some()); + + // Wait for background outcome processing + tokio::time::sleep(Duration::from_millis(200)).await; + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.thread_history.len(), 1); + assert_eq!(mission.status, MissionStatus::Active); // not completed + } + + #[tokio::test] + async fn outcome_processing_extracts_next_focus() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response( + Arc::clone(&store) as Arc, + "Accomplished: Analyzed the codebase\n\nNext focus: Write tests for the auth module\nGoal achieved: no", + ); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "Test Coverage", + "Increase test coverage to 80%", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + mgr.fire_mission(id, "test-user", None).await.unwrap(); + tokio::time::sleep(Duration::from_millis(200)).await; + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + // next_focus should be extracted from the response + assert_eq!( + mission.current_focus.as_deref(), + Some("Write tests for the auth module"), + "next_focus should be extracted from FINAL response" + ); + // approach_history should have one entry + assert_eq!(mission.approach_history.len(), 1); + assert!(mission.approach_history[0].contains("Accomplished")); + } + + #[tokio::test] + async fn outcome_processing_detects_goal_achieved() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response( + Arc::clone(&store) as Arc, + "Coverage is now 82%!\n\nNext focus: none\nGoal achieved: yes", + ); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "Coverage Mission", + "Get to 80% coverage", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Set success criteria + { + let mut missions = store.missions.write().await; + if let Some(m) = missions.get_mut(&id) { + m.success_criteria = Some("coverage >= 80%".into()); + } + } + + mgr.fire_mission(id, "test-user", None).await.unwrap(); + tokio::time::sleep(Duration::from_millis(200)).await; + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!( + mission.status, + MissionStatus::Completed, + "mission should be completed when goal is achieved" + ); + } + + #[tokio::test] + async fn mission_evolves_via_direct_outcome_processing() { + // Test the outcome processing directly without relying on + // background task timing. + let store: Arc = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + + // Create a mission + let mission = Mission::new( + project_id, + "test-user", + "Coverage", + "Increase coverage to 80%", + MissionCadence::Manual, + ); + let id = mission.id; + store.save_mission(&mission).await.unwrap(); + + // Simulate fire 1 outcome + let outcome1 = ThreadOutcome::Completed { + response: Some( + "Found 3 uncovered modules.\n\nNext focus: Write tests for db module\nGoal achieved: no".into(), + ), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome1) + .await + .unwrap(); + + let mission = store.load_mission(id).await.unwrap().unwrap(); + assert_eq!( + mission.current_focus.as_deref(), + Some("Write tests for db module") + ); + assert_eq!(mission.approach_history.len(), 1); + assert_eq!(mission.status, MissionStatus::Active); + + // Simulate fire 2 outcome + let outcome2 = ThreadOutcome::Completed { + response: Some( + "Added 15 tests for db module.\n\nNext focus: Write tests for tools module\nGoal achieved: no".into(), + ), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome2) + .await + .unwrap(); + + let mission = store.load_mission(id).await.unwrap().unwrap(); + assert_eq!( + mission.current_focus.as_deref(), + Some("Write tests for tools module"), + "focus should evolve between outcomes" + ); + assert_eq!(mission.approach_history.len(), 2); + + // Simulate fire 3 — goal achieved + let outcome3 = ThreadOutcome::Completed { + response: Some("Coverage is 82%!\n\nGoal achieved: yes".into()), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome3) + .await + .unwrap(); + + let mission = store.load_mission(id).await.unwrap().unwrap(); + assert_eq!( + mission.status, + MissionStatus::Completed, + "mission should complete when goal achieved" + ); + assert_eq!(mission.approach_history.len(), 3); + } + + #[tokio::test] + async fn fire_with_trigger_payload() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response( + Arc::clone(&store) as Arc, + "Processed the webhook event.\n\nNext focus: none\nGoal achieved: no", + ); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "GitHub Triage", + "Triage incoming issues", + MissionCadence::Webhook { + path: "github".into(), + secret: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + let payload = serde_json::json!({ + "action": "opened", + "issue": { + "title": "Bug: login fails", + "number": 42 + } + }); + + let thread_id = mgr + .fire_mission(id, "test-user", Some(payload.clone())) + .await + .unwrap(); + assert!(thread_id.is_some()); + + tokio::time::sleep(Duration::from_millis(200)).await; + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.last_trigger_payload, Some(payload)); + assert_eq!(mission.threads_today, 1); + } + + #[tokio::test] + async fn fire_on_system_event_matches_cadence() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response(Arc::clone(&store) as Arc, "done"); + let project_id = ProjectId::new(); + + // Create an OnSystemEvent mission + mgr.create_mission( + project_id, + "test-user", + "self-improve", + "improve prompts", + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "thread_completed_with_issues".into(), + filters: std::collections::HashMap::new(), + }, + Vec::new(), + ) + .await + .unwrap(); + + let spawned = mgr + .fire_on_system_event( + "engine", + "thread_completed_with_issues", + "test-user", + Some(serde_json::json!({"issues": []})), + ) + .await + .unwrap(); + assert_eq!(spawned.len(), 1, "should fire the matching mission"); + } + + #[tokio::test] + async fn fire_on_system_event_ignores_non_matching() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response(Arc::clone(&store) as Arc, "done"); + let project_id = ProjectId::new(); + + // Create an OnSystemEvent mission for a different event + mgr.create_mission( + project_id, + "test-user", + "webhook handler", + "handle webhooks", + MissionCadence::OnSystemEvent { + source: "github".into(), + event_type: "push".into(), + filters: std::collections::HashMap::new(), + }, + Vec::new(), + ) + .await + .unwrap(); + + let spawned = mgr + .fire_on_system_event("engine", "thread_completed_with_issues", "test-user", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 0, "should not fire non-matching mission"); + } + + #[tokio::test] + async fn fire_on_system_event_skips_manual_and_cron() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response(Arc::clone(&store) as Arc, "done"); + let project_id = ProjectId::new(); + + mgr.create_mission( + project_id, + "test-user", + "manual", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + mgr.create_mission( + project_id, + "test-user", + "cron", + "goal", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + let spawned = mgr + .fire_on_system_event("engine", "thread_completed_with_issues", "test-user", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 0); + } + + #[tokio::test] + async fn self_improvement_outcome_saves_prompt_overlay() { + let store: Arc = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + + let mut mission = Mission::new( + project_id, + "test-user", + "self-improve", + "improve prompts", + MissionCadence::OnSystemEvent { + source: "engine".into(), + event_type: "thread_completed_with_issues".into(), + filters: std::collections::HashMap::new(), + }, + ); + mission.metadata = serde_json::json!({"self_improvement": true}); + let id = mission.id; + store.save_mission(&mission).await.unwrap(); + + // Enable self-modification for this test so prompt additions are applied + unsafe { std::env::set_var("ORCHESTRATOR_SELF_MODIFY", "true") }; + + let response = r#"{"prompt_additions": ["9. Never call web_fetch — use http() instead."], "fix_patterns": [], "level": 1}"#; + let outcome = ThreadOutcome::Completed { + response: Some(response.into()), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome) + .await + .unwrap(); + + unsafe { std::env::remove_var("ORCHESTRATOR_SELF_MODIFY") }; + + // Verify prompt overlay was saved + let docs = store.list_memory_docs(project_id, "system").await.unwrap(); + let overlay = docs + .iter() + .find(|d| d.title == crate::executor::prompt::PREAMBLE_OVERLAY_TITLE); + assert!(overlay.is_some(), "prompt overlay should be saved"); + assert!(overlay.unwrap().content.contains("Never call web_fetch")); + } + + #[tokio::test] + async fn self_improvement_outcome_saves_fix_patterns() { + let store: Arc = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + + let mut mission = Mission::new( + project_id, + "test-user", + "self-improve", + "improve prompts", + MissionCadence::Manual, + ); + mission.metadata = serde_json::json!({"self_improvement": true}); + let id = mission.id; + store.save_mission(&mission).await.unwrap(); + + let response = r#"{"prompt_additions": [], "fix_patterns": [{"pattern": "Tool xyz not found", "strategy": "Add alias xyz -> x-y-z", "location": "effect_adapter"}]}"#; + let outcome = ThreadOutcome::Completed { + response: Some(response.into()), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome) + .await + .unwrap(); + + let docs = store.list_memory_docs(project_id, "system").await.unwrap(); + let patterns = docs.iter().find(|d| d.title == FIX_PATTERN_DB_TITLE); + assert!(patterns.is_some(), "fix patterns should be saved"); + assert!(patterns.unwrap().content.contains("Tool xyz not found")); + // Should also contain seed patterns + assert!(patterns.unwrap().content.contains("NameError")); + } + + #[tokio::test] + async fn non_self_improvement_mission_skips_structured_output() { + let store: Arc = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + + let mission = Mission::new( + project_id, + "test-user", + "regular", + "do stuff", + MissionCadence::Manual, + ); + let id = mission.id; + store.save_mission(&mission).await.unwrap(); + + // Even if the response has JSON, it should not create overlays + let response = r#"{"prompt_additions": ["should not appear"], "level": 1}"#; + let outcome = ThreadOutcome::Completed { + response: Some(response.into()), + }; + process_mission_outcome(&store, id, ThreadId::new(), &outcome) + .await + .unwrap(); + + let docs = store.list_memory_docs(project_id, "system").await.unwrap(); + assert!(docs.is_empty(), "non-SI mission should not create overlay"); + } + + #[tokio::test] + async fn ensure_self_improvement_mission_creates_on_first_call() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .ensure_self_improvement_mission(project_id, "test-user") + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.name, "self-improvement"); + assert!(is_self_improvement_mission(&mission)); + assert!(matches!( + mission.cadence, + MissionCadence::OnSystemEvent { .. } + )); + assert_eq!(mission.max_threads_per_day, 5); + assert_eq!(mission.user_id, "test-user"); + + // Fix pattern database should be seeded + let docs = store.list_memory_docs(project_id, "system").await.unwrap(); + let patterns = docs.iter().find(|d| d.title == FIX_PATTERN_DB_TITLE); + assert!(patterns.is_some(), "fix patterns should be seeded"); + assert!(patterns.unwrap().content.contains("NameError")); + } + + #[tokio::test] + async fn ensure_self_improvement_mission_idempotent() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id1 = mgr + .ensure_self_improvement_mission(project_id, "test-user") + .await + .unwrap(); + let id2 = mgr + .ensure_self_improvement_mission(project_id, "test-user") + .await + .unwrap(); + + assert_eq!(id1, id2, "should return the same mission ID"); + + // Should only have one mission + let missions = store.list_missions(project_id, "test-user").await.unwrap(); + assert_eq!(missions.len(), 1); + } + + #[tokio::test] + async fn daily_budget_enforced() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager_with_response(Arc::clone(&store) as Arc, "done"); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "budget test", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Set max_threads_per_day to 1 + { + let mut missions = store.missions.write().await; + if let Some(m) = missions.get_mut(&id) { + m.max_threads_per_day = 1; + } + } + + // First fire — should work + let t1 = mgr.fire_mission(id, "test-user", None).await.unwrap(); + assert!(t1.is_some()); + + tokio::time::sleep(Duration::from_millis(100)).await; + + // Second fire — should be blocked by budget + let t2 = mgr.fire_mission(id, "test-user", None).await.unwrap(); + assert!( + t2.is_none(), + "second fire should be blocked by daily budget" + ); + } + + // ── Multi-tenancy tests ──────────────────────────────────── + + #[tokio::test] + async fn per_user_learning_missions_are_isolated() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Bootstrap learning missions for two different users + mgr.ensure_learning_missions(project_id, "alice") + .await + .unwrap(); + mgr.ensure_learning_missions(project_id, "bob") + .await + .unwrap(); + + // Each user should see only their own missions + let alice_missions = store.list_missions(project_id, "alice").await.unwrap(); + let bob_missions = store.list_missions(project_id, "bob").await.unwrap(); + + assert_eq!(alice_missions.len(), bob_missions.len()); + assert!( + alice_missions.len() >= 3, + "at least 3 learning missions per user" + ); + + // No overlap in mission IDs + let alice_ids: std::collections::HashSet<_> = alice_missions.iter().map(|m| m.id).collect(); + let bob_ids: std::collections::HashSet<_> = bob_missions.iter().map(|m| m.id).collect(); + assert!( + alice_ids.is_disjoint(&bob_ids), + "alice and bob should have separate mission instances" + ); + + // Verify user_id is set correctly on all missions + assert!(alice_missions.iter().all(|m| m.user_id == "alice")); + assert!(bob_missions.iter().all(|m| m.user_id == "bob")); + } + + #[tokio::test] + async fn pause_resume_does_not_cross_users() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Create a mission for alice + let alice_id = mgr + .create_mission( + project_id, + "alice", + "alice-task", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Create a mission for bob + let bob_id = mgr + .create_mission( + project_id, + "bob", + "bob-task", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Alice pauses her own mission — should succeed + mgr.pause_mission(alice_id, "alice").await.unwrap(); + let alice_mission = mgr.get_mission(alice_id).await.unwrap().unwrap(); + assert_eq!(alice_mission.status, MissionStatus::Paused); + + // Bob's mission should be unaffected + let bob_mission = mgr.get_mission(bob_id).await.unwrap().unwrap(); + assert_eq!(bob_mission.status, MissionStatus::Active); + + // Bob tries to resume alice's mission — should fail + let result = mgr.resume_mission(alice_id, "bob").await; + assert!( + result.is_err(), + "bob should not be able to resume alice's mission" + ); + assert!( + matches!(result.unwrap_err(), EngineError::AccessDenied { .. }), + "should be AccessDenied" + ); + + // Alice resumes her own mission — should succeed + mgr.resume_mission(alice_id, "alice").await.unwrap(); + let alice_mission = mgr.get_mission(alice_id).await.unwrap().unwrap(); + assert_eq!(alice_mission.status, MissionStatus::Active); + } + + #[tokio::test] + async fn user_cannot_pause_another_users_learning_mission() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Bootstrap per-user learning missions + mgr.ensure_learning_missions(project_id, "alice") + .await + .unwrap(); + mgr.ensure_learning_missions(project_id, "bob") + .await + .unwrap(); + + // Get alice's self-improvement mission + let alice_missions = store.list_missions(project_id, "alice").await.unwrap(); + let alice_self_imp = alice_missions + .iter() + .find(|m| is_self_improvement_mission(m)) + .expect("alice should have a self-improvement mission"); + + // Bob tries to pause alice's self-improvement — should fail + let result = mgr.pause_mission(alice_self_imp.id, "bob").await; + assert!( + matches!(result.unwrap_err(), EngineError::AccessDenied { .. }), + "bob cannot pause alice's learning mission" + ); + + // Alice pauses her own — should succeed + mgr.pause_mission(alice_self_imp.id, "alice").await.unwrap(); + let m = mgr.get_mission(alice_self_imp.id).await.unwrap().unwrap(); + assert_eq!(m.status, MissionStatus::Paused); + + // Bob's self-improvement should still be active + let bob_missions = store.list_missions(project_id, "bob").await.unwrap(); + let bob_self_imp = bob_missions + .iter() + .find(|m| is_self_improvement_mission(m)) + .unwrap(); + assert_eq!(bob_self_imp.status, MissionStatus::Active); + } + + #[tokio::test] + async fn system_mission_visible_to_all_via_with_shared() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Create a system mission (admin-installed shared mission) + let system_id = mgr + .create_mission( + project_id, + "system", + "shared-monitoring", + "monitor uptime", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Create a user mission + let _user_id = mgr + .create_mission( + project_id, + "alice", + "alice-task", + "do stuff", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Alice's list_missions (strict) only shows her own + let alice_strict = store.list_missions(project_id, "alice").await.unwrap(); + assert_eq!(alice_strict.len(), 1); + + // list_missions_with_shared shows both alice's and system's + let alice_shared = store + .list_missions_with_shared(project_id, "alice") + .await + .unwrap(); + assert_eq!(alice_shared.len(), 2); + assert!(alice_shared.iter().any(|m| m.id == system_id)); + + // Bob sees only the system mission (no personal missions) + let bob_shared = store + .list_missions_with_shared(project_id, "bob") + .await + .unwrap(); + assert_eq!(bob_shared.len(), 1); + assert_eq!(bob_shared[0].id, system_id); + } + + #[tokio::test] + async fn shared_mission_management_is_open_at_engine_layer() { + // Contract pinned by this test (matches the doc-comment on + // `resume_mission` and the ownership tightening in PR #2126/#2130): + // + // "Shared missions can only be managed by shared owners + // (system user)." + // + // i.e. shared (system-owned) missions are NOT manageable by regular + // users at the engine layer. The web handler used to be expected to + // gate admin-role; the engine now enforces shared-owner identity + // directly so the contract holds even when the engine is called + // outside the web handler. + // + // The user-vs-user case for non-shared missions is covered by + // `pause_resume_does_not_cross_users`. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // "system" maps to OwnerId::Shared via LEGACY_SHARED_OWNER_ID. + let system_id = mgr + .create_mission( + project_id, + "system", + "shared-mission", + "shared goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + let mission = mgr.get_mission(system_id).await.unwrap().unwrap(); + assert!( + mission.owner_id().is_shared(), + "missions owned by 'system' must be classified as shared" + ); + + // Regular users CANNOT pause a shared mission — engine returns + // AccessDenied. + let alice_pause = mgr.pause_mission(system_id, "alice").await; + assert!( + matches!(alice_pause, Err(EngineError::AccessDenied { .. })), + "regular users must not pause shared missions; got {:?}", + alice_pause + ); + + // The system user (canonical shared-owner identity) can manage it. + mgr.pause_mission(system_id, "system").await.unwrap(); + let m = mgr.get_mission(system_id).await.unwrap().unwrap(); + assert_eq!(m.status, MissionStatus::Paused); + + // Regular users also cannot resume. + let bob_resume = mgr.resume_mission(system_id, "bob").await; + assert!( + matches!(bob_resume, Err(EngineError::AccessDenied { .. })), + "regular users must not resume shared missions; got {:?}", + bob_resume + ); + + // System user resume works. + mgr.resume_mission(system_id, "system").await.unwrap(); + let m = mgr.get_mission(system_id).await.unwrap().unwrap(); + assert_eq!(m.status, MissionStatus::Active); + } + + #[tokio::test] + async fn fire_mission_ownership_check() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Create alice's mission + let alice_id = mgr + .create_mission( + project_id, + "alice", + "alice-only", + "private goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + // Bob cannot fire alice's mission + let result = mgr.fire_mission(alice_id, "bob", None).await; + assert!( + matches!(result.unwrap_err(), EngineError::AccessDenied { .. }), + "bob cannot fire alice's mission" + ); + + // Alice can fire her own + let tid = mgr.fire_mission(alice_id, "alice", None).await.unwrap(); + assert!(tid.is_some()); + } + + #[tokio::test] + async fn fire_on_system_event_scoped_to_user() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Bootstrap per-user learning missions + mgr.ensure_learning_missions(project_id, "alice") + .await + .unwrap(); + mgr.ensure_learning_missions(project_id, "bob") + .await + .unwrap(); + + // Count active missions for each user + let alice_missions = store.list_missions(project_id, "alice").await.unwrap(); + let bob_missions = store.list_missions(project_id, "bob").await.unwrap(); + let alice_self_imp = alice_missions + .iter() + .find(|m| is_self_improvement_mission(m)) + .unwrap(); + let bob_self_imp = bob_missions + .iter() + .find(|m| is_self_improvement_mission(m)) + .unwrap(); + + // Pause bob's self-improvement + mgr.pause_mission(bob_self_imp.id, "bob").await.unwrap(); + + // Fire system event as alice — should fire alice's missions, not bob's + let payload = serde_json::json!({"source_thread_id": "test", "goal": "test"}); + let spawned = mgr + .fire_on_system_event( + "engine", + "thread_completed_with_issues", + "alice", + Some(payload), + ) + .await + .unwrap(); + + // Should have fired alice's self-improvement (active) but not bob's (paused) + assert!(!spawned.is_empty(), "alice's self-improvement should fire"); + + // Verify spawned thread belongs to alice + tokio::time::sleep(Duration::from_millis(50)).await; + for tid in &spawned { + if let Some(thread) = store.load_thread(*tid).await.unwrap() { + assert_eq!( + thread.user_id, "alice", + "spawned thread should belong to alice" + ); + } + } + + // Verify bob's self-improvement is still paused and was not fired + let bob_m = mgr.get_mission(bob_self_imp.id).await.unwrap().unwrap(); + assert_eq!(bob_m.status, MissionStatus::Paused); + assert!( + bob_m.thread_history.is_empty(), + "bob's paused mission should not have spawned threads" + ); + + // Alice's should have recorded the thread + let alice_m = mgr.get_mission(alice_self_imp.id).await.unwrap().unwrap(); + assert!( + !alice_m.thread_history.is_empty(), + "alice's mission should have recorded the spawned thread" + ); + } + + /// Helper: create an event mission with the reactive-default guardrails + /// disabled so the test can fire it repeatedly without tripping cooldown + /// or daily caps. Patterns are caller-supplied; everything else stays + /// at the engine defaults *except* the guardrails we explicitly null out. + async fn create_unguarded_event_mission( + mgr: &MissionManager, + project_id: ProjectId, + user_id: &str, + name: &str, + pattern: &str, + channel: Option<&str>, + ) -> MissionId { + let id = mgr + .create_mission( + project_id, + user_id, + name, + "react to events", + MissionCadence::OnEvent { + event_pattern: pattern.to_string(), + channel: channel.map(String::from), + }, + Vec::new(), + ) + .await + .unwrap(); + // Disable reactive defaults for tests that want to assert the + // matcher behavior without tripping cooldown / max_concurrent. + mgr.update_mission( + id, + user_id, + MissionUpdate { + cooldown_secs: Some(0), + max_concurrent: Some(0), + max_threads_per_day: Some(0), + ..Default::default() + }, + ) + .await + .unwrap(); + id + } + + #[tokio::test] + async fn fire_on_message_event_matches_pattern_and_channel_filter() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Mission with a channel-scoped message event trigger. + let id = create_unguarded_event_mission( + &mgr, + project_id, + "alice", + "PR review nudge", + "review requested", + Some("github"), + ) + .await; + + // Wrong channel — should NOT fire even though pattern matches. + let spawned = mgr + .fire_on_message_event("slack", "review requested on PR #42", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "wrong channel should not fire"); + + // Right channel, wrong pattern — should NOT fire. + let spawned = mgr + .fire_on_message_event("github", "build green", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "wrong pattern should not fire"); + + // Right channel, right pattern — SHOULD fire. + let spawned = mgr + .fire_on_message_event( + "github", + "review requested on PR #42", + "alice", + Some(serde_json::json!({"pr": 42})), + ) + .await + .unwrap(); + assert_eq!( + spawned.len(), + 1, + "matching event should fire exactly one mission" + ); + + // Channel filter is case-insensitive. + let spawned = mgr + .fire_on_message_event("GitHub", "review requested again", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1, "channel match should be case-insensitive"); + + // Mission's thread history should now reflect both fires. + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.thread_history.len(), 2); + } + + #[tokio::test] + async fn fire_on_message_event_without_channel_filter_matches_any_channel() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Mission with no channel filter — should match any channel. + create_unguarded_event_mission( + &mgr, + project_id, + "alice", + "Universal pattern", + "deploy now", + None, + ) + .await; + + for channel in &["github", "slack", "gateway", "repl"] { + let spawned = mgr + .fire_on_message_event(channel, "please deploy now thanks", "alice", None) + .await + .unwrap(); + assert_eq!( + spawned.len(), + 1, + "no channel filter should match channel {channel}" + ); + } + } + + #[tokio::test] + async fn fire_on_message_event_respects_owner_scope() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Alice owns a mission. + create_unguarded_event_mission(&mgr, project_id, "alice", "Alice mission", "ping", None) + .await; + + // Bob fires the event with a matching pattern — should NOT fire + // alice's mission (per-user scoping). + let spawned = mgr + .fire_on_message_event("gateway", "ping", "bob", None) + .await + .unwrap(); + assert!( + spawned.is_empty(), + "events from other users must not fire missions they don't own" + ); + + // Alice fires the event — SHOULD fire her mission. + let spawned = mgr + .fire_on_message_event("gateway", "ping", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1); + } + + #[tokio::test] + async fn fire_on_webhook_matches_path() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + mgr.create_mission( + project_id, + "alice", + "GitHub webhook", + "Handle GitHub events", + MissionCadence::Webhook { + path: "github".into(), + secret: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + // Wrong path — should NOT fire. + let spawned = mgr.fire_on_webhook("slack", "alice", None).await.unwrap(); + assert!(spawned.is_empty()); + + // Right path — SHOULD fire. + let spawned = mgr + .fire_on_webhook( + "github", + "alice", + Some(serde_json::json!({"action": "opened"})), + ) + .await + .unwrap(); + assert_eq!(spawned.len(), 1); + } + + /// Regression for the substring-match flooding bug: + /// `text.contains("review requested")` would match unrelated phrases + /// like "I just reviewed your request" — way too loose. The matcher + /// is now regex-based, so word-boundary-aware patterns no longer + /// flood on accidental substrings. + #[tokio::test] + async fn fire_on_message_event_uses_regex_with_word_boundaries() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Word-boundary regex for "deploy". + create_unguarded_event_mission( + &mgr, + project_id, + "alice", + "Deploy watcher", + r"\bdeploy\b", + None, + ) + .await; + + // Should NOT match: "deployed" / "deployment" / "redeploy". + for noisy in &[ + "I just deployed the change", + "the deployment finished", + "going to redeploy later", + ] { + let spawned = mgr + .fire_on_message_event("gateway", noisy, "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "regex with \\b must not match: {noisy}"); + } + + // SHOULD match: standalone "deploy". + let spawned = mgr + .fire_on_message_event("gateway", "please deploy now", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1, "standalone 'deploy' must match"); + } + + /// Regression: an OnEvent mission created via `create_mission` without + /// explicit guardrails must inherit reactive defaults (cooldown 300s, + /// max_concurrent 1, daily cap 24) so accidentally-loose patterns + /// can't burn the LLM budget. + #[tokio::test] + async fn event_triggered_missions_get_reactive_defaults() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "Default reactive mission", + "react", + MissionCadence::OnEvent { + event_pattern: "anything".into(), + channel: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!( + mission.cooldown_secs, 300, + "OnEvent missions default to a 5-minute cooldown" + ); + assert_eq!( + mission.max_concurrent, 1, + "OnEvent missions default to single-instance" + ); + assert_eq!( + mission.max_threads_per_day, 24, + "OnEvent missions default to 24 fires/day" + ); + } + + /// Manual / Cron missions retain the prior generous defaults — they + /// are self-paced and don't risk flooding from external events. + #[tokio::test] + async fn manual_and_cron_missions_keep_proactive_defaults() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let manual_id = mgr + .create_mission( + project_id, + "alice", + "manual", + "do it on demand", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + let manual = mgr.get_mission(manual_id).await.unwrap().unwrap(); + assert_eq!(manual.cooldown_secs, 0); + assert_eq!(manual.max_concurrent, 0); + assert_eq!(manual.max_threads_per_day, 10); + + let cron_id = mgr + .create_mission( + project_id, + "alice", + "cron", + "every six hours", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + let cron = mgr.get_mission(cron_id).await.unwrap().unwrap(); + assert_eq!(cron.cooldown_secs, 0); + assert_eq!(cron.max_concurrent, 0); + assert_eq!(cron.max_threads_per_day, 10); + } + + /// The per-user sliding-window rate limiter must refuse fires once + /// the cap is reached and recover after the window slides past. + #[tokio::test] + async fn per_user_rate_limit_blocks_excess_fires() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc).with_rate_limit( + FireRateLimit { + max_fires: 3, + window: std::time::Duration::from_secs(60), + }, + ); + let project_id = ProjectId::new(); + + create_unguarded_event_mission( + &mgr, + project_id, + "alice", + "rate-limited mission", + r"go", + None, + ) + .await; + + // First 3 fires should succeed; the 4th should be silently dropped. + for i in 0..3 { + let spawned = mgr + .fire_on_message_event("gateway", "go", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1, "fire {i} should succeed"); + } + let spawned = mgr + .fire_on_message_event("gateway", "go", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "rate-limited fire should be dropped"); + } + + /// `BudgetGate::allow_mission_fire` returning false must abort the + /// fire without spawning a thread or recording history. + #[tokio::test] + async fn budget_gate_can_refuse_mission_fires() { + struct DenyAll; + #[async_trait::async_trait] + impl BudgetGate for DenyAll { + async fn allow_mission_fire(&self, _user_id: &str, _mission_id: MissionId) -> bool { + false + } + } + + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc) + .with_budget_gate(Arc::new(DenyAll)); + let project_id = ProjectId::new(); + + let id = + create_unguarded_event_mission(&mgr, project_id, "alice", "blocked", r"go", None).await; + + let spawned = mgr + .fire_on_message_event("gateway", "go", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "BudgetGate denial must block the fire"); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.thread_history.is_empty(), + "denied fire must not record any threads" + ); + } + + /// Updating a mission must evict its cached compiled regex so the next + /// match attempt picks up the new pattern. + #[tokio::test] + async fn updating_event_pattern_invalidates_regex_cache() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = + create_unguarded_event_mission(&mgr, project_id, "alice", "swappable", r"alpha", None) + .await; + + // Initial pattern matches "alpha". + let spawned = mgr + .fire_on_message_event("gateway", "alpha", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1); + + // Swap the cadence to a new pattern. + mgr.update_mission( + id, + "alice", + MissionUpdate { + cadence: Some(MissionCadence::OnEvent { + event_pattern: r"beta".into(), + channel: None, + }), + ..Default::default() + }, + ) + .await + .unwrap(); + + // The old pattern must no longer match. + let spawned = mgr + .fire_on_message_event("gateway", "alpha", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "stale regex cache must be evicted"); + + // The new pattern must match. + let spawned = mgr + .fire_on_message_event("gateway", "beta", "alice", None) + .await + .unwrap(); + assert_eq!(spawned.len(), 1, "new pattern must take effect"); + } + + // ── routine-fix-history regression tests ───────────────────────── + // + // Tests in this section pin invariants whose v1 routine analogs were + // historically broken (or whose fix went into a v1 routine code path + // that has no v2 equivalent — we add them here to make sure missions + // never regress the same bug). + + /// Mirrors v1 routine fix #1372 / #1374: a fired mission with + /// `max_concurrent = N` and N already-running threads must refuse to + /// fire again. The check is in `fire_mission` after the cooldown gate. + /// This test pins it through the public surface so a future refactor + /// can't drop the check without failing here. + #[tokio::test] + async fn fire_mission_blocks_when_max_concurrent_reached() { + use crate::types::thread::{Thread, ThreadConfig, ThreadType}; + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "single instance", + "do exactly one thing at a time", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + // Set max_concurrent=1 explicitly (Manual missions default to 0). + mgr.update_mission( + id, + "alice", + MissionUpdate { + max_concurrent: Some(1), + ..Default::default() + }, + ) + .await + .unwrap(); + + // Pre-seed a Running thread for this mission so the next fire + // sees max_concurrent already saturated. ThreadType::Mission with + // the default `Created` state is non-terminal in + // count_running_threads (which only treats Done/Failed as + // terminal). + let thread = Thread::new( + "preseeded", + ThreadType::Mission, + project_id, + "alice", + ThreadConfig::default(), + ); + let preseeded_id = thread.id; + store.save_thread(&thread).await.unwrap(); + let mut mission = mgr.get_mission(id).await.unwrap().unwrap(); + mission.thread_history.push(preseeded_id); + store.save_mission(&mission).await.unwrap(); + + // Fire — should be refused with Ok(None), not an error. + let outcome = mgr.fire_mission(id, "alice", None).await.unwrap(); + assert!( + outcome.is_none(), + "max_concurrent=1 with one running thread must block the next fire" + ); + + // The mission's thread_history must NOT have grown. + let after = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!( + after.thread_history.len(), + 1, + "blocked fire must not record a new thread" + ); + } + + /// Mirrors v1 routine fix #1321: notification summaries must be + /// truncated before broadcasting so a runaway response can't OOM + /// chat-channel adapters or saturate SSE buffers. The full text stays + /// in `mission.approach_history` untruncated. + #[test] + fn truncate_notification_text_caps_long_strings() { + let huge = "x".repeat(MAX_NOTIFICATION_RESPONSE_BYTES * 3); + let truncated = truncate_notification_text(&huge); + assert!( + truncated.len() <= MAX_NOTIFICATION_RESPONSE_BYTES + 4, + "truncated text must fit within the cap (plus the ellipsis byte): got {}", + truncated.len() + ); + assert!( + truncated.ends_with('…'), + "truncation must end with an ellipsis" + ); + + let small = "fits within the cap"; + assert_eq!( + truncate_notification_text(small), + small, + "strings under the cap must pass through unchanged" + ); + } + + /// Mirrors v1 routine fix's `floor_char_boundary` change: truncation + /// MUST NOT split a multi-byte UTF-8 sequence. The naive approach + /// (`&s[..MAX]`) would panic on a multi-byte character that straddles + /// the byte index. + #[test] + fn truncate_notification_text_is_utf8_safe() { + // Construct a string where a multi-byte char straddles the byte cap. + // "ñ" is 2 bytes (0xC3 0xB1). We want byte position MAX_BYTES to + // land in the middle of one. + let prefix = "a".repeat(MAX_NOTIFICATION_RESPONSE_BYTES - 1); + let mut input = prefix; + input.push('ñ'); // 2 bytes — second byte is at MAX_BYTES + input.push_str(&"b".repeat(100)); + assert!(input.len() > MAX_NOTIFICATION_RESPONSE_BYTES); + + // Must not panic — the bug would slice a multi-byte char in half. + let truncated = truncate_notification_text(&input); + // And the result must be valid UTF-8 (it's a String, so by + // construction it is — but the assertion makes the invariant + // explicit). + assert!(truncated.is_char_boundary(truncated.len())); + // The 'ñ' must NOT have been split: either it's in the result + // wholly, or it was dropped wholly. + assert!( + !truncated.ends_with('a'), + "truncation should have stopped at the multi-byte char boundary, not after it" + ); + } + + /// Mirrors v1 routine fix #1255: when a mission is deleted (the v2 + /// analog of `routine_delete`), its compiled regex cache entry MUST + /// be evicted so a future mission with the same id can't accidentally + /// pick up a stale pattern. This pins the eviction call already in + /// `complete_mission`. + #[tokio::test] + async fn complete_mission_evicts_event_regex_cache() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = create_unguarded_event_mission( + &mgr, + project_id, + "alice", + "to be deleted", + r"hello", + None, + ) + .await; + + // Force regex compile + cache populate. + let _ = mgr + .fire_on_message_event("gateway", "hello", "alice", None) + .await + .unwrap(); + assert!( + mgr.event_regex_cache.read().await.contains_key(&id), + "regex cache should hold the compiled pattern after first match" + ); + + mgr.complete_mission(id).await.unwrap(); + assert!( + !mgr.event_regex_cache.read().await.contains_key(&id), + "complete_mission must evict the cached compiled regex" + ); + } + + /// Mirrors v1 routine fix #1374: failure-path outcomes must produce a + /// notification, not silently swallow the error. Without this, a + /// failed mission run leaves the user with no signal that anything + /// went wrong. + #[tokio::test] + async fn failed_outcome_emits_error_notification() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "may fail", + "do the risky thing", + MissionCadence::Manual, + vec!["gateway".to_string()], + ) + .await + .unwrap(); + + let mut rx = mgr.subscribe_notifications(); + + let synthetic_thread_id = crate::types::thread::ThreadId::new(); + process_mission_outcome_and_notify( + &(Arc::clone(&store) as Arc), + id, + synthetic_thread_id, + &ThreadOutcome::Failed { + error: "container exited 137".into(), + }, + mgr.notification_tx_for_test(), + None, + ) + .await + .unwrap(); + + let notification = rx + .try_recv() + .expect("Failed outcome must emit a notification"); + assert!(notification.is_error, "is_error flag must be set"); + assert_eq!(notification.notify_channels, vec!["gateway".to_string()]); + assert!( + notification + .response + .as_deref() + .is_some_and(|r| r.contains("container exited 137")), + "notification response must surface the underlying error message; got {:?}", + notification.response + ); + + // Same for MaxIterations — historically the silent-fail case. + process_mission_outcome_and_notify( + &(Arc::clone(&store) as Arc), + id, + synthetic_thread_id, + &ThreadOutcome::MaxIterations, + mgr.notification_tx_for_test(), + None, + ) + .await + .unwrap(); + + let notification = rx + .try_recv() + .expect("MaxIterations must emit a notification"); + assert!(notification.is_error, "MaxIterations must set is_error"); + } + + #[tokio::test] + async fn outcome_processor_reconciles_missing_fire_accounting() { + // Regression: when `fire_mission`'s post-spawn `save_mission` fails, + // the persisted mission is missing the new thread_id, threads_today + // bump, last_fire_at stamp, and (for cron) advanced next_fire_at. + // The outcome processor must reconcile these fields the next time + // it runs so the durable state catches up — otherwise tick re-fires + // against the stale schedule once the in-memory cooldown elapses. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Create a cron mission with `next_fire_at` already in the past, to + // mimic the post-failed-fire state directly. (Going through + // `fire_mission` with a fault-injecting store would require a new + // TestStore variant; this is the equivalent end state.) + let id = mgr + .create_mission( + project_id, + "alice", + "reconcile-test", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + { + let mut missions = store.missions.write().await; + let mission = missions.get_mut(&id).unwrap(); + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::seconds(120)); + mission.threads_today = 0; + mission.thread_history.clear(); + mission.last_fire_at = None; + } + + // Run the outcome processor with a thread_id that the persisted + // mission has never seen — exactly the state a failed `save_mission` + // would leave us in. + let orphan_thread_id = crate::types::thread::ThreadId::new(); + // Pass an explicit `original_fire_at` so the test exercises the + // production-equivalent path where fire_mission's instant flows + // through the watcher into reconcile (instead of falling back to + // `now`). + let original_fire_at = chrono::Utc::now() - chrono::Duration::seconds(30); + process_mission_outcome_and_notify( + &(Arc::clone(&store) as Arc), + id, + orphan_thread_id, + &ThreadOutcome::Completed { + response: Some("done".into()), + }, + mgr.notification_tx_for_test(), + Some(original_fire_at), + ) + .await + .unwrap(); + + let reloaded = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + reloaded.thread_history.contains(&orphan_thread_id), + "outcome processor must idempotently append the missing thread_id" + ); + assert_eq!( + reloaded.threads_today, 1, + "threads_today must catch up after reconcile" + ); + assert_eq!( + reloaded.last_fire_at, + Some(original_fire_at), + "last_fire_at must be reconciled to the original fire instant, not `now`" + ); + assert!( + reloaded + .next_fire_at + .is_some_and(|next| next > chrono::Utc::now()), + "next_fire_at must be advanced past now() after reconcile, got {:?}", + reloaded.next_fire_at + ); + + // Reconcile is idempotent: replaying with the same thread_id must + // not double-count threads_today or duplicate the history entry. + process_mission_outcome_and_notify( + &(Arc::clone(&store) as Arc), + id, + orphan_thread_id, + &ThreadOutcome::Completed { + response: Some("done".into()), + }, + mgr.notification_tx_for_test(), + Some(original_fire_at), + ) + .await + .unwrap(); + let reloaded = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!( + reloaded + .thread_history + .iter() + .filter(|t| **t == orphan_thread_id) + .count(), + 1, + "thread_history must not duplicate on replay" + ); + assert_eq!( + reloaded.threads_today, 1, + "threads_today must not double-count on replay" + ); + } + + /// A pattern that fails to compile (or exceeds the size cap) must be + /// logged and never match — it must not panic, hang, or fall through + /// to a substring search. + #[tokio::test] + async fn invalid_event_regex_never_matches() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // `[` is not a valid regex; compilation must fail. + create_unguarded_event_mission(&mgr, project_id, "alice", "broken pattern", "[", None) + .await; + + let spawned = mgr + .fire_on_message_event("gateway", "anything", "alice", None) + .await + .unwrap(); + assert!(spawned.is_empty(), "invalid regex must not match anything"); + } + + #[tokio::test] + async fn ensure_learning_missions_idempotent_per_user() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Call twice for the same user + mgr.ensure_learning_missions(project_id, "alice") + .await + .unwrap(); + mgr.ensure_learning_missions(project_id, "alice") + .await + .unwrap(); + + // Should not create duplicates + let alice_missions = store.list_missions(project_id, "alice").await.unwrap(); + let self_imp_count = alice_missions + .iter() + .filter(|m| is_self_improvement_mission(m)) + .count(); + assert_eq!( + self_imp_count, 1, + "should not duplicate self-improvement mission" + ); + } + + // ── Cron scheduling tests (#1944) ───────────────────────── + + #[tokio::test] + async fn create_cron_mission_sets_next_fire_at() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "cron test", + "periodic goal", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "cron mission should have next_fire_at computed on creation" + ); + assert!( + mission.next_fire_at.unwrap() > chrono::Utc::now(), + "next_fire_at should be in the future" + ); + } + + #[tokio::test] + async fn create_manual_mission_has_no_next_fire_at() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "manual test", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_none(), + "manual mission should not have next_fire_at" + ); + } + + #[tokio::test] + async fn fire_mission_advances_next_fire_at() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "cron advance", + "periodic goal", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + // Move next_fire_at to the past so tick fires it + { + let mut missions = store.missions.write().await; + if let Some(mission) = missions.get_mut(&id) { + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::seconds(60)); + } + } + + let spawned = mgr.tick("test-user").await.unwrap(); + assert_eq!(spawned.len(), 1); + + // After firing, next_fire_at should be advanced to the future + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "next_fire_at should be set after firing" + ); + assert!( + mission.next_fire_at.unwrap() > chrono::Utc::now(), + "next_fire_at should be strictly in the future after firing" + ); + } + + #[tokio::test] + async fn resume_cron_mission_recomputes_next_fire_at() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "cron resume", + "periodic goal", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + // Pause the mission — this clears it from active list + mgr.pause_mission(id, "test-user").await.unwrap(); + + // Manually set next_fire_at to a stale past value + { + let mut missions = store.missions.write().await; + if let Some(mission) = missions.get_mut(&id) { + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::hours(24)); + } + } + + // Resume — should recompute next_fire_at + mgr.resume_mission(id, "test-user").await.unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "resume should recompute next_fire_at for cron missions" + ); + assert!( + mission.next_fire_at.unwrap() > chrono::Utc::now(), + "recomputed next_fire_at should be in the future" + ); + } + + #[tokio::test] + async fn update_mission_manual_to_cron_sets_next_fire_at() { + // Regression: a Manual -> Cron switch left next_fire_at = None and the + // mission never fired. update_mission must recompute the schedule. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "starts manual", + "goal", + MissionCadence::Manual, + Vec::new(), + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!(mission.next_fire_at.is_none()); + + mgr.update_mission( + id, + "test-user", + MissionUpdate { + cadence: Some(MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }), + ..Default::default() + }, + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "Manual -> Cron update should compute next_fire_at" + ); + assert!( + mission.next_fire_at.unwrap() > chrono::Utc::now(), + "next_fire_at should be in the future" + ); + } + + #[tokio::test] + async fn update_mission_cron_to_manual_clears_next_fire_at() { + // Regression: a stale next_fire_at must be cleared when switching away + // from Cron, otherwise the ticker could fire a non-cron mission. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "starts cron", + "goal", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + assert!( + mgr.get_mission(id) + .await + .unwrap() + .unwrap() + .next_fire_at + .is_some() + ); + + mgr.update_mission( + id, + "test-user", + MissionUpdate { + cadence: Some(MissionCadence::Manual), + ..Default::default() + }, + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_none(), + "non-cron cadence must clear next_fire_at" + ); + } + + #[tokio::test] + async fn create_cron_mission_with_timezone_uses_tz_for_schedule() { + // Regression: every other cron test in this file passes timezone: None, + // so the tz path is only exercised at the unit level inside types/mission. + // This test threads a real ValidTimezone through MissionManager and + // asserts the resulting next_fire_at differs from the UTC equivalent — + // proving the bridge router → mission_create → next_cron_fire chain + // actually honors the user's timezone end-to-end. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + let tz = crate::types::mission::ValidTimezone::parse("America/New_York").unwrap(); + + let id_tz = mgr + .create_mission( + project_id, + "test-user", + "tz-aware", + "fires at 9am NY local", + MissionCadence::Cron { + expression: "0 9 * * *".into(), + timezone: Some(tz), + }, + Vec::new(), + ) + .await + .unwrap(); + + let id_utc = mgr + .create_mission( + project_id, + "test-user", + "tz-naive", + "fires at 9am UTC", + MissionCadence::Cron { + expression: "0 9 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + let m_tz = mgr.get_mission(id_tz).await.unwrap().unwrap(); + let m_utc = mgr.get_mission(id_utc).await.unwrap().unwrap(); + let next_tz = m_tz.next_fire_at.expect("tz cron should have next_fire_at"); + let next_utc = m_utc + .next_fire_at + .expect("utc cron should have next_fire_at"); + + // 9am NY = 13:00 or 14:00 UTC depending on DST; 9am UTC = 09:00 UTC. + use chrono::Timelike; + assert_ne!( + next_tz.hour(), + next_utc.hour(), + "tz-aware and tz-naive cron schedules must produce different UTC instants" + ); + let tz_hour = next_tz.hour(); + assert!( + tz_hour == 13 || tz_hour == 14, + "9am NY should land on UTC 13 or 14, got {tz_hour}" + ); + assert_eq!(next_utc.hour(), 9, "9am UTC should land on UTC 9"); + } + + #[tokio::test] + async fn update_mission_cron_expression_change_recomputes_next_fire_at() { + // Regression: changing the cron expression must reset the schedule. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "cron edit", + "goal", + MissionCadence::Cron { + // Year-locked to 2099 so the next fire is deterministically + // far in the future regardless of the calendar date the + // test runs on. The original `0 0 1 1 *` ("once a year on + // Jan 1") was racy around New Year's, when the yearly + // schedule's next fire could land within seconds and + // invert the `after < before` ordering below. + expression: "0 0 0 1 1 * 2099".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + let before = mgr.get_mission(id).await.unwrap().unwrap().next_fire_at; + + mgr.update_mission( + id, + "test-user", + MissionUpdate { + cadence: Some(MissionCadence::Cron { + expression: "* * * * *".into(), // every minute + timezone: None, + }), + ..Default::default() + }, + ) + .await + .unwrap(); + + let after = mgr.get_mission(id).await.unwrap().unwrap().next_fire_at; + assert!(after.is_some()); + assert_ne!( + before, after, + "schedule must be recomputed on cadence change" + ); + assert!( + after.unwrap() < before.unwrap(), + "every-minute schedule should fire sooner than once-a-year" + ); + } + + #[tokio::test] + async fn fire_mission_with_corrupt_cron_expression_does_not_orphan_thread() { + // Regression: previously fire_mission used `?` on next_cron_fire after + // spawning the thread. A persisted mission with a corrupt cron string + // would spawn the thread, then abort fire_mission with an Err — leaving + // the thread running with no entry in thread_history, no incremented + // budget, and (when also reordered) no outcome watcher installed. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "corrupt cron", + "goal", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + // Capture the original next_fire_at — we expect fire to *preserve* it + // (rather than replace with None or recompute) when the expression + // can't be parsed. + let original_next = mgr + .get_mission(id) + .await + .unwrap() + .unwrap() + .next_fire_at + .expect("create should populate next_fire_at"); + + // Corrupt the persisted expression directly in the test store. + { + let mut missions = store.missions.write().await; + if let Some(m) = missions.get_mut(&id) + && let MissionCadence::Cron { + ref mut expression, .. + } = m.cadence + { + *expression = "this is not a cron".to_string(); + } + } + + // Fire must succeed despite the corrupt expression. + let thread_id = mgr + .fire_mission(id, "test-user", None) + .await + .expect("fire_mission must not fail on corrupt cron"); + assert!(thread_id.is_some(), "fire should spawn a thread"); + let thread_id = thread_id.unwrap(); + + // The mission record must reflect the fire: thread tracked + budget + // incremented. Without the fix, save_mission was never reached. + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.thread_history.contains(&thread_id), + "thread should be recorded in thread_history" + ); + assert_eq!( + mission.threads_today, 1, + "threads_today should be incremented even if next_fire_at couldn't recompute" + ); + // next_fire_at should be preserved (not cleared) since we couldn't + // compute a new one. + assert_eq!( + mission.next_fire_at, + Some(original_next), + "next_fire_at must be preserved when next_cron_fire fails" + ); + } + + #[tokio::test] + async fn resume_mission_preserves_concurrent_field_changes() { + // Regression: resume_mission used to do update_mission_status() then a + // separate load+save round-trip to recompute next_fire_at. Now it does + // a single mutate-and-save with the mission already loaded for the + // ownership check, eliminating the extra interleave window. + // + // We can't deterministically exercise the TOCTOU window in a unit + // test, but we can assert the new contract: resume_mission writes the + // mission's other fields (e.g. threads_today) faithfully and does not + // depend on a separate update_mission_status round-trip succeeding. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "test-user", + "resume preserve", + "goal", + MissionCadence::Cron { + expression: "0 */6 * * *".into(), + timezone: None, + }, + Vec::new(), + ) + .await + .unwrap(); + + mgr.pause_mission(id, "test-user").await.unwrap(); + + // Simulate a concurrent writer that bumps threads_today between pause + // and resume. With the old two-write resume path, the second + // load+save could clobber this. With the single-save path it cannot + // be clobbered by THIS resume call. + { + let mut missions = store.missions.write().await; + if let Some(m) = missions.get_mut(&id) { + m.threads_today = 7; + m.goal = "concurrently updated goal".to_string(); + } + } + + mgr.resume_mission(id, "test-user").await.unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(mission.status, MissionStatus::Active); + // The concurrent update happened *before* resume_mission's load, so + // the resume should observe and preserve those values rather than + // resetting to creation-time defaults. + assert_eq!( + mission.threads_today, 7, + "resume must not reset threads_today to a stale value" + ); + assert_eq!( + mission.goal, "concurrently updated goal", + "resume must not clobber goal updated before its load" + ); + assert!( + mission.next_fire_at.is_some(), + "resume should still recompute next_fire_at for cron" + ); + } + + #[tokio::test] + async fn ensure_mission_by_metadata_with_cron_cadence_computes_next_fire_at() { + // Regression: ensure_mission_by_metadata used to construct + // Mission::new + save_mission directly, bypassing the next_fire_at + // computation that create_mission performs. Today every caller passes + // OnSystemEvent so the bug is latent, but a future caller passing + // Cron would silently re-introduce the original `next_fire_at = None` + // bug that #1944 fixes. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .ensure_mission_by_metadata( + project_id, + "test-user", + "synthetic_cron", + "synthetic-cron", + "synthetic goal", + MissionCadence::Cron { + expression: "0 9 * * *".into(), + timezone: None, + }, + "synthetic criteria", + 3, + ) + .await + .unwrap(); + + let mission = mgr.get_mission(id).await.unwrap().unwrap(); + assert!( + mission.next_fire_at.is_some(), + "ensure_mission_by_metadata must compute next_fire_at for Cron cadence" + ); + assert!( + mission.next_fire_at.unwrap() > chrono::Utc::now(), + "next_fire_at must be in the future" + ); + } + + #[test] + fn conversation_insights_count_only_done_threads() { + assert!(should_count_for_conversation_insights(ThreadState::Done)); + assert!(!should_count_for_conversation_insights(ThreadState::Failed)); + } + + #[test] + fn build_skill_gap_payload_uses_active_skill_provenance() { + let project_id = ProjectId::new(); + let mut thread = Thread::new( + "repair a github workflow", + ThreadType::Foreground, + project_id, + "alice", + ThreadConfig::default(), + ); + thread.state = ThreadState::Done; + let skill_doc_id = DocId::new(); + thread + .set_active_skills(&[ActiveSkillProvenance { + doc_id: skill_doc_id, + name: "github-pr-workflow".to_string(), + version: 3, + snippet_names: vec!["list_prs".to_string()], + force_activated: false, + }]) + .unwrap(); + thread.add_event(crate::types::event::EventKind::ActionFailed { + step_id: StepId::new(), + action_name: "shell".to_string(), + call_id: "call_1".to_string(), + error: "gh auth status: not authenticated".to_string(), + params_summary: None, + }); + + let trace = crate::executor::trace::build_trace(&thread); + let active_skills = thread.active_skills(); + let payload = build_skill_gap_payload(&thread, &trace, &active_skills).unwrap(); + + assert_eq!( + payload["active_skills"][0]["doc_id"], + serde_json::Value::String(skill_doc_id.0.to_string()) + ); + let hints = payload["repair_hints"].as_array().unwrap(); + assert!( + hints + .iter() + .any(|hint| hint.as_str() == Some("missing_prerequisite")), + "repair hints should include missing_prerequisite: {payload}" + ); + } + + #[test] + fn build_skill_gap_payload_preserves_recoverable_auth_prerequisite_hints() { + let project_id = ProjectId::new(); + let mut thread = Thread::new( + "repair a github workflow", + ThreadType::Foreground, + project_id, + "alice", + ThreadConfig::default(), + ); + thread.state = ThreadState::Done; + thread + .set_active_skills(&[ActiveSkillProvenance { + doc_id: DocId::new(), + name: "github-pr-workflow".to_string(), + version: 3, + snippet_names: vec![], + force_activated: false, + }]) + .unwrap(); + thread.add_event(crate::types::event::EventKind::ActionFailed { + step_id: StepId::new(), + action_name: "shell".to_string(), + call_id: "call_1".to_string(), + error: "authentication required for credential github".to_string(), + params_summary: None, + }); + + let trace = crate::executor::trace::build_trace(&thread); + let payload = build_skill_gap_payload(&thread, &trace, &thread.active_skills()).unwrap(); + let hints = payload["repair_hints"].as_array().unwrap(); + + assert!( + hints + .iter() + .any(|hint| hint.as_str() == Some("missing_prerequisite")), + "recoverable auth failures should still produce missing_prerequisite: {payload}" + ); + } + + #[test] + fn learning_terminal_state_accepts_failed_threads() { + let failed_event = crate::types::event::EventKind::StateChanged { + from: ThreadState::Running, + to: ThreadState::Failed, + reason: Some("boom".into()), + }; + assert_eq!( + learning_terminal_state(&failed_event), + Some(ThreadState::Failed) + ); + + let done_event = crate::types::event::EventKind::StateChanged { + from: ThreadState::Completed, + to: ThreadState::Done, + reason: None, + }; + assert_eq!( + learning_terminal_state(&done_event), + Some(ThreadState::Done) + ); + } + + #[test] + fn thread_completed_successfully_requires_done_without_action_failures() { + let project_id = ProjectId::new(); + + let mut clean_thread = Thread::new( + "clean success", + ThreadType::Foreground, + project_id, + "alice", + ThreadConfig::default(), + ); + clean_thread.state = ThreadState::Done; + let clean_trace = crate::executor::trace::build_trace(&clean_thread); + assert!(thread_completed_successfully(&clean_thread, &clean_trace)); + + let mut failing_thread = Thread::new( + "tool failure", + ThreadType::Foreground, + project_id, + "alice", + ThreadConfig::default(), + ); + failing_thread.state = ThreadState::Done; + failing_thread.add_event(crate::types::event::EventKind::ActionFailed { + step_id: StepId::new(), + action_name: "shell".to_string(), + call_id: "call_1".to_string(), + error: "gh auth status: not authenticated".to_string(), + params_summary: Some("gh auth status".to_string()), + }); + let failing_trace = crate::executor::trace::build_trace(&failing_thread); + assert!(!thread_completed_successfully( + &failing_thread, + &failing_trace + )); + } + + #[tokio::test] + async fn process_skill_repair_output_updates_skill_and_records_repair() { + let store = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + let skill_doc = make_skill_doc(project_id, "alice", "github-pr-workflow"); + let skill_doc_id = skill_doc.id; + store.save_memory_doc(&skill_doc).await.unwrap(); + + let mut mission = Mission::new( + project_id, + "alice", + "skill-repair", + SKILL_REPAIR_GOAL, + MissionCadence::Manual, + ); + mission.metadata = serde_json::json!({"skill_repair": true}); + mission.last_trigger_payload = Some(serde_json::json!({ + "source_thread_id": "thread-123", + "active_skills": [{ + "doc_id": skill_doc_id, + "name": "github-pr-workflow", + "version": 1, + "snippet_names": [], + "force_activated": false + }] + })); + + let response = serde_json::json!({ + "doc_id": skill_doc_id, + "repair_type": "missing_verification", + "summary": "Added a smoke-test step after the gh command.", + "updated_content": "1. Run `gh auth status`\n2. Run the PR command\n3. Verify with `gh pr view`", + "description": "GitHub PR workflow with auth and verification", + }) + .to_string(); + + process_skill_repair_output(&(store.clone() as Arc), &mission, &response) + .await + .unwrap(); + + let updated = store.load_memory_doc(skill_doc_id).await.unwrap().unwrap(); + let meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(meta.version, 2); + assert_eq!(meta.parent_version, Some(1)); + assert_eq!( + updated.content, + "1. Run `gh auth status`\n2. Run the PR command\n3. Verify with `gh pr view`" + ); + assert_eq!(meta.repairs.len(), 1); + assert_eq!( + meta.repairs[0].repair_type, + SkillRepairType::MissingVerification + ); + assert_eq!( + meta.repairs[0].source_thread_id.as_deref(), + Some("thread-123") + ); + assert_eq!(meta.revisions.len(), 1); + assert_eq!(meta.revisions[0].content, "Original skill content"); + } + + #[tokio::test] + async fn process_skill_repair_output_rejects_stale_trigger_version() { + let store = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + let mut skill_doc = make_skill_doc(project_id, "alice", "github-pr-workflow"); + let skill_doc_id = skill_doc.id; + skill_doc.content = "Skill content already updated to v2".to_string(); + let mut meta: V2SkillMetadata = serde_json::from_value(skill_doc.metadata.clone()).unwrap(); + meta.version = 2; + meta.parent_version = Some(1); + skill_doc.metadata = serde_json::to_value(&meta).unwrap(); + store.save_memory_doc(&skill_doc).await.unwrap(); + + let mut mission = Mission::new( + project_id, + "alice", + "skill-repair", + SKILL_REPAIR_GOAL, + MissionCadence::Manual, + ); + mission.metadata = serde_json::json!({"skill_repair": true}); + mission.last_trigger_payload = Some(serde_json::json!({ + "source_thread_id": "thread-123", + "active_skills": [{ + "doc_id": skill_doc_id, + "name": "github-pr-workflow", + "version": 1, + "snippet_names": [], + "force_activated": false + }] + })); + + let response = serde_json::json!({ + "doc_id": skill_doc_id, + "repair_type": "missing_verification", + "summary": "Stale repair output.", + "updated_content": "1. Run the stale command\n2. Verify it" + }) + .to_string(); + + let err = + process_skill_repair_output(&(store.clone() as Arc), &mission, &response) + .await + .unwrap_err(); + match err { + EngineError::Skill { reason } => assert!( + reason.contains("version conflict"), + "expected version conflict, got: {reason}" + ), + other => panic!("expected skill error, got: {other:?}"), + } + + let updated = store.load_memory_doc(skill_doc_id).await.unwrap().unwrap(); + let updated_meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(updated.content, "Skill content already updated to v2"); + assert_eq!(updated_meta.version, 2); + assert!(updated_meta.repairs.is_empty()); + } + + #[tokio::test] + async fn process_skill_repair_output_rejects_empty_content() { + let store = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + let skill_doc = make_skill_doc(project_id, "alice", "github-pr-workflow"); + let skill_doc_id = skill_doc.id; + store.save_memory_doc(&skill_doc).await.unwrap(); + + let mut mission = Mission::new( + project_id, + "alice", + "skill-repair", + SKILL_REPAIR_GOAL, + MissionCadence::Manual, + ); + mission.metadata = serde_json::json!({"skill_repair": true}); + mission.last_trigger_payload = Some(serde_json::json!({ + "source_thread_id": "thread-123", + "active_skills": [{ + "doc_id": skill_doc_id, + "name": "github-pr-workflow", + "version": 1, + "snippet_names": [], + "force_activated": false + }] + })); + + let response = serde_json::json!({ + "doc_id": skill_doc_id, + "repair_type": "missing_verification", + "summary": "This should be rejected.", + "updated_content": " " + }) + .to_string(); + + let err = + process_skill_repair_output(&(store.clone() as Arc), &mission, &response) + .await + .unwrap_err(); + match err { + EngineError::Skill { reason } => assert!( + reason.contains("empty updated_content"), + "expected empty-content validation, got: {reason}" + ), + other => panic!("expected skill error, got: {other:?}"), + } + + let updated = store.load_memory_doc(skill_doc_id).await.unwrap().unwrap(); + let updated_meta: V2SkillMetadata = serde_json::from_value(updated.metadata).unwrap(); + assert_eq!(updated.content, "Original skill content"); + assert_eq!(updated_meta.version, 1); + assert!(updated_meta.repairs.is_empty()); + } + + #[tokio::test] + async fn process_skill_repair_output_rejects_shared_skill_updates() { + let store = Arc::new(TestStore::new()); + let project_id = ProjectId::new(); + let skill_doc = make_skill_doc(project_id, shared_owner_id(), "github-pr-workflow"); + let skill_doc_id = skill_doc.id; + store.save_memory_doc(&skill_doc).await.unwrap(); + + let mut mission = Mission::new( + project_id, + "alice", + "skill-repair", + SKILL_REPAIR_GOAL, + MissionCadence::Manual, + ); + mission.metadata = serde_json::json!({"skill_repair": true}); + mission.last_trigger_payload = Some(serde_json::json!({ + "source_thread_id": "thread-123", + "active_skills": [{ + "doc_id": skill_doc_id, + "name": "github-pr-workflow", + "version": 1, + "snippet_names": [], + "force_activated": false + }] + })); + + let response = serde_json::json!({ + "doc_id": skill_doc_id, + "repair_type": "missing_verification", + "summary": "Attempted shared skill update.", + "updated_content": "1. Verify auth\n2. Run the command" + }) + .to_string(); + + let err = + process_skill_repair_output(&(store.clone() as Arc), &mission, &response) + .await + .unwrap_err(); + match err { + EngineError::AccessDenied { user_id, entity } => { + assert_eq!(user_id, "alice"); + assert!(entity.contains(&skill_doc_id.0.to_string())); + } + other => panic!("expected access denied, got: {other:?}"), + } + + let unchanged = store.load_memory_doc(skill_doc_id).await.unwrap().unwrap(); + let meta: V2SkillMetadata = serde_json::from_value(unchanged.metadata).unwrap(); + assert_eq!(unchanged.content, "Original skill content"); + assert_eq!(meta.version, 1); + assert!(meta.repairs.is_empty()); + } + + #[tokio::test] + async fn dedup_event_does_not_evict_entries_from_other_missions() { + // Regression for the cross-mission dedup window collision: a + // mission with a *short* window must not be able to evict a fresh + // entry belonging to a mission with a *longer* window. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + + let mission_a = MissionId::new(); + let mission_b = MissionId::new(); + + // Mission B (long window) gets a stale entry for key "y" — set it + // 120 seconds in the past so a 60s-window check would consider it + // expired but a 3600s-window check still considers it fresh. + { + let mut table = mgr.dedup_table.write().await; + table.insert( + (mission_b, "y".to_string()), + chrono::Utc::now() - chrono::Duration::seconds(120), + ); + } + + // Mission A (short 60s window) fires for an unrelated key. + let first = mgr.dedup_event(mission_a, "x", 60).await; + assert!(!first, "first sighting of (A, x) should not be flagged"); + + // Mission B's entry must survive — its own window is 3600s, and + // 120s < 3600s, so the next dedup call from B for "y" should still + // see it as a duplicate. + let b_again = mgr.dedup_event(mission_b, "y", 3600).await; + assert!( + b_again, + "(B, y) is 120s old with a 3600s window — must still register as duplicate after A's call" + ); + } + + #[tokio::test] + async fn user_rate_slot_not_consumed_by_failed_fire() { + // Regression for the rate-limiter self-DoS: when fire_mission + // refuses (e.g. budget/concurrent gate) the per-user slot must + // remain available so sustained refusals don't lock the user out. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + + // Sanity: an empty window allows fires. + assert!(mgr.check_user_rate("alice").await); + + // Drive a fire that's guaranteed to early-out before record_user_rate. + // The simplest deterministic refusal is `fire_mission` against a + // mission whose owner doesn't match — that returns AccessDenied + // before reaching the rate check, so it doesn't exercise the + // rate-limit path. Instead, drive `record_user_rate` and + // `check_user_rate` directly to pin the contract: a check that + // doesn't get followed by a record leaves the slot free. + let allowed_before = mgr.check_user_rate("alice").await; + assert!(allowed_before); + + // Snapshot the queue size — must be unchanged after a check-only. + let snapshot_after_check = { + let log = mgr.user_fire_log.read().await; + log.get("alice").map(|q| q.len()).unwrap_or(0) + }; + assert_eq!( + snapshot_after_check, 0, + "check_user_rate must not consume a slot on its own" + ); + + // After a successful fire would have called record_user_rate the + // queue grows by exactly one. + mgr.record_user_rate("alice").await; + let snapshot_after_record = { + let log = mgr.user_fire_log.read().await; + log.get("alice").map(|q| q.len()).unwrap_or(0) + }; + assert_eq!( + snapshot_after_record, 1, + "record_user_rate must append exactly one entry" + ); + } + + #[test] + fn build_skill_gap_payload_skips_read_only_shell_workflows() { + let project_id = ProjectId::new(); + let mut thread = Thread::new( + "inspect github pull requests", + ThreadType::Foreground, + project_id, + "alice", + ThreadConfig::default(), + ); + thread.state = ThreadState::Done; + thread + .set_active_skills(&[ActiveSkillProvenance { + doc_id: DocId::new(), + name: "github-pr-workflow".to_string(), + version: 1, + snippet_names: vec![], + force_activated: false, + }]) + .unwrap(); + thread.add_event(crate::types::event::EventKind::ActionExecuted { + step_id: StepId::new(), + action_name: "shell".to_string(), + call_id: "call_1".to_string(), + params_summary: Some("gh pr list --repo nearai/ironclaw".to_string()), + duration_ms: 15, + }); + + let trace = crate::executor::trace::build_trace(&thread); + assert!( + build_skill_gap_payload(&thread, &trace, &thread.active_skills()).is_none(), + "read-only shell workflows should not trigger skill repair" + ); + } + + // ── next_cron_fire_required + cooldown regression tests ────── + + /// A 7-field cron expression year-locked to a year that's already in the + /// past. `cron::Schedule` parses it cleanly but `upcoming(...).next()` + /// returns `None`, which is exactly the `Ok(None)` case the + /// `next_cron_fire_required` helper guards against. + const PAST_YEAR_CRON: &str = "0 0 0 1 1 * 2020"; + + #[tokio::test] + async fn create_mission_rejects_unschedulable_cron() { + // Regression: previously `create_mission` accepted Ok(None) and + // persisted an Active mission with `next_fire_at = None` — the + // exact failure mode of #1944. Now it must fail fast. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let err = mgr + .create_mission( + project_id, + "alice", + "unschedulable", + "g", + MissionCadence::Cron { + expression: PAST_YEAR_CRON.into(), + timezone: None, + }, + vec![], + ) + .await + .expect_err("create_mission must reject cron with no upcoming fire time"); + + assert!( + matches!(err, EngineError::InvalidCadence { .. }), + "expected InvalidCadence, got: {err:?}" + ); + + // No mission should be persisted, no entry in active. + assert!(store.missions.read().await.is_empty()); + } + + #[tokio::test] + async fn update_mission_rejects_switch_to_unschedulable_cron() { + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "manual-then-cron", + "g", + MissionCadence::Manual, + vec![], + ) + .await + .unwrap(); + + let err = mgr + .update_mission( + id, + "alice", + MissionUpdate { + cadence: Some(MissionCadence::Cron { + expression: PAST_YEAR_CRON.into(), + timezone: None, + }), + ..Default::default() + }, + ) + .await + .expect_err("update_mission must reject cron with no upcoming fire time"); + + assert!(matches!(err, EngineError::InvalidCadence { .. })); + // Original Manual cadence should be preserved on the persisted record. + let reloaded = mgr.get_mission(id).await.unwrap().unwrap(); + assert!(matches!(reloaded.cadence, MissionCadence::Manual)); + } + + #[tokio::test] + async fn resume_mission_rejects_unschedulable_cron() { + // Build a paused cron mission whose schedule is fine, then mutate the + // persisted record to a year-locked expression and try to resume. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "resume-bad", + "g", + MissionCadence::Cron { + expression: "0 9 * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + mgr.pause_mission(id, "alice").await.unwrap(); + + // Tamper with the persisted cadence to simulate a stored mission that + // can no longer fire (e.g. operator edited the database, or year-locked + // schedule rolled past). + { + let mut missions = store.missions.write().await; + if let Some(m) = missions.get_mut(&id) { + m.cadence = MissionCadence::Cron { + expression: PAST_YEAR_CRON.into(), + timezone: None, + }; + } + } + + let err = mgr + .resume_mission(id, "alice") + .await + .expect_err("resume_mission must reject cron with no upcoming fire time"); + assert!(matches!(err, EngineError::InvalidCadence { .. })); + + // Mission must remain paused — resume failed before any state change. + let reloaded = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!(reloaded.status, MissionStatus::Paused); + } + + #[tokio::test] + async fn pause_and_complete_drop_cooldown_entry() { + // Regression: `last_fire_attempt` was previously only ever inserted, + // never pruned. Pausing or completing a mission must drop its + // cooldown entry so the in-memory map can't grow unbounded. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + // Mission A — paused after a fire. + let id_a = mgr + .create_mission( + project_id, + "alice", + "pause-cleanup", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + mgr.fire_mission(id_a, "alice", None).await.unwrap(); + assert!(mgr.last_fire_attempt.read().await.contains_key(&id_a)); + mgr.pause_mission(id_a, "alice").await.unwrap(); + assert!( + !mgr.last_fire_attempt.read().await.contains_key(&id_a), + "pause_mission must drop the cooldown entry" + ); + + // Mission B — completed after a fire. + let id_b = mgr + .create_mission( + project_id, + "alice", + "complete-cleanup", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + mgr.fire_mission(id_b, "alice", None).await.unwrap(); + assert!(mgr.last_fire_attempt.read().await.contains_key(&id_b)); + mgr.complete_mission(id_b).await.unwrap(); + assert!( + !mgr.last_fire_attempt.read().await.contains_key(&id_b), + "complete_mission must drop the cooldown entry" + ); + } + + #[tokio::test] + async fn tick_cooldown_suppresses_re_fire_on_save_failure() { + // Regression for the runaway-re-fire concern: when save_mission fails + // after a successful spawn, the persisted `next_fire_at` AND + // `last_fire_at` stay at their pre-fire values, but the in-memory + // `last_fire_attempt[mid]` is set to the new fire instant. The + // mismatch between in-memory and persisted `last_fire_at` is what + // tells tick to arm the cooldown — without that signal every + // subsequent tick would re-fire the same mission and spawn + // duplicate threads up to the daily budget. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "cooldown", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + + // First fire arms the in-memory `last_fire_attempt` map. + let first = mgr.fire_mission(id, "alice", None).await.unwrap(); + assert!(first.is_some(), "first fire should spawn a thread"); + + // Simulate the post-save-failure state explicitly: rewind + // `next_fire_at` into the past, reset `threads_today` so the budget + // can't be what's blocking, AND clobber `last_fire_at` so it no + // longer matches the in-memory `last_fire_attempt[mid]` instant. + // Together these mimic exactly the state a failed `save_mission` + // call would leave: in-memory recorded the fire, the store didn't. + { + let mut missions = store.missions.write().await; + let mission = missions.get_mut(&id).unwrap(); + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::seconds(120)); + mission.threads_today = 0; + mission.last_fire_at = None; + } + + // tick must suppress the second fire — it can prove the persisted + // record is stale because in-memory last_fire_attempt holds an + // instant the persisted `last_fire_at` doesn't. + let spawned = mgr.tick("alice").await.unwrap(); + assert!( + spawned.is_empty(), + "tick must skip mission whose persisted last_fire_at is stale, got: {spawned:?}" + ); + } + + #[tokio::test] + async fn fire_mission_arms_cooldown_before_save_mission() { + // Race regression: a concurrent tick observing the state between + // `save_mission` completion and the in-memory cooldown insert would + // see no cooldown entry, evaluate the mismatch check to false, and + // (if next_fire_at is in the past) re-fire immediately. Fix: insert + // the cooldown entry BEFORE calling save_mission. + // + // We verify the order by gating save_mission with a oneshot channel + // and asserting `last_fire_attempt[mid]` is already populated while + // save is still in flight. + use tokio::sync::Notify; + + let store = Arc::new(TestStore::new()); + let mgr = Arc::new(make_mission_manager(Arc::clone(&store) as Arc)); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "race-test", + "g", + MissionCadence::Cron { + expression: "0 9 * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + + // Block the next save_mission. The first save_mission call below + // (from create_mission's path) already happened — `block_next_save_mission` + // installs the gate AFTER create, so it only catches fire_mission's save. + let release = store.block_next_save_mission().await; + let started = Arc::new(Notify::new()); + let started_clone = Arc::clone(&started); + let store_clone = Arc::clone(&store); + // Spawn a watcher that translates `save_mission_started` into our own + // `started` notification. We can't share the TestStore's Notify across + // tasks via `notified()` cleanly without a permit, so wrap it. + tokio::spawn(async move { + store_clone.save_mission_started.notified().await; + started_clone.notify_one(); + }); + + // Spawn fire_mission in a task — it will block inside save_mission. + let mgr_clone = Arc::clone(&mgr); + let fire_task = tokio::spawn(async move { + mgr_clone + .fire_mission(id, "alice", None) + .await + .expect("fire should succeed once save is unblocked") + }); + + // Wait until save_mission has begun (inside the gate). + started.notified().await; + + // At this point save_mission is parked. The cooldown MUST already be + // armed because the fix inserts before save. + assert!( + mgr.last_fire_attempt.read().await.contains_key(&id), + "last_fire_attempt[mid] must be populated before save_mission begins; \ + a concurrent tick in this window would otherwise see no cooldown entry" + ); + + // Unblock save and let fire_mission complete. + release.send(()).unwrap(); + let thread_id = fire_task.await.unwrap(); + assert!(thread_id.is_some(), "fire should spawn a thread"); + } + + #[tokio::test] + async fn tick_does_not_re_fire_corrupted_cron_within_cooldown_window() { + // Regression: when `next_cron_fire(expression)` returns Err inside + // fire_mission (corrupted persisted expression), the previous code + // stamped `last_fire_at = fire_instant` anyway. Save then succeeded + // with last_fire_at matching the in-memory value, so tick's + // mismatch detector saw "save succeeded" and the cooldown was + // never armed. With `next_fire_at` still in the past (preserved + // because the cron crate couldn't compute a new value), every + // subsequent tick re-fired the same mission until `max_threads_per_day` + // was exhausted. + // + // Fix: only stamp `last_fire_at = fire_instant` when cron actually + // advanced. On a parse error, leave persisted last_fire_at at its + // OLD value so the in-memory vs persisted mismatch arms the + // cooldown via the same code path as a save failure. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "corrupt-cron", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + + // Corrupt the persisted expression. fire_mission's cron advance + // call will fail and (with the fix) leave last_fire_at at the OLD + // value the test fixture started with. + { + let mut missions = store.missions.write().await; + let mission = missions.get_mut(&id).unwrap(); + if let MissionCadence::Cron { + ref mut expression, .. + } = mission.cadence + { + *expression = "this is not a cron".to_string(); + } + // Force next_fire_at into the past so should_fire is true. + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::seconds(60)); + } + + // First fire spawns a thread successfully despite the corrupt cron + // (regression test `fire_mission_with_corrupt_cron_expression_does_not_orphan_thread` + // pins this behavior). next_fire_at stays at its past value. + let first = mgr.fire_mission(id, "alice", None).await.unwrap(); + assert!( + first.is_some(), + "first fire should spawn despite corrupt cron" + ); + + // tick must NOT re-fire the corrupted mission. Without the fix the + // cooldown was not armed (persisted last_fire_at == fire_instant == + // in-memory), and tick would call fire_mission again every cycle. + let spawned = mgr.tick("alice").await.unwrap(); + assert!( + spawned.is_empty(), + "tick must not re-fire a mission whose cron advance failed; \ + cooldown should be armed via last_fire_at vs in-memory mismatch, \ + got: {spawned:?}" + ); + + // Sanity: the persisted last_fire_at is still its pre-fire value + // (None for a freshly-created mission whose first fire failed to + // advance), confirming the mismatch-arming mechanism. + let reloaded = mgr.get_mission(id).await.unwrap().unwrap(); + assert_eq!( + reloaded.last_fire_at, None, + "last_fire_at must NOT be stamped when cron advance failed" + ); + } + + #[tokio::test] + async fn tick_does_not_throttle_high_frequency_cron_after_successful_fire() { + // Regression for the inverse failure mode: the cooldown must NOT + // throttle a normally-firing high-frequency cron. An earlier + // implementation armed the cooldown unconditionally on every + // successful fire, which silently dropped roughly half of the + // events for `* * * * *` (every-minute) crons because the 60 s + // tick interval fell inside the 90 s cooldown window. The fix: + // only arm the cooldown when the persisted `last_fire_at` does + // NOT match the in-memory `last_fire_attempt` value — i.e. only + // in the failed-save regime. + let store = Arc::new(TestStore::new()); + let mgr = make_mission_manager(Arc::clone(&store) as Arc); + let project_id = ProjectId::new(); + + let id = mgr + .create_mission( + project_id, + "alice", + "high-freq", + "g", + MissionCadence::Cron { + expression: "* * * * *".into(), + timezone: None, + }, + vec![], + ) + .await + .unwrap(); + + // First fire records the in-memory cooldown entry AND persists + // `last_fire_at = fire_instant`. The two values are the same + // (`fire_mission` uses a single `Utc::now()` for both writes). + let first = mgr.fire_mission(id, "alice", None).await.unwrap(); + assert!(first.is_some(), "first fire should spawn a thread"); + + // Mimic "tick runs ~1 minute later, schedule advanced normally": + // rewind `next_fire_at` to a moment in the past that is STRICTLY + // LATER than the fire instant. Crucially, leave `last_fire_at` + // alone — it still equals `last_fire_attempt[mid]`, so the + // cooldown's mismatch detector says "save succeeded, do not + // throttle." Reset `threads_today` so the daily budget isn't + // what's blocking. + { + let mut missions = store.missions.write().await; + let mission = missions.get_mut(&id).unwrap(); + // 1 ms earlier than now, but still later than the original + // fire instant since fire_mission ran microseconds ago. + mission.next_fire_at = Some(chrono::Utc::now() - chrono::Duration::milliseconds(1)); + mission.threads_today = 0; + } + + let spawned = mgr.tick("alice").await.unwrap(); + assert_eq!( + spawned.len(), + 1, + "tick must fire a high-frequency cron after a successful fire — \ + cooldown must not throttle the success path, got spawned={spawned:?}" + ); + } +} diff --git a/crates/ironclaw_engine/src/runtime/mod.rs b/crates/ironclaw_engine/src/runtime/mod.rs new file mode 100644 index 00000000000..a10822ecc14 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/mod.rs @@ -0,0 +1,18 @@ +//! Thread lifecycle management. +//! +//! - [`ThreadManager`] — top-level orchestrator for spawning and supervising threads +//! - [`ThreadTree`] — parent-child relationship tracking +//! - [`messaging`] — inter-thread signal channel + +pub mod conversation; +pub mod lease_refresh; +pub mod manager; +pub mod messaging; +pub mod mission; +pub mod tree; + +pub use conversation::ConversationManager; +pub use manager::ThreadManager; +pub use messaging::ThreadOutcome; +pub use mission::MissionManager; +pub use tree::ThreadTree; diff --git a/crates/ironclaw_engine/src/runtime/tree.rs b/crates/ironclaw_engine/src/runtime/tree.rs new file mode 100644 index 00000000000..3a754dea6a4 --- /dev/null +++ b/crates/ironclaw_engine/src/runtime/tree.rs @@ -0,0 +1,129 @@ +//! Thread tree — parent-child relationship tracking. + +use std::collections::HashMap; + +use crate::types::thread::ThreadId; + +/// Manages parent-child thread relationships. +/// +/// Simple in-memory tree. Threads form a forest (multiple roots). +#[derive(Debug, Default)] +pub struct ThreadTree { + /// child → parent + parents: HashMap, + /// parent → children (ordered by insertion) + children: HashMap>, +} + +impl ThreadTree { + pub fn new() -> Self { + Self::default() + } + + /// Register a parent-child relationship. + pub fn add_child(&mut self, parent_id: ThreadId, child_id: ThreadId) { + self.parents.insert(child_id, parent_id); + self.children.entry(parent_id).or_default().push(child_id); + } + + /// Get the parent of a thread, if any. + pub fn parent_of(&self, thread_id: ThreadId) -> Option { + self.parents.get(&thread_id).copied() + } + + /// Get the children of a thread. + pub fn children_of(&self, thread_id: ThreadId) -> &[ThreadId] { + self.children + .get(&thread_id) + .map(Vec::as_slice) + .unwrap_or(&[]) + } + + /// Walk up the tree to collect all ancestors (parent, grandparent, ...). + pub fn ancestors(&self, thread_id: ThreadId) -> Vec { + let mut result = Vec::new(); + let mut current = thread_id; + while let Some(parent) = self.parents.get(¤t) { + result.push(*parent); + current = *parent; + } + result + } + + /// Remove a thread from the tree. Does not remove its children. + pub fn remove(&mut self, thread_id: ThreadId) { + if let Some(parent) = self.parents.remove(&thread_id) + && let Some(siblings) = self.children.get_mut(&parent) + { + siblings.retain(|id| *id != thread_id); + } + // Orphan any children (their parent_id entries become stale) + self.children.remove(&thread_id); + } + + /// Check if a thread is a root (no parent). + pub fn is_root(&self, thread_id: ThreadId) -> bool { + !self.parents.contains_key(&thread_id) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn add_and_query() { + let mut tree = ThreadTree::new(); + let parent = ThreadId::new(); + let child1 = ThreadId::new(); + let child2 = ThreadId::new(); + + tree.add_child(parent, child1); + tree.add_child(parent, child2); + + assert_eq!(tree.parent_of(child1), Some(parent)); + assert_eq!(tree.parent_of(child2), Some(parent)); + assert_eq!(tree.children_of(parent).len(), 2); + assert!(tree.is_root(parent)); + assert!(!tree.is_root(child1)); + } + + #[test] + fn ancestors_walk_up() { + let mut tree = ThreadTree::new(); + let root = ThreadId::new(); + let mid = ThreadId::new(); + let leaf = ThreadId::new(); + + tree.add_child(root, mid); + tree.add_child(mid, leaf); + + let ancestors = tree.ancestors(leaf); + assert_eq!(ancestors, vec![mid, root]); + } + + #[test] + fn remove_detaches_from_parent() { + let mut tree = ThreadTree::new(); + let parent = ThreadId::new(); + let child = ThreadId::new(); + + tree.add_child(parent, child); + tree.remove(child); + + assert_eq!(tree.parent_of(child), None); + assert!(tree.children_of(parent).is_empty()); + } + + #[test] + fn children_of_unknown_returns_empty() { + let tree = ThreadTree::new(); + assert!(tree.children_of(ThreadId::new()).is_empty()); + } + + #[test] + fn ancestors_of_root_is_empty() { + let tree = ThreadTree::new(); + assert!(tree.ancestors(ThreadId::new()).is_empty()); + } +} diff --git a/crates/ironclaw_engine/src/traits/effect.rs b/crates/ironclaw_engine/src/traits/effect.rs new file mode 100644 index 00000000000..1fed99bd6c7 --- /dev/null +++ b/crates/ironclaw_engine/src/traits/effect.rs @@ -0,0 +1,65 @@ +//! Effect executor trait. +//! +//! The engine delegates actual action execution to the host through this +//! trait. The main crate implements it by wrapping `ToolRegistry` and +//! `SafetyLayer` — the engine itself has no knowledge of specific tools. + +use crate::types::capability::{ActionDef, CapabilityLease}; +use crate::types::error::EngineError; +use crate::types::project::ProjectId; +use crate::types::step::{ActionResult, StepId}; +use crate::types::thread::{ThreadId, ThreadType}; +use ironclaw_common::ValidTimezone; + +/// Contextual information about the thread requesting an effect. +/// +/// Passed to the executor so it can make context-dependent decisions +/// (e.g. different tool behavior in background vs foreground threads). +#[derive(Debug, Clone)] +pub struct ThreadExecutionContext { + pub thread_id: ThreadId, + pub thread_type: ThreadType, + pub project_id: ProjectId, + pub user_id: String, + pub step_id: StepId, + pub current_call_id: Option, + /// The channel this thread's conversation originated from (e.g. "gateway", "repl"). + /// Used by mission_create to default `notify_channels` to the current channel. + pub source_channel: Option, + /// Validated IANA timezone of the user (e.g. "America/New_York"). + /// Used by mission_create to default cron timezone, and exposed to CodeAct scripts. + pub user_timezone: Option, +} + +/// Abstraction over capability action execution. +/// +/// The main crate implements this by wrapping its `ToolRegistry`, `SafetyLayer`, +/// and tool execution pipeline. The engine calls `execute_action` and gets back +/// a result — all safety, sanitization, and actual tool invocation happens in +/// the host. +#[async_trait::async_trait] +pub trait EffectExecutor: Send + Sync { + /// Execute a capability action. + /// + /// The executor is responsible for: + /// 1. Looking up the actual tool implementation + /// 2. Validating parameters + /// 3. Applying safety checks (sanitization, leak detection) + /// 4. Executing the tool + /// 5. Returning the result + async fn execute_action( + &self, + action_name: &str, + parameters: serde_json::Value, + lease: &CapabilityLease, + context: &ThreadExecutionContext, + ) -> Result; + + /// List available actions given the current set of active leases. + /// + /// Used to build the action definitions sent to the LLM. + async fn available_actions( + &self, + leases: &[CapabilityLease], + ) -> Result, EngineError>; +} diff --git a/crates/ironclaw_engine/src/traits/llm.rs b/crates/ironclaw_engine/src/traits/llm.rs new file mode 100644 index 00000000000..972161553c2 --- /dev/null +++ b/crates/ironclaw_engine/src/traits/llm.rs @@ -0,0 +1,61 @@ +//! LLM backend trait. +//! +//! The engine's abstraction over language model providers. Deliberately +//! simpler than the main crate's `LlmProvider` — the engine only needs +//! to make completion calls. Cost tracking, caching, retry, and circuit +//! breaking are host concerns handled by the bridge adapter. + +use std::collections::HashMap; + +use crate::types::capability::ActionDef; +use crate::types::error::EngineError; +use crate::types::message::ThreadMessage; +use crate::types::step::{LlmResponse, TokenUsage}; + +/// Configuration for a single LLM call. +#[derive(Debug, Clone, Default)] +pub struct LlmCallConfig { + /// Maximum tokens to generate. + pub max_tokens: Option, + /// Sampling temperature. + pub temperature: Option, + /// When true, the LLM should not return action calls. + pub force_text: bool, + /// Depth in the recursive call tree (0 = root, 1+ = sub-call). + /// Implementations can use this to route to cheaper models for sub-calls. + pub depth: u32, + /// Optional per-call model override. When set, the bridge adapter forwards + /// this to the underlying `LlmProvider` via `CompletionRequest::model`. + /// Providers that don't support per-request overrides will fall back to + /// their configured model and log a warning. + pub model: Option, + /// Opaque metadata forwarded to the LLM provider. + pub metadata: HashMap, +} + +/// Output from a single LLM call. +#[derive(Debug, Clone)] +pub struct LlmOutput { + pub response: LlmResponse, + pub usage: TokenUsage, +} + +/// Abstraction over language model providers. +/// +/// The main crate implements this by wrapping its `LlmProvider` trait, +/// converting between `ThreadMessage` and `ChatMessage`. +#[async_trait::async_trait] +pub trait LlmBackend: Send + Sync { + /// Call the LLM with conversation messages and available action definitions. + /// + /// Returns either a text response or a set of action calls. + async fn complete( + &self, + messages: &[ThreadMessage], + actions: &[ActionDef], + config: &LlmCallConfig, + ) -> Result; + + /// The model identifier (e.g. "gpt-4", "claude-opus-4-20250514"). + fn model_name(&self) -> &str; +} diff --git a/crates/ironclaw_engine/src/traits/mod.rs b/crates/ironclaw_engine/src/traits/mod.rs new file mode 100644 index 00000000000..21345426a1f --- /dev/null +++ b/crates/ironclaw_engine/src/traits/mod.rs @@ -0,0 +1,9 @@ +//! External dependency traits. +//! +//! The engine defines these traits; the host (main ironclaw crate) +//! implements them via bridge adapters over existing infrastructure. + +pub mod effect; +pub mod llm; +pub mod store; +pub mod workspace; diff --git a/crates/ironclaw_engine/src/traits/store.rs b/crates/ironclaw_engine/src/traits/store.rs new file mode 100644 index 00000000000..83dadda3019 --- /dev/null +++ b/crates/ironclaw_engine/src/traits/store.rs @@ -0,0 +1,204 @@ +//! Storage trait for engine persistence. +//! +//! Defines CRUD operations for all engine types. The main crate implements +//! this by wrapping its dual-backend `Database` trait (PostgreSQL + libSQL). + +use crate::types::capability::{CapabilityLease, LeaseId}; +use crate::types::conversation::{ConversationId, ConversationSurface}; +use crate::types::error::EngineError; +use crate::types::event::ThreadEvent; +use crate::types::memory::{DocId, MemoryDoc}; +use crate::types::mission::{Mission, MissionId, MissionStatus}; +use crate::types::project::{Project, ProjectId}; +use crate::types::step::Step; +use crate::types::thread::{Thread, ThreadId, ThreadState}; +use crate::types::{is_shared_owner, shared_owner_candidates}; + +/// Persistence abstraction for the engine. +#[async_trait::async_trait] +pub trait Store: Send + Sync { + // ── Thread operations ─────────────────────────────────── + + async fn save_thread(&self, thread: &Thread) -> Result<(), EngineError>; + async fn load_thread(&self, id: ThreadId) -> Result, EngineError>; + async fn list_threads( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError>; + async fn update_thread_state( + &self, + id: ThreadId, + state: ThreadState, + ) -> Result<(), EngineError>; + + // ── Step operations ───────────────────────────────────── + + async fn save_step(&self, step: &Step) -> Result<(), EngineError>; + async fn load_steps(&self, thread_id: ThreadId) -> Result, EngineError>; + + // ── Event operations ──────────────────────────────────── + + async fn append_events(&self, events: &[ThreadEvent]) -> Result<(), EngineError>; + async fn load_events(&self, thread_id: ThreadId) -> Result, EngineError>; + + // ── Project operations ────────────────────────────────── + + async fn save_project(&self, project: &Project) -> Result<(), EngineError>; + async fn load_project(&self, id: ProjectId) -> Result, EngineError>; + async fn list_projects(&self, user_id: &str) -> Result, EngineError> { + Err(EngineError::Store { + reason: format!("Store::list_projects not implemented for user '{user_id}'"), + }) + } + async fn list_all_projects(&self) -> Result, EngineError> { + Err(EngineError::Store { + reason: "Store::list_all_projects not implemented".into(), + }) + } + + // ── Conversation operations ───────────────────────────── + + async fn save_conversation( + &self, + conversation: &ConversationSurface, + ) -> Result<(), EngineError> { + Err(EngineError::Store { + reason: format!( + "Store::save_conversation not implemented for conversation '{}'", + conversation.id + ), + }) + } + async fn load_conversation( + &self, + id: ConversationId, + ) -> Result, EngineError> { + Err(EngineError::Store { + reason: format!("Store::load_conversation not implemented for '{id}'"), + }) + } + async fn list_conversations( + &self, + user_id: &str, + ) -> Result, EngineError> { + Err(EngineError::Store { + reason: format!("Store::list_conversations not implemented for user '{user_id}'"), + }) + } + + // ── Memory doc operations ─────────────────────────────── + + async fn save_memory_doc(&self, doc: &MemoryDoc) -> Result<(), EngineError>; + async fn load_memory_doc(&self, id: DocId) -> Result, EngineError>; + async fn list_memory_docs( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError>; + + /// List memory docs visible to a user: their own docs + shared docs. + /// + /// This is the "shared space" pattern: admins can install skills and + /// knowledge under the shared owner id, and they're visible to all users + /// alongside their personal docs. Used for skill listing, context + /// retrieval, and any place where shared knowledge should be accessible. + async fn list_memory_docs_with_shared( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + if is_shared_owner(user_id) { + return self.list_shared_memory_docs(project_id).await; + } + let mut docs = self.list_memory_docs(project_id, user_id).await?; + docs.extend(self.list_shared_memory_docs(project_id).await?); + Ok(docs) + } + + async fn list_shared_memory_docs( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + let mut docs = Vec::new(); + for owner_id in shared_owner_candidates() { + docs.extend(self.list_memory_docs(project_id, owner_id).await?); + } + docs.sort_by_key(|doc| doc.id.0); + docs.dedup_by_key(|doc| doc.id); + Ok(docs) + } + + // ── Capability lease operations ───────────────────────── + + async fn save_lease(&self, lease: &CapabilityLease) -> Result<(), EngineError>; + async fn load_active_leases( + &self, + thread_id: ThreadId, + ) -> Result, EngineError>; + async fn revoke_lease(&self, lease_id: LeaseId, reason: &str) -> Result<(), EngineError>; + + // ── Mission operations ─────────────────────────────────── + + async fn save_mission(&self, mission: &Mission) -> Result<(), EngineError>; + async fn load_mission(&self, id: MissionId) -> Result, EngineError>; + async fn list_missions( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError>; + async fn update_mission_status( + &self, + id: MissionId, + status: MissionStatus, + ) -> Result<(), EngineError>; + + /// List missions visible to a user: their own + shared missions. + /// + /// Shared learning missions (self-improvement, skill-extraction, etc.) are + /// created under the shared owner id and should be visible/manageable by all + /// users through the API. + async fn list_missions_with_shared( + &self, + project_id: ProjectId, + user_id: &str, + ) -> Result, EngineError> { + if is_shared_owner(user_id) { + return self.list_shared_missions(project_id).await; + } + let mut missions = self.list_missions(project_id, user_id).await?; + missions.extend(self.list_shared_missions(project_id).await?); + Ok(missions) + } + + async fn list_shared_missions( + &self, + project_id: ProjectId, + ) -> Result, EngineError> { + let mut missions = Vec::new(); + for owner_id in shared_owner_candidates() { + missions.extend(self.list_missions(project_id, owner_id).await?); + } + missions.sort_by_key(|mission| mission.id.0); + missions.dedup_by_key(|mission| mission.id); + Ok(missions) + } + + // ── Admin operations (system-level, cross-tenant) ────────── + + /// List all threads in a project regardless of user. + /// Used by: recovery, background thread resume at startup. + async fn list_all_threads(&self, project_id: ProjectId) -> Result, EngineError> { + Err(EngineError::Store { + reason: format!("Store::list_all_threads not implemented for project '{project_id}'"), + }) + } + + /// List all missions in a project regardless of user. + /// Used by: cron ticker, event listener, bootstrap. + async fn list_all_missions(&self, project_id: ProjectId) -> Result, EngineError> { + Err(EngineError::Store { + reason: format!("Store::list_all_missions not implemented for project '{project_id}'"), + }) + } +} diff --git a/crates/ironclaw_engine/src/traits/workspace.rs b/crates/ironclaw_engine/src/traits/workspace.rs new file mode 100644 index 00000000000..a2fbccb49a5 --- /dev/null +++ b/crates/ironclaw_engine/src/traits/workspace.rs @@ -0,0 +1,20 @@ +//! Workspace document reader. +//! +//! Used by the mission runtime to load `context_paths` files into a fired +//! mission's meta-prompt. The host (main `ironclaw` crate) implements this +//! over the existing `Workspace` API. +//! +//! Kept deliberately small: just enough surface to read a single document +//! by relative path. The engine does not write to the workspace. + +use crate::types::error::EngineError; + +/// Reads workspace documents by path. Implementations must be tenant-safe: +/// the workspace they wrap is the one belonging to the mission's owner. +#[async_trait::async_trait] +pub trait WorkspaceReader: Send + Sync { + /// Read a document by relative workspace path. Returns the document body + /// as a string. Implementations should return an error rather than panic + /// when the file does not exist or cannot be decoded. + async fn read_doc(&self, path: &str) -> Result; +} diff --git a/crates/ironclaw_engine/src/types/capability.rs b/crates/ironclaw_engine/src/types/capability.rs new file mode 100644 index 00000000000..d8cde387be1 --- /dev/null +++ b/crates/ironclaw_engine/src/types/capability.rs @@ -0,0 +1,348 @@ +//! Capability — the unit of effect. +//! +//! A capability bundles actions (tools), knowledge (skills), and policies +//! (hooks) into a single installable/activatable unit. Capabilities are +//! granted to threads via leases. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use uuid::Uuid; + +use crate::types::thread::ThreadId; + +// ── Granted actions ──────────────────────────────────────── + +/// Which actions a lease grants access to. +/// +/// `All` means the lease covers every action in the capability (wildcard). +/// `Specific` restricts the lease to the listed action names. +/// +/// Serializes as a JSON array for backward compatibility: `[]` = All, +/// `["a","b"]` = Specific. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GrantedActions { + /// Wildcard — covers all actions in the capability. + All, + /// Restricted to specific action names. + Specific(Vec), +} + +impl GrantedActions { + /// Check whether a specific action is covered. + pub fn covers(&self, action_name: &str) -> bool { + match self { + GrantedActions::All => true, + GrantedActions::Specific(actions) => actions.iter().any(|a| a == action_name), + } + } + + /// Returns true if this is a wildcard grant. + pub fn is_all(&self) -> bool { + matches!(self, GrantedActions::All) + } + + /// Returns the specific actions, or an empty slice for wildcard. + pub fn actions(&self) -> &[String] { + match self { + GrantedActions::All => &[], + GrantedActions::Specific(actions) => actions, + } + } +} + +impl Serialize for GrantedActions { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + match self { + GrantedActions::All => Vec::::new().serialize(serializer), + GrantedActions::Specific(v) => v.serialize(serializer), + } + } +} + +impl<'de> Deserialize<'de> for GrantedActions { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let v = Vec::::deserialize(deserializer)?; + if v.is_empty() { + Ok(GrantedActions::All) + } else { + Ok(GrantedActions::Specific(v)) + } + } +} + +/// Strongly-typed lease identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct LeaseId(pub Uuid); + +impl LeaseId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for LeaseId { + fn default() -> Self { + Self::new() + } +} + +// ── Effect types ──────────────────────────────────────────── + +/// Classification of side effects that an action may produce. +/// Used by the policy engine for allow/deny decisions. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub enum EffectType { + /// Read from local filesystem or workspace. + ReadLocal, + /// Read from external APIs (no mutation). + ReadExternal, + /// Write to local filesystem or workspace. + WriteLocal, + /// Write to external services (create PR, send email). + WriteExternal, + /// Authenticated API call requiring credentials. + CredentialedNetwork, + /// Code execution or shell access. + Compute, + /// Financial operations (payments, transfers). + Financial, +} + +// ── Action definition ─────────────────────────────────────── + +/// Definition of a single action within a capability. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ActionDef { + /// Action name (e.g. "create_issue", "web_fetch"). + pub name: String, + /// Human-readable description. + pub description: String, + /// JSON Schema for parameters. + pub parameters_schema: serde_json::Value, + /// Effect types this action may produce. + pub effects: Vec, + /// Whether this action requires user approval before execution. + pub requires_approval: bool, +} + +// ── Capability ────────────────────────────────────────────── + +/// A capability — bundles actions, knowledge, and policies. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Capability { + /// Capability name (e.g. "github", "deployment"). + pub name: String, + /// Human-readable description. + pub description: String, + /// Executable actions (replaces tools). + pub actions: Vec, + /// Domain knowledge blocks (replaces skills). + pub knowledge: Vec, + /// Policy rules (replaces hooks). + pub policies: Vec, +} + +// ── Policy ────────────────────────────────────────────────── + +/// A named policy rule within a capability. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PolicyRule { + pub name: String, + pub condition: PolicyCondition, + pub effect: PolicyEffect, +} + +/// When a policy rule applies. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum PolicyCondition { + /// Always applies. + Always, + /// Applies when the action name exactly matches the pattern. + ActionMatches { pattern: String }, + /// Applies when the action has a specific effect type. + EffectTypeIs(EffectType), +} + +/// What the policy engine decides. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum PolicyEffect { + Allow, + Deny, + RequireApproval, +} + +// ── Capability lease ──────────────────────────────────────── + +/// A time/use-limited grant of capability access to a thread. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CapabilityLease { + pub id: LeaseId, + /// The thread this lease is granted to. + pub thread_id: ThreadId, + /// Which capability this lease covers. + pub capability_name: String, + /// Which actions from the capability are granted. + pub granted_actions: GrantedActions, + /// When the lease was granted. + pub granted_at: DateTime, + /// When the lease expires (None = no expiry). + pub expires_at: Option>, + /// Maximum number of action invocations (None = unlimited). + pub max_uses: Option, + /// Remaining invocations (None = unlimited). + pub uses_remaining: Option, + /// Whether the lease has been explicitly revoked. + pub revoked: bool, + /// Why the lease was revoked (for audit trail). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub revoked_reason: Option, +} + +impl CapabilityLease { + /// Check whether this lease is currently valid. + pub fn is_valid(&self) -> bool { + if self.revoked { + return false; + } + if let Some(expires_at) = self.expires_at + && Utc::now() >= expires_at + { + return false; + } + if let Some(remaining) = self.uses_remaining + && remaining == 0 + { + return false; + } + true + } + + /// Check whether a specific action is covered by this lease. + pub fn covers_action(&self, action_name: &str) -> bool { + self.granted_actions.covers(action_name) + } + + /// Consume one use of this lease. Returns false if no uses remain. + pub fn consume_use(&mut self) -> bool { + if let Some(ref mut remaining) = self.uses_remaining { + if *remaining == 0 { + return false; + } + *remaining -= 1; + } + true + } + + /// Refund one previously consumed use when execution was interrupted + /// before the action actually completed. + pub fn refund_use(&mut self) { + if let (Some(max_uses), Some(remaining)) = (self.max_uses, self.uses_remaining.as_mut()) + && *remaining < max_uses + { + *remaining += 1; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn make_lease() -> CapabilityLease { + CapabilityLease { + id: LeaseId::new(), + thread_id: ThreadId::new(), + capability_name: "test".into(), + granted_actions: GrantedActions::All, + granted_at: Utc::now(), + expires_at: None, + max_uses: None, + uses_remaining: None, + revoked: false, + revoked_reason: None, + } + } + + #[test] + fn valid_lease() { + let lease = make_lease(); + assert!(lease.is_valid()); + } + + #[test] + fn revoked_lease_is_invalid() { + let mut lease = make_lease(); + lease.revoked = true; + assert!(!lease.is_valid()); + } + + #[test] + fn expired_lease_is_invalid() { + let mut lease = make_lease(); + lease.expires_at = Some(Utc::now() - chrono::Duration::seconds(10)); + assert!(!lease.is_valid()); + } + + #[test] + fn exhausted_lease_is_invalid() { + let mut lease = make_lease(); + lease.max_uses = Some(1); + lease.uses_remaining = Some(0); + assert!(!lease.is_valid()); + } + + #[test] + fn consume_use_decrements() { + let mut lease = make_lease(); + lease.max_uses = Some(2); + lease.uses_remaining = Some(2); + assert!(lease.consume_use()); + assert_eq!(lease.uses_remaining, Some(1)); + assert!(lease.consume_use()); + assert_eq!(lease.uses_remaining, Some(0)); + assert!(!lease.consume_use()); + } + + #[test] + fn unlimited_consume_always_succeeds() { + let mut lease = make_lease(); + for _ in 0..100 { + assert!(lease.consume_use()); + } + } + + #[test] + fn refund_use_restores_budget_up_to_max() { + let mut lease = make_lease(); + lease.max_uses = Some(2); + lease.uses_remaining = Some(2); + assert!(lease.consume_use()); + assert_eq!(lease.uses_remaining, Some(1)); + lease.refund_use(); + assert_eq!(lease.uses_remaining, Some(2)); + lease.refund_use(); + assert_eq!(lease.uses_remaining, Some(2)); + } + + #[test] + fn covers_action_empty_grants_all() { + let lease = make_lease(); + assert!(lease.covers_action("anything")); + } + + #[test] + fn covers_action_with_specific_grants() { + let mut lease = make_lease(); + lease.granted_actions = + GrantedActions::Specific(vec!["create_issue".into(), "list_prs".into()]); + assert!(lease.covers_action("create_issue")); + assert!(lease.covers_action("list_prs")); + assert!(!lease.covers_action("delete_repo")); + } +} diff --git a/crates/ironclaw_engine/src/types/conversation.rs b/crates/ironclaw_engine/src/types/conversation.rs new file mode 100644 index 00000000000..7a16ecee170 --- /dev/null +++ b/crates/ironclaw_engine/src/types/conversation.rs @@ -0,0 +1,263 @@ +//! Conversation surface — the UI layer, separate from execution. +//! +//! A conversation is a stream of entries visible to the user. Threads +//! (the execution units) run independently and produce entries that +//! appear in conversations. One conversation can have multiple active +//! threads; one thread can outlive its originating conversation. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::thread::ThreadId; + +/// Strongly-typed conversation identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct ConversationId(pub Uuid); + +impl ConversationId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for ConversationId { + fn default() -> Self { + Self::new() + } +} + +impl std::fmt::Display for ConversationId { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.0) + } +} + +/// Strongly-typed entry identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct EntryId(pub Uuid); + +impl EntryId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for EntryId { + fn default() -> Self { + Self::new() + } +} + +/// Who sent a conversation entry. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub enum EntrySender { + /// The human user. + User, + /// The agent (from a specific thread). + Agent { thread_id: ThreadId }, + /// System notification (thread started, completed, etc.). + System, +} + +/// A single entry in a conversation — a message visible to the user. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ConversationEntry { + pub id: EntryId, + pub sender: EntrySender, + pub content: String, + /// Which thread produced this entry (if any). + pub origin_thread_id: Option, + pub timestamp: DateTime, + /// Optional metadata (channel-specific formatting, attachments, etc.). + pub metadata: serde_json::Value, +} + +impl ConversationEntry { + /// Create a user entry. + pub fn user(content: impl Into) -> Self { + Self { + id: EntryId::new(), + sender: EntrySender::User, + content: content.into(), + origin_thread_id: None, + timestamp: Utc::now(), + metadata: serde_json::Value::Null, + } + } + + /// Create an agent entry from a thread. + pub fn agent(thread_id: ThreadId, content: impl Into) -> Self { + Self { + id: EntryId::new(), + sender: EntrySender::Agent { thread_id }, + content: content.into(), + origin_thread_id: Some(thread_id), + timestamp: Utc::now(), + metadata: serde_json::Value::Null, + } + } + + /// Create a system notification entry. + pub fn system(content: impl Into) -> Self { + Self { + id: EntryId::new(), + sender: EntrySender::System, + content: content.into(), + origin_thread_id: None, + timestamp: Utc::now(), + metadata: serde_json::Value::Null, + } + } + + /// Create a system notification linked to a thread. + pub fn system_for_thread(thread_id: ThreadId, content: impl Into) -> Self { + Self { + id: EntryId::new(), + sender: EntrySender::System, + content: content.into(), + origin_thread_id: Some(thread_id), + timestamp: Utc::now(), + metadata: serde_json::Value::Null, + } + } +} + +/// A conversation surface — the UI-facing view of a chat. +/// +/// Conversations are NOT execution boundaries. They are streams of entries +/// that may come from multiple concurrent threads. A user can start a new +/// thread while another is still running, and both produce entries in the +/// same conversation. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ConversationSurface { + pub id: ConversationId, + /// Which channel this conversation is on (e.g. "telegram", "web", "cli"). + pub channel: String, + /// The user who owns this conversation. + pub user_id: String, + /// All entries in chronological order. + pub entries: Vec, + /// Currently active (non-terminal) thread IDs. + pub active_threads: Vec, + /// Metadata (channel-specific state, external thread IDs, etc.). + pub metadata: serde_json::Value, + pub created_at: DateTime, + pub updated_at: DateTime, +} + +impl ConversationSurface { + pub fn new(channel: impl Into, user_id: impl Into) -> Self { + let now = Utc::now(); + Self { + id: ConversationId::new(), + channel: channel.into(), + user_id: user_id.into(), + entries: Vec::new(), + active_threads: Vec::new(), + metadata: serde_json::Value::Null, + created_at: now, + updated_at: now, + } + } + + /// Add an entry and update the timestamp. + pub fn add_entry(&mut self, entry: ConversationEntry) { + self.entries.push(entry); + self.updated_at = Utc::now(); + } + + /// Register a thread as active in this conversation. + pub fn track_thread(&mut self, thread_id: ThreadId) { + if !self.active_threads.contains(&thread_id) { + self.active_threads.push(thread_id); + } + } + + /// Remove a thread from the active list (it completed or failed). + pub fn untrack_thread(&mut self, thread_id: ThreadId) { + self.active_threads.retain(|id| *id != thread_id); + } + + /// Get the most recent entry, if any. + pub fn last_entry(&self) -> Option<&ConversationEntry> { + self.entries.last() + } + + /// Get all entries from a specific thread. + pub fn entries_for_thread(&self, thread_id: ThreadId) -> Vec<&ConversationEntry> { + self.entries + .iter() + .filter(|e| e.origin_thread_id == Some(thread_id)) + .collect() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn conversation_lifecycle() { + let mut conv = ConversationSurface::new("telegram", "user_123"); + assert!(conv.entries.is_empty()); + assert!(conv.active_threads.is_empty()); + + // User sends a message + conv.add_entry(ConversationEntry::user("Hello!")); + assert_eq!(conv.entries.len(), 1); + + // Thread starts + let tid = ThreadId::new(); + conv.track_thread(tid); + conv.add_entry(ConversationEntry::system_for_thread(tid, "Thread started")); + assert_eq!(conv.active_threads.len(), 1); + + // Agent responds + conv.add_entry(ConversationEntry::agent(tid, "Hi there!")); + assert_eq!(conv.entries.len(), 3); + + // Thread completes + conv.untrack_thread(tid); + conv.add_entry(ConversationEntry::system_for_thread( + tid, + "Thread completed", + )); + assert!(conv.active_threads.is_empty()); + assert_eq!(conv.entries.len(), 4); + } + + #[test] + fn multiple_concurrent_threads() { + let mut conv = ConversationSurface::new("web", "user_456"); + + let t1 = ThreadId::new(); + let t2 = ThreadId::new(); + + conv.track_thread(t1); + conv.track_thread(t2); + assert_eq!(conv.active_threads.len(), 2); + + conv.add_entry(ConversationEntry::agent(t1, "Research result A")); + conv.add_entry(ConversationEntry::agent(t2, "Research result B")); + conv.add_entry(ConversationEntry::agent(t1, "More from A")); + + let t1_entries = conv.entries_for_thread(t1); + assert_eq!(t1_entries.len(), 2); + + let t2_entries = conv.entries_for_thread(t2); + assert_eq!(t2_entries.len(), 1); + + conv.untrack_thread(t1); + assert_eq!(conv.active_threads.len(), 1); + } + + #[test] + fn track_thread_is_idempotent() { + let mut conv = ConversationSurface::new("cli", "user"); + let tid = ThreadId::new(); + conv.track_thread(tid); + conv.track_thread(tid); + assert_eq!(conv.active_threads.len(), 1); + } +} diff --git a/crates/ironclaw_engine/src/types/error.rs b/crates/ironclaw_engine/src/types/error.rs new file mode 100644 index 00000000000..7ab42db496e --- /dev/null +++ b/crates/ironclaw_engine/src/types/error.rs @@ -0,0 +1,135 @@ +//! Engine error types. + +use std::fmt; + +use crate::types::capability::EffectType; +use crate::types::thread::{ThreadId, ThreadState}; + +/// Top-level engine error. +#[derive(Debug, thiserror::Error)] +pub enum EngineError { + #[error("thread error: {0}")] + Thread(#[from] ThreadError), + + #[error("step error: {0}")] + Step(#[from] StepError), + + #[error("capability error: {0}")] + Capability(#[from] CapabilityError), + + #[error("store error: {reason}")] + Store { reason: String }, + + #[error("LLM error: {reason}")] + Llm { reason: String }, + + #[error("effect execution error: {reason}")] + Effect { reason: String }, + + #[error("invalid cadence: {reason}")] + InvalidCadence { reason: String }, + + #[error("invalid state transition: {from} -> {to}")] + InvalidTransition { from: ThreadState, to: ThreadState }, + + #[error("thread not found: {0}")] + ThreadNotFound(ThreadId), + + #[error("project not found: {0}")] + ProjectNotFound(ProjectId), + + #[error("lease not found: {lease_id}")] + LeaseNotFound { lease_id: String }, + + #[error("lease expired for capability: {capability_name}")] + LeaseExpired { capability_name: String }, + + #[error("lease denied: {reason}")] + LeaseDenied { reason: String }, + + #[error("max iterations reached: {limit}")] + MaxIterations { limit: usize }, + + #[error("token limit exceeded: {used} of {limit}")] + TokenLimitExceeded { used: u64, limit: u64 }, + + #[error("consecutive error threshold exceeded: {count} errors (limit: {threshold})")] + ConsecutiveErrors { count: u32, threshold: u32 }, + + #[error("thread timeout: {elapsed:?} of {limit:?}")] + Timeout { + elapsed: std::time::Duration, + limit: std::time::Duration, + }, + + #[error("skill error: {reason}")] + Skill { reason: String }, + + #[error("access denied: user '{user_id}' cannot access {entity}")] + AccessDenied { user_id: String, entity: String }, + + #[error("gate paused: {gate_name} requires {action_name}")] + GatePaused { + gate_name: String, + action_name: String, + call_id: String, + parameters: Box, + resume_kind: Box, + resume_output: Option>, + }, +} + +use crate::types::project::ProjectId; + +/// Thread-specific errors. +#[derive(Debug, thiserror::Error)] +pub enum ThreadError { + #[error("thread already running: {0}")] + AlreadyRunning(ThreadId), + + #[error("thread is in terminal state: {0}")] + Terminal(ThreadState), + + #[error("cannot spawn child: parent thread {0} is not running")] + ParentNotRunning(ThreadId), +} + +/// Step-specific errors. +#[derive(Debug, thiserror::Error)] +pub enum StepError { + #[error("step timed out after {0:?}")] + Timeout(std::time::Duration), + + #[error("action not permitted by capability lease: {action}")] + ActionDenied { action: String }, +} + +/// Capability-specific errors. +#[derive(Debug, thiserror::Error)] +pub enum CapabilityError { + #[error("capability not found: {0}")] + NotFound(String), + + #[error("effect type {effect:?} not permitted by policy")] + EffectDenied { effect: EffectType }, +} + +// Display impls for types used in error messages that don't already impl Display. + +impl fmt::Display for ThreadId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +impl fmt::Display for ThreadState { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{self:?}") + } +} + +impl fmt::Display for ProjectId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} diff --git a/crates/ironclaw_engine/src/types/event.rs b/crates/ironclaw_engine/src/types/event.rs new file mode 100644 index 00000000000..4c707dcdd0d --- /dev/null +++ b/crates/ironclaw_engine/src/types/event.rs @@ -0,0 +1,224 @@ +//! Event sourcing types. +//! +//! Every significant action within a thread is recorded as an event. +//! This enables replay, debugging, reflection, and trace-based testing. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::capability::LeaseId; + +/// Generate a short human-readable summary of tool parameters for display. +/// +/// For `http`: shows the URL. For `web_search`: shows the query. +/// For other tools: shows the first string argument, truncated. +/// Returns `None` for empty or unrecognizable params. +pub fn summarize_params(action_name: &str, params: &serde_json::Value) -> Option { + let summary = match action_name { + "http" | "web_fetch" => params + .get("url") + .and_then(|v| v.as_str()) + .map(|u| truncate(u, 80)), + "web_search" | "llm_context" => params + .get("query") + .and_then(|v| v.as_str()) + .map(|q| truncate(q, 60)), + "memory_search" => params + .get("query") + .and_then(|v| v.as_str()) + .map(|q| truncate(q, 60)), + "memory_write" => params + .get("target") + .and_then(|v| v.as_str()) + .map(|t| t.to_string()), + "memory_read" => params + .get("path") + .and_then(|v| v.as_str()) + .map(|p| p.to_string()), + "shell" => params + .get("command") + .and_then(|v| v.as_str()) + .map(|c| truncate(c, 60)), + "message" => params + .get("content") + .and_then(|v| v.as_str()) + .map(|c| truncate(c, 40)), + _ => { + // Generic: show first string value + if let Some(obj) = params.as_object() { + obj.values() + .find_map(|v| v.as_str()) + .map(|s| truncate(s, 50)) + } else { + None + } + } + }; + summary.filter(|s| !s.is_empty()) +} + +fn truncate(s: &str, max: usize) -> String { + if s.len() <= max { + s.to_string() + } else { + // Find a safe UTF-8 boundary + let mut end = max.min(s.len()); + while end > 0 && !s.is_char_boundary(end) { + end -= 1; + } + format!("{}...", &s[..end]) // safety: end is validated by is_char_boundary loop above + } +} +use crate::types::step::{StepId, TokenUsage}; +use crate::types::thread::{ThreadId, ThreadState}; + +/// Strongly-typed event identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct EventId(pub Uuid); + +impl EventId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for EventId { + fn default() -> Self { + Self::new() + } +} + +/// A recorded event in a thread's execution history. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ThreadEvent { + pub id: EventId, + pub thread_id: ThreadId, + pub timestamp: DateTime, + pub kind: EventKind, +} + +impl ThreadEvent { + pub fn new(thread_id: ThreadId, kind: EventKind) -> Self { + Self { + id: EventId::new(), + thread_id, + timestamp: Utc::now(), + kind, + } + } +} + +/// The specific kind of event that occurred. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum EventKind { + // ── Thread lifecycle ──────────────────────────────────── + StateChanged { + from: ThreadState, + to: ThreadState, + reason: Option, + }, + + // ── Step lifecycle ────────────────────────────────────── + StepStarted { + step_id: StepId, + }, + StepCompleted { + step_id: StepId, + tokens: TokenUsage, + }, + StepFailed { + step_id: StepId, + error: String, + }, + + // ── Action execution ──────────────────────────────────── + ActionExecuted { + step_id: StepId, + action_name: String, + call_id: String, + duration_ms: u64, + /// Short human-readable summary of parameters (e.g., URL for http tool). + #[serde(default, skip_serializing_if = "Option::is_none")] + params_summary: Option, + }, + ActionFailed { + step_id: StepId, + action_name: String, + call_id: String, + error: String, + /// Short human-readable summary of parameters. + #[serde(default, skip_serializing_if = "Option::is_none")] + params_summary: Option, + }, + + // ── Capability leases ─────────────────────────────────── + LeaseGranted { + lease_id: LeaseId, + capability_name: String, + }, + LeaseRevoked { + lease_id: LeaseId, + reason: String, + }, + LeaseExpired { + lease_id: LeaseId, + }, + + // ── Messages ──────────────────────────────────────────── + MessageAdded { + role: String, + content_preview: String, + }, + + // ── Thread tree ───────────────────────────────────────── + ChildSpawned { + child_id: ThreadId, + goal: String, + }, + ChildCompleted { + child_id: ThreadId, + }, + + // ── Approval flow ─────────────────────────────────────── + ApprovalRequested { + action_name: String, + call_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + parameters: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + description: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + allow_always: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + gate_name: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + params_summary: Option, + }, + ApprovalReceived { + call_id: String, + approved: bool, + }, + + // ── Self-improvement ────────────────────────────────────── + SelfImprovementStarted, + SelfImprovementComplete { + prompt_updated: bool, + patterns_added: usize, + }, + SelfImprovementFailed { + error: String, + }, + + // ── Skill activation ─────────────────────────────────────── + SkillActivated { + skill_names: Vec, + }, + + // ── Orchestrator versioning ─────────────────────────────── + OrchestratorRollback { + from_version: u64, + to_version: u64, + reason: String, + }, +} diff --git a/crates/ironclaw_engine/src/types/memory.rs b/crates/ironclaw_engine/src/types/memory.rs new file mode 100644 index 00000000000..6d8db9be27f --- /dev/null +++ b/crates/ironclaw_engine/src/types/memory.rs @@ -0,0 +1,110 @@ +//! Memory documents — the unit of durable knowledge. +//! +//! Memory docs are structured knowledge produced by reflection on completed +//! threads. They are project-scoped and used for context building (retrieval, +//! not replay of raw history). + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::project::ProjectId; +use crate::types::thread::ThreadId; + +use super::{OwnerId, default_user_id}; + +/// Strongly-typed document identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct DocId(pub Uuid); + +impl DocId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for DocId { + fn default() -> Self { + Self::new() + } +} + +/// The kind of knowledge a memory document captures. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum DocType { + /// What a thread accomplished. + Summary, + /// Durable learning from experience. + Lesson, + /// Detected problem for follow-up. + Issue, + /// Missing capability request. + Spec, + /// Working memory / scratch notes. + Note, + /// Reusable skill with activation metadata and optional code snippets. + Skill, + /// Structured execution plan with steps, status, and progress tracking. + Plan, +} + +/// A memory document — structured durable knowledge. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct MemoryDoc { + pub id: DocId, + pub project_id: ProjectId, + /// Tenant isolation: the user who owns this document. + #[serde(default = "default_user_id")] + pub user_id: String, + pub doc_type: DocType, + pub title: String, + pub content: String, + pub source_thread_id: Option, + pub tags: Vec, + pub metadata: serde_json::Value, + pub created_at: DateTime, + pub updated_at: DateTime, +} + +impl MemoryDoc { + pub fn new( + project_id: ProjectId, + user_id: impl Into, + doc_type: DocType, + title: impl Into, + content: impl Into, + ) -> Self { + let now = Utc::now(); + Self { + id: DocId::new(), + project_id, + user_id: user_id.into(), + doc_type, + title: title.into(), + content: content.into(), + source_thread_id: None, + tags: Vec::new(), + metadata: serde_json::Value::Object(serde_json::Map::new()), + created_at: now, + updated_at: now, + } + } + + pub fn with_source_thread(mut self, thread_id: ThreadId) -> Self { + self.source_thread_id = Some(thread_id); + self + } + + pub fn with_tags(mut self, tags: Vec) -> Self { + self.tags = tags; + self + } + + pub fn owner_id(&self) -> OwnerId<'_> { + OwnerId::from_user_id(&self.user_id) + } + + pub fn is_owned_by(&self, user_id: &str) -> bool { + self.owner_id().matches_user(user_id) + } +} diff --git a/crates/ironclaw_engine/src/types/message.rs b/crates/ironclaw_engine/src/types/message.rs new file mode 100644 index 00000000000..d3e6c424806 --- /dev/null +++ b/crates/ironclaw_engine/src/types/message.rs @@ -0,0 +1,109 @@ +//! Thread messages — the engine's own message type. +//! +//! Simpler than the main crate's `ChatMessage`. Bridge adapters handle +//! conversion between `ThreadMessage` and `ChatMessage`. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; + +use crate::types::provenance::Provenance; +use crate::types::step::ActionCall; + +/// Role of a message participant. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum MessageRole { + System, + User, + Assistant, + /// Result from a capability action (replaces "Tool" role). + ActionResult, +} + +/// A message in a thread's conversation history. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ThreadMessage { + pub role: MessageRole, + pub content: String, + pub provenance: Provenance, + /// For ActionResult messages: the call ID this is responding to. + pub action_call_id: Option, + /// For ActionResult messages: the action name. + pub action_name: Option, + /// For Assistant messages: actions the LLM wants to execute. + pub action_calls: Option>, + pub timestamp: DateTime, +} + +impl ThreadMessage { + /// Create a system message. + pub fn system(content: impl Into) -> Self { + Self { + role: MessageRole::System, + content: content.into(), + provenance: Provenance::System, + action_call_id: None, + action_name: None, + action_calls: None, + timestamp: Utc::now(), + } + } + + /// Create a user message. + pub fn user(content: impl Into) -> Self { + Self { + role: MessageRole::User, + content: content.into(), + provenance: Provenance::User, + action_call_id: None, + action_name: None, + action_calls: None, + timestamp: Utc::now(), + } + } + + /// Create an assistant text message. + pub fn assistant(content: impl Into) -> Self { + Self { + role: MessageRole::Assistant, + content: content.into(), + provenance: Provenance::LlmGenerated, + action_call_id: None, + action_name: None, + action_calls: None, + timestamp: Utc::now(), + } + } + + /// Create an assistant message with action calls. + pub fn assistant_with_actions(content: Option, calls: Vec) -> Self { + Self { + role: MessageRole::Assistant, + content: content.unwrap_or_default(), + provenance: Provenance::LlmGenerated, + action_call_id: None, + action_name: None, + action_calls: Some(calls), + timestamp: Utc::now(), + } + } + + /// Create an action result message. + pub fn action_result( + call_id: impl Into, + action_name: impl Into, + content: impl Into, + ) -> Self { + let name: String = action_name.into(); + Self { + role: MessageRole::ActionResult, + content: content.into(), + provenance: Provenance::ToolOutput { + action_name: name.clone(), + }, + action_call_id: Some(call_id.into()), + action_name: Some(name), + action_calls: None, + timestamp: Utc::now(), + } + } +} diff --git a/crates/ironclaw_engine/src/types/mission.rs b/crates/ironclaw_engine/src/types/mission.rs new file mode 100644 index 00000000000..19edf9de529 --- /dev/null +++ b/crates/ironclaw_engine/src/types/mission.rs @@ -0,0 +1,665 @@ +//! Missions — long-running goals that spawn threads over time. +//! +//! A mission represents an ongoing objective that periodically spawns +//! threads to make progress. Missions can run on a schedule (cron), +//! in response to events, or be triggered manually. + +use std::collections::HashMap; +use std::str::FromStr; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::error::EngineError; +use crate::types::project::ProjectId; +use crate::types::thread::ThreadId; + +use super::{OwnerId, default_user_id}; + +pub use ironclaw_common::ValidTimezone; + +/// Strongly-typed mission identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct MissionId(pub Uuid); + +impl MissionId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for MissionId { + fn default() -> Self { + Self::new() + } +} + +impl std::fmt::Display for MissionId { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.0) + } +} + +/// Lifecycle status of a mission. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum MissionStatus { + /// Mission is actively spawning threads on cadence. + Active, + /// Mission is paused — no new threads will be spawned. + Paused, + /// Mission has achieved its goal. + Completed, + /// Mission has been abandoned or failed irrecoverably. + Failed, +} + +/// How a mission triggers new threads. +/// +/// The engine defines the trigger *types*. The bridge/host implements the +/// actual trigger infrastructure (cron tickers, webhook endpoints, event +/// matchers). The engine just needs to be told "fire this mission now." +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum MissionCadence { + /// Spawn on a cron schedule (e.g., "0 */6 * * *" for every 6 hours). + Cron { + expression: String, + #[serde( + default, + deserialize_with = "ironclaw_common::deserialize_option_lenient" + )] + timezone: Option, + }, + /// Spawn in response to a channel message matching a regex pattern. + /// `channel`, when set, restricts firing to messages from a specific + /// channel name (case-insensitive). + OnEvent { + event_pattern: String, + #[serde(default)] + channel: Option, + }, + /// Spawn in response to a structured system event (from tools or external). + /// `filters`, when non-empty, requires every key/value pair to match + /// against the event payload's top-level fields exactly. + OnSystemEvent { + source: String, + event_type: String, + #[serde(default)] + filters: HashMap, + }, + /// Spawn when an external webhook is received at a registered path. + /// The bridge registers the webhook endpoint and routes payloads here. + Webhook { + path: String, + secret: Option, + }, + /// Only spawn when manually triggered (via mission_fire tool or API). + Manual, +} + +/// A mission — a long-running goal that spawns threads over time. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Mission { + pub id: MissionId, + pub project_id: ProjectId, + /// Tenant isolation: the user who owns this mission. + #[serde(default = "default_user_id")] + pub user_id: String, + pub name: String, + /// Optional human-readable description (separate from the goal statement). + /// Routine `description` fields map here. + #[serde(default)] + pub description: Option, + pub goal: String, + pub status: MissionStatus, + pub cadence: MissionCadence, + + // ── Evolving strategy ── + /// What the next thread should focus on (updated after each thread). + pub current_focus: Option, + /// What approaches have been tried and what happened. + pub approach_history: Vec, + + // ── Progress tracking ── + /// History of threads spawned by this mission. + pub thread_history: Vec, + /// Optional criteria for declaring the mission complete. + pub success_criteria: Option, + + // ── Notification ── + /// Channels to notify when a mission thread completes (e.g. "gateway", "repl"). + /// Empty means no proactive notification (results only in approach_history). + #[serde(default)] + pub notify_channels: Vec, + /// Optional per-channel user/recipient target for notifications. Maps from + /// routine `delivery.user`. When `None`, the channel's last-seen + /// recipient is used. + #[serde(default)] + pub notify_user: Option, + + // ── Context preloading ── + /// Workspace paths whose contents are loaded into the thread's meta-prompt + /// when the mission fires (e.g. `["MEMORY.md", "context/profile.json"]`). + /// Maps from routine `execution.context_paths`. + #[serde(default)] + pub context_paths: Vec, + + // ── Budget / guardrails ── + /// Maximum threads per day (0 = unlimited). + pub max_threads_per_day: u32, + /// Threads spawned today (reset daily by the cron ticker). + pub threads_today: u32, + /// Cooldown between firings, in seconds. 0 = no cooldown. Maps from + /// routine `guardrails.cooldown_secs`. + #[serde(default)] + pub cooldown_secs: u64, + /// Maximum number of mission threads that may be running concurrently + /// (in non-terminal states). 0 = unlimited. Maps from routine + /// `guardrails.max_concurrent`. + #[serde(default)] + pub max_concurrent: u32, + /// Deduplication window for event-triggered firings, in seconds. 0 = no + /// dedup. When set, identical event-key payloads within this window are + /// suppressed. Maps from routine `guardrails.dedup_window`. + #[serde(default)] + pub dedup_window_secs: u64, + /// Timestamp of the most recent successful fire. Used by cooldown + /// enforcement. + #[serde(default)] + pub last_fire_at: Option>, + + // ── Trigger payload ── + /// Payload from the most recent trigger (webhook body, event data, etc.). + /// Injected into the thread's context so the code can access it. + pub last_trigger_payload: Option, + + pub metadata: serde_json::Value, + pub created_at: DateTime, + pub updated_at: DateTime, + /// When the next thread should be spawned (for Cron cadence). + pub next_fire_at: Option>, +} + +impl Mission { + pub fn new( + project_id: ProjectId, + user_id: impl Into, + name: impl Into, + goal: impl Into, + cadence: MissionCadence, + ) -> Self { + let now = Utc::now(); + + // Event-triggered cadences (OnEvent / OnSystemEvent / Webhook) are + // *reactive*: a single noisy channel can fire them on every + // matching message. Cron / Manual cadences are *proactive* and + // self-paced. Set tighter defaults for the reactive variants so a + // mission created without explicit guardrails cannot accidentally + // flood the LLM if its pattern is too loose. The routine alias + // path overrides these via post-create update when the LLM + // supplies explicit guardrails / advanced settings. + let is_reactive = matches!( + cadence, + MissionCadence::OnEvent { .. } + | MissionCadence::OnSystemEvent { .. } + | MissionCadence::Webhook { .. } + ); + let (default_max_threads_per_day, default_cooldown_secs, default_max_concurrent) = + if is_reactive { + // 5-minute cooldown + 24/day cap + single-instance — same + // floor v1 routine_create used for event-driven routines. + (24, 300, 1) + } else { + // Existing defaults for cron/manual missions; no cooldown, + // no concurrency cap, generous daily budget. + (10, 0, 0) + }; + + Self { + id: MissionId::new(), + project_id, + user_id: user_id.into(), + name: name.into(), + description: None, + goal: goal.into(), + status: MissionStatus::Active, + cadence, + current_focus: None, + approach_history: Vec::new(), + thread_history: Vec::new(), + success_criteria: None, + notify_channels: Vec::new(), + notify_user: None, + context_paths: Vec::new(), + max_threads_per_day: default_max_threads_per_day, + threads_today: 0, + cooldown_secs: default_cooldown_secs, + max_concurrent: default_max_concurrent, + dedup_window_secs: 0, + last_fire_at: None, + last_trigger_payload: None, + metadata: serde_json::Value::Object(serde_json::Map::new()), + created_at: now, + updated_at: now, + next_fire_at: None, + } + } + + pub fn with_success_criteria(mut self, criteria: impl Into) -> Self { + self.success_criteria = Some(criteria.into()); + self + } + + pub fn owner_id(&self) -> OwnerId<'_> { + OwnerId::from_user_id(&self.user_id) + } + + pub fn is_owned_by(&self, user_id: &str) -> bool { + self.owner_id().matches_user(user_id) + } + + /// Record that a thread was spawned for this mission. + pub fn record_thread(&mut self, thread_id: ThreadId) { + self.thread_history.push(thread_id); + self.updated_at = Utc::now(); + } + + /// Whether the mission is in a terminal state. + pub fn is_terminal(&self) -> bool { + matches!( + self.status, + MissionStatus::Completed | MissionStatus::Failed + ) + } +} + +/// Normalize a cron expression to the 7-field format expected by the `cron` crate. +/// +/// Field formats accepted: +/// - **5-field** (standard Vixie cron): `min hr dom mon dow` — prepend `0` +/// (seconds) and append `*` (year). +/// - **6-field**: assumed to be `sec min hr dom mon dow` (the `cron` crate's +/// native format minus year) and append `*` (year). **Note:** this is *not* +/// the Quartz `min hr dom mon dow year` interpretation. A user passing +/// `"0 9 * * * 2027"` intending "at 09:00 every day in 2027" will instead +/// get "at second 0 of minute 9 of every hour, every day, every year". Use +/// the explicit 7-field form `0 0 9 * * * 2027` to disambiguate. +/// - **7-field**: `sec min hr dom mon dow year` — passed through unchanged. +/// +/// Returns an error for any other field count rather than passing the input +/// through to `cron::Schedule::from_str`, which would surface a confusing +/// low-level parse error. +fn normalize_cron_expression(expression: &str) -> Result { + let trimmed = expression.trim(); + let fields: Vec<&str> = trimmed.split_whitespace().collect(); + match fields.len() { + 5 => Ok(format!("0 {} *", fields.join(" "))), + 6 => { + // Disambiguate the Quartz-style 6-field form. A user (or LLM) + // typing `"0 9 * * * 2027"` almost certainly means + // "at 09:00 every day in 2027" (Quartz: `min hr dom mon dow year`), + // not "at second 0 of minute 9, every hour, every day, dow=2027". + // The cron crate would treat the year-shaped final field as a + // (nonsensical) day-of-week and silently produce a wrong schedule. + // Reject early with a message that points at the explicit + // 7-field form so the caller can fix it instead of debugging a + // schedule that never fires. + if let Some(last) = fields.last() + && is_year_field(last) + { + return Err(EngineError::InvalidCadence { + reason: format!( + "ambiguous 6-field cron expression '{expression}': the trailing '{last}' \ + looks like a year. The 6-field form is `sec min hr dom mon dow`, NOT the \ + Quartz `min hr dom mon dow year`. Use the explicit 7-field form \ + `0 {} {} {} {} {} {last}` to mean 'at the given time in {last}'.", + fields[0], fields[1], fields[2], fields[3], fields[4] + ), + }); + } + Ok(format!("{} *", fields.join(" "))) + } + 7 => Ok(trimmed.to_string()), + n => Err(EngineError::InvalidCadence { + reason: format!( + "invalid cron expression '{expression}': expected 5, 6, or 7 fields, got {n}" + ), + }), + } +} + +/// True if `field` is a literal 4-digit year in the plausible cron range. +/// +/// Used to detect the Quartz-style `min hr dom mon dow year` mistake in +/// 6-field input. Range chosen to cover the cron crate's accepted year span +/// without firing on field values that happen to be 4 digits but mean +/// something else (none of the standard cron field ranges produce 4-digit +/// literals). +fn is_year_field(field: &str) -> bool { + field.len() == 4 + && field.bytes().all(|b| b.is_ascii_digit()) + && field + .parse::() + .is_ok_and(|y| (1970..=2099).contains(&y)) +} + +/// Parse a cron expression and compute the next fire time from now. +/// +/// Accepts standard 5-field, 6-field, or 7-field cron expressions (auto-normalized). +/// When a [`ValidTimezone`] is provided, the schedule is evaluated in that +/// timezone and the result is converted back to UTC. Otherwise UTC is used. +/// +/// Cron parse failures return [`EngineError::InvalidCadence`] (validation, not +/// storage), so callers can map them to user-facing errors. +pub fn next_cron_fire( + expression: &str, + timezone: Option<&ValidTimezone>, +) -> Result>, EngineError> { + let normalized = normalize_cron_expression(expression)?; + let schedule = + cron::Schedule::from_str(&normalized).map_err(|e| EngineError::InvalidCadence { + reason: format!("invalid cron expression '{expression}': {e}"), + })?; + if let Some(vtz) = timezone { + Ok(schedule + .upcoming(vtz.tz()) + .next() + .map(|dt| dt.with_timezone(&Utc))) + } else { + Ok(schedule.upcoming(Utc).next()) + } +} + +/// Like [`next_cron_fire`], but treats `Ok(None)` as a validation error. +/// +/// `next_cron_fire` returns `Ok(None)` for cron expressions that are +/// syntactically valid but will never fire again (e.g. `0 0 9 * * * 2020` — +/// year-locked to a year that's already passed). At lifecycle entry points +/// (`create_mission`, cadence updates, `resume_mission`) this is the same +/// failure mode as the original #1944 bug: an Active mission with +/// `next_fire_at = None` that the ticker can never pick up. Surface it as +/// `InvalidCadence` so callers fail fast and the operator gets a clear error. +/// +/// `fire_mission` and `bootstrap_project` intentionally tolerate `Ok(None)` +/// (logged) and should keep using `next_cron_fire` directly — the thread is +/// already running or the data is already persisted, and aborting would do +/// more harm than logging. +pub fn next_cron_fire_required( + expression: &str, + timezone: Option<&ValidTimezone>, +) -> Result, EngineError> { + next_cron_fire(expression, timezone)?.ok_or_else(|| EngineError::InvalidCadence { + reason: format!( + "cron expression '{expression}' has no upcoming fire time (year-locked or otherwise unschedulable)" + ), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use chrono::{Datelike, Timelike}; + + #[test] + fn rejects_four_field_cron() { + // Four-field input is not a recognized cron format. Surface a clear + // error rather than passing through to a low-level parse failure. + let err = next_cron_fire("* * * *", None).unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("expected 5, 6, or 7 fields"), "got: {msg}"); + } + + #[test] + fn accepts_five_field_cron() { + let next = next_cron_fire("0 9 * * *", None).unwrap(); + assert!(next.is_some(), "5-field cron should produce a fire time"); + } + + #[test] + fn next_cron_fire_respects_timezone() { + // "0 9 * * *" in America/New_York should produce a UTC instant whose + // wall-clock time in NY is 09:00 on some date — and the resulting UTC + // hour should differ from a UTC-evaluated schedule (since NY is offset + // from UTC year-round). + let tz = ValidTimezone::parse("America/New_York").unwrap(); + let in_ny = next_cron_fire("0 9 * * *", Some(&tz)) + .unwrap() + .expect("schedule should produce a fire time"); + let in_utc = next_cron_fire("0 9 * * *", None) + .unwrap() + .expect("schedule should produce a fire time"); + + // NY 09:00 in UTC is either 13:00 (EDT) or 14:00 (EST). UTC 09:00 is 09:00. + let ny_utc_hour = in_ny.hour(); + assert!( + ny_utc_hour == 13 || ny_utc_hour == 14, + "NY 09:00 should map to UTC 13 or 14, got {ny_utc_hour}" + ); + assert_eq!(in_utc.hour(), 9, "UTC schedule should fire at hour 9"); + assert_ne!( + in_ny.hour(), + in_utc.hour(), + "tz-aware and tz-naive schedules should differ" + ); + + // Sanity: result is a real future date, not the epoch. Compare + // against `Utc::now()` so the assertion stays stable across calendar + // years rather than being pinned to a hard-coded threshold. + assert!(in_ny > Utc::now(), "next cron fire must be in the future"); + } + + #[test] + fn normalize_six_field_cron() { + // 6-field (with seconds) should be accepted. + let next = next_cron_fire("0 0 9 * * *", None).unwrap(); + assert!(next.is_some()); + } + + #[test] + fn six_field_cron_is_sec_min_hr_dom_mon_dow_not_quartz_with_year() { + // Pin the 6-field interpretation: `sec min hr dom mon dow`, NOT the + // Quartz-style `min hr dom mon dow year`. A 6-field input gets `*` + // appended for the year position. This test exists so a future change + // doesn't silently flip the interpretation and break existing + // missions. + let normalized = normalize_cron_expression("0 0 9 * * *").unwrap(); + assert_eq!( + normalized, "0 0 9 * * * *", + "6-field input must be treated as `sec min hr dom mon dow` and appended with `*` (year)" + ); + + // Sanity: the resulting schedule fires at 09:00:00 wall-clock daily. + let next = next_cron_fire("0 0 9 * * *", None).unwrap().unwrap(); + assert_eq!(next.hour(), 9); + assert_eq!(next.minute(), 0); + assert_eq!(next.second(), 0); + } + + #[test] + fn normalize_seven_field_cron() { + // 7-field (sec min hr dom mon dow year) should pass through. + let next = next_cron_fire("0 0 9 * * * 2027", None).unwrap(); + assert!(next.is_some()); + } + + #[test] + fn six_field_cron_with_year_shaped_last_field_is_rejected() { + // A user (or LLM) typing the Quartz-style `min hr dom mon dow year` + // form gets a clear error pointing at the explicit 7-field form, + // rather than a silently misparsed schedule. The 6-field form is + // `sec min hr dom mon dow`, so `2027` would otherwise be interpreted + // as a (nonsensical) day-of-week. + let err = next_cron_fire("0 9 * * * 2027", None).unwrap_err(); + let msg = err.to_string(); + assert!( + matches!(err, EngineError::InvalidCadence { .. }), + "expected InvalidCadence, got: {err:?}" + ); + assert!( + msg.contains("looks like a year") && msg.contains("0 0 9 * * * 2027"), + "error should explain Quartz ambiguity and suggest 7-field form, got: {msg}" + ); + + // Cover all year boundaries. + for year in ["1970", "1999", "2000", "2026", "2099"] { + let expr = format!("0 0 * * * {year}"); + assert!( + matches!( + next_cron_fire(&expr, None), + Err(EngineError::InvalidCadence { .. }) + ), + "year {year} should be rejected as Quartz-style ambiguity" + ); + } + + // Out-of-range 4-digit values are NOT treated as years and fall + // through to the regular 6-field interpretation (which the cron + // crate may then reject for its own reasons). + let normalized = normalize_cron_expression("0 0 9 * * 1969"); + assert!( + normalized.is_ok(), + "1969 (out of year range) should not trigger the Quartz heuristic" + ); + + // 5-field cron with a literal day-of-week numeric value must still + // work — the year heuristic only applies to 6-field input. + assert!(next_cron_fire("0 9 * * 3", None).unwrap().is_some()); + } + + #[test] + fn invalid_cron_returns_invalid_cadence_error() { + // Cron parse errors are validation errors, not store errors. + let err = next_cron_fire("not a cron", None).unwrap_err(); + assert!( + matches!(err, EngineError::InvalidCadence { .. }), + "expected InvalidCadence, got: {err:?}" + ); + + let err = next_cron_fire("nope nope nope nope nope", None).unwrap_err(); + assert!(matches!(err, EngineError::InvalidCadence { .. })); + } + + // ── DST tests (#1944) ───────────────────────────────────── + // + // The whole point of carrying user_timezone through the engine is so that + // cron schedules respect DST. These tests pin the cron crate's behavior on + // the two tricky transitions in `America/New_York`: + // + // * Spring-forward: 2027-03-14 02:00 jumps to 03:00. Local times in + // [02:00, 03:00) do not exist on that day. + // * Fall-back: 2027-11-07 02:00 jumps back to 01:00. Local times in + // [01:00, 02:00) occur twice (once EDT, once EST). + // + // We don't test specific calendar dates (those would rot); instead we use + // explicit reference instants via the `cron` crate's `after()` method to + // assert behavior in a year-independent way. + + use chrono::TimeZone; + + fn schedule_after( + expression: &str, + tz: &ValidTimezone, + after_utc: DateTime, + ) -> DateTime { + let normalized = normalize_cron_expression(expression).unwrap(); // safety: test helper + let schedule = cron::Schedule::from_str(&normalized).unwrap(); // safety: test helper + let after_local = after_utc.with_timezone(&tz.tz()); + schedule + .after(&after_local) + .next() + .expect("schedule should produce a fire time") // safety: test helper + .with_timezone(&Utc) + } + + #[test] + fn dst_spring_forward_skips_missing_local_hour() { + // 2027-03-14 in America/New_York: clocks jump 02:00 -> 03:00 EDT. + // A cron at "30 2 * * *" requests a wall-clock time that does not + // exist on that day. The cron crate skips that occurrence and fires + // on the next valid day at 02:30 (which is then EDT, UTC-4). + let tz = ValidTimezone::parse("America/New_York").unwrap(); + + // Reference: 2027-03-13 00:00 UTC = 2027-03-12 19:00 EST, well + // before the spring-forward day. We just need a stable anchor. + let after = Utc.with_ymd_and_hms(2027, 3, 13, 0, 0, 0).unwrap(); + let fire = schedule_after("30 2 * * *", &tz, after); + + // The first fire on 2027-03-13 is 02:30 EST = 07:30 UTC. The next + // fire would be 2027-03-14 02:30 — but that doesn't exist on DST + // day, so the schedule skips to 2027-03-15 02:30 EDT = 06:30 UTC. + // Whichever the cron crate picks, it must NOT land in the missing + // local interval [02:00, 03:00) on 2027-03-14. + let fire_local = fire.with_timezone(&tz.tz()); + if fire_local.year() == 2027 && fire_local.month() == 3 && fire_local.day() == 14 { + // If it lands on DST day, the wall-clock hour must be >= 3 (EDT). + assert!( + fire_local.hour() >= 3, + "fire on DST day must not be in skipped [02:00, 03:00) window, got {fire_local}" + ); + } + // Sanity: the result is a real future instant. + assert!(fire > after); + } + + #[test] + fn dst_fall_back_picks_one_of_overlapping_hours() { + // 2027-11-07 in America/New_York: clocks jump 02:00 EDT -> 01:00 EST. + // Local times in [01:00, 02:00) occur twice. A cron at "30 1 * * *" + // could fire at 01:30 EDT (05:30 UTC) or 01:30 EST (06:30 UTC). + // The cron crate picks one consistently — we just assert it picks + // exactly one and that the result is correct in UTC. + let tz = ValidTimezone::parse("America/New_York").unwrap(); + let after = Utc.with_ymd_and_hms(2027, 11, 6, 12, 0, 0).unwrap(); + let fire = schedule_after("30 1 * * *", &tz, after); + + let fire_local = fire.with_timezone(&tz.tz()); + // Whatever date the cron crate lands on, the local time must be 01:30. + assert_eq!( + fire_local.hour(), + 1, + "expected hour 1 local, got {fire_local}" + ); + assert_eq!( + fire_local.minute(), + 30, + "expected minute 30 local, got {fire_local}" + ); + + // And the UTC instant must be exactly one of the two valid 01:30 NY + // instants on the fall-back day, OR a 01:30 NY on a neighbouring day. + // Either way, converting back must round-trip to the same wall clock. + let round_trip = fire.with_timezone(&tz.tz()); + assert_eq!(round_trip, fire_local); + } + + #[test] + fn dst_aware_schedule_advances_correctly_across_transition() { + // Across a DST transition the absolute UTC interval between two + // consecutive 09:00 local fires shifts by an hour. This is the + // "load-bearing tz" property the PR exists to enable. + let tz = ValidTimezone::parse("America/New_York").unwrap(); + // Pick an anchor in EST (winter, before spring-forward). + let anchor = Utc.with_ymd_and_hms(2027, 3, 1, 0, 0, 0).unwrap(); + let normalized = normalize_cron_expression("0 9 * * *").unwrap(); + let schedule = cron::Schedule::from_str(&normalized).unwrap(); + let anchor_local = anchor.with_timezone(&tz.tz()); + + // Take 30 consecutive fires — long enough to cross spring-forward. + let fires: Vec<_> = schedule.after(&anchor_local).take(30).collect(); + assert_eq!(fires.len(), 30); + + // All fires must be at 09:00 local wall clock, regardless of DST. + for f in &fires { + assert_eq!(f.hour(), 9, "every fire must be 09:00 local, got {f}"); + } + + // The UTC hour shifts when crossing DST: 09:00 EST = 14:00 UTC, + // 09:00 EDT = 13:00 UTC. Both must appear across the 30-day window. + let utc_hours: std::collections::BTreeSet = + fires.iter().map(|f| f.with_timezone(&Utc).hour()).collect(); + assert!( + utc_hours.contains(&13) && utc_hours.contains(&14), + "30-day window straddling spring-forward should contain both 13:00 and 14:00 UTC fires; got {utc_hours:?}" + ); + } +} diff --git a/crates/ironclaw_engine/src/types/mod.rs b/crates/ironclaw_engine/src/types/mod.rs new file mode 100644 index 00000000000..c7d94e95b95 --- /dev/null +++ b/crates/ironclaw_engine/src/types/mod.rs @@ -0,0 +1,82 @@ +//! Core type definitions for the engine. +//! +//! All data structures live here. No async, no I/O — just types and +//! validation logic. + +use std::borrow::Cow; + +pub mod capability; +pub mod conversation; +pub mod error; +pub mod event; +pub mod memory; +pub mod message; +pub mod mission; +pub mod project; +pub mod provenance; +pub mod step; +pub mod thread; + +pub const LEGACY_SHARED_OWNER_ID: &str = "system"; +pub const SHARED_OWNER_ID: &str = "__shared__"; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum OwnerId<'a> { + Shared, + User(Cow<'a, str>), +} + +/// Default user_id for backwards-compatible deserialization of records +/// created before multi-tenant isolation was added. +pub(crate) fn default_user_id() -> String { + "legacy".to_string() +} + +pub fn shared_owner_id() -> &'static str { + SHARED_OWNER_ID +} + +pub fn is_shared_owner(user_id: &str) -> bool { + user_id == SHARED_OWNER_ID || user_id == LEGACY_SHARED_OWNER_ID +} + +impl<'a> OwnerId<'a> { + pub fn from_user_id(user_id: &'a str) -> Self { + if is_shared_owner(user_id) { + Self::Shared + } else { + Self::User(Cow::Borrowed(user_id)) + } + } + + pub fn is_shared(&self) -> bool { + matches!(self, Self::Shared) + } + + pub fn matches_user(&self, user_id: &str) -> bool { + matches!(self, Self::User(owner) if owner == user_id) + } + + pub fn as_user_id(&self) -> &str { + match self { + Self::Shared => shared_owner_id(), + Self::User(user_id) => user_id.as_ref(), + } + } +} + +pub fn shared_owner_candidates() -> [&'static str; 2] { + [SHARED_OWNER_ID, LEGACY_SHARED_OWNER_ID] +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn owner_id_maps_shared_and_user_values() { + assert!(OwnerId::from_user_id(SHARED_OWNER_ID).is_shared()); + assert!(OwnerId::from_user_id(LEGACY_SHARED_OWNER_ID).is_shared()); + assert!(OwnerId::from_user_id("alice").matches_user("alice")); + } +} diff --git a/crates/ironclaw_engine/src/types/project.rs b/crates/ironclaw_engine/src/types/project.rs new file mode 100644 index 00000000000..0ea95de6254 --- /dev/null +++ b/crates/ironclaw_engine/src/types/project.rs @@ -0,0 +1,67 @@ +//! Project — the unit of context. +//! +//! A project is a persistent domain of work that scopes memory documents, +//! threads, and missions. Examples: "IronClaw architecture", "deployment system". + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use super::{OwnerId, default_user_id}; + +/// Strongly-typed project identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct ProjectId(pub Uuid); + +impl ProjectId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for ProjectId { + fn default() -> Self { + Self::new() + } +} + +/// A project — the unit of context scoping. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Project { + pub id: ProjectId, + /// Tenant isolation: the user who owns this project. + #[serde(default = "default_user_id")] + pub user_id: String, + pub name: String, + pub description: String, + pub metadata: serde_json::Value, + pub created_at: DateTime, + pub updated_at: DateTime, +} + +impl Project { + pub fn new( + user_id: impl Into, + name: impl Into, + description: impl Into, + ) -> Self { + let now = Utc::now(); + Self { + id: ProjectId::new(), + user_id: user_id.into(), + name: name.into(), + description: description.into(), + metadata: serde_json::Value::Object(serde_json::Map::new()), + created_at: now, + updated_at: now, + } + } + + pub fn owner_id(&self) -> OwnerId<'_> { + OwnerId::from_user_id(&self.user_id) + } + + pub fn is_owned_by(&self, user_id: &str) -> bool { + self.owner_id().matches_user(user_id) + } +} diff --git a/crates/ironclaw_engine/src/types/provenance.rs b/crates/ironclaw_engine/src/types/provenance.rs new file mode 100644 index 00000000000..9b3fb8fa7cf --- /dev/null +++ b/crates/ironclaw_engine/src/types/provenance.rs @@ -0,0 +1,25 @@ +//! Provenance tracking for data flow analysis. +//! +//! Every data value can be tagged with its origin. The policy engine uses +//! provenance at effect boundaries to enforce taint-based security rules. +//! Phase 1: types only; enforcement comes in Phase 4. + +use serde::{Deserialize, Serialize}; + +use crate::types::memory::DocId; + +/// The origin of a piece of data. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub enum Provenance { + /// Direct user input. + User, + /// System prompt, configuration. + #[default] + System, + /// Result from a capability action. + ToolOutput { action_name: String }, + /// Generated by the LLM. + LlmGenerated, + /// Retrieved from project memory. + MemoryRetrieval { doc_id: DocId }, +} diff --git a/crates/ironclaw_engine/src/types/step.rs b/crates/ironclaw_engine/src/types/step.rs new file mode 100644 index 00000000000..f09bcb0ed43 --- /dev/null +++ b/crates/ironclaw_engine/src/types/step.rs @@ -0,0 +1,166 @@ +//! Step — the unit of execution within a thread. +//! +//! Each step corresponds to one LLM call plus its subsequent action +//! executions. This replaces the implicit "iteration" counter in the +//! existing `run_agentic_loop`. + +use std::time::Duration; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::thread::ThreadId; + +/// Strongly-typed step identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct StepId(pub Uuid); + +impl StepId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for StepId { + fn default() -> Self { + Self::new() + } +} + +/// Status of a step within its lifecycle. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum StepStatus { + Pending, + LlmCalling, + Executing, + Completed, + Failed, +} + +/// Which execution tier handles the step's code/actions. +/// +/// Monty is the sole CodeAct/RLM executor. WASM and Docker are used for +/// third-party tool isolation and thread sandboxing (Phase 8), not for +/// running LLM-generated Python. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum ExecutionTier { + /// Structured tool calls (JSON action calls from LLM). + Structured, + /// Embedded Python via Monty (CodeAct/RLM pattern). + Scripting, +} + +/// A single execution step within a thread. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Step { + pub id: StepId, + pub thread_id: ThreadId, + /// 1-indexed sequence within the thread. + pub sequence: usize, + pub status: StepStatus, + pub tier: ExecutionTier, + pub llm_response: Option, + pub action_results: Vec, + pub tokens_used: TokenUsage, + pub started_at: DateTime, + pub completed_at: Option>, +} + +impl Step { + pub fn new(thread_id: ThreadId, sequence: usize) -> Self { + Self { + id: StepId::new(), + thread_id, + sequence, + status: StepStatus::Pending, + tier: ExecutionTier::Structured, + llm_response: None, + action_results: Vec::new(), + tokens_used: TokenUsage::default(), + started_at: Utc::now(), + completed_at: None, + } + } +} + +// ── LLM response types ───────────────────────────────────── + +/// Response from the LLM: text, action calls, or executable code. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum LlmResponse { + /// Final text response. + Text(String), + /// One or more action calls (with optional reasoning text). + ActionCalls { + calls: Vec, + content: Option, + }, + /// Executable Python code (CodeAct). Tool calls happen as function + /// calls within the code; the runtime suspends at each one and + /// delegates to the EffectExecutor. + Code { + code: String, + content: Option, + }, +} + +/// A request from the LLM to execute a capability action. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ActionCall { + /// Unique call identifier (echoed in the result). + pub id: String, + /// Action name (e.g. "web_fetch", "create_issue"). + pub action_name: String, + /// Action parameters as JSON. + pub parameters: serde_json::Value, +} + +/// Result of executing a capability action. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ActionResult { + /// The call ID this result corresponds to. + pub call_id: String, + /// The action that was executed. + pub action_name: String, + /// Output value. + pub output: serde_json::Value, + /// Whether this result represents an error. + pub is_error: bool, + /// How long the action took. + #[serde(with = "duration_millis")] + pub duration: Duration, +} + +/// Token usage for a single LLM call. +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize)] +pub struct TokenUsage { + pub input_tokens: u64, + pub output_tokens: u64, + pub cache_read_tokens: u64, + pub cache_write_tokens: u64, + /// USD cost for this call (populated by LlmBackend if cost data is available). + pub cost_usd: f64, +} + +impl TokenUsage { + pub fn total(&self) -> u64 { + self.input_tokens + self.output_tokens + } +} + +/// Serde helper for Duration as milliseconds. +mod duration_millis { + use std::time::Duration; + + use serde::{Deserialize, Deserializer, Serializer}; + + pub fn serialize(d: &Duration, s: S) -> Result { + s.serialize_u64(d.as_millis() as u64) + } + + pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result { + let millis = u64::deserialize(d)?; + Ok(Duration::from_millis(millis)) + } +} diff --git a/crates/ironclaw_engine/src/types/thread.rs b/crates/ironclaw_engine/src/types/thread.rs new file mode 100644 index 00000000000..c2390d027e2 --- /dev/null +++ b/crates/ironclaw_engine/src/types/thread.rs @@ -0,0 +1,539 @@ +//! Thread — the unit of work. +//! +//! A thread is a bounded task or investigation. It unifies the concepts of +//! Session (interactive conversation), Job (background work), Routine +//! (scheduled execution), and Sub-agent (delegated reasoning) into a single +//! abstraction with a shared state machine. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::types::capability::LeaseId; +use crate::types::error::EngineError; +use crate::types::event::{EventKind, ThreadEvent}; +use crate::types::memory::DocId; +use crate::types::message::ThreadMessage; +use crate::types::project::ProjectId; + +use super::{OwnerId, default_user_id}; + +/// Strongly-typed thread identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct ThreadId(pub Uuid); + +impl ThreadId { + pub fn new() -> Self { + Self(Uuid::new_v4()) + } +} + +impl Default for ThreadId { + fn default() -> Self { + Self::new() + } +} + +// ── State machine ─────────────────────────────────────────── + +/// Thread lifecycle state. +/// +/// ```text +/// Created → Running → Waiting → Running (resume) +/// → Suspended → Running (resume) +/// → Completed → Done +/// → Failed +/// ``` +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub enum ThreadState { + /// Thread has been created but not yet started. + Created, + /// Thread is actively executing steps. + Running, + /// Waiting for external input (user approval, child completion). + Waiting, + /// Paused by system (resource pressure, priority preemption). + Suspended, + /// Execution finished successfully. + Completed, + /// Fully finished (terminal). + Done, + /// Terminal failure. + Failed, +} + +impl ThreadState { + /// Check whether a transition to `target` is valid. + pub fn can_transition_to(self, target: Self) -> bool { + matches!( + (self, target), + // From Created + (Self::Created, Self::Running) + | (Self::Created, Self::Failed) + // From Running + | (Self::Running, Self::Waiting) + | (Self::Running, Self::Suspended) + | (Self::Running, Self::Completed) + | (Self::Running, Self::Failed) + // From Waiting + | (Self::Waiting, Self::Running) + | (Self::Waiting, Self::Failed) + // From Suspended + | (Self::Suspended, Self::Running) + | (Self::Suspended, Self::Failed) + // From Completed + | (Self::Completed, Self::Done) + ) + } + + /// Whether this state is terminal (no further transitions possible). + pub fn is_terminal(self) -> bool { + matches!(self, Self::Done | Self::Failed) + } + + /// Whether this state represents active work. + pub fn is_active(self) -> bool { + matches!(self, Self::Running | Self::Waiting) + } +} + +// ── Thread type ───────────────────────────────────────────── + +/// The nature of the work a thread performs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum ThreadType { + /// Interactive conversation with a user. + Foreground, + /// Background research or sub-task. + Research, + /// Long-running goal that spawns threads over time. + Mission, +} + +// ── Thread configuration ──────────────────────────────────── + +/// Execution parameters for a thread. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ThreadConfig { + /// Maximum number of LLM call iterations. + pub max_iterations: usize, + /// Maximum wall-clock duration for the thread. + pub max_duration: Option, + /// Whether to detect and nudge on tool intent without action calls. + pub enable_tool_intent_nudge: bool, + /// Maximum number of tool intent nudges per thread. + pub max_tool_intent_nudges: u32, + + // ── Budget controls (Phase 4, from RLM cross-reference) ── + /// Maximum cumulative input+output tokens before termination. + pub max_tokens_total: Option, + /// Maximum consecutive steps with errors before termination. + /// Resets to 0 on any successful step (matching official RLM behavior). + pub max_consecutive_errors: Option, + /// Model context limit in tokens (for compaction threshold calculation). + /// Default: 128,000. Used to trigger compaction at 85% usage. + pub model_context_limit: usize, + /// Whether to enable automatic compaction when context grows large. + pub enable_compaction: bool, + /// Compaction threshold as fraction of model_context_limit (0.0-1.0). + /// Default: 0.85 (matching official RLM). + pub compaction_threshold: f64, + /// Maximum cumulative USD cost before termination. + /// Requires the LlmBackend to populate `TokenUsage::cost_usd`. + pub max_budget_usd: Option, + /// Depth of this thread in the recursive call tree. + /// Root threads are depth 0. Sub-calls via rlm_query() increment depth. + pub depth: u32, + /// Maximum recursion depth for rlm_query() sub-calls. + pub max_depth: u32, +} + +impl Default for ThreadConfig { + fn default() -> Self { + Self { + max_iterations: 50, + max_duration: None, + enable_tool_intent_nudge: true, + max_tool_intent_nudges: 2, + max_tokens_total: None, + max_consecutive_errors: Some(5), + max_budget_usd: None, + model_context_limit: 128_000, + enable_compaction: false, + compaction_threshold: 0.85, + depth: 0, + max_depth: 1, + } + } +} + +/// Provenance for a skill that was active during thread execution. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ActiveSkillProvenance { + pub doc_id: DocId, + pub name: String, + pub version: u32, + #[serde(default)] + pub snippet_names: Vec, + #[serde(default)] + pub force_activated: bool, +} + +const ACTIVE_SKILLS_METADATA_KEY: &str = "active_skills"; + +// ── Thread ────────────────────────────────────────────────── + +/// A thread — the unit of work. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Thread { + pub id: ThreadId, + pub goal: String, + pub thread_type: ThreadType, + pub state: ThreadState, + pub project_id: ProjectId, + /// Tenant isolation: the user who owns this thread. + #[serde(default = "default_user_id")] + pub user_id: String, + pub parent_id: Option, + pub config: ThreadConfig, + /// User-visible transcript for the thread. + pub messages: Vec, + /// Internal execution transcript used by the orchestrator for inference, + /// tool traces, compaction, and resumable working state. + #[serde(default)] + pub internal_messages: Vec, + pub events: Vec, + pub capability_leases: Vec, + pub metadata: serde_json::Value, + pub created_at: DateTime, + pub updated_at: DateTime, + pub completed_at: Option>, + pub step_count: usize, + pub total_tokens_used: u64, + /// Cumulative USD cost across all steps. + pub total_cost_usd: f64, +} + +impl Thread { + /// Create a new thread in the `Created` state. + pub fn new( + goal: impl Into, + thread_type: ThreadType, + project_id: ProjectId, + user_id: impl Into, + config: ThreadConfig, + ) -> Self { + let now = Utc::now(); + Self { + id: ThreadId::new(), + goal: goal.into(), + thread_type, + state: ThreadState::Created, + project_id, + user_id: user_id.into(), + parent_id: None, + config, + messages: Vec::new(), + internal_messages: Vec::new(), + events: Vec::new(), + capability_leases: Vec::new(), + metadata: serde_json::Value::Object(serde_json::Map::new()), + created_at: now, + updated_at: now, + completed_at: None, + step_count: 0, + total_tokens_used: 0, + total_cost_usd: 0.0, + } + } + + /// Create a child thread with a parent reference. + pub fn with_parent(mut self, parent_id: ThreadId) -> Self { + self.parent_id = Some(parent_id); + self + } + + pub fn owner_id(&self) -> OwnerId<'_> { + OwnerId::from_user_id(&self.user_id) + } + + pub fn is_owned_by(&self, user_id: &str) -> bool { + self.owner_id().matches_user(user_id) + } + + /// Persist active skill provenance in thread metadata. + pub fn set_active_skills( + &mut self, + active_skills: &[ActiveSkillProvenance], + ) -> Result<(), EngineError> { + let metadata = self + .metadata + .as_object_mut() + .ok_or_else(|| EngineError::Store { + reason: "thread metadata is not a JSON object".into(), + })?; + metadata.insert( + ACTIVE_SKILLS_METADATA_KEY.into(), + serde_json::to_value(active_skills).map_err(|e| EngineError::Store { + reason: format!("failed to serialize active skill provenance: {e}"), + })?, + ); + self.updated_at = Utc::now(); + Ok(()) + } + + /// Load active skill provenance from thread metadata. + pub fn active_skills(&self) -> Vec { + self.metadata + .get(ACTIVE_SKILLS_METADATA_KEY) + .cloned() + .and_then(|value| serde_json::from_value(value).ok()) + .unwrap_or_default() + } + + /// Transition to a new state, recording an event. + pub fn transition_to( + &mut self, + new_state: ThreadState, + reason: Option, + ) -> Result<(), EngineError> { + if !self.state.can_transition_to(new_state) { + return Err(EngineError::InvalidTransition { + from: self.state, + to: new_state, + }); + } + + let event = ThreadEvent::new( + self.id, + EventKind::StateChanged { + from: self.state, + to: new_state, + reason, + }, + ); + self.events.push(event); + self.state = new_state; + self.updated_at = Utc::now(); + + if new_state == ThreadState::Completed || new_state == ThreadState::Done { + self.completed_at = Some(Utc::now()); + } + + Ok(()) + } + + /// Add an event to this thread's log. + pub fn add_event(&mut self, kind: EventKind) { + self.events.push(ThreadEvent::new(self.id, kind)); + self.updated_at = Utc::now(); + } + + /// Add a message to this thread's conversation. + pub fn add_message(&mut self, message: ThreadMessage) { + let preview = if message.content.chars().count() > 80 { + let p: String = message.content.chars().take(80).collect(); + format!("{p}...") + } else { + message.content.clone() + }; + self.add_event(EventKind::MessageAdded { + role: format!("{:?}", message.role), + content_preview: preview, + }); + self.messages.push(message); + } + + /// Add a message to the internal execution transcript without exposing it + /// as a user-visible conversation message. + pub fn add_internal_message(&mut self, message: ThreadMessage) { + self.internal_messages.push(message); + self.updated_at = Utc::now(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::memory::DocId; + + fn make_thread() -> Thread { + Thread::new( + "test goal", + ThreadType::Foreground, + ProjectId::new(), + "test-user", + ThreadConfig::default(), + ) + } + + // ── State machine tests ───────────────────────────────── + + #[test] + fn default_config_has_concrete_consecutive_error_limit() { + // Regression: a None default serializes to null and makes the Python + // orchestrator's `consecutive_action_errors >= max_consecutive_errors + 2` + // guard crash with TypeError on the first action error, since + // dict.get("key", 5) returns None (not 5) when the key is present + // with a null value. + let config = ThreadConfig::default(); + assert_eq!(config.max_consecutive_errors, Some(5)); + } + + #[test] + fn created_can_transition_to_running() { + assert!(ThreadState::Created.can_transition_to(ThreadState::Running)); + } + + #[test] + fn created_can_transition_to_failed() { + assert!(ThreadState::Created.can_transition_to(ThreadState::Failed)); + } + + #[test] + fn created_cannot_transition_to_completed() { + assert!(!ThreadState::Created.can_transition_to(ThreadState::Completed)); + } + + #[test] + fn running_can_transition_to_waiting() { + assert!(ThreadState::Running.can_transition_to(ThreadState::Waiting)); + } + + #[test] + fn running_can_transition_to_suspended() { + assert!(ThreadState::Running.can_transition_to(ThreadState::Suspended)); + } + + #[test] + fn running_can_transition_to_completed() { + assert!(ThreadState::Running.can_transition_to(ThreadState::Completed)); + } + + #[test] + fn running_can_transition_to_failed() { + assert!(ThreadState::Running.can_transition_to(ThreadState::Failed)); + } + + #[test] + fn waiting_can_resume_to_running() { + assert!(ThreadState::Waiting.can_transition_to(ThreadState::Running)); + } + + #[test] + fn suspended_can_resume_to_running() { + assert!(ThreadState::Suspended.can_transition_to(ThreadState::Running)); + } + + #[test] + fn completed_can_transition_to_done() { + assert!(ThreadState::Completed.can_transition_to(ThreadState::Done)); + } + + #[test] + fn done_is_terminal() { + assert!(ThreadState::Done.is_terminal()); + assert!(!ThreadState::Done.can_transition_to(ThreadState::Running)); + } + + #[test] + fn failed_is_terminal() { + assert!(ThreadState::Failed.is_terminal()); + assert!(!ThreadState::Failed.can_transition_to(ThreadState::Running)); + } + + #[test] + fn running_is_active() { + assert!(ThreadState::Running.is_active()); + } + + #[test] + fn waiting_is_active() { + assert!(ThreadState::Waiting.is_active()); + } + + #[test] + fn created_is_not_active() { + assert!(!ThreadState::Created.is_active()); + } + + // ── Thread lifecycle tests ────────────────────────────── + + #[test] + fn new_thread_is_created() { + let t = make_thread(); + assert_eq!(t.state, ThreadState::Created); + assert!(t.events.is_empty()); + assert!(t.messages.is_empty()); + } + + #[test] + fn valid_transition_succeeds() { + let mut t = make_thread(); + assert!(t.transition_to(ThreadState::Running, None).is_ok()); + assert_eq!(t.state, ThreadState::Running); + assert_eq!(t.events.len(), 1); + } + + #[test] + fn invalid_transition_fails() { + let mut t = make_thread(); + let result = t.transition_to(ThreadState::Completed, None); + assert!(result.is_err()); + assert_eq!(t.state, ThreadState::Created); + } + + #[test] + fn full_lifecycle_created_to_done() { + let mut t = make_thread(); + t.transition_to(ThreadState::Running, None).unwrap(); + t.transition_to(ThreadState::Completed, Some("finished".into())) + .unwrap(); + t.transition_to(ThreadState::Done, None).unwrap(); + assert!(t.state.is_terminal()); + assert_eq!(t.events.len(), 3); + assert!(t.completed_at.is_some()); + } + + #[test] + fn add_message_records_event() { + let mut t = make_thread(); + t.add_message(ThreadMessage::user("hello")); + assert_eq!(t.messages.len(), 1); + assert_eq!(t.events.len(), 1); + match &t.events[0].kind { + EventKind::MessageAdded { role, .. } => assert_eq!(role, "User"), + other => panic!("unexpected event: {other:?}"), + } + } + + #[test] + fn child_thread_has_parent() { + let parent = make_thread(); + let child = Thread::new( + "child goal", + ThreadType::Research, + parent.project_id, + "test-user", + ThreadConfig::default(), + ) + .with_parent(parent.id); + assert_eq!(child.parent_id, Some(parent.id)); + } + + #[test] + fn active_skill_provenance_roundtrips_through_metadata() { + let mut thread = make_thread(); + let skills = vec![ActiveSkillProvenance { + doc_id: DocId::new(), + name: "github-pr-workflow".to_string(), + version: 3, + snippet_names: vec!["list_prs".to_string()], + force_activated: true, + }]; + + thread.set_active_skills(&skills).unwrap(); + + assert_eq!(thread.active_skills(), skills); + } +} diff --git a/crates/ironclaw_gateway/Cargo.toml b/crates/ironclaw_gateway/Cargo.toml new file mode 100644 index 00000000000..04e55cadbe5 --- /dev/null +++ b/crates/ironclaw_gateway/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "ironclaw_gateway" +version = "0.1.0" +edition = "2024" +rust-version = "1.92" +description = "Gateway frontend assets, layout configuration, and widget extension system for IronClaw" +license = "MIT OR Apache-2.0" +publish = false + +[package.metadata.dist] +dist = false + +[dependencies] +serde = { version = "1", features = ["derive"] } +serde_json = "1" +thiserror = "2" +tracing = "0.1" diff --git a/crates/ironclaw_gateway/src/assets.rs b/crates/ironclaw_gateway/src/assets.rs new file mode 100644 index 00000000000..76a5434fe4b --- /dev/null +++ b/crates/ironclaw_gateway/src/assets.rs @@ -0,0 +1,54 @@ +//! Embedded static assets for the IronClaw web gateway. +//! +//! All frontend files are compiled into the binary via `include_str!()` / +//! `include_bytes!()`. The web gateway serves these as the default baseline; +//! workspace-stored customizations (layout config, widgets, CSS overrides) +//! are layered on top at runtime. + +// ==================== Core Files ==================== + +/// Main HTML page (SPA shell). +pub const INDEX_HTML: &str = include_str!("../static/index.html"); + +/// Main application JavaScript. +pub const APP_JS: &str = include_str!("../static/app.js"); + +/// Base stylesheet. +pub const STYLE_CSS: &str = include_str!("../static/style.css"); + +/// Theme initialization script (runs synchronously in `` to prevent FOUC). +pub const THEME_INIT_JS: &str = include_str!("../static/theme-init.js"); + +/// Favicon. +pub const FAVICON_ICO: &[u8] = include_bytes!("../static/favicon.ico"); + +// ==================== Internationalization ==================== + +/// i18n core library. +pub const I18N_INDEX_JS: &str = include_str!("../static/i18n/index.js"); + +/// English translations. +pub const I18N_EN_JS: &str = include_str!("../static/i18n/en.js"); + +/// Chinese (Simplified) translations. +pub const I18N_ZH_CN_JS: &str = include_str!("../static/i18n/zh-CN.js"); + +/// Korean translations. +pub const I18N_KO_JS: &str = include_str!("../static/i18n/ko.js"); + +/// i18n integration with the app. +pub const I18N_APP_JS: &str = include_str!("../static/i18n-app.js"); + +// ==================== Admin Panel ==================== + +/// Shared theme tokens (CSS custom properties). +pub const THEME_CSS: &str = include_str!("../static/theme.css"); + +/// Admin panel HTML shell. +pub const ADMIN_HTML: &str = include_str!("../static/admin.html"); + +/// Admin panel stylesheet. +pub const ADMIN_CSS: &str = include_str!("../static/admin.css"); + +/// Admin panel JavaScript. +pub const ADMIN_JS: &str = include_str!("../static/admin.js"); diff --git a/crates/ironclaw_gateway/src/bundle.rs b/crates/ironclaw_gateway/src/bundle.rs new file mode 100644 index 00000000000..0b615df6ad4 --- /dev/null +++ b/crates/ironclaw_gateway/src/bundle.rs @@ -0,0 +1,757 @@ +//! Frontend bundle assembly. +//! +//! Combines the embedded base HTML with workspace customizations (layout +//! config, widgets, CSS overrides) into the final served page. + +use crate::layout::LayoutConfig; +use crate::widget::{WidgetManifest, scope_css}; + +/// Escape HTML special characters to prevent XSS in text content. +fn escape_html(s: &str) -> String { + s.replace('&', "&") + .replace('<', "<") + .replace('>', ">") +} + +/// Escape HTML attribute value (includes quotes). +fn escape_html_attr(s: &str) -> String { + s.replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) +} + +/// Rewrite any occurrence of `needle` (ASCII, case-insensitive) in `s` by +/// inserting a backslash between the leading `<` and `/`, turning `` or ``, which let a `` sequence inside a color + // value close the tag early and inject arbitrary HTML. Apply the same + // escape here so branding stays in lock-step with the other paths. + let css_vars = bundle.layout.branding.to_css_vars(); + if !css_vars.is_empty() { + let safe_vars = escape_tag_close(&css_vars, "{}", safe_vars)); + } + + // --- Body injections --- + + // Layout config as global variable. JSON strings can contain `` + // (serde_json does not escape `<` or `/` by default), so neutralize any + // ` tag. + // + // The `nonce` attribute carries the [`NONCE_PLACEHOLDER`] sentinel — the + // gateway swaps it for a fresh per-response nonce that matches the + // response's `Content-Security-Policy` header. Without the nonce the + // browser blocks this inline script under the gateway's CSP. + match serde_json::to_string(&bundle.layout) { + Ok(layout_json) => { + let safe_layout = escape_tag_close(&layout_json, "window.__IRONCLAW_LAYOUT__ = {safe_layout};" + )); + } + Err(e) => { + // `LayoutConfig` and every nested type derive `Serialize` cleanly, + // so this branch is unreachable on well-typed input. Surface it + // anyway — a silent drop here would mean the customized HTML + // ships without `window.__IRONCLAW_LAYOUT__`, and the IIFE in + // `app.js` would no-op all branding/tab/chat customizations + // without leaving a trace. A loud warn at the failure site is + // cheap insurance against a future refactor that introduces a + // serialization-fallible field. + tracing::warn!( + error = %e, + "failed to serialize LayoutConfig for window.__IRONCLAW_LAYOUT__ injection — \ + customizations will not apply" + ); + } + } + + // Widget CSS (scoped) and JS + for widget in &bundle.widgets { + if let Some(ref css) = widget.css { + let scoped = scope_css(css, &widget.manifest.id); + if !scoped.trim().is_empty() { + // Neutralize any `` would otherwise break out of the tag. + // + // No nonce needed: the gateway's CSP allows `'unsafe-inline'` + // for `style-src`. Scripts are the only nonce-gated tags. + let safe_css = escape_tag_close(&scoped, "{}", + escape_html_attr(&widget.manifest.id), + safe_css + )); + } + } + + // Widget JS inlined (avoids auth issues with ` to prevent tag breakout (XSS) and + // stamp the CSP nonce placeholder so the gateway can authorize this + // script under its `script-src 'nonce-…'` policy. + let safe_js = escape_tag_close(&widget.js, "\n{}\n", + NONCE_PLACEHOLDER, + escape_html_attr(&widget.manifest.id), + safe_js + )); + } + + // Custom CSS + if let Some(ref custom_css) = bundle.custom_css + && !custom_css.trim().is_empty() + { + // Same reasoning as widget CSS — neutralize `{}", safe_custom)); + } + + // --- Assemble --- + + let mut result = base_html.to_string(); + + // Inject before + if !head_injections.is_empty() { + let head_block = head_injections.join("\n"); + if let Some(pos) = result.rfind("") { + result.insert_str(pos, &format!("\n{}\n", head_block)); + } + } + + // Override if branding title is set (HTML-escaped to prevent XSS) + if let Some(ref title) = bundle.layout.branding.title + && let Some(start) = result.find("<title>") + && let Some(end) = result[start..].find("") + { + let end = start + end + "".len(); + result.replace_range( + start..end, + &format!("{}", escape_html(title)), + ); + } + + // Inject before + if !body_injections.is_empty() { + let body_block = body_injections.join("\n"); + if let Some(pos) = result.rfind("") { + result.insert_str(pos, &format!("\n{}\n", body_block)); + } + } + + result +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::layout::*; + use crate::widget::*; + + const MINIMAL_HTML: &str = + "IronClaw"; + + #[test] + fn test_assemble_index_no_customizations() { + let bundle = FrontendBundle::default(); + let result = assemble_index(MINIMAL_HTML, &bundle); + // Layout config is always injected (even when default/empty) + assert!(result.contains("window.__IRONCLAW_LAYOUT__")); + // No branding overrides or custom CSS + assert!(!result.contains("--color-primary")); + assert!(!result.contains("data-custom-css")); + } + + #[test] + fn test_assemble_index_branding_title() { + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + title: Some("Acme AI".to_string()), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!(result.contains("Acme AI")); + assert!(!result.contains("IronClaw")); + } + + #[test] + fn test_assemble_index_branding_colors() { + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + colors: Some(BrandingColors { + primary: Some("#0066cc".to_string()), + accent: None, + }), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!(result.contains("--color-primary: #0066cc;")); + } + + #[test] + fn test_assemble_index_layout_config_injected() { + let bundle = FrontendBundle { + layout: LayoutConfig { + tabs: TabConfig { + hidden: Some(vec!["routines".to_string()]), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!(result.contains("window.__IRONCLAW_LAYOUT__")); + assert!(result.contains("routines")); + } + + #[test] + fn test_assemble_index_widget_script() { + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "dashboard".to_string(), + name: "Dashboard".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "console.log('hello');".to_string(), + css: Some(".panel { color: red; }".to_string()), + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!(result.contains("data-widget=\"dashboard\"")); + assert!(result.contains("console.log('hello');")); + assert!(result.contains("data-widget=\"dashboard\"")); + assert!(result.contains("[data-widget=\"dashboard\"] .panel")); + } + + #[test] + fn test_assemble_index_custom_css() { + let bundle = FrontendBundle { + custom_css: Some("body { background: #111; }".to_string()), + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!(result.contains("data-custom-css")); + assert!(result.contains("background: #111;")); + } + + // ==================== Security Tests ==================== + + #[test] + fn test_assemble_index_title_xss_escaped() { + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + title: Some("".to_string()), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Title should be HTML-escaped, not rendered as a script tag + assert!(result.contains("<script>alert(1)</script>")); + assert!(!result.contains("<script>")); + } + + #[test] + fn test_assemble_index_widget_js_script_breakout_escaped() { + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "evil".to_string(), + name: "Evil Widget".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "var x = '</script><script>alert(1)</script>';".to_string(), + css: None, + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // </script> in widget JS should be escaped to prevent tag breakout + assert!(!result.contains("</script><script>alert(1)")); + assert!(result.contains("<\\/script>")); + } + + #[test] + fn test_assemble_index_layout_script_carries_nonce_placeholder() { + // Every injected <script> must carry the nonce placeholder so the + // gateway can rotate it per response. Without this attribute, the + // browser blocks the script under the gateway's strict CSP. + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + title: Some("Acme".to_string()), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + assert!( + result.contains(&format!("<script nonce=\"{NONCE_PLACEHOLDER}\">")), + "layout JSON script must carry nonce placeholder, got: {result}" + ); + } + + #[test] + fn test_assemble_index_widget_script_carries_nonce_placeholder() { + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "dashboard".to_string(), + name: "Dashboard".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "console.log('hi');".to_string(), + css: None, + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Widget script must have BOTH the type=module and the nonce attribute. + assert!( + result.contains(&format!( + "<script type=\"module\" nonce=\"{NONCE_PLACEHOLDER}\" data-widget=\"dashboard\">" + )), + "widget script tag must carry nonce placeholder, got: {result}" + ); + // The placeholder must be a substring the caller can substitute. The + // sentinel string itself must NOT be a valid nonce — it should never + // accidentally appear elsewhere in well-formed HTML. + assert!(NONCE_PLACEHOLDER.starts_with("__")); + assert!(!NONCE_PLACEHOLDER.contains(' ')); + } + + #[test] + fn test_assemble_index_widget_style_has_no_nonce() { + // Inline <style> blocks don't need a nonce — the gateway's CSP allows + // 'unsafe-inline' for style-src. Adding nonce there would be dead + // weight. This test pins that decision so a future change doesn't + // accidentally start nonce-gating styles. + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "styled".to_string(), + name: "Styled".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "// noop".to_string(), + css: Some(".panel { color: red; }".to_string()), + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Find the <style data-widget="styled"> tag and assert no nonce attr. + let style_tag = "<style data-widget=\"styled\">"; + assert!(result.contains(style_tag)); + assert!( + !result.contains("<style data-widget=\"styled\" nonce="), + "<style> tags must not carry nonce attributes" + ); + } + + #[test] + fn test_assemble_index_layout_json_script_breakout_escaped() { + // Layout branding title containing `</script>` must not break out + // of the `window.__IRONCLAW_LAYOUT__` script injection. + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + title: Some("evil</script><script>alert(1)</script>".to_string()), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // The layout <script> tag must not contain a raw `</script>` closer + // from the title — it must be neutralized as `<\/script>`. + // Title itself is injected into <title> HTML-escaped (a separate code path), + // but it's also present inside the JSON string in window.__IRONCLAW_LAYOUT__. + let layout_start = result.find("window.__IRONCLAW_LAYOUT__").unwrap(); + let layout_end = result[layout_start..].find("</script>").unwrap() + layout_start; + let layout_script = &result[layout_start..layout_end]; + // Between the opening `<script>` and the first real `</script>`, the + // raw breakout payload must not appear. + assert!( + !layout_script.contains("</script>"), + "raw </script> inside layout JSON broke out of the script tag" + ); + assert!(layout_script.contains("<\\/script>")); + } + + #[test] + fn test_assemble_index_widget_css_style_breakout_escaped() { + // Widget CSS containing `</style>` (e.g., via a content: "…" string) + // must not break out of the <style> tag. + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "evil-css".to_string(), + name: "Evil CSS Widget".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "// safe".to_string(), + css: Some( + ".x::before { content: \"</style><script>alert(1)</script>\"; }".to_string(), + ), + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // The widget <style> block must not contain a raw `</style>` from + // the CSS string literal. + let style_start = result.find("data-widget=\"evil-css\">").unwrap(); + // `style_start` points into the opening <style> tag's attribute; the + // first `</style>` after this marker must be the tag's real closer. + let rest = &result[style_start..]; + let first_close = rest.find("</style>").unwrap(); + let body = &rest[..first_close]; + assert!( + !body.contains("</style>"), + "raw </style> inside widget CSS broke out of the style tag" + ); + assert!(body.contains("<\\/style>")); + } + + #[test] + fn test_assemble_index_branding_style_breakout_escaped() { + // Defense in depth for the branding CSS-vars injection point. + // The `BrandingConfig` color validator (in `layout.rs`) is the + // primary defense and strips anything containing `</style>` + // before it ever reaches `assemble_index`, so the head `<style>` + // block for branding should never even be emitted in this case. + // + // This test locks in BOTH contracts: + // + // 1. A hostile color value is dropped before it lands in the + // head — no `--color-primary` declaration appears at all. + // 2. The only place the raw breakout string appears in the final + // document is the layout-config `<script>` (which + // `escape_tag_close` already handled for the `</script>` + // sequence); it must NOT appear inside any head `<style>` + // block that would render the injection as HTML. + // + // If either the validator or the bundle-level escape regresses, + // this test fails with a useful diagnostic. + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + colors: Some(BrandingColors { + primary: Some("red</style><script>alert(1)</script>".to_string()), + accent: None, + }), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + + // Contract 1: validator drops the hostile primary value. + assert!( + !result.contains("--color-primary"), + "hostile branding color must be dropped by the validator; got: {result}" + ); + + // Contract 2: no `<style>` tag in the head section contains the + // raw `</style>` breakout. The head runs from `<head>` up to + // `</head>`; search that slice for any `<style>` block emitted + // by branding and verify none contains the raw close-tag. + if let Some(head_end) = result.find("</head>") { + let head = &result[..head_end]; + let mut search = head; + while let Some(style_open) = search.find("<style") { + let after_open = &search[style_open..]; + let body_start = after_open + .find('>') + .map(|i| i + 1) + .unwrap_or(after_open.len()); + let body_rest = &after_open[body_start..]; + let body_end = body_rest.find("</style>").unwrap_or(body_rest.len()); + let body = &body_rest[..body_end]; + assert!( + !body.contains("</style>"), + "raw </style> inside head <style> block: {body}" + ); + // Advance past this block for any subsequent matches. + search = &body_rest[body_end.min(body_rest.len())..]; + } + } + } + + #[test] + fn test_assemble_index_custom_css_style_breakout_escaped() { + // Custom workspace CSS containing `</style>` must not break out. + let bundle = FrontendBundle { + custom_css: Some("body { color: red; } </style><script>alert(1)</script>".to_string()), + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + let style_start = result.find("data-custom-css>").unwrap(); + let rest = &result[style_start..]; + let first_close = rest.find("</style>").unwrap(); + let body = &rest[..first_close]; + assert!( + !body.contains("</style>"), + "raw </style> inside custom CSS broke out of the style tag" + ); + assert!(body.contains("<\\/style>")); + } + + #[test] + fn test_escape_tag_close_case_insensitive() { + // HTML parsers treat `</SCRIPT>` and `</script >` the same as + // `</script>`, so the escape must be case-insensitive. + assert_eq!( + escape_tag_close("a </SCRIPT> b", "</script"), + "a <\\/SCRIPT> b" + ); + assert_eq!( + escape_tag_close("a </Script\n> b", "</script"), + "a <\\/Script\n> b" + ); + // Unrelated `<` and `/` characters must be untouched. + assert_eq!(escape_tag_close("<div>x</div>", "</script"), "<div>x</div>"); + } + + #[test] + fn test_escape_tag_close_multibyte_safe() { + // Must not panic on multi-byte UTF-8 characters adjacent to the needle. + let input = "日本語</script>日本語"; + let out = escape_tag_close(input, "</script"); + assert!(out.contains("<\\/script>")); + assert!(out.contains("日本語")); + } + + #[test] + fn test_assemble_index_widget_id_xss_escaped() { + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "x\" onload=\"alert(1)".to_string(), + name: "XSS Widget".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: "// safe".to_string(), + css: None, + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Widget ID in attributes should be escaped + assert!(result.contains(""")); + assert!(!result.contains("onload=\"alert(1)\"")); + } + + // ==================== Edge Case Tests ==================== + + #[test] + fn test_escape_html_basic() { + assert_eq!(escape_html("<b>bold</b>"), "<b>bold</b>"); + assert_eq!(escape_html("a & b"), "a & b"); + assert_eq!(escape_html("safe text"), "safe text"); + assert_eq!(escape_html(""), ""); + } + + #[test] + fn test_escape_html_attr_quotes() { + assert_eq!( + escape_html_attr("value\"with\"quotes"), + "value"with"quotes" + ); + } + + #[test] + fn test_assemble_index_missing_head_body_tags() { + // Gracefully handles malformed HTML (no </head> or </body>) + let html = "<html><body>content</body></html>"; + let bundle = FrontendBundle { + layout: LayoutConfig { + branding: BrandingConfig { + title: Some("Test".to_string()), + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let result = assemble_index(html, &bundle); + // Should still contain layout config (injected before </body>) + assert!(result.contains("window.__IRONCLAW_LAYOUT__")); + } + + #[test] + fn test_assemble_index_empty_widget_js() { + let bundle = FrontendBundle { + widgets: vec![ResolvedWidget { + manifest: WidgetManifest { + id: "empty".to_string(), + name: "Empty Widget".to_string(), + slot: WidgetSlot::Tab, + icon: None, + position: None, + }, + js: String::new(), + css: None, + }], + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Empty JS should still produce a script tag (widget registers itself) + assert!(result.contains("data-widget=\"empty\"")); + } + + #[test] + fn test_assemble_index_empty_custom_css_skipped() { + let bundle = FrontendBundle { + custom_css: Some(" \n ".to_string()), + ..Default::default() + }; + let result = assemble_index(MINIMAL_HTML, &bundle); + // Whitespace-only custom CSS should be skipped + assert!(!result.contains("data-custom-css")); + } +} diff --git a/crates/ironclaw_gateway/src/layout.rs b/crates/ironclaw_gateway/src/layout.rs new file mode 100644 index 00000000000..06d5c7a64aa --- /dev/null +++ b/crates/ironclaw_gateway/src/layout.rs @@ -0,0 +1,818 @@ +//! Layout configuration types for frontend customization. +//! +//! A [`LayoutConfig`] is stored as `.system/gateway/layout.json` in the +//! workspace. It controls branding, tab visibility/order, chat features, and +//! per-widget configuration. All fields are optional with sensible defaults. + +use std::collections::HashMap; + +use serde::{Deserialize, Serialize}; + +/// Top-level layout configuration. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct LayoutConfig { + /// Branding overrides (title, logo, colors). + #[serde(default)] + pub branding: BrandingConfig, + + /// Tab bar configuration. + #[serde(default)] + pub tabs: TabConfig, + + /// Chat panel configuration. + #[serde(default)] + pub chat: ChatConfig, + + /// Per-widget instance configuration (keyed by widget ID). + #[serde(default)] + pub widgets: HashMap<String, WidgetInstanceConfig>, +} + +/// Branding overrides for the gateway UI. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct BrandingConfig { + /// Page title (replaces default "IronClaw"). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub title: Option<String>, + + /// Subtitle shown below the title. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub subtitle: Option<String>, + + /// URL to a logo image. Always read via [`Self::safe_logo_url`] — + /// the field is `pub(crate)` so external Rust callers must route + /// through the validating getter, and the [`skip_unsafe_url`] serde + /// predicate drops unsafe values from the JSON output so the JS + /// side (`window.__IRONCLAW_LAYOUT__` and + /// `GET /api/frontend/layout`) never sees them. + #[serde(default, skip_serializing_if = "skip_unsafe_url")] + pub(crate) logo_url: Option<String>, + + /// URL to a custom favicon. Same access discipline as + /// [`Self::logo_url`] — read via [`Self::safe_favicon_url`]. + #[serde(default, skip_serializing_if = "skip_unsafe_url")] + pub(crate) favicon_url: Option<String>, + + /// Color overrides (injected as CSS custom properties on `:root`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub colors: Option<BrandingColors>, +} + +/// Serde `skip_serializing_if` predicate for branding URL fields. +/// Returns `true` (drop the field from JSON output) when the value is +/// missing, empty, or fails [`is_safe_url`]. +/// +/// This closes the wire-format leg of the URL validation: even if a +/// future intra-crate Rust caller bypasses the `safe_logo_url` / +/// `safe_favicon_url` getters and writes a hostile value into the +/// `pub(crate)` field directly, the JSON serialized to the JS side +/// (`window.__IRONCLAW_LAYOUT__`) and the response body of +/// `GET /api/frontend/layout` simply omit the field entirely — no +/// `null`, no `javascript:` payload, nothing for a future consumer to +/// inadvertently render. Belt-and-braces with the type-level visibility +/// downgrade. +/// +/// `skip_serializing_if` predicates take a `&Option<String>` and return +/// `bool`; we negate the "is present and safe" check so the field is +/// dropped on every other branch. +fn skip_unsafe_url(value: &Option<String>) -> bool { + !value.as_deref().is_some_and(is_safe_url) +} + +/// Color overrides for the UI theme. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct BrandingColors { + /// Primary brand color (e.g., `"#0066cc"`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub primary: Option<String>, + + /// Accent color. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub accent: Option<String>, +} + +/// Tab bar layout configuration. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct TabConfig { + /// Ordered list of tab IDs to display (built-in + widget tabs). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub order: Option<Vec<String>>, + + /// Tab IDs to hide from the tab bar. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub hidden: Option<Vec<String>>, + + /// Default tab to show on load. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub default_tab: Option<String>, +} + +/// Chat panel feature flags. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct ChatConfig { + /// Show suggestion chips below the input. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub suggestions: Option<bool>, + + /// Enable image upload in the chat input. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub image_upload: Option<bool>, + + /// Opt in to converting inline JSON-shaped fragments in assistant + /// messages into styled data cards (`upgradeInlineJson` in `app.js`). + /// + /// Disabled by default because the heuristic pattern-matches any + /// balanced `{...}` in rendered markdown — prose containing JSON-like + /// text (`"yes, set the value to {x: 1, y: 2}"`) gets false-positive + /// rewritten into a card. Operators that drive structured data + /// through chat (e.g., a workflow that emits real JSON in every + /// reply) can flip this on; everyone else gets prose left alone. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub upgrade_inline_json: Option<bool>, +} + +/// Per-widget instance configuration. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct WidgetInstanceConfig { + /// Whether this widget is enabled. Defaults to `true` so a layout entry + /// that only customizes a widget's `config` (and omits `enabled`) does + /// not silently disable the widget. + #[serde(default = "default_true")] + pub enabled: bool, + + /// Arbitrary widget-specific configuration passed to `widget.init()`. + #[serde(default)] + pub config: serde_json::Value, +} + +impl Default for WidgetInstanceConfig { + fn default() -> Self { + Self { + enabled: true, + config: serde_json::Value::Null, + } + } +} + +fn default_true() -> bool { + true +} + +impl BrandingConfig { + /// Generate CSS custom property overrides for injection into `:root`. + /// + /// Color values are run through [`is_safe_css_color`] before + /// interpolation so a hostile `layout.json` cannot break out of the + /// `:root {}` block (e.g. + /// `red; } .chat-input[value^="s"] { background: url(...) }`) or close + /// the surrounding `<style>` tag. Invalid values are silently dropped + /// so the rest of the branding config still applies. + pub fn to_css_vars(&self) -> String { + let mut vars = Vec::new(); + if let Some(ref colors) = self.colors { + if let Some(ref primary) = colors.primary + && is_safe_css_color(primary) + { + vars.push(format!("--color-primary: {};", primary)); + } + if let Some(ref accent) = colors.accent + && is_safe_css_color(accent) + { + vars.push(format!("--color-accent: {};", accent)); + } + } + if vars.is_empty() { + String::new() + } else { + format!(":root {{ {} }}", vars.join(" ")) + } + } + + /// Return [`Self::logo_url`] if it passes [`is_safe_url`], otherwise + /// `None`. + /// + /// `logo_url` is currently a passthrough field — no consumer in the + /// browser runtime reads it yet — but it is exposed via + /// `GET /api/frontend/layout` and lands inside the + /// `window.__IRONCLAW_LAYOUT__` JSON island. The first consumer that + /// renders it (most likely as `<img src="…">` or + /// `<link rel="icon" href="…">`) would inherit a footgun if a + /// `layout.json` could ship `javascript:`/`data:` URIs unfiltered. + /// Routing every consumer through this getter — mirroring the + /// `to_css_vars` precedent for branding colors — keeps the validation + /// at the type layer so a future caller can't accidentally bypass it + /// by reading the field directly. + pub fn safe_logo_url(&self) -> Option<&str> { + self.logo_url.as_deref().filter(|v| is_safe_url(v)) + } + + /// Return [`Self::favicon_url`] if it passes [`is_safe_url`], otherwise + /// `None`. Same rationale as [`Self::safe_logo_url`]. + pub fn safe_favicon_url(&self) -> Option<&str> { + self.favicon_url.as_deref().filter(|v| is_safe_url(v)) + } +} + +/// Return `true` if `value` is a safe widget identifier for use in HTML +/// attributes, CSS attribute selectors, and workspace path segments. +/// +/// Widget ids land in three places where the surrounding syntax matters: +/// +/// 1. **HTML attributes** like `data-widget="<id>"` — already protected +/// by `escape_html_attr` at the bundle layer, but a defense-in-depth +/// failure of that escape on a hostile id is the kind of cascading +/// bug we want to make impossible at the type level. +/// 2. **CSS attribute selectors** in `scope_css`'s +/// `[data-widget="<id>"]` prefix — this is the un-escaped path the +/// paranoid review flagged. A literal `"` or `]` in the id would +/// close the selector and inject an arbitrary CSS rule. +/// 3. **Workspace path segments** in +/// `.system/gateway/widgets/{id}/index.js` — `is_safe_segment` is the +/// primary defense here, but the two checks should agree on what +/// "safe" means. +/// +/// The accepted form is intentionally narrow: a single ASCII alphanumeric +/// followed by zero or more `[a-zA-Z0-9._-]`, capped at 64 chars. This +/// covers every existing widget fixture (`skills-viewer`, `dashboard_v2`, +/// `a.b.c`, `widget-1`) while making CSS / HTML / path injection +/// impossible by construction. Operators who need a broader charset +/// can lobby for it in a follow-up; widening a regex is a one-line +/// change, narrowing one after a release is a breaking change. +pub fn is_safe_widget_id(value: &str) -> bool { + if value.is_empty() || value.len() > 64 { + return false; + } + let mut chars = value.chars(); + // First char must be alphanumeric so an id can never look like an + // option flag (`-foo`), a hidden file (`.foo`), or a separator + // fragment. + let Some(first) = chars.next() else { + return false; + }; + if !first.is_ascii_alphanumeric() { + return false; + } + // Subsequent chars: alphanumeric, dot, hyphen, underscore. + chars.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_')) +} + +/// Return `true` if `value` is a syntactically-safe URL for use in `<img +/// src>` / `<link href>` / similar HTML attribute contexts. +/// +/// Accepts a conservative subset: +/// +/// * **HTTPS / HTTP absolute URLs**: `https://example.com/logo.png`, +/// `http://intranet.local/icon.svg`. HTTP is allowed (not just HTTPS) so +/// intranet and dev deployments aren't gratuitously broken; the gateway +/// itself enforces TLS at the network layer where appropriate. +/// * **Site-relative paths**: `/static/logo.png`, `/foo/bar.svg`. Must +/// start with a single `/`, NOT `//` (protocol-relative — those can be +/// hijacked into a different scheme by the browser's URL parser). +/// +/// Rejects: +/// +/// * `javascript:`, `data:`, `vbscript:`, `file:`, `blob:`, and any other +/// non-HTTP(S) absolute scheme. These are the classic +/// `<img src="javascript:…">` / data-URI tracking-pixel vectors. +/// * Strings containing characters that could break out of an HTML +/// attribute (`<`, `>`, `"`, `'`, backtick, backslash) or terminate it +/// prematurely (NUL, newline, carriage return, tab). +/// * Empty / whitespace-only values, and anything > 2048 bytes (matches +/// the de-facto Chrome / Apache URL length cap; longer values are +/// either pathological or an exfil vector). +/// +/// Strict URL parsing is **not** the goal — this validator favors +/// rejecting suspicious shapes over preserving every legal RFC 3986 form. +/// A user who needs an exotic URL can pre-encode it; the gateway's job is +/// to refuse anything an attacker could weaponize against an unsuspecting +/// future consumer. +pub(crate) fn is_safe_url(value: &str) -> bool { + // Length check runs against the RAW input, not the trimmed view, so a + // pathological 4 KB value padded with leading/trailing whitespace can't + // sneak past the cap by collapsing to a short URL after trim. The cap + // is intentionally a guard against exfil-shaped payloads, not a guard + // against the resolved URL itself, so the right thing to count is what + // the caller actually wrote. + if value.len() > 2048 { + return false; + } + let v = value.trim(); + if v.is_empty() { + return false; + } + // Reject HTML-attribute breakout vectors and any control character + // that could be smuggled through copy-paste from a hostile source. + if v.bytes().any(|b| { + matches!( + b, + b'<' | b'>' | b'"' | b'\'' | b'`' | b'\\' | b'\0' | b'\n' | b'\r' | b'\t' + ) + }) { + return false; + } + // Site-relative path. Must start with a single `/`, NOT `//` + // (protocol-relative URLs are scheme-flippable in the browser URL + // parser and historically have been a source of CSP-bypass tricks). + if let Some(rest) = v.strip_prefix('/') { + return !rest.starts_with('/'); + } + // Otherwise must be an absolute http(s) URL. Lowercase the scheme + // prefix for the comparison only — the rest of the URL is left as-is + // because path/query case can be semantically meaningful. + let lower = v.to_ascii_lowercase(); + lower.starts_with("https://") || lower.starts_with("http://") +} + +/// Return `true` if `value` is a syntactically-safe CSS color literal. +/// +/// Accepts a conservative subset of CSS color syntax: +/// +/// * Hex literals: `#rgb`, `#rgba`, `#rrggbb`, `#rrggbbaa`. +/// * Functional notation: `rgb(...)`, `rgba(...)`, `hsl(...)`, `hsla(...)`, +/// `hwb(...)`, `lab(...)`, `lch(...)`, `oklab(...)`, `oklch(...)`, +/// `color(...)`. +/// * CSS named colors (alphabetic identifiers only). +/// +/// Anything containing characters that could break out of a CSS property +/// value (`;`, `{`, `}`, `<`, `>`, backslash, newline, quotes) or the +/// `url(` prefix is rejected regardless of surface syntax. The primary +/// goal is to keep attacker-controlled values from escaping the +/// `:root { … }` block or the enclosing `<style>` tag — strict CSS Color +/// Module conformance is *not* a goal, so this will reject some valid but +/// unusual inputs (e.g. `color-mix(...)`) by design. +pub(crate) fn is_safe_css_color(value: &str) -> bool { + let v = value.trim(); + if v.is_empty() || v.len() > 128 { + return false; + } + // Reject any character that could terminate the declaration, close the + // surrounding block, break out of the `<style>` tag, or start a CSS + // comment (`*` handles both `/*` and `*/` because both require the + // asterisk; the bare `/` used in `rgb(0 0 0 / 50%)` stays legal). + if v.bytes().any(|b| { + matches!( + b, + b';' | b'{' | b'}' | b'<' | b'>' | b'"' | b'\'' | b'\\' | b'*' | b'\n' | b'\r' | b'\t' + ) + }) { + return false; + } + let lower = v.to_ascii_lowercase(); + // `url(...)` references — even inside function args — can point at + // arbitrary origins and leak request metadata. Never allow them in a + // branding color value. + if lower.contains("url(") { + return false; + } + // Hex literal: `#` followed by 3/4/6/8 hex digits. + if let Some(hex) = v.strip_prefix('#') { + let len_ok = matches!(hex.len(), 3 | 4 | 6 | 8); + let chars_ok = hex.chars().all(|c| c.is_ascii_hexdigit()); + return len_ok && chars_ok; + } + // Functional notation: `ident(...)` where `ident` is a recognized + // color function and the body contains only digits, letters, spaces, + // commas, dots, percent signs, and parentheses. The outer parens must + // balance and the value must end with `)`. + if let Some(open) = v.find('(') { + let ident = &lower[..open]; + let func_ok = matches!( + ident, + "rgb" | "rgba" | "hsl" | "hsla" | "hwb" | "lab" | "lch" | "oklab" | "oklch" | "color" + ); + if !func_ok { + return false; + } + if !v.ends_with(')') { + return false; + } + let body = &v[open + 1..v.len() - 1]; + let body_ok = body.chars().all(|c| { + c.is_ascii_alphanumeric() || matches!(c, ' ' | ',' | '.' | '%' | '+' | '-' | '/') + }); + // `/` is allowed inside functional color syntax (e.g. + // `rgb(0 0 0 / 50%)`), but we already rejected the outer `url(` + // form above and the enclosing parens are required — a bare `/` + // inside the function body cannot escape the declaration. + return body_ok; + } + // Named color or CSS-wide keyword: alphabetic identifier only. + v.chars().all(|c| c.is_ascii_alphabetic()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_layout_config_default_is_empty() { + let config = LayoutConfig::default(); + assert!(config.branding.title.is_none()); + assert!(config.tabs.order.is_none()); + assert!(config.widgets.is_empty()); + } + + #[test] + fn test_layout_config_roundtrip() { + let json = serde_json::json!({ + "branding": { "title": "Acme AI", "colors": { "primary": "#0066cc" } }, + "tabs": { "order": ["chat", "memory"], "hidden": ["routines"] }, + "widgets": { "dashboard": { "enabled": true, "config": { "refresh": 30 } } } + }); + let config: LayoutConfig = serde_json::from_value(json).unwrap(); + assert_eq!(config.branding.title.as_deref(), Some("Acme AI")); + assert_eq!(config.tabs.hidden.as_ref().map(|h| h.len()), Some(1)); + assert!(config.widgets.get("dashboard").is_some_and(|w| w.enabled)); + } + + #[test] + fn test_branding_css_vars_empty() { + let branding = BrandingConfig::default(); + assert!(branding.to_css_vars().is_empty()); + } + + #[test] + fn test_branding_css_vars_with_colors() { + let branding = BrandingConfig { + colors: Some(BrandingColors { + primary: Some("#0066cc".to_string()), + accent: Some("#ff6b00".to_string()), + }), + ..Default::default() + }; + let css = branding.to_css_vars(); + assert!(css.contains("--color-primary: #0066cc;")); + assert!(css.contains("--color-accent: #ff6b00;")); + } + + #[test] + fn test_widget_instance_enabled_defaults_to_true() { + // A layout entry that customizes config but omits `enabled` must + // NOT silently disable the widget — that was the old bug. + let json = serde_json::json!({ "config": { "refresh": 30 } }); + let cfg: WidgetInstanceConfig = serde_json::from_value(json).unwrap(); + assert!(cfg.enabled, "enabled should default to true"); + } + + #[test] + fn test_widget_instance_default_impl_is_enabled() { + // The programmatic default must match the deserialized default. + assert!(WidgetInstanceConfig::default().enabled); + } + + #[test] + fn test_widget_instance_explicit_false_respected() { + let json = serde_json::json!({ "enabled": false }); + let cfg: WidgetInstanceConfig = serde_json::from_value(json).unwrap(); + assert!(!cfg.enabled); + } + + #[test] + fn test_partial_deserialization() { + let json = serde_json::json!({"branding": {"title": "Test"}}); + let config: LayoutConfig = serde_json::from_value(json).unwrap(); + assert_eq!(config.branding.title.as_deref(), Some("Test")); + assert!(config.chat.suggestions.is_none()); + } + + #[test] + fn test_is_safe_css_color_accepts_common_forms() { + // Hex literals of every supported length. + assert!(is_safe_css_color("#fff")); + assert!(is_safe_css_color("#fff0")); + assert!(is_safe_css_color("#0066cc")); + assert!(is_safe_css_color("#0066ccaa")); + // Functional notation, including modern `rgb(... / alpha)` syntax. + assert!(is_safe_css_color("rgb(0, 0, 0)")); + assert!(is_safe_css_color("rgba(10, 20, 30, 0.5)")); + assert!(is_safe_css_color("rgb(0 0 0 / 50%)")); + assert!(is_safe_css_color("hsl(200, 50%, 50%)")); + assert!(is_safe_css_color("oklch(0.7 0.15 200)")); + // Named colors / keywords. + assert!(is_safe_css_color("red")); + assert!(is_safe_css_color("transparent")); + // Leading/trailing whitespace is tolerated. + assert!(is_safe_css_color(" #fff ")); + } + + #[test] + fn test_is_safe_css_color_rejects_injection_vectors() { + // Declaration termination would let an attacker add a new + // declaration or close the `:root {}` block. + assert!(!is_safe_css_color("red;")); + assert!(!is_safe_css_color("red; } .chat-input { background: red }")); + // `<style>` tag breakout. + assert!(!is_safe_css_color("red</style><script>alert(1)</script>")); + assert!(!is_safe_css_color("#fff</STYLE>")); + // `url(...)` can pull from arbitrary origins. + assert!(!is_safe_css_color("url(https://attacker.example/leak)")); + assert!(!is_safe_css_color("rgb(url(x), 0, 0)")); + // CSS comments could hide payload from casual readers. + assert!(!is_safe_css_color("red /* ok */ ")); + assert!(!is_safe_css_color("red*/")); + // Quotes / backslash / newline are never legal in a color value. + assert!(!is_safe_css_color("\"#fff\"")); + assert!(!is_safe_css_color("#fff\\")); + assert!(!is_safe_css_color("#fff\nbad")); + // Unknown functions are rejected even with balanced parens. + assert!(!is_safe_css_color("expression(1)")); + // Empty or absurdly long values are rejected. + assert!(!is_safe_css_color("")); + assert!(!is_safe_css_color(" ")); + assert!(!is_safe_css_color(&"#".repeat(200))); + } + + #[test] + fn test_is_safe_widget_id_accepts_existing_fixtures() { + // Every widget id used in this PR's test fixtures and the + // FRONTEND.md examples must remain valid — narrowing the regex + // after these have shipped would be a breaking change. + for id in [ + "skills-viewer", + "dashboard", + "dashboard_v2", + "widget-1", + "a.b.c", + "evil", // hostile-payload fixtures pick valid ids on purpose + "evil-css", + "styled", + "empty", + "real-id", + "spoofed-id", + "x", + "0", + "abc123", + "a", + ] { + assert!( + is_safe_widget_id(id), + "fixture widget id {id:?} must remain valid" + ); + } + } + + #[test] + fn test_is_safe_widget_id_rejects_injection_payloads() { + // CSS attribute-selector breakout — the paranoid review's P-W4 + // example. A `"` or `]` would close the `[data-widget="…"]` + // prefix in `scope_css` and let the rest of the id inject + // arbitrary CSS rules. + assert!(!is_safe_widget_id("x\"],.evil{color:red}[x")); + assert!(!is_safe_widget_id("a]")); + assert!(!is_safe_widget_id("a\"")); + // HTML attribute breakout shapes (escape_html_attr would catch + // them, but we want defense in depth at the type level too). + assert!(!is_safe_widget_id("a><script>alert(1)</script>")); + assert!(!is_safe_widget_id("a onerror=alert(1)")); + // Path traversal / separators (already caught by + // `is_safe_segment` in handlers/frontend.rs, but the two checks + // should agree on what's safe). + assert!(!is_safe_widget_id("..")); + assert!(!is_safe_widget_id("a/b")); + assert!(!is_safe_widget_id("a\\b")); + assert!(!is_safe_widget_id("a\0b")); + // Whitespace, control chars, non-ASCII. + assert!(!is_safe_widget_id("a b")); + assert!(!is_safe_widget_id("a\nb")); + assert!(!is_safe_widget_id("日本語")); + // Leading non-alphanumeric — id can't look like a flag, hidden + // file, or separator fragment. + assert!(!is_safe_widget_id("-foo")); + assert!(!is_safe_widget_id(".foo")); + assert!(!is_safe_widget_id("_foo")); + // Empty / overlong. + assert!(!is_safe_widget_id("")); + assert!(!is_safe_widget_id(&"a".repeat(65))); + // 64 chars exactly is the limit. + assert!(is_safe_widget_id(&"a".repeat(64))); + } + + #[test] + fn test_is_safe_url_accepts_common_forms() { + // Absolute HTTPS — the default operator path. + assert!(is_safe_url("https://example.com/logo.png")); + assert!(is_safe_url("https://cdn.example.com/path/to/icon.svg?v=2")); + // Absolute HTTP — intranet/dev usability. The gateway enforces + // TLS at the network layer where appropriate, so blocking plain + // HTTP here would be punishing for development setups. + assert!(is_safe_url("http://intranet.local/x.png")); + // Site-relative — for assets served by the gateway itself. + assert!(is_safe_url("/static/logo.png")); + assert!(is_safe_url("/foo/bar/baz.svg")); + // Tolerated whitespace from sloppy hand-edits. + assert!(is_safe_url(" https://example.com/logo.png ")); + } + + #[test] + fn test_is_safe_url_rejects_injection_vectors() { + // The classic `<img src=javascript:>` vector. Case-insensitive + // check covers `JavaScript:`, `JAVASCRIPT:`, etc. + assert!(!is_safe_url("javascript:alert(1)")); + assert!(!is_safe_url("JavaScript:alert(1)")); + assert!(!is_safe_url("JAVASCRIPT:alert(1)")); + // `data:` is the tracking-pixel / payload-stash vector. + assert!(!is_safe_url("data:text/html,<script>alert(1)</script>")); + assert!(!is_safe_url("data:image/svg+xml;base64,PHN2Zy8+")); + // Other historically-abused schemes. + assert!(!is_safe_url("vbscript:msgbox(1)")); + assert!(!is_safe_url("file:///etc/passwd")); + assert!(!is_safe_url("blob:https://attacker.example/x")); + // Protocol-relative URLs are scheme-flippable in the browser + // URL parser and have historically been a CSP-bypass source. + assert!(!is_safe_url("//attacker.example/logo.png")); + // HTML-attribute breakout vectors. + assert!(!is_safe_url( + "https://x.example/\"><script>alert(1)</script>" + )); + assert!(!is_safe_url("https://x.example/<img>")); + assert!(!is_safe_url("https://x.example/'onerror='alert(1)")); + assert!(!is_safe_url("https://x.example/`backtick`")); + // Control characters that could be smuggled through copy-paste. + assert!(!is_safe_url("https://x.example/\nhost")); + assert!(!is_safe_url("https://x.example/\rhost")); + assert!(!is_safe_url("https://x.example/\tpath")); + assert!(!is_safe_url("https://x.example/\0null")); + // Empty / whitespace-only. + assert!(!is_safe_url("")); + assert!(!is_safe_url(" ")); + // Length cap. 2049 is one byte over the 2048-byte limit. + // `"https://example.com/"` is 20 chars, so 2029 trailing chars + // brings the total to 2049 — one over. + let too_long = format!("https://example.com/{}", "a".repeat(2029)); + assert_eq!(too_long.len(), 2049); + assert!(!is_safe_url(&too_long)); + // And exactly 2048 must still pass. + let at_limit = format!("https://example.com/{}", "a".repeat(2028)); + assert_eq!(at_limit.len(), 2048); + assert!(is_safe_url(&at_limit)); + // The length cap counts the RAW input, not the trimmed view, so a + // pathological short URL padded with leading/trailing whitespace + // up past the cap is still rejected. Without the raw-length check + // the trim() would collapse this to a 24-char URL and silently + // pass — defeating the exfil-shape guard the cap exists for. + let padded = format!(" https://example.com/x{} ", " ".repeat(2030)); + assert!(padded.len() > 2048); + assert!( + !is_safe_url(&padded), + "raw length must be checked before trim" + ); + // No scheme at all (not relative either). + assert!(!is_safe_url("example.com/logo.png")); + // Single `/` is technically a valid root path, but the + // contract is "site-relative path"; bare `/` is fine. + assert!(is_safe_url("/")); + } + + #[test] + fn test_branding_safe_logo_url_filters_invalid() { + // safe_logo_url is the contract any future consumer must use. + // It must return None when the underlying field is missing, + // empty, whitespace-only, or a hostile scheme — and the original + // string when it's a legal HTTPS / HTTP / site-relative URL. + let safe = BrandingConfig { + logo_url: Some("https://example.com/logo.png".to_string()), + ..Default::default() + }; + assert_eq!(safe.safe_logo_url(), Some("https://example.com/logo.png")); + + let hostile = BrandingConfig { + logo_url: Some("javascript:alert(1)".to_string()), + ..Default::default() + }; + assert!( + hostile.safe_logo_url().is_none(), + "javascript: scheme must be dropped by safe_logo_url" + ); + + let relative = BrandingConfig { + logo_url: Some("/static/logo.png".to_string()), + ..Default::default() + }; + assert_eq!(relative.safe_logo_url(), Some("/static/logo.png")); + + let absent = BrandingConfig { + logo_url: None, + ..Default::default() + }; + assert!(absent.safe_logo_url().is_none()); + } + + #[test] + fn test_branding_serialize_drops_hostile_urls() { + // The wire-format leg of the URL validation. The Rust field is + // `pub(crate)` so external code must use the safe getters, but a + // future *intra-crate* caller could still write a hostile value + // into the field directly. The custom serializer ensures that + // hostile value never reaches the JS side via + // `window.__IRONCLAW_LAYOUT__` or `GET /api/frontend/layout`. + let hostile = BrandingConfig { + title: Some("Acme".to_string()), + logo_url: Some("javascript:alert(1)".to_string()), + favicon_url: Some("data:text/html,<script>alert(1)</script>".to_string()), + ..Default::default() + }; + let json = serde_json::to_string(&hostile).expect("serialize"); + + // Title survives — it's a separate field with its own escape + // path (HTML-escaped at injection time). + assert!(json.contains("\"title\":\"Acme\"")); + + // Hostile URL fields must NOT appear in the JSON output. The + // skip_serializing_if + custom serializer combo means they're + // omitted entirely (not present as `null`). + assert!( + !json.contains("logo_url"), + "logo_url with javascript: scheme must be dropped from JSON: {json}" + ); + assert!( + !json.contains("favicon_url"), + "favicon_url with data: scheme must be dropped from JSON: {json}" + ); + assert!( + !json.contains("javascript:"), + "javascript: payload must not appear anywhere in serialized output: {json}" + ); + assert!( + !json.contains("data:text"), + "data: payload must not appear anywhere in serialized output: {json}" + ); + } + + #[test] + fn test_branding_serialize_preserves_safe_urls() { + // Safe URLs must round-trip through the serializer unchanged so + // legitimate operator branding still reaches the JS side. + let safe = BrandingConfig { + logo_url: Some("https://example.com/logo.png".to_string()), + favicon_url: Some("/favicon.ico".to_string()), + ..Default::default() + }; + let json = serde_json::to_string(&safe).expect("serialize"); + assert!(json.contains("\"logo_url\":\"https://example.com/logo.png\"")); + assert!(json.contains("\"favicon_url\":\"/favicon.ico\"")); + } + + #[test] + fn test_branding_safe_favicon_url_filters_invalid() { + // Same contract as safe_logo_url; covers the parallel field so a + // future consumer can never accidentally route favicon through a + // bypass while logo is correctly validated. + let safe = BrandingConfig { + favicon_url: Some("/favicon.ico".to_string()), + ..Default::default() + }; + assert_eq!(safe.safe_favicon_url(), Some("/favicon.ico")); + + let hostile = BrandingConfig { + favicon_url: Some("data:image/x-icon;base64,AA==".to_string()), + ..Default::default() + }; + assert!( + hostile.safe_favicon_url().is_none(), + "data: scheme must be dropped by safe_favicon_url" + ); + } + + #[test] + fn test_chat_upgrade_inline_json_defaults_to_none() { + // The opt-in flag must default to `None` (== not set, treated as + // off) so an existing layout.json without the field doesn't + // suddenly start rewriting prose into JSON cards after upgrade. + let cfg: ChatConfig = serde_json::from_str("{}").unwrap(); + assert!(cfg.upgrade_inline_json.is_none()); + } + + #[test] + fn test_chat_upgrade_inline_json_roundtrips_explicit_true() { + let cfg: ChatConfig = serde_json::from_str(r#"{"upgrade_inline_json": true}"#).unwrap(); + assert_eq!(cfg.upgrade_inline_json, Some(true)); + // Round-trip through serialize so the JS-visible JSON shape is + // pinned: explicit `true` survives, default `None` is omitted. + let json = serde_json::to_string(&cfg).unwrap(); + assert!(json.contains("\"upgrade_inline_json\":true")); + let omitted: ChatConfig = ChatConfig::default(); + let json = serde_json::to_string(&omitted).unwrap(); + assert!(!json.contains("upgrade_inline_json")); + } + + #[test] + fn test_branding_css_vars_drops_unsafe_colors() { + // A hostile `layout.json` must not be able to slip a `;`-terminated + // or tag-breakout color past `to_css_vars`. Both fields are set; + // only the safe one should appear. + let branding = BrandingConfig { + colors: Some(BrandingColors { + primary: Some("red; } .chat-input { background: red".to_string()), + accent: Some("#ff6b00".to_string()), + }), + ..Default::default() + }; + let css = branding.to_css_vars(); + assert!( + !css.contains("chat-input"), + "primary injection leaked: {css}" + ); + assert!(!css.contains("--color-primary"), "primary must be dropped"); + assert!(css.contains("--color-accent: #ff6b00;")); + } +} diff --git a/crates/ironclaw_gateway/src/lib.rs b/crates/ironclaw_gateway/src/lib.rs new file mode 100644 index 00000000000..f8836819cd7 --- /dev/null +++ b/crates/ironclaw_gateway/src/lib.rs @@ -0,0 +1,38 @@ +//! IronClaw Gateway — frontend assets, layout configuration, and widget +//! extension system. +//! +//! This crate owns the complete frontend served by the IronClaw web gateway: +//! +//! - **Embedded assets** (`assets` module): HTML, JS, CSS, i18n files compiled +//! into the binary for zero-dependency serving. +//! - **Layout configuration** (`layout` module): Branding, tab order, feature +//! flags — customizable per-tenant via workspace. +//! - **Widget system** (`widget` module): Self-contained frontend components +//! that plug into named slots in the UI. +//! - **Bundle assembly** (`bundle` module): Combines base assets with workspace +//! customizations into the final served HTML. + +pub mod assets; +mod bundle; +mod layout; +mod widget; + +pub use bundle::{FrontendBundle, NONCE_PLACEHOLDER, ResolvedWidget, assemble_index}; +pub use layout::{ + BrandingColors, BrandingConfig, ChatConfig, LayoutConfig, TabConfig, WidgetInstanceConfig, + is_safe_widget_id, +}; +pub use widget::{WidgetManifest, WidgetSlot, scope_css}; + +/// Errors from frontend operations. +#[derive(Debug, thiserror::Error)] +pub enum FrontendError { + #[error("Layout configuration is invalid: {reason}")] + InvalidLayout { reason: String }, + + #[error("Widget '{id}' not found")] + WidgetNotFound { id: String }, + + #[error("Widget manifest is invalid: {reason}")] + InvalidManifest { reason: String }, +} diff --git a/crates/ironclaw_gateway/src/widget.rs b/crates/ironclaw_gateway/src/widget.rs new file mode 100644 index 00000000000..669ba8942d9 --- /dev/null +++ b/crates/ironclaw_gateway/src/widget.rs @@ -0,0 +1,387 @@ +//! Widget system types and utilities. +//! +//! Widgets are self-contained frontend components that plug into named +//! [`WidgetSlot`]s in the UI. Each widget has a manifest (`manifest.json`) +//! and implementation files (`index.js`, optional `style.css`). + +use serde::{Deserialize, Serialize}; + +/// Widget manifest — metadata about a widget component. +/// +/// Stored as `.system/gateway/widgets/{id}/manifest.json` in the workspace. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct WidgetManifest { + /// Unique widget identifier (must be a valid HTML attribute value). + pub id: String, + + /// Human-readable widget name. + pub name: String, + + /// Where this widget is rendered in the UI. + pub slot: WidgetSlot, + + /// Optional icon identifier (CSS class or emoji). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub icon: Option<String>, + + /// Positioning hint (e.g., `"after:memory"`, `"before:jobs"`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub position: Option<String>, +} + +/// Named insertion points in the UI where widgets can be rendered. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum WidgetSlot { + /// Full tab panel (adds a new tab to the tab bar). + Tab, + /// Banner area above the chat message list. + ChatHeader, + /// Area below the chat input. + ChatFooter, + /// Extra action buttons next to the send button. + ChatActions, + /// Right sidebar panel. + Sidebar, + /// Left side of the status bar. + StatusLeft, + /// Right side of the status bar. + StatusRight, + /// Additional section in the Settings tab. + SettingsSection, + /// Custom inline renderer for structured data in chat messages. + /// Registered via `IronClaw.registerChatRenderer()` on the browser side. + ChatRenderer, +} + +/// Prefix every CSS selector with `[data-widget="{widget_id}"]` for style isolation. +/// +/// This prevents widget styles from bleeding into the main app or other widgets. +/// The widget container element gets `data-widget="{id}"` set by the runtime. +/// +/// The parser tracks brace depth and handles nested grouping at-rules like +/// `@media`, `@supports`, `@container`, `@layer`, `@document`, and `@scope`: +/// selectors inside those blocks are scoped the same way as top-level selectors. +/// Other at-rules (`@keyframes`, `@font-face`, `@page`, etc.) are passed through +/// verbatim — their bodies contain declarations or keyframe selectors that +/// must not be prefixed with `[data-widget=…]`. +/// +/// This is a brace-aware text transform, not a real CSS parser: it does not +/// understand CSS strings, comments, or the CSS Nesting spec. Widget CSS +/// should avoid putting `{`/`}` inside string literals or comments. +/// +/// # Example +/// +/// ``` +/// use ironclaw_gateway::scope_css; +/// +/// let scoped = scope_css(".title { color: red; }", "my-widget"); +/// assert!(scoped.contains("[data-widget=\"my-widget\"] .title")); +/// ``` +pub fn scope_css(css: &str, widget_id: &str) -> String { + let prefix = format!("[data-widget=\"{}\"]", widget_id); + let mut result = String::with_capacity(css.len() + css.len() / 4); + + // Stack entry kind — `true` means the enclosing block is a rule list + // (top level, inside `@media`, etc.), so the next `{` opens a rule whose + // selector should be scoped. `false` means the enclosing block holds + // declarations (or opaque at-rule content like `@keyframes`), so we copy + // everything verbatim while still balancing braces. + let mut stack: Vec<bool> = Vec::new(); + let mut current_selector = String::new(); + + for ch in css.chars() { + // Top-of-stack — default to "rule list" when empty (top level). + let in_rule_list = stack.last().copied().unwrap_or(true); + + if !in_rule_list { + // Verbatim mode: copy chars through but track nesting so the + // matching `}` pops the right frame. This keeps `@keyframes` + // and other opaque at-rules intact. + match ch { + '{' => { + stack.push(false); + result.push('{'); + } + '}' => { + stack.pop(); + result.push('}'); + } + _ => result.push(ch), + } + continue; + } + + match ch { + '{' => { + let selector = current_selector.trim(); + if is_grouping_atrule(selector) { + // Emit the at-rule header as-is; its body holds more rules + // (selectors inside will be scoped on the next iteration). + result.push_str(selector); + result.push_str(" {"); + stack.push(true); + } else { + // Regular rule — scope each comma-separated selector. A + // non-grouping at-rule (like `@keyframes`) is passed through + // and its body is treated as opaque. + let parts: Vec<String> = selector + .split(',') + .map(str::trim) + .filter(|s| !s.is_empty()) + .map(|s| { + if s.starts_with('@') { + s.to_string() + } else { + format!("{} {}", prefix, s) + } + }) + .collect(); + result.push_str(&parts.join(", ")); + result.push_str(" {"); + stack.push(false); + } + current_selector.clear(); + } + '}' => { + // Close an enclosing rule list (e.g., the outer `}` of `@media`). + // A stray `}` at the very top level is malformed but we pass + // it through rather than dropping it. + stack.pop(); + result.push('}'); + current_selector.clear(); + } + _ => current_selector.push(ch), + } + } + + // Any trailing unterminated content (malformed CSS) is passed through. + if !current_selector.trim().is_empty() { + result.push_str(¤t_selector); + } + + result +} + +/// `true` if the given CSS fragment is a grouping at-rule whose body contains +/// more rules (not declarations). Selectors inside these at-rules should be +/// scoped recursively. +fn is_grouping_atrule(selector: &str) -> bool { + let s = selector.trim_start(); + if !s.starts_with('@') { + return false; + } + // The at-rule name is everything after `@` up to the first whitespace, + // `(`, or `{`. Compare case-insensitively. + let name: String = s[1..] + .chars() + .take_while(|c| !c.is_whitespace() && *c != '(' && *c != '{') + .collect::<String>() + .to_ascii_lowercase(); + matches!( + name.as_str(), + "media" | "supports" | "container" | "layer" | "document" | "scope" + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_widget_manifest_roundtrip() { + let json = serde_json::json!({ + "id": "dashboard", + "name": "Analytics Dashboard", + "slot": "tab", + "icon": "chart-bar", + "position": "after:memory" + }); + let manifest: WidgetManifest = serde_json::from_value(json).unwrap(); + assert_eq!(manifest.id, "dashboard"); + assert_eq!(manifest.slot, WidgetSlot::Tab); + assert_eq!(manifest.icon.as_deref(), Some("chart-bar")); + } + + #[test] + fn test_widget_slot_serialization() { + assert_eq!( + serde_json::to_string(&WidgetSlot::ChatHeader).unwrap(), + "\"chat_header\"" + ); + assert_eq!( + serde_json::to_string(&WidgetSlot::SettingsSection).unwrap(), + "\"settings_section\"" + ); + } + + #[test] + fn test_scope_css_basic() { + let input = ".title { color: red; }"; + let result = scope_css(input, "my-widget"); + assert!(result.contains("[data-widget=\"my-widget\"] .title")); + assert!(result.contains("color: red;")); + } + + #[test] + fn test_scope_css_multiple_selectors() { + let input = ".a, .b { margin: 0; }"; + let result = scope_css(input, "w"); + assert!(result.contains("[data-widget=\"w\"] .a")); + assert!(result.contains("[data-widget=\"w\"] .b")); + } + + #[test] + fn test_scope_css_multiple_rules() { + let input = ".a { color: red; } .b { color: blue; }"; + let result = scope_css(input, "w"); + assert!(result.contains("[data-widget=\"w\"] .a")); + assert!(result.contains("[data-widget=\"w\"] .b")); + } + + #[test] + fn test_scope_css_empty() { + assert_eq!(scope_css("", "w"), ""); + } + + /// Count `{` / `}` in a string and return `(open, close)`. A well-formed + /// CSS output must have equal counts. + fn brace_counts(s: &str) -> (usize, usize) { + (s.matches('{').count(), s.matches('}').count()) + } + + #[test] + fn test_scope_css_at_rule_not_prefixed() { + // The @media rule itself should not be prefixed. + let input = "@media (max-width: 768px) { .mobile { display: block; } }"; + let result = scope_css(input, "w"); + assert!(!result.contains("[data-widget=\"w\"] @media")); + } + + #[test] + fn test_scope_css_media_query_inner_selector_scoped() { + // Selectors nested inside @media must still be scoped to the widget. + let input = "@media (max-width: 768px) { .mobile { display: block; } }"; + let result = scope_css(input, "w"); + assert!( + result.contains("[data-widget=\"w\"] .mobile"), + "expected inner .mobile to be scoped, got: {result}" + ); + // Declarations must be preserved. + assert!(result.contains("display: block;")); + // Braces must balance — this is the regression check for the old + // single-bool parser that dropped the outer `}`. + let (open, close) = brace_counts(&result); + assert_eq!(open, close, "unbalanced braces in: {result}"); + assert_eq!(open, 2); + } + + #[test] + fn test_scope_css_nested_supports_and_media() { + // @supports wrapping @media wrapping a rule — three levels of nesting. + let input = + "@supports (display: grid) { @media (min-width: 600px) { .grid { display: grid; } } }"; + let result = scope_css(input, "w"); + assert!(result.contains("[data-widget=\"w\"] .grid")); + let (open, close) = brace_counts(&result); + assert_eq!(open, close, "unbalanced braces in: {result}"); + assert_eq!(open, 3); + } + + #[test] + fn test_scope_css_keyframes_passthrough() { + // @keyframes bodies hold keyframe-selectors (0%, 100%), NOT element + // selectors — they must not be prefixed with [data-widget=…]. And + // the nested { } inside must be correctly balanced. + let input = "@keyframes fade { 0% { opacity: 0; } 100% { opacity: 1; } }"; + let result = scope_css(input, "w"); + assert!(!result.contains("[data-widget=\"w\"] 0%")); + assert!(!result.contains("[data-widget=\"w\"] 100%")); + assert!(result.contains("@keyframes fade")); + assert!(result.contains("opacity: 0;")); + assert!(result.contains("opacity: 1;")); + let (open, close) = brace_counts(&result); + assert_eq!(open, close, "unbalanced braces in: {result}"); + } + + #[test] + fn test_scope_css_sibling_rules_inside_media() { + // Two sibling rules inside a single @media block. + let input = "@media (max-width: 768px) { .a { color: red; } .b { color: blue; } }"; + let result = scope_css(input, "w"); + assert!(result.contains("[data-widget=\"w\"] .a")); + assert!(result.contains("[data-widget=\"w\"] .b")); + let (open, close) = brace_counts(&result); + assert_eq!(open, close); + } + + #[test] + fn test_scope_css_balanced_after_complex_input() { + // Mix of top-level rules, @media, and @keyframes. + let input = " + .header { color: red; } + @media (max-width: 768px) { + .header { font-size: 14px; } + .nav, .footer { padding: 0; } + } + @keyframes spin { from { transform: rotate(0); } to { transform: rotate(360deg); } } + "; + let result = scope_css(input, "w"); + let (open, close) = brace_counts(&result); + assert_eq!(open, close, "unbalanced braces in complex input: {result}"); + assert!(result.contains("[data-widget=\"w\"] .header")); + assert!(result.contains("[data-widget=\"w\"] .nav")); + assert!(result.contains("[data-widget=\"w\"] .footer")); + assert!(!result.contains("[data-widget=\"w\"] from")); + assert!(!result.contains("[data-widget=\"w\"] to")); + } + + #[test] + fn test_scope_css_preserves_declarations() { + let input = ".box { padding: 10px; margin: 5px; }"; + let result = scope_css(input, "w"); + assert!(result.contains("padding: 10px;")); + assert!(result.contains("margin: 5px;")); + } + + #[test] + fn test_scope_css_widget_id_with_special_chars() { + let result = scope_css(".x { color: red; }", "my-widget_v2"); + assert!(result.contains("[data-widget=\"my-widget_v2\"] .x")); + } + + #[test] + fn test_widget_slot_all_variants_serialize() { + // Ensure all slot variants round-trip through serde + let slots = vec![ + WidgetSlot::Tab, + WidgetSlot::ChatHeader, + WidgetSlot::ChatFooter, + WidgetSlot::ChatActions, + WidgetSlot::Sidebar, + WidgetSlot::StatusLeft, + WidgetSlot::StatusRight, + WidgetSlot::SettingsSection, + WidgetSlot::ChatRenderer, + ]; + for slot in slots { + let json = serde_json::to_string(&slot).unwrap(); + let back: WidgetSlot = serde_json::from_str(&json).unwrap(); + assert_eq!(slot, back); + } + } + + #[test] + fn test_widget_manifest_minimal() { + // Manifest with only required fields + let json = serde_json::json!({ + "id": "test", + "name": "Test Widget", + "slot": "tab" + }); + let manifest: WidgetManifest = serde_json::from_value(json).unwrap(); + assert_eq!(manifest.id, "test"); + assert!(manifest.icon.is_none()); + assert!(manifest.position.is_none()); + } +} diff --git a/crates/ironclaw_gateway/static/admin.css b/crates/ironclaw_gateway/static/admin.css new file mode 100644 index 00000000000..d38e81ef524 --- /dev/null +++ b/crates/ironclaw_gateway/static/admin.css @@ -0,0 +1,749 @@ +/* IronClaw Admin Panel */ + +/* Shared theme tokens are loaded from /theme.css. */ + +/* === Base Reset === */ +*, *::before, *::after { + margin: 0; + padding: 0; + box-sizing: border-box; +} + +body { + font-family: 'DM Sans', -apple-system, BlinkMacSystemFont, sans-serif; + font-size: var(--text-base); + color: var(--text); + background: var(--bg); + line-height: 1.5; + min-height: 100vh; +} + +/* === Auth Screen === */ +#auth-screen, #access-denied { + display: flex; + align-items: center; + justify-content: center; + min-height: 100vh; + padding: var(--space-4); +} + +.auth-card { + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-8); + width: 100%; + max-width: 380px; + box-shadow: var(--shadow-card); +} + +.auth-brand { + display: flex; + align-items: center; + gap: var(--space-3); + font-size: var(--text-xl); + font-weight: 600; + margin-bottom: var(--space-6); + color: var(--text); +} + +.auth-badge, .sidebar-badge { + font-size: var(--text-xs); + font-weight: 500; + background: var(--accent-subtle); + color: var(--accent); + padding: 2px 8px; + border-radius: 9999px; + vertical-align: middle; +} + +.auth-error { + background: var(--danger-subtle); + border: 1px solid var(--danger-border-subtle); + color: var(--danger); + padding: var(--space-3); + border-radius: var(--radius); + font-size: var(--text-sm); + margin-bottom: var(--space-4); +} + +.auth-form { + display: flex; + flex-direction: column; + gap: var(--space-3); +} + +.auth-form input { + padding: 10px 14px; + background: var(--bg); + border: 1px solid var(--border); + border-radius: var(--radius); + color: var(--text); + font-size: var(--text-base); + font-family: var(--font-mono); + outline: none; + transition: border-color 0.2s; +} + +.auth-form input:focus { + border-color: var(--accent); + box-shadow: 0 0 0 3px var(--focus-ring); +} + +/* === Buttons === */ +.btn-primary { + background: var(--accent); + color: var(--text-on-accent); + border: none; + padding: 10px 20px; + border-radius: var(--radius); + font-size: var(--text-base); + font-weight: 500; + cursor: pointer; + transition: opacity 0.2s; +} + +.btn-primary:hover { opacity: 0.9; } +.btn-primary:disabled { opacity: 0.5; cursor: not-allowed; } + +.btn-secondary { + background: var(--bg-tertiary); + color: var(--text); + border: 1px solid var(--border); + padding: 10px 20px; + border-radius: var(--radius); + font-size: var(--text-base); + font-weight: 500; + cursor: pointer; + transition: background 0.2s; +} + +.btn-secondary:hover { background: var(--hover-subtle); } + +.btn-small { + padding: 4px 10px; + font-size: var(--text-xs); + border-radius: 6px; + border: 1px solid var(--border); + background: var(--bg-secondary); + color: var(--text); + cursor: pointer; + transition: background 0.2s; + white-space: nowrap; +} + +.btn-small:hover { background: var(--bg-tertiary); } + +.btn-danger { + border-color: var(--danger); + color: var(--danger); +} + +.btn-danger:hover { + background: var(--danger); + color: var(--text-on-danger); +} + +/* === App Layout === */ +#app { + display: flex; + min-height: 100vh; +} + +/* === Sidebar === */ +#sidebar { + width: 220px; + min-width: 220px; + background: var(--bg-secondary); + border-right: 1px solid var(--border); + display: flex; + flex-direction: column; + padding: 0; +} + +.sidebar-brand { + display: flex; + align-items: center; + gap: var(--space-2); + padding: var(--space-4) var(--space-4); + font-size: var(--text-lg); + font-weight: 600; + border-bottom: 1px solid var(--border); + color: var(--text); +} + +.sidebar-nav { + flex: 1; + padding: var(--space-2) 0; + display: flex; + flex-direction: column; +} + +.nav-link { + display: flex; + align-items: center; + gap: var(--space-3); + padding: 10px var(--space-4); + color: var(--text-secondary); + text-decoration: none; + font-size: var(--text-sm); + font-weight: 500; + transition: color 0.2s, background 0.2s; + border: none; + background: none; + cursor: pointer; + width: 100%; + text-align: left; + font-family: inherit; +} + +.nav-link:hover { + color: var(--text); + background: var(--hover-surface); +} + +.nav-link.active { + color: var(--accent); + background: var(--accent-subtle); +} + +.nav-link-soon { + opacity: 0.5; +} + +.nav-link-soon::after { + content: "soon"; + font-size: 9px; + text-transform: uppercase; + letter-spacing: 0.5px; + background: var(--bg-tertiary); + color: var(--text-muted); + padding: 1px 6px; + border-radius: 9999px; + margin-left: auto; +} + +.sidebar-footer { + border-top: 1px solid var(--border); + padding: var(--space-2) 0; +} + +.nav-link-back { color: var(--text-muted); } +.nav-link-logout { color: var(--text-muted); } + +/* === Main Content === */ +#content { + flex: 1; + padding: var(--space-6); + overflow-y: auto; + min-height: 100vh; +} + +/* === Page Header === */ +.page-header { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: var(--space-6); + flex-wrap: wrap; + gap: var(--space-3); +} + +.page-header h1 { + font-size: var(--text-2xl); + font-weight: 600; +} + +/* === Dashboard Metric Cards === */ +.metrics-grid { + display: grid; + grid-template-columns: repeat(4, 1fr); + gap: var(--space-4); + margin-bottom: var(--space-6); +} + +.metric-card { + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-4) var(--space-6); +} + +.metric-label { + font-size: var(--text-xs); + color: var(--text-secondary); + text-transform: uppercase; + letter-spacing: 0.5px; + font-weight: 500; + margin-bottom: var(--space-1); +} + +.metric-value { + font-size: var(--text-2xl); + font-weight: 700; + color: var(--text); + font-family: var(--font-mono); +} + +.metric-value.accent { color: var(--accent); } +.metric-value.warning { color: var(--warning); } +.metric-value.danger { color: var(--danger); } + +/* === Data Tables === */ +.data-table { + width: 100%; + border-collapse: collapse; +} + +.data-table th { + padding: 10px 12px; + text-align: left; + border-bottom: 1px solid var(--border); + font-size: var(--text-xs); + font-weight: 500; + color: var(--text-secondary); + text-transform: uppercase; + letter-spacing: 0.5px; +} + +.data-table td { + padding: 10px 12px; + border-bottom: 1px solid var(--border); + font-size: var(--text-sm); + color: var(--text); +} + +.data-table tr:hover td { + background: var(--hover-surface); +} + +.data-table .mono { + font-family: var(--font-mono); + font-size: var(--text-xs); +} + +.data-table .actions { + display: flex; + gap: var(--space-1); + flex-wrap: wrap; +} + +/* === Badges === */ +.badge { + display: inline-block; + padding: 2px 10px; + border-radius: 9999px; + font-size: var(--text-xs); + font-weight: 500; + line-height: 1.5; +} + +.badge-admin { + background: var(--accent-subtle); + color: var(--accent); +} + +.badge-member { + background: var(--hover-surface); + color: var(--text-secondary); +} + +.badge-active { + background: var(--accent-subtle); + color: var(--accent); +} + +.badge-suspended { + background: var(--danger-subtle); + color: var(--danger); +} + +.badge-deactivated { + background: var(--hover-surface); + color: var(--text-muted); +} + +/* === Search & Filters === */ +.toolbar { + display: flex; + align-items: center; + gap: var(--space-3); + margin-bottom: var(--space-4); + flex-wrap: wrap; +} + +.search-input { + padding: 8px 14px; + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius); + color: var(--text); + font-size: var(--text-sm); + min-width: 240px; + outline: none; + transition: border-color 0.2s; +} + +.search-input:focus { + border-color: var(--accent); + box-shadow: 0 0 0 3px var(--focus-ring); +} + +.filter-btn { + padding: 6px 14px; + border: 1px solid var(--border); + border-radius: 9999px; + background: var(--bg-secondary); + color: var(--text-secondary); + font-size: var(--text-xs); + cursor: pointer; + transition: all 0.2s; +} + +.filter-btn:hover { border-color: var(--accent); color: var(--text); } +.filter-btn.active { background: var(--accent-subtle); border-color: var(--accent-border-subtle); color: var(--accent); } + +/* === Forms === */ +.form-card { + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-6); + margin-bottom: var(--space-6); +} + +.form-row { + display: flex; + gap: var(--space-3); + align-items: end; + flex-wrap: wrap; + margin-bottom: var(--space-3); +} + +.form-group { + display: flex; + flex-direction: column; + gap: var(--space-1); +} + +.form-group label { + font-size: var(--text-xs); + font-weight: 500; + color: var(--text-secondary); +} + +.form-group input, +.form-group select { + padding: 8px 12px; + background: var(--bg); + border: 1px solid var(--border); + border-radius: var(--radius); + color: var(--text); + font-size: var(--text-sm); + outline: none; + transition: border-color 0.2s; +} + +.form-group input:focus, +.form-group select:focus { + border-color: var(--accent); + box-shadow: 0 0 0 3px var(--focus-ring); +} + +.form-group select { + cursor: pointer; + appearance: none; + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='12' height='12' viewBox='0 0 24 24' fill='none' stroke='%23a1a1aa' stroke-width='2'%3E%3Cpolyline points='6 9 12 15 18 9'/%3E%3C/svg%3E"); + background-repeat: no-repeat; + background-position: right 10px center; + padding-right: 30px; +} + +/* === Token Banner === */ +.token-banner { + background: var(--accent-subtle); + border: 1px solid var(--accent-border-subtle); + border-radius: var(--radius); + padding: var(--space-4); + margin-bottom: var(--space-4); +} + +.token-banner p { + font-size: var(--text-sm); + margin-bottom: var(--space-2); + color: var(--text); +} + +.token-banner .token-value { + font-family: var(--font-mono); + font-size: var(--text-sm); + background: var(--bg); + padding: var(--space-2) var(--space-3); + border-radius: 6px; + word-break: break-all; + display: flex; + align-items: center; + justify-content: space-between; + gap: var(--space-2); +} + +.token-banner .token-value code { + flex: 1; + overflow: hidden; + text-overflow: ellipsis; +} + +/* === Detail Page === */ +.detail-grid { + display: grid; + grid-template-columns: 1fr 1fr; + gap: var(--space-6); + margin-bottom: var(--space-6); +} + +.detail-card { + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-6); +} + +.detail-card h2 { + font-size: var(--text-lg); + font-weight: 600; + margin-bottom: var(--space-4); + color: var(--text); +} + +.detail-row { + display: flex; + justify-content: space-between; + padding: var(--space-2) 0; + border-bottom: 1px solid var(--border); + font-size: var(--text-sm); +} + +.detail-row:last-child { border-bottom: none; } + +.detail-label { + color: var(--text-secondary); + font-weight: 500; +} + +.detail-value { + color: var(--text); + text-align: right; +} + +/* === Usage Bars === */ +.usage-bar-cell { + display: flex; + align-items: center; + gap: var(--space-2); +} + +.usage-bar { + height: 6px; + background: var(--accent); + border-radius: 3px; + min-width: 2px; +} + +/* === Period Selector === */ +.period-selector { + display: flex; + gap: 2px; + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius); + padding: 2px; +} + +.period-btn { + padding: 6px 16px; + border: none; + background: none; + color: var(--text-secondary); + font-size: var(--text-sm); + font-weight: 500; + cursor: pointer; + border-radius: 6px; + transition: all 0.2s; +} + +.period-btn:hover { color: var(--text); } + +.period-btn.active { + background: var(--accent-subtle); + color: var(--accent); +} + +/* === Modal === */ +.modal-overlay { + position: fixed; + inset: 0; + background: var(--bg-overlay); + display: flex; + align-items: center; + justify-content: center; + z-index: 1000; + padding: var(--space-4); +} + +.modal-card { + background: var(--bg-secondary); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-6); + max-width: 480px; + width: 100%; + box-shadow: var(--shadow-card); +} + +.modal-card h2 { + font-size: var(--text-lg); + font-weight: 600; + margin-bottom: var(--space-4); +} + +.modal-actions { + display: flex; + gap: var(--space-3); + justify-content: flex-end; + margin-top: var(--space-6); +} + +/* === Empty & Loading States === */ +.empty-state { + text-align: center; + padding: 60px var(--space-4); + color: var(--text-secondary); +} + +.empty-state svg { + margin-bottom: var(--space-4); + opacity: 0.3; +} + +.empty-state p { + font-size: var(--text-sm); +} + +.coming-soon { + text-align: center; + padding: 80px var(--space-4); +} + +.coming-soon h2 { + font-size: var(--text-xl); + font-weight: 600; + color: var(--text-secondary); + margin-bottom: var(--space-3); +} + +.coming-soon p { + color: var(--text-muted); + font-size: var(--text-sm); +} + +.loading { + text-align: center; + padding: 60px var(--space-4); + color: var(--text-secondary); +} + +.error-message { + background: var(--danger-subtle); + border: 1px solid var(--danger-border-subtle); + color: var(--danger); + padding: var(--space-4); + border-radius: var(--radius); + font-size: var(--text-sm); +} + +/* === Breadcrumb === */ +.breadcrumb { + display: flex; + align-items: center; + gap: var(--space-2); + font-size: var(--text-sm); + color: var(--text-secondary); + margin-bottom: var(--space-4); +} + +.breadcrumb a { + color: var(--text-secondary); + text-decoration: none; +} + +.breadcrumb a:hover { color: var(--accent); } +.breadcrumb .sep { opacity: 0.5; } + +/* === Responsive === */ +@media (max-width: 768px) { + #app { + flex-direction: column; + } + + #sidebar { + width: 100%; + min-width: 100%; + border-right: none; + border-bottom: 1px solid var(--border); + } + + .sidebar-nav { + flex-direction: row; + overflow-x: auto; + padding: 0 var(--space-2); + } + + .nav-link { + white-space: nowrap; + padding: var(--space-3) var(--space-3); + } + + .sidebar-footer { + display: flex; + border-top: none; + padding: 0 var(--space-2); + } + + .sidebar-footer .nav-link { + flex: 1; + justify-content: center; + } + + #content { + padding: var(--space-4); + min-height: auto; + } + + .metrics-grid { + grid-template-columns: repeat(2, 1fr); + } + + .detail-grid { + grid-template-columns: 1fr; + } + + .data-table { + display: block; + overflow-x: auto; + } + + .search-input { + min-width: 100%; + } + + .toolbar { + flex-direction: column; + align-items: stretch; + } +} + +@media (max-width: 480px) { + .metrics-grid { + grid-template-columns: 1fr; + } +} diff --git a/crates/ironclaw_gateway/static/admin.html b/crates/ironclaw_gateway/static/admin.html new file mode 100644 index 00000000000..6989c76a040 --- /dev/null +++ b/crates/ironclaw_gateway/static/admin.html @@ -0,0 +1,98 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover"> + <meta name="robots" content="noindex, nofollow"> + <title>IronClaw Admin + + + + + + + +

+
+
+ + + + IronClaw Admin +
+ +
+ + +
+
+
+ + + + + + + + + + diff --git a/crates/ironclaw_gateway/static/admin.js b/crates/ironclaw_gateway/static/admin.js new file mode 100644 index 00000000000..a35ac936c39 --- /dev/null +++ b/crates/ironclaw_gateway/static/admin.js @@ -0,0 +1,951 @@ +/* IronClaw Admin Panel */ + +// TODO(#1968): Inline style attributes throughout this file bypass the +// theme token system in admin.css. Migrate to CSS classes that reference +// CSS custom properties (--space-*, --text-*, --accent, etc.). + +(function () { + 'use strict'; + + // --------------------------------------------------------------------------- + // State + // --------------------------------------------------------------------------- + + var token = ''; + var oidcProxyAuth = false; + var currentProfile = null; + + // --------------------------------------------------------------------------- + // Helpers + // --------------------------------------------------------------------------- + + function escapeHtml(str) { + if (!str) return ''; + return String(str) + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); + } + + function formatNumber(n) { + if (n == null) return '0'; + return Number(n).toLocaleString(); + } + + function formatTokenCount(n) { + if (n == null || n === 0) return '0'; + if (n >= 1000000) return (n / 1000000).toFixed(1) + 'M'; + if (n >= 1000) return (n / 1000).toFixed(1) + 'K'; + return String(n); + } + + function formatCost(v) { + if (v == null) return '$0.00'; + var n = parseFloat(v); + if (isNaN(n)) return '$0.00'; + return '$' + n.toFixed(2); + } + + function formatUptime(secs) { + if (!secs) return '0s'; + var d = Math.floor(secs / 86400); + var h = Math.floor((secs % 86400) / 3600); + var m = Math.floor((secs % 3600) / 60); + if (d > 0) return d + 'd ' + h + 'h'; + if (h > 0) return h + 'h ' + m + 'm'; + return m + 'm'; + } + + function formatRelativeTime(iso) { + if (!iso) return 'Never'; + var diff = (Date.now() - new Date(iso).getTime()) / 1000; + if (diff < 0) diff = 0; + if (diff < 60) return 'Just now'; + if (diff < 3600) return Math.floor(diff / 60) + 'm ago'; + if (diff < 86400) return Math.floor(diff / 3600) + 'h ago'; + if (diff < 2592000) return Math.floor(diff / 86400) + 'd ago'; + return new Date(iso).toLocaleDateString(); + } + + function statusBadge(status) { + var cls = 'badge badge-' + escapeHtml(status || 'active'); + return '' + escapeHtml(status || 'active') + ''; + } + + function roleBadge(role) { + var cls = 'badge badge-' + escapeHtml(role || 'member'); + return '' + escapeHtml(role || 'member') + ''; + } + + function truncateId(id) { + if (!id) return ''; + return id.length > 12 ? id.slice(0, 12) + '\u2026' : id; + } + + // --------------------------------------------------------------------------- + // API + // --------------------------------------------------------------------------- + + function apiFetch(path, options) { + var opts = {}; + if (options) { + for (var k in options) { + if (Object.prototype.hasOwnProperty.call(options, k)) { + opts[k] = options[k]; + } + } + } + opts.headers = Object.assign({}, opts.headers || {}); + if (token && !oidcProxyAuth) { + opts.headers['Authorization'] = 'Bearer ' + token; + } + if (opts.body && typeof opts.body === 'object') { + opts.headers['Content-Type'] = 'application/json'; + opts.body = JSON.stringify(opts.body); + } + return fetch(path, opts).then(function (res) { + if (!res.ok) { + return res.text().then(function (body) { + var err = new Error(body || res.status + ' ' + res.statusText); + err.status = res.status; + throw err; + }); + } + if (res.status === 204) return null; + return res.json(); + }); + } + + // --------------------------------------------------------------------------- + // Auth + // --------------------------------------------------------------------------- + + function showAuth() { + document.getElementById('auth-screen').style.display = 'flex'; + document.getElementById('access-denied').style.display = 'none'; + document.getElementById('app').style.display = 'none'; + } + + function showAccessDenied() { + document.getElementById('auth-screen').style.display = 'none'; + document.getElementById('access-denied').style.display = 'flex'; + document.getElementById('app').style.display = 'none'; + } + + function showApp() { + document.getElementById('auth-screen').style.display = 'none'; + document.getElementById('access-denied').style.display = 'none'; + document.getElementById('app').style.display = 'flex'; + } + + function logout() { + token = ''; + oidcProxyAuth = false; + currentProfile = null; + sessionStorage.removeItem('ironclaw_token'); + showAuth(); + } + + function authenticate(t) { + oidcProxyAuth = false; + token = t; + return apiFetch('/api/profile').then(function (profile) { + currentProfile = profile; + if (profile.role !== 'admin') { + token = ''; + showAccessDenied(); + return false; + } + // Security note: sessionStorage is readable by any XSS payload running + // in this origin. We accept this risk because: (1) the token is session- + // scoped and cleared on tab close, (2) the CSP restricts script-src to + // 'self' only (no inline scripts), (3) migration to httpOnly cookies + // needs server-side session management (larger effort, tracked for + // follow-up). + sessionStorage.setItem('ironclaw_token', t); + showApp(); + route(); + return true; + }).catch(function (err) { + token = ''; + throw err; + }); + } + + function autoAuth() { + // Check sessionStorage + var saved = sessionStorage.getItem('ironclaw_token'); + if (saved) { + authenticate(saved).catch(function () { showAuth(); }); + return; + } + + // Check implicit auth (e.g. OIDC proxy cookie) by probing profile directly. + apiFetch('/api/profile').then(function (profile) { + oidcProxyAuth = true; + token = ''; + currentProfile = profile; + if (profile.role !== 'admin') { + showAccessDenied(); + } else { + showApp(); + route(); + } + }).catch(function () { + oidcProxyAuth = false; + showAuth(); + }); + } + + // --------------------------------------------------------------------------- + // Router + // --------------------------------------------------------------------------- + + function parseHash() { + var hash = window.location.hash || '#/'; + if (hash.charAt(0) === '#') hash = hash.slice(1); + if (!hash || hash.charAt(0) !== '/') hash = '/'; + return hash; + } + + function route() { + var path = parseHash(); + var content = document.getElementById('content'); + + // Update active nav link + var links = document.querySelectorAll('.nav-link[data-route]'); + for (var i = 0; i < links.length; i++) { + var r = links[i].getAttribute('data-route'); + var isActive = path === r || (r !== '/' && path.indexOf(r) === 0); + links[i].classList.toggle('active', isActive); + } + + // Route dispatch + if (path === '/') { + renderDashboard(content); + } else if (path === '/users') { + renderUsers(content); + } else if (path.indexOf('/users/') === 0) { + var userId = decodeURIComponent(path.slice(7)); + renderUserDetail(content, userId); + } else if (path === '/usage') { + renderUsage(content, 'day'); + } else if (path === '/workspaces') { + renderStub(content, 'Workspaces', 'Workspace management is coming soon.', '#1607'); + } else if (path === '/invitations') { + renderStub(content, 'Invitations', 'Invitation management is coming soon.', '#1608'); + } else { + content.innerHTML = '

Page not found

'; + } + } + + // --------------------------------------------------------------------------- + // Pages + // --------------------------------------------------------------------------- + + // --- Dashboard --- + + function renderDashboard(el) { + el.innerHTML = '
Loading dashboard...
'; + + Promise.all([ + apiFetch('/api/admin/usage/summary'), + apiFetch('/api/admin/users') + ]).then(function (results) { + var summary = results[0]; + var rawUsers = results[1] || {}; + var users = Array.isArray(rawUsers) ? rawUsers : (rawUsers.users || []); + + var u = summary.users || {}; + var j = summary.jobs || {}; + var usage = summary.usage_30d || {}; + + var html = '

Dashboard

'; + + // Metrics + html += '
'; + html += metricCard('Total Users', formatNumber(u.total)); + html += metricCard('Active Users', formatNumber(u.active), 'accent'); + html += metricCard('Suspended', formatNumber(u.suspended), u.suspended > 0 ? 'danger' : ''); + html += metricCard('Admins', formatNumber(u.admins)); + html += metricCard('Total Jobs', formatNumber(j.total)); + html += metricCard('30d LLM Calls', formatNumber(usage.llm_calls)); + html += metricCard('30d Cost', formatCost(usage.total_cost), 'accent'); + html += metricCard('Uptime', formatUptime(summary.uptime_seconds)); + html += '
'; + + // Recent users table + var recent = users.slice().sort(function (a, b) { + var ta = a.last_active_at || a.created_at || ''; + var tb = b.last_active_at || b.created_at || ''; + return tb.localeCompare(ta); + }).slice(0, 5); + + html += '
'; + html += '

Recent Users

'; + if (recent.length === 0) { + html += '

No users yet

'; + } else { + html += ''; + html += ''; + html += ''; + for (var i = 0; i < recent.length; i++) { + var ru = recent[i]; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + } + html += '
NameRoleStatusJobsLast Active
' + escapeHtml(ru.display_name) + '' + roleBadge(ru.role) + '' + statusBadge(ru.status) + '' + formatNumber(ru.job_count) + '' + formatRelativeTime(ru.last_active_at) + '
'; + } + html += '
'; + + el.innerHTML = html; + }).catch(function (err) { + el.innerHTML = '
Failed to load dashboard: ' + escapeHtml(err.message) + '
'; + }); + } + + function metricCard(label, value, cls) { + return '
' + + '
' + escapeHtml(label) + '
' + + '
' + escapeHtml(value) + '
' + + '
'; + } + + // --- Users List --- + + var usersCache = null; + var usersFilter = 'all'; + var usersSearch = ''; + + function renderUsers(el) { + el.innerHTML = '
Loading users...
'; + usersCache = null; + usersFilter = 'all'; + usersSearch = ''; + + apiFetch('/api/admin/users').then(function (raw) { + usersCache = Array.isArray(raw) ? raw : (raw && raw.users ? raw.users : []); + renderUsersPage(el); + }).catch(function (err) { + el.innerHTML = '
Failed to load users: ' + escapeHtml(err.message) + '
'; + }); + } + + function renderUsersPage(el) { + var users = filterUsers(usersCache || []); + + var html = '

Users

'; + html += ''; + html += '
'; + + // Create user form (hidden by default) + html += ''; + + // Token banner + html += ''; + + // Toolbar + html += '
'; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += '
'; + + // Table + if (users.length === 0) { + html += '

No users found

'; + } else { + html += ''; + html += ''; + html += ''; + html += ''; + for (var i = 0; i < users.length; i++) { + var u = users[i]; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + } + html += '
IDNameEmailRoleStatusJobsCostLast ActiveActions
' + escapeHtml(truncateId(u.id)) + '' + escapeHtml(u.display_name) + '' + escapeHtml(u.email || '') + '' + roleBadge(u.role) + '' + statusBadge(u.status) + '' + formatNumber(u.job_count) + '' + formatCost(u.total_cost) + '' + formatRelativeTime(u.last_active_at) + ''; + if (u.status === 'active') { + html += ''; + } else { + html += ''; + } + if (u.role === 'admin') { + html += ''; + } else { + html += ''; + } + html += ''; + html += '
'; + } + + el.innerHTML = html; + + // Search input handler + var searchEl = document.getElementById('users-search'); + if (searchEl) { + searchEl.addEventListener('input', function () { + usersSearch = searchEl.value; + renderUsersPage(el); + }); + searchEl.focus(); + searchEl.setSelectionRange(usersSearch.length, usersSearch.length); + } + } + + function filterUsers(users) { + var result = users; + if (usersFilter === 'active') { + result = result.filter(function (u) { return u.status === 'active'; }); + } else if (usersFilter === 'suspended') { + result = result.filter(function (u) { return u.status === 'suspended'; }); + } else if (usersFilter === 'admin') { + result = result.filter(function (u) { return u.role === 'admin'; }); + } + if (usersSearch) { + var q = usersSearch.toLowerCase(); + result = result.filter(function (u) { + return (u.display_name && u.display_name.toLowerCase().indexOf(q) >= 0) || + (u.email && u.email.toLowerCase().indexOf(q) >= 0) || + (u.id && u.id.toLowerCase().indexOf(q) >= 0); + }); + } + return result; + } + + // --- User Detail --- + + function renderUserDetail(el, userId) { + el.innerHTML = '
Loading user...
'; + + Promise.all([ + apiFetch('/api/admin/users/' + encodeURIComponent(userId)), + apiFetch('/api/admin/usage?user_id=' + encodeURIComponent(userId) + '&period=month') + ]).then(function (results) { + var user = results[0]; + var usageData = results[1]; + + var html = '
Users/' + escapeHtml(user.display_name) + '
'; + + html += '

' + escapeHtml(user.display_name) + '

'; + html += '
'; + if (user.status === 'active') { + html += ''; + } else { + html += ''; + } + html += ''; + html += ''; + html += '
'; + + // Token banner slot + html += ''; + + // Profile + Stats grid + html += '
'; + + // Profile card + html += '

Profile

'; + html += detailRowRawHtml('ID', '' + escapeHtml(user.id) + ''); + html += detailRow('Email', user.email || 'Not set'); + html += detailRowRawHtml('Role', roleBadge(user.role)); + html += detailRowRawHtml('Status', statusBadge(user.status)); + html += detailRow('Created', formatRelativeTime(user.created_at)); + html += detailRow('Last Login', formatRelativeTime(user.last_login_at)); + if (user.created_by) { + html += detailRowRawHtml('Created By', '' + escapeHtml(truncateId(user.created_by)) + ''); + } + html += '
'; + + // Stats card + html += '

Summary

'; + html += detailRow('Jobs', formatNumber(user.job_count)); + html += detailRow('Total Cost', formatCost(user.total_cost)); + html += detailRow('Last Active', formatRelativeTime(user.last_active_at)); + html += '
'; + + html += '
'; + + // Role management + html += '
'; + html += '

Role Management

'; + html += '
'; + html += '
'; + html += '
'; + html += '
'; + html += '
'; + + // Usage table + var entries = (usageData && usageData.usage) || []; + html += '
'; + html += '

Usage (Last 30 Days)

'; + if (entries.length === 0) { + html += '

No usage data

'; + } else { + html += ''; + html += ''; + html += ''; + for (var i = 0; i < entries.length; i++) { + var e = entries[i]; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + } + html += '
ModelCallsInput TokensOutput TokensCost
' + escapeHtml(e.model) + '' + formatNumber(e.call_count) + '' + formatTokenCount(e.input_tokens) + '' + formatTokenCount(e.output_tokens) + '' + formatCost(e.total_cost) + '
'; + } + html += '
'; + + el.innerHTML = html; + }).catch(function (err) { + el.innerHTML = '
Users/Error
' + + '
Failed to load user: ' + escapeHtml(err.message) + '
'; + }); + } + + function detailRow(label, value) { + return '
' + escapeHtml(label) + '' + escapeHtml(value == null ? '' : String(value)) + '
'; + } + + // SAFETY: valueHtml is injected as raw HTML — callers MUST pre-escape any + // user-supplied content via escapeHtml() to prevent XSS. Prefer detailRow() + // for plain-text values; use this variant only when the value contains + // trusted markup (badges, , etc.). + function detailRowRawHtml(label, valueHtml) { + return '
' + escapeHtml(label) + '' + valueHtml + '
'; + } + + // --- Usage --- + + function renderUsage(el, period) { + el.innerHTML = '
Loading usage data...
'; + + apiFetch('/api/admin/usage?period=' + encodeURIComponent(period)).then(function (data) { + var entries = (data && data.usage) || []; + + var html = '

Usage

'; + html += '
'; + html += ''; + html += ''; + html += ''; + html += '
'; + + if (entries.length === 0) { + html += '

No usage data for this period

'; + el.innerHTML = html; + return; + } + + // Aggregate by user + var byUser = {}; + var maxCost = 0; + for (var i = 0; i < entries.length; i++) { + var e = entries[i]; + if (!byUser[e.user_id]) { + byUser[e.user_id] = { user_id: e.user_id, calls: 0, input_tokens: 0, output_tokens: 0, cost: 0 }; + } + byUser[e.user_id].calls += e.call_count || 0; + byUser[e.user_id].input_tokens += e.input_tokens || 0; + byUser[e.user_id].output_tokens += e.output_tokens || 0; + byUser[e.user_id].cost += parseFloat(e.total_cost) || 0; + } + + var userList = Object.keys(byUser).map(function (k) { return byUser[k]; }); + userList.sort(function (a, b) { return b.cost - a.cost; }); + + for (var j = 0; j < userList.length; j++) { + if (userList[j].cost > maxCost) maxCost = userList[j].cost; + } + + // Summary row + var totalCalls = 0, totalInput = 0, totalOutput = 0, totalCostVal = 0; + for (var k = 0; k < userList.length; k++) { + totalCalls += userList[k].calls; + totalInput += userList[k].input_tokens; + totalOutput += userList[k].output_tokens; + totalCostVal += userList[k].cost; + } + + html += '
'; + html += metricCard('Total Calls', formatNumber(totalCalls)); + html += metricCard('Input Tokens', formatTokenCount(totalInput)); + html += metricCard('Output Tokens', formatTokenCount(totalOutput)); + html += metricCard('Total Cost', formatCost(totalCostVal.toFixed(2)), 'accent'); + html += '
'; + + // Per-user table + html += '
'; + html += '

Per-User Breakdown

'; + html += ''; + html += ''; + html += ''; + for (var m = 0; m < userList.length; m++) { + var uu = userList[m]; + var pct = maxCost > 0 ? (uu.cost / maxCost * 100) : 0; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + } + html += '
UserCallsInput TokensOutput TokensCost
' + escapeHtml(truncateId(uu.user_id)) + '' + formatNumber(uu.calls) + '' + formatTokenCount(uu.input_tokens) + '' + formatTokenCount(uu.output_tokens) + '' + formatCost(uu.cost.toFixed(2)) + '
'; + + // Per-model table + var byModel = {}; + for (var n = 0; n < entries.length; n++) { + var em = entries[n]; + if (!byModel[em.model]) { + byModel[em.model] = { model: em.model, calls: 0, input_tokens: 0, output_tokens: 0, cost: 0 }; + } + byModel[em.model].calls += em.call_count || 0; + byModel[em.model].input_tokens += em.input_tokens || 0; + byModel[em.model].output_tokens += em.output_tokens || 0; + byModel[em.model].cost += parseFloat(em.total_cost) || 0; + } + + var modelList = Object.keys(byModel).map(function (k) { return byModel[k]; }); + modelList.sort(function (a, b) { return b.cost - a.cost; }); + + html += '
'; + html += '

Per-Model Breakdown

'; + html += ''; + html += ''; + html += ''; + for (var p = 0; p < modelList.length; p++) { + var mm = modelList[p]; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + html += ''; + } + html += '
ModelCallsInput TokensOutput TokensCost
' + escapeHtml(mm.model) + '' + formatNumber(mm.calls) + '' + formatTokenCount(mm.input_tokens) + '' + formatTokenCount(mm.output_tokens) + '' + formatCost(mm.cost.toFixed(2)) + '
'; + + el.innerHTML = html; + }).catch(function (err) { + el.innerHTML = '
Failed to load usage: ' + escapeHtml(err.message) + '
'; + }); + } + + // --- Stub pages --- + + function renderStub(el, title, desc, issue) { + el.innerHTML = '
' + + '

' + escapeHtml(title) + '

' + + '

' + escapeHtml(desc) + '

' + + '

Tracking: ' + escapeHtml(issue) + '

' + + '
'; + } + + // --------------------------------------------------------------------------- + // Actions (event delegation) + // --------------------------------------------------------------------------- + + function handleAction(target) { + var action = target.getAttribute('data-action'); + if (!action) return; + + var id = target.getAttribute('data-id'); + var content = document.getElementById('content'); + + switch (action) { + case 'show-create-form': + var form = document.getElementById('create-user-form'); + if (form) form.style.display = 'block'; + break; + + case 'hide-create-form': + var formH = document.getElementById('create-user-form'); + if (formH) formH.style.display = 'none'; + break; + + case 'create-user': + createUser(content); + break; + + case 'suspend': + apiFetch('/api/admin/users/' + encodeURIComponent(id) + '/suspend', { method: 'POST' }) + .then(function () { refreshCurrentPage(); }) + .catch(function (err) { alert('Failed to suspend: ' + err.message); }); + break; + + case 'activate': + apiFetch('/api/admin/users/' + encodeURIComponent(id) + '/activate', { method: 'POST' }) + .then(function () { refreshCurrentPage(); }) + .catch(function (err) { alert('Failed to activate: ' + err.message); }); + break; + + case 'change-role': + var newRole = target.getAttribute('data-role'); + apiFetch('/api/admin/users/' + encodeURIComponent(id), { + method: 'PATCH', + body: { role: newRole } + }).then(function () { refreshCurrentPage(); }) + .catch(function (err) { alert('Failed to change role: ' + err.message); }); + break; + + case 'save-role': + var sel = document.getElementById('role-select'); + if (sel) { + apiFetch('/api/admin/users/' + encodeURIComponent(id), { + method: 'PATCH', + body: { role: sel.value } + }).then(function () { refreshCurrentPage(); }) + .catch(function (err) { alert('Failed to save role: ' + err.message); }); + } + break; + + case 'create-token': + var userName = target.getAttribute('data-name') || 'user'; + var tokenName = prompt('Token name for ' + userName + ':'); + if (!tokenName) return; + apiFetch('/api/tokens', { + method: 'POST', + body: { name: tokenName, user_id: id } + }).then(function (res) { + showTokenBanner(res.token || res.plaintext_token); + }).catch(function (err) { alert('Failed to create token: ' + err.message); }); + break; + + case 'delete-user': + var name = target.getAttribute('data-name') || id; + showConfirmModal( + 'Delete User', + 'Are you sure you want to delete "' + name + '"? This action cannot be undone.', + 'Delete', + function () { + apiFetch('/api/admin/users/' + encodeURIComponent(id), { method: 'DELETE' }) + .then(function () { window.location.hash = '#/users'; }) + .catch(function (err) { alert('Failed to delete: ' + err.message); }); + } + ); + break; + + case 'filter': + usersFilter = target.getAttribute('data-filter') || 'all'; + renderUsersPage(content); + break; + + case 'period': + var period = target.getAttribute('data-period') || 'day'; + renderUsage(content, period); + break; + + case 'copy-token': + var tokenVal = target.getAttribute('data-token'); + if (tokenVal && navigator.clipboard) { + navigator.clipboard.writeText(tokenVal); + target.textContent = 'Copied!'; + setTimeout(function () { target.textContent = 'Copy'; }, 2000); + } + break; + + case 'modal-close': + closeModal(); + break; + } + } + + function createUser(el) { + var nameEl = document.getElementById('new-user-name'); + var emailEl = document.getElementById('new-user-email'); + var roleEl = document.getElementById('new-user-role'); + if (!nameEl || !nameEl.value.trim()) { + alert('Display name is required'); + return; + } + var body = { display_name: nameEl.value.trim(), role: roleEl ? roleEl.value : 'member' }; + if (emailEl && emailEl.value.trim()) body.email = emailEl.value.trim(); + + apiFetch('/api/admin/users', { method: 'POST', body: body }).then(function (res) { + var formEl = document.getElementById('create-user-form'); + if (formEl) formEl.style.display = 'none'; + if (nameEl) nameEl.value = ''; + if (emailEl) emailEl.value = ''; + var createdToken = res && (res.token || res.plaintext_token); + + // Reload users list + apiFetch('/api/admin/users').then(function (raw) { + usersCache = Array.isArray(raw) ? raw : (raw && raw.users ? raw.users : []); + renderUsersPage(el); + if (createdToken) { + showTokenBanner(createdToken); + } + }); + }).catch(function (err) { + alert('Failed to create user: ' + err.message); + }); + } + + function showTokenBanner(tokenValue) { + var banner = document.getElementById('user-token-banner'); + if (!banner) return; + banner.innerHTML = '
' + + '

Token created! Copy this now — it will not be shown again.

' + + '
' + escapeHtml(tokenValue) + '' + + '
' + + '

Use this token in the admin login field.

' + + '
'; + banner.style.display = 'block'; + } + + function refreshCurrentPage() { + route(); + } + + // --------------------------------------------------------------------------- + // Modal + // --------------------------------------------------------------------------- + + /** + * Show a confirmation modal dialog. + * + * All string parameters are HTML-escaped via escapeHtml() before insertion + * into the DOM, so callers do not need to pre-sanitise user-supplied strings. + * + * @param {string} title - Dialog title (escaped before rendering). + * @param {string} message - Dialog body text (escaped before rendering). + * @param {string} confirmText - Text for the confirm button (escaped before rendering). + * @param {Function} onConfirm - Callback invoked when the user confirms. + */ + function showConfirmModal(title, message, confirmText, onConfirm) { + var overlay = document.getElementById('modal-overlay'); + var content = document.getElementById('modal-content'); + if (!overlay || !content) return; + + content.innerHTML = '

' + escapeHtml(title) + '

' + + '

' + escapeHtml(message) + '

' + + '
' + + '' + + '' + + '
'; + overlay.style.display = 'flex'; + + var confirmBtn = document.getElementById('modal-confirm'); + if (confirmBtn) { + confirmBtn.onclick = function () { + closeModal(); + onConfirm(); + }; + } + } + + function closeModal() { + var overlay = document.getElementById('modal-overlay'); + if (overlay) overlay.style.display = 'none'; + } + + // --------------------------------------------------------------------------- + // Event Listeners + // --------------------------------------------------------------------------- + + document.addEventListener('click', function (e) { + var target = e.target; + // Walk up to find data-action + while (target && target !== document) { + if (target.getAttribute && target.getAttribute('data-action')) { + e.preventDefault(); + handleAction(target); + return; + } + target = target.parentElement; + } + }); + + document.addEventListener('keydown', function (e) { + if (e.key === 'Escape') closeModal(); + }); + + // Modal overlay click to close + var overlay = document.getElementById('modal-overlay'); + if (overlay) { + overlay.addEventListener('click', function (e) { + if (e.target === overlay) closeModal(); + }); + } + + // Auth form + var connectBtn = document.getElementById('connect-btn'); + if (connectBtn) { + connectBtn.addEventListener('click', function () { + var input = document.getElementById('token-input'); + var errEl = document.getElementById('auth-error'); + if (!input || !input.value.trim()) return; + + connectBtn.disabled = true; + connectBtn.textContent = 'Connecting...'; + + authenticate(input.value.trim()).catch(function (err) { + if (errEl) { + errEl.textContent = 'Authentication failed: ' + err.message; + errEl.style.display = 'block'; + } + connectBtn.disabled = false; + connectBtn.textContent = 'Connect'; + }); + }); + } + + var tokenInput = document.getElementById('token-input'); + if (tokenInput) { + tokenInput.addEventListener('keydown', function (e) { + if (e.key === 'Enter' && connectBtn) connectBtn.click(); + }); + } + + // Logout buttons + var logoutBtn = document.getElementById('logout-btn'); + if (logoutBtn) logoutBtn.addEventListener('click', logout); + var logoutDenied = document.getElementById('logout-btn-denied'); + if (logoutDenied) logoutDenied.addEventListener('click', logout); + + // Hash-based routing + window.addEventListener('hashchange', function () { + if (document.getElementById('app').style.display !== 'none') { + route(); + } + }); + + // --------------------------------------------------------------------------- + // Init + // --------------------------------------------------------------------------- + + autoAuth(); + +})(); diff --git a/src/channels/web/static/app.js b/crates/ironclaw_gateway/static/app.js similarity index 56% rename from src/channels/web/static/app.js rename to crates/ironclaw_gateway/static/app.js index ddcfc828e9b..f84d9e5735e 100644 --- a/src/channels/web/static/app.js +++ b/crates/ironclaw_gateway/static/app.js @@ -68,11 +68,12 @@ document.getElementById('settings-theme-toggle')?.addEventListener('click', () = const btn = document.getElementById('settings-theme-toggle'); if (btn) { const mode = localStorage.getItem('ironclaw-theme') || 'system'; - btn.textContent = 'Theme: ' + mode.charAt(0).toUpperCase() + mode.slice(1); + btn.textContent = I18n.t('theme.label', { mode: mode.charAt(0).toUpperCase() + mode.slice(1) }); } }); let token = ''; +let oidcProxyAuth = false; let eventSource = null; let logEventSource = null; let currentTab = 'chat'; @@ -89,11 +90,134 @@ let pairingPollInterval = null; let unreadThreads = new Map(); // thread_id -> unread count let _loadThreadsTimer = null; const JOB_EVENTS_CAP = 500; +const JOB_EVENTS_MAX_JOBS = 50; +const MAX_DOM_MESSAGES = 200; const MEMORY_SEARCH_QUERY_MAX_LENGTH = 100; let stagedImages = []; let authFlowPending = false; let _ghostSuggestion = ''; let currentSettingsSubtab = 'inference'; +let generatedImagesByThread = new Map(); +const GENERATED_IMAGE_THREAD_CACHE_CAP = 20; +const GENERATED_IMAGES_PER_THREAD_CAP = 8; + +// --- Hash-based URL Navigation --- +// +// Encodes navigation state in window.location.hash so refreshing +// the page restores the current tab, thread, memory file, job detail, etc. +// +// Hash format: #/{tab}[/{detail}[/{subtab}]] +// #/chat → chat tab, assistant thread +// #/chat/{threadId} → chat tab, specific thread +// #/memory → memory tab, tree root +// #/memory/{path/to/file} → memory tab, specific file +// #/jobs → jobs list +// #/jobs/{jobId} → job detail +// #/routines → routines list +// #/routines/{id} → routine detail +// #/settings/{subtab} → settings tab with specific sub-tab +// #/logs → logs tab + +/** Suppress hash-change handling while we're programmatically updating. */ +let _suppressHashChange = false; + +/** Update the URL hash to reflect current navigation state. */ +function updateHash() { + if (_suppressHashChange) return; + var parts = [currentTab]; + + switch (currentTab) { + case 'chat': + if (currentThreadId && currentThreadId !== assistantThreadId) { + parts.push(currentThreadId); + } + break; + case 'memory': + if (typeof currentMemoryPath === 'string' && currentMemoryPath) { + parts.push(currentMemoryPath); + } + break; + case 'jobs': + if (typeof currentJobId !== 'undefined' && currentJobId) { + parts.push(currentJobId); + } + break; + case 'routines': + if (typeof currentRoutineId !== 'undefined' && currentRoutineId) { + parts.push(currentRoutineId); + } + break; + case 'settings': + if (currentSettingsSubtab && currentSettingsSubtab !== 'inference') { + parts.push(currentSettingsSubtab); + } + break; + } + + var hash = '#/' + parts.join('/'); + if (window.location.hash !== hash) { + window.history.replaceState(null, '', hash); + } +} + +/** Parse the current URL hash into navigation state. */ +function parseHash() { + var hash = window.location.hash || ''; + if (!hash.startsWith('#/')) return null; + var parts = hash.substring(2).split('/'); + return { + tab: parts[0] || 'chat', + detail: parts.slice(1).join('/') || null, + }; +} + +/** + * Restore navigation state from the URL hash. + * Called once after authentication and on hashchange events. + */ +function restoreFromHash() { + var state = parseHash(); + if (!state) return; + + // Suppress hash updates while restoring — switchTab/readMemoryFile/etc. + // each call updateHash(), which would overwrite the full hash before + // the detail part is restored. + _suppressHashChange = true; + + // Switch tab + if (state.tab && state.tab !== currentTab) { + switchTab(state.tab); + } + + // Restore detail state within the tab + if (state.detail) { + switch (state.tab) { + case 'chat': + // Defer thread switch until threads are loaded + window._pendingThreadRestore = state.detail; + break; + case 'memory': + readMemoryFile(state.detail); + break; + case 'jobs': + openJobDetail(state.detail); + break; + case 'routines': + openRoutineDetail(state.detail); + break; + case 'settings': + switchSettingsSubtab(state.detail); + break; + } + } + + _suppressHashChange = false; +} + +window.addEventListener('hashchange', function() { + if (_suppressHashChange) return; + restoreFromHash(); +}); // --- Streaming Debounce State --- let _streamBuffer = ''; @@ -104,9 +228,36 @@ const STREAM_DEBOUNCE_MS = 50; let _connectionLostTimer = null; let _connectionLostAt = null; let _reconnectAttempts = 0; +let _lastSseEventId = null; + +// --- Turn Response Tracking State --- +// Safety net for lost SSE response events (see #2079): tracks whether we +// received a `response` event for the current turn so that a "Done" status +// arriving without one can trigger a history reload. +const DONE_WITHOUT_RESPONSE_TIMEOUT_MS = 1500; +// Single-thread tracking is intentional: background thread events are already +// filtered out by `isCurrentThread`, so only the active thread's turn state +// matters here. Per-thread state is unnecessary. +let _turnResponseReceived = false; +let _doneWithoutResponseTimer = null; + +// Clean up connection-level timers and buffers. +// Called before creating a new connection, on tab hide, and on page unload +// to prevent leaked intervals/timeouts from accumulating across reconnects. +// Note: _doneWithoutResponseTimer is intentionally NOT cleared here — it is a +// turn-level concern managed by the onopen and response handlers (#2079). +function cleanupConnectionState() { + if (_streamDebounceTimer) { clearInterval(_streamDebounceTimer); _streamDebounceTimer = null; } + _streamBuffer = ''; + if (_connectionLostTimer) { clearTimeout(_connectionLostTimer); _connectionLostTimer = null; } + if (jobListRefreshTimer) { clearTimeout(jobListRefreshTimer); jobListRefreshTimer = null; } + if (_loadThreadsTimer) { clearTimeout(_loadThreadsTimer); _loadThreadsTimer = null; } + if (gatewayStatusInterval) { clearInterval(gatewayStatusInterval); gatewayStatusInterval = null; } +} // --- Send Cooldown State --- let _sendCooldown = false; +let _recentLocalPairingApprovals = new Map(); // --- Slash Commands --- @@ -140,6 +291,82 @@ let _activityThinking = null; // --- Auth --- +// Common post-auth initialization shared by token auth and OIDC auto-auth. +function initApp() { + var authScreen = document.getElementById('auth-screen'); + var app = document.getElementById('app'); + // Cross-fade: fade out auth screen, then show app + if (authScreen) authScreen.style.opacity = '0'; + // Show app container (invisible — opacity:0 in CSS) so layout computes + app.style.display = 'flex'; + // Position tab indicator instantly (no transition) before fade-in + var indicator = document.getElementById('tab-indicator'); + if (indicator) indicator.style.transition = 'none'; + updateTabIndicator(); + // Force layout so the instant position is applied, then restore transition + if (indicator) { + void indicator.offsetLeft; + indicator.style.transition = ''; + } + // Now fade in + app.classList.add('visible'); + // Hide auth screen after fade-out transition completes + setTimeout(function() { if (authScreen) authScreen.style.display = 'none'; }, 300); + // Strip token and log_level from URL so they're not visible in the address bar + var cleaned = new URL(window.location); + var urlLogLevel = cleaned.searchParams.get('log_level'); + cleaned.searchParams.delete('token'); + cleaned.searchParams.delete('log_level'); + window.history.replaceState({}, '', cleaned.pathname + cleaned.search + cleaned.hash); + connectSSE(); + connectLogSSE(); + startGatewayStatusPolling(); + // Fetch user profile and render avatar + account menu. + apiFetch('/api/profile').then(function(profile) { + if (!profile) return; + window._currentUser = profile; + // Hide admin tabs for non-admin users. + if (profile.role !== 'admin') { + var usersTab = document.querySelector('[data-settings-subtab="users"]'); + if (usersTab) usersTab.style.display = 'none'; + } + // Render avatar. + var avatarImg = document.getElementById('user-avatar-img'); + var avatarInitials = document.getElementById('user-avatar-initials'); + var displayName = profile.display_name || profile.email || profile.id || '?'; + if (avatarInitials) { + avatarInitials.textContent = displayName.charAt(0).toUpperCase(); + } + if (profile.avatar_url && avatarImg) { + avatarImg.referrerPolicy = 'no-referrer'; + avatarImg.onload = function() { + if (avatarInitials) avatarInitials.style.display = 'none'; + }; + avatarImg.src = profile.avatar_url; + avatarImg.removeAttribute('hidden'); + } + // Populate dropdown. + var nameEl = document.getElementById('user-dropdown-name'); + var emailEl = document.getElementById('user-dropdown-email'); + var roleEl = document.getElementById('user-dropdown-role'); + if (nameEl) nameEl.textContent = profile.display_name || profile.id; + if (emailEl) emailEl.textContent = profile.email || ''; + if (roleEl) roleEl.textContent = profile.role; + }).catch(function() {}); + checkTeeStatus(); + loadThreads(); + loadMemoryTree(); + loadJobs(); + // Restore navigation state from URL hash (tab, thread, memory file, etc.) + restoreFromHash(); + // Apply URL log_level param if present, otherwise just sync the dropdown + if (urlLogLevel) { + setServerLogLevel(urlLogLevel); + } else { + loadServerLogLevel(); + } +} + function authenticate() { token = document.getElementById('token-input').value.trim(); if (!token) { @@ -151,51 +378,14 @@ function authenticate() { const connectBtn = document.getElementById('auth-connect-btn'); if (connectBtn) { connectBtn.disabled = true; - connectBtn.textContent = 'Connecting...'; + connectBtn.textContent = I18n.t('auth.connecting'); } // Test the token against the health-ish endpoint (chat/threads requires auth) apiFetch('/api/chat/threads') .then(() => { sessionStorage.setItem('ironclaw_token', token); - const authScreen = document.getElementById('auth-screen'); - const app = document.getElementById('app'); - // Cross-fade: fade out auth screen, then show app - if (authScreen) authScreen.style.opacity = '0'; - // Show app container (invisible — opacity:0 in CSS) so layout computes - app.style.display = 'flex'; - // Position tab indicator instantly (no transition) before fade-in - const indicator = document.getElementById('tab-indicator'); - if (indicator) indicator.style.transition = 'none'; - updateTabIndicator(); - // Force layout so the instant position is applied, then restore transition - if (indicator) { - void indicator.offsetLeft; - indicator.style.transition = ''; - } - // Now fade in - app.classList.add('visible'); - // Hide auth screen after fade-out transition completes - setTimeout(() => { if (authScreen) authScreen.style.display = 'none'; }, 300); - // Strip token and log_level from URL so they're not visible in the address bar - const cleaned = new URL(window.location); - const urlLogLevel = cleaned.searchParams.get('log_level'); - cleaned.searchParams.delete('token'); - cleaned.searchParams.delete('log_level'); - window.history.replaceState({}, '', cleaned.pathname + cleaned.search); - connectSSE(); - connectLogSSE(); - startGatewayStatusPolling(); - checkTeeStatus(); - loadThreads(); - loadMemoryTree(); - loadJobs(); - // Apply URL log_level param if present, otherwise just sync the dropdown - if (urlLogLevel) { - setServerLogLevel(urlLogLevel); - } else { - loadServerLogLevel(); - } + initApp(); }) .catch(() => { sessionStorage.removeItem('ironclaw_token'); @@ -206,7 +396,7 @@ function authenticate() { // Reset Connect button on error if (connectBtn) { connectBtn.disabled = false; - connectBtn.textContent = 'Connect'; + connectBtn.textContent = I18n.t('auth.connect'); } }); } @@ -215,10 +405,159 @@ document.getElementById('token-input').addEventListener('keydown', (e) => { if (e.key === 'Enter') authenticate(); }); +// Close SSE connections on page unload to free the browser's connection pool. +// Without this, stale SSE connections from prior page loads linger and exhaust +// the HTTP/1.1 per-origin connection limit (6), blocking API fetch calls. +window.addEventListener('beforeunload', () => { + cleanupConnectionState(); + if (eventSource) { eventSource.close(); eventSource = null; } + if (logEventSource) { logEventSource.close(); logEventSource = null; } +}); + +// Pause SSE when the browser tab is hidden (another tab is focused) and resume +// when it becomes visible again. This frees connection slots for other tabs +// running the gateway — without this, each tab holds 1-2 SSE connections and +// the 3rd tab exhausts the browser's per-origin limit. +document.addEventListener('visibilitychange', () => { + if (document.hidden) { + cleanupConnectionState(); + if (eventSource) { eventSource.close(); eventSource = null; } + if (logEventSource) { logEventSource.close(); logEventSource = null; } + } else if (token) { + connectSSE(); + startGatewayStatusPolling(); + if (currentTab === 'logs') connectLogSSE(); + } +}); + +// --- Social login (OAuth + NEAR wallet) --- + +// Show the token form (used as fallback when no OAuth providers are available). +function showTokenForm() { + var tokenForm = document.getElementById('auth-token-form'); + if (tokenForm) { + tokenForm.style.display = ''; + var input = document.getElementById('token-input'); + if (input) input.focus(); + } +} + +// Discover enabled providers and show corresponding buttons. +fetch('/auth/providers', { credentials: 'include' }) + .then(function(r) { return r.ok ? r.json() : { providers: [] }; }) + .then(function(data) { + var providers = data.providers || []; + if (providers.length === 0) { showTokenForm(); return; } + // Store NEAR network for the wallet connector. + if (data.near_network) window._nearNetwork = data.near_network; + var social = document.getElementById('auth-social'); + if (social) social.style.display = ''; + providers.forEach(function(p) { + var btn = document.getElementById('auth-' + p + '-btn'); + if (!btn) return; + btn.style.display = ''; + if (p === 'near') { + btn.addEventListener('click', authenticateWithNear); + } else { + btn.addEventListener('click', function() { window.location = '/auth/login/' + p; }); + } + }); + // When social providers are available, collapse the token form + // and show the "or use a token" divider instead. + var tokenForm = document.getElementById('auth-token-form'); + var tokenDivider = document.getElementById('auth-token-divider'); + if (tokenForm && tokenDivider) { + tokenForm.style.display = 'none'; + tokenDivider.style.display = ''; + tokenDivider.style.cursor = 'pointer'; + tokenDivider.addEventListener('click', function() { + tokenForm.style.display = ''; + tokenDivider.style.display = 'none'; + var input = document.getElementById('token-input'); + if (input) input.focus(); + }); + } + }) + .catch(function() { showTokenForm(); }); + +// NEAR wallet authentication via near-connect. +async function authenticateWithNear() { + var nearBtn = document.getElementById('auth-near-btn'); + var errEl = document.getElementById('auth-error'); + if (nearBtn) { nearBtn.disabled = true; nearBtn.textContent = I18n.t('auth.connectingWallet'); } + if (errEl) errEl.textContent = ''; + + try { + // 1. Get challenge nonce from the server. + var challengeResp = await fetch('/auth/near/challenge', { credentials: 'include' }); + if (!challengeResp.ok) throw new Error('Failed to get challenge'); + var challenge = await challengeResp.json(); + + // 2. Load near-connect dynamically if not already loaded. + if (!window._nearConnector) { + var mod = await import('https://esm.sh/@hot-labs/near-connect@0.11'); + var network = window._nearNetwork || 'mainnet'; + window._nearConnector = new mod.NearConnector({ network: network }); + } + var connector = window._nearConnector; + + // 3. Connect wallet and request signature. + if (nearBtn) nearBtn.textContent = I18n.t('auth.signWithWallet'); + var wallet = await connector.connect(); + var accounts = await wallet.getAccounts(); + if (!accounts || accounts.length === 0) throw new Error('No NEAR account found'); + + var accountId = accounts[0].accountId; + + // Convert hex nonce to Uint8Array for signMessage. + var nonceBytes = new Uint8Array(challenge.nonce.match(/.{2}/g).map(function(b) { return parseInt(b, 16); })); + + var signed = await wallet.signMessage({ + message: challenge.message, + recipient: challenge.recipient || 'ironclaw', + nonce: nonceBytes, + }); + + // 4. Send signature to server for verification. + if (nearBtn) nearBtn.textContent = I18n.t('auth.verifying'); + var verifyResp = await fetch('/auth/near/verify', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ + account_id: accountId, + public_key: signed.publicKey, + signature: signed.signature, + nonce: challenge.nonce, + }), + }); + + if (!verifyResp.ok) { + var errText = await verifyResp.text(); + throw new Error(errText || 'Verification failed'); + } + + await verifyResp.json(); + + // 5. Rely on the HttpOnly session cookie created by the backend. + token = ''; + sessionStorage.removeItem('ironclaw_token'); + initApp(); + } catch (err) { + if (errEl) errEl.textContent = err.message || 'NEAR wallet login failed'; + if (nearBtn) { nearBtn.disabled = false; nearBtn.textContent = I18n.t('auth.social.near'); } + } +} + // Note: main event listener registration is at the bottom of this file (search // "Event Listener Registration"). Do NOT add duplicate listeners here. -// Auto-authenticate from URL param or saved session +// Auto-authenticate from URL param, saved session, or OIDC proxy header. +// +// When behind a reverse proxy that injects auth (e.g., AWS ALB with OIDC), +// the proxy already authenticates every request. We probe /api/gateway/status +// without a token — if the proxy's header lets us through, skip the login +// screen entirely. (function autoAuth() { const params = new URLSearchParams(window.location.search); const urlToken = params.get('token'); @@ -227,15 +566,28 @@ document.getElementById('token-input').addEventListener('keydown', (e) => { authenticate(); return; } + // Restore OIDC proxy mode from session. + if (sessionStorage.getItem('ironclaw_oidc') === '1') { + oidcProxyAuth = true; + } const saved = sessionStorage.getItem('ironclaw_token'); if (saved) { document.getElementById('token-input').value = saved; - // Hide auth screen immediately to prevent flash, authenticate() will - // restore it if the token turns out to be invalid. document.getElementById('auth-screen').style.display = 'none'; document.getElementById('app').style.display = 'flex'; authenticate(); + return; } + // Probe for proxy-injected OIDC auth (no token needed from the client). + fetch('/api/gateway/status', { credentials: 'include' }).then(function(r) { + if (r.ok) { + oidcProxyAuth = true; + sessionStorage.setItem('ironclaw_oidc', '1'); + document.getElementById('auth-screen').style.display = 'none'; + document.getElementById('app').style.display = 'flex'; + initApp(); + } + }).catch(function() { /* proxy auth not available, show login */ }); })(); // --- API helper --- @@ -243,7 +595,10 @@ document.getElementById('token-input').addEventListener('keydown', (e) => { function apiFetch(path, options) { const opts = options || {}; opts.headers = opts.headers || {}; - opts.headers['Authorization'] = 'Bearer ' + token; + // In OIDC mode the reverse proxy provides auth; skip the Authorization header. + if (token && !oidcProxyAuth) { + opts.headers['Authorization'] = 'Bearer ' + token; + } if (opts.body && typeof opts.body === 'object') { opts.headers['Content-Type'] = 'application/json'; opts.body = JSON.stringify(opts.body); @@ -351,15 +706,46 @@ function updateRestartButtonVisibility() { // --- SSE --- -function connectSSE() { - if (eventSource) eventSource.close(); +function rememberSseEventId(event) { + if (!event || !event.lastEventId) return; + _lastSseEventId = event.lastEventId; + window.__e2e = window.__e2e || {}; + window.__e2e.lastSseEventId = event.lastEventId; +} - eventSource = new EventSource('/api/chat/events?token=' + encodeURIComponent(token)); +function connectSSE(lastEventIdOverride) { + if (eventSource) eventSource.close(); + cleanupConnectionState(); + + // In OIDC mode the reverse proxy provides auth; no query token needed. + let chatSseUrl = (token && !oidcProxyAuth) + ? '/api/chat/events?token=' + encodeURIComponent(token) + : '/api/chat/events'; + const lastEventId = lastEventIdOverride || _lastSseEventId; + if (lastEventId) { + chatSseUrl += (chatSseUrl.includes('?') ? '&' : '?') + + 'last_event_id=' + encodeURIComponent(lastEventId); + } + eventSource = new EventSource(chatSseUrl); + + const addTrackedEventListener = (eventType, handler) => { + eventSource.addEventListener(eventType, (event) => { + rememberSseEventId(event); + handler(event); + }); + }; eventSource.onopen = () => { document.getElementById('sse-dot').classList.remove('disconnected'); - document.getElementById('sse-status').textContent = I18n.t('status.connected'); + var statusEl = document.getElementById('sse-status'); + if (statusEl) statusEl.textContent = I18n.t('status.connected'); _reconnectAttempts = 0; + // Clear stale turn-tracking state from before the disconnect + _turnResponseReceived = false; + if (_doneWithoutResponseTimer) { + clearTimeout(_doneWithoutResponseTimer); + _doneWithoutResponseTimer = null; + } // Dismiss connection-lost banner and show reconnected flash if (_connectionLostTimer) { @@ -369,7 +755,7 @@ function connectSSE() { const lostBanner = document.getElementById('connection-banner'); if (lostBanner) { const wasDisconnectedLong = _connectionLostAt && (Date.now() - _connectionLostAt > 10000); - lostBanner.textContent = 'Reconnected'; + lostBanner.textContent = I18n.t('connection.reconnected'); lostBanner.className = 'connection-banner connection-banner-success'; setTimeout(() => { lostBanner.remove(); }, 2000); _connectionLostAt = null; @@ -400,12 +786,13 @@ function connectSSE() { eventSource.onerror = () => { _reconnectAttempts++; document.getElementById('sse-dot').classList.add('disconnected'); - document.getElementById('sse-status').textContent = I18n.t('status.reconnecting'); + var statusEl2 = document.getElementById('sse-status'); + if (statusEl2) statusEl2.textContent = I18n.t('status.reconnecting'); // Update existing banner with attempt count const existingBanner = document.getElementById('connection-banner'); if (existingBanner && existingBanner.classList.contains('connection-banner-warning')) { - existingBanner.textContent = 'Connection lost. Reconnecting... (attempt ' + _reconnectAttempts + ')'; + existingBanner.textContent = I18n.t('connection.reconnecting', { count: _reconnectAttempts }); } // Start connection-lost banner timer (3s delay) @@ -416,13 +803,44 @@ function connectSSE() { // Only show if still disconnected const dot = document.getElementById('sse-dot'); if (dot?.classList.contains('disconnected')) { - showConnectionBanner('Connection lost. Reconnecting... (attempt ' + _reconnectAttempts + ')', 'warning'); + showConnectionBanner(I18n.t('connection.reconnecting', { count: _reconnectAttempts }), 'warning'); } }, 3000); } }; - eventSource.addEventListener('response', (e) => { + // Forward all SSE events to registered widget handlers. + // Wraps addEventListener to intercept every named event and dispatch + // to widget subscribers before the built-in handler runs. + // Must run before any addTrackedEventListener calls so the wrapper is in place. + // + // NOTE: Only NAMED events (those dispatched via `addEventListener('foo', …)` + // by the gateway, see `SseEvent` in `src/channels/web/types.rs`) are + // forwarded. The generic `eventSource.onmessage` handler is intentionally + // NOT wrapped because the IronClaw gateway never emits SSE frames without + // an `event:` field — every frame carries a typed name (`response`, + // `tool_started`, `gate_required`, etc.). Widget authors should subscribe + // to those typed events via `IronClaw.api.on('', handler)` + // rather than relying on the generic message channel; if a widget needs + // an untyped stream it must open its own `EventSource`. + var _origAddEventListener = eventSource.addEventListener.bind(eventSource); + eventSource.addEventListener = function(type, listener, opts) { + _origAddEventListener(type, function(e) { + // Dispatch to widget handlers + if (IronClaw.api && e.data) { + try { + var parsed = JSON.parse(e.data); + IronClaw.api._dispatch(type, parsed); + } catch (parseErr) { + console.warn('[IronClaw] SSE parse error for event', type, parseErr); + } + } + // Call original handler + listener(e); + }, opts); + }; + + addTrackedEventListener('response', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) { if (data.thread_id) { @@ -444,8 +862,14 @@ function connectSSE() { const streamingMsg = document.querySelector('.message.assistant[data-streaming="true"]'); if (streamingMsg) streamingMsg.removeAttribute('data-streaming'); + _turnResponseReceived = true; + if (_doneWithoutResponseTimer) { + clearTimeout(_doneWithoutResponseTimer); + _doneWithoutResponseTimer = null; + } finalizeActivityGroup(); addMessage('assistant', data.content); + pruneOldMessages(); enableChatInput(); // Refresh thread list so new titles appear after first message loadThreads(); @@ -456,7 +880,7 @@ function connectSSE() { } }); - eventSource.addEventListener('thinking', (e) => { + addTrackedEventListener('thinking', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) { if (data.thread_id) debouncedLoadThreads(); @@ -466,7 +890,7 @@ function connectSSE() { showActivityThinking(data.message); }); - eventSource.addEventListener('suggestions', (e) => { + addTrackedEventListener('suggestions', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; if (data.suggestions && data.suggestions.length > 0) { @@ -474,13 +898,13 @@ function connectSSE() { } }); - eventSource.addEventListener('tool_started', (e) => { + addTrackedEventListener('tool_started', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; addToolCard(data.name); }); - eventSource.addEventListener('tool_completed', (e) => { + addTrackedEventListener('tool_completed', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; completeToolCard(data.name, data.success, data.error, data.parameters); @@ -491,13 +915,13 @@ function connectSSE() { } }); - eventSource.addEventListener('tool_result', (e) => { + addTrackedEventListener('tool_result', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; setToolCardOutput(data.name, data.preview); }); - eventSource.addEventListener('stream_chunk', (e) => { + addTrackedEventListener('stream_chunk', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; finalizeActivityGroup(); @@ -511,6 +935,10 @@ function connectSSE() { } if (lastAssistant) lastAssistant.setAttribute('data-streaming', 'true'); + // Mark turn as having received content so the Done safety net + // does not trigger a spurious loadHistory() for streaming responses. + _turnResponseReceived = true; + // Accumulate chunks and debounce rendering at 50ms intervals _streamBuffer += data.content; // Force flush when buffer exceeds 10K chars to prevent memory buildup @@ -528,7 +956,7 @@ function connectSSE() { } }); - eventSource.addEventListener('status', (e) => { + addTrackedEventListener('status', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) { if (data.thread_id) debouncedLoadThreads(); @@ -541,15 +969,28 @@ function connectSSE() { if (data.message === 'Done' || data.message === 'Awaiting approval') { finalizeActivityGroup(); enableChatInput(); + // Safety net (#2079): if "Done" arrives but we never received a + // `response` event for this turn, the message may have been lost + // (broadcast lag, proxy buffering, brief SSE disconnect). Reload + // history after a short delay so the user sees the answer. + if (!_turnResponseReceived && data.message === 'Done') { + if (!_doneWithoutResponseTimer) { + _doneWithoutResponseTimer = setTimeout(() => { + _doneWithoutResponseTimer = null; + if (currentThreadId) loadHistory(); + }, DONE_WITHOUT_RESPONSE_TIMEOUT_MS); + } + } + _turnResponseReceived = false; } }); - eventSource.addEventListener('job_started', (e) => { + addTrackedEventListener('job_started', (e) => { const data = JSON.parse(e.data); showJobCard(data); }); - eventSource.addEventListener('approval_needed', (e) => { + addTrackedEventListener('approval_needed', (e) => { const data = JSON.parse(e.data); const hasThread = !!data.thread_id; const forCurrentThread = !hasThread || isCurrentThread(data.thread_id); @@ -566,26 +1007,47 @@ function connectSSE() { if (currentTab === 'settings') refreshCurrentSettingsTab(); }); - eventSource.addEventListener('auth_required', (e) => { + addTrackedEventListener('auth_required', (e) => { handleAuthRequired(JSON.parse(e.data)); }); - eventSource.addEventListener('auth_completed', (e) => { + addTrackedEventListener('auth_completed', (e) => { const data = JSON.parse(e.data); handleAuthCompleted(data); }); - eventSource.addEventListener('extension_status', (e) => { + addTrackedEventListener('pairing_required', (e) => { + const data = JSON.parse(e.data); + handlePairingRequired(data); + }); + + addTrackedEventListener('pairing_completed', (e) => { + const data = JSON.parse(e.data); + handlePairingCompleted(data); + }); + + addTrackedEventListener('gate_required', (e) => { + const data = JSON.parse(e.data); + handleGateRequired(data); + }); + + addTrackedEventListener('gate_resolved', (e) => { + const data = JSON.parse(e.data); + handleGateResolved(data); + }); + + addTrackedEventListener('extension_status', (e) => { if (currentTab === 'settings') refreshCurrentSettingsTab(); }); - eventSource.addEventListener('image_generated', (e) => { + addTrackedEventListener('image_generated', (e) => { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; - addGeneratedImage(data.data_url, data.path); + rememberGeneratedImage(data.thread_id, data.event_id, data.data_url, data.path); + addGeneratedImage(data.data_url, data.path, data.event_id); }); - eventSource.addEventListener('error', (e) => { + addTrackedEventListener('error', (e) => { if (e.data) { const data = JSON.parse(e.data); if (!isCurrentThread(data.thread_id)) return; @@ -595,37 +1057,44 @@ function connectSSE() { } }); - eventSource.addEventListener('turn_cost', (e) => { - const event = JSON.parse(e.data); - if (!isCurrentThread(event.thread_id)) return; - // Add cost badge below last assistant message - const messages = document.querySelectorAll('.message.assistant'); - const lastMsg = messages[messages.length - 1]; - const tokens = (event.input_tokens || 0) + (event.output_tokens || 0); - if (lastMsg && tokens > 0) { - const badge = document.createElement('div'); - badge.className = 'turn-cost-badge'; - const cost = event.cost_usd ? ' \u00b7 ' + event.cost_usd : ''; - badge.textContent = tokens.toLocaleString() + ' tokens' + cost; - lastMsg.appendChild(badge); - } - }); - // Job event listeners (activity stream for all sandbox jobs) const jobEventTypes = [ 'job_message', 'job_tool_use', 'job_tool_result', 'job_status', 'job_result' ]; for (const evtType of jobEventTypes) { - eventSource.addEventListener(evtType, (e) => { + addTrackedEventListener(evtType, (e) => { const data = JSON.parse(e.data); const jobId = data.job_id; if (!jobId) return; - if (!jobEvents.has(jobId)) jobEvents.set(jobId, []); - const events = jobEvents.get(jobId); + // Move jobId to end of Map insertion order (LRU: most-recent last). + // delete+set keeps the Map ordered by last-access time so that + // keys().next() always yields the least-recently-used entry in O(1). + const existing = jobEvents.get(jobId); + if (existing) jobEvents.delete(jobId); + const events = existing || []; + jobEvents.set(jobId, events); events.push({ type: evtType, data: data, ts: Date.now() }); // Cap per-job events to prevent memory leak while (events.length > JOB_EVENTS_CAP) events.shift(); + // Cap total tracked jobs — evict the least-recently-used entry (O(1)). + // Skip currentJobId so the user's actively-viewed job detail panel + // doesn't go empty when many other jobs fire events. + if (jobEvents.size > JOB_EVENTS_MAX_JOBS) { + let evicted = false; + for (const k of jobEvents.keys()) { + if (k !== currentJobId) { + jobEvents.delete(k); + evicted = true; + break; + } + } + // Fallback: if every entry is currentJobId (impossible in practice), + // evict the first key to maintain the cap. + if (!evicted) { + jobEvents.delete(jobEvents.keys().next().value); + } + } // If the Activity tab is currently visible for this job, refresh it refreshActivityTab(jobId); // Auto-refresh job list when on jobs tab (debounced) @@ -639,6 +1108,13 @@ function connectSSE() { } }); } + + // Plan progress checklist + addTrackedEventListener('plan_update', (e) => { + const data = JSON.parse(e.data); + if (data.thread_id && !isCurrentThread(data.thread_id)) return; + renderPlanChecklist(data); + }); } // Check if an SSE event belongs to the currently viewed thread. @@ -703,9 +1179,14 @@ function clearSuggestionChips() { function sendMessage() { clearSuggestionChips(); removeWelcomeCard(); + _turnResponseReceived = false; + if (_doneWithoutResponseTimer) { + clearTimeout(_doneWithoutResponseTimer); + _doneWithoutResponseTimer = null; + } const input = document.getElementById('chat-input'); if (authFlowPending) { - showToast('Complete the auth step before sending chat messages.', 'info'); + showToast(I18n.t('chat.authRequiredBeforeSend'), 'info'); const tokenField = document.querySelector('.auth-card .auth-token-input input'); if (tokenField) tokenField.focus(); return; @@ -718,7 +1199,40 @@ function sendMessage() { const content = input.value.trim(); if (!content && stagedImages.length === 0) return; + // Intercept approval keywords when an unresolved approval card is pending. + // Find the most recent unresolved card for the current thread (resolved cards + // linger 1.5s before removal; cards from other threads must not be matched). + const approvalCards = Array.from(document.querySelectorAll('.approval-card')); + const approvalCard = approvalCards.reverse().find(card => { + if (card.querySelector('.approval-resolved')) return false; + const cardThreadId = card.getAttribute('data-thread-id'); + return !cardThreadId || cardThreadId === currentThreadId; + }); + if (approvalCard && content) { + const lower = content.toLowerCase(); + let action = null; + if (['yes', 'y', 'approve', 'ok', '/approve', '/yes', '/y'].includes(lower)) { + action = 'approve'; + } else if (['always', 'a', 'yes always', 'approve always', '/always', '/a'].includes(lower)) { + action = 'always'; + } else if (['no', 'n', 'deny', 'reject', 'cancel', '/deny', '/no', '/n'].includes(lower)) { + action = 'deny'; + } + if (action) { + input.value = ''; + autoResizeTextarea(input); + input.focus(); + const requestId = approvalCard.getAttribute('data-request-id'); + const threadId = approvalCard.getAttribute('data-thread-id'); + if (requestId) { + sendApprovalAction(requestId, action, threadId); + } + return; + } + } + const userMsg = addMessage('user', content || '(images attached)'); + pruneOldMessages(); input.value = ''; autoResizeTextarea(input); input.focus(); @@ -736,7 +1250,7 @@ function sendMessage() { }).catch((err) => { // Handle rate limiting (429) if (err.status === 429) { - showToast('Rate limited. Please wait.', 'error'); + showToast(I18n.t('chat.rateLimited'), 'error'); _sendCooldown = true; const sendBtn = document.getElementById('send-btn'); if (sendBtn) sendBtn.disabled = true; @@ -752,7 +1266,7 @@ function sendMessage() { const retryLink = document.createElement('a'); retryLink.className = 'retry-link'; retryLink.href = '#'; - retryLink.textContent = 'Retry'; + retryLink.textContent = I18n.t('common.retry'); retryLink.addEventListener('click', (e) => { e.preventDefault(); if (userMsg.parentNode) userMsg.parentNode.removeChild(userMsg); @@ -809,11 +1323,11 @@ function handleImageFiles(files) { Array.from(files).forEach(file => { if (!file.type.startsWith('image/')) return; if (file.size > MAX_IMAGE_SIZE_BYTES) { - alert(`Image "${file.name}" exceeds 5 MB limit (${(file.size / 1024 / 1024).toFixed(1)} MB)`); + alert(I18n.t('chat.imageTooBig', { name: file.name, size: (file.size / 1024 / 1024).toFixed(1) })); return; } if (stagedImages.length >= MAX_STAGED_IMAGES) { - alert(`Maximum ${MAX_STAGED_IMAGES} images allowed per message`); + alert(I18n.t('chat.maxImages', { n: MAX_STAGED_IMAGES })); return; } const reader = new FileReader(); @@ -865,17 +1379,25 @@ chatMessagesEl.addEventListener('copy', (e) => { e.clipboardData.setData('text/plain', text); }); -function addGeneratedImage(dataUrl, path) { - const container = document.getElementById('chat-messages'); +function createGeneratedImageElement(dataUrl, path, eventId) { const card = document.createElement('div'); card.className = 'generated-image-card'; + if (eventId) { + card.dataset.imageEventId = eventId; + } - const img = document.createElement('img'); - img.className = 'generated-image'; - img.src = dataUrl; - img.alt = 'Generated image'; - - card.appendChild(img); + if (isSafeGeneratedImageDataUrl(dataUrl)) { + const img = document.createElement('img'); + img.className = 'generated-image'; + img.src = dataUrl; + img.alt = 'Generated image'; + card.appendChild(img); + } else { + const placeholder = document.createElement('div'); + placeholder.className = 'generated-image-placeholder'; + placeholder.textContent = 'Generated image unavailable in history payload'; + card.appendChild(placeholder); + } if (path) { const pathLabel = document.createElement('div'); @@ -884,8 +1406,76 @@ function addGeneratedImage(dataUrl, path) { card.appendChild(pathLabel); } + return card; +} + +function isSafeGeneratedImageDataUrl(dataUrl) { + return typeof dataUrl === 'string' && /^data:image\//i.test(dataUrl); +} + +function hasRenderedGeneratedImage(container, eventId) { + if (!eventId) return false; + return Array.from(container.querySelectorAll('.generated-image-card')).some((card) => { + return card.dataset.imageEventId === eventId; + }); +} + +function addGeneratedImage(dataUrl, path, eventId, shouldScroll = true) { + const container = document.getElementById('chat-messages'); + if (hasRenderedGeneratedImage(container, eventId)) { + return; + } + const card = createGeneratedImageElement(dataUrl, path, eventId); container.appendChild(card); - container.scrollTop = container.scrollHeight; + if (shouldScroll) { + container.scrollTop = container.scrollHeight; + } +} + +function rememberGeneratedImage(threadId, eventId, dataUrl, path) { + if (!threadId || !eventId || !isSafeGeneratedImageDataUrl(dataUrl)) return; + const normalizedPath = path || null; + let images = generatedImagesByThread.get(threadId); + if (!images) { + if (generatedImagesByThread.size >= GENERATED_IMAGE_THREAD_CACHE_CAP) { + const oldestThreadId = generatedImagesByThread.keys().next().value; + if (oldestThreadId) { + generatedImagesByThread.delete(oldestThreadId); + } + } + images = []; + generatedImagesByThread.set(threadId, images); + } else { + // Refresh insertion order so recently viewed/updated threads stay cached. + generatedImagesByThread.delete(threadId); + generatedImagesByThread.set(threadId, images); + } + if (images.some(img => img.eventId === eventId)) { + return; + } + images.push({ eventId, dataUrl, path: normalizedPath }); + while (images.length > GENERATED_IMAGES_PER_THREAD_CAP) { + images.shift(); + } +} + +function getRememberedGeneratedImage(threadId, eventId) { + if (!threadId || !eventId) return null; + const images = generatedImagesByThread.get(threadId); + if (!images) return null; + return images.find(img => img.eventId === eventId) || null; +} + +function resolveGeneratedImageForRender(threadId, image) { + const normalizedPath = image.path || null; + if (image.data_url) { + return { dataUrl: image.data_url, path: normalizedPath }; + } + const remembered = getRememberedGeneratedImage(threadId, image.event_id); + if (remembered) { + return { dataUrl: remembered.dataUrl, path: remembered.path }; + } + return { dataUrl: null, path: normalizedPath }; } // --- Slash Autocomplete --- @@ -952,16 +1542,22 @@ function filterSlashCommands(value) { } } -function sendApprovalAction(requestId, action) { - apiFetch('/api/chat/approval', { +function sendApprovalAction(requestId, action, threadId) { + const card = document.querySelector('.approval-card[data-request-id="' + requestId + '"]'); + const targetThreadId = threadId || (card ? card.getAttribute('data-thread-id') : null) || currentThreadId; + apiFetch('/api/chat/gate/resolve', { method: 'POST', - body: { request_id: requestId, action: action, thread_id: currentThreadId }, + body: { + request_id: requestId, + thread_id: targetThreadId, + resolution: action === 'deny' ? 'denied' : 'approved', + always: action === 'always', + }, }).catch((err) => { addMessage('system', 'Failed to send approval: ' + err.message); }); // Disable buttons and show confirmation on the card - const card = document.querySelector('.approval-card[data-request-id="' + requestId + '"]'); if (card) { const buttons = card.querySelectorAll('.approval-actions button'); buttons.forEach((btn) => { @@ -970,7 +1566,7 @@ function sendApprovalAction(requestId, action) { const actions = card.querySelector('.approval-actions'); const label = document.createElement('span'); label.className = 'approval-resolved'; - const labelText = action === 'approve' ? 'Approved' : action === 'always' ? 'Always approved' : 'Denied'; + const labelText = action === 'approve' ? I18n.t('approval.approved') : action === 'always' ? I18n.t('approval.alwaysApproved') : I18n.t('approval.denied'); label.textContent = labelText; actions.appendChild(label); // Remove the card after showing the confirmation briefly @@ -1013,6 +1609,299 @@ function sanitizeRenderedHtml(html) { return ''; } +// ==================== Structured Data Rendering ==================== +// +// Detects JSON objects and key-value data in assistant messages and +// renders them as styled cards instead of raw text. Also supports +// extensible chat renderers via IronClaw.registerChatRenderer(). + +/** + * Post-process a .message-content element to upgrade structured data into cards. + * Runs registered chat renderers first, then falls back to built-in JSON detection. + */ +function upgradeStructuredData(contentEl) { + // 1. Run registered chat renderers. + // + // Each registered renderer receives the live `.message-content` element + // and the textContent. The renderer is allowed to mutate the element — + // attach event listeners, set data attributes, swap inner DOM — but any + // HTML it injects must still pass DOMPurify before it reaches the user. + // `renderMarkdown` already runs `sanitizeRenderedHtml` on the markdown + // output BEFORE this function is called, but a renderer that does + // `contentEl.innerHTML = '
...'` would + // bypass that sanitization step entirely. Re-run the sanitizer on + // whatever the renderer leaves behind so the same HTML allowlist + // applies regardless of how the content got there. + // + // CSP already blocks ` + and ' ' and end of content.'), + (re.compile(r"long response", re.IGNORECASE), + ("First paragraph. " * 200 + "\n\n" + "Second paragraph. " * 200)), + # For tool intent nudge test: first response expresses intent without tool call + (re.compile(r"search intent", re.IGNORECASE), + "Let me search for that information now."), + # After the orchestrator sends its nudge, recover with a final completion. + # The exact nudge prefix is "You said you would perform an action..." — + # see `signals_tool_intent` + the nudge append in + # `crates/ironclaw_engine/orchestrator/default.py`. Match either the new + # phrasing or the legacy "You expressed intent" so older deployments + # still work. + (re.compile(r"You said you would perform an action|You expressed intent", re.IGNORECASE), + "I found the information you requested."), ] DEFAULT_RESPONSE = "I understand your request." +TOOL_FAILURE_TRIGGER = re.compile(r"issue 1780 tool failure", re.IGNORECASE) +TRUNCATED_TOOL_CALL_TRIGGER = re.compile( + r"issue 1780 truncated tool call", + re.IGNORECASE, +) +EMPTY_REPLY_TRIGGER = re.compile(r"issue 1780 empty reply", re.IGNORECASE) +LOOP_FOREVER_TRIGGER = re.compile(r"issue 1780 loop forever", re.IGNORECASE) + TOOL_CALL_PATTERNS = [ (re.compile(r"echo (.+)", re.IGNORECASE), "echo", lambda m: {"message": m.group(1)}), + ( + re.compile(r"loop until cap", re.IGNORECASE), + "echo", + lambda _: {"message": "loop-until-cap"}, + ), ( re.compile(r"make approval post (?P