Skip to content

chore: promote staging to staging-promote/7234700c-23635804857 (2026-03-28 08:09 UTC) - #1724

Merged
henrypark133 merged 42 commits into
staging-promote/7234700c-23635804857from
staging-promote/8f8cb7f7-23680994633
Mar 30, 2026
Merged

henrypark133 merged 42 commits into
staging-promote/7234700c-23635804857from
staging-promote/8f8cb7f7-23680994633

Conversation

@ironclaw-ci

@ironclaw-ci ironclaw-ci Bot commented Mar 28, 2026 •

Copy link
Copy Markdown
Contributor

Auto-promotion from staging CI

Batch range: 2f4eb08613cefff1af8b7b1a475fda00c84dd855..8f8cb7f7b1767d28f62ec74a14bf8eb74dd1097a
Promotion branch: staging-promote/8f8cb7f7-23680994633
Base: staging-promote/7234700c-23635804857
Triggered by: Staging CI batch at 2026-03-28 08:09 UTC

Commits in this batch (1):

Current commits in this promotion (1)

Current base: staging-promote/7234700c-23635804857
Current head: staging-promote/8f8cb7f7-23680994633
Current range: origin/staging-promote/7234700c-23635804857..origin/staging-promote/8f8cb7f7-23680994633

Auto-updated by staging promotion metadata workflow

Waiting for gates:

  • Tests: pending
  • E2E: pending
  • Claude Code review: pending (will post comments on this PR)

Auto-created by staging-ci workflow

…i-tenant isolation (#1626)

* feat: complete multi-tenant isolation — per-user budgets, model selection, heartbeat cycling

Finishes the remaining isolation work from phases 2–4 of #59:

Phase 2 (DB scoping): Fix /status and /list commands to use _for_user
DB variants instead of global queries that leaked cross-user job data.

Phase 3 (Runtime isolation): Per-user workspace in routine engine's
spawn_fire so lightweight routines run in the correct user context.
Per-user daily cost tracking in CostGuard with configurable budget via
MAX_COST_PER_USER_PER_DAY_CENTS. Multi-user heartbeat that cycles
through all users with routines, auto-detected from GATEWAY_USER_TOKENS.

Phase 4 (Provider/tools): Per-user model selection via preferred_model
setting — looked up from SettingsStore on first iteration, threaded
through ReasoningContext.model_override to CompletionRequest. Works
with providers that support per-request model overrides (NearAI).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use selected_model setting key to match /model command persistence

The dispatcher was reading "preferred_model" but the /model command
(merged from staging) persists to "selected_model". Since set_setting
is already per-user scoped, using the same key makes /model work as
the per-user model override in multi-tenant mode.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: heartbeat hygiene, /model multi-tenant guard, RigAdapter model override

Three follow-up fixes for multi-tenant isolation:

1. Multi-user heartbeat now runs memory hygiene per user before each
   heartbeat check, matching single-user heartbeat behavior.

2. /model command in multi-tenant mode only persists to per-user
   settings (selected_model) without calling set_model() on the shared
   LlmProvider. The per-request model_override in the dispatcher reads
   from the same setting. Added multi_tenant flag to AgentConfig
   (auto-detected from GATEWAY_USER_TOKENS).

3. RigAdapter now supports per-request model overrides by injecting the
   model name into rig-core's additional_params. OpenAI/Anthropic/Ollama
   API servers use last-key-wins for duplicate JSON keys, so the override
   takes effect via serde's flatten serialization order.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review — cost model attribution, heartbeat concurrency, pruning

Fixes from review comments on #1614:

- Cost tracking now uses the override model name (not active_model_name)
  when a per-user model override is active, for accurate attribution.
- Multi-user heartbeat runs per-user checks concurrently via JoinSet
  instead of sequentially, preventing one slow user from blocking others.
- Per-user failure counts tracked independently; users exceeding
  max_failures are skipped (matching single-user semantics).
- per_user_daily_cost HashMap pruned on day rollover to prevent
  unbounded growth in long-lived deployments.
- Doc comment fixed: says "routines" not "active routines".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: /status ownership, model persistence scoping, heartbeat robustness

Addresses second round of PR review on #1614:

- /status <job_id> DB path now validates job.user_id == requesting user
  before returning data (was missing ownership check, security fix).

- persist_selected_model takes user_id param instead of owner_id, and
  skips .env/TOML writes in multi-tenant mode (these are shared global
  files). handle_system_command now receives user_id from caller.

- JoinSet collection handles Err(JoinError) explicitly instead of
  silently dropping panicked tasks.

- Notification forwarder extracts owner_id from response metadata in
  multi-tenant mode for per-user routing instead of broadcasting to
  the agent owner.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: cost pricing, fire_manual workspace, heartbeat concurrency cap

Round 3 review fixes:

- Cost tracking passes None for cost_per_token when model override is
  active, letting CostGuard look up pricing by model name instead of
  using the default provider's rates (serrrfirat).

- fire_manual() now uses per-user workspace, matching spawn_fire()
  pattern (serrrfirat).

- Removed MULTI_TENANT env var — multi-tenant mode is auto-detected
  solely from GATEWAY_USER_TOKENS presence (serrrfirat + Copilot).

- Multi-user heartbeat capped at 8 concurrent tasks to avoid flooding
  the LLM provider (serrrfirat + Copilot).

- Fixed inject_model_override doc comment accuracy (Copilot).

- Added comment explaining multi-tenant notification routing priority
  (Copilot).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: user-scoped webhook endpoint for multi-tenant isolation

Adds POST /api/webhooks/u/{user_id}/{path} — a user-scoped webhook
endpoint that filters the routine lookup by user_id, preventing
cross-user webhook triggering when paths collide.

The existing /api/webhooks/{path} endpoint remains unchanged for
backward compatibility in single-user deployments.

Changes:
- get_webhook_routine_by_path gains user_id: Option<&str> param
- Both postgres and libsql implementations add AND user_id = ? filter
  when user_id is provided
- New webhook_trigger_user_scoped_handler extracts (user_id, path)
  from URL and passes to shared fire_webhook_inner logic
- Route registered on public router (webhooks are called by external
  services that can't send bearer tokens)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(db): add UserStore trait with users, api_tokens, invitations tables

Foundation for DB-backed user management (#1605):

- UserRecord, ApiTokenRecord, InvitationRecord types in db/mod.rs
- UserStore sub-trait (17 methods) added to Database supertrait
- PostgreSQL migration V14__users.sql (users, api_tokens, invitations)
- libSQL schema + incremental migration V14
- Full implementations for both PgBackend (via Store delegation) and
  LibSqlBackend (direct SQL in libsql/users.rs)
- authenticate_token JOINs api_tokens+users with active/non-revoked
  checks; has_any_users for bootstrap detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(web): DB-backed auth, user/token/invitation API handlers

Adds the web gateway layer for DB-backed user management (#1605):

Auth refactor:
- CombinedAuthState wraps env-var tokens (MultiAuthState) + optional
  DbAuthenticator for DB-backed token lookup with LRU cache (60s TTL,
  1024 max entries)
- auth_middleware tries env-var tokens first, then DB fallback
- From<MultiAuthState> impl for backward compatibility
- main.rs wires with_db_auth when database is available

API handlers (12 new endpoints):
- /api/admin/users — CRUD: create, list, detail, update, suspend, activate
- /api/tokens — create (returns plaintext once), list, revoke
- /api/invitations — create, list, accept (creates user + first token)

Token creation: 32 random bytes → hex plaintext, SHA-256 hash stored.
Invitation accept: validates hash + pending + not expired, creates
user record and first API token atomically.

All test files updated for CombinedAuthState type change.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: startup env-var user migration + UserStore integration tests

Completes the DB-backed user management feature (#1605):

- Startup migration: when GATEWAY_USER_TOKENS is set and the users
  table is empty, inserts env-var users + hashed tokens into DB.
  Logs deprecation notice when DB already has users.
- hash_token made pub for reuse in migration code.
- 10 integration tests for UserStore (libsql file-backed):
  - has_any_users bootstrap detection
  - create/get/get_by_email/list/update user lifecycle
  - token create → authenticate → revoke → reject cycle
  - suspended user tokens rejected
  - wrong-user token revoke returns false
  - invitation create → accept → user created
  - record_login and record_token_usage timestamps
- libSQL migration: removed FK constraints from V14 (incompatible
  with execute_batch inside transactions). Tables in both base SCHEMA
  and incremental migration for fresh and existing databases.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: remove GATEWAY_USER_TOKENS, fix review feedback

GATEWAY_USER_TOKENS never went to production — replaced entirely by
DB-backed user management via /api/admin/users and /api/tokens.

Removed:
- UserTokenConfig struct and GATEWAY_USER_TOKENS env var parsing
- user_tokens field from GatewayConfig
- GatewayChannel::new_multi_auth() constructor
- Env-var user migration block in main.rs (~90 lines)
- multi_tenant auto-detection from GATEWAY_USER_TOKENS (now runtime
  via db.has_any_users() in app.rs)

Review fixes (zmanian):
- User ID generation: UUID instead of display-name derivation (#1)
- Invitation accept moved to public router (no auth needed) (#3)
- libSQL get_invitation_by_hash aligned with postgres: filters
  status='pending' AND expires_at > now (#4)
- UUID parse: returns DatabaseError::Serialization instead of
  unwrap_or_default (#7)
- PostgreSQL SELECT * replaced with explicit column lists (#8)
- Sort order aligned (both backends use DESC) (#6)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add role-based access control (admin/member)

Adds a `role` field (admin|member) to user management:

Schema:
- `role TEXT NOT NULL DEFAULT 'member'` added to users table in both
  PostgreSQL V14 migration and libSQL schema/incremental migration
- UserRecord gains `role: String` field
- UserIdentity gains `role: String` field, populated from DB in
  DbAuthenticator and defaulting to "admin" for single-user mode

Access control:
- AdminUser extractor: returns 403 Forbidden if role != "admin"
- /api/admin/users/* handlers: require AdminUser (create, list,
  detail, update, suspend, activate)
- POST /api/invitations: requires AdminUser (only admins can invite)
- User creation accepts optional "role" param (defaults to "member")
- Invitation acceptance creates users with "member" role

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(web): add Users admin tab to web UI

Adds a Users tab to the web gateway UI for managing users, tokens,
and roles without needing direct API calls.

Features:
- User list table with ID, name, email, role, status, created date
- Create user form with display name, email, role selector
- Suspend/activate actions per user
- Create API token for any user (shows plaintext once with copy button)
- Role badges (admin highlighted, member muted)
- Non-admin users see "Admin access required" message
- Keyboard shortcut: Cmd/Ctrl+5 switches to Users tab

CSS:
- Reuses routines-table styles for the user list
- Badge, token-display, btn-small, btn-danger, btn-primary components

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move Users to Settings subtab, bootstrap admin user on first run

- Moved Users from top-level tab to Settings sidebar subtab (under
  Skills, before Theme toggle)
- On first startup with empty users table, automatically creates an
  admin user from GATEWAY_USER_ID config with a corresponding API
  token from GATEWAY_AUTH_TOKEN. This ensures the owner appears in
  the Users panel immediately.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: user creation shows token, + Token works, no password save popup

Three UI/UX fixes:

1. Create user now generates an initial API token and shows it in a
   copy-able banner instead of triggering the browser's password save
   dialog. Uses autocomplete="off" and type="text" for email field.

2. "+ Token" button works: exposed createTokenForUser/suspendUser/
   activateUser on window for inline onclick handlers in dynamically
   generated table rows. Token creation uses showTokenBanner helper.

3. Admin token creation: POST /api/tokens now accepts optional
   "user_id" field when the requesting user is admin, allowing
   token creation for other users from the Users panel.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use event delegation for user action buttons (CSP compliance)

Inline onclick handlers are blocked by the Content-Security-Policy
(script-src 'self' without 'unsafe-inline'). Switched to data-action
attributes with a delegated click listener on the users table.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add i18n for Users subtab, show login link on user creation

- Added 'settings.users' i18n key for English and Chinese
- Token banner now shows a full login link (domain/?token=xxx)
  with a Copy Link button, plus the raw token below
- Login link works automatically via existing ?token= auto-auth

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: token hash mismatch — hash hex string, not raw bytes

Critical auth bug: token creation hashed the raw 32 bytes
(hasher.update(token_bytes)) but authentication hashed the hex-encoded
string (hash_token(candidate) where candidate is the hex string the
user sends). This meant newly created tokens could never authenticate.

Fixed all 4 token creation sites (users, tokens, invitations create,
invitations accept) to use hash_token(&plaintext_token) which hashes
the hex string consistently with the auth lookup path.

Removed now-unused sha2::Digest imports from handlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: remove invitation system

The invitation flow is redundant — admin create user already generates
a token and shows a login link. Invitations add complexity without
value until email integration exists.

Removed:
- InvitationRecord struct and 4 UserStore trait methods
- invitations table from V14 migration (postgres + both libsql schemas)
- PostgreSQL Store methods (create/get/accept/list invitations)
- libSQL UserStore invitation methods + row_to_invitation helper
- invitations.rs handler file (212 lines)
- /api/invitations routes (create, list, accept)
- test_invitation_lifecycle test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: user deletion, self-service profile, per-user job limits, usage API

Four multi-tenancy improvements:

1. User deletion cascade (DELETE /api/admin/users/{id}):
   Deletes user and all data across 11 user-scoped tables (settings,
   secrets, routines, memory, jobs, conversations, etc.). Admin only.

2. Self-service profile (GET/PATCH /api/profile):
   Users can read and update their own display_name and metadata
   without admin privileges.

3. Per-user job concurrency (MAX_JOBS_PER_USER env var):
   Scheduler checks active_jobs_for(user_id) before dispatch.
   Prevents one user from exhausting all job slots.

4. Usage reporting (GET /api/admin/usage?user_id=X&period=day|week|month):
   Aggregates LLM costs from llm_calls via agent_jobs.user_id.
   Returns per-user, per-model breakdown of calls, tokens, and cost.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add TenantCtx for compile-time tenant isolation

Implements zmanian's architectural proposal from #1614 review:
two-tier scoped database access (TenantScope/AdminScope) so handler
code cannot accidentally bypass tenant scoping.

TenantScope (default): wraps user_id + Arc<dyn Database>, auto-binds
user_id on every operation. ID-based lookups return None for cross-
tenant resources. No escape hatch — forgetting to scope is a compile
error.

AdminScope (explicit opt-in): cross-tenant access for system-level
components (heartbeat, routine engine, self-repair, scheduler, worker).

TenantCtx bundles TenantScope + workspace + cost guard + per-user
rate limiting. Constructed once per request in handle_message, threaded
through all command handlers and ChatDelegate.

Key changes:
- New src/tenant.rs (~920 lines): TenantScope, AdminScope, TenantCtx,
  TenantRateState, TenantRateRegistry
- All command handlers: user_id: &str → ctx: &TenantCtx
- ChatDelegate: cost check/record/settings via self.tenant
- System components: store field changed to AdminScope
- Config: TENANT_MAX_LLM_CONCURRENT, TENANT_MAX_JOBS_CONCURRENT env vars
- Fixes bug: /status <job_id> cross-tenant leak (now auto-filtered)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR #1626 review feedback — bounded LRU cache, admin auth, FK cleanup

- Replace HashMap with lru::LruCache in DbAuthenticator so the token
  cache is hard-bounded at 1024 entries (evicts LRU, not just expired)
- Gate admin user endpoints (list/detail/update/suspend/activate) with
  AdminUser extractor so members get 403 instead of full access
- Add api_tokens to libSQL delete_user cleanup list to prevent orphaned
  tokens (libSQL has no FK cascade)
- Add regression tests for all three fixes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update CA certificates in runtime Docker image

Ensures the root certificate bundle is current so TLS handshakes
to services like Supabase succeed on Railway.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve CI failures — formatting, no-panics check

- Run cargo fmt on test code
- Replace .expect() with const NonZeroUsize in DbAuthenticator
- Add // safety: comments for test-only code in multi_tenant.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: switch PostgreSQL TLS from rustls to native-tls

rustls with rustls-native-certs fails TLS handshake on Railway's
slim container (empty or stale root cert store). native-tls delegates
to OpenSSL on Linux which handles system certs more reliably.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Adding user management api

* feat: admin secrets provisioning API + API documentation

- Add PUT/GET/DELETE /api/admin/users/{id}/secrets/{name} endpoints for
  application backends to provision per-user secrets (AES-256-GCM encrypted)
- Add secrets_store field to GatewayState with builder wiring
- Create docs/USER_MANAGEMENT_API.md with full API spec covering users,
  secrets, tokens, profile, and usage endpoints
- Update web gateway CLAUDE.md route table

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add CatchPanicLayer to capture handler panics

Without this, panics in async handlers silently drop the connection
and the edge proxy returns a generic 503. Now panics are caught,
logged, and returned as 500 with the panic message.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address second-round review — transactional delete, overflow, error logging

- C1: Wrap PostgreSQL delete_user() in a transaction so partial cleanup
  can't leave users in a half-deleted state
- M2: Add job_events to delete cleanup (both backends) — FK to
  agent_jobs without CASCADE would cause FK violation
- H1/M4: Cap expires_in_days to 36500 before i64 cast (tokens + secrets)
- H2: Validate target user exists before creating admin token to prevent
  orphan tokens on libSQL
- H3: Log DB errors in DbAuthenticator::authenticate() instead of
  silently swallowing them as 401

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert to rustls with webpki-roots fallback for PostgreSQL TLS

native-tls/OpenSSL caused silent crashes (segfaults in C code) during
DB writes on Railway containers. Switch back to rustls but add
webpki-roots as a fallback when system certs are missing, which was
the original TLS handshake failure on slim container images.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update Cargo.lock for rustls + webpki-roots

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* debug: add /api/debug/db-write endpoint to diagnose user insert failure

Temporary diagnostic endpoint that tests DB INSERT to users table
with full error logging. No auth required. Will be removed after
debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use cargo-chef in Dockerfile for dependency caching

Splits the build into planner/deps/builder stages. Dependencies are
only recompiled when Cargo.toml or Cargo.lock change. Source-only
changes skip straight to the final build stage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* debug: add tracing to users_create_handler

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: guard created_by FK in user creation handler

The auth identity user_id (from owner_id scope) may not match any
user row in the DB, causing a FK violation on the created_by column.
Check that the referenced user exists before setting created_by.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: collapse GATEWAY_USER_ID into IRONCLAW_OWNER_ID

Remove the separate GATEWAY_USER_ID config. The gateway now uses
IRONCLAW_OWNER_ID (config.owner_id) directly for auth identity,
bootstrap user creation, and workspace scoping.

Previously, with_owner_scope() rebinds the auth identity to owner_id
while keeping default_sender_id as the gateway user_id. This caused
a FK constraint violation when creating users because the auth
identity ("default") didn't match any user in the DB ("nearai").

Changes:
- Remove GATEWAY_USER_ID env var and gateway_user_id from settings
- Remove user_id field from GatewayConfig
- Add owner_id parameter to GatewayChannel::new()
- Remove with_owner_scope() method
- Remove default_sender_id from GatewayState
- Remove sender override logic in chat/approval handlers
- Remove debug endpoint and tracing from prior debugging
- Update all tests and E2E fixtures

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: hide Users tab for non-admins, remove auth hint text

- Fetch /api/profile after login and hide the Users settings tab
  when the user's role is not admin
- Remove the "Enter the GATEWAY_AUTH_TOKEN" hint from the login page
  since tokens are now managed via the admin panel, not .env files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review feedback (auth 503, token expiry, CORS PATCH)

- DB auth errors now return 503 instead of 401 so outages are
  distinguishable from invalid tokens (serrrfirat H3)
- Cap expires_in_days to 36500 before i64 cast to prevent negative
  duration from u64 overflow (serrrfirat H1)
- Add PATCH to CORS allowed methods for profile/user update
  endpoints (Copilot)
- Stop leaking panic details in CatchPanicLayer response body

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: harden multi-tenant isolation — review fixes from #1614

- Add conversation ownership checks in TenantScope: add_conversation_message,
  touch_conversation, list_conversation_messages (+ paginated),
  update_conversation_metadata_field, get_conversation_metadata now return
  NotFound for conversations not owned by the tenant (cross-tenant data leak)
- Fix multi-user heartbeat: clear notify_user_id per runner so notifications
  persist to the correct user, not the shared config target
- Move hygiene tasks into bounded JoinSet instead of unbounded tokio::spawn
- Revert send_notification to private visibility (only used within module)
- Use effective_model_name() for cost attribution in dispatcher so providers
  that ignore per-request model overrides report the actual model used
- Fix inject_model_override doc comment; add 3 unit tests
- Fix heartbeat doc comment ("routines" not "active routines")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add Jobs, Cost, Last Active columns to admin Users table

Add UserSummaryStats struct and user_summary_stats() batch query to the
UserStore trait (both PostgreSQL and libSQL backends). The admin users
list endpoint now fetches per-user aggregates (job count, total LLM
spend, most recent activity) in a single query and includes them inline
in the response. The frontend Users table displays three new columns.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review comments and CI formatting failures

CI fixes:
- cargo fmt fixes in cli/mod.rs and db/tls.rs

Security/correctness (from Copilot + serrrfirat + pranavraja99 reviews):
- Token create: reject expires_in_days > 36500 with 400 instead of silent clamp
- Token create: return 404 when admin targets non-existent user
- User create: map duplicate email constraint violations to 409 Conflict
- User create: remove unnecessary DB roundtrip for created_by (use AdminUser directly)
- DB auth: log warn on DB lookup failures instead of silently swallowing errors
- libSQL: add FK constraints on users.created_by and api_tokens.user_id

Config fixes:
- agent.multi_tenant: resolve from AGENT_MULTI_TENANT env var instead of hardcoding false
- heartbeat.multi_tenant: fix doc comment to match actual env-var-based behavior

UI fix:
- showTokenBanner: pass correct title ("Token created!" vs "User created!")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address remaining review comments (round 2)

- Secrets handlers: normalize name to lowercase before store operations,
  validate target user_id exists (returns 404 if not found)
- libSQL: propagate cost parsing errors instead of unwrap_or_default()
  in both user_usage_stats and user_summary_stats
- users_list_handler: propagate user_summary_stats DB errors (was
  silently swallowed with unwrap_or_default)
- loadUsers: distinguish 401/403 (admin required) from other errors
- Docs: fix users.id type (TEXT not UUID), remove "invitation flow"
  from V14 migration comment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: i18n for Users tab, atomic user+token creation, transactional delete_user

i18n:
- Add 31 translation keys for all Users tab strings (en + zh-CN)
- Wire data-i18n attributes on HTML elements (headings, buttons, inputs,
  table headers, empty state)
- Replace all hard-coded strings in app.js with I18n.t() calls

Atomic user+token creation:
- Add create_user_with_token() to UserStore trait
- PostgreSQL: wraps both INSERTs in conn.transaction() with auto-rollback
- libSQL: wraps in explicit BEGIN/COMMIT with ROLLBACK on error
- Handler uses single atomic call instead of two separate operations

Transactional delete_user for libSQL:
- Wrap multi-table DELETE cascade in BEGIN/COMMIT transaction
- ROLLBACK on any error to prevent partial cleanup / inconsistent state
- Matches the PostgreSQL implementation which already used transactions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert V14 migration to match deployed checksum [skip-regression-check]

Refinery checksums applied migrations — editing V14__users.sql after
it was already applied causes deployment failures. Revert the cosmetic
comment changes (added in df40b22) to restore the original checksum.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: bootstrap onboarding flow for multi-tenant users

The bootstrap greeting and workspace seeding only ran for the owner
workspace at startup, so new users created via the admin API never
received the welcome message or identity files (BOOTSTRAP.md, SOUL.md,
AGENTS.md, USER.md, etc.).

Three fixes:
- tenant_ctx(): seed per-user workspace on first creation via
  seed_if_empty(), which writes identity files and sets
  bootstrap_pending when the workspace is truly fresh
- handle_message(): check take_bootstrap_pending() on the tenant
  workspace (not the owner workspace) and persist the greeting to
  the user's own assistant conversation + broadcast via SSE
- WorkspacePool: seed new per-user workspaces in the web gateway
  so memory tools also see identity files immediately

The existing single-user bootstrap in Agent::run() is preserved for
non-multi-tenant deployments.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address remaining PR review comments (round 3)

- Docs: fix metadata description from "merge patch" to "full replacement"
- Secrets: reject expires_in_days > 36500 with 400 (was silently clamped)
- libSQL: CAST(SUM(cost) AS TEXT) in user_usage_stats and user_summary_stats
  to prevent SQLite numeric coercion from crashing get_text() — this was
  the root cause of the Copilot "SUM returns numeric type" comments
- Add 3 regression tests: user_summary_stats (empty + with data) and
  user_usage_stats (multi-model aggregation)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add role change support for users (admin/member toggle)

- Add update_user_role() to UserStore trait + both backends (PostgreSQL
  and libSQL)
- Extend PATCH /api/admin/users/{id} to accept optional "role" field
  with validation (must be "admin" or "member")
- Add "Make Admin" / "Make Member" toggle button in Users table actions
- Add i18n keys for role change (en + zh-CN)
- Update API docs to document the role field on PATCH
- Fix test helpers to use fmt_ts() for timestamps (was using SQLite
  datetime('now') which produces incompatible format for string comparison)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: show live LLM spend in Users table instead of only DB-recorded costs [skip-regression-check]

Chat turns record LLM cost in CostGuard (in-memory) but don't create
agent_jobs/llm_calls DB rows — those are only written for background
jobs. The Users table was querying only from DB, so it showed $0.00
for users who only chatted.

Now supplements DB stats with CostGuard.daily_spend_for_user() —
the same source displayed in the status bar token counter. Shows
whichever is larger (DB historical total vs live daily spend).

Also falls back to last_login_at for "Last Active" when no DB job
activity exists.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: persist chat LLM calls to DB and fix usage stats query

Two root causes for zero usage stats:

1. ChatDelegate only recorded LLM costs to CostGuard (in-memory) —
   never to the llm_calls DB table. Added DB persistence via
   TenantScope.record_llm_call() after each chat LLM call, with
   job_id=NULL and conversation_id=thread_id.

2. user_summary_stats query only joined agent_jobs→llm_calls, missing
   chat calls (which have job_id=NULL). Redesigned query to start from
   llm_calls and resolve user_id via COALESCE(agent_jobs.user_id,
   conversations.user_id) — covers both job and chat LLM calls.

Both PostgreSQL and libSQL queries updated. TenantScope gets
record_llm_call() method. Tests updated for new query semantics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review comments — input validation, cost semantics, panic safety [skip-regression-check]

- Validate display_name: trim whitespace, reject empty strings (create + update)
- Validate metadata: must be a JSON object, return 400 if not (admin + profile)
- secrets_list_handler: verify target user_id exists before listing
- Cost display: use DB total directly (chat calls now persist to DB),
  remove confusing max(db,live) CostGuard fallback
- CatchPanicLayer: truncate panic payload to 200 chars in log to limit
  potential sensitive data exposure

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address Copilot round 5 — docs, secrets consistency, token name, provider field [skip-regression-check]

- Docs: users.id note updated to "typically UUID v4 strings (bootstrap
  admin may use a custom ID)"
- secrets_list_handler: return 503 when DB store is None (was falling
  through to list secrets without user validation)
- tokens_create: trim + reject empty token name (matching display_name
  pattern)
- LlmCallRecord.provider: use llm_backend ("nearai","openai") instead
  of model_name() which returns the model identifier
- user_summary_stats zero-LLM users: acceptable — handler already falls
  back to 0 cost and last_login_at for missing entries

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: DB auth returns 503 on outage, scheduler counts only blocking jobs

From serrrfirat review:
- DB auth: return Err(()) on database errors so middleware returns 503
  instead of silently returning Ok(None) → 401 (auth miss)
- Scheduler: add parallel_blocking_count_for() that uses
  is_parallel_blocking() (Pending/InProgress/Stuck) instead of
  is_active() for per-user concurrency — Completed/Submitted jobs
  no longer count against MAX_JOBS_PER_USER

From Copilot:
- CLAUDE.md: fix secrets route paths from {id} to {user_id}
- token_hash: use .as_slice() instead of .to_vec() to avoid
  heap allocation on every token auth/creation call

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: immediate auth cache invalidation on security-critical actions (zmanian review #6)

Add DbAuthenticator::invalidate_user() that evicts all cached entries
for a user. Called after:
- Suspend user (immediate lockout, was 60s delay)
- Activate user (immediate access restoration)
- Role change (admin↔member takes effect immediately)
- Token revocation (revoked token can't be reused from cache)

The DbAuthenticator is shared (via Clone, which Arc-clones the cache)
between the auth middleware and GatewayState, so handlers can evict
entries from the same cache the middleware reads.

Also from zmanian's review:
- Items 1-5, 7-11 were already resolved in prior commits
- Item 12 (String→enum for status/role) is deferred as a broader refactor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: last-admin protection, usage stats for chat calls, UTF-8 safe panic truncation

Last-admin protection:
- Suspend, delete, and role-demotion of the last active admin now
  return 409 Conflict instead of succeeding and locking out the admin API
- Helper is_last_admin() checks active admin count before destructive ops

Usage stats:
- user_usage_stats() now includes chat LLM calls (job_id=NULL) by
  joining via conversations.user_id, matching user_summary_stats()
- Both PostgreSQL and libSQL queries updated

Panic handler:
- Use floor_char_boundary(200) instead of byte-index [..200] to
  prevent panic on multi-byte UTF-8 characters in panic messages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: workspace seed race, bootstrap atomicity, email trim, secrets upsert response [skip-regression-check]

- WorkspacePool: await seed_if_empty() synchronously after inserting
  into cache (drop lock first to avoid blocking), so callers see
  identity files immediately instead of racing a background task
- Bootstrap admin: use create_user_with_token() for atomic user+token
  creation, matching the admin create endpoint
- Email: trim whitespace, treat empty as None to prevent " " being
  stored and breaking uniqueness
- Secrets PUT: report "updated" vs "created" based on prior existence
- Last token_hash.to_vec() → .as_slice() in authenticate_token

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: disable unscoped webhook endpoint in multi-tenant mode [skip-regression-check]

The original /api/webhooks/{path} endpoint looks up routines across all
users. In multi-tenant mode, anyone who knows the webhook path + secret
could trigger another user's routine. Now returns 410 Gone with a
message pointing to the scoped endpoint /api/webhooks/u/{user_id}/{path}.

Detection uses state.db_auth.is_some() — present only when DB-backed
auth is enabled (multi-tenant). Single-user deployments are unaffected.

From: standardtoaster review comment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: webhook multi-tenant check, secrets error propagation, stale doc comment [skip-regression-check]

- Webhook: use workspace_pool.is_some() instead of db_auth.is_some()
  for multi-tenant detection — db_auth is set for any DB deployment,
  workspace_pool is only set when has_any_users() was true at startup
- Secrets: propagate exists() errors instead of unwrap_or(false) so
  backend outages surface as 500 rather than incorrect "created" status
- Config: fix stale workspace_read_scopes comment referencing user_id

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added scope: agent Agent core (agent loop, router, scheduler) scope: channel/cli TUI / CLI channel scope: channel/web Web gateway channel scope: db Database trait / abstraction scope: db/postgres PostgreSQL backend scope: db/libsql libSQL / Turso backend scope: llm LLM integration scope: config Configuration scope: sandbox Docker sandbox scope: docs Documentation scope: dependencies Dependency updates size: XL 500+ changed lines risk: high Safety, secrets, auth, or critical infrastructure contributor: core 20+ merged PRs labels Mar 28, 2026
henrypark133 and others added 14 commits March 28, 2026 14:46
* Clean up extension credentials on uninstall

* Address PR review feedback

* Cover channel webhook secrets on uninstall

* Harden tool secret cleanup detection
…rse_timestamp tests (#1700)

* fix(db): add tracing warn for naive timestamp fallback and improve parse_timestamp tests

* style: fix formatting
)

* fix(oauth): tighten legacy state validation and fallback handling

* style: fix formatting

* refactor: separate validation checks for clearer error messages
- Implement broadcast_dm() that creates a DM channel with the target
  user (POST /users/@me/channels, cached by Discord) and sends the
  message to it
- Extract DISCORD_API_BASE constant for all Discord REST API URLs
- Extract send_channel_message() shared helper to deduplicate message
  posting between on_respond and broadcast_dm
- Add snowflake validation on user_id before API calls
- Fix pre-existing clippy redundant_closure warning
- Use typed DmChannelResponse struct instead of serde_json::Value

Closes no specific issue — completes the previously stubbed on_broadcast.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…PTY (#1678)

- Add pty-process crate (MIT, tokio async support) for PTY allocation
- Spawn claude CLI with pty-process::Command::arg() chaining instead of
  building a shell string for script -qfc
- Eliminates all shell injection surfaces: prompt, model, session_id
  are passed via execve, never interpreted by a shell
- Keep stderr on separate pipe to prevent NDJSON parse breakage
  (pty-process attaches PTY to all fds by default)
- Gate PTY behind #[cfg(unix)] with direct-spawn fallback for Windows CI
- Read stdout from PTY master (implements tokio::io::AsyncRead)
- Add regression tests: arg vector construction + PTY allocation

Addresses review feedback from zmanian and gemini-code-assist.

Co-authored-by: j-bloggs <j-bloggs@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#1677)

* fix(worker): treat empty LLM response after text output as completion

When a job's LLM produces a substantive text response (e.g., formatted
results from a routine) and the next LLM call returns empty or errors,
the worker now treats this as successful completion instead of
continuing the loop until failure.

Previously, empty responses always triggered TextAction::Continue,
causing the loop to re-call the LLM. The LLM had nothing more to say,
so the provider returned "Response contained no message or tool call
(empty)". This made routine jobs that successfully produced results
report as "failed".

The fix adds a `has_text_response` flag to JobDelegate:
- After any non-empty text response: flag is set
- Empty text after flag is set: treated as completion
- LLM errors (select_tools/respond_with_tools) after flag: treated
  as completion instead of propagating
- Empty text before any output: still retries (rate-limit backoff)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(worker): restrict error swallowing to EmptyResponse variant only

- Add LlmError::EmptyResponse variant for when LLM returns no content
- Update nearai_chat and github_copilot providers to emit EmptyResponse
  instead of InvalidResponse for empty/no-choice responses
- try_complete_on_error now only swallows EmptyResponse (not AuthFailed,
  ContextLengthExceeded, Http, Io, etc.)
- Extract is_completion_eligible_error as testable pure function
- Log mark_completed errors at warn level instead of silently dropping
- Add EmptyResponse to retry and circuit breaker transient classifications
- Rewrite test to exercise real classification logic against all variants

Addresses review feedback from zmanian and gemini-code-assist.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(worker): extract mark_completed_or_warn helper to DRY completion logic

Extract shared mark-completed + warn-on-failure pattern into a single
helper method used by both try_complete_on_error and handle_text_response.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: j-bloggs <j-bloggs@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(routines): persist full LLM transcript and remove sandbox gate for full_job

Routine execution output was invisible — routine_fire returned a one-liner,
routine_history had no actual output, and the conversation thread contained
only a summary. Full-job routines also hard-failed without Docker.

Three fixes:

1. **Full transcript persistence**: execute_lightweight now persists every
   message (prompt, LLM responses, tool calls with params, tool results) to
   the routine's conversation thread as it executes, not just a summary
   after the fact.

2. **Routine output visibility**: routine_history includes conversation_id
   and recent_output messages. routine_fire tells the user to check
   routine_history. Web detail page has a "View Execution Thread" button
   that navigates to the chat tab. ROUTINE_OK stores "No issues found"
   instead of None. Full-job summary pulls actual job output instead of
   generic "Job X finished".

3. **Remove SandboxReadiness gate**: full_job routines dispatch through the
   scheduler like regular /job commands — no Docker required. The
   SandboxReadiness enum is removed entirely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply cargo fmt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(worker): treat AutonomousUnavailable tool errors as recoverable

The job worker crashed the entire job when a tool was denied for
autonomous execution (e.g. secret_list). The error was already recorded
in reason_ctx for the LLM to see, but process_tool_result_job returned
Err which propagated through the agentic loop and terminated the job.

Now all tool errors (including AutonomousUnavailable) return Ok,
letting the LLM see the denial and try a different approach.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): sanitize tool names for OpenAI Codex Responses API

The Codex API requires tool names to match `^[a-zA-Z0-9_-]+$` but
MCP/extension tools can have dots in their names (e.g. `mcp.server.tool`).
This caused HTTP 400 errors when the job worker sent tool calls back
to the LLM.

Sanitize tool names in both `convert_tool_definition` and
`convert_message` (function_call items) by replacing invalid characters
with underscores.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(routines): inject execution context into full_job description [skip-regression-check]

When a full_job routine dispatches a job, the LLM had no context that
it was already executing inside a routine. It wasted iterations on
infrastructure (discovering tools, creating routines, setting up auth)
instead of doing the actual work.

Prepend a clear directive to the job description telling the LLM that
tools and the routine are already configured, and to execute the task
directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): auto-refresh expired OAuth tokens on access [skip-regression-check]

When IronClaw restarts, MCP servers fail with "Secret has expired"
because get_access_token() checks token expiry locally and returns an
error before any HTTP request is made — so the existing 401-retry
refresh logic never triggers.

Now get_access_token() catches SecretError::Expired and automatically
calls refresh_access_token() using the stored refresh token. If the
refresh succeeds, the new token is returned transparently. If it fails,
the error message includes both the expiry and the refresh failure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): align refresh token naming and set expiry on stored tokens

Two bugs prevented MCP OAuth token auto-refresh on restart:

1. Naming mismatch: the hosted OAuth flow stored the refresh token as
   `{token_secret_name}_refresh_token` (e.g. `mcp_notion_access_token_refresh_token`)
   but `McpServerConfig::refresh_token_secret_name()` returned
   `mcp_notion_refresh_token`. The refresh token was there but unfindable.

2. Missing expiry: `store_tokens` in auth.rs never called `with_expiry()`
   even though `AccessToken::expires_in` was available. Combined with the
   fix from the previous commit (auto-refresh on Expired), tokens stored
   via the MCP auth flow will now also trigger refresh correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(web): show activity and transitions for agent jobs in job detail [skip-regression-check]

The job events endpoint only checked sandbox jobs for ownership,
returning 404 for agent jobs dispatched from routines. The detail
handler also returned empty transitions for agent jobs.

- events handler: fall back to agent job ownership check
- detail handler: populate transitions from job's state history

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(routines): expose max_iterations for full_job routines (default 25)

The max_iterations parameter was hardcoded to 10 and not configurable
via routine_create or routine_update, causing complex tasks to hit the
iteration cap.

- Add max_iterations to full_job execution schema (1-200, default 25)
- Thread it through parse → build → RoutineAction
- Support updating via routine_update
- Raise default from 10 to 25

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(routines): break self-dialogue loop after full_job plan execution

After plan execution, the completion-check Q&A ("Is the job complete?" /
"No, not complete...") was left in the message context, causing the
agentic loop to repeat the same analysis instead of calling tools.

Replace the stale dialogue with an action-oriented continuation prompt
that instructs the LLM to use tools for remaining work. Also strip
<suggestions> tags from all job output since they're only meaningful
for interactive chat sessions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(repl): prevent test hang in single-message mode

In single-message mode, start() stored a clone of the mpsc sender in
self.msg_tx for approval injection. After the thread sent /quit and
exited, the stored clone kept the stream alive, so stream.next()
blocked forever in the test assertion that the stream ends.

Skip storing the sender in single-message mode since interactive
approval is not needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(jobs): treat text responses as final answer in agentic loop

When the LLM produces a non-empty text response with no tool intent
(already filtered by the nudge mechanism), it is the job's final
answer. Previously, handle_text_response only exited the loop if the
text matched rigid completion phrases like "job is complete". Natural
summaries like "Weekly review completed and saved to Notion" were
added to context and the loop continued, causing the LLM to restate
the same summary until max_iterations was hit.

Now any non-empty text response marks the job complete and stops the
loop, matching the chat dispatcher behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(tests): reduce skills catalog network failure test from 10s to 1s

The test_search_returns_error_on_network_failure test connects to an
unreachable RFC 5737 TEST-NET IP and waited for the full 10s production
REQUEST_TIMEOUT. Add with_url_and_timeout test helper and use a 1s
timeout instead. [skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): accept 'message' as alias for 'content' in message tool

LLMs frequently call the message tool with {"message": "..."} instead
of {"content": "..."}. Fall back to the 'message' key when 'content'
is missing to avoid InvalidParameters errors during autonomous job
execution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): attach thread_id for gateway broadcast in message tool

When the message tool broadcasts to all channels (channel=null), it
sent an OutgoingResponse without a thread_id. The gateway silently
dropped these messages (returned Ok but never sent the SSE event),
so they appeared in repl but not in the web UI.

The thread_id was only populated when channel was explicitly "gateway".
Now it is always populated from notify_thread_id metadata, so
broadcast_all delivers to the gateway correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): return error instead of silently dropping messages

Gateway broadcast() and respond() previously returned Ok(()) when
thread_id was missing, silently swallowing the message. Callers
(message tool, agent loop) believed delivery succeeded when it didn't.

Now returns ChannelError::MissingRoutingTarget so callers can detect
and report the failure. Four regression tests verify the contract:
respond/broadcast with and without thread_id.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve rebase conflicts with staging

Restore sandbox_readiness field removed by pre-rebase commits (staging
still uses it). Update repl test to match staging's single-message
behavior (no longer sends /quit). Add missing reasoning field to
ToolCall in codex test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): log error when routine conversation lookup fails

The routine_history tool silently swallowed errors from
get_or_create_routine_conversation, returning empty output without
any diagnostic logging. Add tracing::warn so failures are visible
in logs. [skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR #1650 review comments

- E2E test: accept submitted/accepted as success states in job assertion
- TimeTool: remove operation from required schema (defaults to "now")
- jobs handler: log DB errors server-side, return generic message to client
- routines handler: use read-only find_routine_conversation on GET
- codex provider: reverse-map sanitized tool names so MCP tools resolve

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address zmanian review feedback on PR #1650

- MCP refresh token: fall back to legacy secret name (mcp_{name}_refresh_token)
  so existing users don't need to re-authenticate after the naming fix
- Job worker: replace fragile messages.pop() with truncate-to-saved-count
  to avoid maintenance hazard if message flow changes
- Document cost implications of max_iterations 10->25 default bump
- Revert Cargo.toml dist profile change (thin LTO comment, codegen-units=16)
  as it's unrelated to this PR

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve rebase conflicts and address new Copilot comments

- Fix no_silent_drop tests for updated GatewayConfig (user_id moved to
  GatewayChannel::new second arg, user_tokens removed)
- Fix handle_text_response param name (_reason_ctx -> reason_ctx)
- Fix missing has_text_response field in test JobDelegate
- Propagate row.get errors in find_routine_conversation instead of
  unwrap_or_default
- Only fall back to legacy refresh token name on NotFound/Expired,
  propagate real errors (DB, decryption)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(discord): restore gateway channel flow in wasm

* chore(discord): bump channel version to 0.2.1

* fix(discord): address review feedback on gateway channel PR

- Add #[serde(default)] to DiscordMessageMetadata for backward compat
  with old Option<String> serialized metadata
- Restore mention polling alongside Gateway (on_poll processes gateway
  events first, then runs poll_for_mentions if configured)
- Update on_respond to handle source_message_id with message_reference
  for mention-poll reply threading
- Implement Gateway presence status: dnd before pairing, online after
- Implement Gateway resume (OP 6) with session_id tracking, falling
  back to fresh identify on Invalid Session (OP 9)
- Extract WebsocketSessionState and spawn_websocket_poll to reduce
  nesting in start_websocket_runtime
- Simplify should_apply_dm_pairing tautology
- Remove completed plan docs
- Fix clippy items_after_test_module in extensions handler

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(discord): address review findings in gateway channel PR

- Fix gateway presence always showing "online" by filtering empty
  owner_id strings from workspace store reads
- Fix interaction followup using POST instead of PATCH to
  /messages/@original, which left deferred "thinking" state unresolved
- Restore mention-poll pagination (up to 5 pages of 100 messages)
- Remove dead ed25519-dalek and hex dependencies from WASM crate
- Remove unused _channel_id parameter from remember_processed_id
- Clean up redundant let binding in send_pairing_reply

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(discord): address second-round review findings

- Log warning when gateway event queue JSON fails to deserialize
  instead of silently returning empty (zmanian review item 1)
- Defer presence update from OP 10 Hello to after OP 0 READY, per
  Discord gateway protocol which requires READY before non-Identify
  commands (zmanian review item 2)
- Add 0-25% random jitter to websocket reconnect backoff per Discord's
  reconnection recommendations (zmanian suggestion)
- Extract WebsocketPollContext struct to replace 19-parameter
  spawn_websocket_poll function (zmanian suggestion)
- Document intent bitmask 4609 = GUILDS + GUILD_MESSAGES +
  DIRECT_MESSAGES in capabilities JSON (zmanian suggestion)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: zhyaoyu <zhyaoyu@aliyun.com>
Co-authored-by: ilblackdragon@gmail.com <ilblackdragon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#1667)

Add support for bundle layouts where directories without SKILL.md are
recursed into to find nested skills (e.g., skills/my-org/skill-a/SKILL.md).

- Add configurable max_scan_depth (SKILLS_MAX_SCAN_DEPTH env, default 3)
- Recurse into subdirectories lacking SKILL.md up to depth limit
- Share remaining discovery cap across recursive levels
- Replace try_exists + read_dir with single read_dir (eliminates TOCTOU)
- Box::pin recursive async calls for correct future sizing

Closes #1664

Co-authored-by: Rajul Bhatnagar <brajul@amazon.com>
* Clarify message tool and channel setup guidance

* Add target format hints to proactive messaging prompt

* Clarify search and message tool edge cases

* Fix stale tool_search e2e assertion

* Update src/llm/reasoning.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Tighten prompt guidance for message replies

* Format prompt guidance assertions

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: pin staging ci jobs to a single tested sha

* chore(ci): retrigger regression gate

[skip-regression-check]

---------

Co-authored-by: Firat Sertgoz <f@nuff.tech>
ilblackdragon and others added 19 commits March 29, 2026 23:51
… gate regressions (#1746)

* fix: resolve 11 test failures from multi-tenant bootstrap and sandbox gate regressions

Three root causes fixed:

1. Per-user bootstrap greeting in tests: After the multi-tenant isolation
   PR, `tenant_ctx("test-user")` creates a per-user workspace that seeds
   BOOTSTRAP.md and triggers an unwanted bootstrap greeting. This threw
   off response counting and caused message-drain races in 7 e2e tests.
   Fix: pre-seed the "test-user" workspace in the test rig DB so the
   first tenant_ctx call finds existing documents.

2. Sandbox gate blocking full_job routines: The full_job reliability
   overhaul (#1650) intended to remove the SandboxReadiness gate from
   execute_full_job (since full_job routines dispatch through the
   scheduler, not Docker). The gate was accidentally re-added during
   rebase, breaking 4 routine tests. Fix: remove the gate and clean up
   the unused sandbox_readiness field from EngineContext.

3. Owner-gate tests expecting old failure path: Two tests expected
   RunStatus::Failed from the sandbox gate. With the gate removed, the
   tool is now blocked at execution time by the approval context and the
   job completes normally. Fix: update traces and assertions to match the
   new behavior (RunStatus::Ok, owner_gate_count == 0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: keep DockerUnavailable gate for full_job routines

Only remove the DisabledByConfig gate — when sandbox is enabled but
Docker is unavailable, full_job routines should still fail rather than
silently running without the expected sandbox isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR review feedback

- Remove unused `include_completion` param from `owner_gate_trace()`
  and update all 5 call sites
- Use `.expect()` instead of `let _ =` on `seed_if_empty()` in test rig
  to surface seeding failures early
- Rename owner-gate tests from `_blocks_` to `_denies_tool_` to clarify
  the denial-with-success semantics

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: let per-user bootstrap fire naturally, filter in TestRig

Instead of pre-seeding the "test-user" workspace to prevent the
per-user bootstrap greeting, let it happen naturally and make the
TestRig resilient to it. `wait_for_responses` now transparently
filters bootstrap greetings from the response stream:

- Normal tests: all greetings filtered (bootstrap_greetings_to_keep=0)
- `.with_bootstrap()` tests: 1 greeting kept (the startup greeting),
  additional per-user duplicates filtered

Also updates owner-gate test section headers to match the
denial-with-success semantics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: assert tool denial event in owner-gate tests

The owner-gate denial tests previously only checked RunStatus::Ok +
owner_gate_count == 0, which could pass if the tool was never called
at all. Now both tests also verify that a tool_result event with
success=false exists for "owner_gate" in the job's event log,
confirming the tool was attempted and blocked by the approval context.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply rustfmt to collapsed function signature

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use async lock in bootstrap filter loop, add DisabledByConfig unit test

- Switch TestRig polling loop from `captured_responses()` (try_lock,
  panics on contention) to `captured_responses_async()` (async lock,
  safe under concurrent response pushing)
- Add unit test asserting DisabledByConfig does NOT match the
  DockerUnavailable gate (verifies the intended behavior change)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(slack): respond to thread replies in channels without requiring @mention

Two fixes:

1. Host bug: `on_respond` callback never committed workspace writes or
   injected workspace reader, unlike all other WASM callbacks. Any WASM
   channel persisting state during on_respond silently lost data.

2. Slack WASM channel: track threads where the bot has participated via
   workspace storage. When a message event arrives in a channel thread
   the bot previously replied to, process it without requiring @mention.

Closes #1404

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(slack): log workspace_write error instead of silently discarding

Address code review feedback: handle the Result from workspace_write
when tracking thread participation, logging a warning on failure
instead of using `let _ =` which would silently swallow errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(slack): harden thread reply tracking

---------

Co-authored-by: synner88 <29090601+synner88@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Firat Sertgoz <f@nuff.tech>
Co-authored-by: firat.sertgoz <firat.sertgoz@near.ai>
…esh (#1756)

* fix(routines): clone Arc before await in web handler event cache refresh (#1076)

Address review: drop superseded ticker changes, keep only the .cloned()
fix that prevents holding RwLockReadGuard across .await in toggle/delete
handlers. Add regression test for web toggle disabling a system_event
routine.

Closes #1076

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use explicit block to drop RwLockReadGuard before await

Address review feedback: in Rust 2024, `if let` scrutinee temporaries
live through the body, so the `.cloned()` approach still held the
RwLockReadGuard across `refresh_event_cache().await`. Extract into an
explicit block to ensure the guard is dropped, matching the existing
pattern in `routines_trigger_handler`.

Also add retry loop for `routine_by_name` in the integration test to
avoid flakiness from potential race conditions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1762)

* test(e2e): align wasm reinstall expectations with uninstall cleanup

* test(e2e): clarify wasm reinstall fixture semantics
…2238

chore: promote staging to staging-promote/d567d94c-23755250735 (2026-03-30 18:33 UTC)
…0735

chore: promote staging to staging-promote/d0f7862a-23733136790 (2026-03-30 16:19 UTC)
…6790

chore: promote staging to staging-promote/c75dea0e-23731217718 (2026-03-30 07:31 UTC)
…7718

chore: promote staging to staging-promote/8acdd080-23726030902 (2026-03-30 06:33 UTC)
…0902

chore: promote staging to staging-promote/368d2f52-23720185295 (2026-03-30 03:01 UTC)
…5295

chore: promote staging to staging-promote/70214c4a-23719079615 (2026-03-29 22:05 UTC)
…9615

chore: promote staging to staging-promote/86389dab-23706696435 (2026-03-29 21:07 UTC)
…6435

chore: promote staging to staging-promote/e0e530e6-23703082447 (2026-03-29 10:07 UTC)
…2447

chore: promote staging to staging-promote/a8e83210-23702343584 (2026-03-29 06:21 UTC)
…3584

chore: promote staging to staging-promote/8a320ae9-23693265249 (2026-03-29 05:32 UTC)
…5249

chore: promote staging to staging-promote/fd41bdf4-23691145719 (2026-03-28 20:05 UTC)
…5719

chore: promote staging to staging-promote/de5a1c7b-23688974037 (2026-03-28 18:06 UTC)
…4037

chore: promote staging to staging-promote/9bb19a98-23687925861 (2026-03-28 16:06 UTC)
…5861

chore: promote staging to staging-promote/9ba10eac-23686921981 (2026-03-28 15:07 UTC)
…1981

chore: promote staging to staging-promote/8f8cb7f7-23680994633 (2026-03-28 14:09 UTC)
@henrypark133
henrypark133 merged commit 71418bb into staging-promote/7234700c-23635804857 Mar 30, 2026
11 of 12 checks passed
@henrypark133
henrypark133 deleted the staging-promote/8f8cb7f7-23680994633 branch March 30, 2026 19:33
@github-actions github-actions Bot added scope: channel/wasm WASM channel runtime scope: tool/builtin Built-in tools scope: tool/wasm WASM tool sandbox scope: tool/mcp MCP client scope: worker Container worker scope: extensions Extension management scope: pairing Pairing mode scope: ci CI/CD workflows labels Mar 30, 2026
drchirag1991 pushed a commit to drchirag1991/ironclaw that referenced this pull request Apr 8, 2026
…3680994633

chore: promote staging to staging-promote/184d60ba-23635804857 (2026-03-28 08:09 UTC)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: high Safety, secrets, auth, or critical infrastructure scope: agent Agent core (agent loop, router, scheduler) scope: channel/cli TUI / CLI channel scope: channel/wasm WASM channel runtime scope: channel/web Web gateway channel scope: ci CI/CD workflows scope: config Configuration scope: db/libsql libSQL / Turso backend scope: db/postgres PostgreSQL backend scope: db Database trait / abstraction scope: dependencies Dependency updates scope: docs Documentation scope: extensions Extension management scope: llm LLM integration scope: pairing Pairing mode scope: sandbox Docker sandbox scope: tool/builtin Built-in tools scope: tool/mcp MCP client scope: tool/wasm WASM tool sandbox scope: worker Container worker size: XL 500+ changed lines staging-promotion

Projects

None yet

Development

Successfully merging this pull request may close these issues.