Skip to content

feat(tools): production-grade coding tools, file history, and skills - #2025

Merged
ilblackdragon merged 21 commits into
stagingfrom
feat/coding-tools-v2
Apr 10, 2026
Merged

ilblackdragon merged 21 commits into
stagingfrom
feat/coding-tools-v2

Conversation

@ilblackdragon

Copy link
Copy Markdown
Member

Summary

  • Add glob tool for fast file pattern matching (sorted by mtime, excludes .git/node_modules/target)
  • Add grep tool wrapping ripgrep with structured output modes (content/files_with_matches/count), pagination, and context lines
  • Add file_undo tool backed by in-memory FileHistory snapshots (supports binary files via Vec<u8>)
  • Enhance read_file with device path blocking (on resolved paths), binary detection, size limits
  • Enhance apply_patch with uniqueness validation and file history integration
  • Add bundled skills: coding, commit, code-review

Security hardening (from review)

  • Device/proc path blocking runs after validate_path() on resolved paths, preventing traversal bypass
  • Glob tool rejects absolute patterns and .. segments; strip_prefix defense-in-depth on all matches
  • Glob sync I/O wrapped in spawn_blocking
  • Grep tool injects ctx.extra_env matching ShellTool policy
  • Content mode path relativization uses per-line strip_prefix (not global string replace)
  • files_with_matches sorts globally before pagination

Supersedes #1841 (rebranched from staging to avoid v2-architecture merge conflicts).

Test plan

  • 64 unit tests pass across glob, grep, file, file_history modules
  • cargo fmt --all -- --check clean
  • cargo clippy --all-features zero warnings
  • check_no_panics.py passes

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings April 5, 2026 03:18
@github-actions github-actions Bot added scope: tool Tool infrastructure scope: tool/builtin Built-in tools scope: docs Documentation scope: dependencies Dependency updates size: XL 500+ changed lines risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs labels Apr 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the toolchain for “coding agent” workflows by adding dedicated search/discovery tools (glob, grep), file-level undo via in-memory snapshots, and tightening/standardizing filesystem safety behaviors. It also bundles several skills to guide agent behavior for coding, commits, and code review.

Changes:

  • Add new built-in tools: glob (pattern-based file discovery) and grep (ripgrep-backed search with structured outputs).
  • Add file_undo backed by a shared in-memory FileHistory, and integrate snapshotting into write_file / apply_patch.
  • Harden and extend filesystem tools (read_file size/line limits, binary detection, device/proc blocking; apply_patch uniqueness validation; shared excluded-dir list).

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/tools/registry.rs Registers new dev tools and injects shared FileHistory into file-modifying tools.
src/tools/builtin/path_utils.rs Introduces DEFAULT_EXCLUDED_DIRS constant for consistent filesystem-walk exclusions.
src/tools/builtin/mod.rs Exposes new built-in modules/tools (glob/grep/file_history).
src/tools/builtin/grep_tool.rs Implements grep tool with multiple output modes, truncation, and pagination.
src/tools/builtin/glob_tool.rs Implements glob tool with mtime sorting, default exclusions, and sandbox checks.
src/tools/builtin/file.rs Enhances read_file safety/limits; integrates file history into write_file / apply_patch; adds apply_patch validation.
src/tools/builtin/file_history.rs Adds FileHistory snapshot store and file_undo tool implementation.
skills/commit/SKILL.md Adds “commit” skill guidance for commit-message workflow.
skills/coding/SKILL.md Adds “coding” skill guidance emphasizing disciplined tool usage.
skills/code-review/SKILL.md Adds “code-review” skill guidance for review workflow.
Cargo.toml Adds glob dependency.
Cargo.lock Locks glob dependency for the workspace.
Comments suppressed due to low confidence (1)

src/tools/builtin/file.rs:218

  • lines[start_line..end_line] can panic when offset is greater than total_lines (e.g., offset=9999 on a short file), because start_line becomes > total_lines and the slice is out of bounds. Clamp start_line to total_lines (or return an empty result / InvalidParameters) before slicing to avoid crashing the tool.
            (total_lines, false)
        };

        let selected_lines: Vec<String> = lines[start_line..end_line]
            .iter()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/tools/builtin/grep_tool.rs Outdated
Comment on lines +20 to +30
/// Maximum output size before truncation (64KB, same as ShellTool).
const MAX_OUTPUT_SIZE: usize = 64 * 1024;

/// Default head limit for output lines/entries.
const DEFAULT_HEAD_LIMIT: usize = 250;

/// Safe environment variables forwarded to rg (same policy as ShellTool).
const SAFE_ENV_VARS: &[&str] = &[
"PATH", "HOME", "LANG", "LC_ALL", "LC_CTYPE", "TERM", "USER", "LOGNAME", "TMPDIR", "TMP",
"TEMP",
];

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says this uses the “same policy as ShellTool”, but the allowed env var list here is much narrower than ShellTool’s SAFE_ENV_VARS (e.g., missing PWD, LC_MESSAGES, XDG_* and toolchain vars). Either reuse the ShellTool allowlist (or move it to a shared helper) or update the comment/policy so they can’t silently diverge.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — removed the local copy. grep_tool now imports SAFE_ENV_VARS directly from shell.rs (made pub(crate)) so they can't diverge.

Comment thread src/tools/builtin/grep_tool.rs Outdated
Comment on lines +300 to +317
// Collect all file entries with mtime, sort globally, then paginate
let all_lines = &lines;
let mut file_entries: Vec<(String, SystemTime)> =
Vec::with_capacity(all_lines.len());
for line in all_lines {
let path = line.trim();
if path.is_empty() {
continue;
}
let mtime = tokio::fs::metadata(path)
.await
.and_then(|m| m.modified())
.unwrap_or(UNIX_EPOCH);
let relative = std::path::Path::new(path)
.strip_prefix(&search_path)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| path.to_string());
file_entries.push((relative, mtime));

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In files_with_matches mode this does an awaited tokio::fs::metadata call per match line, sequentially. For large result sets this can be very slow. Consider sorting without mtimes (or only fetching mtimes for the page being returned), or parallelize metadata collection (e.g., stream + buffer_unordered) / run it in spawn_blocking.

Suggested change
// Collect all file entries with mtime, sort globally, then paginate
let all_lines = &lines;
let mut file_entries: Vec<(String, SystemTime)> =
Vec::with_capacity(all_lines.len());
for line in all_lines {
let path = line.trim();
if path.is_empty() {
continue;
}
let mtime = tokio::fs::metadata(path)
.await
.and_then(|m| m.modified())
.unwrap_or(UNIX_EPOCH);
let relative = std::path::Path::new(path)
.strip_prefix(&search_path)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| path.to_string());
file_entries.push((relative, mtime));
// Collect all file entries with mtime, sort globally, then paginate.
// Metadata lookup is done with bounded parallelism to avoid
// sequentially awaiting one filesystem call per matched file.
let all_paths: Vec<String> = lines
.iter()
.map(|line| line.trim())
.filter(|path| !path.is_empty())
.map(|path| path.to_string())
.collect();
let mut file_entries: Vec<(String, SystemTime)> =
Vec::with_capacity(all_paths.len());
let mut join_set = tokio::task::JoinSet::new();
let mut pending = all_paths.into_iter();
let max_concurrency = 64usize;
while join_set.len() < max_concurrency {
let Some(path) = pending.next() else {
break;
};
let search_path = search_path.clone();
join_set.spawn(async move {
let mtime = tokio::fs::metadata(&path)
.await
.and_then(|m| m.modified())
.unwrap_or(UNIX_EPOCH);
let relative = std::path::Path::new(&path)
.strip_prefix(&search_path)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| path.clone());
(relative, mtime)
});
}
while let Some(join_result) = join_set.join_next().await {
if let Ok(entry) = join_result {
file_entries.push(entry);
}
if let Some(path) = pending.next() {
let search_path = search_path.clone();
join_set.spawn(async move {
let mtime = tokio::fs::metadata(&path)
.await
.and_then(|m| m.modified())
.unwrap_or(UNIX_EPOCH);
let relative = std::path::Path::new(&path)
.strip_prefix(&search_path)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| path.clone());
(relative, mtime)
});
}

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — replaced sequential await loop with tokio::task::JoinSet bounded to 64 concurrent metadata lookups.

Comment thread src/tools/builtin/glob_tool.rs Outdated
Comment on lines +104 to +108
if pattern.contains("..") {
return Err(ToolError::InvalidParameters(
"Glob patterns containing '..' are not allowed.".to_string(),
));
}

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pattern.contains("..") rejects any pattern containing the substring ".." (e.g. foo..bar), even when it’s not a parent-directory segment. If the intent is to block traversal segments, consider checking parsed path components for Component::ParentDir (handling both / and \\) rather than a substring match.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — now uses std::path::Component::ParentDir check instead of substring match, so patterns like foo..bar are no longer falsely rejected.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several new development tools, including GlobTool for pattern-based file discovery, GrepTool for content searching, and a FileHistory system with FileUndoTool for reverting changes. It also adds skill definitions for code review, coding best practices, and git commits. Existing tools like ReadFileTool and ApplyPatchTool were enhanced with security features such as blocked device paths, binary file detection, and uniqueness validation. Review feedback suggests utilizing dedicated path manipulation APIs instead of string operations for relativizing file paths to ensure correctness.

Comment on lines +366 to +373
// "content" mode — relativize paths per-line using strip_prefix
// to avoid false positives from substring replacement in file content
let search_prefix = format!("{}/", search_path.display());
let content: String = paginated
.iter()
.map(|line| line.strip_prefix(search_prefix.as_str()).unwrap_or(line))
.collect::<Vec<_>>()
.join("\n");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Relativizing paths by stripping a prefix is prone to errors if the prefix is not exactly matched. Consider using pathdiff::diff_paths or path.strip_prefix on the path object itself rather than string manipulation to ensure correctness.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code already uses std::path::Path::new(path).strip_prefix(&search_path) (Path object, not string manipulation) in files_with_matches and count modes. The content mode uses string prefix stripping which is safe since the prefix is the search_path directory with trailing slash.

Comment thread src/tools/builtin/file.rs Outdated
)));
}

// Binary file detection: read first 8KB and check for null bytes

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Severity

apply_patch now supports UTF-16LE text via the encoding-aware helpers, but read_file still probes for NUL bytes and then falls back to read_to_string() as UTF-8.

That means valid UTF-16LE text files will be rejected as "binary" here, and because apply_patch now requires read_file first, those files become impossible to patch through the intended flow.

read_file should reuse the same encoding-aware read path as apply_patch, then render the decoded text with line numbers.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — binary detection now checks for UTF-16LE BOM (FF FE) before the null-byte check, and read_file uses the encoding-aware read path from file_edit_guard::read_file_with_encoding() so UTF-16LE files are properly decoded.

Comment thread src/tools/builtin/file.rs Outdated
// Record the read for staleness detection
if let Some(ref read_state) = self.read_state {
let mtime = metadata.modified().unwrap_or(std::time::UNIX_EPOCH);
let partial = has_explicit_range;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High Severity

This records a default read_file call as a full read even when the response was truncated to the first 2000 lines.

partial is only derived from offset/limit, but the new default cap also produces a partial view. After reading the first 2000 lines of a 3000-line file, apply_patch / write_file will pass the guard and allow edits against unseen content.

partial should also be set when truncated_by_default is true, and there should be a test that a default-truncated read blocks later edits until the full file is read.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — partial is now set when has_explicit_range || truncated_by_default, so the default 2000-line cap correctly marks the read as partial and blocks edits until the full file is read.

Comment thread src/tools/builtin/grep_tool.rs Outdated
// Build output based on mode
let result = match output_mode {
"files_with_matches" => {
// Collect all file entries with mtime, sort globally, then paginate

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Severity

files_with_matches says it sorts globally by mtime before pagination, but by this point the raw rg output has already been truncated to 64KB.

So on large result sets you're only sorting the first chunk of ripgrep output, not the full match set. Newer files that fall after the first 64KB can never be returned, even if they should have been first after the mtime sort.

This mode needs a non-truncating collection path before the sort, or a different truncation strategy that happens after the full file list has been gathered.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the 64KB truncation happening before mtime sort. This is a known limitation — ripgrep itself streams output and we truncate at the collection layer. For the common case (< 64KB of matched file paths) the sort is correct. For pathological cases the user can narrow the search pattern. Will note this as a known limitation.

Comment thread src/tools/registry.rs
/// capabilities needed for the software builder. Call this after
/// `register_builtin_tools()` to enable code generation features.
pub fn register_dev_tools(&self) {
let file_history = shared_file_history();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High Severity

read_state and file_history are being allocated once inside register_dev_tools() and then shared through the registry. In this app, the registry itself is shared across sessions/threads, so the new stateful behavior is no longer scoped to a single conversation.

That creates two correctness problems:

  1. A read in session A can satisfy the "read before edit" guard for session B if they touch the same path.
  2. file_undo can restore a snapshot created by a different session, even though the tool description says it only undoes changes from the current session.

This needs to be keyed by session/thread/job, or injected from per-session context instead of living in registry-global state.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — both ReadFileState and FileHistory are now keyed by job_id (from JobContext). A read in session A cannot satisfy the guard for session B, and file_undo only restores snapshots from the same job. Added test_read_state_isolated_across_jobs test to verify.

ilblackdragon and others added 6 commits April 5, 2026 08:39
…ing skills

Add dedicated coding tools inspired by Claude Code's architecture to make
IronClaw a more effective coding assistant:

New tools:
- GlobTool: fast file pattern matching via `glob` crate, sorted by mtime,
  with default exclusions (.git, node_modules, target, etc.)
- GrepTool: content search wrapping ripgrep with 3 output modes
  (content, files_with_matches, count), pagination, and context lines
- FileUndoTool: restore files to pre-modification state using in-memory
  file history snapshots

Enhanced tools:
- ReadFileTool: 10MB limit, 2000-line default, binary detection, device
  path blocking (/dev/zero, /proc/*/fd/*)
- ApplyPatchTool: uniqueness validation (error on ambiguous matches),
  workspace path rejection, 10MB size limit, file history integration
- WriteFileTool: file history integration for undo support

Updated tool descriptions to guide LLM behavior (prefer apply_patch over
write_file, always read before editing, use glob/grep instead of shell).

New skills:
- coding: best practices for code editing, search, and file operations
- commit: git commit message generation workflow
- review: code review workflow with structured checklist

Shared infrastructure:
- DEFAULT_EXCLUDED_DIRS constant in path_utils.rs
- FileHistory module with SharedFileHistory for cross-tool snapshots

66 new tests covering all tools, edge cases, and regression scenarios.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… fixes

- Move device path blocking after validate_path() to prevent traversal bypass
- Add /proc/kcore, /proc/kmem to blocked paths
- Reject absolute patterns and '..' in glob tool, add strip_prefix defense
- Wrap glob sync I/O in spawn_blocking to avoid blocking tokio executor
- Sort files_with_matches globally before pagination in grep tool
- Add default exclusions for node_modules/target in grep tool
- Inject ctx.extra_env into rg environment matching ShellTool policy
- Use per-line strip_prefix for content mode path relativization
- Change FileSnapshot.content_before to Vec<u8> for binary file support
- Log snapshot errors with tracing::debug instead of silently discarding
- Fix skill name mismatch: code-review → review to match directory

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Aligns the directory name with the manifest name (code-review) to prevent
incorrect override/dedup behavior in the bundled-skill loader. The name
stays "code-review" since other domains may also need review-type skills.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ng, encoding preservation

Add file_edit_guard module with production-grade safeguards for file editing:
- ReadFileState tracks file reads with mtime for staleness detection
- 4-level fuzzy matching fallback (exact → whitespace-normalized → quote-normalized → both)
- UTF-16LE BOM detection and line ending style preservation (LF/CRLF/CR)
- Read-before-edit enforcement for ApplyPatch and WriteFile tools
- No-op edit rejection (old_string == new_string)
- Shared state injection via Arc<RwLock<>> across ReadFile, WriteFile, ApplyPatch

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lism, security

- Session-scoped state: ReadFileState and FileHistory now keyed by job_id
  so concurrent sessions sharing the same registry don't leak state (#2025)
- Parallel metadata: grep files_with_matches uses JoinSet (max 64 concurrency)
  instead of sequential await per file for mtime sorting
- Shared env allowlist: grep_tool imports SAFE_ENV_VARS from shell.rs
  (made pub(crate)) instead of maintaining a divergent copy
- Glob traversal: uses Component::ParentDir check instead of substring ".."
  match, so patterns like "foo..bar" are no longer falsely rejected
- UTF-16LE in read_file: binary detection skips null-byte check for files
  with UTF-16LE BOM; read_file uses encoding-aware read path
- Partial flag: default 2000-line truncation now marks read as partial,
  preventing edits against unseen content
- write_file guard softened: staleness check logs warning instead of
  hard error (full-file replacement has lower risk than apply_patch)
- Updated e2e trace to include read_file before apply_patch
- Updated expected tool list in schema validation tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 5, 2026 09:06
@ilblackdragon
ilblackdragon force-pushed the feat/coding-tools-v2 branch from b91390c to b3eaa23 Compare April 5, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 16 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

src/tools/builtin/file.rs:226

  • offset is documented as 1-indexed, but start_line is computed directly from offset without clamping to total_lines. If the caller passes an offset beyond EOF, lines[start_line..end_line] will panic (start > end). Consider clamping start_line to total_lines (and returning 0 lines shown) so out-of-range offsets are handled safely.
        let start_line = if offset > 0 {
            offset.saturating_sub(1)
        } else {
            0
        };

        let (end_line, truncated_by_default) = if let Some(lim) = limit {
            ((start_line + lim as usize).min(total_lines), false)
        } else if !has_explicit_range && total_lines > DEFAULT_LINE_LIMIT {
            // Apply default 2000-line limit when no offset/limit specified
            (DEFAULT_LINE_LIMIT.min(total_lines), true)
        } else {
            (total_lines, false)
        };

        let selected_lines: Vec<String> = lines[start_line..end_line]
            .iter()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/tools/builtin/file.rs
Comment on lines 150 to +164
let path = validate_path(path_str, self.base_dir.as_deref())?;

// Block device paths that would hang or produce infinite output.
// Check the resolved path to prevent bypasses via relative paths or symlinks.
let resolved_str = path.to_string_lossy();
if BLOCKED_DEVICE_PATHS
.iter()
.any(|p| resolved_str.starts_with(p))
|| (resolved_str.starts_with("/proc/") && resolved_str.contains("/fd/"))
{
return Err(ToolError::InvalidParameters(format!(
"Reading device/proc paths is not allowed: {}",
path.display()
)));
}

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Device/proc blocking relies only on the resolved path from validate_path(). Because validate_path() canonicalizes existing absolute paths, /proc/self/fd/* will typically resolve to something like /dev/pts/* (or a pipe), which bypasses the /proc/.../fd check and may block/hang reads. Consider also checking the original path_str (pre-canonicalization) for /proc/*/fd/ (and similar like /dev/fd/) and rejecting those regardless of how they resolve.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point about canonicalization resolving /proc/self/fd/* to /dev/pts/*. The current check runs on the resolved path which catches the common cases. Adding pre-canonicalization checks would improve defense-in-depth — will address in a follow-up.

Comment thread src/tools/builtin/file.rs
Comment on lines +372 to +376
// Snapshot existing file before overwriting (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

write_file snapshots the entire existing file contents before overwriting. There’s no size guard here, so overwriting a very large file could read it fully into memory and store it in FileHistory, causing high memory usage/DoS. Suggest checking existing file size (metadata.len) before snapshotting, and either (a) refuse, (b) cap snapshots to a maximum size, or (c) store an on-disk temp snapshot instead of in-memory for large files.

Suggested change
// Snapshot existing file before overwriting (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);
// Snapshot existing file before overwriting (for file_undo).
// Guard against unbounded memory usage by skipping snapshots for large files.
const MAX_HISTORY_SNAPSHOT_SIZE_BYTES: u64 = 10 * 1024 * 1024;
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
match fs::metadata(&path).await {
Ok(metadata) if metadata.is_file() => {
if metadata.len() <= MAX_HISTORY_SNAPSHOT_SIZE_BYTES {
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);
}
} else {
tracing::debug!(
"skipping file_history snapshot before write_file for {}: file size {} exceeds limit {}",
path.display(),
metadata.len(),
MAX_HISTORY_SNAPSHOT_SIZE_BYTES
);
}
}
Ok(_) => {
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);
}
}
Err(_) => {
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);
}
}

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid concern. The snapshot is bounded by MAX_PATCH_SIZE (10MB) for apply_patch, but write_file doesn't have the same guard on the existing file. Will add a size cap on snapshots in a follow-up — files over a threshold should skip snapshotting or use on-disk temp files.

Comment thread src/tools/builtin/file.rs
Comment on lines +356 to +366
// Staleness check for existing files — log a warning but don't hard-fail.
// write_file replaces the entire file content, so the risk of overwriting
// unseen content is lower than apply_patch (which does substring replacement).
if path.exists()
&& let Some(ref read_state) = self.read_state
{
let current_mtime = fs::metadata(&path)
.await
.and_then(|m| m.modified())
.unwrap_or(std::time::UNIX_EPOCH);
let state = read_state.read().await;

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

path.exists() performs blocking filesystem I/O via std, and this runs inside an async execute() method. Prefer using tokio::fs::try_exists() / fs::metadata().await.is_ok() (you already call fs::metadata below) to avoid blocking the async runtime.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — replaced the blocking path.exists() with the fs::metadata(&path).await call that immediately follows it. The metadata result now serves double duty (existence check + mtime).

Comment thread src/tools/builtin/file.rs
Comment on lines +745 to 802
// Try to find the old_string, with fuzzy matching fallbacks
let (match_count, actual_old_string, match_method) = {
let (count, method) = file_edit_guard::count_matches(&content, old_string);
if count > 0 {
// Determine the actual string to replace
let actual = if method == MatchMethod::Exact {
old_string.to_string()
} else if let Some(m) = file_edit_guard::find_match(&content, old_string) {
m.actual
} else {
old_string.to_string()
};
(count, actual, method)
} else {
// Provide a helpful error with the old_string included
let preview = if old_string.len() > 200 {
format!("{}...", &old_string[..200])
} else {
old_string.to_string()
};
return Err(ToolError::ExecutionFailed(format!(
"String to replace not found in {}.\n\
old_string:\n{}\n\n\
Make sure old_string matches the file content exactly, \
including whitespace and indentation.",
path.display(),
preview
)));
}
};

// Uniqueness validation
if !replace_all && match_count > 1 {
return Err(ToolError::ExecutionFailed(format!(
"Could not find the specified text in {}. Make sure old_string matches exactly.",
"Found {} matches for the specified text in {}. \
Provide more context in old_string to make it unique, or set replace_all=true.",
match_count,
path.display()
)));
}

// Apply replacement
// Snapshot before modification (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "apply_patch", 0).await {
tracing::debug!("file_history snapshot failed before apply_patch: {}", e);
}
}

// Apply replacement using the actual matched string
let new_content = if replace_all {
content.replace(old_string, new_string)
content.replace(&actual_old_string, new_string)
} else {
content.replacen(old_string, new_string, 1)
content.replacen(&actual_old_string, new_string, 1)
};

// Count replacements
let replacements = if replace_all {
content.matches(old_string).count()
} else {
1
};
let replacements = if replace_all { match_count } else { 1 };

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fuzzy matching + replace_all can produce incorrect behavior. match_count is computed on normalized content, but the replacement uses content.replace(&actual_old_string, ...) where actual_old_string is a single concrete substring from the first match. If multiple fuzzy matches exist with different original formatting (e.g., differing trailing whitespace / quote variants), match_count may be >1 but replace() may update fewer occurrences, and replacements will be reported incorrectly. Consider either rejecting replace_all when match_method != Exact, or implementing a replacement strategy that replaces all normalized matches while preserving original spans, and compute replacements from the actual number of edits performed.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good edge case. When match_method != Exact, the actual_old_string is derived from the first match and may not match all normalized variants. For now, replace_all with fuzzy matching will replace however many instances match the concrete string. Rejecting replace_all for non-exact matches would be safer — will evaluate in a follow-up.

Comment on lines +309 to +321
// Seed initial batch
for path in pending.by_ref().take(max_concurrency) {
let sp = search_path.clone();
join_set.spawn(async move {
let mtime = tokio::fs::metadata(&path)
.await
.and_then(|m| m.modified())
.unwrap_or(UNIX_EPOCH);
let relative = std::path::Path::new(&path)
.strip_prefix(&sp)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| path.clone());
(relative, mtime)

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In files_with_matches mode, ripgrep typically returns paths relative to the current working directory (or to the search root), but this code calls tokio::fs::metadata(&path) on the raw string. If path is relative, metadata will be resolved relative to the process CWD, which can fail and force mtime=UNIX_EPOCH, breaking sort-by-mtime and potentially mis-ordering results. Consider resolving each path against search_path (e.g., let abs = if Path::new(&path).is_absolute() { ... } else { search_path.join(&path) }) before metadata() and before strip_prefix.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We pass --path-separator / and the full absolute search_path to rg, so output paths are absolute. But you're right that defensively resolving relative paths would be more robust. Will address.

Comment on lines +370 to +377
for line in &paginated {
if let Some((file, count_str)) = line.rsplit_once(':') {
let count = count_str.trim().parse::<u64>().unwrap_or(0);
let relative = std::path::Path::new(file)
.strip_prefix(&search_path)
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|_| file.to_string());
total += count;

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to files_with_matches, count mode assumes file strings can be strip_prefix'd against search_path, but ripgrep output is often relative, so strip_prefix(&search_path) will fail and absolute/odd paths may be returned. Consider resolving file against search_path first (or using pathdiff-style relativization from an absolute path) so returned paths are consistently relative to the search root.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above — rg receives an absolute search path so output is absolute, but defensive resolution would be better. Will address together with the files_with_matches fix.

Comment on lines +161 to +172
// Build rg command
let mut cmd = Command::new("rg");
cmd.env_clear();
for (key, val) in safe_env() {
cmd.env(&key, &val);
}

// Re-inject explicitly approved environment from the job context,
// mirroring ShellTool's behavior so tools can access required credentials.
for (key, val) in _ctx.extra_env.as_ref() {
cmd.env(key, val);
}

Copilot AI Apr 5, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool shells out to rg, but the repo’s sandbox container image (docker/sandbox.Dockerfile) doesn’t appear to install ripgrep. If grep is intended to run inside that sandbox (ToolDomain::Container), it will always fail at runtime with the “rg is not installed” error. Consider adding ripgrep to the sandbox image build (or otherwise guaranteeing rg is present in the execution environment) so the tool is usable in production.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — ripgrep needs to be added to the sandbox Dockerfile. Will address in a separate PR for the container image.

ilblackdragon and others added 2 commits April 5, 2026 12:58
…rite_file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The check_no_panics.py lexer misinterpreted Rust lifetimes ('static) as
char literal starts, causing in_char state to persist across lines and
hide all subsequent brace-delimited blocks — including #[cfg(test)] mod
tests. Reset in_char at line boundaries since Rust char literals cannot
span lines.

https://claude.ai/code/session_012bJjER6L5zSAFd9BqYMUmC
Copilot AI review requested due to automatic review settings April 5, 2026 14:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Remove redundant double-pass through .lines() — the first
collect+join was a no-op since .lines() already handles line endings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 10, 2026 09:44
@ilblackdragon

Copy link
Copy Markdown
Member Author

Updates after latest staging merge

Merged latest staging and addressed the following:

Fixed in this push

  1. CI formatting fixed — cargo fmt diffs from merge resolved
  2. Wasmtime 43 cache config — removed enabled = true from TOML (wasmtime 43 dropped that field); updated test assertions
  3. Removed .fmt-test — accidental test artifact deleted
  4. Simplified strip_trailing_whitespace — removed redundant double .lines() pass in file_edit_guard.rs

Self-review items status

  • Remove /.fmt-test — Done
  • turn_number plumbing — Still always 0 from callers. Low priority since it's used in undo output and tests but doesn't affect correctness. Will address when we add proper turn tracking.
  • Total-bytes cap in file_history — Deferred to follow-up (tracked)
  • Simplify strip_trailing_whitespace — Done
  • grep mtime sort optimization — Deferred to follow-up

Pre-existing test failures

tool_error_recovery and workspace_semantic_search in e2e_advanced_traces fail on staging too (before our branch). The trace fixtures expect tool call counts that don't match the current agent dispatcher behavior — unrelated to this PR.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/tools/builtin/file.rs Outdated
Comment on lines +793 to +794
let preview = if old_string.len() > 200 {
format!("{}...", &old_string[..200])

Copilot AI Apr 10, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

old_string[..200] can panic on non-ASCII input because Rust string slicing requires a UTF-8 char boundary. If a user provides an old_string containing multi-byte characters and length > 200 bytes, this will crash the tool. Truncate the preview on a char boundary (e.g., walk back to is_char_boundary, or build the preview via chars().take(...)).

Suggested change
let preview = if old_string.len() > 200 {
format!("{}...", &old_string[..200])
let preview = if old_string.chars().count() > 200 {
format!("{}...", old_string.chars().take(200).collect::<String>())

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 489612d — now uses old_string.chars().take(200).collect::<String>() for char-boundary-safe truncation.

Comment thread src/tools/builtin/file.rs
Comment on lines +389 to +395
// Snapshot existing file before overwriting (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "write_file", 0).await {
tracing::debug!("file_history snapshot failed before write_file: {}", e);
}
}

Copilot AI Apr 10, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

write_file snapshots the existing file by reading it fully into memory with no size guard. Overwriting a very large file can cause high memory usage (and the snapshot queue holds up to 50 entries), which is a realistic resource-exhaustion/DoS vector. Add a maximum snapshot size (or skip snapshotting / store snapshots on disk) based on file metadata before calling FileHistory::snapshot.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 489612d — added MAX_SNAPSHOT_FILE_SIZE (10MB) check in FileHistory::snapshot(). Files exceeding the limit are silently skipped with a debug log.

Comment thread src/tools/builtin/file.rs
Comment on lines +818 to +824
// Snapshot before modification (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "apply_patch", 0).await {
tracing::debug!("file_history snapshot failed before apply_patch: {}", e);
}
}

Copilot AI Apr 10, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FileHistory::snapshot is always called with turn_number = 0 here, so file_undo will report restored_from_turn: 0 regardless of when the edit happened. If turn numbers aren’t available at tool-execution time, consider removing/renaming this field (e.g., to restored_from_snapshot_id), making it optional, or plumbing a real turn counter into the tool context so this metadata is accurate.

Suggested change
// Snapshot before modification (for file_undo)
if let Some(ref history) = self.file_history {
let mut h = history.write().await;
if let Err(e) = h.snapshot(ctx.job_id, &path, "apply_patch", 0).await {
tracing::debug!("file_history snapshot failed before apply_patch: {}", e);
}
}
// Snapshot before modification (for file_undo) is intentionally skipped here.
// This path does not have an accurate turn number available, and passing a
// hard-coded value would record misleading `restored_from_turn` metadata.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 489612d — turn_number parameter removed entirely. FileHistory now auto-assigns monotonically increasing sequence_number values.

.unwrap_or(DEFAULT_MAX_RESULTS as u64) as usize;

let start = std::time::Instant::now();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Severity — Missing is_sensitive_path check

GlobTool does not check is_sensitive_path() on the search root or matched paths. The other file tools (ReadFileTool, WriteFileTool, ApplyPatchTool, ListDirTool) all guard against credential directories (.ssh, .aws, .gnupg, etc.), but GlobTool skips this check.

This means {"pattern": "*", "path": "~/.ssh"} would enumerate files in sensitive directories. Not a blocker since GlobTool only returns paths (not content) and read_file would still block reading them, but it is a defense-in-depth gap.

Suggested fix: Add is_sensitive_path(&search_root) after validate_path, and filter matched paths with is_sensitive_path inside the spawn_blocking loop (mirroring ListDirTool's skip logic).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 489612d — added is_sensitive_path check on the search root after validate_path, plus per-match filtering inside the spawn_blocking loop (mirroring ListDirTool's skip logic).

.and_then(|v| v.as_bool())
.unwrap_or(false);
let type_filter = params.get("type_filter").and_then(|v| v.as_str());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Severity — Missing is_sensitive_path check

GrepTool does not check is_sensitive_path() on the search path. Same defense-in-depth gap as the GlobTool comment, but slightly higher risk here because GrepTool returns file content (in content and count modes), not just paths.

Example: {"pattern": "AKIA", "path": "/home/user/.aws"} could surface AWS access key IDs in grep output.

Not a blocker — requires the LLM to actively target credential directories, and sandboxed deployments mitigate this — but worth closing the gap for parity with the other file tools.

Suggested fix: Add is_sensitive_path(&search_path) check after validate_path, consistent with ReadFileTool and ListDirTool.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 489612d — added is_sensitive_path check on the search path after validate_path, consistent with the other file tools.

serrrfirat
serrrfirat previously approved these changes Apr 10, 2026

@serrrfirat serrrfirat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — solid work. Path validation, env scrubbing, fuzzy matching, and the read-before-edit guard are all well-implemented.

Two inline comments on missing is_sensitive_path checks in GlobTool and GrepTool — defense-in-depth gaps, not blockers. Can be addressed in a follow-up.

- Add is_sensitive_path checks to GlobTool and GrepTool, matching the
  defense-in-depth posture of ReadFileTool/WriteFileTool/ListDirTool
- Fix UTF-8 panicking byte-index slice in apply_patch error preview
  (old_string[..200] → chars().take(200))
- Add 10MB size guard on file_history snapshots to prevent memory
  exhaustion from snapshotting large files
- Replace dead turn_number field with auto-incrementing sequence_number
  in FileHistory — callers no longer pass a hardcoded 0
- Fix glob mtime test flakiness by increasing sleep to 1100ms (above
  1s filesystem granularity)
- Fix emoji test to actually include emoji/non-ASCII content

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ilblackdragon
ilblackdragon merged commit 2cc5546 into staging Apr 10, 2026
15 checks passed
@ilblackdragon
ilblackdragon deleted the feat/coding-tools-v2 branch April 10, 2026 16:37
@claude

claude Bot commented Apr 10, 2026

Copy link
Copy Markdown

Code review

Found 8 issues:

  1. [HIGH:87] Unbounded memory growth in FileHistory - up to 50 snapshots × 10MB = 500MB per registry instance

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file_history.rs#L22-L27

The FileHistory stores snapshots in a VecDeque with max 50 snapshots. While there's protection against files >10MB, under concurrent load with many job_ids, old entries aren't cleaned up from the ReadFileState HashMap. At production scale (10K+ short-lived jobs), this accumulates unbounded memory. Recommendation: attach read-state to JobContext lifecycle for cleanup or implement LRU eviction.

  1. [MEDIUM:75] Off-by-one error in fuzzy byte offset mapping for patch application

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file_edit_guard.rs#L164-L177

In map_normalized_char_to_original_byte(), when handling the newline after a line:

if normalized_char_idx == normalized_seen + 1 {
    return Some(original_byte + 1);  // original_byte is at line.len(), not the newline
}

This returns a byte position pointing after the newline, not at the newline itself. This causes fuzzy-matched patches with replace_all=true to extract incorrect byte ranges, potentially corrupting files when the old_string has trailing whitespace differences.

  1. [MEDIUM:70] Path traversal TOCTOU between validation and write

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file.rs#L159-L175

The validate_path() function canonicalizes symlinks, but between validation and the actual write in apply_patch or write_file, an attacker could replace a real file with a symlink. Mitigations exist (layered validation), but recommend re-validating immediately before write or using O_NOFOLLOW.

  1. [MEDIUM:65] Unbounded allocation in ripgrep metadata collection spawns excessive tasks

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/grep_tool.rs#L312-L353

For files_with_matches mode, metadata is collected with 64 concurrent tasks BEFORE truncation to max_results. A malicious glob could match thousands of files and spike FD/memory usage. Recommendation: truncate results before metadata collection or lower concurrency bound.

  1. [MEDIUM:60] Missing HashSet optimization for path exclusion checks

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/glob_tool.rs#L42-L51

The DEFAULT_EXCLUDED_DIRS is a slice (&[&str]), searched linearly. With 200 glob results × 50 path components × O(6) lookup, this is acceptable now but should use phf::Map or once_cell::Lazy<HashSet> for production scale.

  1. [MEDIUM:50] UTF-16LE decoder silently discards incomplete final byte

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file_edit_guard.rs#L408-L411

The decoder uses chunks_exact(2) which silently drops any remaining odd byte:

let u16s: Vec<u16> = data
    .chunks_exact(2)
    .map(|pair| u16::from_le_bytes([pair[0], pair[1]]))
    .collect();

If a UTF-16LE file has an odd number of bytes after BOM stripping, the final byte is lost. Recommendation: validate (data.len() % 2) == 0 or use chunks(2) with error handling.

  1. [LOW:65] Silent failures in file history snapshots only logged at debug level

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file.rs#L223-L230

File snapshot failures in apply_patch and write_file are caught and logged at debug level, allowing edits to proceed with partial undo capability. This is intentional (per comments), but users won't know file history is broken. Recommendation: consider a warning-level log or capability flag.

  1. [LOW:50] ReadFileState HashMap grows unbounded with job_ids

https://github.com/anthropics/ironclaw/blob/45cece552df2517c64fb14ab8a73c714d79f429b/src/tools/builtin/file_edit_guard.rs#L26-L50

The ReadFileState::update_mtime() silently does nothing if a file was never read. Combined with unbounded HashMap growth, long-running services accumulate entries. Recommendation: add max-entries bound or LRU eviction, or attach state to JobContext for lifecycle management.

ilblackdragon added a commit that referenced this pull request Apr 10, 2026
…ismatch

The file_history snapshot/restore tests were failing on macOS because
`/var` is a symlink to `/private/var`. `snapshot()` stored the
original path (`/var/folders/.../code.rs`), but `execute()` called
`validate_path()` which canonicalizes to `/private/var/folders/...`.
The path comparison in `restore_latest` mismatched, returning "No
file history found" even though the snapshot existed.

Fix: canonicalize paths consistently at both the storage boundary
(`snapshot()`) and the lookup boundary (`latest_snapshot_for()`,
`snapshots_for()`, `restore_latest()`). A shared `canonical()` helper
handles the non-existent-file case (write_file's "new file" path)
by canonicalizing the parent directory and joining the filename —
the parent always exists even when the file itself doesn't yet.

This was a pre-existing staging failure from PR #2025 that affected
all macOS developers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
henrypark133 added a commit that referenced this pull request Apr 10, 2026
…ce race (#2209)

* fix(workspace): collapse reindex delete+insert into one transaction to close TOCTOU race

Two concurrent reindexers for the same document could both DELETE the
existing chunks, then both try to INSERT chunk_index 0, hitting the
UNIQUE (document_id, chunk_index) constraint and failing with
"database is locked" or constraint violation. The delete and inserts
were separate libsql transactions with async points between them.

Add `WorkspaceStore::replace_chunks(document_id, &[ChunkWrite])` that
runs DELETE + N INSERTs inside a single BEGIN IMMEDIATE transaction
(not the default DEFERRED — DEFERRED bypasses busy_timeout on the
first write contention). The libsql impl, postgres impl, and the
in-memory storage variant all go through the new method, and
`Workspace::reindex_document` builds the `ChunkWrite` Vec (with
embeddings) up front so nothing async happens between the delete and
the insert loop.

Regression test in `workspace::versioning_tests` spawns 4 concurrent
writers against the same document on a multi-thread runtime and
asserts last-writer-wins without UNIQUE collisions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(auth): resolve display name + extension target from action when surfacing auth gates

The engine's `ResumeKind::Authentication` only carries `credential_name`
(e.g. `google_oauth_token`), which was being used as both the
user-facing display string AND as the first argument to
`submit_auth_token`. Two failure modes:

1. Display: users saw "google_oauth_token" in the auth-required prompt
   instead of the friendly extension name "google-drive-tool".
2. Routing: `submit_auth_token` expects an *extension* name and walks
   the extension's capabilities file to find the actual secret. Passing
   `google_oauth_token` directly fails closed with "Extension not
   installed: google_oauth_token", trapping the user in a re-auth loop
   on every paste.

`bridge/router.rs` now resolves the actual extension via
`tools.provider_extension_for_tool(action_name)` for both the gate
display path and the `submit_auth_token` call. Built-in tools, HTTP,
and skill credentials still fall back to the credential name (the
existing behaviour for those callers).

`extensions/manager.rs` fixes three more auth-readiness traps surfaced
by the v2 Drive trace:

- All capabilities lookups (`auth_wasm_tool`, channel activate, setup
  schema, configure, explicit secret query, upgrader) now go through
  `load_tool_capabilities` / `load_channel_capabilities` so a tool
  installed under the legacy hyphen filename
  (`google-drive-tool.capabilities.json`) is still resolved when
  looked up by canonical underscore name. The pre-v0.23 layout
  silently reported `no_auth_required` and bypassed the gate
  entirely.

- `activate_wasm_tool` now uses `existing_extension_file_path` for
  both the `.wasm` and `.capabilities.json` lookups so the legacy
  hyphen filename is resolved here as well. Without this, the
  upstream `determine_installed_kind` happily reported the extension
  as installed via its own alias check, but `activate_wasm_tool`
  then failed with `NotInstalled` — the readiness probe fell back to
  "treat as ready" and the agent ended up calling a tool that
  couldn't activate, hit a 401/403, and looped trying to recover.

- `configure()` post-activation OAuth cleanup now skips deletion when
  the caller is *also* providing a fresh credential in the same
  `secrets` map. The previous behaviour wrote the user's pasted token
  then immediately deleted it (along with `_scopes` /
  `_refresh_token` siblings), causing the resume to hit the wrapper
  with `token_exists=false` and re-fire the gate forever. Explicit
  Reconfigure (empty secrets map) still wipes the records to kick
  off a fresh OAuth dance.

`config/mod.rs` test config now seeds a deterministic 32-byte master
key so replay-mode tests that touch credentials get a working secrets
store out of the box without each test having to build its own.

Three regression tests in `extensions::manager::tests`:
- `test_activate_wasm_tool_finds_legacy_hyphen_alias`
- `test_auth_wasm_tool_finds_legacy_hyphen_alias`
- `test_configure_preserves_oauth_token_when_caller_provides_it`
- `test_configure_clears_oauth_token_for_reconfigure_flow`

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): canonicalize MCP tool identifiers to snake_case at registration

MCP tool names commonly contain dashes (e.g. Notion's `notion-search`),
and so do user-supplied server names (`my-server`). The runtime
converges on snake_case identifiers per `ToolRegistry::resolve_name`,
and LLMs (Codex / GPT-5 in particular) silently normalize tool names
to valid Python identifiers by converting dashes to underscores. The
old code built the registry key as `format!("{server}_{tool}")` and
preserved any dashes from the original tool name, so the registry got
`notion_notion-search` while the LLM emitted `notion_notion_search` —
direct lookup missed and the legacy alias fallback (which only goes
underscores → dashes) couldn't reconstruct the mixed-separator form
either, leaving every Notion MCP tool unreachable.

Add `mcp_tool_id(server, tool)` in `tools/mcp/client.rs` that does
`format!(...).replace('-', "_")`, re-export it from `tools/mcp/mod.rs`,
and use it in:

- `McpClient::create_tools` — the prefixed_name on every wrapped MCP
  tool now agrees with what the LLM will emit
- `ExtensionManager::activate_mcp` — `tool_names` is now sourced from
  `tool_impls.iter().map(|t| t.name())` instead of being independently
  rebuilt from the raw McpTool list, eliminating drift between
  registered names and reported names
- `ExtensionManager::latent_actions_for_mcp_server` — latent provider
  actions surfaced before activation use the same canonical form

The original (possibly hyphenated) `t.name` is still preserved on the
wrapper's inner `McpTool` and used verbatim when forwarding the
`tools/call` request to the MCP server, so MCP protocol compatibility
is unchanged — the canonicalization is internal-only.

5 regression tests in `tools::mcp::client::tests`:
- `test_mcp_tool_id_canonicalizes_dashed_tool_name`
- `test_mcp_tool_id_canonicalizes_dashed_server_name`
- `test_mcp_tool_id_passthrough_for_already_canonical_names`
- `test_create_tools_canonicalizes_dashed_mcp_tool_names` (drives
  `create_tools` end-to-end via MockTransport)
- `test_create_tools_round_trips_through_registry_resolve_name`
  (caller-level test per `.claude/rules/testing.md` — registers the
  wrapped tools in a real `ToolRegistry` and asserts that
  `resolve_name("notion_notion_search")` returns the registered tool
  via the direct HashMap path, not via the legacy alias fallback)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): flatten top-level schema unions for OpenAI + symmetric Python/Rust action_calls round-trip

Two related bugs in the LLM ↔ engine boundary, both surfaced by the
GitHub Copilot MCP and the v2 orchestrator:

1. **Top-level schema flatten.** OpenAI's tool API rejects schemas
   whose top level isn't `type: "object"` or that contain top-level
   `oneOf`/`anyOf`/`allOf`/`enum`/`not`, with HTTP 400:

       Invalid schema for function '<name>': schema must have type
       'object' and not have 'oneOf'/'anyOf'/'allOf'/'enum'/'not' at
       the top level.

   The GitHub Copilot MCP's `github` tool uses top-level `oneOf` for
   action dispatch, so the agent was 400-ing the moment it tried to
   enumerate tools. `normalize_schema_strict` (already shared between
   the OpenAI Codex provider and `RigAdapter::convert_tools`) now
   short-circuits when it sees a forbidden top-level construct,
   replacing `parameters` with a permissive object envelope
   (`{type: "object", properties: {}, additionalProperties: true,
   required: []}`) and appending the original schema to the tool
   description as advisory text (truncated on a char boundary at 1500
   bytes). The MCP server still validates the actual shape on its
   end, so the tool keeps working — we just lose API-level schema
   enforcement and the LLM has to read variant structure from the
   description.

   The function signature changes to take `&mut String` for the
   description (to append the hint). Both call sites
   (`openai_codex_provider::convert_tool_definition` and
   `rig_adapter::convert_tools`) pass an owned clone through.

   This is slightly lossy for Anthropic users on tools with top-level
   unions (Claude could have handled the union natively), but
   keeping a single normalizer for all rig-based providers is simpler
   than threading per-provider flags through the adapter, and the
   description hint preserves the variant info Claude needs.

2. **Python ↔ Rust `action_calls` field-name mismatch.** The Python
   orchestrator (`default.py`) appends assistant messages with
   `action_calls=calls` where each call is shaped
   `{name, call_id, params}` (the friendly Python names produced by
   `orchestrator.rs:handle_llm_complete`). The reverse parser
   `json_to_thread_messages` tried to deserialize via
   `serde_json::from_value::<Vec<ActionCall>>`, which expects the
   canonical Rust field names `{action_name, id, parameters}`. The
   deserialize fails, but `.ok()` swallows the error and the
   assistant message comes back with `action_calls = None`. Every
   subsequent tool result then looks orphaned to
   `sanitize_tool_messages` and gets rewritten as a user message,
   losing the model's ability to reason about prior tool calls.

   Introduce a private `PythonActionCall` interchange struct as the
   single source of truth for the field naming convention, with
   bidirectional `From` conversions. Both call sites
   (Rust → Python serialization + Python → Rust deserialization)
   now go through `action_calls_to_python_json` /
   `python_json_to_action_calls`, so any future field addition only
   needs to touch one struct definition.

   `ActionCall` itself is unchanged — adding `#[serde(rename = ...)]`
   would have invalidated every persisted Step record and ThreadEvent.

11 regression tests:
- `rig_adapter::tests::test_normalize_schema_strict_*` (6 tests)
  covering pass-through, top-level oneOf flatten, anyOf/allOf/enum/not
  flatten, non-object replacement, nested-oneOf preservation, and
  char-boundary truncation
- `rig_adapter::tests::test_convert_tools_handles_top_level_oneof_dispatcher`
  (caller-level test driving `convert_tools` end to end)
- `openai_codex_provider::tests::test_convert_tool_definition_handles_top_level_oneof_dispatcher`
  (caller-level test driving the codex provider path)
- `executor::orchestrator::tests::python_action_call_round_trips_through_serde`
- `executor::orchestrator::tests::action_calls_to_python_json_uses_python_field_names`
- `executor::orchestrator::tests::python_json_to_action_calls_parses_python_field_names`
- `executor::orchestrator::tests::python_json_to_action_calls_rejects_canonical_field_names`
  (guards against silent shape drift)
- `executor::orchestrator::tests::json_to_thread_messages_preserves_action_calls_from_python_orchestrator`
  (caller-level test feeding the literal JSON shape `default.py`
  produces, asserting the assistant message's `action_calls` survive
  the round-trip with matching call_ids)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(live): add Drive auth-gate round-trip live test + supporting harness pieces

End-to-end smoke test for the post-flight auth gate path that the
recent auth-postflight commits stitched together. Two phases:

  Phase A: delete the developer's real `google_oauth_token` (and the
  refresh token) from the test rig's libsql DB while keeping the
  `_scopes` companion record so `auth_wasm_tool`'s scope expansion
  check doesn't fire on the re-store. Send a Drive search prompt and
  assert the agent emits `StatusUpdate::AuthRequired` within one
  iteration. The expected path:
    1. agent calls `google-drive-tool { action: "list_files" }`
    2. wrapper's `resolve_host_credentials` reports
       `missing_required = ["google_oauth_token"]`
    3. wrapper fails closed with the
       "requires credentials that are not configured" message
    4. effect adapter's post-flight branch fires
       `auth::postflight::detect_post_call_auth_failure`
    5. matcher hits the `requires credentials + not configured` pair
       (commit cd8b68de)
    6. detector calls `ensure_extension_ready(.., ExplicitAuth)` →
       `EnsureReadyOutcome::NeedsAuth`
    7. `EngineError::GatePaused { resume_kind: Authentication }`
       bubbles to the orchestrator
    8. router stores it in `pending_gates` and emits
       `StatusUpdate::AuthRequired`

  Phase B: re-insert the captured token via `secrets_store()`, send
  the synthetic value as a follow-up message. The v2 router treats the
  next user message after an auth gate as
  `GateResolution::CredentialProvided`, which calls `submit_auth_token`
  (idempotent overwrite of what we just inserted) then
  `execute_pending_gate_action` → `execute_resolved_pending_action`,
  and the original Drive call replays. The test asserts the resume
  ran (additional tool activity + a follow-up response).

Live-tier only (`#[ignore]`); skipped outside `IRONCLAW_LIVE_TEST=1`.
The test deliberately does NOT commit a recorded trace fixture: any
trace would inevitably capture the bearer token, real Drive file
metadata, and HTTP headers — all PII that's hard to scrub safely.
Hermetic regression coverage for the underlying alias-aware
capabilities bug lives in
`test_auth_wasm_tool_finds_legacy_hyphen_alias`.

Supporting harness changes:

- `LiveTestHarnessBuilder::with_no_trace_recording()` — opt-out flag
  for tests that touch real credentials. Live mode still runs against
  the real LLM but no fixture is committed; replay mode builds a stub
  harness so the test can detect the mode and skip gracefully without
  panicking on a missing fixture.

- `LiveTestHarness::finish_turns(&[(user_input, responses)])` —
  multi-turn variant of `finish` for tests that span an auth-gate
  round-trip (prompt → AuthRequired → token → resume). The session
  log renders all turns in order so a reader can follow the full
  conversation, not just the first prompt. Status events are still
  rendered once at the top because the rig doesn't tag them with a
  turn boundary.

- Session log formatter now renders `StatusUpdate::AuthRequired` and
  `StatusUpdate::AuthCompleted` so the gate is visible in the log.

- `TestRig::secrets_store()` and `TestRig::owner_id()` accessors so
  live tests can manipulate credentials directly under the same scope
  the agent loop uses.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): log on python_json_to_action_calls deserialize failure

Address PR #2209 review: the helper used `serde_json::from_value(...).ok()?`
which is the exact `.ok()` swallow pattern the parent commit set out to
fix. If a future Python orchestrator patch ever drifts the action_calls
shape (extra required field, rename, partial migration), the helper would
silently return None and every subsequent tool result would look orphaned
to `sanitize_tool_messages` again — with no operator-visible signal at all.

Replace with an explicit match that emits a `warn!` (with the parse error
and the offending JSON value) on the failure path so the breadcrumb is
visible the moment any drift happens. The `None` return is preserved so
existing callers and the
`python_json_to_action_calls_rejects_canonical_field_names` test still
hold.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(test): generate random master key per Config::for_testing call

Address PR #2209 review: the previous fix hardcoded
`0123456789abcdef...0123456789abcdef` as the AES-256-GCM master key
inside `pub fn for_testing`. The function is `pub` (gated only behind
`#[cfg(feature = "libsql")]`, not `#[cfg(test)]`, because integration
tests in `tests/*.rs` are separate crates compiled against the lib's
non-test surface), which meant every developer building with libsql
had a publicly-known master key sitting in their process — and the
constant was now baked into Git history forever.

Replace with `generate_test_master_key()`, a private helper that pulls
32 bytes from `rand::thread_rng()` and hex-encodes them. Each call
returns a fresh key. Tests don't need cross-process determinism: each
test creates its own temp DB and the secrets store is born fresh on
every call anyway. `rand 0.8` is already a direct workspace dependency
so no Cargo changes are needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): normalize all non-identifier characters in mcp_tool_id

Address PR #2209 review: `mcp_tool_id` only handled `-` → `_`, but the
MCP spec doesn't actually constrain tool names to OpenAI's
`^[a-zA-Z0-9_-]{1,64}$` regex — a server could legally return
`notion.search`, `notion:create_issue`, `files/read`, or names with
spaces or non-ASCII characters. The same LLM normalization that bites on
`-` will bite on `.` and `:` too, and `extract_server_name` only strips
`.` from the host portion of a URL, leaving the tool portion of the
prefixed name unprotected.

Replace the single `.replace('-', "_")` with a `chars().map()` pass that
sends every non-`[A-Za-z0-9_]` character to `_`. This handles dashes,
dots, colons, slashes, spaces, and unicode in one shot — and since the
chars iterator yields one Rust char per code point, multi-byte
characters become a single `_` rather than splitting weirdly.

New regression test `test_mcp_tool_id_normalizes_non_identifier_chars`
covers dot, colon, slash, space, and multi-byte unicode inputs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): make flatten_top_level hint keyword-aware

Address PR #2209 review: the description hint appended by
`flatten_top_level` was a one-size-fits-all "pick one variant and pass
its fields as a flat object". That's correct for top-level `oneOf` and
`anyOf`, but actively misleading for the other forbidden constructs:

- `allOf` — the LLM should pass fields from ALL variants combined,
  not pick one
- `enum` — the LLM should pass one of the listed literal values,
  not "fields"
- `not` — the LLM should pass any object that does NOT match the
  constraint

Extract `FORBIDDEN_TOP_LEVEL` to a module-level constant (now shared
between `needs_top_level_flatten` and a new `detect_forbidden_top_level`
helper) and add `schema_flatten_hint_intro(detected)` which branches on
the actual keyword that triggered the flatten and returns a precise
intro string. Falls back to a "free-form object" message when the
schema wasn't an object at all (no recognized forbidden keyword, just
the wrong top-level type).

New regression test `test_normalize_schema_strict_hint_is_keyword_aware`
asserts that each of the 5 keywords produces a hint containing the
expected discriminating phrase.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(workspace): serialize postgres replace_chunks via FOR UPDATE on parent doc

Address PR #2209 review (Copilot, src/workspace/repository.rs:350):
the libsql `replace_chunks` is fine because `BEGIN IMMEDIATE` acquires
the writer lock at transaction start, but the postgres path used the
default-isolation `BEGIN` which is not equivalent. Two concurrent
reindexers running under separate snapshots can both DELETE (each sees
its own pre-delete state, neither sees the other's), then race to
INSERT chunk_index 0 and hit the `UNIQUE (document_id, chunk_index)`
constraint.

Add `SELECT 1 FROM memory_documents WHERE id = $1 FOR UPDATE` at the
top of the transaction. The `FOR UPDATE` row lock is per-document,
ties to the existing parent row (FK already in place from
`memory_chunks.document_id`), and is released automatically on
commit/rollback. Concurrent reindexers for the same doc now serialize
on the parent row and last-writer-wins cleanly.

Picked `FOR UPDATE` over `pg_advisory_xact_lock` because it's the
row-locking primitive PG operators expect when reading the code, and
it doesn't introduce a hash function dependency for the lock key.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): merge top-level union variants into flatten_top_level envelope

Address PR #2209 review (gemini-code-assist, src/llm/rig_adapter.rs:296):
after flattening a top-level oneOf/anyOf/allOf, the LLM was left with
`properties: {}` and could only read variant fields from the description
hint. That works but it's lossy — the LLM can't do schema-based
reasoning about which fields exist, and the description hint is
truncated to 1500 bytes so deeply-nested schemas are unreadable.

Add `merge_top_level_variant_properties` which walks the union variants,
collects every property they declare, and returns a single map. The
flatten envelope now uses that map instead of empty `{}`, so the LLM
sees structured field hints. `additionalProperties: true` and
`required: []` are preserved, so strict-mode validation stays disabled
and the LLM is free to mix fields across variants — the upstream MCP
server enforces the actual constraints on its end.

First-write wins on conflicting types: if two variants declare the
same field with different schemas, the first variant's schema is kept.
The full original schema still goes into the description hint, so the
ambiguous case is recoverable from there.

Two new regression tests:
- `test_normalize_schema_strict_merges_variant_properties` exercises a
  GitHub-Copilot-shaped tool with two variants that share a
  discriminator and asserts every field from every variant survives.
- `test_normalize_schema_strict_merge_first_write_wins_on_conflict`
  pins the documented conflict-resolution behaviour.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(router): extract resolve_extension_for_action helper for 3 dup sites

Address PR #2209 review (henrypark133, src/bridge/router.rs:1606): the
`provider_extension_for_tool + unwrap_or_else(credential_name)` pattern
was implemented in three places — once via the
`resolve_auth_gate_display_name` helper at line ~64, and twice inlined
in `resolve_gate` (line ~1603) and `await_thread_outcome` (line ~2779).
The inline sites couldn't use the helper because they'd already
destructured `credential_name` from the `ResumeKind` match and needed
the result for `submit_auth_token`, not just display.

Extract the core into `async fn resolve_extension_for_action(tools,
action_name, credential_fallback) -> String`. Make
`resolve_auth_gate_display_name` a thin wrapper that handles the
non-Authentication ResumeKind variants. Both inline sites now call the
helper directly with the destructured `credential_name`. The two
inline-site comment blocks that explained the rationale are collapsed
into shorter "see helper for full rationale" pointers since the doc
on `resolve_extension_for_action` carries the full explanation now.

Three sites collapse to one implementation. The auth display + routing
logic now has a single source of truth.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): warn on post-normalization tool name collisions in create_tools

Address PR #2209 review (serrrfirat, src/tools/mcp/client.rs:682):
after the broader `mcp_tool_id` char normalization (commit 18d4ce48),
two MCP tools whose names differ only by `-` vs `_` (e.g. `search-all`
and `search_all`) collide on the same registry key. The second
`ToolRegistry::register` call silently shadows the first with no
signal at all — operators debugging an unreachable tool would have
zero breadcrumb to discover the collision.

Add collision detection in `McpClient::create_tools` itself, where we
still have both the original tool name and the normalized id. Build a
`HashMap<normalized_id, original_name>` while iterating, and emit a
`tracing::warn!` when two distinct originals collide on the same id.
The warn carries the normalized id, both colliding original names,
and the server name, so an operator can immediately see which upstream
tools to rename. Behaviour is unchanged — the second tool still wins
on register, matching what the LLM would emit anyway since it
normalizes both names to the same string.

The collision detection is scoped to a single MCP server's tool list
because cross-server collisions have different registry-key prefixes
(`server_a_foo` vs `server_b_foo`) and can't actually shadow each
other. This is the right level — `ToolRegistry::register` itself
doesn't have access to the pre-normalization name and couldn't emit
this signal even if we wanted it there.

New regression test
`test_create_tools_handles_post_normalization_collision` drives a
MockTransport that lists `search-all` and `search_all`, asserts both
wrappers are produced with the same `Tool::name()`, registers them in
a real `ToolRegistry`, and asserts last-write-wins on shadow.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(engine): drop entries from action_calls_to_python_json on failure instead of injecting null

Address PR #2209 review (serrrfirat,
crates/ironclaw_engine/src/executor/orchestrator.rs:1920): the previous
helper used `unwrap_or_else(|_| Value::Null)` which silently corrupts
the array when serialization fails. The Python orchestrator
(`default.py`) accesses `c.get("name")` / `c.get("call_id")` /
`c.get("params")` on each entry, so a `null` would crash with a Python
`AttributeError` and lose the entire LLM step — and the fallback
contradicts this PR's own stated goal of not silently swallowing
errors.

Replace with `filter_map` so a failed entry is dropped from the output
rather than corrupting it. The warn log on the failure path is
preserved (and now also includes `action_name` for easier
correlation). Python's tool-result loop iterates
`range(len(results))` against the same shortened call list so a
missing entry is benign.

Note: the failure path is essentially unreachable for the
`PythonActionCall` shape (`String + String + Value` all
infallible-to-serialize) but the contract should still be safe — the
helper will be touched again when the Python interchange shape
evolves and we don't want a future maintainer to discover this trap
the hard way.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(engine): summarize action_calls in warn log to avoid leaking PII

Address PR #2209 review (serrrfirat,
crates/ironclaw_engine/src/executor/orchestrator.rs:1999): the
`python_json_to_action_calls` warn log emitted `value = %value` which
dumps the full action_calls JSON array on parse failure. Tool params
can carry user PII (search queries, file names, email content,
conversation text), and the warn fires precisely when the Python ↔
Rust shape drifts — exactly the moment operators will be grepping
logs and shipping output to log aggregators (Datadog, CloudWatch,
Sentry).

Add `summarize_action_calls_for_log` which builds a structural-only
summary: array length and the keys of the first entry. The keys
themselves are static field names (`name`, `call_id`, `params`), not
user data. The shape summary is enough to debug a drift (operator can
see whether the shape is roughly right and which fields are missing)
without exposing any of the actual parameter contents.

Edge cases handled:
- empty array → "empty array"
- non-array value (Python passed wrong shape) → "non-array value of
  type <kind>" via a small `json_value_type_name` helper
- entries that aren't objects → "<not an object>" rather than
  attempting to walk them

Two regression tests:
- `summarize_action_calls_for_log_does_not_leak_user_pii` builds an
  intentionally PII-laden value with salary spreadsheet queries,
  credentials, and "private message about layoffs" content, asserts
  none of the user-content strings appear in the summary, AND that
  even the upstream tool name doesn't leak (operator-level intent
  signal).
- `summarize_action_calls_for_log_handles_edge_cases` pins the
  empty/string/object/null fallback paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* incorporate #2227: factory server_name normalization, registry bidirectional alias, WASM/channel loader normalization, legacy token fallback

Merge the non-overlapping changes from PR #2227
(fix-tool-name-hyphen-normalization) so this PR supersedes it. Our PR
already fixed the core issue (mcp_tool_id canonicalization + stricter
non-identifier-char normalization), but #2227 adds valuable
defense-in-depth and compatibility layers that we didn't cover:

- `tools/mcp/factory.rs` — normalize `server.name` at the factory
  boundary (before any branch including the OAuth early-return). This
  ensures the secret name, session key, and tool prefix all use the
  same underscore-only form. Our PR normalized only at the tool-id
  level which left the server_name itself hyphenated in the session
  manager and token secret store.

- `tools/mcp/client.rs` — normalize `new_with_name` so callers that
  pass a hyphenated server name get consistent behavior even when
  bypassing the factory.

- `tools/mcp/config.rs` + `tools/mcp/auth.rs` — legacy token secret
  name fallback for pre-normalization tokens. When checking if an MCP
  server is authenticated, if the canonical (underscore) secret name
  doesn't exist, try the legacy (hyphenated) form. This prevents
  forcing re-auth on existing users who stored tokens under the old
  hyphenated server name before upgrade.

- `tools/registry.rs` — bidirectional `resolve_key` helper that tries
  exact → hyphen→underscore → underscore→hyphen aliases in `get`,
  `has`, `unregister`, `resolve_name`, `get_resolved`,
  `provider_extension_for_tool`, and `tool_definitions_for_actions`.
  Defense-in-depth: even if a tool somehow ends up registered with a
  mixed-separator name (edge case, stale DB, manual insertion), the
  registry will still find it. 4 new regression tests cover both
  alias directions, get_resolved, and unregister via alias.

- `tools/wasm/loader.rs` + `channels/wasm/loader.rs` — `load_from_dir`
  and `discover_*` functions normalize hyphenated filenames (file stem
  → replace('-', "_")). Dev tool install name changed from
  `{name}-tool` to `{name}_tool`.

Conflict resolution: manager.rs (kept our version with comment),
client.rs create_tools (kept our `mcp_tool_id` which is strictly
better — handles ALL non-identifier chars, not just dashes), client.rs
tests (kept our comprehensive MockTransport-based test suite, dropped
#2227's simpler duplicate). All other hunks from #2227 applied cleanly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(manager): normalize server name prefix in starts_with tool-list filters

Address PR #2209 review (henrypark133, Critical C1): the 3
`starts_with(&format!("{}_", name))` filters in `activate_mcp`
(already-active fast path), `list()`, and `remove()` used the raw
(possibly hyphenated) server name, while `mcp_tool_id` normalizes the
tool registry keys to underscores-only. A hyphenated server name like
`my-server` produced a prefix `my-server_` that matched zero tools
(they're all `my_server_*`), returning empty tool lists and failing to
unregister on extension removal.

Fix: use `crate::tools::mcp::mcp_tool_id(name, "")` as the prefix.
This produces `my_server_` from `my-server`, matching the registered
keys exactly. All 3 sites now use the same normalization as tool
registration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): accept array type containing "object" in needs_top_level_flatten

Address PR #2209 review (henrypark133, Critical C2):
`needs_top_level_flatten` only matched `JsonValue::String("object")`
for the type check. A top-level `"type": ["object", "null"]` (valid
JSON Schema for a nullable object, produced by some upstream providers
and `make_nullable`) triggered `bad_type = true` and the schema was
flattened, silently discarding all its properties.

Extend the check to also accept `JsonValue::Array` when any element is
the string `"object"`. This prevents unnecessary flattening of schemas
that are semantically object-typed but use the array form for
nullability.

New regression test:
`test_normalize_schema_strict_does_not_flatten_nullable_object_type`

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): update seen_ids on collision so 3rd collision reports against 2nd

Address PR #2209 review (henrypark133, Nit N1): the collision detection
in create_tools skipped the `seen_ids.insert` on the collision branch,
so a 3rd colliding tool would report against the 1st original name
instead of the 2nd (the actual shadow). Added the insert inside the
warning branch so subsequent collisions report the correct chain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(wasm): fix CI type mismatch + add string-matching fallback for wrapped traps

Address PR #2209 CI failure and Copilot review comments:

1. **CI fix (E0308):** `classify_trap_error` took `anyhow::Error` but
   wasmtime 43's `call_execute` returns `wasmtime::Error` (a distinct
   type in `wasmtime_internal_core`). Changed the signature to accept
   `wasmtime::Error` directly. This is also more correct — accepting
   the native error type preserves type information that a lossy
   `.into()` conversion would strip, making the structured `Trap`
   downcast more reliable.

2. **String-matching fallback (Copilot, wrapper.rs:1107):** The doc
   claimed "falls back to string matching" but the implementation only
   did the structured downcast. Added a string-matching fallback that
   checks the full Display chain for "all fuel consumed", "out of fuel",
   "OutOfFuel", and "unreachable" when the downcast fails. This covers
   the case where component-model glue or host wrappers bury the Trap
   inside layers that `downcast_ref` can't see through. New regression
   test `trap_classification_fuel_via_string_fallback` exercises this
   path using a plain `wasmtime::Error::msg` wrapper.

3. **Stale doc comment (Copilot, test_rig.rs:1282):** Updated
   `secrets_store()` doc to reflect that most test rigs now have a
   working secrets store because `Config::for_testing()` generates a
   random master key per call. `None` only occurs with a config
   override that explicitly disables secrets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review: tighten unreachable trap match, cap schema serialization, document legacy auth

Address PR #2209 review (serrrfirat, 4 comments):

1. `wrapper.rs:1131` — tightened the string fallback from bare
   `contains("unreachable")` to `contains("unreachable code")` /
   `"UnreachableCodeReached"` / `"wasm trap: unreachable"`. The old
   match would false-positive on HTTP errors like "endpoint was
   unreachable" or "server unreachable: connection refused", replacing
   the real diagnostic chain with a generic message.

2. `rig_adapter.rs:326` — added `count_json_nodes` pre-check (cheap
   recursive walk, no alloc) before calling `serde_json::to_string`.
   A malicious MCP server with a many-MB schema would have triggered
   a proportional allocation even though we only keep 1500 bytes.
   Schemas over 5000 nodes skip serialization entirely and get a
   "(schema too large to inline)" placeholder instead.

3. `auth.rs:1202` — documented that the legacy token fallback
   intentionally uses bare `get_decrypted` (no refresh). The path is
   transitional: users re-auth once and get migrated to the canonical
   naming scheme. Wiring refresh through the legacy path adds
   complexity for a self-healing compat layer.

4. `rig_adapter.rs:171` — Anthropic lossiness was already documented
   in the `normalize_schema_strict` doc comment (lines 164-170).
   Reply-only; per-provider flag is a follow-up.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): ensure array items is a JSON Schema object for OpenAI strict mode

OpenAI rejects array-typed properties whose `items` field is missing,
boolean (`true`), or any non-object value with:

  "array schema items is not an object"

Schema generators like schemars produce `{"type": "array"}` (no items)
or `{"type": "array", "items": true}` for `Vec<serde_json::Value>`,
which is valid JSON Schema but violates OpenAI's strict-mode rules.
The google_docs_tool's `requests: Vec<serde_json::Value>` field
triggered this on every tool enumeration.

In `normalize_schema_recursive`, detect array-typed properties and
ensure `items` is a JSON Schema object before recursing. Missing or
non-object `items` are replaced with `{}` (accept any item). Object
`items` are left untouched and recursed into as before.

Regression test covers all three cases: missing items, boolean items
(`true`), and well-formed items (must not be clobbered).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): add post-normalization validation to catch schema rules the normalizer misses

The schema_validator module already knew the "array items must be an
object" rule (Rule 8, line 218), had a test for it
(test_array_missing_items_fails, line 315), and would have caught the
google_docs_tool 400 — but it was only wired into CI tests against
built-in tools, never applied to WASM/MCP tool schemas or to the
output of normalize_schema_strict.

The root cause pattern: we're playing whack-a-mole with OpenAI's
undocumented strict-mode rules, adding fixes one at a time when a new
tool exposes a schema shape the normalizer doesn't handle. Each time,
the fix is a runtime 400 in production that takes a PR cycle to fix.

The structural fix: run validate_strict_schema as a debug-level
post-check after normalization. If the normalizer missed something,
the diagnostic appears in local logs immediately (before the schema
even reaches the LLM provider), giving developers a local breadcrumb
instead of a runtime 400 from OpenAI. The schema still goes through
(the tool remains usable), and the LLM provider surfaces the 400 if
OpenAI actually rejects it — but now the cause is instantly visible
in `RUST_LOG=ironclaw::llm=debug` output.

This also means that any future normalizer rule we add gets automatic
regression coverage: if the normalizer introduces a bug that violates
a rule the validator knows about, the debug log fires on every tool
call in dev mode.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): normalize merged properties on flatten path + silence null action_calls warn

Two runtime issues from the latest deploy:

1. The flatten path in `normalize_schema_strict` short-circuited with
   `return schema`, skipping BOTH the recursive normalizer AND the
   post-normalization validator. The merged properties copied from
   union variants were raw — a `Vec<serde_json::Value>` field like
   google_docs_tool's `requests` kept its bare `{"type": "array"}`
   without `items`, and OpenAI rejected it with "array schema items
   is not an object" on every tool-using call.

   Fix: the flatten path now normalizes each merged property
   individually via `normalize_schema_recursive(prop_schema)` before
   returning the envelope. The top-level envelope stays permissive
   (`additionalProperties: true`, `required: []`) so the LLM can
   mix fields across variants, but each property's internal schema
   gets the full treatment (array items, nested objects, etc.).

   The post-normalization validator also runs on both paths now
   (no early return before it).

   Regression test:
   `test_normalize_schema_strict_flatten_normalizes_merged_array_items`
   mimics the google_docs_tool shape (tagged enum with `oneOf`, one
   variant containing an items-less array) and asserts the merged
   `requests` property has `items` as an object after normalization.

2. `python_json_to_action_calls` warn log fired on every text-only
   assistant message with "invalid type: null, expected a sequence"
   because Python's `action_calls: null` (legitimate "no tool calls"
   signal) was passed to the parser. Added a `.filter(|v| !v.is_null())`
   before the parser call in `json_to_thread_messages` so null is
   treated the same as "key absent" — no parse attempt, no false
   alarm. The warn only fires for genuinely malformed data now.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): replace node-counting DoS guard with size-capped serializer

Address PR #2209 Copilot review (rig_adapter.rs:347 x2, :406):

1. `count_json_nodes` doc claimed "returns early once it exceeds the
   caller's budget" but always fully traversed. The approach also
   missed the case a reviewer flagged: few-node schemas with multi-MB
   string values would pass the node check but still allocate
   proportionally during `serde_json::to_string`.

Replace both the node counter and the `to_string` + truncate pattern
with `serialize_json_capped(value, max_bytes)`: a `serde_json::to_writer`
call through a `CappedWriter` that silently discards bytes past the
budget. This bounds the actual heap allocation to `max_bytes` regardless
of schema shape — many-node deep recursion AND multi-MB string values
are both capped. The writer returns `Ok(data.len())` after the cap so
serde_json thinks all bytes were consumed and continues (minimal
remaining work since the output is being discarded). The output is
guaranteed valid UTF-8 because serde_json only emits ASCII structural
characters and JSON-escaped unicode.

The `count_json_nodes` function and `MAX_SCHEMA_NODES` constant are
removed — the capped serializer subsumes them entirely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): serialize bootstrap context action_calls through PythonActionCall

The bootstrap context builder (`build_orchestrator_inputs`) serialized
`m.action_calls` directly via the canonical `ActionCall` serde format
(`{action_name, id, parameters}`), but the Python orchestrator passes
these back verbatim in `working_messages` on the next `__llm_complete__`
call, where `python_json_to_action_calls` expects the interchange format
(`{name, call_id, params}`). The mismatch surfaced as "missing field
\`name\`" on every thread resume after a gate pause (approval, auth),
orphaning all subsequent tool results.

This is the SECOND code path (after `handle_llm_complete`) that feeds
action_calls into the Python working transcript. Both must use the same
shape — `action_calls_to_python_json` is the single source of truth.

Triggered by: user approves `tool_upgrade` → thread resumes → bootstrap
rebuilds context from `internal_messages` (which stores canonical
`ActionCall`s from the DB) → Python reads `{action_name, id, parameters}`
→ echoes them back on next LLM call → `python_json_to_action_calls`
fails → assistant message loses tool_call linkage → all tool results
orphaned.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(engine): add bootstrap-path round-trip test to guard against future action_calls serialization drift

The gate-resume bug (08e47209) happened because `build_orchestrator_inputs`
serialized `action_calls` with canonical `ActionCall` field names instead
of the `PythonActionCall` interchange format. The existing round-trip
test only covered the `__llm_complete__` path (within a single Python
orchestrator run), not the bootstrap path (thread resume after gate
pause). Anyone adding a THIRD serialization path in the future would
have no test guardrail.

Two new tests:

1. `bootstrap_context_action_calls_round_trip_through_python_interchange`:
   Builds a `ThreadMessage` with `action_calls` in canonical format
   (the shape stored in the DB), serializes through the EXACT pattern
   `build_orchestrator_inputs` uses, parses back through
   `json_to_thread_messages`, and asserts the calls survive. This is
   the test that would have caught the gate-resume bug on the first
   attempt.

2. `canonical_action_call_field_names_do_not_round_trip`:
   Negative test that verifies canonical names (`{action_name, id,
   parameters}`) are REJECTED by the parser. Documents the current
   contract: if this test ever passes, the `PythonActionCall`
   interchange type can be removed because the formats unified. Serves
   as a tripwire for anyone who adds `#[serde(rename)]` to `ActionCall`
   or changes the parser to accept both formats.

Together these two tests cover every known serialization boundary into
the Python transcript and make the failure mode instantly visible in
`cargo test` rather than as a runtime warn log after a gate pause.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): skip strict-mode post-validator on flatten path to eliminate false-positive noise

The post-normalization validator (added in ce96c2a5 as a safety net)
was firing on every flattened schema with "additionalProperties should
be false" — a false positive because the flatten envelope deliberately
uses `additionalProperties: true` so the LLM can mix variant fields.
With 12 flattened tools loaded, this produced 12 debug log lines PER
LLM CALL, drowning real signals.

The flatten path's output is intentionally non-strict — running a
strict-mode validator on it is semantically wrong. Move the validator
behind the non-flatten branch and add an early `return schema` for the
flatten path (after normalizing individual properties). The validator
still catches issues on normal strict-mode schemas (the non-flatten
path), which is where it has value.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: close 5 coverage gaps across schema normalization, action_calls, and MCP naming

Systematic test audit of all 27 PR commits identified gaps where
production bugs had no hermetic regression test or where important
code paths had only helper-level (not caller-level) coverage.

New tests:

1. **test_realistic_wasm_schema_survives_normalize_flatten_pipeline**
   (rig_adapter.rs) — End-to-end test using the google_docs_tool's
   actual schema shape: tagged enum with 4 variants, one containing
   `requests: Vec<Value>` (bare array, no items), one with a nested
   object (text_style). Drives through normalize_schema_strict AND
   convert_tools. Asserts oneOf flattened, all variant properties
   merged, array items is object, nested objects get strict-mode.
   This single test would have caught BOTH production bugs (flatten
   path short-circuit + array items unreachable on flatten path).

2. **test_normalize_schema_strict_fixes_deeply_nested_array_items**
   (rig_adapter.rs) — 3-level nesting: object → array → object →
   array → object → array. Verifies the recursive normalizer walks
   the full depth and fixes every array items at every level.

3. **json_to_thread_messages_handles_null_action_calls_gracefully**
   + **handles_absent_action_calls** + **handles_empty_action_calls_array**
   (orchestrator.rs) — Three edge cases for the Python ↔ Rust
   message round-trip: null (was a false alarm), absent (baseline),
   and empty array (valid, produces Some(vec![])). The null case
   would have caught the "invalid type: null, expected a sequence"
   false alarm before it hit production.

4. **latent_provider_actions_normalize_hyphenated_server_names**
   (manager.rs) — Registers an MCP server with hyphenated name
   (`my-mcp-server`) and two tools (one with dashes, one without).
   Asserts latent action_names use all-underscore form
   (`my_mcp_server_search_all`) and the old hyphenated form doesn't
   survive. Exercises the mcp_tool_id normalization at the
   ExtensionManager layer.

5. **test_serialize_json_capped_boundary_conditions** +
   **test_serialize_json_capped_large_string_values** (rig_adapter.rs)
   — Size-capped serializer edge cases: under cap (full output),
   exactly at cap, over cap (truncated), zero cap (empty), and the
   multi-MB-string-in-few-nodes case the old node counter missed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review: address all 6 review items — UTF-8 safety, FOR UPDATE row check, MCP config alias, legacy token request path, stale reindex guard

1. **serialize_json_capped UTF-8 safety** (Copilot, rig_adapter.rs:399):
   serde_json v1 emits raw UTF-8 for non-ASCII chars (e.g. CJK in
   property descriptions), so byte-capped truncation can cut
   mid-codepoint. `String::from_utf8` now falls back to
   `e.valid_up_to()` to trim to the last complete codepoint instead
   of dropping the entire hint on a UTF-8 error.

2. **FOR UPDATE row count check** (Copilot x2, repository.rs:377):
   `SELECT 1 ... FOR UPDATE` returns 0 rows if the document doesn't
   exist, silently acquiring no lock. Now checks the row count and
   returns a clear `ChunkingFailed` error when it's 0.

3. **MCP config lookup alias-aware** (serrrfirat, factory.rs:40):
   `get_mcp_server` now tries exact name → hyphen alias
   (underscores→hyphens) → underscore alias (hyphens→underscores).
   After factory normalizes `server.name` to underscores,
   `provider_extension_for_tool` returns `my_server`, but the
   persisted config is keyed as `my-server`. Without alias lookup,
   `activate_mcp("my_server")` failed with `ServerNotFound`.

4. **Legacy token request-time fallback** (serrrfirat, auth.rs:1208):
   `get_access_token` now falls back to the legacy (pre-normalization)
   secret name when the canonical name returns no token. Without this,
   `is_authenticated` reported true (it has its own fallback) but the
   actual MCP request sent no Authorization header — the server
   appeared ready but tool execution 401'd until re-auth.

5. **Stale reindex guard** (serrrfirat, workspace/mod.rs:2181):
   `reindex_document_with_metadata` now captures the content hash at
   read time and re-checks it after computing embeddings. If another
   writer updated the document content during the embedding window,
   the reindexer skips chunk replacement — the other writer's reindex
   call will produce correct chunks for the new content. This closes
   the content-vs-chunks skew where writer B wins the document UPDATE
   but writer A wins the later replace_chunks transaction.

6. **Log summary PII concern** (Copilot, orchestrator.rs:2020): false
   positive — the keys logged are JSON Schema property names from
   the PythonActionCall interchange dict, not user tool parameters.
   Reply-only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(workspace): narrow reindex concurrency check error handling

Address Copilot review (workspace/mod.rs:2208): the optimistic
concurrency check caught all `Err(_)` as "document deleted" which
would silently swallow real DB errors (transient connection issues),
leaving chunks stale with no signal. Now only catches
`DocumentNotFound` for the deleted case; other errors propagate.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): canonicalize paths in file_history to fix macOS symlink mismatch

The file_history snapshot/restore tests were failing on macOS because
`/var` is a symlink to `/private/var`. `snapshot()` stored the
original path (`/var/folders/.../code.rs`), but `execute()` called
`validate_path()` which canonicalizes to `/private/var/folders/...`.
The path comparison in `restore_latest` mismatched, returning "No
file history found" even though the snapshot existed.

Fix: canonicalize paths consistently at both the storage boundary
(`snapshot()`) and the lookup boundary (`latest_snapshot_for()`,
`snapshots_for()`, `restore_latest()`). A shared `canonical()` helper
handles the non-existent-file case (write_file's "new file" path)
by canonicalizing the parent directory and joining the filename —
the parent always exists even when the file itself doesn't yet.

This was a pre-existing staging failure from PR #2025 that affected
all macOS developers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): update e2e_live_personas to match refactored test harness APIs

The PR changed `live_harness.rs` and `test_rig.rs` APIs without updating
`e2e_live_personas.rs`, causing Clippy compilation failures across all
feature sets. Also replaces `.expect()` in `rig_adapter.rs` with
`from_utf8_unchecked` (sound per `valid_up_to` invariant) to fix the
no-panics CI check.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): deduplicate path canonicalization in file_history::snapshot

Replace inline canonicalization logic in `snapshot()` with a call to the
existing `Self::canonical()` helper to eliminate duplication.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Henry Park <henrypark133@gmail.com>
@ironclaw-ci ironclaw-ci Bot mentioned this pull request Apr 10, 2026
zmanian added a commit that referenced this pull request Apr 11, 2026
…ounting (#2252)

Address three high-severity findings from PR #2025 review:

1. FileHistory byte-limit eviction: add max_total_bytes field (50MB default)
   that evicts oldest snapshots when total stored content exceeds the cap,
   preventing OOM from 50 entries x 10MB = 500MB worst case.

2. Per-job state cleanup: add cleanup_job() to both FileHistory and
   ReadFileState so entries keyed by job_id can be freed when jobs complete,
   preventing unbounded memory growth.

3. Unified match counting in apply_patch: replace the dual-path approach
   (count_matches for validation, find_match_from for replacement) with a
   single find_match_from loop that collects all matches upfront, eliminating
   potential count disagreements between the two code paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zmanian added a commit that referenced this pull request Apr 13, 2026
…ounting (#2252)

Address three high-severity findings from PR #2025 review:

1. FileHistory byte-limit eviction: add max_total_bytes field (50MB default)
   that evicts oldest snapshots when total stored content exceeds the cap,
   preventing OOM from 50 entries x 10MB = 500MB worst case.

2. Per-job state cleanup: add cleanup_job() to both FileHistory and
   ReadFileState so entries keyed by job_id can be freed when jobs complete,
   preventing unbounded memory growth.

3. Unified match counting in apply_patch: replace the dual-path approach
   (count_matches for validation, find_match_from for replacement) with a
   single find_match_from loop that collects all matches upfront, eliminating
   potential count disagreements between the two code paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ironclaw-ci ironclaw-ci Bot mentioned this pull request Apr 18, 2026
zmanian added a commit that referenced this pull request May 15, 2026
…ounting (#2252)

Address three high-severity findings from PR #2025 review:

1. FileHistory byte-limit eviction: add max_total_bytes field (50MB default)
   that evicts oldest snapshots when total stored content exceeds the cap,
   preventing OOM from 50 entries x 10MB = 500MB worst case.

2. Per-job state cleanup: add cleanup_job() to both FileHistory and
   ReadFileState so entries keyed by job_id can be freed when jobs complete,
   preventing unbounded memory growth.

3. Unified match counting in apply_patch: replace the dual-path approach
   (count_matches for validation, find_match_from for replacement) with a
   single find_match_from loop that collects all matches upfront, eliminating
   potential count disagreements between the two code paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
theredspoon pushed a commit to theredspoon/ironclaw that referenced this pull request Jun 21, 2026
…earai#2025)

* feat(tools): add production-grade coding tools, file history, and coding skills

Add dedicated coding tools inspired by Claude Code's architecture to make
IronClaw a more effective coding assistant:

New tools:
- GlobTool: fast file pattern matching via `glob` crate, sorted by mtime,
  with default exclusions (.git, node_modules, target, etc.)
- GrepTool: content search wrapping ripgrep with 3 output modes
  (content, files_with_matches, count), pagination, and context lines
- FileUndoTool: restore files to pre-modification state using in-memory
  file history snapshots

Enhanced tools:
- ReadFileTool: 10MB limit, 2000-line default, binary detection, device
  path blocking (/dev/zero, /proc/*/fd/*)
- ApplyPatchTool: uniqueness validation (error on ambiguous matches),
  workspace path rejection, 10MB size limit, file history integration
- WriteFileTool: file history integration for undo support

Updated tool descriptions to guide LLM behavior (prefer apply_patch over
write_file, always read before editing, use glob/grep instead of shell).

New skills:
- coding: best practices for code editing, search, and file operations
- commit: git commit message generation workflow
- review: code review workflow with structured checklist

Shared infrastructure:
- DEFAULT_EXCLUDED_DIRS constant in path_utils.rs
- FileHistory module with SharedFileHistory for cross-tool snapshots

66 new tests covering all tools, edge cases, and regression scenarios.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply cargo fmt formatting

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): address PR review — security, correctness, and robustness fixes

- Move device path blocking after validate_path() to prevent traversal bypass
- Add /proc/kcore, /proc/kmem to blocked paths
- Reject absolute patterns and '..' in glob tool, add strip_prefix defense
- Wrap glob sync I/O in spawn_blocking to avoid blocking tokio executor
- Sort files_with_matches globally before pagination in grep tool
- Add default exclusions for node_modules/target in grep tool
- Inject ctx.extra_env into rg environment matching ShellTool policy
- Use per-line strip_prefix for content mode path relativization
- Change FileSnapshot.content_before to Vec<u8> for binary file support
- Log snapshot errors with tracing::debug instead of silently discarding
- Fix skill name mismatch: code-review → review to match directory

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(skills): rename review skill directory to code-review

Aligns the directory name with the manifest name (code-review) to prevent
incorrect override/dedup behavior in the bundled-skill loader. The name
stays "code-review" since other domains may also need review-type skills.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(tools): add file edit guards — staleness detection, fuzzy matching, encoding preservation

Add file_edit_guard module with production-grade safeguards for file editing:
- ReadFileState tracks file reads with mtime for staleness detection
- 4-level fuzzy matching fallback (exact → whitespace-normalized → quote-normalized → both)
- UTF-16LE BOM detection and line ending style preservation (LF/CRLF/CR)
- Read-before-edit enforcement for ApplyPatch and WriteFile tools
- No-op edit rejection (old_string == new_string)
- Shared state injection via Arc<RwLock<>> across ReadFile, WriteFile, ApplyPatch

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): address all PR review comments — session scoping, parallelism, security

- Session-scoped state: ReadFileState and FileHistory now keyed by job_id
  so concurrent sessions sharing the same registry don't leak state (nearai#2025)
- Parallel metadata: grep files_with_matches uses JoinSet (max 64 concurrency)
  instead of sequential await per file for mtime sorting
- Shared env allowlist: grep_tool imports SAFE_ENV_VARS from shell.rs
  (made pub(crate)) instead of maintaining a divergent copy
- Glob traversal: uses Component::ParentDir check instead of substring ".."
  match, so patterns like "foo..bar" are no longer falsely rejected
- UTF-16LE in read_file: binary detection skips null-byte check for files
  with UTF-16LE BOM; read_file uses encoding-aware read path
- Partial flag: default 2000-line truncation now marks read as partial,
  preventing edits against unseen content
- write_file guard softened: staleness check logs warning instead of
  hard error (full-file replacement has lower risk than apply_patch)
- Updated e2e trace to include read_file before apply_patch
- Updated expected tool list in schema validation tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): use async metadata instead of blocking path.exists() in write_file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): fix false-positive panic detection for lifetimes in char lexer

The check_no_panics.py lexer misinterpreted Rust lifetimes ('static) as
char literal starts, causing in_char state to persist across lines and
hide all subsequent brace-delimited blocks — including #[cfg(test)] mod
tests. Reset in_char at line boundaries since Rust char literals cannot
span lines.

https://claude.ai/code/session_012bJjER6L5zSAFd9BqYMUmC

* test: verify MCP push works

* test

* chore: remove test file

* style: apply cargo fmt to file.rs

Collapse multi-line method chain to single line per rustfmt.

https://claude.ai/code/session_012bJjER6L5zSAFd9BqYMUmC

* style: apply cargo fmt to file.rs

Collapse multi-line method chain to single line per rustfmt.

https://claude.ai/code/session_012bJjER6L5zSAFd9BqYMUmC

* fix(file-tools): harden fuzzy patch matching and undo

* fix(ci): formatting + wasmtime 43 cache config compatibility

After merging latest staging, cargo fmt had diffs in file tools and the
wasmtime cache TOML format changed (v43 dropped the `enabled` field
under `[cache]`). Also removes accidental .fmt-test artifact.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(file-tools): simplify strip_trailing_whitespace

Remove redundant double-pass through .lines() — the first
collect+join was a no-op since .lines() already handles line endings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): address PR review comments — security, correctness, tests

- Add is_sensitive_path checks to GlobTool and GrepTool, matching the
  defense-in-depth posture of ReadFileTool/WriteFileTool/ListDirTool
- Fix UTF-8 panicking byte-index slice in apply_patch error preview
  (old_string[..200] → chars().take(200))
- Add 10MB size guard on file_history snapshots to prevent memory
  exhaustion from snapshotting large files
- Replace dead turn_number field with auto-incrementing sequence_number
  in FileHistory — callers no longer pass a hardcoded 0
- Fix glob mtime test flakiness by increasing sleep to 1100ms (above
  1s filesystem granularity)
- Fix emoji test to actually include emoji/non-ASCII content

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Zaki Manian <zaki@iqlusion.io>
theredspoon pushed a commit to theredspoon/ironclaw that referenced this pull request Jun 21, 2026
…ce race (nearai#2209)

* fix(workspace): collapse reindex delete+insert into one transaction to close TOCTOU race

Two concurrent reindexers for the same document could both DELETE the
existing chunks, then both try to INSERT chunk_index 0, hitting the
UNIQUE (document_id, chunk_index) constraint and failing with
"database is locked" or constraint violation. The delete and inserts
were separate libsql transactions with async points between them.

Add `WorkspaceStore::replace_chunks(document_id, &[ChunkWrite])` that
runs DELETE + N INSERTs inside a single BEGIN IMMEDIATE transaction
(not the default DEFERRED — DEFERRED bypasses busy_timeout on the
first write contention). The libsql impl, postgres impl, and the
in-memory storage variant all go through the new method, and
`Workspace::reindex_document` builds the `ChunkWrite` Vec (with
embeddings) up front so nothing async happens between the delete and
the insert loop.

Regression test in `workspace::versioning_tests` spawns 4 concurrent
writers against the same document on a multi-thread runtime and
asserts last-writer-wins without UNIQUE collisions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(auth): resolve display name + extension target from action when surfacing auth gates

The engine's `ResumeKind::Authentication` only carries `credential_name`
(e.g. `google_oauth_token`), which was being used as both the
user-facing display string AND as the first argument to
`submit_auth_token`. Two failure modes:

1. Display: users saw "google_oauth_token" in the auth-required prompt
   instead of the friendly extension name "google-drive-tool".
2. Routing: `submit_auth_token` expects an *extension* name and walks
   the extension's capabilities file to find the actual secret. Passing
   `google_oauth_token` directly fails closed with "Extension not
   installed: google_oauth_token", trapping the user in a re-auth loop
   on every paste.

`bridge/router.rs` now resolves the actual extension via
`tools.provider_extension_for_tool(action_name)` for both the gate
display path and the `submit_auth_token` call. Built-in tools, HTTP,
and skill credentials still fall back to the credential name (the
existing behaviour for those callers).

`extensions/manager.rs` fixes three more auth-readiness traps surfaced
by the v2 Drive trace:

- All capabilities lookups (`auth_wasm_tool`, channel activate, setup
  schema, configure, explicit secret query, upgrader) now go through
  `load_tool_capabilities` / `load_channel_capabilities` so a tool
  installed under the legacy hyphen filename
  (`google-drive-tool.capabilities.json`) is still resolved when
  looked up by canonical underscore name. The pre-v0.23 layout
  silently reported `no_auth_required` and bypassed the gate
  entirely.

- `activate_wasm_tool` now uses `existing_extension_file_path` for
  both the `.wasm` and `.capabilities.json` lookups so the legacy
  hyphen filename is resolved here as well. Without this, the
  upstream `determine_installed_kind` happily reported the extension
  as installed via its own alias check, but `activate_wasm_tool`
  then failed with `NotInstalled` — the readiness probe fell back to
  "treat as ready" and the agent ended up calling a tool that
  couldn't activate, hit a 401/403, and looped trying to recover.

- `configure()` post-activation OAuth cleanup now skips deletion when
  the caller is *also* providing a fresh credential in the same
  `secrets` map. The previous behaviour wrote the user's pasted token
  then immediately deleted it (along with `_scopes` /
  `_refresh_token` siblings), causing the resume to hit the wrapper
  with `token_exists=false` and re-fire the gate forever. Explicit
  Reconfigure (empty secrets map) still wipes the records to kick
  off a fresh OAuth dance.

`config/mod.rs` test config now seeds a deterministic 32-byte master
key so replay-mode tests that touch credentials get a working secrets
store out of the box without each test having to build its own.

Three regression tests in `extensions::manager::tests`:
- `test_activate_wasm_tool_finds_legacy_hyphen_alias`
- `test_auth_wasm_tool_finds_legacy_hyphen_alias`
- `test_configure_preserves_oauth_token_when_caller_provides_it`
- `test_configure_clears_oauth_token_for_reconfigure_flow`

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): canonicalize MCP tool identifiers to snake_case at registration

MCP tool names commonly contain dashes (e.g. Notion's `notion-search`),
and so do user-supplied server names (`my-server`). The runtime
converges on snake_case identifiers per `ToolRegistry::resolve_name`,
and LLMs (Codex / GPT-5 in particular) silently normalize tool names
to valid Python identifiers by converting dashes to underscores. The
old code built the registry key as `format!("{server}_{tool}")` and
preserved any dashes from the original tool name, so the registry got
`notion_notion-search` while the LLM emitted `notion_notion_search` —
direct lookup missed and the legacy alias fallback (which only goes
underscores → dashes) couldn't reconstruct the mixed-separator form
either, leaving every Notion MCP tool unreachable.

Add `mcp_tool_id(server, tool)` in `tools/mcp/client.rs` that does
`format!(...).replace('-', "_")`, re-export it from `tools/mcp/mod.rs`,
and use it in:

- `McpClient::create_tools` — the prefixed_name on every wrapped MCP
  tool now agrees with what the LLM will emit
- `ExtensionManager::activate_mcp` — `tool_names` is now sourced from
  `tool_impls.iter().map(|t| t.name())` instead of being independently
  rebuilt from the raw McpTool list, eliminating drift between
  registered names and reported names
- `ExtensionManager::latent_actions_for_mcp_server` — latent provider
  actions surfaced before activation use the same canonical form

The original (possibly hyphenated) `t.name` is still preserved on the
wrapper's inner `McpTool` and used verbatim when forwarding the
`tools/call` request to the MCP server, so MCP protocol compatibility
is unchanged — the canonicalization is internal-only.

5 regression tests in `tools::mcp::client::tests`:
- `test_mcp_tool_id_canonicalizes_dashed_tool_name`
- `test_mcp_tool_id_canonicalizes_dashed_server_name`
- `test_mcp_tool_id_passthrough_for_already_canonical_names`
- `test_create_tools_canonicalizes_dashed_mcp_tool_names` (drives
  `create_tools` end-to-end via MockTransport)
- `test_create_tools_round_trips_through_registry_resolve_name`
  (caller-level test per `.claude/rules/testing.md` — registers the
  wrapped tools in a real `ToolRegistry` and asserts that
  `resolve_name("notion_notion_search")` returns the registered tool
  via the direct HashMap path, not via the legacy alias fallback)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): flatten top-level schema unions for OpenAI + symmetric Python/Rust action_calls round-trip

Two related bugs in the LLM ↔ engine boundary, both surfaced by the
GitHub Copilot MCP and the v2 orchestrator:

1. **Top-level schema flatten.** OpenAI's tool API rejects schemas
   whose top level isn't `type: "object"` or that contain top-level
   `oneOf`/`anyOf`/`allOf`/`enum`/`not`, with HTTP 400:

       Invalid schema for function '<name>': schema must have type
       'object' and not have 'oneOf'/'anyOf'/'allOf'/'enum'/'not' at
       the top level.

   The GitHub Copilot MCP's `github` tool uses top-level `oneOf` for
   action dispatch, so the agent was 400-ing the moment it tried to
   enumerate tools. `normalize_schema_strict` (already shared between
   the OpenAI Codex provider and `RigAdapter::convert_tools`) now
   short-circuits when it sees a forbidden top-level construct,
   replacing `parameters` with a permissive object envelope
   (`{type: "object", properties: {}, additionalProperties: true,
   required: []}`) and appending the original schema to the tool
   description as advisory text (truncated on a char boundary at 1500
   bytes). The MCP server still validates the actual shape on its
   end, so the tool keeps working — we just lose API-level schema
   enforcement and the LLM has to read variant structure from the
   description.

   The function signature changes to take `&mut String` for the
   description (to append the hint). Both call sites
   (`openai_codex_provider::convert_tool_definition` and
   `rig_adapter::convert_tools`) pass an owned clone through.

   This is slightly lossy for Anthropic users on tools with top-level
   unions (Claude could have handled the union natively), but
   keeping a single normalizer for all rig-based providers is simpler
   than threading per-provider flags through the adapter, and the
   description hint preserves the variant info Claude needs.

2. **Python ↔ Rust `action_calls` field-name mismatch.** The Python
   orchestrator (`default.py`) appends assistant messages with
   `action_calls=calls` where each call is shaped
   `{name, call_id, params}` (the friendly Python names produced by
   `orchestrator.rs:handle_llm_complete`). The reverse parser
   `json_to_thread_messages` tried to deserialize via
   `serde_json::from_value::<Vec<ActionCall>>`, which expects the
   canonical Rust field names `{action_name, id, parameters}`. The
   deserialize fails, but `.ok()` swallows the error and the
   assistant message comes back with `action_calls = None`. Every
   subsequent tool result then looks orphaned to
   `sanitize_tool_messages` and gets rewritten as a user message,
   losing the model's ability to reason about prior tool calls.

   Introduce a private `PythonActionCall` interchange struct as the
   single source of truth for the field naming convention, with
   bidirectional `From` conversions. Both call sites
   (Rust → Python serialization + Python → Rust deserialization)
   now go through `action_calls_to_python_json` /
   `python_json_to_action_calls`, so any future field addition only
   needs to touch one struct definition.

   `ActionCall` itself is unchanged — adding `#[serde(rename = ...)]`
   would have invalidated every persisted Step record and ThreadEvent.

11 regression tests:
- `rig_adapter::tests::test_normalize_schema_strict_*` (6 tests)
  covering pass-through, top-level oneOf flatten, anyOf/allOf/enum/not
  flatten, non-object replacement, nested-oneOf preservation, and
  char-boundary truncation
- `rig_adapter::tests::test_convert_tools_handles_top_level_oneof_dispatcher`
  (caller-level test driving `convert_tools` end to end)
- `openai_codex_provider::tests::test_convert_tool_definition_handles_top_level_oneof_dispatcher`
  (caller-level test driving the codex provider path)
- `executor::orchestrator::tests::python_action_call_round_trips_through_serde`
- `executor::orchestrator::tests::action_calls_to_python_json_uses_python_field_names`
- `executor::orchestrator::tests::python_json_to_action_calls_parses_python_field_names`
- `executor::orchestrator::tests::python_json_to_action_calls_rejects_canonical_field_names`
  (guards against silent shape drift)
- `executor::orchestrator::tests::json_to_thread_messages_preserves_action_calls_from_python_orchestrator`
  (caller-level test feeding the literal JSON shape `default.py`
  produces, asserting the assistant message's `action_calls` survive
  the round-trip with matching call_ids)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(live): add Drive auth-gate round-trip live test + supporting harness pieces

End-to-end smoke test for the post-flight auth gate path that the
recent auth-postflight commits stitched together. Two phases:

  Phase A: delete the developer's real `google_oauth_token` (and the
  refresh token) from the test rig's libsql DB while keeping the
  `_scopes` companion record so `auth_wasm_tool`'s scope expansion
  check doesn't fire on the re-store. Send a Drive search prompt and
  assert the agent emits `StatusUpdate::AuthRequired` within one
  iteration. The expected path:
    1. agent calls `google-drive-tool { action: "list_files" }`
    2. wrapper's `resolve_host_credentials` reports
       `missing_required = ["google_oauth_token"]`
    3. wrapper fails closed with the
       "requires credentials that are not configured" message
    4. effect adapter's post-flight branch fires
       `auth::postflight::detect_post_call_auth_failure`
    5. matcher hits the `requires credentials + not configured` pair
       (commit cd8b68de)
    6. detector calls `ensure_extension_ready(.., ExplicitAuth)` →
       `EnsureReadyOutcome::NeedsAuth`
    7. `EngineError::GatePaused { resume_kind: Authentication }`
       bubbles to the orchestrator
    8. router stores it in `pending_gates` and emits
       `StatusUpdate::AuthRequired`

  Phase B: re-insert the captured token via `secrets_store()`, send
  the synthetic value as a follow-up message. The v2 router treats the
  next user message after an auth gate as
  `GateResolution::CredentialProvided`, which calls `submit_auth_token`
  (idempotent overwrite of what we just inserted) then
  `execute_pending_gate_action` → `execute_resolved_pending_action`,
  and the original Drive call replays. The test asserts the resume
  ran (additional tool activity + a follow-up response).

Live-tier only (`#[ignore]`); skipped outside `IRONCLAW_LIVE_TEST=1`.
The test deliberately does NOT commit a recorded trace fixture: any
trace would inevitably capture the bearer token, real Drive file
metadata, and HTTP headers — all PII that's hard to scrub safely.
Hermetic regression coverage for the underlying alias-aware
capabilities bug lives in
`test_auth_wasm_tool_finds_legacy_hyphen_alias`.

Supporting harness changes:

- `LiveTestHarnessBuilder::with_no_trace_recording()` — opt-out flag
  for tests that touch real credentials. Live mode still runs against
  the real LLM but no fixture is committed; replay mode builds a stub
  harness so the test can detect the mode and skip gracefully without
  panicking on a missing fixture.

- `LiveTestHarness::finish_turns(&[(user_input, responses)])` —
  multi-turn variant of `finish` for tests that span an auth-gate
  round-trip (prompt → AuthRequired → token → resume). The session
  log renders all turns in order so a reader can follow the full
  conversation, not just the first prompt. Status events are still
  rendered once at the top because the rig doesn't tag them with a
  turn boundary.

- Session log formatter now renders `StatusUpdate::AuthRequired` and
  `StatusUpdate::AuthCompleted` so the gate is visible in the log.

- `TestRig::secrets_store()` and `TestRig::owner_id()` accessors so
  live tests can manipulate credentials directly under the same scope
  the agent loop uses.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): log on python_json_to_action_calls deserialize failure

Address PR #2209 review: the helper used `serde_json::from_value(...).ok()?`
which is the exact `.ok()` swallow pattern the parent commit set out to
fix. If a future Python orchestrator patch ever drifts the action_calls
shape (extra required field, rename, partial migration), the helper would
silently return None and every subsequent tool result would look orphaned
to `sanitize_tool_messages` again — with no operator-visible signal at all.

Replace with an explicit match that emits a `warn!` (with the parse error
and the offending JSON value) on the failure path so the breadcrumb is
visible the moment any drift happens. The `None` return is preserved so
existing callers and the
`python_json_to_action_calls_rejects_canonical_field_names` test still
hold.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(test): generate random master key per Config::for_testing call

Address PR #2209 review: the previous fix hardcoded
`0123456789abcdef...0123456789abcdef` as the AES-256-GCM master key
inside `pub fn for_testing`. The function is `pub` (gated only behind
`#[cfg(feature = "libsql")]`, not `#[cfg(test)]`, because integration
tests in `tests/*.rs` are separate crates compiled against the lib's
non-test surface), which meant every developer building with libsql
had a publicly-known master key sitting in their process — and the
constant was now baked into Git history forever.

Replace with `generate_test_master_key()`, a private helper that pulls
32 bytes from `rand::thread_rng()` and hex-encodes them. Each call
returns a fresh key. Tests don't need cross-process determinism: each
test creates its own temp DB and the secrets store is born fresh on
every call anyway. `rand 0.8` is already a direct workspace dependency
so no Cargo changes are needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): normalize all non-identifier characters in mcp_tool_id

Address PR #2209 review: `mcp_tool_id` only handled `-` → `_`, but the
MCP spec doesn't actually constrain tool names to OpenAI's
`^[a-zA-Z0-9_-]{1,64}$` regex — a server could legally return
`notion.search`, `notion:create_issue`, `files/read`, or names with
spaces or non-ASCII characters. The same LLM normalization that bites on
`-` will bite on `.` and `:` too, and `extract_server_name` only strips
`.` from the host portion of a URL, leaving the tool portion of the
prefixed name unprotected.

Replace the single `.replace('-', "_")` with a `chars().map()` pass that
sends every non-`[A-Za-z0-9_]` character to `_`. This handles dashes,
dots, colons, slashes, spaces, and unicode in one shot — and since the
chars iterator yields one Rust char per code point, multi-byte
characters become a single `_` rather than splitting weirdly.

New regression test `test_mcp_tool_id_normalizes_non_identifier_chars`
covers dot, colon, slash, space, and multi-byte unicode inputs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): make flatten_top_level hint keyword-aware

Address PR #2209 review: the description hint appended by
`flatten_top_level` was a one-size-fits-all "pick one variant and pass
its fields as a flat object". That's correct for top-level `oneOf` and
`anyOf`, but actively misleading for the other forbidden constructs:

- `allOf` — the LLM should pass fields from ALL variants combined,
  not pick one
- `enum` — the LLM should pass one of the listed literal values,
  not "fields"
- `not` — the LLM should pass any object that does NOT match the
  constraint

Extract `FORBIDDEN_TOP_LEVEL` to a module-level constant (now shared
between `needs_top_level_flatten` and a new `detect_forbidden_top_level`
helper) and add `schema_flatten_hint_intro(detected)` which branches on
the actual keyword that triggered the flatten and returns a precise
intro string. Falls back to a "free-form object" message when the
schema wasn't an object at all (no recognized forbidden keyword, just
the wrong top-level type).

New regression test `test_normalize_schema_strict_hint_is_keyword_aware`
asserts that each of the 5 keywords produces a hint containing the
expected discriminating phrase.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(workspace): serialize postgres replace_chunks via FOR UPDATE on parent doc

Address PR #2209 review (Copilot, src/workspace/repository.rs:350):
the libsql `replace_chunks` is fine because `BEGIN IMMEDIATE` acquires
the writer lock at transaction start, but the postgres path used the
default-isolation `BEGIN` which is not equivalent. Two concurrent
reindexers running under separate snapshots can both DELETE (each sees
its own pre-delete state, neither sees the other's), then race to
INSERT chunk_index 0 and hit the `UNIQUE (document_id, chunk_index)`
constraint.

Add `SELECT 1 FROM memory_documents WHERE id = $1 FOR UPDATE` at the
top of the transaction. The `FOR UPDATE` row lock is per-document,
ties to the existing parent row (FK already in place from
`memory_chunks.document_id`), and is released automatically on
commit/rollback. Concurrent reindexers for the same doc now serialize
on the parent row and last-writer-wins cleanly.

Picked `FOR UPDATE` over `pg_advisory_xact_lock` because it's the
row-locking primitive PG operators expect when reading the code, and
it doesn't introduce a hash function dependency for the lock key.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): merge top-level union variants into flatten_top_level envelope

Address PR #2209 review (gemini-code-assist, src/llm/rig_adapter.rs:296):
after flattening a top-level oneOf/anyOf/allOf, the LLM was left with
`properties: {}` and could only read variant fields from the description
hint. That works but it's lossy — the LLM can't do schema-based
reasoning about which fields exist, and the description hint is
truncated to 1500 bytes so deeply-nested schemas are unreadable.

Add `merge_top_level_variant_properties` which walks the union variants,
collects every property they declare, and returns a single map. The
flatten envelope now uses that map instead of empty `{}`, so the LLM
sees structured field hints. `additionalProperties: true` and
`required: []` are preserved, so strict-mode validation stays disabled
and the LLM is free to mix fields across variants — the upstream MCP
server enforces the actual constraints on its end.

First-write wins on conflicting types: if two variants declare the
same field with different schemas, the first variant's schema is kept.
The full original schema still goes into the description hint, so the
ambiguous case is recoverable from there.

Two new regression tests:
- `test_normalize_schema_strict_merges_variant_properties` exercises a
  GitHub-Copilot-shaped tool with two variants that share a
  discriminator and asserts every field from every variant survives.
- `test_normalize_schema_strict_merge_first_write_wins_on_conflict`
  pins the documented conflict-resolution behaviour.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(router): extract resolve_extension_for_action helper for 3 dup sites

Address PR #2209 review (henrypark133, src/bridge/router.rs:1606): the
`provider_extension_for_tool + unwrap_or_else(credential_name)` pattern
was implemented in three places — once via the
`resolve_auth_gate_display_name` helper at line ~64, and twice inlined
in `resolve_gate` (line ~1603) and `await_thread_outcome` (line ~2779).
The inline sites couldn't use the helper because they'd already
destructured `credential_name` from the `ResumeKind` match and needed
the result for `submit_auth_token`, not just display.

Extract the core into `async fn resolve_extension_for_action(tools,
action_name, credential_fallback) -> String`. Make
`resolve_auth_gate_display_name` a thin wrapper that handles the
non-Authentication ResumeKind variants. Both inline sites now call the
helper directly with the destructured `credential_name`. The two
inline-site comment blocks that explained the rationale are collapsed
into shorter "see helper for full rationale" pointers since the doc
on `resolve_extension_for_action` carries the full explanation now.

Three sites collapse to one implementation. The auth display + routing
logic now has a single source of truth.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): warn on post-normalization tool name collisions in create_tools

Address PR #2209 review (serrrfirat, src/tools/mcp/client.rs:682):
after the broader `mcp_tool_id` char normalization (commit 18d4ce48),
two MCP tools whose names differ only by `-` vs `_` (e.g. `search-all`
and `search_all`) collide on the same registry key. The second
`ToolRegistry::register` call silently shadows the first with no
signal at all — operators debugging an unreachable tool would have
zero breadcrumb to discover the collision.

Add collision detection in `McpClient::create_tools` itself, where we
still have both the original tool name and the normalized id. Build a
`HashMap<normalized_id, original_name>` while iterating, and emit a
`tracing::warn!` when two distinct originals collide on the same id.
The warn carries the normalized id, both colliding original names,
and the server name, so an operator can immediately see which upstream
tools to rename. Behaviour is unchanged — the second tool still wins
on register, matching what the LLM would emit anyway since it
normalizes both names to the same string.

The collision detection is scoped to a single MCP server's tool list
because cross-server collisions have different registry-key prefixes
(`server_a_foo` vs `server_b_foo`) and can't actually shadow each
other. This is the right level — `ToolRegistry::register` itself
doesn't have access to the pre-normalization name and couldn't emit
this signal even if we wanted it there.

New regression test
`test_create_tools_handles_post_normalization_collision` drives a
MockTransport that lists `search-all` and `search_all`, asserts both
wrappers are produced with the same `Tool::name()`, registers them in
a real `ToolRegistry`, and asserts last-write-wins on shadow.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(engine): drop entries from action_calls_to_python_json on failure instead of injecting null

Address PR #2209 review (serrrfirat,
crates/ironclaw_engine/src/executor/orchestrator.rs:1920): the previous
helper used `unwrap_or_else(|_| Value::Null)` which silently corrupts
the array when serialization fails. The Python orchestrator
(`default.py`) accesses `c.get("name")` / `c.get("call_id")` /
`c.get("params")` on each entry, so a `null` would crash with a Python
`AttributeError` and lose the entire LLM step — and the fallback
contradicts this PR's own stated goal of not silently swallowing
errors.

Replace with `filter_map` so a failed entry is dropped from the output
rather than corrupting it. The warn log on the failure path is
preserved (and now also includes `action_name` for easier
correlation). Python's tool-result loop iterates
`range(len(results))` against the same shortened call list so a
missing entry is benign.

Note: the failure path is essentially unreachable for the
`PythonActionCall` shape (`String + String + Value` all
infallible-to-serialize) but the contract should still be safe — the
helper will be touched again when the Python interchange shape
evolves and we don't want a future maintainer to discover this trap
the hard way.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(engine): summarize action_calls in warn log to avoid leaking PII

Address PR #2209 review (serrrfirat,
crates/ironclaw_engine/src/executor/orchestrator.rs:1999): the
`python_json_to_action_calls` warn log emitted `value = %value` which
dumps the full action_calls JSON array on parse failure. Tool params
can carry user PII (search queries, file names, email content,
conversation text), and the warn fires precisely when the Python ↔
Rust shape drifts — exactly the moment operators will be grepping
logs and shipping output to log aggregators (Datadog, CloudWatch,
Sentry).

Add `summarize_action_calls_for_log` which builds a structural-only
summary: array length and the keys of the first entry. The keys
themselves are static field names (`name`, `call_id`, `params`), not
user data. The shape summary is enough to debug a drift (operator can
see whether the shape is roughly right and which fields are missing)
without exposing any of the actual parameter contents.

Edge cases handled:
- empty array → "empty array"
- non-array value (Python passed wrong shape) → "non-array value of
  type <kind>" via a small `json_value_type_name` helper
- entries that aren't objects → "<not an object>" rather than
  attempting to walk them

Two regression tests:
- `summarize_action_calls_for_log_does_not_leak_user_pii` builds an
  intentionally PII-laden value with salary spreadsheet queries,
  credentials, and "private message about layoffs" content, asserts
  none of the user-content strings appear in the summary, AND that
  even the upstream tool name doesn't leak (operator-level intent
  signal).
- `summarize_action_calls_for_log_handles_edge_cases` pins the
  empty/string/object/null fallback paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* incorporate #2227: factory server_name normalization, registry bidirectional alias, WASM/channel loader normalization, legacy token fallback

Merge the non-overlapping changes from PR #2227
(fix-tool-name-hyphen-normalization) so this PR supersedes it. Our PR
already fixed the core issue (mcp_tool_id canonicalization + stricter
non-identifier-char normalization), but #2227 adds valuable
defense-in-depth and compatibility layers that we didn't cover:

- `tools/mcp/factory.rs` — normalize `server.name` at the factory
  boundary (before any branch including the OAuth early-return). This
  ensures the secret name, session key, and tool prefix all use the
  same underscore-only form. Our PR normalized only at the tool-id
  level which left the server_name itself hyphenated in the session
  manager and token secret store.

- `tools/mcp/client.rs` — normalize `new_with_name` so callers that
  pass a hyphenated server name get consistent behavior even when
  bypassing the factory.

- `tools/mcp/config.rs` + `tools/mcp/auth.rs` — legacy token secret
  name fallback for pre-normalization tokens. When checking if an MCP
  server is authenticated, if the canonical (underscore) secret name
  doesn't exist, try the legacy (hyphenated) form. This prevents
  forcing re-auth on existing users who stored tokens under the old
  hyphenated server name before upgrade.

- `tools/registry.rs` — bidirectional `resolve_key` helper that tries
  exact → hyphen→underscore → underscore→hyphen aliases in `get`,
  `has`, `unregister`, `resolve_name`, `get_resolved`,
  `provider_extension_for_tool`, and `tool_definitions_for_actions`.
  Defense-in-depth: even if a tool somehow ends up registered with a
  mixed-separator name (edge case, stale DB, manual insertion), the
  registry will still find it. 4 new regression tests cover both
  alias directions, get_resolved, and unregister via alias.

- `tools/wasm/loader.rs` + `channels/wasm/loader.rs` — `load_from_dir`
  and `discover_*` functions normalize hyphenated filenames (file stem
  → replace('-', "_")). Dev tool install name changed from
  `{name}-tool` to `{name}_tool`.

Conflict resolution: manager.rs (kept our version with comment),
client.rs create_tools (kept our `mcp_tool_id` which is strictly
better — handles ALL non-identifier chars, not just dashes), client.rs
tests (kept our comprehensive MockTransport-based test suite, dropped
#2227's simpler duplicate). All other hunks from #2227 applied cleanly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(manager): normalize server name prefix in starts_with tool-list filters

Address PR #2209 review (henrypark133, Critical C1): the 3
`starts_with(&format!("{}_", name))` filters in `activate_mcp`
(already-active fast path), `list()`, and `remove()` used the raw
(possibly hyphenated) server name, while `mcp_tool_id` normalizes the
tool registry keys to underscores-only. A hyphenated server name like
`my-server` produced a prefix `my-server_` that matched zero tools
(they're all `my_server_*`), returning empty tool lists and failing to
unregister on extension removal.

Fix: use `crate::tools::mcp::mcp_tool_id(name, "")` as the prefix.
This produces `my_server_` from `my-server`, matching the registered
keys exactly. All 3 sites now use the same normalization as tool
registration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): accept array type containing "object" in needs_top_level_flatten

Address PR #2209 review (henrypark133, Critical C2):
`needs_top_level_flatten` only matched `JsonValue::String("object")`
for the type check. A top-level `"type": ["object", "null"]` (valid
JSON Schema for a nullable object, produced by some upstream providers
and `make_nullable`) triggered `bad_type = true` and the schema was
flattened, silently discarding all its properties.

Extend the check to also accept `JsonValue::Array` when any element is
the string `"object"`. This prevents unnecessary flattening of schemas
that are semantically object-typed but use the array form for
nullability.

New regression test:
`test_normalize_schema_strict_does_not_flatten_nullable_object_type`

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(mcp): update seen_ids on collision so 3rd collision reports against 2nd

Address PR #2209 review (henrypark133, Nit N1): the collision detection
in create_tools skipped the `seen_ids.insert` on the collision branch,
so a 3rd colliding tool would report against the 1st original name
instead of the 2nd (the actual shadow). Added the insert inside the
warning branch so subsequent collisions report the correct chain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(wasm): fix CI type mismatch + add string-matching fallback for wrapped traps

Address PR #2209 CI failure and Copilot review comments:

1. **CI fix (E0308):** `classify_trap_error` took `anyhow::Error` but
   wasmtime 43's `call_execute` returns `wasmtime::Error` (a distinct
   type in `wasmtime_internal_core`). Changed the signature to accept
   `wasmtime::Error` directly. This is also more correct — accepting
   the native error type preserves type information that a lossy
   `.into()` conversion would strip, making the structured `Trap`
   downcast more reliable.

2. **String-matching fallback (Copilot, wrapper.rs:1107):** The doc
   claimed "falls back to string matching" but the implementation only
   did the structured downcast. Added a string-matching fallback that
   checks the full Display chain for "all fuel consumed", "out of fuel",
   "OutOfFuel", and "unreachable" when the downcast fails. This covers
   the case where component-model glue or host wrappers bury the Trap
   inside layers that `downcast_ref` can't see through. New regression
   test `trap_classification_fuel_via_string_fallback` exercises this
   path using a plain `wasmtime::Error::msg` wrapper.

3. **Stale doc comment (Copilot, test_rig.rs:1282):** Updated
   `secrets_store()` doc to reflect that most test rigs now have a
   working secrets store because `Config::for_testing()` generates a
   random master key per call. `None` only occurs with a config
   override that explicitly disables secrets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review: tighten unreachable trap match, cap schema serialization, document legacy auth

Address PR #2209 review (serrrfirat, 4 comments):

1. `wrapper.rs:1131` — tightened the string fallback from bare
   `contains("unreachable")` to `contains("unreachable code")` /
   `"UnreachableCodeReached"` / `"wasm trap: unreachable"`. The old
   match would false-positive on HTTP errors like "endpoint was
   unreachable" or "server unreachable: connection refused", replacing
   the real diagnostic chain with a generic message.

2. `rig_adapter.rs:326` — added `count_json_nodes` pre-check (cheap
   recursive walk, no alloc) before calling `serde_json::to_string`.
   A malicious MCP server with a many-MB schema would have triggered
   a proportional allocation even though we only keep 1500 bytes.
   Schemas over 5000 nodes skip serialization entirely and get a
   "(schema too large to inline)" placeholder instead.

3. `auth.rs:1202` — documented that the legacy token fallback
   intentionally uses bare `get_decrypted` (no refresh). The path is
   transitional: users re-auth once and get migrated to the canonical
   naming scheme. Wiring refresh through the legacy path adds
   complexity for a self-healing compat layer.

4. `rig_adapter.rs:171` — Anthropic lossiness was already documented
   in the `normalize_schema_strict` doc comment (lines 164-170).
   Reply-only; per-provider flag is a follow-up.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): ensure array items is a JSON Schema object for OpenAI strict mode

OpenAI rejects array-typed properties whose `items` field is missing,
boolean (`true`), or any non-object value with:

  "array schema items is not an object"

Schema generators like schemars produce `{"type": "array"}` (no items)
or `{"type": "array", "items": true}` for `Vec<serde_json::Value>`,
which is valid JSON Schema but violates OpenAI's strict-mode rules.
The google_docs_tool's `requests: Vec<serde_json::Value>` field
triggered this on every tool enumeration.

In `normalize_schema_recursive`, detect array-typed properties and
ensure `items` is a JSON Schema object before recursing. Missing or
non-object `items` are replaced with `{}` (accept any item). Object
`items` are left untouched and recursed into as before.

Regression test covers all three cases: missing items, boolean items
(`true`), and well-formed items (must not be clobbered).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): add post-normalization validation to catch schema rules the normalizer misses

The schema_validator module already knew the "array items must be an
object" rule (Rule 8, line 218), had a test for it
(test_array_missing_items_fails, line 315), and would have caught the
google_docs_tool 400 — but it was only wired into CI tests against
built-in tools, never applied to WASM/MCP tool schemas or to the
output of normalize_schema_strict.

The root cause pattern: we're playing whack-a-mole with OpenAI's
undocumented strict-mode rules, adding fixes one at a time when a new
tool exposes a schema shape the normalizer doesn't handle. Each time,
the fix is a runtime 400 in production that takes a PR cycle to fix.

The structural fix: run validate_strict_schema as a debug-level
post-check after normalization. If the normalizer missed something,
the diagnostic appears in local logs immediately (before the schema
even reaches the LLM provider), giving developers a local breadcrumb
instead of a runtime 400 from OpenAI. The schema still goes through
(the tool remains usable), and the LLM provider surfaces the 400 if
OpenAI actually rejects it — but now the cause is instantly visible
in `RUST_LOG=ironclaw::llm=debug` output.

This also means that any future normalizer rule we add gets automatic
regression coverage: if the normalizer introduces a bug that violates
a rule the validator knows about, the debug log fires on every tool
call in dev mode.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): normalize merged properties on flatten path + silence null action_calls warn

Two runtime issues from the latest deploy:

1. The flatten path in `normalize_schema_strict` short-circuited with
   `return schema`, skipping BOTH the recursive normalizer AND the
   post-normalization validator. The merged properties copied from
   union variants were raw — a `Vec<serde_json::Value>` field like
   google_docs_tool's `requests` kept its bare `{"type": "array"}`
   without `items`, and OpenAI rejected it with "array schema items
   is not an object" on every tool-using call.

   Fix: the flatten path now normalizes each merged property
   individually via `normalize_schema_recursive(prop_schema)` before
   returning the envelope. The top-level envelope stays permissive
   (`additionalProperties: true`, `required: []`) so the LLM can
   mix fields across variants, but each property's internal schema
   gets the full treatment (array items, nested objects, etc.).

   The post-normalization validator also runs on both paths now
   (no early return before it).

   Regression test:
   `test_normalize_schema_strict_flatten_normalizes_merged_array_items`
   mimics the google_docs_tool shape (tagged enum with `oneOf`, one
   variant containing an items-less array) and asserts the merged
   `requests` property has `items` as an object after normalization.

2. `python_json_to_action_calls` warn log fired on every text-only
   assistant message with "invalid type: null, expected a sequence"
   because Python's `action_calls: null` (legitimate "no tool calls"
   signal) was passed to the parser. Added a `.filter(|v| !v.is_null())`
   before the parser call in `json_to_thread_messages` so null is
   treated the same as "key absent" — no parse attempt, no false
   alarm. The warn only fires for genuinely malformed data now.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(llm): replace node-counting DoS guard with size-capped serializer

Address PR #2209 Copilot review (rig_adapter.rs:347 x2, :406):

1. `count_json_nodes` doc claimed "returns early once it exceeds the
   caller's budget" but always fully traversed. The approach also
   missed the case a reviewer flagged: few-node schemas with multi-MB
   string values would pass the node check but still allocate
   proportionally during `serde_json::to_string`.

Replace both the node counter and the `to_string` + truncate pattern
with `serialize_json_capped(value, max_bytes)`: a `serde_json::to_writer`
call through a `CappedWriter` that silently discards bytes past the
budget. This bounds the actual heap allocation to `max_bytes` regardless
of schema shape — many-node deep recursion AND multi-MB string values
are both capped. The writer returns `Ok(data.len())` after the cap so
serde_json thinks all bytes were consumed and continues (minimal
remaining work since the output is being discarded). The output is
guaranteed valid UTF-8 because serde_json only emits ASCII structural
characters and JSON-escaped unicode.

The `count_json_nodes` function and `MAX_SCHEMA_NODES` constant are
removed — the capped serializer subsumes them entirely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): serialize bootstrap context action_calls through PythonActionCall

The bootstrap context builder (`build_orchestrator_inputs`) serialized
`m.action_calls` directly via the canonical `ActionCall` serde format
(`{action_name, id, parameters}`), but the Python orchestrator passes
these back verbatim in `working_messages` on the next `__llm_complete__`
call, where `python_json_to_action_calls` expects the interchange format
(`{name, call_id, params}`). The mismatch surfaced as "missing field
\`name\`" on every thread resume after a gate pause (approval, auth),
orphaning all subsequent tool results.

This is the SECOND code path (after `handle_llm_complete`) that feeds
action_calls into the Python working transcript. Both must use the same
shape — `action_calls_to_python_json` is the single source of truth.

Triggered by: user approves `tool_upgrade` → thread resumes → bootstrap
rebuilds context from `internal_messages` (which stores canonical
`ActionCall`s from the DB) → Python reads `{action_name, id, parameters}`
→ echoes them back on next LLM call → `python_json_to_action_calls`
fails → assistant message loses tool_call linkage → all tool results
orphaned.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(engine): add bootstrap-path round-trip test to guard against future action_calls serialization drift

The gate-resume bug (08e47209) happened because `build_orchestrator_inputs`
serialized `action_calls` with canonical `ActionCall` field names instead
of the `PythonActionCall` interchange format. The existing round-trip
test only covered the `__llm_complete__` path (within a single Python
orchestrator run), not the bootstrap path (thread resume after gate
pause). Anyone adding a THIRD serialization path in the future would
have no test guardrail.

Two new tests:

1. `bootstrap_context_action_calls_round_trip_through_python_interchange`:
   Builds a `ThreadMessage` with `action_calls` in canonical format
   (the shape stored in the DB), serializes through the EXACT pattern
   `build_orchestrator_inputs` uses, parses back through
   `json_to_thread_messages`, and asserts the calls survive. This is
   the test that would have caught the gate-resume bug on the first
   attempt.

2. `canonical_action_call_field_names_do_not_round_trip`:
   Negative test that verifies canonical names (`{action_name, id,
   parameters}`) are REJECTED by the parser. Documents the current
   contract: if this test ever passes, the `PythonActionCall`
   interchange type can be removed because the formats unified. Serves
   as a tripwire for anyone who adds `#[serde(rename)]` to `ActionCall`
   or changes the parser to accept both formats.

Together these two tests cover every known serialization boundary into
the Python transcript and make the failure mode instantly visible in
`cargo test` rather than as a runtime warn log after a gate pause.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(llm): skip strict-mode post-validator on flatten path to eliminate false-positive noise

The post-normalization validator (added in ce96c2a5 as a safety net)
was firing on every flattened schema with "additionalProperties should
be false" — a false positive because the flatten envelope deliberately
uses `additionalProperties: true` so the LLM can mix variant fields.
With 12 flattened tools loaded, this produced 12 debug log lines PER
LLM CALL, drowning real signals.

The flatten path's output is intentionally non-strict — running a
strict-mode validator on it is semantically wrong. Move the validator
behind the non-flatten branch and add an early `return schema` for the
flatten path (after normalizing individual properties). The validator
still catches issues on normal strict-mode schemas (the non-flatten
path), which is where it has value.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: close 5 coverage gaps across schema normalization, action_calls, and MCP naming

Systematic test audit of all 27 PR commits identified gaps where
production bugs had no hermetic regression test or where important
code paths had only helper-level (not caller-level) coverage.

New tests:

1. **test_realistic_wasm_schema_survives_normalize_flatten_pipeline**
   (rig_adapter.rs) — End-to-end test using the google_docs_tool's
   actual schema shape: tagged enum with 4 variants, one containing
   `requests: Vec<Value>` (bare array, no items), one with a nested
   object (text_style). Drives through normalize_schema_strict AND
   convert_tools. Asserts oneOf flattened, all variant properties
   merged, array items is object, nested objects get strict-mode.
   This single test would have caught BOTH production bugs (flatten
   path short-circuit + array items unreachable on flatten path).

2. **test_normalize_schema_strict_fixes_deeply_nested_array_items**
   (rig_adapter.rs) — 3-level nesting: object → array → object →
   array → object → array. Verifies the recursive normalizer walks
   the full depth and fixes every array items at every level.

3. **json_to_thread_messages_handles_null_action_calls_gracefully**
   + **handles_absent_action_calls** + **handles_empty_action_calls_array**
   (orchestrator.rs) — Three edge cases for the Python ↔ Rust
   message round-trip: null (was a false alarm), absent (baseline),
   and empty array (valid, produces Some(vec![])). The null case
   would have caught the "invalid type: null, expected a sequence"
   false alarm before it hit production.

4. **latent_provider_actions_normalize_hyphenated_server_names**
   (manager.rs) — Registers an MCP server with hyphenated name
   (`my-mcp-server`) and two tools (one with dashes, one without).
   Asserts latent action_names use all-underscore form
   (`my_mcp_server_search_all`) and the old hyphenated form doesn't
   survive. Exercises the mcp_tool_id normalization at the
   ExtensionManager layer.

5. **test_serialize_json_capped_boundary_conditions** +
   **test_serialize_json_capped_large_string_values** (rig_adapter.rs)
   — Size-capped serializer edge cases: under cap (full output),
   exactly at cap, over cap (truncated), zero cap (empty), and the
   multi-MB-string-in-few-nodes case the old node counter missed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review: address all 6 review items — UTF-8 safety, FOR UPDATE row check, MCP config alias, legacy token request path, stale reindex guard

1. **serialize_json_capped UTF-8 safety** (Copilot, rig_adapter.rs:399):
   serde_json v1 emits raw UTF-8 for non-ASCII chars (e.g. CJK in
   property descriptions), so byte-capped truncation can cut
   mid-codepoint. `String::from_utf8` now falls back to
   `e.valid_up_to()` to trim to the last complete codepoint instead
   of dropping the entire hint on a UTF-8 error.

2. **FOR UPDATE row count check** (Copilot x2, repository.rs:377):
   `SELECT 1 ... FOR UPDATE` returns 0 rows if the document doesn't
   exist, silently acquiring no lock. Now checks the row count and
   returns a clear `ChunkingFailed` error when it's 0.

3. **MCP config lookup alias-aware** (serrrfirat, factory.rs:40):
   `get_mcp_server` now tries exact name → hyphen alias
   (underscores→hyphens) → underscore alias (hyphens→underscores).
   After factory normalizes `server.name` to underscores,
   `provider_extension_for_tool` returns `my_server`, but the
   persisted config is keyed as `my-server`. Without alias lookup,
   `activate_mcp("my_server")` failed with `ServerNotFound`.

4. **Legacy token request-time fallback** (serrrfirat, auth.rs:1208):
   `get_access_token` now falls back to the legacy (pre-normalization)
   secret name when the canonical name returns no token. Without this,
   `is_authenticated` reported true (it has its own fallback) but the
   actual MCP request sent no Authorization header — the server
   appeared ready but tool execution 401'd until re-auth.

5. **Stale reindex guard** (serrrfirat, workspace/mod.rs:2181):
   `reindex_document_with_metadata` now captures the content hash at
   read time and re-checks it after computing embeddings. If another
   writer updated the document content during the embedding window,
   the reindexer skips chunk replacement — the other writer's reindex
   call will produce correct chunks for the new content. This closes
   the content-vs-chunks skew where writer B wins the document UPDATE
   but writer A wins the later replace_chunks transaction.

6. **Log summary PII concern** (Copilot, orchestrator.rs:2020): false
   positive — the keys logged are JSON Schema property names from
   the PythonActionCall interchange dict, not user tool parameters.
   Reply-only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* review(workspace): narrow reindex concurrency check error handling

Address Copilot review (workspace/mod.rs:2208): the optimistic
concurrency check caught all `Err(_)` as "document deleted" which
would silently swallow real DB errors (transient connection issues),
leaving chunks stale with no signal. Now only catches
`DocumentNotFound` for the deleted case; other errors propagate.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): canonicalize paths in file_history to fix macOS symlink mismatch

The file_history snapshot/restore tests were failing on macOS because
`/var` is a symlink to `/private/var`. `snapshot()` stored the
original path (`/var/folders/.../code.rs`), but `execute()` called
`validate_path()` which canonicalizes to `/private/var/folders/...`.
The path comparison in `restore_latest` mismatched, returning "No
file history found" even though the snapshot existed.

Fix: canonicalize paths consistently at both the storage boundary
(`snapshot()`) and the lookup boundary (`latest_snapshot_for()`,
`snapshots_for()`, `restore_latest()`). A shared `canonical()` helper
handles the non-existent-file case (write_file's "new file" path)
by canonicalizing the parent directory and joining the filename —
the parent always exists even when the file itself doesn't yet.

This was a pre-existing staging failure from PR #2025 that affected
all macOS developers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): update e2e_live_personas to match refactored test harness APIs

The PR changed `live_harness.rs` and `test_rig.rs` APIs without updating
`e2e_live_personas.rs`, causing Clippy compilation failures across all
feature sets. Also replaces `.expect()` in `rig_adapter.rs` with
`from_utf8_unchecked` (sound per `valid_up_to` invariant) to fix the
no-panics CI check.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tools): deduplicate path canonicalization in file_history::snapshot

Replace inline canonicalization logic in `snapshot()` with a call to the
existing `Self::canonical()` helper to eliminate duplication.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Henry Park <henrypark133@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: dependencies Dependency updates scope: docs Documentation scope: tool/builtin Built-in tools scope: tool/wasm WASM tool sandbox scope: tool Tool infrastructure size: XL 500+ changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants