perf(core): Automated performance tuning by Claude#1295
perf(core): Automated performance tuning by Claude#1295
Conversation
⚡ Performance Benchmark
Details
History
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request focuses on significant performance enhancements across several critical paths within the packager pipeline. By parallelizing asynchronous operations, optimizing data structures for lookups, and refining string processing algorithms, the changes aim to provide measurable speedups, particularly for repositories containing a large number of files. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1295 +/- ##
==========================================
- Coverage 87.13% 82.17% -4.96%
==========================================
Files 115 116 +1
Lines 4367 5693 +1326
Branches 1015 1387 +372
==========================================
+ Hits 3805 4678 +873
- Misses 562 1015 +453 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Deploying repomix with
|
| Latest commit: |
3f84dc5
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://134a3083.repomix.pages.dev |
| Branch Preview URL: | https://perf-auto-perf-tuning.repomix.pages.dev |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces several well-executed performance optimizations across different parts of the codebase, as detailed in the summary. The changes, including parallelizing Git operations, converting a list to a Set for O(1) lookups, optimizing line counting with indexOf, and hoisting regex compilations, directly address identified hot paths and show measurable speedups. The implementation of these optimizations is correct and follows best practices for performance in TypeScript. No further issues or improvement opportunities were identified based on the provided changes and the performance-focused objective of the pull request.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Key Optimization 16: Lazy-load Zod, optimize tree generation, and cache git repo checksThree targeted optimizations:
Module import time benchmark (20 runs, importing defaultAction.js):
Local benchmark (15 runs, packing repomix repo):
Note: Full pipeline benchmark variance (~200ms IQR) masks the module-level improvement. The 42ms faster module loading allows the worker process to start earlier, overlapping more initialization with Zod loading. CI benchmarks with controlled environment will show clearer results. |
Key Optimization 17: Lazy-load strip-comments, is-binary-path, and isbinaryfileDefer loading three modules from worker module startup to first use:
Total: ~20ms removed from the worker's critical module loading path. The worker process now loads these modules during file collection (I/O-bound phase) instead of during startup (CPU-bound module resolution), allowing the worker to be ready to receive tasks sooner. Local benchmark (15 runs, packing repomix repo):
Note: Local benchmark variance (~170ms IQR) exceeds the expected ~20ms improvement. The P75 improved by -80ms. CI benchmarks with controlled environment will provide more accurate measurement. |
Key Optimization 18: Overlap metrics with write, skip redundant sort, and optimize hot pathsFour targeted optimizations:
Local benchmark (15 runs, packing repomix repo):
Note: Local benchmark variance (~5ms IQR) is comparable to the improvement. The metrics/write overlap and security worker ext fix primarily benefit larger repos and CI environments where disk I/O and security checks take longer. |
Key Optimization 19: Fix token counter special tokens, estimate output tokens, and overlap file metricsThree targeted optimizations:
Local benchmark (15 runs, packing repomix repo):
|
Key Optimization 20: Overlap git-based file sorting with security checkMove Previously, When suspicious files are found (rare, <1% of repos), the pre-sorted array is filtered to remove flagged files. Filtering preserves sort order, so no re-sort is needed. Pipeline change: Local benchmark (15 runs, packing repomix repo):
|
yamadashy
force-pushed
the
perf/auto-perf-tuning
branch
from
fb048aa to
6861f46
Compare
Key Optimization 21: Lazy-load minimatch, parallelize file search I/O, and simplify permission checkFive targeted optimizations to reduce file search and permission checking overhead:
Local benchmark (15 runs, packing repomix repo):
What was investigated but not implemented:
|
Key Optimization 22: Batch security check tasks to reduce IPC overheadReduce structured clone serialization overhead by batching multiple files per worker IPC round-trip:
What was investigated but not used:
Local benchmark (25 runs with 3 warmup, packing repomix repo):
Security check stage timing (single profiled run):
|
Key Optimization 23: Increase file collection concurrency and optimize result partitioningThree targeted optimizations:
Local benchmark (10 runs, packing repomix repo, separate processes):
Note: Warm-cache sequential runs show less difference because filesystem caching reduces I/O latency. The cold-cache scenario (separate process invocations) represents the typical user experience. What was investigated but not changed
|
Key Optimization 26: Pre-compute lowercase in sort comparators and eliminate output template intermediatesThree targeted optimizations to reduce allocation pressure and GC overhead:
What was investigated but not implemented:
Local benchmark (20 runs, 3 warmup, ~1010 files):
The tighter variance (IQR: 27ms vs 48ms) suggests reduced GC pressure from fewer transient allocations. |
yamadashy
force-pushed
the
perf/auto-perf-tuning
branch
from
49a2598 to
75be4f7
Compare
Move regex patterns from inside function bodies to module-level constants to avoid repeated compilation on every file processed. For a repo with 1000 files, this eliminates 7000 regex compilations per run. - Hoist dataUriPattern, standaloneBase64Pattern to module scope - Hoist base64ValidCharsPattern, hasNumbers/UpperCase/LowerCase/SpecialChars - Add lastIndex reset for global-flag regexes before each use Cherry-picked optimization from PR #1295 (3/3 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace O(n²) string concatenation with O(n) array accumulation pattern in treeToString and treeToStringWithLineCounts. For repos with 1000+ files, the old code copied the entire accumulated string on each append, while the new code pushes fragments and joins once at the end. - Extract treeToStringInner/treeToStringWithLineCountsInner helpers - Move sortTreeNodes call into generateFileTree for single sort at build time - Retain sort guard in treeToString/_isRoot for direct callers Cherry-picked optimization from PR #1295 (3/3 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ntrustedFiles Replace O(n*m) Array.some() linear scan with Set.has() for O(n+m) filtering. Pre-builds a Set of suspicious file paths for constant-time lookups during the filter pass. Cherry-picked optimization from PR #1295 (3/3 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace three separate .filter() passes over security results with a single for-of loop using switch statement. Also skip filterOutUntrustedFiles entirely when no suspicious files are found (the common ~99% case). - Change let to const for result arrays (populated via push) - Short-circuit avoids Set construction + filter over all raw files Cherry-picked optimization from PR #1295 (2/3 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Convert static imports of initAction, mcpAction, remoteAction, and versionAction to dynamic import() at their use sites. The default pack path (95%+ of invocations) now avoids loading MCP server, git clone, and init action module trees entirely. Also inline isExplicitRemoteUrl prefix check to avoid loading git-url-parse module for non-remote runs. PR #1295 reports -66% module import time (358ms → 123ms). Cherry-picked optimization (4/5 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove log-update dependency (and its wrap-ansi → string-width chain, ~49ms module load) in favor of direct process.stderr.write with ANSI \x1B[2K\r for single-line in-place updates. The spinner only ever writes single lines, so log-update's multi-line and terminal-width handling was unnecessary overhead. Cherry-picked optimization from PR #1295 (4/5 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…arse Add Promise-based Map cache to isGitRepository() keyed by directory. When getGitDiffs and getGitLogs run concurrently, both call isGitRepository on the same directory — the cache ensures only one git rev-parse process is spawned instead of multiple. Cache is bypassed when custom deps are provided (test mocks). Cherry-picked optimization from PR #1295 (4/5 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove file content from the worker→main process IPC response since the main process only uses processedFiles[].path for the token count tree reporter. For a typical repo with 1000 files averaging 4KB each, this avoids ~4MB of structured clone serialization. Cherry-picked optimization from PR #1295 (4/5 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wrap sequential getGitDiffs() and getGitLogs() calls in Promise.all() since both are independent git subprocess operations. Saves the duration of the shorter call (~5-20ms) by overlapping their I/O. Cherry-picked optimization from PR #1295 (3/5 reviewer consensus). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Key Optimization 27: Tighten BasicAuth pre-filter to require scheme://...@ on same lineThe security pre-filter's BasicAuth check previously used two separate This caused ~93% false positives: 189 out of 195 files passing the pre-filter were sent to the expensive secretlint worker thread despite having no actual BasicAuth credentials. Fix: Replace the separate Benchmark (repomix self-pack, 981 files):
All 10 files matching both old and new patterns are true positives with |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Cache BPE token counts per file in a module-level Map keyed by encoding:path:charCount. On warm MCP/server pack() calls where files haven't changed, this eliminates the worker thread round-trip entirely (IPC serialization + BPE tokenization), returning cached results in ~0.07ms instead of ~39ms. The file content cache in fileRead.ts already validates freshness via mtime+size, so by the time metrics are computed, content is known-fresh. charCount acts as a lightweight change detector — if file content changes, its length almost certainly changes, invalidating the cache entry. For partial cache hits (some files changed), only the changed files are sent to the worker, reducing IPC payload proportionally. Cache is bounded to 5000 entries with FIFO eviction. Both worker-thread and main-thread fallback paths share the same cache. Warm pack() benchmark (25 runs, packing repomix repo ~1009 files): | | Median | p25 | p75 | |---|---|---|---| | Before | 97.1ms | 94.0ms | 105.9ms | | After | 92.9ms | 89.1ms | 98.3ms | | **Improvement** | **-4.2ms (-4.3%)** | | | The modest overall improvement is because file metrics were already overlapped with the security check in the parallel block. The real win is the phase-level improvement: | Phase | Before | After | Improvement | |---|---|---|---| | file-metrics | 38.99ms | 0.07ms | **-38.9ms (-99.8%)** | This frees the metrics worker thread for other work and eliminates ~80KB of IPC serialization overhead per warm call. The improvement compounds in scenarios where the security check is also fast (cached workers, few files matching the pre-filter), as the metrics phase was previously the longer branch in the parallel block. https://claude.ai/code/session_015KjXDgxLV8VmRWST6R4J1H
…cross pack() calls Five targeted optimizations to reduce redundant work on warm MCP/server pack() calls: 1. Cache processed files in processFilesMainThread (fileProcess.ts): On warm runs where file content hasn't changed (validated by raw content length — fileRead.ts already validates by mtime+size), skip the per-file trim() and object allocation loop. Cache is invalidated when processing config options change (truncateBase64, removeEmptyLines, etc.). 2. Cache tree string in formatPackToolResponse (mcpToolRuntime.ts): The MCP response includes a directory structure tree generated via generateTreeString(safeFilePaths, []) which takes ~11ms for 1000 files. On repeated pack() calls with the same file list, the cached tree is returned immediately. Validated by file count + first/last file paths as a fast change-detection heuristic. 3. Cache summary strings in createRenderContext (outputGenerate.ts): The header, purpose, guidelines, and notes strings depend only on config and instruction — not on file content. Cache them across pack() calls using config reference identity check. 4. Replace flatMap with direct loops in renderGroups (outputSplit.ts): Avoids intermediate array allocations when collecting processedFiles and allFilePaths across groups for the split output path. 5. Use concat instead of spread in calculateSelectiveFileMetrics.ts: Replace [...cachedResults, ...newResults] with cachedResults.concat(newResults) to avoid spread's intermediate iterator + copy overhead. MCP pack + response benchmark (50 runs, packing repomix repo ~1011 files): | | Baseline | After | Improvement | |----------|----------|---------|---------------------| | Median | 120.6ms | 116.9ms | -3.7ms (-3.1%) | | p25 | 116.9ms | 113.4ms | -3.5ms (-3.0%) | | p75 | 123.6ms | 121.6ms | -2.0ms (-1.6%) | Warm pack() benchmark (50 runs): | | Baseline | After | Improvement | |----------|----------|---------|---------------------| | Median | 114.8ms | 114.7ms | -0.1ms (within noise) | The improvement is concentrated in the MCP response path (tree cache ~11ms savings amortized across pipeline overhead). pack()-only path shows no measurable change since the tree generation runs in formatPackToolResponse, not in pack() itself. https://claude.ai/code/session_0134ro4Edgvmz42f5xXCYc3N
…ild process Previously, every MCP tool call (pack_codebase, pack_remote_repository, generate_skill) spawned a fresh child process via the `quiet: true` code path in defaultAction.ts. This meant each call paid the full cold startup cost (~580ms): child process spawn (~50ms), module loading (~200ms), and uncached pack() execution (~330ms). All 59 rounds of warm-path caching optimizations (file content cache, processed file cache, token count cache, worker pool reuse, search result cache) were completely wasted because each child process started with empty caches. Add `_inProcess` flag to CliOptions that MCP tools set to bypass the child process. pack() now runs directly in the MCP server process, enabling all module-level caches to persist across repeated tool calls. Memory remains bounded by existing cache limits (200MB file content, 5000 processed files, 5000 token counts, 16 search entries). The first MCP call still pays the cold cost (~580ms), but subsequent calls benefit from warm caches: file content validated by statSync instead of re-read, processed files returned from cache, token counts cached per-file, security/metrics worker pools pre-warmed. MCP pack_codebase benchmark (25 runs, packing repomix repo ~1009 files): | | Child Process (old) | In-Process Warm (new) | Improvement | |---|---|---|---| | Median | 570.3ms | 141.2ms | -429.1ms (-75.2%) | | p25 | 566.5ms | 134.4ms | -432.1ms | | p75 | 582.0ms | 152.0ms | -430.0ms | Cold first run unchanged at ~580ms (no caches populated yet). https://claude.ai/code/session_01FzMsphkoBQmcYVJRgQD92Y
Speculatively start the `git ls-files` subprocess during Zod config validation (~43ms) so the subprocess (~33ms) completes before searchFiles is called. The pre-started result is passed through pack() → searchFiles() which reuses it instead of spawning a new subprocess. - WHY: git ls-files only needs rootDir, not the full config. Starting it during config loading overlaps ~30ms of subprocess I/O with the ~43ms Zod validation phase, saving ~20-30ms on the critical path. - DECISION: Uses static import of node:child_process (execFile) to avoid dynamic import overhead contending with config module loading. - CONSTRAINT: Only pre-starts for single-directory mode (common case) and non-stdin mode. Multi-directory and stdin modes skip the optimization to keep logic simple. - SAFETY: Read-only git operation, no side effects. If config disables useGitignore, the pre-started result is simply ignored. For non-git repos, the promise catches and returns [], causing searchFiles to fall back to globby as usual. Interleaved A/B benchmark (8 runs each, packing repomix repo): WITH pre-start: median 488ms WITHOUT pre-start: median 519ms Improvement: ~31ms (-6%) https://claude.ai/code/session_01Tv33sxNbfhNMjtYARjPwmz
…e counting with write I/O Two optimizations targeting the warm pack() path (MCP/website server): 1. Sync cache probe in collectFiles: On warm runs, 95-100% of files hit the content cache. Previously, all files went through an async promise pool (~1000 async function frames + Promise resolutions) even when every readRawFileCached call was synchronous (statSync + Map lookup). Now, a plain for loop calls probeFileCache() synchronously for all files first. Only cache misses (typically 0-10 files on warm runs) enter the async pool for actual I/O. This eliminates ~1000 unnecessary Promise allocations and microtask resolutions. Collection time (warm, ~1010 files): ~32ms → ~12ms (-62%) 2. Overlap output line counting with disk write: The ~3.5ms indexOf-based line count scan (120K lines, 3.7MB) previously ran sequentially after the Promise.all(metrics, write) completed. Now it runs inside the Promise.all, so the CPU-bound scan overlaps with the I/O-bound disk write instead of adding to it. Also adds picospinner dependency required after rebase onto main (spinner refactoring in d1904df). Benchmark (25 warm runs, 5 warmup, ~1010 files): | Metric | Before | After | Change | |------------------|--------|-------|----------------| | Trimmed mean | 88ms | 83ms | -5ms (-5.7%) | | Median | 85ms | 82ms | -3ms (-3.5%) | | Collection phase | 32ms | 12ms | -20ms (-62%) | The modest total improvement despite large collection savings is because the security worker IPC round-trip (~21ms) is now the pipeline bottleneck in the parallel block, absorbing much of the time freed by faster collection. https://claude.ai/code/session_016ZtGEn6BAAbEY9hSk3iPTL
…-5MB allocation Two optimizations targeting the warm pack() hot path: 1. Cache security check results across pack() calls (securityCheck.ts): On warm MCP/server runs, file content hasn't changed since the last check. Cache results keyed by filePath + contentLength (validated by the upstream file content cache via mtime+size). When all tasks hit the cache, the worker IPC is skipped entirely — saving ~18ms of structured clone serialization + secretlint regex matching per warm call. 2. Stream output parts to disk without joining (outputStyles, writeOutputToDisk): Native renderers (xml, markdown, plain) now return string[] instead of joining ~6000 parts into a single 3-5MB contiguous string. The write path uses a WriteStream where stream.write() buffers synchronously (no per-part async overhead), and the metrics path already handles string[] via outputParts normalization. This eliminates the peak allocation of the full output string and reduces GC pressure during the write phase. Parsable styles (parsable-xml, json) still return string since they use library serializers (fast-xml-builder, JSON.stringify) that produce strings. Benchmark results (15 runs, 3 warmup, ~1010 files): | Metric | Before | After | Improvement | |---|---|---|---| | pack() trimmed mean | 96.0ms | 76.5ms | -19.5ms (-20.3%) | | pack() median | 97.5ms | 76.5ms | -21.0ms (-21.5%) | | Startup median | 73ms | 70ms | -3ms (-4.1%) | All 1085 tests pass, lint clean. https://claude.ai/code/session_01JmLDDWguPj8PEcdAQWvRE2
…ip unchanged disk writes Three optimizations targeting the warm pack() hot path: 1. **Cache-first security pre-filter** (securityCheck.ts): The SECRET_TRIGGER_PATTERN regex scanned all ~988 file contents (~3.6MB) on every warm pack() call, taking ~16ms even though all results were already cached. Now checks the security result cache BEFORE running the pre-filter, and caches pre-filter rejections (null results) so files that don't contain secret patterns are never re-scanned. On warm runs, the cache check loop runs in ~0.3ms (Map lookups only), completely eliminating the 16ms regex scan. 2. **Cache tree string across pack() calls** (fileTreeGenerate.ts): The directory tree string is deterministic given the same file list. On warm MCP/server runs where no files changed, the tree is identical. Cache validated by file count + first/last path + empty dir count + root count. Saves ~1.5ms per warm call. 3. **Skip disk write when output unchanged** (writeOutputToDisk.ts): On warm runs where file content hasn't changed, the output is identical. Track the total character count of the last write and skip re-writing 3-5MB to disk when unchanged. Verify file still exists via statSync to guard against external deletion. Saves ~10ms of I/O per warm call. pack() benchmark (25 warm runs, 5 warmup, ~988 files): | | Before | After | Improvement | |---|---|---|---| | Trimmed mean | 55.2ms | 21.1ms | **-34.1ms (-61.8%)** | | Median | 55.2ms | 19.8ms | **-35.4ms (-64.1%)** | https://claude.ai/code/session_015HARP7Uqx3mMjmjCkvXUoZ
…r pack) Replace async promisePool with synchronous readFileSync loop for cache-miss file reads during collectFiles. readFileSync avoids ~1000 Promise allocations, libuv threadpool scheduling, and microtask overhead per cold run. Key changes: - Add readRawFileSync() to fileRead.ts for synchronous UTF-8 file reading - collectFiles sync fast-read path uses readRawFileSync for cache misses - Non-UTF-8 files (~1%) fall back to async readRawFile with jschardet - Test mocks use the original async path via deps identity check Micro-benchmark (readFileSync vs async promisePool, 1000 files): readFileSync loop: 16ms promisePool(128): 120ms Speedup: 8x pack() benchmark (3 rounds each, in-process, packing repomix repo ~1009 files): | | Before | After | Improvement | |---|---|---|---| | Cold (avg) | 588ms | 528ms | -60ms (-10.2%) | | Warm (median) | 62ms | 65ms | ~same | CLI benchmark (15 runs, 3 warmup): | | Before | After | Improvement | |---|---|---|---| | Median | 841ms | 745ms | -96ms (-11.4%) | WHY: Async file reading via fs.readFile creates one Promise per file, each scheduled through libuv's 4-thread pool. With 1000 files, Promise allocation + microtask resolution + threadpool contention dominate the file collection phase. readFileSync bypasses all of this, going directly to the kernel where the VFS page cache serves recently-accessed inodes in ~0.016ms each. CONSTRAINT: readFileSync blocks the event loop, but this is acceptable because: (1) CLI processes are single-request and exit immediately after pack() (2) MCP/server warm runs have 95-100% cache hits via the existing sync probeFileCache path — only a few changed files use readFileSync https://claude.ai/code/session_01YS9ryAW6UvS7s6Y14UfqUN
…picospinner Pre-start metrics and security worker pools ~60ms earlier by beginning tinypool import at cliRun.ts module load time instead of inside pack(). The BPE table warmup (~300ms) now overlaps with Commander parsing, version logging, defaultAction import, and config loading — reducing idle wait from ~140ms to ~80ms. Also lazy-load picospinner via dynamic import() so the module is only loaded when the spinner is actually started (TTY mode). Non-TTY paths (--version, --quiet, --stdout, piped output, benchmarks) skip the ~2-3ms module load entirely. Implementation: - cliRun.ts: Module-level speculative import of processConcurrency.js starts tinypool loading during Commander setup - defaultAction.ts: Uses pre-loaded processConcurrency to create worker pools immediately, storing them in packager.ts module-level cache via new setPreWarmedMetricsPool/setPreWarmedSecurityPool exports - packager.ts: New setter functions for pre-warming the cached pools from outside pack() - cliSpinner.ts: Lazy-load picospinner in constructor, make start() async Benchmark (10 runs, 2 warmup, packing repomix repo ~1009 files): | | Before | After | Improvement | |---|---|---|---| | Median | 544ms | 481ms | **-63ms (-11.6%)** | https://claude.ai/code/session_01WcatA4CtbjGGN7EHJJtRSS
Conducted comprehensive performance investigation across 5 parallel scopes: 1. I/O & Filesystem operations 2. Memory allocation & GC pressure 3. Algorithms & data structures 4. Dependencies & startup time 5. Pipeline structure & parallelism All 10 high-priority optimization candidates identified are already implemented on this branch: ✅ O(n²) sortedFilePathsByDir → Map-based O(n) lookup ✅ O(n*m) filterOutUntrustedFiles → Set-based O(1) lookup ✅ localeCompare in sortTreeNodes → string operators (~3x faster) ✅ String += in treeToString → array accumulation (O(n²) → O(n)) ✅ calculateMarkdownDelimiter flatMap+match → single-pass charCodeAt ✅ calculateFileLineCounts match(/\n/g) → indexOf loop ✅ Sequential git diffs+logs → Promise.all ✅ Sequential permission checks → optimized single readdir ✅ Sequential split output writes → Promise.all ✅ Clipboard + disk write → Promise.all Additional already-done optimizations: ✅ tiktoken (WASM) replaced with gpt-tokenizer (pure JS) ✅ isBinaryPath check before fs.stat ✅ Lazy-load jschardet, iconv-lite, clipboardy, Handlebars ✅ Search result cache validated via .git/index mtime ✅ Per-file token count cache, security result cache ✅ Processed files + tree string + summary context cache ✅ Sync fast-path for cached file collection ✅ Pre-warm worker pools during config loading ✅ readFileSync for cold-run file collection Current benchmark results (~1009 files, repomix repo): Warm pack() (10 runs, median): 59.6ms Cold pack() (single run): 534ms CLI end-to-end (15 runs, median): 89ms Warm file search (cached): 0.18ms Remaining time dominated by fundamental operations: - File search validation: ~0.2ms (cached via .git/index mtime) - File collection statSync: ~12ms (mtime+size cache validation) - Metrics worker overhead: ~24ms (IPC even when tokens cached) - Security check: ~0.3ms (cached by content hash) No further optimizations found that would provide measurable improvement at the 1000-file scale. https://claude.ai/code/session_01SDk99Mp2WesN3JERkdCux8
… faster) Add a pack result cache that short-circuits the full processing pipeline when all inputs are unchanged between consecutive pack() calls. On warm MCP/server runs, file search, collection (stat validation), and git operations are the only work needed — processFiles, security check, metrics calculation, output generation, and all Promise.all orchestration are skipped entirely. Cache validation uses: - Config object identity (reference check) - File list identity (count + first/last path heuristic) - File content freshness (0 cache misses from collectFiles stat validation) - Git state identity (diff + log content lengths) Changes: - fileCollect.ts: Add `cacheMissCount` field to FileCollectResults so packager can detect when all files were served from cache (0 misses = nothing changed) - packager.ts: Add PackResultCacheEntry storing the last successful PackResult with its input signature. On cache hit, return immediately after collectFiles. pack() benchmark (20 warm runs, 3 warmup, ~987 files): | Metric | Before | After | Improvement | |--------------|---------|--------|----------------------| | Median | 25.5ms | 3.4ms | -22.1ms (-86.7%) | | Trimmed mean | 25.5ms | 3.4ms | -22.1ms (-86.7%) | | Min | 20.4ms | 2.9ms | -17.5ms | The fast path costs ~3.4ms (searchFiles 0.05ms + collectFiles stat validation 3ms + git await 0.1ms + cache check 0.05ms), versus ~25ms for the full pipeline. https://claude.ai/code/session_01LqhtHwcBu4dRJHx3JERArz
yamadashy
force-pushed
the
perf/auto-perf-tuning
branch
from
178778b to
3b0a2fd
Compare
Three targeted fixes: 1. Fix countOutputLines for string[] output parts (packager.ts): The string[] code path started each part's line count at 1, but parts are concatenated directly (no separator). This over-counted by (numParts) lines — ~6000 for a typical output with ~6000 parts. Now just counts newlines across all parts with count starting at 1. 2. Batch mkdir in website server ZIP extraction (fileUtils.ts): Per-file fs.mkdir was called for every file in the ZIP (~1000 calls). Pre-collect unique parent directories and batch-create them before writing files — matching the pattern already used in processZipFile.ts. Reduces ~1000 mkdir syscalls to ~100 for typical ZIPs. 3. Remove redundant fs.access in website server file copy (fileUtils.ts): fs.copyFile already fails with a clear error if the source doesn't exist, making the pre-check fs.access call unnecessary. Benchmark (CLI, 5 runs, 2 warmup, packing repomix repo ~1009 files): Median: 510ms (no regression from these fixes) Startup (--version, 10 runs, 2 warmup): Median: 72ms All 1090 tests pass, lint clean. https://claude.ai/code/session_01XeaZajSv4SYfQsz8dHjary
Partial cherry-pick from commit 75bec9e (#1295). Changes included: - Replace Zod instanceof check with duck typing in errorHandle.ts to avoid eagerly importing Zod on every CLI invocation (-22% startup time) - Replace O(n²) reduce+spread with flatMap in outputGenerate.ts - Remove redundant Set wrapping where inputs are already disjoint - Parallelize disk write and clipboard copy in produceOutput.ts - Remove unnecessary sort of file change counts in outputSort.ts - Add missing await to freeTokenCounters in calculateMetricsWorker.ts Excluded from cherry-pick: - tokenCounterFactory.ts (depends on gpt-tokenizer migration) - filePathSort.ts / fileTreeGenerate.ts (localeCompare changes risk altering sort order for non-ASCII file paths) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
intent(startup-perf): cherry-pick strip-comments lazy-loading from #1295 to reduce worker startup overhead decision(cherry-pick): partial cherry-pick — only strip-comments lazy-loading, excluding fileRead.ts changes that depend on prior restructuring commits rejected(fileRead-changes): lazy-loading of is-binary-path and isbinaryfile from same commit — deep conflicts with main due to file reading restructure constraint(imports): main branch still uses parseFile import from treeSitter — must keep alongside new ensureStripCommentsLoaded import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Partial cherry-pick from commit 75bec9e (#1295). Changes included: - Replace Zod instanceof check with duck typing in errorHandle.ts to avoid eagerly importing Zod on every CLI invocation (-22% startup time) - Replace O(n²) reduce+spread with flatMap in outputGenerate.ts - Remove redundant Set wrapping where inputs are already disjoint - Parallelize disk write and clipboard copy in produceOutput.ts - Remove unnecessary sort of file change counts in outputSort.ts - Add missing await to freeTokenCounters in calculateMetricsWorker.ts Excluded from cherry-pick: - tokenCounterFactory.ts (depends on gpt-tokenizer migration) - filePathSort.ts / fileTreeGenerate.ts (localeCompare changes risk altering sort order for non-ASCII file paths) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… faster server requests The website server's processZipFile and remoteRepo handlers were spawning a child process for each pack() call due to quiet: true without _inProcess flag. Each child process paid ~500ms of overhead (Node.js startup + ESM module re-loading + worker pool warmup for gpt-tokenizer BPE + @secretlint/core). Set _inProcess: true (matching the pattern already used by MCP tools) to run pack() directly in the server process. This reuses module-level cached worker pools across requests, eliminating the per-request spawn + warmup overhead. All module-level caches are bounded (200MB file content, 5000 entries for metrics/security/processing, 16 entries for search results), so memory growth is controlled in long-running server processes. Benchmark (5 runs, 2 warmup, packing repomix repo ~983 files): | Mode | Median | |---|---| | In-Process (_inProcess: true) | 122.4ms | | Child Process (before) | 581.2ms | | **Improvement** | **-458.8ms (-78.9%)** | https://claude.ai/code/session_018a2JAZXzPHMc5F2bb3kPLY
Partial cherry-pick from commit 75bec9e (#1295). Changes included: - Replace Zod instanceof check with duck typing in errorHandle.ts to avoid eagerly importing Zod on every CLI invocation (-22% startup time) - Replace O(n²) reduce+spread with flatMap in outputGenerate.ts - Remove redundant Set wrapping where inputs are already disjoint - Parallelize disk write and clipboard copy in produceOutput.ts - Remove unnecessary sort of file change counts in outputSort.ts - Add missing await to freeTokenCounters in calculateMetricsWorker.ts Excluded from cherry-pick: - tokenCounterFactory.ts (depends on gpt-tokenizer migration) - filePathSort.ts / fileTreeGenerate.ts (localeCompare changes risk altering sort order for non-ASCII file paths) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Partial cherry-pick from commit 75bec9e (#1295). Changes included: - Replace Zod instanceof check with duck typing in errorHandle.ts to avoid eagerly importing Zod on every CLI invocation (-22% startup time) - Replace O(n²) reduce+spread with flatMap in outputGenerate.ts - Remove redundant Set wrapping where inputs are already disjoint - Parallelize disk write and clipboard copy in produceOutput.ts - Remove unnecessary sort of file change counts in outputSort.ts - Add missing await to freeTokenCounters in calculateMetricsWorker.ts Excluded from cherry-pick: - tokenCounterFactory.ts (depends on gpt-tokenizer migration) - filePathSort.ts / fileTreeGenerate.ts (localeCompare changes risk altering sort order for non-ASCII file paths) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
intent(startup-perf): cherry-pick strip-comments lazy-loading from #1295 to reduce worker startup overhead decision(cherry-pick): partial cherry-pick — only strip-comments lazy-loading, excluding fileRead.ts changes that depend on prior restructuring commits rejected(fileRead-changes): lazy-loading of is-binary-path and isbinaryfile from same commit — deep conflicts with main due to file reading restructure constraint(imports): main branch still uses parseFile import from treeSitter — must keep alongside new ensureStripCommentsLoaded import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Partial cherry-pick from commit 75bec9e (#1295). Changes included: - Replace Zod instanceof check with duck typing in errorHandle.ts to avoid eagerly importing Zod on every CLI invocation (-22% startup time) - Replace O(n²) reduce+spread with flatMap in outputGenerate.ts - Remove redundant Set wrapping where inputs are already disjoint - Parallelize disk write and clipboard copy in produceOutput.ts - Remove unnecessary sort of file change counts in outputSort.ts - Add missing await to freeTokenCounters in calculateMetricsWorker.ts Excluded from cherry-pick: - tokenCounterFactory.ts (depends on gpt-tokenizer migration) - filePathSort.ts / fileTreeGenerate.ts (localeCompare changes risk altering sort order for non-ASCII file paths) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
intent(startup-perf): cherry-pick strip-comments lazy-loading from #1295 to reduce worker startup overhead decision(cherry-pick): partial cherry-pick — only strip-comments lazy-loading, excluding fileRead.ts changes that depend on prior restructuring commits rejected(fileRead-changes): lazy-loading of is-binary-path and isbinaryfile from same commit — deep conflicts with main due to file reading restructure constraint(imports): main branch still uses parseFile import from treeSitter — must keep alongside new ensureStripCommentsLoaded import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
intent(startup-perf): cherry-pick strip-comments lazy-loading from #1295 to reduce worker startup overhead decision(cherry-pick): partial cherry-pick — only strip-comments lazy-loading, excluding fileRead.ts changes that depend on prior restructuring commits rejected(fileRead-changes): lazy-loading of is-binary-path and isbinaryfile from same commit — deep conflicts with main due to file reading restructure constraint(imports): main branch still uses parseFile import from treeSitter — must keep alongside new ensureStripCommentsLoaded import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
yamadashy
force-pushed
the
perf/auto-perf-tuning
branch
2 times, most recently
from
69f42b3 to
e7755c4
Compare
# Conflicts: # package-lock.json # package.json # src/cli/prompts/skillPrompts.ts # src/config/configLoad.ts # src/core/file/fileRead.ts # src/core/metrics/calculateMetrics.ts # src/core/output/outputGenerate.ts # src/core/output/outputSort.ts # src/core/packager.ts # src/core/packager/produceOutput.ts # src/core/skill/packSkill.ts # src/core/skill/skillStyle.ts # src/core/skill/skillTechStack.ts # src/core/skill/writeSkillOutput.ts # src/mcp/tools/grepRepomixOutputTool.ts # tests/core/packager.test.ts # tests/core/packager/splitOutput.test.ts
3 participants
Summary
Fresh performance optimization pass on current
main, focusing on startup time reduction and algorithmic improvements.Key Optimization 1: Lazy-load CLI actions for 62% faster startup
All 5 CLI action handlers (
defaultAction,initAction,mcpAction,remoteAction,versionAction) were eagerly imported at startup, forcing Node.js to parse ~1,200 lines of action code plus their transitive dependencies (configLoad,packager, git modules,@clack/prompts, MCP SDK, etc.) regardless of which command was executed.Replaced static imports with dynamic
import()so each action module is only loaded when its code path is reached.Startup benchmark (
--version, 15 runs):Key Optimization 2: Lazy-load jschardet and iconv-lite
Only ~1% of source files need encoding detection (non-UTF-8). These modules (~130KB combined) were eagerly imported but rarely used. Now loaded via dynamic
import()only when UTF-8 decode fails.Also moved
isBinaryPathcheck beforefs.stat()to skip filesystem I/O entirely for obvious binary extensions (.png, .jpg, etc.).Key Optimization 3: Fix O(n²) file path regrouping
sortedFilePathsByDirinpackager.tsusedArray.find()+Array.includes()inside.filter(), causing O(n²) complexity for large file sets. Replaced withMap-based O(n) lookup.Key Optimization 4: Parallelize git diff and git log operations
getGitDiffsandgetGitLogswere awaited sequentially despite being independent I/O operations. Now run concurrently viaPromise.all().Key Optimization 5: Reduce GC pressure across hot paths
+=) with array accumulation (push+join). String+=in recursive loops causes O(n²) copying; array accumulation is O(n).string.includes+ charCode scan) to skip ~95% of files that have no base64 data.Set-based O(1) lookup instead ofArray.some()O(n) scan.flatMap+reduce(creates intermediate arrays) with single-passcharCodeAtloop.content.match(/\\n/g)(allocates array of all matches) withindexOfloop.split/map/joinwith regexcontent.replace(/[ \\t]+$/gm, '').split/filter/joinwith regexcontent.replace(/^\\s*\\n/gm, '').Key Optimization 28: Sync fast-path for cached file collection
On warm MCP/server runs, 95-100% of files hit the content cache. Previously all ~1000 files went through an async promise pool (~1000 async function frames + Promise resolutions) even when every
readRawFileCachedcall was synchronous (statSync + Map lookup). Now a plain for loop callsprobeFileCache()synchronously first, and only cache misses enter the async pool.Also overlaps the output line count scan (~3.5ms for 120K lines) with the disk write I/O inside
Promise.allinstead of running it sequentially after.Collection phase (warm, ~1010 files): ~32ms → ~12ms (-62%)
Key Optimization 29: Cache security results and stream output parts
Two optimizations targeting the warm pack() hot path:
Cache security check results across pack() calls (securityCheck.ts): On warm MCP/server runs, file content hasn't changed since the last check. Cache results keyed by filePath + contentLength (validated by the upstream file content cache via mtime+size). When all tasks hit the cache, the worker IPC is skipped entirely — saving ~18ms of structured clone serialization + secretlint regex matching per warm call.
Stream output parts to disk without joining (outputStyles, writeOutputToDisk): Native renderers (xml, markdown, plain) now return
string[]instead of joining ~6000 parts into a single 3-5MB contiguous string. The write path uses a WriteStream wherestream.write()buffers synchronously (no per-part async overhead), and the metrics path already handlesstring[]via outputParts normalization. This eliminates the peak allocation of the full output string and reduces GC pressure during the write phase.Key Optimization 30: Skip security pre-filter regex, cache tree string, and skip unchanged disk writes
Three optimizations targeting the warm pack() hot path:
Cache-first security pre-filter (securityCheck.ts): The
SECRET_TRIGGER_PATTERNregex scanned all ~988 file contents (~3.6MB) on every warm pack() call, taking ~16ms even though all results were already cached. Now checks the security result cache BEFORE running the pre-filter, and caches pre-filter rejections (null results) so files that don't contain secret patterns are never re-scanned. On warm runs, the cache check loop runs in ~0.3ms (Map lookups only), completely eliminating the 16ms regex scan.Cache tree string across pack() calls (fileTreeGenerate.ts): The directory tree string is deterministic given the same file list. On warm MCP/server runs where no files changed, the tree is identical. Cache validated by file count + first/last path + empty dir count + root count. Saves ~1.5ms per warm call.
Skip disk write when output unchanged (writeOutputToDisk.ts): On warm runs where file content hasn't changed, the output is identical. Track the total character count of the last write and skip re-writing 3-5MB to disk when unchanged. Verify file still exists via statSync to guard against external deletion. Saves ~10ms of I/O per warm call.
pack() benchmark (25 warm runs, 5 warmup, ~988 files):
Key Optimization 31: Use readFileSync for cold-run file collection (~11% faster pack)
Replace async promisePool with synchronous readFileSync loop for cache-miss file reads during collectFiles. Async
fs.readFilecreates one Promise per file, each scheduled through libuv's 4-thread pool. With ~1000 files, Promise allocation + microtask resolution + threadpool contention dominate the file collection phase.readFileSyncbypasses all of this, going directly to the kernel where the VFS page cache serves recently-accessed inodes in ~0.016ms each.Non-UTF-8 files (~1%) fall back to async readRawFile with jschardet encoding detection.
Micro-benchmark (1000 files, 7.8MB total):
pack() benchmark (3 rounds, in-process, ~1009 files):
CLI benchmark (15 runs, 3 warmup):
Key Optimization 32: Pre-warm worker pools during config loading and lazy-load picospinner
Pre-start metrics and security worker pools ~60ms earlier by beginning tinypool import at cliRun.ts module load time instead of inside pack(). The BPE table warmup (~300ms) now overlaps with Commander parsing, version logging, defaultAction import, and config loading — reducing idle wait from ~140ms to ~80ms.
Also lazy-load picospinner via dynamic
import()so the module is only loaded when the spinner is actually started (TTY mode). Non-TTY paths (--version, --quiet, --stdout, piped output) skip the ~2-3ms module load entirely.CLI benchmark (10 runs, 2 warmup, packing repomix repo ~1009 files):
Key Optimization 33: Cache entire pack() result for warm MCP/server runs
On warm MCP/server runs where file list, file content, git state, and config are all unchanged between consecutive pack() calls, the entire pipeline output is identical. Added a pack result cache that short-circuits the full processing pipeline after just searchFiles + collectFiles (stat validation) + git await.
When the cache hits, processFiles, security check, metrics calculation, output generation, disk write, and all Promise.all orchestration overhead (~20ms total) are skipped entirely.
Cache validation uses config object identity, file list identity (count + first/last path heuristic), file content freshness (0 cache misses from collectFiles stat validation), and git state identity (diff + log content lengths).
pack() benchmark (20 warm runs, properly warmed, ~987 files):
Key Optimization 34: Fix output line over-count and batch ZIP mkdir
Three targeted fixes:
Fix countOutputLines for string[] output parts (packager.ts): The string[] code path started each part's line count at 1 (
partLines = 1), but parts are concatenated directly (no separator between them). This over-counted by ~(numParts) lines — approximately 6000 for a typical output with ~6000 parts. Now just counts newlines across all parts with count starting at 1 for the first line.Batch mkdir in website server ZIP extraction (fileUtils.ts): Per-file
fs.mkdirwas called for every file in the ZIP (~1000 calls). Pre-collect unique parent directories and batch-create them before writing files — matching the pattern already used in processZipFile.ts. Reduces ~1000 mkdir syscalls to ~100 for typical ZIPs (~15-30ms saved).Remove redundant fs.access in website server file copy (fileUtils.ts):
fs.copyFilealready fails with a clear error if the source doesn't exist, making the pre-checkfs.access()call unnecessary. Eliminates 1 syscall per copy operation.Key Optimization 35: Run website server pack() in-process instead of child process (~79% faster)
The website server's
processZipFileandremoteRepohandlers were spawning a child process for each pack() call due toquiet: truewithout the_inProcessflag. Each child process paid ~500ms of overhead: Node.js startup + ESM module re-loading + worker pool warmup (gpt-tokenizer BPE tables + @secretlint/core initialization).Set
_inProcess: true(matching the pattern already used by MCP tools) to run pack() directly in the server process. This reuses module-level cached worker pools across requests, eliminating the per-request spawn + warmup overhead. All module-level caches are bounded (200MB file content, 5000 entries for metrics/security/processing), so memory growth is controlled.Server-like benchmark (5 runs, 2 warmup, packing repomix repo ~983 files):
Benchmark results
Startup benchmark (
--version, 10 runs, 2 warmup):pack() benchmark (20 warm runs, properly warmed, ~987 files):
Server-like execution (warm, ~983 files):
Checklist
npm run test(1090 tests pass)npm run lint(clean)https://claude.ai/code/session_018a2JAZXzPHMc5F2bb3kPLY