fix(workers): resilient + zero-copy ingestion worker pool — prevent analyze hangs on TS-root-scale loads

gitnexus/README.md

@@ -359,6 +359,16 @@ npx gitnexus analyze
 
 For repositories with very large source files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget. The default is **8388608 bytes (8 MB)**.
 
+### Worker pool resilience tuning
+
+Three env vars expose the pool's resilience layers (respawn budget, cumulative-timeout cap, circuit breaker). Defaults are tuned for typical repos; bump them when an analyze legitimately needs more retries, or lower them to fail-fast on a known-bad shape.
+
+| Variable                                          | Default                   | Effect                                                                                                                            |
+| ------------------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT`            | `3`                       | Max replacement spawns per slot before the slot is dropped from the active rotation.                                              |
+| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS`        | `5 × subBatchTimeoutMs`   | Total retry wall-time budget per job before quarantining. Bounds exponentially-growing retry waits.                              |
+| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD`    | `max(3, poolSize)`        | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, dispatches require a fresh pool.            |
+
 ## Privacy
 
 - All processing happens locally on your machine

gitnexus/bench/parse-throughput.md

@@ -0,0 +1,175 @@
+# Parse-throughput benchmark (scaffold)
+
+> **Status: methodology + harness scaffold, no measurement data yet.**
+> The Latest measurement table below contains `_TBD_` placeholders.
+> This file ships intentionally without numbers — populating it
+> requires a dedicated bench-pass against the U6 fixture (and ideally
+> a real-world TS-root-scale repo) on consistent hardware, which is
+> tracked as future work rather than gated on PR #1693's merge.
+> Until the table is populated, the load-bearing perf-regression
+> protection lives in `gitnexus/test/integration/parse-impl-large-fixture.test.ts`
+> (U6, 30 s wall-clock budget via `Promise.race`).
+
+Tracks `runChunkedParseAndResolve` wall-clock + peak heap on a synthetic
+fixture so PR #1693's "analyze no longer hangs on TS-root-shaped loads"
+claim is measurable, not just asserted by smoke tests. The harness
+recipe below is deliberately small enough to re-run in a few minutes
+when the bench-pass is undertaken.
+
+---
+
+## Methodology
+
+### Fixture
+
+Synthetic TypeScript repo, _not_ a clone of microsoft/TypeScript. CI cost
+of cloning real-world repos is prohibitive; the synthetic shape exercises
+the same pipeline paths (chunking, deferred extraction, cross-chunk
+imports + heritage) without the disk-I/O overhead. Larger numbers can be
+manually captured against real repos and cross-referenced here, but the
+authoritative regression-tracking shape is the synthetic fixture so runs
+are reproducible across hardware.
+
+The fixture matches the structure pinned by
+`gitnexus/test/integration/parse-impl-large-fixture.test.ts` (U6):
+
+- 15 small modules (`mod0.ts` … `mod14.ts`), one exported function each.
+- 1 dense `complex.ts` with 30 functions + 1 class + 1 interface.
+- 1 `index.ts` re-exporting every symbol from every module.
+
+`GITNEXUS_CHUNK_BYTE_BUDGET=64` forces multi-chunk parsing on this small
+fixture — without that override the whole thing fits in one chunk and
+the deferred-extraction path is not exercised end-to-end.
+
+### What to measure
+
+| Metric                      | How                                                                              |
+| --------------------------- | -------------------------------------------------------------------------------- |
+| Wall-clock total            | `Date.now()` delta around `runChunkedParseAndResolve`                            |
+| Peak heap                   | Sample `process.memoryUsage().heapUsed` every 50 ms during the run; keep the max |
+| Chunks observed             | Count distinct `Parsing chunk X/Y` progress messages                             |
+| `getStats()` final snapshot | Quarantined paths, dropped slots, breaker state                                  |
+
+### Hardware shape (record alongside each measurement)
+
+- OS + version
+- CPU model + logical core count
+- RAM
+- Node version
+- gitnexus commit SHA (so the snapshot is anchored to a tree, not "main")
+
+---
+
+## Harness recipe
+
+The U6 test (`test/integration/parse-impl-large-fixture.test.ts`) is the
+checked-in mini-benchmark — it exercises the same fixture and bounds the
+wall-clock at 30 s via `Promise.race`. To produce a richer snapshot for
+this doc, run it under instrumentation:
+
+```bash
+# From the gitnexus/ subdir:
+cd gitnexus
+# Single-threaded baseline (sequential fallback):
+npx vitest run test/integration/parse-impl-large-fixture.test.ts --reporter=verbose
+
+# Worker-pool path (requires built dist/ — pre-built by `npm run build`):
+npm run build && \
+  GITNEXUS_WORKER_POOL_SIZE=4 \
+  GITNEXUS_PARSE_CHUNK_CONCURRENCY=2 \
+  GITNEXUS_VERBOSE=1 \
+  npx vitest run test/integration/parse-impl-large-fixture.test.ts --reporter=verbose
+```
+
+For peak-heap sampling, wrap the dispatch call in a Node script that
+polls `process.memoryUsage()`. A future helper at
+`gitnexus/bench/scripts/parse-throughput.ts` would automate this — the
+plan's stretch goal. Until that lands, capture peak heap manually via:
+
+```bash
+node --inspect=0 \
+  --require ./scripts/heap-sampler.js \
+  ./node_modules/.bin/vitest run test/integration/parse-impl-large-fixture.test.ts
+```
+
+---
+
+## Latest measurement
+
+> _No measurement data has been collected yet — this file is the
+> methodology + harness scaffold. The single recorded data point is the
+> U6 wall-clock smoke baseline below; the worker-pool rows are
+> placeholders for future bench-pass output._
+
+The U6 integration test (`gitnexus/test/integration/parse-impl-large-fixture.test.ts`)
+was observed completing the synthetic fixture in **~6 seconds** under
+the sequential path (`skipWorkers: true`) on the development machine,
+well under the 30 s `Promise.race` wall-clock budget. That number is a
+smoke baseline only — recorded here for reference, not as a regression
+target.
+
+| Path                                       | files/s | wall-clock           | peak heap | chunks | quarantined |
+| ------------------------------------------ | ------- | -------------------- | --------- | ------ | ----------- |
+| Sequential (`skipWorkers: true`, U6 smoke) | _TBD_   | ~6 s _(observation)_ | _TBD_     | 17     | 0           |
+| Worker pool, `--workers 4`, concurrency 2  | _TBD_   | _TBD_                | _TBD_     | _TBD_  | 0           |
+| Worker pool, `--workers 1`, concurrency 1  | _TBD_   | _TBD_                | _TBD_     | _TBD_  | 0           |
+
+**Hardware:** _TBD — record OS, CPU, RAM, Node version, gitnexus SHA at
+the time of the bench-pass that populates the table above._
+
+---
+
+## Operator-tuning quick reference
+
+Cross-links to the env vars documented in the [README](../../README.md#environment-variables).
+Use this section as a starting point when the benchmark numbers above
+suggest a tuning opportunity for your hardware shape.
+
+- **CPU-bound, big repo, lots of cores:** raise `GITNEXUS_WORKER_POOL_SIZE`
+  past the default cap of 16. The 16-worker cap exists because past that
+  point main-thread merge / extraction dominates; if you've measurably
+  ruled that out, the env var lifts the cap explicitly. (See
+  `worker-pool.ts` `DEFAULT_POOL_SIZE_CAP`.)
+- **Slow files (large minified JS, deep TS types):** raise
+  `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` past 30 000 ms. The cumulative
+  budget is 5× this value (U10 pins this) so a 60 s idle timeout permits
+  300 s of total retry-and-split wall-clock before quarantining the file.
+- **Constrained container (cgroup CPU limit):** the pool now uses
+  `os.availableParallelism()` (U3 H2), which honors cgroup limits — no
+  manual `GITNEXUS_WORKER_POOL_SIZE` override needed unless the auto-
+  resolved value is too aggressive for your I/O budget.
+- **Long-running host (eval-server, MCP daemon) running back-to-back
+  analyzes:** `--workers` is now threaded through `AnalyzeOptions`
+  (U2 B2), so per-invocation sizing is honored without `process.env`
+  state leaking across calls. `GITNEXUS_VERBOSE` is similarly snapshot/
+  restore-bracketed.
+
+---
+
+## What this benchmark does NOT measure
+
+- **Real-repo performance.** The synthetic fixture is sized for CI; it
+  doesn't exercise the cumulative-load shape (50k files, occasional
+  pathological file) that drove the original PR #1693 hang report. Real-
+  repo numbers should be captured ad-hoc against the user's target repo
+  and cross-referenced here only as supplementary evidence.
+- **Worker-pool resilience under real crashes.** That's verified by the
+  `worker-pool.test.ts` integration tests (real `process.exit`, real
+  `error` events, real protocol violations) and the unit suite. The
+  benchmark cares about throughput on the happy path.
+- **IPC repack throughput.** Phase 3 of the PR #1693 plan introduces a
+  transferList + binary wire-format IPC repack (U16-U17). Once that
+  lands, an `IPC repack` row should be added to the "Latest measurement"
+  table above with before/after numbers on the same hardware.
+
+---
+
+## Related artifacts
+
+- Plan: `docs/plans/2026-05-20-001-feat-pr1693-resilience-hardening-and-ipc-repack-plan.md`
+- Integration test (mini-benchmark with wall-clock guard): `gitnexus/test/integration/parse-impl-large-fixture.test.ts` (U6)
+- Operator env-var reference: `README.md` → Environment variables
+- Resilience layer tests: `gitnexus/test/unit/worker-pool-resilience.test.ts`,
+  `worker-pool-cumulative-timeout.test.ts`,
+  `worker-pool-windows-quarantine.test.ts`,
+  `worker-pool-slot-generation.test.ts`

gitnexus/src/cli/analyze.ts

@@ -148,7 +148,7 @@ function ensureHeap(): boolean {
       stdio: 'inherit',
       env: { ...process.env, NODE_OPTIONS: `${nodeOpts} ${HEAP_FLAG}`.trim() },
     });
-  } catch (e: any) {
+  } catch (e: unknown) {
     if (childProcessLikelyOom(e)) {
       cliError(
         `  Analysis likely ran out of memory.\n` +
@@ -159,11 +159,50 @@ function ensureHeap(): boolean {
         { recoveryHint: 'heap-oom-respawn' },
       );
     }
-    process.exitCode = e.status ?? 1;
+    const status =
+      typeof e === 'object' && e !== null && 'status' in e && typeof e.status === 'number'
+        ? e.status
+        : 1;
+    process.exitCode = status;
   }
   return true;
 }
 
+/**
+ * GITNEXUS_* env vars that `analyzeCommand` writes for backward-compatible
+ * downstream consumption. Snapshotted at function entry and restored in the
+ * finally block so that programmatic callers (tests, long-running hosts)
+ * don't see leaked state across invocations. `GITNEXUS_WORKER_POOL_SIZE` is
+ * NOT in this list: that knob is threaded through `runFullAnalysis` options
+ * (see `workerPoolSize` plumbing) so the CLI never has to mutate `process.env`
+ * for it in the first place.
+ */
+const ANALYZE_CLI_ENV_KEYS = [
+  'GITNEXUS_VERBOSE',
+  'GITNEXUS_MAX_FILE_SIZE',
+  'GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS',
+  'GITNEXUS_EMBEDDING_THREADS',
+  'GITNEXUS_EMBEDDING_BATCH_SIZE',
+  'GITNEXUS_EMBEDDING_SUB_BATCH_SIZE',
+  'GITNEXUS_EMBEDDING_DEVICE',
+] as const;
+
+type AnalyzeEnvSnapshot = Record<(typeof ANALYZE_CLI_ENV_KEYS)[number], string | undefined>;
+
+const snapshotAnalyzeEnv = (): AnalyzeEnvSnapshot => {
+  const snap = {} as AnalyzeEnvSnapshot;
+  for (const k of ANALYZE_CLI_ENV_KEYS) snap[k] = process.env[k];
+  return snap;
+};
+
+const restoreAnalyzeEnv = (snap: AnalyzeEnvSnapshot): void => {
+  for (const k of ANALYZE_CLI_ENV_KEYS) {
+    const v = snap[k];
+    if (v === undefined) delete process.env[k];
+    else process.env[k] = v;
+  }
+};
+
 export interface AnalyzeOptions {
   force?: boolean;
   repairFts?: boolean;
@@ -226,6 +265,8 @@ export interface AnalyzeOptions {
   maxFileSize?: string;
   /** Override worker sub-batch idle timeout in seconds. */
   workerTimeout?: string;
+  /** Parse worker pool size; 0 disables workers (sequential fallback). */
+  workers?: string;
   embeddingThreads?: string;
   embeddingBatchSize?: string;
   embeddingSubBatchSize?: string;
@@ -259,6 +300,22 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
   // a stack trace and a non-zero exit code instead of a silent exit 0.
   installFatalHandlers();
 
+  // Snapshot the GITNEXUS_* env vars that the impl writes for downstream
+  // consumption, so they don't leak across `analyzeCommand` invocations in
+  // programmatic callers (tests, long-running hosts). `process.exit(0)` on
+  // the success path bypasses `finally` — intentional: when the process is
+  // exiting, restoration is moot. For early-return paths (validation
+  // errors) and the alreadyUpToDate fast path the finally restores the
+  // pre-call values.
+  const envSnap = snapshotAnalyzeEnv();
+  try {
+    await analyzeCommandImpl(inputPath, options);
+  } finally {
+    restoreAnalyzeEnv(envSnap);
+  }
+};
+
+const analyzeCommandImpl = async (inputPath?: string, options?: AnalyzeOptions): Promise<void> => {
   if (options?.verbose) {
     process.env.GITNEXUS_VERBOSE = '1';
   }
@@ -279,6 +336,26 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
     );
   }
 
+  // `--workers` is threaded through `runFullAnalysis` options → PipelineOptions
+  // → createWorkerPool, intentionally bypassing the GITNEXUS_WORKER_POOL_SIZE
+  // env channel so this CLI surface never mutates `process.env` for pool size.
+  // Tests can therefore re-invoke analyzeCommand with different --workers
+  // values back-to-back and observe the value they passed, not whatever the
+  // previous call leaked.
+  let workerPoolSize: number | undefined;
+  if (options?.workers !== undefined) {
+    const parsedWorkers = Number(options.workers);
+    if (!Number.isInteger(parsedWorkers) || parsedWorkers < 0) {
+      cliError(
+        '  --workers must be a non-negative integer. ' +
+          'Pass 0 to disable the worker pool (sequential fallback).\n',
+      );
+      process.exitCode = 1;
+      return;
+    }
+    workerPoolSize = parsedWorkers;
+  }
+
   // Parse `--embeddings [limit]`: `true` → default cap, string → numeric cap
   // (0 disables the cap entirely). Validated up here so failures match the
   // sibling-validation pattern (exit before bar.start() — otherwise
@@ -551,6 +628,10 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
         // be able to accept the duplicate name without also paying the
         // cost of a full pipeline re-index. See #829 review round 2.
         allowDuplicateName: options?.allowDuplicateName,
+        // Worker pool size threaded from --workers, replacing the previous
+        // GITNEXUS_WORKER_POOL_SIZE env mutation. `undefined` defers to the
+        // env / auto-formula fallback inside the pipeline.
+        workerPoolSize,
       },
       {
         onProgress: (_phase, percent, message) => {
@@ -688,7 +769,7 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
     }
 
     console.log('');
-  } catch (err: any) {
+  } catch (err: unknown) {
     clearInterval(elapsedTimer);
     process.removeListener('SIGINT', sigintHandler);
     console.log = origLog;
@@ -698,7 +779,7 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
     console.error = origError;
     bar.stop();
 
-    const msg = err.message || String(err);
+    const msg = err instanceof Error ? err.message : String(err);
 
     // Registry name-collision from --name (#829) — surface as an
     // actionable error rather than a generic stack-trace.

gitnexus/src/cli/index.ts

@@ -71,6 +71,10 @@ program
     '--worker-timeout <seconds>',
     'Worker sub-batch idle timeout before retry/fallback. Default: 30.',
   )
+  .option(
+    '--workers <n>',
+    'Parse worker pool size. Default: cores-1 capped at 16. Pass 0 to disable workers (sequential).',
+  )
   .option('--embedding-threads <n>', 'Limit local ONNX embedding CPU threads')
   .option('--embedding-batch-size <n>', 'Number of nodes per embedding batch')
   .option('--embedding-sub-batch-size <n>', 'Number of chunks per embedding model call')
@@ -82,6 +86,11 @@ program
       '  GITNEXUS_MAX_FILE_SIZE=N  Override large-file skip threshold (KB). Default 512, max 32768.\n' +
       '  GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N  Worker idle timeout in milliseconds. Default 30000.\n' +
       '  GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N  Worker job byte budget. Default 8388608.\n' +
+      '  GITNEXUS_WORKER_POOL_SIZE=N  Parse worker count override. Default cores-1 capped at 16.\n' +
+      '  GITNEXUS_PARSE_CHUNK_CONCURRENCY=N  Concurrent in-flight parse chunks. Default 2.\n' +
+      '  GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N  Max replacement spawns per slot before drop. Default 3.\n' +
+      '  GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N  Total retry wall-time per job. Default 5x sub-batch timeout.\n' +
+      '  GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N  Per-slot deaths to trip circuit breaker. Default max(3, poolSize).\n' +
       '  GITNEXUS_EMBEDDING_THREADS=N  Limit local ONNX CPU threads for --embeddings.\n' +
       '  GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N  Max embedding chunks for exact-scan fallback. Default 10000.\n' +
       '\nTip: `.gitnexusignore` supports `.gitignore`-style negation. Add e.g.\n' +

gitnexus/src/cli/wiki.ts

@@ -107,6 +107,24 @@ function prompt(question: string, hide = false): Promise<string> {
 }
 
 export const wikiCommand = async (inputPath?: string, options?: WikiCommandOptions) => {
+  // Snapshot GITNEXUS_VERBOSE at entry — wikiCommand mutates it (the impl
+  // below) so cursor-client (process.env-driven) sees the right value during
+  // this run. Restored in finally so back-to-back wiki calls in long-running
+  // hosts don't leak verbose state from one invocation to the next. Pairs
+  // with the same snapshot/restore pattern in `analyzeCommand`.
+  const originalVerbose = process.env.GITNEXUS_VERBOSE;
+  try {
+    await wikiCommandImpl(inputPath, options);
+  } finally {
+    if (originalVerbose === undefined) {
+      delete process.env.GITNEXUS_VERBOSE;
+    } else {
+      process.env.GITNEXUS_VERBOSE = originalVerbose;
+    }
+  }
+};
+
+const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions): Promise<void> => {
   // Set verbose mode globally for cursor-client to pick up
   if (options?.verbose) {
     process.env.GITNEXUS_VERBOSE = '1';