feat(install): toolchain-free tree-sitter via vendored prebuilds

[magyargergo]

Warning

DRAFT — DO NOT MERGE until gitnexus/vendor/tree-sitter-kotlin/prebuilds/ is populated by the build-tree-sitter-prebuilds workflow. Until then Kotlin is vendored with an empty prebuilds/ and no source-build fallback, so a fresh install has no Kotlin (even on toolchain hosts). dart/proto remain fully functional throughout.

Goal

Eliminate the C/C++-toolchain requirement at install for every tree-sitter grammar GitNexus uses ("no operational risk for any tree-sitter"), by generating and vendoring native prebuilds — mirroring the existing vendored tree-sitter-swift.

Why this scope (audit-driven)

Of 15 grammars, only 3 are at risk: dart + proto (vendored but compiled from source at postinstall) and kotlin (third-party optionalDependency, source-only). The other 10 already ship 6 upstream prebuilds and stay dependency-review-tracked — left as npm dependencies. swift already vendors upstream prebuilds. Because dart/proto are already off the dependency graph, only kotlin newly leaves it (the sole new CVE-tracking blind spot; mitigated by a recommended drift-check job).

What this PR does

• .github/workflows/build-tree-sitter-prebuilds.yml — registry-parameterized workflow building {dart,proto,kotlin} × {linux,darwin,win32}-{x64,arm64} natively (no cross-compile), validating each .node loads and parses on its arch, then opening a PR that vendors them. A guard job gates the heavy matrix to run only on dispatch or a real grammar-version change — ordinary code PRs cost zero matrix minutes.
• dart/proto — prefer a committed prebuild; fall back to today's source build when none matches → no regression before prebuilds exist.
• kotlin → vendored (Swift parity; supersedes the merged fix(install): graceful Kotlin optional-grammar install + accurate toolchain docs #2110 optionalDependency mechanism). The ~23 MB parser.c is not vendored (the workflow builds from the published npm package); only node-types.json + bindings + prebuilds are. Removed from optionalDependencies; lockfile regenerated; probe / parser-loader note / README+devcontainer docs / the fix(install): graceful Kotlin optional-grammar install + accurate toolchain docs #2110 tests all updated.

Verification (local)

tsc --noEmit clean · 44 unit tests pass (incl. updated cli-commands + build-tree-sitter-kotlin-probe) · prettier + eslint clean · all build scripts node --check + exit-0 smoke · workflow YAML + embedded scripts validated. The actual cross-platform prebuild load is only provable by dispatching the workflow (that's the point of the first run).

To complete (open decisions)

1. Confirm arm64 hosted runners (ubuntu-24.04-arm, windows-11-arm) are available to this repo's plan — else those legs queue forever and no prebuild PR lands.
2. Pin 3 placeholder action SHAs (setup-python, attest-build-provenance) + zizmor/Scorecard allowlist.
3. RELEASE_APP_* token must have Contents+PRs write for the aggregate job (publish.yml pattern).
4. Dispatch the workflow → it builds the binaries and opens the vendoring PR → populate prebuilds/ here → un-draft.
5. Optional: a weekly drift-check job (recovers the kotlin CVE signal) and closing the tree-sitter-c 4/6-prebuild gap when the tree-sitter 0.21→0.23 runtime bump lands.

🤖 Generated with Claude Code

[vercel]

@magyargergo is attempting to deploy a commit to the NexusCore Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

CI Report

✅ All checks passed

Pipeline Status

Stage	Status	Details
✅ Typecheck	success	tsc --noEmit
✅ Tests	success	unit tests, 3 platforms
✅ E2E	success	gitnexus-web changes only

Test Results

Tests	Passed	Failed	Skipped	Duration
10841	10825	0	16	534s

✅ All 10825 tests passed

16 test(s) skipped — expand for details

• COBOL pipeline benchmark > scales with file count
• C++ ADL emit benchmark > emit phase scales sub-quadratically with co-scaled files and sites
• C++ pipeline benchmark > scales with file count
• C# pipeline benchmark > scales with file count — namespaces spread across the solution
• C# pipeline benchmark > scales with file count — all types in one (global) namespace bucket
• C# pipeline benchmark > scales with file count — all types in one (named) namespace bucket
• Go pipeline benchmark > scales with file count (workers enabled)
• Go pipeline benchmark — worker pool (issue Worker idle timeout kills long Go scope extraction and surfaces as Napi::Error during analyze #1848) > does not quarantine the large generated Go file on sub-batch idle timeout
• Go structural interface detection benchmark > scales linearly with interface × struct count
• Go structural interface detection split-phase benchmark > separates index-build and detection time
• PHP pipeline benchmark > scales with file count (workers enabled)
• Ruby pipeline benchmark > scales with file count (workers enabled)
• Rust pipeline benchmark > scales with file count (workers enabled)
• Vue pipeline benchmark > scales with component count
• run.cjs direct-exec entrypoint (fix(cli): steer docs, skills, and hooks through a CLI-neutral project-local runner (#1939) #1945) > resolves a .cmd shim via the Windows shell branch, passing args and exit code
• buildTypeEnv > known limitations (documented skip tests) > Ruby block parameter: users.each { |user| } — closure param inference, different feature

Code Coverage

Tests

Metric	Coverage	Covered	Base	Delta	Status
Statements	75.07%	35450/47221	N/A%	—	🟢 ███████████████░░░░░
Branches	62.81%	21904/34869	N/A%	—	🟢 ████████████░░░░░░░░
Functions	80.81%	3829/4738	N/A%	—	🟢 ████████████████░░░░
Lines	78.87%	32053/40639	N/A%	—	🟢 ███████████████░░░░░

---

_{📋 View full run · Generated by CI}

[magyargergo]

Added: full prebuild/ABI coverage verification + regression guard

Verified every tree-sitter grammar for native-binding coverage across {linux,darwin,win32}-{x64,arm64} on the ABI we support (Node ≥22 / N-API; tree-sitter language ABI 0.21.1). Method: inspected the installed tarball contents + checked each .node for the N-API entry symbol napi_register_module_v1 on all six platforms (including darwin/win32, by binary inspection — no need to run them).

Result: 10 npm grammars + vendored swift = full 6/6 N-API. The only gaps are dart/proto/kotlin (0/6 — what this PR fixes) and tree-sitter-c 4/6 (missing linux-arm64/win32-arm64) → filed as #2116.

New regression guard (gitnexus/test/unit/prebuild-coverage.test.ts):

• Every vendored grammar must ship a loadable N-API prebuild for all 6 tuples. This is intentionally RED for dart/proto/kotlin right now — it is the must-fill-before-merge gate. Once the build-tree-sitter-prebuilds workflow populates prebuilds/, it goes green. (swift already passes.)
• npm grammars are checked too (catches a future upstream platform drop); tree-sitter-c is allow-listed at 4/6 pending tree-sitter-c ships only 4/6 native prebuilds (missing linux-arm64, win32-arm64) — toolchain needed on ARM #2116, and the guard fails if that gap is silently closed (prompting allow-list removal).

So the tests / ubuntu / coverage check will show the 3 vendored failures until the binaries land — that's the gate working, not a broken test.

[magyargergo]

Tri-review (3 methods, Codex live) — toolchain-free vendored tree-sitter

Methods & engines. GitNexus swarm + Compound-Engineering personas (both Claude) + Codex (the one independent engine, live) + two requested personas (DevOps/release-pipeline, npm native-build). 4 of 6 Claude lanes returned full structured findings (correctness, adversarial, DevOps, npm-native); 2 (risk, test/CI) ended mid-investigation, their domains covered by the others. Claude-lane agreement = "consistent across personas," not independent; the strong signal is Codex+Claude.

What's solid (the reviews validated this). The consolidated build script's required-vs-optional gate + never-exit-non-zero invariant; the Docker prune → re-postinstall → COPY chain (builder has the toolchain, so the .node + node-gyp-build reach runtime); the publish coverage-guard correctly blocking a lean tarball while prebuilds are incomplete; the cost-gating (an ordinary PR triggers zero native matrix — confirmed); least-privilege workflow perms + persist-credentials: false; N-API / ABI-14 consistency. Genuine care here.

🔴 Headline — P1 (resolve before un-drafting/merge)

Unguarded static import C from 'tree-sitter-c' crashes analyze at module-load when C has no binding. tree-sitter-c is now vendored prebuild-only and removed from dependencies, but it has 0/6 committed prebuilds (only .gitkeep). On a toolchain-less host (source-build fails) or under npm install --ignore-scripts (materialize never runs), C ends up with no binding. Three sites import it with a hard static ESM import that runs before parser-loader's optional/degradation machinery — parse-worker.ts:7, core/ingestion/languages/c/query.ts:2 (eager on the main thread via languages/index.ts → c-cpp.ts → c/index.ts), core/group/extractors/include-extractor.ts:5 — so the import throws ERR_MODULE_NOT_FOUND at module-load and crashes the whole run, instead of the clean "C disabled, other languages fine" the parser-loader entry intends. This is the exact bug class #2091/#2093 fixed for Swift/Dart/Kotlin via the lazy getLanguageGrammar pattern; C was left static because it used to be an always-present npm dep.

• Mechanism: adversarial (Claude) lane + coordinator code-read + the #2091/#2093 precedent. Codex independently flagged the --ignore-scripts failure path (its F2), corroborating the install-failure trigger — but it framed the result as "parser loader fails," not the static-import module-load crash, so the crash mechanism itself is single-Claude-lane + coordinator. [code-read]
• Fix: commit C's six prebuilds before un-drafting, or convert the three C imports to the lazy/guarded pattern Swift/Dart/Kotlin already use.

🔴 P0 — supply chain (pre-merge)

attest-build-provenance is pinned to bd77c077… commented # v2.4.0 # PLACEHOLDER-PIN, but real v2.4.0 = e8998f94… (coordinator-verified, GitHub API) — an untagged mid-stream commit, so the SLSA-attestation step runs unvetted action code and the comment misrepresents what runs. The PR's own # PLACEHOLDER-PIN — verify before merge marker flags it as unfinished; the specific finding is that the SHA is wrong, not merely unverified. The sibling setup-python@a26af69b… # v5.6.0 PLACEHOLDER-PIN is, by contrast, a correct pin (coordinator-verified) — only its comment is stale. (Inline on the workflow line.)

Other findings (body)

• P1 (DevOps) — aggregate hard-fails with no graceful skip if RELEASE_APP_ID/RELEASE_APP_PRIVATE_KEY are unset; the "open a PR with prebuilds" feature can't run until the App is provisioned, after a full 6-runner build spends its budget. Gate on secret presence + fall back to open_pr=false artifacts. [code-read]
• P2 (Codex F3 + correctness + npm-native — STRONG) — the publish guard detects source-exclusion only via the vendor/**/src/parser.c toggle; a partial .npmignore edit (e.g. excluding binding.gyp but leaving parser.c commented) passes the guard while shipping an unbuildable grammar. Validate the full source-build set, or assert against npm pack --dry-run --json.
• P2 (Codex F5 + adversarial + correctness — STRONG) — test false-confidence: gitnexus/test/unit/prebuild-coverage.test.ts's strict 6/6 assertion is dormant whenever binding.gyp exists (all grammars have it), so a missing prebuild passes CI silently; and parser-loader-abi.test only drives the lazy loader, so it's blind to the P1 crash. CI is green precisely because CI hosts have a toolchain. Add a per-grammar "fully-prebuilt" gate + a smoke that imports the real static-import surface.
• P2 (npm-native + adversarial) — "toolchain-free" is aspirational today: only Swift has 6/6 prebuilds; c/dart/proto/kotlin are source-build-only until build-tree-sitter-prebuilds.yml runs (itself blocked on the P0 SHA + new runners + RELEASE_APP secrets). This transitional state is what makes the P1 live.
• P2/P3 (DevOps) — gitnexus/package-lock.json in the workflow paths: fires the (cheap) guard on most dependency PRs; the new arm runners (esp. windows-11-arm) are unproven in this repo (failure mode is safe — fail-fast: false + the 6/6 aggregate assertion refuse a partial set); the 30-min build timeout is tight for the 23.6 MB kotlin parser.c.
• P3 (Codex F4 — verify) — aggregate's if: inputs.open_pr != false is null on pull_request events; GHA null-coercion makes the direction non-obvious (over-fire per Codex's read, or under-fire the documented version-change-PR → prebuild-PR flow). Make it explicitly event-gated.
• P3 (npm-native) — promote node-gyp-build/node-addon-api to gitnexus's own dependencies (currently safe under --omit=optional only via the required tree-sitter's transitive edge — see Refuted); add the /std:c11 /utf-8 win block to c's binding.gyp for parity (inert today — no non-ASCII bytes in any parser.c/scanner.c); AGENTS.md:177 still calls kotlin/swift "optional" (stale).
• P3 (correctness) — pre-existing materialize double-rename-failure edge could leave a grammar unmaterialized (very low probability).

✅ Validated / refuted (the reviews doing their job)

• REFUTED — Codex F1 (P1: node-gyp-build optionalDependency → --omit=optional breaks C). The required tree-sitter@0.21.1 declares node-addon-api + node-gyp-build as regular dependencies (lockfile verified), as do 12 other regular tree-sitter-* deps — so node-gyp-build survives --omit=optional and the npm-11 arborist prune, and every vendored grammar's require("node-gyp-build") resolves. The independent engine's plausible P1 is a false positive; two Claude lanes + the coordinator's lockfile check refute it. (The P3 hardening note above is the residual.)
• Docker image, the publish guard's "can't ship a fully-dead grammar," and GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 were all probed and found safe.

CI: ABI gates ×3, packaged-install smoke (ubuntu+windows), lint/format/typecheck/actionlint/zizmor/CodeQL/gitleaks/Trivy all green; build-matrix + "Vendor prebuilds" correctly skip (no version change); a few pending; Vercel = deploy-auth (ignore).

Coverage: read the substantive diff (scripts, parser-loader, workflow, Dockerfile, .npmignore, package.json, vendor binding.gyp/package.json); the generated parser.c/node-types.json/binaries were not line-read.

Automated multi-tool digest (3 methods, Codex live). Verify before acting.

[magyargergo]

P1 (resolve before un-drafting) — this C degradation is bypassed by unguarded static imports. This optional / severity: error entry is meant to turn a missing tree-sitter-c binding into a clean "C disabled, other languages fine." But tree-sitter-c is now vendored prebuild-only with 0/6 committed prebuilds, and three sites import it with a hard static ESM import that runs before this loader and never reaches it: parse-worker.ts:7, core/ingestion/languages/c/query.ts:2 (eager on the main thread via languages/index.ts → c-cpp.ts → c/index.ts), and core/group/extractors/include-extractor.ts:5.

On a toolchain-less host (source-build fails) or under npm install --ignore-scripts (materialize never runs), C has no binding → those imports throw ERR_MODULE_NOT_FOUND at module-load → the whole analyze crashes, never reaching this degradation. This is the exact bug class #2091/#2093 fixed for Swift/Dart/Kotlin via the lazy getLanguageGrammar pattern; C was left static because it used to be an always-present npm dep.

Fix: commit C's six prebuilds before un-drafting, OR convert the three C imports to the lazy/guarded pattern. (Anchored here on the bypassed degradation handler — the actual crash sites are the three import lines above. Adversarial lane traced the mechanism; Codex independently flagged the --ignore-scripts path; verified by code-read + the #2091/#2093 precedent.) [code-read]