perf: add deterministic synthetic benchmark corpus

[seonghobae]

Scope and ownership

Refs #375. This branch is the established single-writer implementation lane for Inkspan's deterministic large-document performance evidence. Protected main remains the only shipped implementation authority. No SHA, merge-base, check result, review count, or workflow state recorded in this PR body is lifecycle authority; refetch those values at decision time.

The lane remains standalone, deterministic, provider-neutral, and network/model independent. It contains deterministic synthetic Markdown/Office fixtures, strict summary/comparison primitives, bounded Markdown and revision-evidence measurement harnesses, retained-memory settling analysis, demo-vendor chunking, an Office render duration/peak-RSS measurement primitive bound to the canonical synthetic fixture lock, and a one-command suite that composes Markdown + revision-evidence measurements. Packed mode measures exact modules extracted from one verified npm artifact and records package/runtime/source/reference-hardware provenance.

This work does not by itself establish production performance budgets, a supported large-document envelope, complete packed-editor/browser/IME/Yjs/Office matrix acquisition, leak freedom, or release acceptance.

Executable contract

Benchmark producers and fixture generators fail closed on unsafe symlink/non-directory ancestors, unsafe leaf targets, pre-existing hard-linked outputs, bounded-input violations, and unverifiable package/module provenance. Packed mode requires the exact package tarball digest and package identity, the active Node runtime identity, the source checkout identity, and a reference-hardware identifier.

benchmarks/run-current-suite.mjs additionally fails closed unless the benchmark checkout cleanliness guard succeeds before delegating to benchmarks/run-current-suite-core.mjs. This closes the false-provenance class where modified or untracked source could otherwise produce evidence labeled with an unchanged HEAD SHA. The failure is bounded and does not disclose dirty file paths.

office/benchmarks/measure_render.py accepts only a committed synthetic Office fixture whose exact byte count and SHA-256 match benchmarks/office-fixtures.lock.json. It requires a clean checkout and verified source revision, rejects unbounded iteration counts before inspecting caller-selected input, uses a fresh Python child process for each render sample, and records render duration plus process peak RSS with p50/p75/p95/max summaries and runtime/reference-hardware provenance. Ordinary evidence contains fixture identity/hash/size and measurements, never the document body or caller path. It performs no network, credential, service, database, or model operation.

Failure-contract / TDD lineage

A direct reproduction against the predecessor implementation established that git rev-parse HEAD alone cannot distinguish clean source from tracked or untracked worktree mutations. The narrow repair added a clean-checkout guard plus isolated temporary-repository contract tests for clean acceptance and dirty-state rejection; packed-suite tests exercise the delegated clean path. The test fixture is isolated from the repository checkout so parallel benchmark tests are not contaminated by a temporary dirty worktree.

The Office measurement contract was added test-first on the canonical performance branch: it requires lock-bound synthetic input, stable privacy-safe rejection of arbitrary/private content, bounded iteration work, isolated repeated samples, duration and peak-RSS evidence, and source/runtime/reference-hardware provenance. The immediately superseded test-only generation was cancelled before terminal hosted RED evidence, so it is lineage rather than passing evidence; current-head verification must be read live and predecessor/cancelled runs never transfer.

Earlier RED/GREEN iterations established output-symlink, hard-link, measurement privacy/resource, suite ordering, failure-atomicity, packed-artifact identity, runtime identity, source-SHA, summary/comparison, and retained-memory-analysis contracts. Historical workflow results document lineage only; they never transfer to a later head or base.

Remaining #375 acceptance work

This PR does not close #375 until protected evidence covers the applicable acceptance boundary. Remaining product work includes:

• repeated acquisition from the actual packed editor/runtime on accepted reference hardware and the committed deterministic corpus;
• trusted interaction, IME, mount/hydration, serialization, revision, autosave, Yjs, and print measurements at the source owner that can produce them without crossing Inkspan's host boundary;
• full supported Office fixture/profile acquisition across DOCX/XLSX/PPTX runtimes, publication-path measurement where applicable, and evidence-derived conversion budgets; the current Office primitive is measurement infrastructure, not a published support envelope;
• an accepted repeated-run retained-memory/leak/settling acquisition protocol rather than analysis of synthetic samples alone;
• support budgets derived from measured evidence rather than raised after a regression;
• accepted support-envelope architecture/operability/README claims;
• appropriately owned bounded PR smoke evidence plus scheduled/manual full evidence; and
• stable over-limit behavior wherever Inkspan owns the bounded API.

Browser/IME work, reference-host integration, release workflow, and organization-required workflow behavior remain with their established owners. Do not create competing source writers merely to make this performance lane appear complete.

Decision-time acceptance rule

Before any readiness, merge, release, closure, ownership, or support-envelope decision, independently refetch and reconcile at least:

• protected main, this PR's exact head, its independently resolved live base, ancestry/divergence, mergeability, and changed paths;
• all open Inkspan PRs/issues that can own or conflict with the affected source boundary;
• formal reviews, last-push approval requirements, and unresolved review threads;
• repository and organization-required workflow runs plus individual jobs and their actual checkout/source SHAs;
• live rulesets/branch governance and required-review policy;
• applicable package, browser, Office, coverage, security, SBOM, provenance, reproducibility, release, rollback, and operability evidence; and
• releases/tags/registry state when making a publication or buyer-facing support claim.

Pending, queued, in-progress, skipped-required, cancelled, absent, neutral, failed, stale, predecessor, wrong-checkout, synthetic-source-only, status-only, model-only, or vacuous evidence is non-passing. Automated comments/reviews are technical input, not qualifying independent approval. Any material head/base/ruleset movement invalidates the corresponding decision evidence.

Do not self-approve, weaken gates, transfer predecessor evidence, fabricate release identity, create a competing CI/security writer, or represent branch behavior as protected-main shipped truth.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Local exact-head validation for 30edf11 (macOS, Node v24.16.0, clean detached worktree with a real frozen pnpm install):\n\n- The 39-file performance contract selection reached 97/99 tests. The initial default os.tmpdir run was an environment artifact: this host exposes /tmp and /var through symlinks, while the ancestry checks intentionally fail closed. Re-running with canonical TMPDIR=/private/tmp removed those 29 symlink-ancestor failures.\n- One remaining failure is the oversized-input contract preload: its Node --import fixture patches fs.readFileSync before Node 24 loads the target module, so the ESM loader itself raises the sentinel before the benchmark can inspect the file size.\n- The packed-artifact suite passed when the Vitest test timeout was raised to 20s; its default 5s timeout expired locally.\n\nNo source was changed and this is validation only, not approval or merge evidence. The PR remains Draft.

[seonghobae]

Pushed test-only portability fixes at exact HEAD d025d45:\n\n- The whole-file-read sentinel now blocks only the sample input path, allowing the Node 24 ESM loader to load the benchmark entrypoint while still proving the benchmark does not read that input with fs.readFileSync.\n- The four locally slow benchmark-process contracts now use a 20s Vitest test timeout, consistent with their existing child-process budgets.\n- In a clean detached worktree with a real frozen pnpm install and TMPDIR=/private/tmp, all 39 performance contract files passed: 99/99 tests.\n\nThe PR remains Draft; this is validation/test maintenance only, not approval or merge evidence.

[seonghobae]

Post-push exact-head check update for d025d45: local performance contracts remain 39/39 files and 99/99 tests passed. Hosted checks currently show 21 success, 7 skipped, 1 in progress (Strix), and 1 failure (opencode-review). The OpenCode failure points to the deleted workflow id 309142954; its run/job log lookup returns GitHub API 404, so it produced no review verdict or code finding. Strix is still pending. No approval or merge evidence is implied.

[seonghobae]

Terminal Strix evidence for exact HEAD d025d45: the job failed closed after three bounded attempts because the provider returned OpenAI API 413 request_too_large for orchestrator/free. The log explicitly reports STRIX_PROVIDER_UNAVAILABLE and no authoritative vulnerability finding; the displayed vulnerability count was 0 but the gate did not pass. This is provider/backend evidence, not a code approval.