Token evidence: RTK coverage study and 18 token-tool currency receipts (sanitized, revision-bound)

[seathatflowsinourveins]

Summary

Publish the two scratch-only PR-EV results so their aggregate numbers, source
references and release observations can be inspected from the repository:
the historical RTK coverage study and all 18 token-tool currency records.

Changes

• Add evidence/artifacts/rtk-coverage-study-20260927/ with the aggregate tables,
  sanitized exactness.out, method and a records table. Preserve numeric
  outcomes except for the removed unsupported-key list, including its counts;
  remove the local recall handle. This repair leaves all three data files unchanged.
• Add evidence/artifacts/token-tool-currency-20260927/ with 18 per-tool JSON
  records, a method and a records table. Retain reviewed pins, scratch claims,
  actual allowlisted live release metadata, channel-specific behind_by values,
  primary-source findings and explicit retrieval dates.
• Add a dated interpretation erratum: discover's 62.4% whole-call-derived
  classification is not the later M-R1 eligible-part gate.
• Record Headroom's newly published v0.39.1, RTK's newer prerelease, changed
  unreleased-source distances and the context-hub Secret-storage practice: per-host 0600 store, agent read guard hook + deny rules, managed-profile install, names-only checker #182 closed/unmerged correction.
• Add tests/test_token_full_save_evidence.py for completeness, source/date
  fields, preserved RTK outcomes, evidence boundaries and identifier patterns.
• Correct T7 attribution to rc.467, whose binary reports 0.49.0; correct the
  cargo-test boundary and remove the unretained installed-version/help claim.
• Recompute every metadata file:line against repository revision
  1c32ad2; add component/version and revision
  controls for all 18 records, including secondary pins.
• Bound query timestamps to latest-release batch starts, mark Serena's main
  distance not re-verified, and label publication checks structural validation.
• Exercise the shared identifier-pattern assertion against six planted
  synthetic samples; retain failure and passing observations separately.

Evidence (with classes)

• Historical local measurement: the supplied 59,041-call RTK study and
  611-observation aggregate classification; no new private population replay.
• Historical local integration / synthetic fixtures: retained exactness
  output; not a new RTK run and not unchanged upstream tests.
• Upstream source/documentation review: live GitHub latest-release checks
  for all 18 tools on 2026-09-27, selected package-channel checks, tagged/commit
  source review and finding-specific primary URLs.
• Structural validation / publication controls: the repair's first run
  failed before receipt corrections (10 tests, 56 subtest failures, exit 1).
  The fresh passing run returned 10 tests OK, exit 0. An intermediate run
  failed twice because the new control used candidate names instead of
  landscape winners' component_id; reading the original entries corrected it.
  These are artifact consistency checks, not native tool execution or adoption.
  The build's older missing-artifact failure had no retained stdout and did not
  demonstrate identifier detection.
• Synthetic discriminating controls: the same assertion used to scan
  publication artifacts rejected each planted email, UUID, personal path,
  bearer, credential-shaped value and session identifier. The unwrapped check
  returned exit 1 with six failures; the permanent unittest requires each
  rejection. No sample values are published in these receipts or logs.
• Independent artifact observation: all three RTK data files equal the
  committed publication bytes; tables.txt still has the original aggregate
  SHA256. All 18 scratch records, pin versions and latest-release snapshots are
  unchanged. Eight metadata files were read from the named repository revision
  and byte-compared with the checkout. rtk git diff --check returned 0.
• Native local leak scans: fresh Gitleaks 8.30.1 results for both artifact
  directories and the unittest module are retained below. The shared pattern
  scan checks all 25 public artifacts. Neither scan establishes detection of
  every sensitive string. No credential stores were read.
• Not performed: live provider execution, new native qualification,
  installations/upgrades, upstream test reruns or fresh tokenizer parity trials.

Retained repair outputs, 2026-09-27: the covering command is
rtk python3 -m unittest tests.test_token_full_save_evidence, preceded by
export TMPDIR=/var/tmp/claude-evidence, at the worktree root. Start
2026-09-27T14:12:05.239823+00:00, end 14:12:05.437827+00:00; exit 0,
stdout empty, stderr:

..........
----------------------------------------------------------------------
Ran 10 tests in 0.036s

OK

The red publication run returned Ran 10 tests in 0.039s and
FAILED (failures=56). The separate planted-control call used rtk python3 -
with the same TMPDIR and shared assertion; exit 1, stdout empty, stderr
excerpts (synthetic fixture, expected rejection):

AssertionError: forbidden identifier pattern: email
AssertionError: forbidden identifier pattern: uuid
AssertionError: forbidden identifier pattern: personal path
AssertionError: forbidden identifier pattern: bearer value
AssertionError: forbidden identifier pattern: credential-shaped value
AssertionError: forbidden identifier pattern: account or connection value
Ran 1 test in 0.003s
FAILED (failures=6)

Acceptance of the final branch (local integration, this host, 2026-09-27):

• Before the repair. The build's failing-module rerun (only the modules that failed in the contended harness suite, with TMPDIR outside /tmp) reported Ran 854 tests and OK (skipped=73).
• At the repair head. validate.sh reported FAILS=0. The full suite ran 6603 tests with failures=1: tests.test_secret_path_guard.test_host_profile_copy_is_verbatim, which fails on branches older than main d022295 because this host installed Claude harness settings: credential-store and destructive-git denies, Opus role agents without frontmatter isolation, rtk-aware guard #402's guard at 14:04Z.
• After an independent repair verification. The coordinator made test_pin_metadata_lines_contain_the_component_version read each cited file at the record's metadata_revision (git show <revision>:<path>), so later line moves on main cannot break it. It also reworded the two METHOD.md passages the verification found inaccurate.
• Registration. The branch's last commit registers this branch's own files in manifests/evidence.json after the replay onto main (docs/lanes.md hot-file protocol).
• Replay onto main bfdc99c (#381 PR-A: token-adoption measurement kernel (M3/M4/M5, RTK eligibility, MCP attempt states) #432 merged) as 3fda98b: content lines identical, 26 files re-registered, validate.sh FAILS=0; tests.test_token_full_save_evidence ran 10 tests, OK.

Fresh Gitleaks returned output: installed rtk gitleaks version returned
8.30.1. For each target below, the command at the worktree root was
rtk gitleaks dir TARGET --config .gitleaks.toml --redact --no-banner --no-color.
Every command returned exit 0 and empty stdout; returned stderr follows.
Only terminal color escapes were stripped; times are the client's local times.

Target	Start / end (UTC, 2026-09-27)	Exit
evidence/artifacts/rtk-coverage-study-20260927	14:16:21.763747 / 14:16:22.366550	0
evidence/artifacts/token-tool-currency-20260927	14:16:22.366700 / 14:16:22.925858	0
tests/test_token_full_save_evidence.py	14:16:22.925929 / 14:16:23.493198	0

10:16AM INF scanned ~28373 bytes (28.37 KB) in 6.9ms
10:16AM INF no leaks found

10:16AM INF scanned ~98434 bytes (98.43 KB) in 18.8ms
10:16AM INF no leaks found

10:16AM INF scanned ~10403 bytes (10.40 KB) in 18.2ms
10:16AM INF no leaks found

SOTA sources

The source selection reuses maintained native release APIs and upstream
implementations; no new runtime mechanism is introduced.

• rtk-ai/rtk v0.50.0, commit
  1d87b8e719ce0a50c223cd93ca64dd16921f9aec: src/main.rs,
  src/hooks/decision.rs, src/discover/mod.rs, src/discover/registry.rs.
• rtk-ai/rtk dev-0.51.0-rc.467,
  commit a89a31494670fcec8ffa20d939dd94c64bd998fb: historical fixture comparison;
  Cargo.toml:3
  confirms the 0.49.0 binary label. The supplied RTK REPORT.md:37-41,193,205,270
  establishes the tested revisions, jq column attribution and cargo-test omission.
• kenn-io/agentsview v0.44.0; reviewed pin 0.43.0. Finding-specific source/PR/commit URLs are in records/agentsview.json.
• akitaonrails/ai-memory v2.4.1; reviewed pin 2.4.1. Finding-specific source/PR/commit URLs are in records/ai-memory.json.
• ast-grep/ast-grep 0.45.3; reviewed pin 0.45.3. Finding-specific source/PR/commit URLs are in records/ast-grep.json.
• ccusage/ccusage v20.0.24; reviewed pin 20.0.24 at ecb676cce27cb5dd0090c7804a5cecc35e8ba805. Finding-specific source/PR/commit URLs are in records/ccusage.json.
• DeusData/codebase-memory-mcp v0.11.0; reviewed pin 0.11.0. Finding-specific source/PR/commit URLs are in records/codebase-memory-mcp.json.
• andrewyng/context-hub v0.1.4; reviewed pin 0.1.4. Finding-specific source/PR/commit URLs are in records/context-hub.json.
• mksglu/context-mode v1.0.169; reviewed pin 1.0.169. Finding-specific source/PR/commit URLs are in records/context-mode.json.
• niieani/gpt-tokenizer 4.0.0; reviewed pin 3.4.0. Finding-specific source/PR/commit URLs are in records/gpt-tokenizer.json.
• headroomlabs-ai/headroom v0.39.1; reviewed pin 0.37.0. Finding-specific source/PR/commit URLs are in records/headroom.json.
• jgravelle/jcodemunch-mcp v1.108.319; reviewed pin 1.108.319. Finding-specific source/PR/commit URLs are in records/jcodemunch-mcp.json.
• microsoft/markitdown v0.1.8; reviewed pin 0.1.8. Finding-specific source/PR/commit URLs are in records/markitdown.json.
• openclaw/mcporter v0.14.1; reviewed pin 0.14.1. Finding-specific source/PR/commit URLs are in records/mcporter.json.
• tobi/qmd v2.8.3; reviewed pin 2.8.3. Finding-specific source/PR/commit URLs are in records/qmd.json.
• yamadashy/repomix v1.18.1; reviewed pin 1.18.1 at 80b4280a9196feace092fc672dfe2b5fac62ef08. Finding-specific source/PR/commit URLs are in records/repomix.json.
• rtk-ai/rtk v0.50.0; reviewed pin 0.50.0. Finding-specific source/PR/commit URLs are in records/rtk.json.
• oraios/serena v1.7.0; reviewed pin 2.0.0.dev0 at c6fbd1c5932df2494ffa0020af5a9fbe80b82143. Finding-specific source/PR/commit URLs are in records/serena.json.
• giancarloerra/SocratiCode v1.15.0; reviewed pin 1.14.0. Finding-specific source/PR/commit URLs are in records/socraticode.json.
• toon-format/toon v4.1.1; reviewed pin 4.1.1. Finding-specific source/PR/commit URLs are in records/toon.json.
• GitHub REST releases
  and gh api: supported metadata retrieval.
• Python unittest and the
  existing tests/test_token_e2e_preregistration.py artifact-contract pattern.
• Gitleaks v8.30.1 README:
  supported directory/file scan.
• docs/acceptance-evidence-policy.md and
  evidence/artifacts/rtk-exclude-widen-20260926/README.md: evidence boundaries and
  records-table style. The full-save plan sections 2, 3.1 and 4.1 define scope.
• Repository commit 1c32ad2: the metadata
  sources named in every record, including manifests/stack.json, platform pin
  files, the landscape winners and tokenizer sources. No pinned version changed.

Review dispositions

All nine findings are accepted and repaired; none is disputed.

1. T7 names v0.50.0 and rc.467 in METHOD.md, its table and the README row.
   The dated erratum cites pinned Cargo.toml and preserves exactness.out bytes.
2. All metadata references were recomputed, include a repository revision,
   and are opened by a unittest that checks component and version, including
   secondary pins. Scratch records retain their original values.
3. Evidence states the final branch's acceptance and registration (above). The
   numeric claim now excludes the removed unsupported-key list and its counts.
4. METHOD.md, README and this PR body classify offline checks as structural
   validation, which proves artifact consistency only.
5. Six planted samples exercise the same assertion and pattern table used by
   the publication scan; their failing observation is retained beside the green run.
6. retrieved_at is the latest-release query's batch start; package/compare
   observations have only a retrieval date. The prior ENOBUFS rerun is disclosed.
7. Serena's main distance is not re-verified. Its historical 30 remains in
   scratch_record; fixed-SHA comparison URLs remain source evidence only.
8. Cargo tests were not run in the original study or by the publisher.
9. The unretained installed-version/help claim was removed; the Gitleaks
   statement points to this PR Evidence section instead of private unit notes.

Review round (2026-09-29, native-agent-stack-10)

Head reviewed: e3678505 (main merged at ba31dcd0), by two model families, read-only, on frozen diffs.

Reviewer	Result	Findings
GPT-6 (gpt-6-astra, effort max, codex exec -s read-only; the model is the -m request, not an observed resolution)	needs_changes	1 major, 2 minor
Claude evidence-reviewer (Opus, max)	needs_changes	2 minor, 3 nit

Repairs, each checked by an independent Opus stack-verifier that re-ran the acceptance commands and proved red-first on the previous source:

• Provenance test no longer skips on a bad path (major). tests/test_token_full_save_evidence.py had turned every failed git show into a skip, so a nonexistent metadata path or revision bypassed validation. A module-level metadata_lines now fails when the revision exists but the path does not, and fails in CI (GITHUB_ACTIONS=true) when the revision is missing (validate.yml checks out full history); it skips only outside CI. Two control tests fail against the old logic. Result: 12 tests OK, 0 skips, with the variable unset and set to true (the second run simulates only the variable, not a CI run).
• Population figures attributed. rtk-coverage-study-20260927/METHOD.md now says the unpublished original report states the 1,945 / 1,606 / 1,594 / 53,895 figures and the window bounds; only the 59,041-call total is also retained in the published aggregates.
• Bucket sum stated. The 22 published bucket rows sum to 25,947 covered parts against 25,954 on the TOTAL line (missed column reconciles at 15,623); METHOD.md records both sums and that the published files do not attribute the 7-part difference. The data file is unchanged.
• Main distances bounded. token-tool-currency-20260927/METHOD.md names the five records (context-mode, ccusage, qmd, repomix, toon) whose 2026-09-27 figures have no retained main observation, and the README no longer says the records retain the observed heads for them. Only jcodemunch-mcp, codebase-memory-mcp and agentsview keep an unreleased_main head. No number changed.

Final head 39e39bcb is 0f0a67fa plus a merge of main b1be50c8 by the hot-file protocol (26 files re-registered). Measured 2026-09-29 on it: python3 -B -m unittest tests.test_token_full_save_evidence 12 tests OK; the three registry tests OK; scripts/validate.py, scripts/evidence_manifest.py --check and scripts/component_matrix.py --check exit 0 (run by the merge script).

Evidence class of the repair: local integration checks measured on 2026-09-29 (unittest, validate.py, evidence_manifest.py --check, component_matrix.py --check, the three registry tests with zizmor on PATH). They are not upstream tests. The RTK data files (tables.txt, frame-summary.txt, exactness.out) and all 18 records/*.json are byte-identical to the reviewed head.

Residuals and not done

• The currency records' metadata_revision is 1c32ad2. Their line coordinates are valid at that revision
  and may differ on main after this lands; the unittest reads the recorded revision.
• retrieved_at provenance differs by batch (METHOD.md): batch a shares batch stamps; batch b's per-record
  stamps are retained, but how they were produced is not.
• The private sample, host-specific fixture script and private transcripts are
  intentionally unpublished; full historical population regeneration is not
  possible from aggregate receipts alone.
• Scratch measurements and historical full-suite counts are not new acceptance.
  Serena's and MarkItDown's historical main-distance claims are not re-verified.
  The command producing the T7 appendix was not retained.
• No new latest-release or compare observations were made in this repair;
  all retained publication snapshots remain dated 2026-09-27.
• No repository pins or runtime configuration changed. manifests/evidence.json
  changes only by re-registration of this branch's files in the last commit,
  which keeps lane:foundation under docs/lanes.md:145-148.
• Not fixed in this round: 16 change entries in nine records carry no date key; the test's IDENTIFIER_PATTERNS are not a subset of scripts/validate.py PRIVATE_CONTENT (each has patterns the other lacks); the transcript counts have no retained output beyond the attribution wording; rtk-coverage-study-20260927/README.md L6 repeats the measurement window without the attribution; a revision that exists but is not a commit object is treated as absent by metadata_lines.

Recommended lane label: lane:foundation.

🤖 Generated with Claude Code

[seathatflowsinourveins]

Review record, 2026-09-29 (native-agent-stack-10). Merge candidate: head 39e39bcb, which is e3678505 plus the repairs below and merges of main (latest b1be50c8).

• Two model families reviewed e3678505, read-only: GPT-6 (gpt-6-astra, effort max, codex exec -s read-only; the model is the -m request, not an observed resolution) returned needs_changes (1 major, 2 minor). A Claude evidence-reviewer (Opus, max) returned needs_changes (2 minor, 3 nit). Nothing was disputed.
• Repair: an Opus isolated-builder fixed the major finding (a provenance test that skipped on any failed git show) and the attribution, bucket-sum and main-distance wording, with red-first evidence. An independent Opus stack-verifier re-ran every acceptance command and the negative controls and returned ready_to_push with three minors. Its README/METHOD wording finding was then fixed by the coordinator and checked against the records by script, not by a second agent.
• Checks on the merge candidate: python3 -m unittest tests.test_token_full_save_evidence 12 tests OK, 0 skips; scripts/validate.py, scripts/evidence_manifest.py --check and scripts/component_matrix.py --check exit 0; the three registry tests pass with zizmor on PATH. These are local integration checks, not upstream tests. The published RTK data files and all 18 records/*.json are byte-identical to the reviewed head. Hosted required checks are read with gh pr checks 409 --required immediately before merging.
• Residuals, listed in the body: 16 change entries without a date key; the test's identifier patterns are not a subset of scripts/validate.py PRIVATE_CONTENT; transcript counts have no retained output beyond the attribution wording.