Repository navigation
Gate A U8: freeze snapshot and drift check for the #381 window W - #502
Merged
seathatflowsinourveins merged 16 commits intoSep 29, 2026
Merged
Conversation
tests/test_freeze_snapshot.py specifies the U8 tool for the #381 window W (README "Procedure" step 2, RUNBOOK "Freeze and preflight"): capture writes a private full record and a sanitized one, compare reports drift among frozen items only, check compares a capture with sealed expectations, list-frozen prints the frozen set. The suite runs against a synthetic host (temporary checkout, home, fake claude/codex/rtk/node/python3/qmd/systemctl on a private PATH, canned repo scripts, a loopback HTTP server). Shapes of the canned output were read from the real commands on 2026-09-29: claude 2.1.284 mcp list and -p json, qmd 2.8.3 status, systemd 255 systemctl --user show, adoption_status.py --client-wiring and --pinned-versions, codex_quota.py --json. MutationControlTests writes a mutant of the tool for each property (environment leak, a collector blind to each item class, informational drift that fails, check that always passes, credential refusal off, a raising missing-tool path) and requires the test for that property to fail on it. tools/token-e2e/freeze_snapshot.py is only the argument surface here and writes empty item lists, so the tests fail on their assertions and not on an import error. The implementation follows in the next commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tools/token-e2e/freeze_snapshot.py is the U8 glue for the #381 window W (README "Procedure" step 2, RUNBOOK "Freeze and preflight"): a read-only, standard-library-only tool that captures the frozen environment as items with an id, class, status, value and method, compares two captures, checks one against sealed expectations and lists the frozen set. No upstream implementation of this glue exists. Rules and their sources: `systemctl --user show UNIT -p PROPERTY` for unit state (systemctl(1), systemd 255); `git --no-optional-locks status --porcelain --untracked-files=no` for a tree-clean read that writes nothing (git(1), git-status(1)); the command outputs observed on 2026-09-29 for claude 2.1.284 mcp list and -p json, qmd 2.8.3 status, adoption_status.py and codex_quota.py. The Claude usage probe adds --verbose (array output whatever the user settings say), --no-session-persistence and --setting-sources project so the one Haiku call runs no user hook or MCP server; the brief's command reads the same rate_limit_event. Privacy: collectors derive only hashes, versions, counts, booleans and model aliases, the private capture stores paths as ~/ or <repo>/ forms, and a final guard refuses (exit 3, nothing written) any string that carries an environment value of eight characters or more, the home or checkout path, or the user or host name. The credential stores are refused by name, symlink and inode. All scanners are linear character scans. The tests are corrected where the failing-first run showed a wrong assumption: the schema id carries no slash (the sanitized capture holds no path character), nested ids use a colon, and the probe flags, binary hash and tokenizer override are asserted. MutationControlTests now compiles each mutant before running it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review of the first implementation found and fixed, each with a test: - a refusal message named the offending item id, and a host-derived family id such as hooks.installed.<name> can carry the user name: a family member is now named by its family, and the message names the environment variables involved (names, never values) so a refusal can be acted on; - the guard would have refused every capture on a host whose user, host or a variable equals a word of the tool's own catalogue (a user called claude): catalogue and structure words are not listed as forbidden; - a configured path that is a FIFO would block the read: only regular files are read (error reason not_regular_file or is_directory); - a parser pin with an unusable file key was read partly: it is now unrecognized_pin as a whole; the pin's default directory is home-relative; - unit-derived config paths must be absolute (a relative one has an unknown base); a configured unit may be informational (paper-trading units run inside a run window) while the nine defaults stay frozen. tools/token-e2e/README.md gives usage, the catalogue table, how the seal, W-open, W-start and W-end captures are used, the configuration, the privacy rules and the limits, with sources (#381 README "Procedure", RUNBOOK freeze rules, systemctl(1) show, git(1) --no-optional-locks, git-status(1) --porcelain). A test matches the table rows against list-frozen in both directions; removing a row or naming a stale id makes it fail. The suite is 99 tests, green on Python 3.13.15 and 3.12.3. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Independent review of the completed unit found four gaps, fixed here: - `list-frozen` without --repo skipped the roots check, so a configuration path such as /etc/hosts was accepted. The brief says never accept a path outside the checkout or the home directory: with no checkout the home directory is the only root and a relative path may not climb out (check_path_policy). A test covers the refusals and the accepted paths. - The brief's literal negative control, a tool variant that prints os.environ, was missing: my leak mutants put the environment into the record, which the guard catches, while nothing guards stdout. A print_environment mutant now fails the privacy test on its stdout scan. - `MutationControlTests` runs its 17 mutants in a thread pool (about 10 s instead of about 55 s) and records the assertion lines from stderr. - A test now covers the Connected-prefix rule of `claude mcp list` statuses. tools/token-e2e/README.md records the residual limits: a connected flag of `claude mcp list` can flip without a file changing, ai-memory and cognee-live are not user units on the workstation host, a build id kept only in a unit's environment is not captured, and the parser items are missing until U1's pin. Against the earlier copies of the tool the current suite fails, which shows what these commits fixed: the first implementation (fe790450) fails 9 tests (a FIFO capture hangs until the 60 s timeout, a refusal that names the user in a host-derived id, a host user called claude refusing every capture, the parser pin read partly, informational units, and others); the hardening commit fails only the path-policy test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
scripts/validate.py rejects a registered file that contains a personal home path (/home/<name> or /Users/<name>, PRIVATE_CONTENT "personal home path"). Two fixtures of tests/test_freeze_snapshot.py used /home/x as an example of non-plain text; they now use /opt/x, which is equally a path and is refused by the same is_plain rule. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The last assertion of FixtureSanityTests.test_fake_commands_print_the_canned_shapes compared a value with a constant that its own else branch produced, so it could never fail. It now asserts that the fixture PATH holds only the fake bin directory and the symlinked utilities, and that claude, codex, rtk, node, python3, qmd, systemctl and git each resolve to the fake: a real command reachable from the fixture would make the whole suite talk to the host. With a real directory appended to the PATH the test fails. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The brief asks for a source, document or section for every rule the tool implements. The README's Sources section now covers each of them: the #381 README "Procedure" and RUNBOOK freeze rules, systemctl(1) show (systemd 255) and the configuration flag each default unit's binary lists in its --help, git(1) --no-optional-locks with git-status(1) --porcelain, the Claude Code CLI reference for the probe flags (--no-session-persistence, --setting-sources, --output-format, --verbose, --model), proc_meminfo(5) MemAvailable, and the output shapes read on 2026-09-29 and kept as test fixtures. It also records the check for an existing tool (osquery, AIDE, Ansible and InSpec are maintained but do not fit: osquery's systemd_units table has no MainPID or NRestarts column) and two limits: `claude mcp list` has no documented machine-readable form, and unifiedWindows in the rate_limit_event is observed on claude 2.1.284 while the Agent SDK reference documents RateLimitInfo without it, so a client update may turn the four Claude capacity items into `missing`. The Tests paragraph now lists the print-environment mutant. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Coordinator delta to the U8 brief, item 3: the behaviour-affecting keys of the user and project settings files join the frozen set. The sealed `claude -p` arms launch with no permission flags, so these settings define every arm's permissions and whether a peer session can message a run's lead. Keys and value sets follow the settings reference fetched 2026-09-29 (https://code.claude.com/docs/en/settings-reference): permissions.defaultMode (default, acceptEdits, plan, auto, dontAsk, bypassPermissions, manual), permissions.allow|deny|ask (counted, never printed), skipDangerousModePermission Prompt, crossSessionInbound (accept, hold, refuse), autoContinueAtUsageLimit, autoUpdatesChannel (latest, stable), ultracode, enableWorkflows, workflowSizeGuideline (unrestricted, small, medium, large), switchModelsOnFlag, model and effortLevel; advisorModel is already an item. These tests are committed before the change and fail against the tool as it stands: one row per key in each of the three settings files (catalogue and values, with values that differ between the user and project fixture), absent and undocumented values as classes and never as values, rules counted and never printed, drift on every key, `check` failing per item, and an environment value equal to a printed setting value not refusing the capture. Mutation controls cover permissions.defaultMode and crossSessionInbound (a collector blind to the key) and permission rules printed with and without the guard; the new rows also name the text their failing assertion must contain. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Coordinator delta to the U8 brief, item 3. Source: the settings reference, fetched 2026-09-29 (https://code.claude.com/docs/en/settings-reference), for the keys and the closed value sets. Each settings file now yields 14 more frozen items, claude.settings.<file>.{permissions_default_mode, permissions_allow_count, permissions_deny_count, permissions_ask_count, skip_dangerous_mode_permission_prompt, cross_session_inbound, auto_continue_at_usage_limit, auto_updates_channel, ultracode, enable_workflows, workflow_size_guideline, switch_models_on_flag, model, effort_level}: a documented value, a boolean, a count of rules or a model alias. An absent key is `unset`, a value outside its set or of another type is `other`, and a rule list that is not a list is an error. The files are the same three the existing derived items read; the brief names user and project, and local costs nothing and closes the same gap for a checkout that has a settings.local.json. Two guard changes come with it. A permission rule can hold a secret, so the rules are read and counted but never printed, and their text is listed as forbidden like an environment value. A model alias or documented value that the tool prints from a settings file may equal an environment value (an advisor alias, ANTHROPIC_MODEL): that no longer refuses the capture, and the documented values join the catalogue text the guard already spares. The README table gains the row; its coverage test matches it in both directions. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The real acceptance capture of this unit ran while the account's five-hour window was over its limit: `claude -p ... --output-format json --verbose` exited 1 with HTTP 429, and its rate_limit_event (status rejected, five_hour utilization 1.06, seven_day 0.53) was in the output array. The tool treated the nonzero exit as a failure and reported all four Claude capacity items as error(exit), dropping the numbers at the one moment the opening rule of a run window (both windows under 30% used) needs them. The test makes the fake claude exit with a configurable status and expects the windows to be read from a rejected call, and an error(exit) for a failed call that returned no usable event. It fails against the tool as it stands. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A `claude -p` call rejected at the session limit exits 1 with HTTP 429 and still returns its rate_limit_event (observed on claude 2.1.284 on 2026-09-29: status rejected, five_hour utilization 1.06, seven_day 0.53). The tool now reads that event from the output of a call that exited nonzero, so the four informational Claude capacity items carry the numbers when the window is over its limit, which is when the opening rule of a run window needs them. A failed call that returned no usable event stays an error, with reason exit. The items are informational, so neither outcome fails compare or check. The README row says so. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review of freeze_snapshot at 89ae6814 found two defects; these tests fail against the tool as it stands (Ran 8 tests, FAILED failures=30 errors=13). list-frozen printed the path a freeze.json extra file names, so an absolute path put the home path and user name on stdout. The new test lists five spellings in text, --all and --json form and requires that no configured text, home path, checkout path or user name reaches stdout and that the extra row carries no path character. tools.parser.* mirrors child-usage.mjs verifiedShellParser (branch claude/pra-u1d-parser-ci-2d-20260929 at 967561cc, lines 656-669) but ignored CHILD_USAGE_SHELL_PARSER and the lockfile, so it reported all_match_pin true for an install the kernel refuses. ParserPinTests give the fixture the real U1 pin shape (packages, install.lockfile) and a package-lock.json, and check the directory order, the lockfile hash and match, the refusals and the pin validation; ParserTests covers lock_matches_pin. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Repairs the two defects the review of 89ae6814 found; the failing-first tests of 01be4f65 now pass (Ran 122 tests, OK). list-frozen no longer prints the path a freeze.json extra file names: its how-to-check line says the configuration names the file and only the private capture records where it lives, so no home path, checkout path or user name reaches stdout. tools.parser.* now checks the install as child-usage.mjs verifiedShellParser does (branch claude/pra-u1d-parser-ci-2d-20260929 at 967561cc, lines 656, 661 and 663-668; identical on claude/pra-u1-cmdpos-2d-20260928 at 2bad732, lines 643-655): the directory comes from the configured parser_dir, then an absolute CHILD_USAGE_SHELL_PARSER that meets the path policy (a relative value and ~ are refused, and nothing derived from the variable is stored), then the pin's default; the pin must carry both packages with a version and an integrity; the lockfile it names (default package-lock.json) is hashed as tools.parser.file.<name> and must list both packages with the pinned version and integrity, or all_match_pin is false. Adds seven mutation controls, each failing its intended assertion: the configured path printed, the variable ignored, its location stored, a relative or tilde value accepted, the configured directory below the variable, the lockfile ignored, and the tokenizer_prefix override ignored (a test the review found had never failed against an earlier tool). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
README of tools/token-e2e after the review of 89ae6814: the catalogue row, configuration and sources for the kernel-order parser check (child-usage.mjs verifiedShellParser at 967561cc and 2bad732), the path-free list-frozen line, and the limits the review found undocumented. Corrections: the Python claim no longer says 3.9 or newer (the tests ran on 3.12.3 and 3.13.15 only) and the host-state claim covers the tool's own writes, because claude mcp list starts servers and claude -p is the client's command. The session-limit probe is described as exiting nonzero, which the retained capture (error exit) and the retained probe shape (status rejected, api_error_status 429) show; the exact code appears only in a commit message. New limits: the report freeze omits tools/token-report/returned_results.js, which token_manifest.py inlines into the rendered HTML (name it under files, as the example now does); the RUNBOOK role-body SHA256 values are the seal owner's to verify against the seal capture; the RUNBOOK names three role copies and the tool hashes four. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The README said the usage probe's flags keep the call from running the user's hooks or MCP servers or leaving a transcript. One direct run of the probe's exact argv on 2026-09-29 (claude 2.1.284) showed what is observable: the isolated call reported no MCP server against the seven that a call with the user settings reported (retained shape haiku.json), and left no project directory under the client's state. The event stream names no hook in either run, so that no user hook ran follows from --setting-sources project (Claude Code CLI reference: the list of setting sources to load) and is inferred, not observed. The text now says so. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Hot-file protocol of docs/lanes.md, applied against this branch's exact base (b528bb5) and recreated after the repair round changed the three files: manifests/evidence.json gains the entries for tools/token-e2e/freeze_snapshot.py, tools/token-e2e/README.md and tests/test_freeze_snapshot.py through host_receipts.register_file, so files[] stays sorted and its hashes are those of the committed bytes. component_matrix.py --write and new_host_grand_list.py --write changed nothing, and scripts/validate.py exits 0 (8003 hashed files, 168 receipts). This commit replaces e55db6be and 4aadc8b6, which sat before the last repair commits, so the manifest is touched in the last commit only. Not taken from origin/main: its manifest lists 75 files that do not exist at this base, so validate.py would report them missing. The protocol must be run again after the rebase onto main that precedes the merge; this commit is valid against b528bb5 only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
seathatflowsinourveins
added
the
lane:foundation
seathatflowsinourveins
deleted the
claude/pra-u8-freeze-snapshot-2d-20260929
branch
September 29, 2026 13:24
5 tasks done
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 29, 2026
…ng tests (#512) * CI: raise the validate and validate-macos job timeouts for the #381 tooling tests Measured after #502 on 8d1be28: validate took 753s of its 1200s limit (run 36572588202) and validate-macos 1148s of its 1800s limit (run 36572588263). The token-E2E measurement tooling that is still to merge (units U1 to U5, U9, U10, U13) adds an estimated 600 to 900s of unit tests on Linux and more on macOS, so the limits become 40 and 60 minutes; they stay hang guards. Precedent: the 2026-09-26 raise from 10 to 20 minutes recorded in the same comment block. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Re-register the two workflow files (hot file) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
tools/token-e2e/freeze_snapshot.py(capture,compare,check,list-frozen), a read-only, Python-standard-library freeze snapshot and drift check for the PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 token-E2E window W (versions, hashes, settings-derived booleans, unit states, gateway route digests, catalogue counts; ID-free sanitized output; nothing on the host is changed), with its README and 122 tests.b2767df2lane:foundation(Gate A unit U8; owner native-agent-stack-2d)tools/token-e2e/freeze_snapshot.py,tools/token-e2e/README.md,tests/test_freeze_snapshot.py,manifests/evidence.json(hot file: main's copy plus the three registrations, last commit)SOTA sources
There is no upstream implementation of this glue. The closest maintained tools were checked on 2026-09-29 and not adopted: osquery (
specs/linux/systemd_units.tablehas noMainPIDorNRestartscolumn,specs/hash.tablecovers files only, and it needs its own binary), AIDE (a file-integrity hash database), and Ansible or InSpec (frameworks that check declared state); none readsclaude,codexorqmd, derives values from a settings file, digests a gateway answer or carries this tool's redaction rules. Each rule follows its own source:evidence/artifacts/token-adoption-e2e-20260926/): the frozen list and the rule that nothing changes while a run window is open.systemctl show(systemd 255, https://www.freedesktop.org/software/systemd/man/255/systemctl.html):--property=for computer-parsable output; each default unit's own binary--helpfor its configuration flag (otelcol--config, Loki-config.file, Prometheus--config.file, Grafanaserver --config), read 2026-09-29.--no-optional-locks(https://git-scm.com/docs/git) andgit status --porcelain --untracked-files=no(https://git-scm.com/docs/git-status).--no-session-persistence,--setting-sources,--output-format,--verbose,--model.MemAvailable.examples/claude-native/workflows/child-usage.mjsverifiedShellParser(U1, not merged at this base; read from branchclaude/pra-u1-cmdpos-2d-20260928at2bad7320, lines 643-655, andclaude/pra-u1d-parser-ci-2d-20260929at967561cc): the directory order, pinned files and lockfile rule thattools.parser.*mirrors.claude mcp listandclaude -p ... --output-format json(claude 2.1.284),qmd status(qmd 2.8.3),scripts/adoption_status.py --client-wiring --json/--pinned-versions --json,scripts/codex_quota.py --json, tool--versionlines.claude mcp listhas no documented machine-readable form, and the Agent SDK reference documentsRateLimitInfowithoutunifiedWindows, so those shapes are observed, not specified.Evidence-class table
local_integrationpython3 -B -m unittest tests.test_freeze_snapshot→Ran 122 tests,OKlocal_integration(private capture, not committed)freeze_snapshot.py capture --label rehearsal --out <scratch> --repo <worktree>(exit 0); leak scan counts onlycompareof a capture with itself reports 0 drift;checkagainst an expected file derived from it passes 252/252; one mutated expected value fails exactly one itemlocal_integrationcompareexit 0;checkexit 0 and exit 1list-frozen; the Python parser-pin mirror disagreeing with the kernel) were fixed with failing-first tests and re-verified by an independent command re-run (13 of 20 claims pass; the rest are handoff-count corrections, an environment-owned scratch directory, and two recorded low residuals)source_reviewnative_provenfor the shapes read;syntheticfixtures for the teststests/test_freeze_snapshot.pyLocal commands run
Residuals recorded (bounded-review-loops: one review, one repair round): (1) the cross-family review is missing for lack of Codex capacity; (2) two low residuals from the re-verify:
list-frozenstill prints a home path when a gateway route path in the configuration is itself a literal home path, and the parser mirror'sall_match_pincan be true for a lockfile with an extra malformed key where the kernel sayshash_mismatch; (3) U1 is unmerged, sotools.parser.*is verified against fixtures and the real installed bytes with the U1 pin read bygit show: after U1 merges, re-readverifiedShellParserand re-run a capture on the merged main; (4) the qmd index counts drift on their own on this host, so a seal or W-end compare reportstools.qmd.*drift unless the owner stops the updater.Decision record
None new: this is a tool for the sealed #381 protocol (Gate A U8). The alternatives checked and why none was adopted are in the README "Sources" section.
Host evidence
Not applicable (no files under
evidence/hosts/).Checklist
permissions: contents: read(no workflow changes).🤖 Generated with Claude Code