Skip to content

Skills trial: remove verification-before-completion under the conflict rule (M4), #381 Amendment 3 - #464

Merged
seathatflowsinourveins merged 3 commits into
mainfrom
claude/remove-verification-before-completion-20260928
Sep 28, 2026
Merged

seathatflowsinourveins merged 3 commits into
mainfrom
claude/remove-verification-before-completion-20260928

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR removes the trial skill verification-before-completion (obra/superpowers@8ca22dba9a94f28898bbce59f2537ff4d87c747d) under the trial's in-force conflict rule, docs/decisions/2026-09-25-skills-trial-and-usage.md:284-286: "gives instructions that conflict with CLAUDE.md/AGENTS.md once actually read in full → remove it immediately".

The conflict:

  • The pinned SKILL.md puts "Previous run" under Not Sufficient for "Tests pass" (L42). It also says "If you haven't run the verification command in this message, you cannot claim it passes" (L20) and "Execute the FULL command (fresh, complete)" (L28).
  • AGENTS.md:16 says "Reuse passing evidence when its inputs still match and run only checks needed for a concrete gap".

Findings. They are recorded in docs/decisions/2026-09-28-delegated-decisions.md (#462) and evidence/artifacts/delegated-decisions-20260928/:

  1. The trial owner's full read.
  2. The practice-sweep reviewer.
  3. GPT-6 (the pinned request was gpt-6-astra at max, read-only): "VERDICT: conflict", disposition "remove now".

Findings 1 and 2 are not shown to be independent. coordination.md records that one reading was shared before the GPT-6 verdict.

Changes

  • Skills manifest: the row moves to excluded, with a dated reason that quotes both sides. The budgets are recomputed: Claude from 7,409 to 7,184, and Codex from 3,054 to 2,829. The template's skillOverrides key is removed.
  • isolated-builder (all three copies, byte-identical, sha256 0f8e0834…): the preload and the body line that named the skill are removed.
  • adoption/hooks/claude/token-lanes-block.builder.md: gains the default block's evidence sentence, so the builder keeps an evidence rule. The token-lanes addendum records why Token-lanes carrier: evidence sentence replaces the verification-skill line (AN-15, after #447) #456's removal no longer applies. The new block sha256 is c81a91c4ffb2fa906feb403d5a11bd294b6cf2db45795d80f9acd662c5004973.
  • Landscape-sweep templates and TEMPLATE_SKILLS: drop the skill. "Evidence before any verdict" stays as plain text, and the prompt hash is updated.
  • Workflow suites: they now pin the builder preload set as ['context-mode:context-mode'].
  • PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 token-E2E preregistration, Amendment 3 (pre-execution, authorised by the Gate A owner):
    • a new isolated-builder.md role-body row 0f8e0834… and a new RUNBOOK seal 135340b6…;
    • the d022295a and Amendment 2 rows kept as history;
    • tests/test_token_e2e_preregistration.py pins both.
  • Current-state prose: updated to match the removal. Dated records stay unedited.

Host steps after merge

These run on this host only, and a receipt comes in a follow-up:

  1. DISABLE_TELEMETRY=1 skills remove verification-before-completion -g -y -a claude-code codex, through the pinned skills 1.7.0 CLI.
    • -a limits the removal to the two agents the installer adds it for (install_skills.py L60-62). Without it, remove targets every known agent.
  2. python3 tools/adoption/install_claude_profile.py --only guard --only agents.
  3. A python3 scripts/skills_status.py read-back.

Review

  • Gate A owner (native-agent-stack-a9) reviewed Amendment 3 at d98ff7d0 and approved it with no blocking findings. It re-derived the role-body and seal hashes and re-ran the preregistration, token-lanes and install-profile tests (109 OK), both node suites and validate.py.

  • GPT-6 cross-family review (gpt-6-astra, max, read-only): VERDICT: clean. Usage: 1,207,847 input tokens (1,094,784 of them cached) and 15,607 output.

  • Independent stack-verifier: it confirmed the tests, all hashes, the budget recompute (7,184 and 2,829, by both the test predicate and the named method) and the registration (26 of 26). A scratch-HOME install_claude_profile.py --home ... --only guard --only agents showed:

    • the installed builder preloads only context-mode:context-mode;
    • its block holds the evidence sentence;
    • the hook injects that block.

    It found two low defects:

    1. Fixed in 29ed7d98: the post-patch-validation exclusion reason still assumed the removed skill.
    2. Kept, and recorded here: Amendment 3 edits three earlier README spots in place: a sentence at L228, the arms-roles table row at L234, and a note in Sealing. Each edit is dated and listed in Amendment 3's change rows, and none changes a hash or seal. Amendment 2 (PR-H (#381) Amendment 2: adopt the #402 role bodies before execution (Opus verifier and builder, builder base binding) #413) set the same precedent, and the Gate A owner approved it. The Sealing rule's "never a silent rewrite" holds; strict append-only does not.

Checks

These are our integration checks, run after merging origin/main at c61e6657:

  • The skills-manifest, skills-status, install-profile, token-lanes, landscape-sweep, docs-consistency, token-E2E-preregistration and grand-dashboard unittests: Ran 350 tests, OK (skipped=7).
  • test-envelope.mjs: 254 of 254 pass. test-contract-mutations.mjs: 74 of 74 pass.
  • The three registry test methods: OK. python3 scripts/validate.py: passed.
  • Failing-first controls on the base: token-lanes failures=2, envelope 253 of 254, skills manifest failures=3, install_claude_profile failures=1, landscape prompt-hash mismatch.

SOTA sources

  • obra/superpowers@8ca22dba9a94f28898bbce59f2537ff4d87c747d, skills/verification-before-completion/SKILL.md (sha256 2befe7fc…), read in full on 2026-09-28. Upstream main has no newer commit touching the skill: the last is 3be5aad3, from 2026-07-24.
  • The upstream skills CLI 1.7.0: remove <name> -g -y -a <agents>.
  • Claude Code sub-agents docs on preloading skills.

🤖 Generated with Claude Code

Scout and others added 2 commits September 28, 2026 13:48
…t rule (M4), #381 Amendment 3

The pinned SKILL.md (obra/superpowers@8ca22dba) says "Previous run" is not sufficient and requires
a fresh full re-run "in this message"; AGENTS.md:16 says to reuse passing evidence whose inputs still
match. The trial's in-force rule (skills-trial :284-286) removes a trial skill whose instructions
conflict with AGENTS.md once read in full. Findings: the trial owner's full read, the practice-sweep
reviewer, and GPT-6 (gpt-6-astra, max) "VERDICT: conflict"; decision record
docs/decisions/2026-09-28-delegated-decisions.md (#462).

- manifest row moved to excluded with a dated reason; budgets recomputed (Claude 7,409 -> 7,184,
  Codex 3,054 -> 2,829); template skillOverrides key removed
- isolated-builder (three copies): preload and body line removed; token-lanes-block.builder.md gains
  the default block's evidence sentence (addendum records why #456's removal no longer applies)
- landscape-sweep templates and TEMPLATE_SKILLS drop the skill; workflow suites pin the new preload set
- #381 token-E2E preregistration Amendment 3 (pre-execution, authorised by the Gate A owner): new
  isolated-builder role-body row and RUNBOOK seal, history rows kept, tests updated

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e protocol

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 28, 2026
…umes the removed skill

Independent verification of #464 (low defect): the reason still read as if verification-before-completion
were installed. The exclusion stands on the fp-check duplication and the Socket Warn.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins
seathatflowsinourveins merged commit c0966da into main Sep 28, 2026
26 of 32 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/remove-verification-before-completion-20260928 branch September 28, 2026 18:29
seathatflowsinourveins pushed a commit that referenced this pull request Sep 28, 2026
…on with main (M4, #464)

Main c0966da (#464) moved verification-before-completion from the
adoption skills to its excluded list under the skills-trial conflict
rule M4. On the rebased tree the unchanged coverage contract and
installer refused the runtime manifest's reuse_ref for it (67 tests,
6 failures, 2 errors).

- Exclude the skill in the runtime manifest with its reviewed pin, the
  M4 and AGENTS.md:16 reason, and the overturn "main re-admits it"; it
  carries no adoption_ref because main no longer lists it in skills.
- New test_verification_before_completion_stays_excluded_until_main_readmits_it
  and the revised superpowers lifecycle test failed first; a source-count
  test ties sources and the research table to the selected skills.
- Recount: 137 skills from 12 sources (superpowers 14), 27 reused
  (16 Codex-disabled on main), 9 exclusions, 15 empty cells (new
  github-ci-fix/extraction-caller gap), 1,020,955 bytes and 36,739
  description characters.
- validation.json: repair_r3.not_relayed becomes the five confirmed
  CONFIRM ids; repair_r4 records the rebase, sources, fail-first and checks.
- State Codex detection precisely ($CODEX_HOME, ~/.codex when unset, or
  /etc/codex) in research.md, the harness-defaults row and a code comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 28, 2026
…on with main (M4, #464)

Main c0966da (#464) moved verification-before-completion from the
adoption skills to its excluded list under the skills-trial conflict
rule M4. On the rebased tree the unchanged coverage contract and
installer refused the runtime manifest's reuse_ref for it (67 tests,
6 failures, 2 errors).

- Exclude the skill in the runtime manifest with its reviewed pin, the
  M4 and AGENTS.md:16 reason, and the overturn "main re-admits it"; it
  carries no adoption_ref because main no longer lists it in skills.
- New test_verification_before_completion_stays_excluded_until_main_readmits_it
  and the revised superpowers lifecycle test failed first; a source-count
  test ties sources and the research table to the selected skills.
- Recount: 137 skills from 12 sources (superpowers 14), 27 reused
  (16 Codex-disabled on main), 9 exclusions, 15 empty cells (new
  github-ci-fix/extraction-caller gap), 1,020,955 bytes and 36,739
  description characters.
- validation.json: repair_r3.not_relayed becomes the five confirmed
  CONFIRM ids; repair_r4 records the rebase, sources, fail-first and checks.
- State Codex detection precisely ($CODEX_HOME, ~/.codex when unset, or
  /etc/codex) in research.md, the harness-defaults row and a code comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Sep 28, 2026
…l manifest (#429)

* Runtime-worker skills program: project-scoped installs and a 138-skill pinned manifest

GPT-6 (astra, max) build. install_skills.py gains --project-dir and --agent
targets with the default global behaviour unchanged, project lock identity and
pinned source-tree checks, and scoped rollback. The runtime-worker manifest pins
138 skills (77 OpenHands extensions v0.25.0, all 15 superpowers lifecycle
skills, 28 existing adoption entries by reference) with a report-only freshness
checker and scheduled workflow. Not installed into any worker yet; invocation
and pruning are measured after install.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills: repair round for the independent Claude review

GPT-6 (astra, max) repair of all seven findings; the blocker (#407's scoped
global rollback) was already resolved at the rebase and is kept byte for
byte. Project rollback now removes exactly the add targets and checks the
canonical folder, project lock entry and Claude link independently,
reporting a retained copy as an error instead of widening the deletion
(skills@7407f389 src/remove.ts:209-333). A project .claude/skills/<name>
that is not a symlink to the canonical folder is local-modified unless
--force (installer.ts:254-264). Every selected source/ref is fetched before
any add; an unavailable gh is reported as unverified, not as a mismatch.
The pruning rule records Claude Code's skill-listing budget and treats
dropped descriptions as unknown. 52 tests pass; five fail-first runs are
recorded in validation.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* CI fix: cover the skills freshness workflow and classify the repair values

Found and fixed through the upstream gh-fix-ci skill (openai/skills@49f948fa)
on the GPT-6 lane, then reviewed by the coordinator:
- tests/test_workflow_security_coverage.py lists
  runtime-worker-skills-freshness.yml; the workflow itself already passes the
  hardening, pinning, permissions and zizmor checks.
- blind_checkout: the review disposition "fixed" joins the closed-vocabulary
  labels (as in #416), and the superseded rollback assertion's disposition is
  data.
- harness-defaults anti-pattern row: run acceptance with TMPDIR outside every
  repository.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test_install_skills: compare resolved paths on both sides for macOS temp dirs

On macOS the temporary directory sits under /var, a symlink to /private/var,
so local.resolve() returned the /private/var form while the expected path kept
the /var form. validate-macos failed 8 subtests of the two new project-install
tests (run 36349458060). Resolve the expected path too, as
tests/adaptive_paper_hermetic.py already does with os.path.realpath.

Reproduced on Linux with TMPDIR set to a symlinked directory: 8 failures
before, OK after; the module passes 38 tests with a plain or symlinked TMPDIR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills repair r2: exclude main's gated security-audit, take reused gates from main

Coordinator direction on the #429 blocker (option b): runtime workers must
not bypass a gate set on main.

- Contract: every skill in adoption/skills/manifest.json is either reused
  by reuse_ref (pins equal to main's, gates not restated) or listed in
  `excluded` with an adoption_ref, a reason and an overturn condition.
  security-audit (#448, 8315274: Codex off, name-only pending the M5c
  bake-off against /security-review) is excluded, and a test fails as soon
  as main's gate on it changes. A synthetic future main skill in neither
  set fails the contract (negative control).
- Gates: install_skills.py resolves each reuse_ref when it reads a
  manifest, refuses an entry whose pin drifted or that restates
  codex_enabled/claude_listing, and takes both gates from main.
  --print-codex-config on the runtime manifest now disables the 16 reused
  skills main keeps off for Codex; before, it printed none. The README
  states that claude_listing is a Claude Code skillOverrides setting that
  the OpenHands SDK does not read (OpenHands/software-agent-sdk@fcc102a6
  skill.py:1123-1130, 182-185, 278-284, 528-538; same blob at da28c773).
- --only naming a pruned project skill fails as pruned, not unknown.
- The README documents that --check-only needs the pinned skills binary,
  and gh sign-in and network in project mode.
- Counts: 138 selected skills from 12 sources (unchanged); 28 of main's 29
  adoption skills reused; 8 exclusions (7 OpenHands, 1 adoption).
- validation.json records the round (fail-first: 9 tests, failures=7,
  errors=2; after the fix: 9 OK) and refreshes every input hash; the
  anti-pattern log gains the dropped-gates row; the 8 changed files are
  re-registered in manifests/evidence.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills: narrow the OpenHands loader claim and the security-audit overturn (R2-6)

The README said OpenHands withholds a skill from the model only through
SKILL.md disable-model-invocation, and research.md said the SDK reads only
SKILL.md frontmatter. load_user_skills also loads OpenHands' own enabled
installed skills (skill.py:936-973 at fcc102a6). Both now say the loader
reads skill directories, SKILL.md frontmatter and its installed-skill enable
state, never Claude Code settings or a skillOverrides key. The
available_skills citation moves from lines 182-185 to 183-186.

The security-audit overturn now names a promotion as codex_enabled true or
claude_listing on; any other change to main's gate still fails the
exclusion test and needs a new review. The test message says so.

validation.json records R2-6, the corrected sources, the red excerpt in log
order, refreshed input hashes and the rerun checks; the five changed files
are re-registered in manifests/evidence.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills records: note the rebase onto main 8d8f79c

research.md and validation.json record the second rebase, onto 8d8f79c
(#460), the merged docs/harness-defaults.md input hash and the checks rerun
on that tree. The manifests/evidence.json re-registration that was part of
this commit was dropped when the branch was rebased onto c1581fa, and is
redone at the tip.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills repair r3: containment and pin checks before add, project scope, name-keyed reuse

Two independent reviews of 8efdc35, a GPT-6 cross-family review
(codex_call.sh, gpt-6-astra max) and a headless Claude session review,
returned changes-needed with no blocker or high finding. Each code item
failed first against the unchanged code:

- GPT6-C1: refuse a project whose .agents/skills, .claude/skills or
  skills-lock.json passes through a symlink, whose selected skill paths
  resolve outside it, or that is --home, before any skills or gh call.
- CL-D1: compare every selected pinned tree with its tree_sha in the
  preflight and refuse before any add; the native in-use guard keeps a
  rolled-back copy while a detected agent such as Codex reads it.
- CL-D3: the runtime manifest carries "scope": "project" and is refused
  without --project-dir; pruned entries are never installed in either mode.
- CL-D6: reuse_ref and adoption_ref name the adoption manifest and match
  by skill name.
- CL-D7: runtime_skill_freshness.py reads the CLI version and ref from
  the manifest's cli pin.
- GPT6-C5: offline zizmor analysis of the freshness workflow, with
  floating-tag, persist-credentials and permissions controls.

Docs: the rendered OpenHands skill list omits <location>; disabled_skills
and the loading flags are its controls; third-party instruction files load
from the work directory and Git root, so the example worker projects move
outside the checkout; rollback retention is certain when Codex is detected;
skills-review-r1.json is labelled unpublished.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Runtime-worker skills repair r4: exclude verification-before-completion with main (M4, #464)

Main c0966da (#464) moved verification-before-completion from the
adoption skills to its excluded list under the skills-trial conflict
rule M4. On the rebased tree the unchanged coverage contract and
installer refused the runtime manifest's reuse_ref for it (67 tests,
6 failures, 2 errors).

- Exclude the skill in the runtime manifest with its reviewed pin, the
  M4 and AGENTS.md:16 reason, and the overturn "main re-admits it"; it
  carries no adoption_ref because main no longer lists it in skills.
- New test_verification_before_completion_stays_excluded_until_main_readmits_it
  and the revised superpowers lifecycle test failed first; a source-count
  test ties sources and the research table to the selected skills.
- Recount: 137 skills from 12 sources (superpowers 14), 27 reused
  (16 Codex-disabled on main), 9 exclusions, 15 empty cells (new
  github-ci-fix/extraction-caller gap), 1,020,955 bytes and 36,739
  description characters.
- validation.json: repair_r3.not_relayed becomes the five confirmed
  CONFIRM ids; repair_r4 records the rebase, sources, fail-first and checks.
- State Codex detection precisely ($CODEX_HOME, ~/.codex when unset, or
  /etc/codex) in research.md, the harness-defaults row and a code comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Rebase onto main c0966da: re-register the PR's files in manifests/evidence.json

Main's evidence.json with the 13 PR files registered at their current
sha256 and byte size (8 added, 5 updated); no other entry changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant