test(.claude/rules): canary for harness-native auto-load empirical test#1163
Merged
test(.claude/rules): canary for harness-native auto-load empirical test#1163
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a canary rule file under .claude/rules/ to empirically test whether Claude Code auto-loads rule files at session start (vs requiring an explicit read), using a unique detection string and a documented test protocol.
Changes:
- Added
.claude/rules/test-canary.mdcontaining the canary detection string and step-by-step protocol for verifying harness auto-load behavior. - Documented interpretation (“pass/fail”) and follow-up actions based on observed behavior.
AceHack
added a commit
that referenced
this pull request
May 1, 2026
…s; multi-harness reframe (rebased clean from main) (#1164) Surgical landing of the loading-taxonomy substrate after the otto/claude-code-loading-taxonomy-2026-05-01 branch hit unresolvable rebase conflicts. This commit applies ONLY the surgical edits against current origin/main: - CLAUDE.md: new ground-rule bullet for the loading taxonomy (three mechanisms across multiple surfaces, doc-supported auto-load with EMPIRICAL VERIFICATION STATUS calibration — unverified in our harness pending canary test in PR #1163). - memory/MEMORY.md: new index row pointing at the memo with the same calibrated language. - memory/feedback_claude_code_loading_taxonomy_*.md (new file): full memo with EMPIRICAL VERIFICATION STATUS section, three-mechanism taxonomy, behavioral-lesson placement rule-of-thumb, factory-owned substrate-discovery.ts as fallback under live-off-the-land preference, multi-harness reframe per maintainer 2026-05-01 ("we are going to have to solve this for every harness... substrate-discovery.ts in claude.md and agents.md"). Substrate corrections this PR makes vs prior unmerged branch: (1) Drops the fragile "five surfaces" count claim (memo actually enumerates 6 + agents = 7). Reframe around the *mechanism* distinction (direct vs lazy vs router vs subagent vs on-demand), which is the load-bearing insight regardless of how surfaces are counted. (2) Downgrades CLAUDE.md bullet language from "verified" to "doc-supported by canonical Anthropic source; rules- auto-load piece specifically is unverified in our harness pending the canary test." Prevents contributors moving lessons into .claude/rules/ on the assumption auto-load works when it might not in our harness. (3) Captures the multi-harness reframe: "live off the land" is per-harness, not global. Each harness uses native where available; substrate-discovery.ts is the factory- owned cross-harness fallback, pointed at from both CLAUDE.md and AGENTS.md. Iterative-improvement angle for skill descriptions. Composes with PR #1163 (canary test) which is the empirical gate that resolves the EMPIRICAL VERIFICATION STATUS calibration. Replaces unmergeable branch otto/claude-code-loading-taxonomy-2026-05-01 which had cumulative rebase conflicts from main advancing underneath. New branch is rebased-clean from current main. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…al test The human maintainer 2026-05-01 calibration challenge surfaced that I was claiming `.claude/rules/*.md` auto-loads in our Claude Code harness based on canonical Anthropic docs alone — without empirical verification. This canary file enables the test: 1. Detection string `RULES_AUTOLOAD_CANARY_2026_05_01_LIVE_OFF_THE_LAND` embedded in the body. Unique, grep-able. 2. Test protocol: restart Claude Code, ask fresh session for the canary string without referencing this file. If session knows the string from auto-loaded context → docs accurate; if session has to Read the file → auto-load doesn't happen in our harness. 3. Alternative: `/memory` slash command per Anthropic docs should list rules files loaded in session. The result discriminates the architectural path: - Pass → "live off the land" works; harness-native discovery is sufficient; substrate-discovery.ts unnecessary. - Fail → harness-native incomplete; need factory-owned tooling (substrate-discovery.ts) or treat .claude/rules/ as ordinary pointer-discovered docs. Composes with PR #1161's loading-taxonomy memo (which now carries the EMPIRICAL VERIFICATION STATUS calibration). Per the human maintainer's preference: "if the skill router works that's pretty agent/anthropic/harness native and i like that" + "live off the land." This test is the gate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…sal + fix markdownlint MD032 (5 Copilot threads + lint failure) Five P1 Copilot review threads on PR #1163: (1, 2, 3) Three dangling-pointer findings on the loading- taxonomy memo file. The findings were valid at PR-open time (memo was on PR #1164 branch, not main). Now resolved by PR #1164 merging — the memo is on main. Rebased canary branch on latest main; no path changes needed (same target file path, just now reachable). (4) substrate-discovery.ts referenced as if it exists; it doesn't yet. Clarified inline: 'proposed in the loading- taxonomy memo's multi-harness reframe section; not yet built — would land as tools/substrate-discovery/discover.ts if the canary fails the test.' (5) Path clarification: this repo uses CLAUDE.md at root, not .claude/CLAUDE.md. Updated reference to clarify both are valid per Anthropic docs but our repo has the root location. Plus markdownlint MD032 (lists need blank lines around) on line 104 — converted the trailing lineage paragraph from list-style to prose to satisfy the lint without changing content. Composes with PR #1167 harness-engineering memo where Osmani's 'AGENTS.md under 60 lines' is the calibration target making canary verification load-bearing for MVP trim work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
AceHack
force-pushed
the
otto/claude-rules-canary-test-2026-05-01
branch
from
0410983 to
92b2e0b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sets up the canary test described in PR #1161's loading-taxonomy memo. Restart Claude Code session, ask for the detection string
RULES_AUTOLOAD_CANARY_2026_05_01_LIVE_OFF_THE_LANDwithout referencing the file — if session knows it from auto-loaded context, harness-native discovery works; if session has to Read the file, it doesn't.Result discriminates 'live off the land' (harness-native sufficient) vs factory-owned substrate-discovery.ts.
🤖 Generated with Claude Code