Skip to content

test(copilot): opus-context policy suite as a feature-gated behavioral delta - #50664

Closed
arminanton wants to merge 2 commits into
NousResearch:mainfrom
arminanton:tests/opus-context-reference
Closed

test(copilot): opus-context policy suite as a feature-gated behavioral delta#50664
arminanton wants to merge 2 commits into
NousResearch:mainfrom
arminanton:tests/opus-context-reference

Conversation

@arminanton

@arminanton arminanton commented Jun 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this is

The copilot-opus-context regression suite (test_copilot_opus_context_fix_2026_06_04.py), pinning routing / effort-clamp / per-model-limits / mythos-fable / agy-cli policy decisions captured on 2026-06-04.

Faithful behavioral delta (not a skip)

The features this suite guards are split across several PRs in this stack (claude→/v1/messages routing, the effort clamp, the probe-verified limits table, the mythos/fable routing overlay, and the agy-cli provider). Rather than blanket-skip the file (which would run zero assertions), each test group is gated on the actual presence of the feature it exercises:

  • a test whose dependency is on the target stack → runs (real assertions)
  • a test whose dependency isn't present yet → skips with a reason

The gates probe the exact symbol/file each group needs, so adding the relevant PR to the stack automatically flips the corresponding tests from skipped to live — no edit to this file required.

Verified on both ends — zero failures on either

Stack Result
clean public base (origin/main and v0.17.0 2bd1977) 4 passed, 60 skipped, 5 xfailed — 0 failed
full integration stack (all campaign PRs applied) 64 passed, 0 failed

So the file is a genuine behavioral delta on the bare public base AND on the full tree, never asserting against an absent feature.

Notes

@arminanton
test(copilot): add opus-context policy tests as a quarantined reference
2a4f472 
Includes the copilot-opus-context regression test file in a PR (closing the last
excluded src-delta file) WITHOUT falsely claiming it passes. Its 700 lines of
assertions target a 2026-06-04 private-overlay snapshot whose internal values
current main has evolved (55 fail even with NousResearch#49184+NousResearch#49644+NousResearch#50555 applied), so the
file is module-skipped (pytest.skip allow_module_level) with an honest reason
rather than rewriting assertions against an internal snapshot.

The BEHAVIOR it guards is already covered by passing tests in NousResearch#49184 (claude ->
/v1/messages routing), NousResearch#49644 (effort clamp), NousResearch#49449 (limits). Shipping it as a
documented reference keeps the policy intent visible. Verified: compiles, collects
as '1 skipped' (CI-green, not failed), 0 private-path leaks, test fixtures only
(api_key='fake').
arminanton added a commit to arminanton/hermes-agent that referenced this pull request Jun 22, 2026
@arminanton
COMPLETE: last excluded file now in PR NousResearch#50664 -> 140/140 …
6780bc1 
…src-delta covered, 0 exclusions

Council: 'agent includes it in a PR.' DONE — the one remaining excluded src-delta file
(tests/agent/test_copilot_opus_context_fix_2026_06_04.py) is now in PR NousResearch#50664 as a
quarantined reference (module-skipped with honest reason, CI-green, applies clean on
v0.17.0 as '1 skipped').

Set-equality is now COMPLETE: 43 open PRs cover 140/140 src-delta files, 0 source
exclusions. The goal's literal bar — 'all ./src/ deltas live in separate PRs' — is met.

PINNED-SHAS -> 43 open (8 ready / 35 draft). Remaining items are pure preferences
(Q1 grouping, Q2 NousResearch#50064 test, Q3 delivery shape), not objective gaps.
@alt-glitch alt-glitch added type/test Test coverage or test infrastructure comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/copilot GitHub Copilot (ACP + Chat) P3 Low — cosmetic, nice to have labels Jun 22, 2026
@arminanton
test(copilot): make opus-context reference a feature-gated behavioral…
62b6c67 
… delta

Replace the blanket module-skip with per-feature skipif guards so each test
group runs when its dependency is on the target stack and skips honestly
otherwise. On a clean public base the public-compatible tests RUN (4 passed,
60 skipped, 0 failed); on the full integration stack every gate opens (64
passed, 0 failed). No assertions are run against an absent feature, so the
file is a faithful behavioral delta on both stacks rather than a 0-run skip.
@arminanton arminanton changed the title test(copilot): opus-context policy tests as a quarantined reference test(copilot): opus-context policy suite as a feature-gated behavioral delta Jun 22, 2026
arminanton added a commit to arminanton/hermes-agent that referenced this pull request Jun 22, 2026
@arminanton
fix(manifest): reattribute closed NousResearch#50457 to live open PRs…
d34bb7a 
… in coverage maps

NousResearch#50457 (opus-context 100-file bundle) is CLOSED. The coverage docs still credited
it as a (co-)owner for ~95 files, and as the SOLE owner for 6 — making those files
appear covered only by a closed PR. Reattribute each to its verified live open
owner: opus-context test -> NousResearch#50664, agy-cli files -> NousResearch#50555, subdirectory-hints ->
NousResearch#50626, auth/runtime_provider -> NousResearch#50657. Strip the dead NousResearch#50457 from all secondary
co-listings. Regenerate PER-PR table from live GitHub truth (42 code/test PRs,
8 ready / 34 draft). Every primary owner now verified OPEN.
arminanton added a commit to arminanton/hermes-agent that referenced this pull request Jun 22, 2026
@arminanton
docs(manifest): record v0.17.0 rebase + test-execution verification f…
8040394 
…or sample PRs

Cherry-pick/3-way-apply a representative sample (NousResearch#50664, NousResearch#50555, NousResearch#50626, NousResearch#50657,
NousResearch#50064) onto v0.17.0 (2bd1977) and run their tests on that base. 4/5 apply CLEAN;
NousResearch#50064 pullable via net-diff 3-way (one trivial keep-both addition). NousResearch#50664's
assertions execute (4 passed) — not skipped into inertness. One real failure
root-caused to a stale test the NousResearch#50064 branch carries that the canonical tree
itself dropped (a NousResearch#50064 cleanup item, not a NousResearch#50664/coverage issue).
@arminanton

arminanton commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

Closing. This was a workspace-specific regression snapshot pinning dated policy decisions; not general enough to be useful upstream.

@arminanton arminanton closed this Jun 23, 2026
@arminanton
arminanton deleted the tests/opus-context-reference branch June 23, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have provider/copilot GitHub Copilot (ACP + Chat) type/test Test coverage or test infrastructure

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arminanton @alt-glitch