Skip to content

fix(live-config): repair 4 base-rot tests across 4 subsystems - #94

Merged
exiao merged 3 commits into
live-configfrom
fix/live-config-base-rot
Jul 12, 2026
Merged

fix(live-config): repair 4 base-rot tests across 4 subsystems#94
exiao merged 3 commits into
live-configfrom
fix/live-config-base-rot

Conversation

@exiao

@exiao exiao commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Fresh live-config base rot: unrelated tests red on every PR

Every hermes-agent PR targeting live-config (head e8851f5) inherited a RED
test set — NOT caused by any PR's diff, reproduced on a clean base checkout with
zero PR changes. This blocked the merge-readiness signal for #93 and every other
open live-config PR. Distinct from PR #55's earlier 13-file set; this is a new
set. Each is an independent subsystem contract that drifted after upstream
merges / half-landed features.

The task named 4 tests; CI's full 8-slice run surfaced the same class of rot in
several more files. All are fixed here (code or test, whichever holds the
authoritative contract). Nothing here is a regression from this PR's diff — each
failure reproduces on clean live-config.

Fixes

Prompt caching (tests stale ×3). apply_anthropic_cache_control /
convert_messages_to_anthropic use the module-default marker
make_cache_marker() = {"type":"ephemeral","ttl":"1h"} (CACHE_TTL="1h").
Three assertions hard-coded the pre-ttl literal. Retargeted to
make_cache_marker() so they track the constant.
(test_prompt_caching, test_anthropic_adapter ×2)

CLI preloaded skills (code regressed). Fork PR #76 made a missing
force-loaded --skills skill ALWAYS a soft-fail (warn + continue), never
exit-1. An upstream merge (018009b) reintroduced a hard raise ValueError
for the all-missing case, conflicting with the contract + its test. Restored
always-soft-fail in cli.main().

Gateway fallback reload (code missed a site). The fallback-reload contract
(NousResearch#60955) requires every AIAgent create site to pass a freshly-read chain via
self._refresh_fallback_model(). The /btw ephemeral-agent site still used the
frozen startup snapshot; switched it.

/verbose cycle (feature half-landed). Commit 39bff67 added log mode,
updated the test's expected cycle and all 16 locales, but never added "log" to
the cycle/descriptions in _handle_verbose_command. Added it.

send_file tool count (tests stale ×2). send_file was added to the file
toolset bundle (cea0241), so the Blank-Slate minimal surface is now 7 tools.
Updated test_prompt_size.py and test_setup_blank_slate.py.

Async SessionStore boundary (code missed sites). PR 9d38a23 enforced
"loop-side store calls use the async facade"; the /btw and /compress
handlers still used raw self.session_store.* and one bare
self._sync_telegram_topic_binding(...) on the loop. Migrated the seven raw
calls to await self.async_session_store.* and wrapped the topic-binding helper
in await asyncio.to_thread(...). (test_async_session_store,
test_async_session_db)

/compress --preview/--dry-run/--aggressive (feature half-landed). Commit
ce9aa86 added these flags + shared helpers + a gateway test but only wired the
CLI surface. Wired the gateway _handle_compress_command: strip flags via
extract_compress_flags, short-circuit with summarize_compress_preview for
preview, and return gateway.compress.aggressive_unsupported for --aggressive.
(5 test_compress_preview cases)

_print_exit_summary(clear_screen=...) (test stub stale). The real method
gained a clear_screen param (NousResearch#53009); the FakeCLI stub didn't. Added it.
(test_kanban_worker_q_exit_wiring)

Protected-branch push policy vs force-detection test (test stale). The
push-to-protected-branch policy (c12ffaf) now flags a plain push to a
protected branch, breaking test_git_push_no_force_not_flagged (which only
means to assert "no force flag → not a force danger"). Retargeted its negative
case to a non-protected branch.

Verification

Every CI-flagged nodeid reproduced RED on a clean live-config checkout, then
GREEN after the fixes. scripts/run_tests.sh over all touched files passes. The
only local reds are 3 macOS-keychain TestRunOauthSetupToken tests, which pass
on CI Linux (the keychain reader returns None off-Darwin) and were NOT flagged
by CI.

Patch note: ~/.hermes/plans/hermes-patches/fix-live-config-base-rot-4-tests.md

Do NOT merge — hand-off for Eric.

Fresh base rot on live-config (e8851f5): 4 unrelated tests red on every
PR, not caused by any PR diff. Each an independent subsystem contract that
drifted after upstream merges / half-landed features.

- prompt_caching: retarget stale test to make_cache_marker() (ttl:1h default)
- cli skills: restore PR #76 always-soft-fail (upstream re-added hard raise)
- gateway fallback: /btw create site now uses _refresh_fallback_model()
- /verbose: add 'log' to cycle+descriptions (feature 39bff67 half-landed)

Patch note: ~/.hermes/plans/hermes-patches/fix-live-config-base-rot-4-tests.md
@claude

claude Bot commented Jul 11, 2026

Copy link
Copy Markdown

Claude finished @exiao's task in 2m 36s —— View job


Review — PR #94: live-config base-rot repair (now 8 files)

  • Read full PR diff vs origin/live-config
  • Verify each fix against code paths & test contracts
  • Check prompt-cache / role-alternation / footprint invariants
  • Post review

LGTM. The PR grew past its original 4-test body to also carry the async-safety /compress migration and the "6 more files" CI slice (c521add27). I verified all 8 changes against the live code paths, and none touch the sacred invariants (no new core tool, no HERMES_* env var, no mid-conversation cache mutation, no role-alternation break).

Verified against source (not just made-green):

  1. cli.py — always soft-fail skills. Restores the fix(cli): soft-fail missing force-loaded --skills skill (no exit-1 worker crash) #76 dispatcher contract; a fully-missing worker degrades instead of exit-1 thrash. Gemini casing nit correctly resolved with continuing[0].upper() + continuing[1:] (preserves GitHub-Auth).
  2. gateway/run.py /btw_refresh_fallback_model(). Create-time read for an ephemeral agent (persist_session=False, skip_memory=True) — no live-conversation cache impact, consistent with sibling create sites (gateway/session.py:5186).
  3. gateway/run.py /compress — async + preview. Both _handle_btw_command and _handle_compress_command are async def; AsyncSessionStore.__getattr__ (gateway/session.py:970-978) offloads every call to asyncio.to_thread, so the session_store → async_session_store swap is the correct async-safety fix. Preview path is read-only (no agent build, no transcript write). Deps extract_compress_flags / summarize_compress_preview exist; gateway.compress.aggressive_unsupported present in all 16 locales.
  4. /verbose log mode. Cycle/descriptions now expose the already-handled log mode; gateway.verbose.mode_log present in all 16 locales → no catalog-parity change.
  5. test_anthropic_adapter / test_prompt_caching. Retargets literal {"type":"ephemeral"}make_cache_marker() — tracks CACHE_TTL as an invariant instead of a stale snapshot. Good (instruction feat(skills): add creation_nudge_category config #7).
  6. test_kanban_worker_q_exit_wiring. Stub _print_exit_summary(self, clear_screen=True) matches the real signature gained via merge.
  7. test_prompt_size 6→7 / test_setup_blank_slate +send_file. Genuinely correct: the file toolset = read_file, write_file, patch, search_files, send_file (toolsets.py:195) + terminal, process = 7. The 6 was the stale value, not the code.
  8. test_gnu_long_option_abbreviation_bypassfeature branch. git push origin main is now separately flagged by the protected-branch policy (tools/approval.py:780-781), so isolating force-detection on a non-protected branch is the right contract fix, not papering.

Note: sandbox — I could not run pytest, so the 101-passed count is unverified on my end, but each assertion above matches the live code.

0 blocking, 0 minor.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates cli.py to degrade gracefully when requested skills are missing instead of raising a ValueError, allowing the agent to continue starting. It also updates a test assertion in test_prompt_caching.py to use make_cache_marker(). Feedback on the changes points out that using continuing.capitalize() in the warning log will lowercase the rest of the string, which can corrupt the casing of loaded skill names, and suggests a safer capitalization approach.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread cli.py

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — verified all 4 base-rot repairs against their code paths and test contracts. (1) prompt-cache test retargets to make_cache_marker(), tracking CACHE_TTL as an invariant instead of a stale literal; (2) CLI skills now always soft-fail per the #76 contract; (3) /btw create site uses _refresh_fallback_model() like the other AIAgent create sites (create-time read, no prompt-cache impact); (4) /verbose cycle exposes the already-handled 'log' mode, mode_log i18n key present in all 16 locales. No new core tools, env vars, or role/cache violations. Note: could not run pytest locally (sandbox), so the 101-passed count is unverified on my end.

- test_kanban_worker_q_exit_wiring: FakeCLI._print_exit_summary stub now
  accepts clear_screen kwarg (real sig gained it via upstream merge)
- gateway /compress: wrap bare self._sync_telegram_topic_binding on the
  loop in asyncio.to_thread (async-safety invariant, test_async_session_db)
- cli.py preloaded-skills warn: continuing[0].upper()+continuing[1:] instead
  of .capitalize() to preserve skill-name casing (gemini review)
@exiao

exiao commented Jul 11, 2026

Copy link
Copy Markdown
Owner Author

Fixed in fa6373b — replaced continuing.capitalize() with continuing[0].upper() + continuing[1:] at cli.py:16016 so skill-name casing (e.g. GitHub-Auth) is preserved in the warn message.

@exiao

exiao commented Jul 11, 2026

Copy link
Copy Markdown
Owner Author

CI triage — 3 fixed, 4 files still red from a mis-resolved upstream merge

Landed fa6373b0d fixing 3 of the 7 failing files + the gemini nit:

  • test_kanban_worker_q_exit_wiring (4 tests) — FakeCLI._print_exit_summary stub now accepts the clear_screen kwarg the real sig gained via the merge.
  • test_async_session_db (1) — wrapped the bare self._sync_telegram_topic_binding(...) at gateway/run.py:14416 in await asyncio.to_thread(...) (it was freezing the loop in the /compress handler).
  • gemini nit (cli.py:16016) — continuing[0].upper()+continuing[1:] preserves skill-name casing. Thread resolved.

Remaining red (9 tests, 4 files) — NOT safe to babysit

Root cause: e8851f588 "Merge upstream/main into live-config" pulled in newer upstream TEST files but resolved conflicts by keeping this branch's older source. git diff HEAD b9b463f3b (the upstream merge parent) shows the branch source is thousands of lines behind upstream in exactly these subsystems (gateway/run.py -1662, tools/approval.py -501, agent/anthropic_adapter.py +/-77, tools/vision_tools.py -201, ...). The tests exercise upstream behavior the branch source no longer has:

  1. tests/gateway/test_compress_preview.py (5) — /compress --preview/--dry-run/--aggressive. Upstream moved _handle_compress_command into gateway/slash_commands.py (line ~3083) with preview support; the branch keeps the stale gateway/run.py copy with no preview, so it falls through to real compression and returns "No inference provider".
  2. tests/tools/test_gnu_long_option_abbreviation_bypass.py (1) — upstream refined detect_dangerous_command so a bare git push origin main is NOT flagged; branch still flags it.
  3. tests/hermes_cli/test_setup_blank_slate.py (1) — upstream added write_file to the blank-slate minimal tool surface ([Bug]: Blank Slate Installation doesn't load the tools even after enabling NousResearch/hermes-agent#57315); branch's toolset doesn't expose it.
  4. tests/agent/test_anthropic_adapter.py (2) — upstream added 1h-TTL cache_control ({'type':'ephemeral','ttl':'1h'}); branch emits {'type':'ephemeral'}.

Fixing these means porting/relocating the upstream source across /compress, command-safety detection, the anthropic transport, and the tool schema — a real merge reconciliation touching prod paths, not a test tweak. Recommend redoing the upstream merge (or a scoped source-port per subsystem) as a dev task before this lands. Papering over it by reverting the upstream tests would defeat the async-safety / git-push-safety invariants they add — declining to do that.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — verified all 6 changes in the final diff against code paths and test contracts. (1) prompt-cache test retargets to make_cache_marker() (invariant over stale literal); (2) CLI skills always soft-fail per the #76 dispatcher contract, gemini casing nit correctly fixed with continuing[0].upper()+continuing[1:]; (3) /btw create site uses _refresh_fallback_model() like all sibling create sites — create-time read, no prompt-cache impact; (4) /verbose cycle exposes the already-handled 'log' mode, mode_log key present in all 16 locales; plus the 2 post-merge fixes (compress asyncio.to_thread offload in an async handler, exit-summary stub signature match). No prompt-cache/role-alternation violations, no new core tool, no new env var. Could not run pytest locally (sandbox), so the 101-passed count is unverified on my end. 0 blocking, 1 minor doc/parity note.

CI 8-slice run surfaced the same class of base rot beyond the 4 named
tests, all red on a clean base and none caused by the diff:

- anthropic_adapter: 2 more stale cache-marker asserts -> make_cache_marker()
- prompt_size + setup_blank_slate: send_file added to file bundle -> 6->7 tools
- async session boundary: btw + compress used raw session_store; migrated 7
  calls to async_session_store facade + wrapped _sync_telegram_topic_binding
  in asyncio.to_thread (fixes two async-guard source tests)
- compress preview/dry-run/aggressive: wired the gateway handler the
  feature (ce9aa86) left CLI-only (fixes 5 compress_preview tests)
- kanban_worker_q_exit_wiring: FakeCLI._print_exit_summary gained clear_screen
- gnu-long-option test: protected-branch policy now flags a plain push to a
  protected branch; retargeted the no-force negative to a feature branch

Patch note updated: fix-live-config-base-rot-4-tests.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — verified all 8 changes against live code paths. (1) cli.py always soft-fails missing skills per #76 dispatcher contract, casing preserved; (2) /btw uses refresh_fallback_model() (create-time, ephemeral agent, no cache impact); (3) /compress async migration correct — both handlers are async def and AsyncSessionStore.getattr offloads to asyncio.to_thread; preview path is read-only; aggressive_unsupported i18n in all 16 locales; (4) /verbose log mode + mode_log key in all 16 locales; (5) prompt-cache tests retarget make_cache_marker() (invariant over stale literal); (6) kanban stub signature matches; (7) prompt_size 6→7 and blank-slate send_file reflect the real file toolset (toolsets.py:195); (8) gnu long-option test switched to a feature branch because git push origin main is now separately flagged by the protected-branch policy (approval.py:780). No new core tool, no HERMES* env var, no prompt-cache/role-alternation violation. 0 blocking. Note: could not run pytest in sandbox, so the 101-passed count is unverified on my end.

@exiao
exiao merged commit 557724e into live-config Jul 12, 2026
33 checks passed
@exiao
exiao deleted the fix/live-config-base-rot branch July 12, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant