test(gateway): make the fallback-chain tests independent of host config - #299
Merged
Conversation
TestResolveRuntimeWithFallback patches `server._load_fallback_model`, but `_resolve_runtime_with_fallback()` consults `_load_effective_fallback_model()` — the policy-filtered chain added by #269. That wrapper only reaches the raw loader when the ambient policy is not "off": def _load_effective_fallback_model(): cfg = _load_cfg() policy = get_fallback_policy(cfg) if policy == "off": return [] chain = _load_fallback_model() or [] CI has no hermes config, so the policy defaults to "any", the stub is reached, and these tests are green there. On a host whose config sets `fallback_policy: off`, the wrapper short-circuits to [], the stub is never consulted, no fallback is ever attempted, and all five tests fail for a reason that has nothing to do with what they assert. Patch what the code actually calls, so the case under test is the one that runs on any host. `test_auth_error_all_fallbacks_fail_raises` additionally pins `_load_cfg`, because it asserts the policy-specific exhaustion wording ("no usable configured backup route remained" is the "any" branch) and would otherwise assert against whatever policy the host happens to have. This is not a CI-visible failure — it is a latent host-config dependency that makes the suite behave differently for a contributor than it does in CI. TestResolveRuntimeWithFallback: 8 passed against a config with `fallback_policy: off`, and 8 passed with no hermes config at all. Before this change the first environment produced 5 failures. This file's two remaining failures (golden transcript, session activate) are owned by #296; this change is disjoint from it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OmarB97
force-pushed
the
fix/tui-gateway-test-drift
branch
from
August 2, 2026 06:15
4cd7690 to
3c801ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Removes a latent host-config dependency in
TestResolveRuntimeWithFallback(5 tests intests/test_tui_gateway_server.py).The tests patch
server._load_fallback_model, but_resolve_runtime_with_fallback()consults_load_effective_fallback_model()— the policy-filtered chain added by #269. That wrapper only reaches the raw loader when the ambient policy is notoff:This is not a CI failure, and I want to be precise about that. CI has no hermes config, so
get_fallback_policy({})returns the"any"default, the wrapper falls through to the patched raw loader, and all five tests pass. They are green onmaintoday.On a host whose
config.yamlsetsfallback_policy: off, the wrapper short-circuits to[], the stub is never consulted, no fallback is ever attempted, and all five fail — asserting nothing about the code they exist to cover. The result is a suite that behaves differently for a contributor than it does in CI, and five tests that are silently inert for everyone else (they pass through a stub that is only reachable by accident of the default policy).The fix is to patch what the code actually calls.
test_auth_error_all_fallbacks_fail_raisesadditionally pins_load_cfg, because it asserts the policy-specific exhaustion wording —"no usable configured backup route remained"is theanybranch — and would otherwise assert against whatever policy the host happens to have.Scope note: this file has two other failures on
main(the golden turn-isolation transcript and the session-activate double). Those are the CI-blocking ones and they are owned by #296, which is already green. This change is disjoint from that PR — different tests, no overlapping lines — so the two compose. Until #296 lands,Python testson this PR will still show those two failures; they are pre-existing onmainand untouched here.Related Issue
No filed issue — found while driving
main's CI back to green after the stale-lockfile fix (#286) unmasked the Python suite.Type of Change
Changes Made
All in
tests/test_tui_gateway_server.py, classTestResolveRuntimeWithFallback:monkeypatch.setattrtargets changed from_load_fallback_modelto_load_effective_fallback_model— the function_resolve_runtime_with_fallback()actually calls.test_auth_error_all_fallbacks_fail_raisespins_load_cfgto{"fallback_policy": "any"}so its assertion on the exhaustion wording is deterministic.How to Test
Green either way after this change; before it, the first command produced 5 failures and the second 0:
HERMES_HOME=/path/to/a/home/whose/config/sets/fallback_policy_off \ .venv/bin/python -m pytest "tests/test_tui_gateway_server.py::TestResolveRuntimeWithFallback" -qenv -u HERMES_HOME .venv/bin/python -m pytest "tests/test_tui_gateway_server.py::TestResolveRuntimeWithFallback" -qBoth now report 8 passed. Whole-file run in a CI-like environment: 382 passed, 2 failed — the two failures being the pre-existing golden-transcript and session-activate cases owned by #296.
Checklist
Code
fix(scope):,feat(scope):, etc.)Documentation & Housekeeping
cli-config.yaml.example— N/ACONTRIBUTING.md/AGENTS.md— N/ARisks / gaps
ruff check tests/test_tui_gateway_server.pyis clean._load_effective_fallback_modelmeans these tests no longer exercise the policy gate itself. That is the correct split — they are about walking the chain onAuthError, and the gate has its own coverage — but it does mean a regression in theoff/local-onlyfiltering would not be caught here.