Skip to content

test(gateway): make the fallback-chain tests independent of host config - #299

Merged
OmarB97 merged 1 commit into
mainfrom
fix/tui-gateway-test-drift
Aug 2, 2026
Merged

test(gateway): make the fallback-chain tests independent of host config#299
OmarB97 merged 1 commit into
mainfrom
fix/tui-gateway-test-drift

Conversation

@OmarB97

@OmarB97 OmarB97 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Removes a latent host-config dependency in TestResolveRuntimeWithFallback (5 tests in tests/test_tui_gateway_server.py).

The tests patch server._load_fallback_model, but _resolve_runtime_with_fallback() consults _load_effective_fallback_model() — the policy-filtered chain added by #269. That wrapper only reaches the raw loader when the ambient policy is not off:

def _load_effective_fallback_model():
    cfg = _load_cfg()
    policy = get_fallback_policy(cfg)
    if policy == "off":
        return []
    chain = _load_fallback_model() or []

This is not a CI failure, and I want to be precise about that. CI has no hermes config, so get_fallback_policy({}) returns the "any" default, the wrapper falls through to the patched raw loader, and all five tests pass. They are green on main today.

On a host whose config.yaml sets fallback_policy: off, the wrapper short-circuits to [], the stub is never consulted, no fallback is ever attempted, and all five fail — asserting nothing about the code they exist to cover. The result is a suite that behaves differently for a contributor than it does in CI, and five tests that are silently inert for everyone else (they pass through a stub that is only reachable by accident of the default policy).

The fix is to patch what the code actually calls. test_auth_error_all_fallbacks_fail_raises additionally pins _load_cfg, because it asserts the policy-specific exhaustion wording — "no usable configured backup route remained" is the any branch — and would otherwise assert against whatever policy the host happens to have.

Scope note: this file has two other failures on main (the golden turn-isolation transcript and the session-activate double). Those are the CI-blocking ones and they are owned by #296, which is already green. This change is disjoint from that PR — different tests, no overlapping lines — so the two compose. Until #296 lands, Python tests on this PR will still show those two failures; they are pre-existing on main and untouched here.

Related Issue

No filed issue — found while driving main's CI back to green after the stale-lockfile fix (#286) unmasked the Python suite.

Type of Change

  • ✅ Tests (adding or improving test coverage)

Changes Made

All in tests/test_tui_gateway_server.py, class TestResolveRuntimeWithFallback:

  • Five monkeypatch.setattr targets changed from _load_fallback_model to _load_effective_fallback_model — the function _resolve_runtime_with_fallback() actually calls.
  • test_auth_error_all_fallbacks_fail_raises pins _load_cfg to {"fallback_policy": "any"} so its assertion on the exhaustion wording is deterministic.
  • A comment at the first patch site explaining the wrapper/policy interaction, so the next person does not re-point it at the raw loader.

How to Test

uv sync --locked --python 3.11 --extra all --extra dev

Green either way after this change; before it, the first command produced 5 failures and the second 0:

HERMES_HOME=/path/to/a/home/whose/config/sets/fallback_policy_off \
  .venv/bin/python -m pytest "tests/test_tui_gateway_server.py::TestResolveRuntimeWithFallback" -q
env -u HERMES_HOME .venv/bin/python -m pytest "tests/test_tui_gateway_server.py::TestResolveRuntimeWithFallback" -q

Both now report 8 passed. Whole-file run in a CI-like environment: 382 passed, 2 failed — the two failures being the pre-existing golden-transcript and session-activate cases owned by #296.

Checklist

Code

Documentation & Housekeeping

  • I've updated relevant documentation — N/A, test-only change
  • I've updated cli-config.yaml.example — N/A
  • I've updated CONTRIBUTING.md / AGENTS.md — N/A
  • I've considered cross-platform impact — N/A, no platform-specific code; this removes an environment dependency
  • I've updated tool descriptions/schemas — N/A

Risks / gaps

TestResolveRuntimeWithFallback patches `server._load_fallback_model`, but
`_resolve_runtime_with_fallback()` consults `_load_effective_fallback_model()`
— the policy-filtered chain added by #269. That wrapper only reaches the raw
loader when the ambient policy is not "off":

    def _load_effective_fallback_model():
        cfg = _load_cfg()
        policy = get_fallback_policy(cfg)
        if policy == "off":
            return []
        chain = _load_fallback_model() or []

CI has no hermes config, so the policy defaults to "any", the stub is reached,
and these tests are green there. On a host whose config sets
`fallback_policy: off`, the wrapper short-circuits to [], the stub is never
consulted, no fallback is ever attempted, and all five tests fail for a reason
that has nothing to do with what they assert.

Patch what the code actually calls, so the case under test is the one that runs
on any host. `test_auth_error_all_fallbacks_fail_raises` additionally pins
`_load_cfg`, because it asserts the policy-specific exhaustion wording ("no
usable configured backup route remained" is the "any" branch) and would
otherwise assert against whatever policy the host happens to have.

This is not a CI-visible failure — it is a latent host-config dependency that
makes the suite behave differently for a contributor than it does in CI.

TestResolveRuntimeWithFallback: 8 passed against a config with
`fallback_policy: off`, and 8 passed with no hermes config at all. Before this
change the first environment produced 5 failures.

This file's two remaining failures (golden transcript, session activate) are
owned by #296; this change is disjoint from it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@OmarB97
OmarB97 force-pushed the fix/tui-gateway-test-drift branch from 4cd7690 to 3c801ca Compare August 2, 2026 06:15
@OmarB97 OmarB97 changed the title test(gateway): repair TUI-gateway tests that drifted from #269/#270 behavior test(gateway): make the fallback-chain tests independent of host config Aug 2, 2026
@OmarB97
OmarB97 merged commit 389a929 into main Aug 2, 2026
33 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant