From 90c568927f3699d29a876fa46d528f2c332c1746 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 03:13:40 +0000 Subject: [PATCH] test(discovery): fix stale test left behind by PR #941's nvidia_nim/nvidia_nim_sub independence fix PR #941 (merged) removed model_discovery._provider_family and its nvidia_nim/nvidia_nim_sub collapsing, correctly updating every test that referenced _provider_family by name. tests/test_discovery_bootstrap_selection.py was missed because its test asserts the collapsed behavior by outcome, not by name: test_bootstrap_selector_treats_nim_primary_and_sub_as_one_outage_domain still asserted select_bootstrap_discovered_agents(...) == [nim_primary, openrouter], i.e. that nim_sub gets displaced by an unrelated provider to preserve "family" diversity -- the exact pool-shrinking result #941 was meant to eliminate. This left the test failing on main itself (verified directly against commit 42da1d53, the #941 merge commit, in an isolated worktree) and therefore failing the full-suite check on every open PR that merges/rebases onto current main. Root of the original premise: nvidia_nim and nvidia_nim_sub were collapsed into one "outage domain" on the unverified assumption that both KV credentials (NVIDIA_NIM_API_KEY / NVIDIA_NIM_API_KEY_SUB) expose the identical NVIDIA NIM model catalog. That was never checked against live evidence -- two different API keys hitting the same /v1/models URL does not guarantee the same result set -- and docs/planning/adrs/0015 already recorded the opposite: "NVIDIA primary and secondary keys are independent provider accounts." #941 independently reached the same conclusion and fixed the production code and five other test files; this fixes the sixth. Renamed to test_bootstrap_selector_keeps_nim_primary_and_sub_independent and corrected the assertion to [nim_primary, nim_sub], matching select_bootstrap_discovered_agents' actual behavior since #941 merged. Verify-revert checked: reverting this test to main's current content reproduces the exact failure; the fix passes. Full suite on this worktree (origin/main @ 42da1d53 + this one-file change): 2817 passed, 1 skipped, 1 failed (tests/test_psychometric_routing.py's fast_mlsirm ModuleNotFoundError -- a pre-existing, unrelated environment gap, not caused by this change). interrogate: 100%. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw --- docs/product-technical-gap-baseline.md | 27 ++++++++++++++++++++- tests/test_discovery_bootstrap_selection.py | 9 ++++--- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index e10aad199..584ece4ff 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -1,6 +1,31 @@ # Contextual Orchestrator: Product & Technical Gap Baseline -## 2026-08-30 provider-catalog-sync: no scheduled run has succeeded in 5 days over one provider; workflow check was too strict +## 2026-08-31 PR #941 left one stale test asserting the removed nvidia_nim/nvidia_nim_sub collapse + +Independently investigated the same premise PR #941 ("fix(discovery): keep credential accounts +independent") corrected: `model_discovery._provider_family` collapsed `nvidia_nim`/`nvidia_nim_sub` +into one family on an unverified assumption that the two KV credentials expose the same NVIDIA NIM +model catalog. `docs/planning/adrs/0015-durable-provider-catalog.md` (accepted 2026-08-22) already +recorded the opposite — "NVIDIA primary and secondary keys are independent provider accounts" — and +nothing in this repo (ADRs, doctoring notes, commit history back to the function's introduction at +`03667a3a`) ever recorded an actual comparison of what each credential's `/v1/models` call returns. +PR #941 merged first with the equivalent fix (deletes `_provider_family` outright rather than keeping +it as an identity seam; both approaches are behaviorally the same: every KV credential is now its own +independent provider/family for pool-diversity selection and outage-tolerance bounding) and updated +every test file its own `_provider_family` grep found. + +One test file was missed because it asserts the collapsed behavior by outcome, not by referencing +`_provider_family` by name: `tests/test_discovery_bootstrap_selection.py::test_bootstrap_selector_treats_nim_primary_and_sub_as_one_outage_domain` +still asserted `select_bootstrap_discovered_agents([nim_sub, openrouter, nim_primary], price_book, 2) +== [nim_primary, openrouter]` — the exact pool-shrinking result the fix was meant to eliminate. +Reproduced failing on a clean checkout of `main` at `42da1d53` (the #941 merge commit itself, not an +artifact of any local branch state): `AssertionError: ... nvidia_nim_sub ... != ... openrouter ...`. +Renamed to `test_bootstrap_selector_keeps_nim_primary_and_sub_independent` and corrected to assert +`[nim_primary, nim_sub]`, matching the behavior `select_bootstrap_discovered_agents` has had since +#941 merged. A full-suite run on this same commit plus only this one-line test fix was clean (see +this entry's PR for exact counts). No other file was affected; `_provider_family` no longer exists +anywhere in the source tree per #941, and a full-repo grep confirms no other test still references the +removed collapsing behavior. `provider-catalog-sync.yml` (run `33312773022`, job `99260685380`) failed with `credential inventory mismatch: ['BYTEZ_API_KEY']`. Traced to `bootstrap_provider_catalog_runtime` diff --git a/tests/test_discovery_bootstrap_selection.py b/tests/test_discovery_bootstrap_selection.py index a6124fee3..cb08b5e08 100644 --- a/tests/test_discovery_bootstrap_selection.py +++ b/tests/test_discovery_bootstrap_selection.py @@ -323,8 +323,11 @@ def test_bootstrap_selector_prefers_provider_diversity_before_duplicates() -> No assert selected == [router_cheapest, nim_model, openai_model] -def test_bootstrap_selector_treats_nim_primary_and_sub_as_one_outage_domain() -> None: - """Two NIM keys must not displace an independently hosted provider.""" +def test_bootstrap_selector_keeps_nim_primary_and_sub_independent() -> None: + """A shared vendor endpoint does not collapse independent credential + accounts: nim_primary and nim_sub each win their own diversity slot ahead + of the unrelated, costlier openrouter candidate. + """ selector = getattr( model_discovery, "select_bootstrap_discovered_agents", @@ -346,7 +349,7 @@ def test_bootstrap_selector_treats_nim_primary_and_sub_as_one_outage_domain() -> 2, ) - assert selected == [nim_primary, openrouter] + assert selected == [nim_primary, nim_sub] def test_bootstrap_selector_is_deterministic_when_every_model_is_unpriced() -> None: