[lukmanc405]

Problem

When Auto-hide failed models is enabled (it defaults to true) and a user runs Test All Models, every non-OK result is hidden — including transient failures like rate-limits and timeouts.

On subscription-tier providers (Kiro, etc.) a single Test All sweep fires requests at 10+ models in parallel, which reliably trips the provider's per-account rate limit. The models that get 429'd or time out are then auto-hidden even though they work perfectly on the next request.

This is more than a cosmetic glitch:

• Hidden state is persisted (modelCompatOverrides / custom-model isHidden) and survives server restarts.
• A hidden model is dropped from GET /v1/models, so OpenAI/Anthropic-compatible clients (and the catalog used by external tools) stop seeing it entirely.
• There is no batch recovery in the UI — the user has to hand-toggle the eye icon on each row, or edit the DB, to bring the models back.

Net effect: one throttled Test All can silently wipe an entire provider's catalog, and the user has no obvious way to tell why their models "disappeared."

Root cause

evaluateTestAllEntry() in providerPageHelpers.ts hid any non-OK entry when the toggle was on:

shouldHide: !ok && autoHideFailed,

The /api/models/test-all response already distinguishes transient failures via the typed rateLimited and isTimeout flags, but they weren't consulted.

Fix

Skip auto-hide for entries flagged rateLimited or isTimeout. They still surface as error on the per-row icon (so the user sees the test didn't pass), but the model stays visible. Genuine, persistent failures (upstream 400 "invalid model", schema mismatch, auth failure) are still auto-hidden — the intended use of the toggle.

const transient = Boolean(entry?.rateLimited || entry?.isTimeout);
return {
  status: ok ? "ok" : "error",
  shouldHide: !ok && autoHideFailed && !transient,
};

Tests

Updated the existing regression suite tests/unit/test-all-model-status.test.ts to assert the new policy (transient failures show error but are not hidden). All 4 cases pass:

✔ ok entry maps to status 'ok' and is never hidden
✔ failed entry maps to status 'error' and hides only when autoHide is on
✔ rate-limited / timeout failures show 'error' but are NOT auto-hidden
✔ missing / null / empty entry is treated as a failure

Repro (before this fix)

1. Connect a subscription-tier provider (e.g. Kiro) with 10+ models.
2. Leave Auto-hide failed models on (default).
3. Click Test All Models. The parallel sweep trips the provider's rate limit; several models 429 / time out.
4. Those models are auto-hidden and vanish from GET /v1/models. Clients can no longer select them until each row is manually un-hidden.

[gemini-code-assist]

Code Review

This pull request updates the auto-hide policy for batch model testing to prevent transient failures (like rate limits or timeouts) from triggering auto-hiding. However, the review identifies a critical bug where skipped or untested models (where the test entry is null or undefined) are also treated as persistent failures and silently auto-hidden. The reviewer suggests explicitly checking for a confirmed 'error' status before hiding a model, and updating the corresponding unit tests to assert that missing or empty entries are not hidden.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Critical Bug: Skipped or Untested Models are Silently Auto-Hidden

When a batch test is run, there are two scenarios where a model's test entry (entry) can be undefined or null:

1. Early Stopping: The batch test runner in src/app/api/models/test-all/route.ts stops early if it encounters consecutive rate limits (CONSECUTIVE_RATE_LIMIT_STOP_THRESHOLD). Any remaining models in the list are skipped and will not have an entry in the response.
2. API/Network Failures: If the /api/models/test-all request returns a non-200 response (e.g., a 500 error or validation error) but succeeds at the HTTP/JSON level, result.results will be undefined, meaning entry is undefined for all models.

Under the current implementation, if entry is undefined or null, ok is evaluated as false, and transient is false. If autoHideFailed is enabled, shouldHide evaluates to true. This causes the client to silently auto-hide all skipped or untested models, permanently removing them from the catalog and /v1/models!

To prevent this, we should only auto-hide a model if we have an explicit, confirmed failure status (status: "error") from the test runner.

  const ok = entry?.status === "ok";
  const isError = entry?.status === "error";
  const transient = Boolean(entry?.rateLimited || entry?.isTimeout);
  return {
    status: ok ? "ok" : "error",
    // Hide only persistent failures. Transient (rate-limited, timeout) are
    // surfaced on the icon but kept visible so a single throttled batch test
    // does not silently wipe the catalog. Untested/skipped models (where entry is
    // undefined/null due to early stop) must not be hidden.
    shouldHide: isError && autoHideFailed && !transient,
  };

[gemini-code-assist]

Update Test for Missing/Null/Empty Entries

Since we should only auto-hide models with an explicit "error" status (to prevent skipped/untested models from being hidden when a batch test stops early), the test "missing / null / empty entry is treated as a failure" at the bottom of this file (lines 57-63) needs to be updated to assert shouldHide: false.

Currently, that test is outside the modified diff hunk, so we cannot provide a direct code suggestion for it, but please update it manually to:

test("missing / null / empty entry is treated as a failure but NOT hidden", () => {
  for (const entry of [undefined, null, {}]) {
    const out = evaluateTestAllEntry(entry, true);
    assert.equal(out.status, "error", `entry=${JSON.stringify(entry)} → error`);
    assert.equal(out.shouldHide, false);
  }
});

[diegosouzapw]

Merged into release/v3.8.25 🎉 — thanks @lukmanc405! Great reasoning — auto-hiding transient (rate-limit/timeout) failures from a parallel Test All silently wiped working models from /v1/models with no easy recovery. Surfacing them as 'error' but keeping them visible (only hiding genuine failures) is the right fix. Validated: 4/4. Ships in v3.8.25. 🙌