fix(discovery): exclude vision-input-only models from orchestrator/free

CHANGELOG.md

@@ -194,6 +194,45 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
   production-default gate returns false on junk reports and on
   `measurement_status=estimated`. Access-list scope is a real ablation
   factor, not a duplicate label.
+- `free_discovered_models()` no longer admits a zero-priced model that
+  declares a non-text input modality (e.g. NVIDIA NIM's
+  `meta/llama-3.2-90b-vision-instruct`, whose Models.dev evidence reports
+  `cost: 0/0` and, misleadingly for this exact deployment, `tool_call: true`)
+  into the general-purpose `orchestrator/free` pool. That pool serves
+  arbitrary request shapes -- including Strix's tool/function-calling
+  requests -- without knowing in advance which capability a request needs, so
+  a free model whose only price-evidenced identity requires an extra input
+  modality is reserved for a caller that explicitly needs it, not a general
+  worker. Fixes the required Strix Security Scan failure reproduced in
+  `ContextualWisdomLab/.github` PR #1198 (run 33325907333, job 99295892400),
+  where every one of 3 independent scan attempts exhausted the pool against
+  this exact agent with an identical HTTP 400 `invalid_request_error`. A
+  model excluded here remains fully discovered and price-evidenced; it is
+  only withheld from the free/tool-calling default pool.
+- Devin's review on PR #933 found the fix above incomplete: `free_discovered_models()`
+  itself is now pure price-based inventory again (every zero-priced model
+  counts, restoring correct `--free-only`, `free_tier_count`, and free-tier
+  data-privacy totals), and a new, separately named selector,
+  `general_free_serving_candidates()`, carries the modality-based exclusion
+  for composing the blind `orchestrator/free` pool specifically. More
+  importantly, that exclusion previously lived only inside the one function
+  this PR touched: `_auto_discover_runtime_agents` (`--auto-discover-model-agents`)
+  and `provider_bootstrap._active_agent_from_discovered` (used by both
+  `bootstrap_provider_runtime` and, through it,
+  `provider_catalog_bootstrap.bootstrap_provider_catalog_runtime`) tag an
+  agent `cost:free` directly from raw price evidence and never consulted it,
+  so the incident model could still reach a live `cost:free` agent through
+  either path. The fix is now a single choke point, `TaskOrchestrator._is_free_agent`,
+  which every `orchestrator/free` selection path shares: an agent whose tags
+  declare a non-text input modality is never treated as free-pool eligible
+  there, regardless of which code built it or how old that agent-pool row is
+  -- while `cost:free` keeps meaning "honest zero price" everywhere else,
+  preserving `provider_catalog_store.py`'s durable `is_free` round trip.
+  Verified against three explicit modality fixtures (text-only, vision-only,
+  and text+image); evaluated and rejected narrowing the exclusion to spare a
+  model that "also declares text" (Devin's other suggestion) because the
+  incident model itself declares both `text` and `image` per Models.dev, so
+  that narrowing would have silently re-admitted it.
 
 ### Changed
 

contextual_orchestrator/__main__.py

@@ -21,6 +21,7 @@
     configured_gateway_source,
     discover_all_models,
     free_discovered_models,
+    general_free_serving_candidates,
     is_routable_discovered_model,
     refresh_price_book,
     select_bootstrap_discovered_agents,
@@ -312,7 +313,13 @@ def _discover_models_command(argv: list[str]) -> None:
     privacy_assessments = []
     if args.analyze_privacy_policies:
         discovered, privacy_assessments = analyze_discovered_privacy_policies(discovered)
+    # free_tier_count and general_free_serving_count are always computed over
+    # the complete `discovered` population, independent of --free-only (which
+    # only filters `reported`, the per-model listing below): both answer a
+    # global "how many, out of everything found" question, matching each
+    # other's population by design rather than "reported"'s row-level filter.
     free_models = free_discovered_models(discovered)
+    general_free_serving_models = general_free_serving_candidates(discovered)
     reported = free_models if args.free_only else discovered
     price_book = PriceBook(InMemoryConfigStore())
     priced_count = refresh_price_book(reported, price_book)
@@ -342,6 +349,7 @@ def _discover_models_command(argv: list[str]) -> None:
     report = {
         "discovered_count": len(reported),
         "free_tier_count": len(free_models),
+        "general_free_serving_count": len(general_free_serving_models),
         "free_data_privacy": {
             status: sum(1 for model in free_models if (
                 "supported" if model.supports_zero_data_retention is True else

contextual_orchestrator/chat_capability.py

@@ -97,6 +97,31 @@ def is_general_chat_agent_model_id(model_id: str) -> bool:
     )
 
 
+def requires_non_text_input(input_modalities: Iterable[str]) -> bool:
+    """Return whether declared input-modality evidence includes non-text input.
+
+    A model whose provider/catalog architecture evidence declares an input
+    modality other than ``"text"`` (e.g. ``image``, ``audio``, ``video``) is a
+    specialized multimodal deployment: a caller cannot use it for an arbitrary
+    request without knowing in advance that the request must carry that extra
+    modality. Absence of modality evidence is not evidence of a multimodal
+    requirement, so an empty ``input_modalities`` iterable never triggers this.
+
+    Shared by ``model_discovery._requires_non_text_input`` (reading
+    ``DiscoveredModel.input_modalities`` directly) and
+    ``orchestrator.TaskOrchestrator._agent_requires_non_text_input`` (reading
+    an agent's persisted ``input:<modality>`` tags, with the ``input:`` prefix
+    already stripped by the caller) so the "what counts as non-text" reading
+    lives in exactly one place -- the two representations of the same catalog
+    evidence can never drift on this question independently of each other.
+    """
+    return any(
+        modality.strip().casefold() != "text"
+        for modality in input_modalities
+        if isinstance(modality, str) and modality.strip()
+    )
+
+
 def is_general_chat_candidate(
     model_id: str,
     *,

contextual_orchestrator/model_discovery.py

@@ -26,7 +26,11 @@
 from typing import TYPE_CHECKING, Any, Literal, Mapping
 from urllib.parse import quote, urlsplit, urlunsplit
 
-from .chat_capability import is_general_chat_agent_model_id, is_general_chat_candidate
+from .chat_capability import (
+    is_general_chat_agent_model_id,
+    is_general_chat_candidate,
+    requires_non_text_input,
+)
 from .credentials import get_credential
 from .orchestrator import ModelAgent, ModelClient
 
@@ -1304,11 +1308,105 @@ def agent_from_discovered(discovered: DiscoveredModel, *, priority: int = 0) ->
     )
 
 
+def _requires_non_text_input(discovered: DiscoveredModel) -> bool:
+    """Return whether catalog evidence shows this model needs non-text input.
+
+    A model whose provider/catalog architecture evidence declares an input
+    modality other than ``"text"`` (e.g. ``image``, ``audio``, ``video``) is a
+    specialized multimodal deployment: a caller cannot use it for an arbitrary
+    request without knowing in advance that the request must carry that extra
+    modality. Absence of modality evidence is not evidence of a multimodal
+    requirement, so an empty ``input_modalities`` tuple never triggers this.
+
+    This is a deliberately conservative reading: catalog fields such as
+    Models.dev's ``modalities.input`` document *supported* inputs, not which
+    ones a given request must supply, so a model that lists ``text`` next to
+    ``image`` still trips this check. ContextualWisdomLab/.github#1198's
+    incident model (NVIDIA NIM's ``meta/llama-3.2-90b-vision-instruct``) is
+    exactly that shape -- Models.dev reports its inputs as ``text`` *and*
+    ``image`` -- yet NIM's live deployment rejected a plain tool-calling
+    request against it three times in a row. With no reliable per-deployment
+    tool-calling signal available (see :func:`general_free_serving_candidates`
+    for the incident writeup), treating "declares any non-text input" as
+    disqualifying for *blind* serving is the only evidence-based reading that
+    actually keeps that incident fixed; a model believed to also serve plain
+    text requests just fine can still be reached through a pool that is not
+    modality-blind (see :func:`general_free_serving_candidates`'s docstring).
+
+    Delegates the actual classification to
+    ``chat_capability.requires_non_text_input``, the single evidence-based
+    rule shared with ``orchestrator.TaskOrchestrator._agent_requires_non_text_input``
+    (which reads an agent's persisted ``input:<modality>`` tags instead of
+    ``DiscoveredModel`` directly) so the two representations of the same
+    catalog evidence cannot drift on this question independently of each
+    other.
+    """
+    return requires_non_text_input(discovered.input_modalities)
+
+
 def free_discovered_models(discovered: list[DiscoveredModel]) -> list[DiscoveredModel]:
-    """Return models whose provider metadata identifies zero-cost inference."""
+    """Return the complete zero-cost model inventory (price evidence only).
+
+    This is pure price-based inventory: every model whose structured
+    provider/catalog pricing evidence is entirely zero, regardless of input
+    or output modality. Reporting surfaces that answer "is this model free"
+    -- the ``discover-models`` CLI's ``--free-only`` report, ``free_tier_count``,
+    and the free-tier data-privacy totals -- need this complete inventory, not
+    a servable subset.
+
+    Fitness for the general-purpose *blind* serving pool (``orchestrator/free``)
+    is a stricter, separate question: see :func:`general_free_serving_candidates`.
+    An earlier revision of this function conflated the two, which silently
+    undercounted genuinely free models that are simply unsuited to
+    capability-blind serving in every "is this model free" report.
+    """
     return [model for model in discovered if model.is_free]
 
 
+def general_free_serving_candidates(
+    discovered: list[DiscoveredModel],
+) -> list[DiscoveredModel]:
+    """Return free models fit for the general-purpose blind serving pool.
+
+    A zero price alone does not certify fitness for arbitrary callers: the
+    free pool (``orchestrator/free``) serves every role and request shape --
+    including tool/function-calling requests -- without knowing in advance
+    which capability a given request will need. Provider pricing can be
+    reliably zero on a model that only a caller who already knows to supply
+    an extra input modality (e.g. an image) could ever use meaningfully;
+    :func:`_requires_non_text_input` excludes exactly those rows here, using
+    catalog evidence discovery already records, not a per-model name rule.
+    Such a model remains fully discovered, fully counted in
+    :func:`free_discovered_models`'s price-based inventory, and eligible for
+    a pool that is not modality-blind (e.g. one built for vision/multimodal
+    tasks) -- it is only withheld from *this* general-purpose free selector.
+
+    Reproduces ContextualWisdomLab/.github#1198's required Strix Security Scan
+    failure (run 33325907333, job 99295892400): NVIDIA NIM's free
+    ``meta/llama-3.2-90b-vision-instruct`` passed every existing chat-capability
+    check, yet NIM's live deployment rejected Strix's tool-calling request
+    against it with a definitive HTTP 400 three independent times in a row --
+    because the free pool had no other candidate to fail over to, this one
+    vision-input model alone exhausted the whole tool-calling pool.
+
+    This is the selector every runtime pool-construction path must apply
+    before treating a discovered model as eligible for blind free serving
+    (e.g. tagging an agent ``cost:free`` in a context where that tag alone
+    drives general-chat ``orchestrator/free`` routing).
+    ``TaskOrchestrator._is_general_free_agent`` additionally re-checks an
+    agent's persisted ``input:<modality>`` tags at selection time -- but only
+    for the capability-blind general chat pool, never for a capability-scoped
+    free route (``_capability_agents``), where that same tag is the expected
+    shape, not a surprise -- so a durable agent-pool row written by an older
+    build, before this exclusion existed, cannot bypass it either.
+    """
+    return [
+        model
+        for model in free_discovered_models(discovered)
+        if not _requires_non_text_input(model)
+    ]
+
+
 def _currency_is_comparable(currency_code: object, default_currency: object) -> bool:
     """Return whether two ISO-style currency codes can be compared directly."""
     return (