fix(vision): forward custom-endpoint credentials + read aux model from config.yaml - #57651
Merged
Conversation
alt-glitch
added
type/bug
Collaborator
Core-team salvage (@teknium1) folding #50685 (@suninrain086) and #53754 (@liuhao1024) with authorship preserved. Supersedes the sibling-gap cluster #43254/#43251 and overlaps #57286/#43297 (runtime-first approach preferred over #43297's config-read). Related, not duplicate — this is the canonical version; the salvaged/competing PRs (#50685, #53754, #43254, #57286, #43297) can be closed in its favor by a maintainer. |
A custom:<name> main provider resolves at runtime to the bare provider id
"custom". In the vision auto-detect chain, the main-provider branch called
resolve_provider_client("custom", ...) WITHOUT explicit_base_url/api_key,
so it returned (None, None) ("no endpoint credentials found") and the whole
chain fell through to OpenRouter/Nous. A user on a custom endpoint with no
aggregator configured then got "No LLM provider configured for task=vision
provider=auto" on every image, even though their main model fully supports
vision.
Recover the live endpoint that set_runtime_main() records each turn
(_RUNTIME_MAIN_BASE_URL/_API_KEY/_API_MODE) and forward it to Step 1, with
a fallback to _resolve_custom_runtime() for non-gateway callers. Mirrors the
existing explicit-base_url branch directly above.
Adds TestResolveVisionCustomProvider covering custom, custom:<name>, and the
no-runtime fallback path.
_handlers for vision_analyze and video_analyze read model name from config.yaml (auxiliary.vision.model / auxiliary.video.model) before falling back to AUXILIARY_VISION_MODEL / AUXILIARY_VIDEO_MODEL env vars. Matches the existing config-first pattern for timeout and temperature in the same file. Fixes #53749
…ision_analyze The salvaged tests from #53754 predate _handle_vision_analyze becoming async and the native fast path; await the handler and force the legacy aux path so the model-resolution assertion is actually exercised.
teknium1
force-pushed
the
fix/aux-vision-custom-creds
branch
from
July 3, 2026 10:44
400e6d4 to
e8ec25d
Compare
This was referenced Jul 3, 2026
This was referenced Jul 9, 2026
This was referenced Jul 29, 2026
Open
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Auxiliary vision credential/config resolution fixed for custom-endpoint users: the vision auto-detect chain now forwards the live custom endpoint credentials instead of returning "No LLM provider configured", and
vision_analyze/video_analyzeread the aux model fromconfig.yamlbefore the legacy env var. Fixes #50685's premise and #53749.Root causes:
resolve_vision_provider_client()auto-branch calledresolve_provider_client("custom", ...)withoutexplicit_base_url/explicit_api_key; the genericcustomid has no built-in endpoint, so resolution fell through to aggregators and failed for custom-main users with no aggregator keys._handle_vision_analyze/_handle_video_analyzeread onlyAUXILIARY_VISION_MODEL/AUXILIARY_VIDEO_MODELenv vars, ignoringauxiliary.vision.model/auxiliary.video.modelin config.yaml.Changes
_RUNTIME_MAIN_BASE_URL/_API_KEY/_API_MODE, falling back to_resolve_custom_runtime()) and forward it to Step 1 of vision auto-detect when main provider iscustom/custom:<name>— cherry-picked from fix(vision): forward custom-endpoint credentials in vision auto-detect #50685 (@suninrain086)auxiliary.vision.model(andauxiliary.video.model→ vision fallback) from config.yaml; env vars remain a legacy override — cherry-picked from fix(vision): read auxiliary model from config.yaml before env var #53754 (@liuhao1024)_handle_vision_analyzeand forced the legacy path past the native fast-path gateValidation
Closes #53749. Salvages #50685 and #53754 with contributor authorship preserved. (#57286 and #43297 overlap this area — reviewed; #50685's runtime-first approach supersedes #43297's config-read approach, and #57286's provider/base_url/api_key forwarding is already covered by the call_llm task-config resolution once the custom-endpoint chain works.)
Infographic