feat(auxiliary): add xAI backend for text and vision tasks

[Julientalbot]

Problem

The auxiliary client resolution chain (compression, session_search, vision, web_extract, etc.) had no direct xAI support. Users with XAI_API_KEY had to rely on either the main provider fallback (only if they were also on xAI as main) or routing through OpenRouter, which defeats the purpose of using xAI direct for cheap/fast side tasks.

In particular, _VISION_AUTO_PROVIDER_ORDER was limited to ("openrouter", "nous"). Users with both XAI_API_KEY and OPENROUTER_API_KEY saw every vision analysis routed through OpenRouter even when their main chat provider was xai — burning OpenRouter credits and adding latency while ignoring Grok 4's native vision support.

Concrete scenario: on my instance, every invocation of the vision tool in my content-machine-daily cron (which analyses candidate photos for LinkedIn posts) was going through OpenRouter → gemini-3-flash-preview instead of staying on xAI, even though Grok 4.20 supports vision natively and the main agent was already on xai.

Solution

Add xAI as a first-class auxiliary backend, following the exact pattern used by the existing _try_openrouter / _try_nous helpers, with a text/vision default-model split.

Changes to agent/auxiliary_client.py

1. New _try_xai(vision: bool = False) helper — reads the credential pool first, then falls back to XAI_API_KEY env var. The vision flag selects a different default model (same pattern as _try_nous(vision=True)).

2. Text vs vision default models:

   • _XAI_TEXT_AUX_MODEL = "grok-4-1-fast-non-reasoning" — fast, cheap (~$0.20/$0.50 per M), no reasoning overhead. Used for compression, session_search, skills_hub, web_extract, etc.
   • _XAI_VISION_MODEL = "grok-4.20-0309-non-reasoning" — the larger/smarter model, used for vision tasks where nuanced image judgments matter (e.g. "is this photo suitable for a professional LinkedIn post?"). Stays on the non-reasoning variant to keep per-call latency and cost sensible — reasoning adds 15-30s and significant output tokens for marginal gains on bounded vision tasks.

3. _XAI_DEFAULT_BASE_URL constant next to the existing _NOUS_* / _ANTHROPIC_* constants.

4. _PROVIDER_ALIASES gains x-ai and x.ai → xai so users can spell the provider any of the three common ways (consistent with the existing z-ai / z.ai → zai aliases).

5. _API_KEY_PROVIDER_AUX_MODELS["xai"] = "grok-4-1-fast-non-reasoning" so the api-key provider fallback path picks the text model as its default. Vision tasks route through _try_xai(vision=True) and pick up grok-4.20 instead.

6. resolve_provider_client("xai") gets an explicit case so text auxiliary tasks with provider: xai work standalone (no dependency on the PROVIDER_REGISTRY lookup — feat(providers): add native xAI provider #7050 adds the registry entry too, but this PR stands alone).

7. _VISION_AUTO_PROVIDER_ORDER gains "xai" at the end so users without OpenRouter or Nous get xAI vision automatically. Back-compat preserved: users with OPENROUTER_API_KEY keep the current behavior.

8. _resolve_strict_vision_backend("xai") gets the matching case and passes vision=True so explicit auxiliary.vision.provider: xai in config.yaml resolves to the xAI backend with the grok-4.20 default (no silent OpenRouter fallback when both are available).

Usage after this change

Zero-config path (works automatically for users on xAI direct without OpenRouter/Nous):

auxiliary:
  vision:
    provider: auto   # will now include xAI in the chain

Explicit path (force xAI vision even when OpenRouter is present):

auxiliary:
  vision:
    provider: xai
    # model: grok-4.20-0309-non-reasoning   # optional, this is the default

Testing

4 new tests in tests/agent/test_auxiliary_client.py:

• test_explicit_xai — resolve_provider_client("xai") picks up XAI_API_KEY and returns the default text aux model.
• test_explicit_xai_alias_x_ai — the x-ai alias normalises to xai.
• test_vision_uses_xai_when_no_openrouter_no_nous — vision auto chain falls back to xAI when nothing else is available, returning the grok-4.20 vision default.
• test_vision_forced_xai_uses_xai_backend — explicit config override routes vision to xAI even when OPENROUTER_API_KEY is set. Verifies the api.x.ai base URL is used, not OpenRouter, and the grok-4.20 model is selected.

Also hardens test_codex_fallback_when_nothing_else and test_forced_main_falls_to_codex by explicitly clearing XAI_API_KEY and other api-key env vars — they were silently depending on the developer shell having no api-key providers set.

$ pytest tests/agent/test_auxiliary_client.py
100 passed in 6.86s

Impact

• Back-compat: existing users with OPENROUTER_API_KEY see no behavior change
• New capability: users on xAI direct get a proper first-class auxiliary backend with a smart text/vision split
• Cost reduction for users configured to prefer xAI — vision calls stop leaking to OpenRouter
• Vision quality: grok-4.20 (rather than the cheap fast variant) for nuanced image judgments
• Complements the existing xAI integration stack:
  • x-grok-conv-id prompt caching in run_agent.py
  • "grok" in TOOL_USE_ENFORCEMENT_MODELS in agent/prompt_builder.py
  • Native xAI provider registration in feat(providers): add native xAI provider #7050
  • DEFAULT_CONTEXT_LENGTHS fallbacks in fix(model_metadata): add xAI Grok context length fallbacks #7039

This PR is self-contained and does not depend on #7039 or #7050, though it composes cleanly with them.

[Julientalbot]

Status update (the linked PRs at the bottom have since resolved): #7039 / #7050 were closed; the xAI provider registration they referenced landed via #10783 (merged 16/04). This PR remains self-contained — it only touches agent/auxiliary_client.py and its tests.

Verified still applicable against current main (just rechecked): _VISION_AUTO_PROVIDER_ORDER at agent/auxiliary_client.py:1776 still only contains ("openrouter", "nous"). Vision auto still skips xAI even after #10783 landed, so users on xAI direct (without OpenRouter/Nous) still cannot get auxiliary vision, and users with both keys still silently route through OpenRouter. The concrete scenario in the PR description (photo-analysis cron on my instance) is still broken today.

Happy to rebase on current main if a reviewer has capacity to look.

[Julientalbot]

Closing — conflicting against main for 10+ days and the auxiliary client has since been restructured upstream. Not worth rebasing 123 LOC against a moving target without review signal.