Skip to content

feat: add image generation fallback controls - #32320

Open
y0shua1ee wants to merge 1 commit into
NousResearch:mainfrom
y0shua1ee:feat/image-gen-fallback-model-command
Open

feat: add image generation fallback controls#32320
y0shua1ee wants to merge 1 commit into
NousResearch:mainfrom
y0shua1ee:feat/image-gen-fallback-model-command

Conversation

@y0shua1ee

@y0shua1ee y0shua1ee commented May 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds first-class image generation fallback controls:

  • image_gen.fallback_providers support for image provider fallback.
  • /image_model CLI/gateway command for switching image provider/model.
  • Explicit fallback model override propagation through OpenAI, OpenAI-Codex, xAI, and FAL providers.
  • Retry gating so fallback only triggers for retryable provider failures.
  • Aggregated provider errors with sensitive details redacted.

This implements the image generation provider-fallback slice of the broader tool-level fallback request. The wider issue also discusses vision fallback, script fallback, and other tool categories, so this PR links it as a partial implementation.

Closest related work checked:

Related Issue

Partially addresses #30455 for image_gen provider fallback and image model switching.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/image_generation_tool.py
    • Adds image generation fallback-provider resolution and retry orchestration.
    • Restricts fallback retries to retryable provider failures.
    • Redacts aggregated provider errors before returning them to the agent/user.
  • hermes_cli/image_model_switch.py
    • Adds image provider/model switch support used by CLI and gateway command paths.
  • cli.py, gateway/run.py, hermes_cli/commands.py
    • Wires /image_model into command dispatch.
  • plugins/image_gen/fal/__init__.py
  • plugins/image_gen/openai-codex/__init__.py
  • plugins/image_gen/openai/__init__.py
  • plugins/image_gen/xai/__init__.py
    • Propagates explicit fallback model overrides through each provider.
  • Tests added/updated:
    • tests/hermes_cli/test_image_model_switch.py
    • tests/gateway/test_image_model_command.py
    • tests/tools/test_image_generation.py
    • tests/tools/test_image_generation_plugin_dispatch.py
    • tests/plugins/image_gen/test_fal_provider.py
    • tests/plugins/image_gen/test_openai_provider.py
    • tests/plugins/image_gen/test_xai_provider.py
    • tests/plugins/image_gen/test_openai_codex_provider.py

How to Test

  1. Run formatting/diff checks:
git diff --cached --check
  1. Run the targeted image fallback and command test suite:
uv run pytest -q tests/hermes_cli/test_image_model_switch.py \
                 tests/gateway/test_image_model_command.py \
                 tests/tools/test_image_generation.py \
                 tests/tools/test_image_generation_plugin_dispatch.py \
                 tests/plugins/image_gen/test_fal_provider.py \
                 tests/plugins/image_gen/test_openai_provider.py \
                 tests/plugins/image_gen/test_xai_provider.py \
                 tests/plugins/image_gen/test_openai_codex_provider.py \
                 tests/gateway/test_command_bypass_active_session.py

Expected result from the branch:

196 passed
  1. Run lint on the touched implementation and test files:
uv run ruff check cli.py gateway/run.py hermes_cli/commands.py \
                  hermes_cli/image_model_switch.py tools/image_generation_tool.py \
                  plugins/image_gen/fal/__init__.py \
                  plugins/image_gen/openai/__init__.py \
                  plugins/image_gen/xai/__init__.py \
                  plugins/image_gen/openai-codex/__init__.py \
                  tests/gateway/test_image_model_command.py \
                  tests/hermes_cli/test_image_model_switch.py \
                  tests/tools/test_image_generation.py \
                  tests/tools/test_image_generation_plugin_dispatch.py \
                  tests/plugins/image_gen/test_fal_provider.py \
                  tests/plugins/image_gen/test_openai_provider.py \
                  tests/plugins/image_gen/test_xai_provider.py \
                  tests/plugins/image_gen/test_openai_codex_provider.py

Expected result:

All checks passed

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS with the targeted tests above; GitHub CI is green

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Targeted tests and lint:

$ uv run pytest -q tests/hermes_cli/test_image_model_switch.py tests/gateway/test_image_model_command.py tests/tools/test_image_generation.py tests/tools/test_image_generation_plugin_dispatch.py tests/plugins/image_gen/test_fal_provider.py tests/plugins/image_gen/test_openai_provider.py tests/plugins/image_gen/test_xai_provider.py tests/plugins/image_gen/test_openai_codex_provider.py tests/gateway/test_command_bypass_active_session.py
196 passed

$ uv run ruff check cli.py gateway/run.py hermes_cli/commands.py hermes_cli/image_model_switch.py tools/image_generation_tool.py plugins/image_gen/fal/__init__.py plugins/image_gen/openai/__init__.py plugins/image_gen/xai/__init__.py plugins/image_gen/openai-codex/__init__.py tests/gateway/test_image_model_command.py tests/hermes_cli/test_image_model_switch.py tests/tools/test_image_generation.py tests/tools/test_image_generation_plugin_dispatch.py tests/plugins/image_gen/test_fal_provider.py tests/plugins/image_gen/test_openai_provider.py tests/plugins/image_gen/test_xai_provider.py tests/plugins/image_gen/test_openai_codex_provider.py
All checks passed

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation labels May 26, 2026
@y0shua1ee
y0shua1ee force-pushed the feat/image-gen-fallback-model-command branch 2 times, most recently from f07ca9e to 2509142 Compare May 29, 2026 02:47
@y0shua1ee

Copy link
Copy Markdown
Contributor Author

Rebased this PR onto current main and resolved the command-registry conflict from the latest Slack slash-command changes.

Current status after the refresh:

  • Head: 250914261
  • Merge state: CLEAN
  • CI: green (18 successful checks, 2 skipped)
  • Local verification:
    • ruff check on touched files passed
    • tests/hermes_cli/test_commands.py: 144 passed
    • targeted image fallback/model-command suite: 196 passed

@y0shua1ee
y0shua1ee force-pushed the feat/image-gen-fallback-model-command branch from 2509142 to c59487f Compare June 4, 2026 13:39
Add image_gen.fallback_providers dispatch with retryable-failure gating, provider/model override propagation, and /image_model switching for CLI and gateway.

Covers fallback routing, provider model resolution, command handling, managed config behavior, and active-session bypass tests.
@y0shua1ee
y0shua1ee force-pushed the feat/image-gen-fallback-model-command branch from c59487f to afe3777 Compare June 4, 2026 13:47

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused fallback routing and retry-gating work. The feature premise still holds on current main: tools/image_generation_tool.py:1295-1401 dispatches to one configured image provider and returns that result without an image-provider fallback chain.

Problems

  • The stale dispatcher replacement loses the source-image parameters added by c02192ff6 (tools/image_generation_tool.py:1274-1279, 1336-1349). In this PR, tools/image_generation_tool.py:1116 accepts only prompt/aspect ratio and line 1198 sends only those plus model, so image-to-image calls cannot reach any fallback provider.
  • The same regression exists in the handler: PR tools/image_generation_tool.py:1269-1283 does not read or forward image_url / reference_image_urls, unlike current main at tools/image_generation_tool.py:1518-1551. The added fallback tests exercise only text-to-image calls (tests/tools/test_image_generation_plugin_dispatch.py:154).

Suggested changes

  • Salvage the fallback loop onto the current image-to-image dispatcher and carry source-image inputs through every attempt; add primary-failure/fallback-success coverage with both image input fields.
  • Reconcile command wiring with the current gateway command surface and preserve the existing hermes tools picker.

Automated hermes-sweeper review.

return any(marker in error for marker in _RETRYABLE_IMAGE_ERROR_MARKERS)


def _dispatch_to_plugin_provider(prompt: str, aspect_ratio: str):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main's dispatcher accepts and forwards image_url and reference_image_urls for image-to-image calls (tools/image_generation_tool.py:1274-1279, 1336-1349). Please retain those parameters here and pass them to every fallback attempt; otherwise this stale replacement drops source images before either the primary or fallback provider sees them.

# Route to a plugin-registered provider if one is active (and it's
# not the in-tree FAL path).
# Route to a plugin-registered provider if one is active.
dispatched = _dispatch_to_plugin_provider(prompt, aspect_ratio)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handler now calls the dispatcher with only prompt and aspect ratio. Current main reads image_url and reference_image_urls and forwards both through plugin, managed-Krea, and FAL routes (tools/image_generation_tool.py:1518-1551). Preserve that forwarding and add a fallback test that asserts the backup provider receives the source-image inputs.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants