Skip to content

feat: add xAI image edit and refresh Imagine provider - #41356

Closed
jovijovi wants to merge 1 commit into
NousResearch:mainfrom
jovijovi:feat/xai-image-edit-v016
Closed

jovijovi wants to merge 1 commit into
NousResearch:mainfrom
jovijovi:feat/xai-image-edit-v016

Conversation

@jovijovi

@jovijovi jovijovi commented Jun 7, 2026

Copy link
Copy Markdown

Summary

Updates the existing xAI image PR into a current upstream-clean patch on top of NousResearch/hermes-agent:main.

This PR now adds:

  • a dedicated image_edit tool surface for edit-capable image providers;
  • optional provider-level supports_edit() / edit(...) contract with unsupported_capability fallback for generate-only providers;
  • xAI Grok Imagine refresh aligned with the current Imagine API:
    • default model: grok-imagine-image-quality;
    • safe fallback for deprecated/unknown model ids;
    • official xAI aspect-ratio wire values plus Hermes aliases;
    • resolution honored for both generation and edit paths;
    • response_format: b64_json to avoid depending on ephemeral imgen.x.ai URLs;
    • single-image edit, xAI Files file_id, multi-image edit up to 3 inputs;
    • optional direct-provider n and storage_options support with privacy-default validation;
    • cached outputs plus metadata echo for debugging.

Compatibility notes:

  • Preserves the current upstream image_generate(image_url=..., reference_image_urls=...) contract by routing xAI image inputs through the same edit implementation.
  • image_edit is exposed in the image_gen toolset and gated at runtime by provider support.
  • Non-edit providers keep the default unsupported-capability response; they are not forced to implement editing.

Verification

Hermes local gates on the rebuilt branch:

uv run --frozen pytest tests/plugins/image_gen/test_xai_provider.py tests/tools/test_image_edit_tool.py -q -o 'addopts='
# 100 passed

uv run --frozen pytest \
  tests/agent/test_image_gen_provider_edit.py \
  tests/plugins/image_gen/test_xai_provider.py \
  tests/tools/test_image_edit_tool.py \
  tests/tools/test_image_generation_plugin_dispatch.py \
  tests/test_toolsets.py \
  tests/tools/test_image_generation_image_to_image.py \
  -q -o 'addopts='
# 165 passed

python3 -m py_compile \
  agent/image_gen_provider.py \
  plugins/image_gen/xai/__init__.py \
  tests/agent/test_image_gen_provider_edit.py \
  tests/plugins/image_gen/test_xai_provider.py \
  tests/tools/test_image_edit_tool.py \
  tools/image_edit_tool.py \
  toolsets.py
# OK

git diff --cached --check
# OK before commit

Independent read-only review:

  • Codex blocker review: PASS
  • Previous blocker found and fixed: edit successes now return modality="image"; direct edit and image_generate(image_url=...) have regression coverage.

Scope hygiene

This update intentionally excludes downstream-only Sachima roadmap/dev-log/runtime-rollout evidence, local media/cache artifacts, private profiles, credentials, and local workspace paths.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have tool/vision Vision analysis and image generation provider/xai xAI (Grok) comp/tools Tool registry, model_tools, toolsets labels Jun 7, 2026
@jovijovi
jovijovi force-pushed the feat/xai-image-edit-v016 branch from 8bcb090 to 2493953 Compare June 20, 2026 01:39
@jovijovi jovijovi changed the title feat: add xAI image edit tool feat: add xAI image edit and refresh Imagine provider Jun 20, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the xAI API refresh work. The useful provider changes need a targeted salvage onto current main rather than a direct merge.

Problems

  • agent/image_gen_provider.py:479 adds a local-file read path without raise_if_read_blocked; plugins/image_gen/xai/__init__.py:234 reaches it. Current main guards xAI reads at plugins/image_gen/xai/__init__.py:140-145 (commit c1826e269) and tests that blocked credentials are never opened.
  • tools/image_edit_tool.py:57-60 checks any available edit provider, but dispatch at :82-85 uses only the active provider. This can expose a tool that necessarily returns unsupported_capability.
  • Current main already deliberately unifies image editing under image_generate (agent/image_gen_provider.py:16-24, commit c02192ff6); adding image_edit to _HERMES_CORE_TOOLS duplicates that model-facing surface.
  • The provider rewrite must retain current image_gen.xai.storage behavior at plugins/image_gen/xai/__init__.py:289-333.

Suggested changes

  • Rebase the provider changes conceptually onto the unified image_generate path, preserve current storage and file-safety behavior, and add the active-provider availability test.

Automated hermes-sweeper review.

f"Image file exceeds {max_bytes // (1024 * 1024)}MB cap: {path}"
)

raw = p.read_bytes()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads a model-supplied path without agent.file_safety.raise_if_read_blocked. Current main guards xAI local-image reads before opening them (plugins/image_gen/xai/__init__.py:140-145, c1826e269); put that guard in this shared helper before any filesystem access and add a no-read credential-store regression test.

Comment thread tools/image_edit_tool.py
from hermes_cli.plugins import _ensure_plugins_discovered

_ensure_plugins_discovered()
for provider in list_providers():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Availability is based on any discovered provider, but dispatch later uses only get_active_provider(). If xAI is available while the selected provider cannot edit, this exposes image_edit for a guaranteed unsupported-capability response. Resolve and check the active provider here instead.


# Optional storage_options pass-through (FR7). Validated for safe shape;
# default-off and never adds public_url on the caller's behalf.
storage_options = kwargs.get("storage_options")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve the current config-driven image_gen.xai.storage path when salvaging this rewrite. Current main builds and sends configured storage options by default at plugins/image_gen/xai/__init__.py:289-333; accepting only direct kwargs would silently drop that existing behavior.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the thorough work here, @jovijovi — closing after review, with two reasons:

  1. The core outcome (xAI image editing) has since landed on main via a different route: the xAI provider now routes generate(image_url=...) to /v1/images/edits (with multi-source support, storage options, and — as of Image picker no longer crashes on stale cross-provider models; xAI Grok catalog goes live-driven (salvage #77238) #89841/Picked image model now reaches xAI generation; OpenRouter picker lists all live image models (salvage #55893) #89890 — edit-capable model selection incl. grok-imagine-image-2.0). Your PR was the earlier articulation of that need and the JSON-body-not-multipart insight matches what shipped.

  2. The image_edit core tool + supports_edit() ABC half doesn't fit our tool-surface policy: every core model tool ships on every API call, and image editing already flows through the existing image_generate tool (image_url / reference_image_urls params), so a second tool would duplicate that surface. The capability signaling your ABC change aimed at exists today via capabilities().

Appreciate the contribution — the editing capability is live and your PR helped map the requirements.

@teknium1 teknium1 closed this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have provider/xai xAI (Grok) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants