Skip to content

feat(image-gen): validate image references - #26968

Closed
BeliefanX wants to merge 4 commits into
NousResearch:mainfrom
BeliefanX:feat/image-gen-reference-validation
Closed

feat(image-gen): validate image references#26968
BeliefanX wants to merge 4 commits into
NousResearch:mainfrom
BeliefanX:feat/image-gen-reference-validation

Conversation

@BeliefanX

@BeliefanX BeliefanX commented May 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a small shared helper module for validating image-reference inputs before providers upload or forward them.
  • Supports http(s) URLs, valid data:image/*;base64,... URLs, and local Hermes image-cache paths with size/type checks.
  • Keeps validation independent from provider edit contracts and concrete provider implementations.

Relationship

Testing

  • venv/bin/python -m py_compile agent/image_reference.py tests/agent/test_image_reference.py
  • venv/bin/python -m pytest tests/agent/test_image_reference.py -q -o 'addopts='
  • git diff --cached --check

@cardtest15-coder

This comment was marked as spam.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint type/security Security vulnerability or hardening and removed type/security Security vulnerability or hardening labels May 16, 2026
@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from 95e2f29 to 8440e89 Compare May 16, 2026 15:20
@BeliefanX

Copy link
Copy Markdown
Contributor Author

Addressed review feedback in the updated stack:

  • Added encoded-length precheck for data URLs before base64 decoding, so oversized inputs fail before allocation/decoding.
  • Switched local path validation to use a single opened file and header-only reads for MIME sniffing instead of reading the whole file.
  • Added regression tests for oversized data URLs before decode, invalid base64 after precheck, and header-only local validation.
  • Kept provider/tool behavior unchanged in this PR; it remains validation-only and inherits the PR1 CI baseline fixes.

@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from 8440e89 to 6e5fa50 Compare May 16, 2026 15:39
@BeliefanX
BeliefanX marked this pull request as ready for review May 16, 2026 15:55
@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from 6e5fa50 to 2bf38b2 Compare May 29, 2026 07:47
@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from 2bf38b2 to 5c1a56b Compare June 6, 2026 12:43
@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from 5c1a56b to a8e0afc Compare June 14, 2026 07:50

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the validation logic and for addressing the encoded-length and header-only-read feedback.

Problems

  • agent/image_reference.py:50 is not reached by the current image-generation path. tools/image_generation_tool.py:1343-1349 only normalizes reference strings before calling provider.generate(). Current providers still load references independently: OpenAI at plugins/image_gen/openai/__init__.py:282-284, xAI at plugins/image_gen/xai/__init__.py:140-151, and OpenRouter at plugins/image_gen/openrouter/__init__.py:99-111. Therefore this PR does not yet enforce its stated validation policy.
  • Integration needs to cover provider-specific contracts. For example, OpenAI currently reads any non-denied local path at plugins/image_gen/openai/__init__.py:149-155, whereas Codex has independent local/data validation at plugins/image_gen/openai-codex/__init__.py:179-237.

Suggested changes

  • Wire validation through a production dispatch/provider path and add dispatch-level tests proving rejected references never reach a provider.
  • Reconcile this helper with the existing unified image resolver in tools/image_source.py:89-160 rather than leaving parallel validation semantics.

Automated hermes-sweeper review.

Comment thread agent/image_reference.py Outdated
"""Typed validation error for image references."""

def __init__(self, message: str, *, error_type: str = "invalid_argument") -> None:
super().__init__(message)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This validator is not wired into the current image-generation path: tools/image_generation_tool.py:1343-1349 only normalizes strings before dispatch, and providers still load or forward references independently. Please integrate it at a production chokepoint (with dispatch-level coverage); otherwise the cache/type/size policy has no runtime effect.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
@BeliefanX
BeliefanX force-pushed the feat/image-gen-reference-validation branch from a8e0afc to e210a75 Compare July 13, 2026 09:41
@BeliefanX

Copy link
Copy Markdown
Contributor Author

Addressed the hermes-sweeper findings in 00a658f:

  • Validation now runs in the production dispatch path before provider calls, including fail-fast handling for unknown providers, unsupported image input, invalid schemes, and provider total-image limits.
  • Local/data sources reuse tools/image_source.py with a bounded remaining-byte budget; HTTPS inputs stay as normalized URLs without eager download or reference reordering.
  • Provider source-count capabilities now match their actual wire limits, so excess references fail instead of being silently truncated.
  • Added handler-level regressions covering provider-not-called, zero-I/O failure paths, URL preservation/order, count limits, and bounded local/data materialization.

Focused verification: 174 tests passed; all required GitHub checks are green. Ready for re-review.

@BeliefanX

BeliefanX commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Closing this stale implementation after reviewing it against current main. The image-input safety boundary remains valuable, but any future version should be rebuilt as a minimal handler-level pre-dispatch gate that reuses the current tools.image_source resolver and current provider contracts.

@BeliefanX BeliefanX closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants