-
Notifications
You must be signed in to change notification settings - Fork 257
feat(deep-research): add image-processing reference skill #285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
zhongxuanwang-nv
wants to merge
2
commits into
NVIDIA-AI-Blueprints:develop
from
zhongxuanwang-nv:feat/skills-ref-image-processing
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
134 changes: 134 additions & 0 deletions
134
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| --- | ||
| name: image-processing | ||
| description: > | ||
| Use this skill to inspect and transform images during research by writing Python/Pillow code and running it in the job-scoped sandbox. This skill is for image metadata (dimensions, mode, format, file size), format conversion, resizing and thumbnails, cropping, rotation, grayscale, and basic color/brightness analysis. Triggers: "image", "PNG", "JPEG", "WEBP", "resize", "downscale", "thumbnail", "convert format", "crop", "rotate", "grayscale", "image dimensions", "image metadata", "EXIF", "dominant color", "brightness". Outputs: JSON/Markdown metadata and analysis notes (plus an optional small base64 thumbnail) returned in your ResearchNotes for synthesis. | ||
| --- | ||
|
|
||
| # Image Processing Skill | ||
|
|
||
| Inspect and transform images with Python/Pillow in the job-scoped sandbox, then emit text-survivable results — metadata, analysis notes, and optional small base64 thumbnails — that can be read and included in the final report. | ||
|
|
||
| ## Required Execution Standard | ||
|
|
||
| To keep results reproducible and report-usable, you MUST: | ||
| 1. **Source the Image Explicitly:** Identify where the image bytes come from (a `/shared/...` file, or base64 provided in the request). The sandbox has no network access, so images cannot be fetched by URL from inside `execute`. | ||
| 2. **Stage Into `/workspace`:** If the image is in `/shared/...`, call `read_file` first and write the bytes into a `/workspace` file (or decode an inline base64 string). Sandbox code cannot open `/shared/...` directly. | ||
| 3. **Process Deterministically:** Call the `execute` tool to run Python/Pillow for metadata, conversion, resizing, cropping, rotation, grayscale, or color analysis. Do not describe image properties in prose without measuring them. | ||
| 4. **Emit Text-Survivable Outputs:** Durable outputs must be text. Include the metadata/analysis (as JSON or a Markdown block) in your returned `ResearchNotes` — e.g. in a `ResearchFinding`'s `evidence` and/or `narrative_notes`. For a visual, you may include a *small* base64 `data:image/...;base64,...` thumbnail in that text; lead with the metadata/analysis, which is always readable. Do not call `write_file`; `run_research_batch` persists your returned notes. | ||
| 5. **Report Caveats:** Note the source of the image, and any lossy conversion, downscaling, stripped metadata, or color-space assumptions. | ||
|
|
||
| **Required Tool Use:** For any image inspection or transformation, this skill requires at least one `execute` call that runs Pillow before writing the final artifacts. | ||
|
|
||
| ## Execution Flow | ||
|
|
||
| 1. Determine the image source (a `/shared/...` path, or inline base64 in the request). | ||
| 2. Stage the bytes into `/workspace`: `read_file` the `/shared` file and write it under `/workspace/input.<ext>`, or `base64`-decode an inline string. Never read from or write to `/shared/...` inside the sandbox process. | ||
| 3. Call the `execute` tool with a Python/Pillow script that: | ||
| - opens the image from `/workspace`, | ||
| - performs the requested inspection or transform, | ||
| - writes any derived image to `/workspace` (not `/shared`), | ||
| - prints metadata/analysis as JSON or Markdown text, and a base64 string only for a small thumbnail. | ||
| 4. Inspect the `execute` output. For fixable errors (bad path, unreadable/decode error), fix the code and call `execute` again. If a required library is missing (e.g. Pillow), report it as a sandbox limitation and stop — do not retry or invent image properties. | ||
| 5. Return the result in your `ResearchNotes` — put the metadata/analysis (and any small base64 thumbnail) into a `ResearchFinding`'s `evidence` and/or `narrative_notes`. Do not call `write_file`/`edit_file`; `run_research_batch` persists your returned notes under `/shared/` automatically. | ||
| 6. In the response or report, cite where the image came from and label any computed or derived values. | ||
|
|
||
| --- | ||
|
|
||
| **Other operations** use the same pattern: `im.crop((left, upper, right, lower))`, | ||
| `im.rotate(deg, expand=True)`, `im.convert("L")` (grayscale), `im.resize((w, h))`. | ||
| Keep any embedded base64 thumbnail small (≤256 px) — base64 inflates size ~33%. | ||
|
|
||
| --- | ||
|
|
||
| ## Example Code Templates | ||
|
|
||
| ### A. Inspect Image Metadata | ||
|
|
||
| Use when you only need dimensions, mode, format, and size. EXIF orientation is applied so | ||
| width/height are the *displayed* dimensions (orientation-tagged photos store them rotated). | ||
|
|
||
| ```python | ||
| import json | ||
| from pathlib import Path | ||
| from PIL import Image, ImageOps | ||
|
|
||
| src = "/workspace/input.png" # staged from /shared or decoded from base64 | ||
| with Image.open(src) as im: | ||
| fmt = im.format # read before transpose (transpose drops .format) | ||
| oriented = ImageOps.exif_transpose(im) # honor EXIF orientation -> displayed size | ||
| meta = { | ||
| "width": oriented.width, # displayed dimensions | ||
| "height": oriented.height, | ||
| "mode": oriented.mode, # e.g. "RGB", "RGBA", "L" | ||
| "format": fmt, # e.g. "PNG", "JPEG" | ||
| "size_bytes": Path(src).stat().st_size, | ||
| } | ||
| print(json.dumps(meta, indent=2)) | ||
| ``` | ||
|
|
||
| ### B. Make a Thumbnail and Emit a Base64 Preview | ||
|
|
||
| Use when the report should show a small inline preview (there is no binary-artifact capture, so embed base64 text). | ||
|
|
||
| ```python | ||
| import base64 | ||
| import io | ||
| import json | ||
| from PIL import Image, ImageOps | ||
|
|
||
| with Image.open("/workspace/input.png") as im: | ||
| im = ImageOps.exif_transpose(im).convert("RGB") # honor EXIF orientation | ||
| im.thumbnail((256, 256)) # preserves aspect ratio | ||
| buf = io.BytesIO() | ||
| im.save(buf, format="PNG") | ||
|
zhongxuanwang-nv marked this conversation as resolved.
|
||
|
|
||
| b64 = base64.b64encode(buf.getvalue()).decode("ascii") | ||
| print(json.dumps({"thumb_width": im.width, "thumb_height": im.height, "b64_len": len(b64)})) | ||
| # Emit the data URI on its own line so the agent can include it in its ResearchNotes summary: | ||
| print(f"data:image/png;base64,{b64}") | ||
| ``` | ||
|
|
||
| The agent then includes that summary in its returned `ResearchNotes` — metadata first | ||
| (source, original WxH/mode/format/size), then the preview as | ||
| ``. | ||
|
|
||
| ### C. Convert Format | ||
|
|
||
| Use to standardize an image to JPEG/PNG/WEBP. | ||
|
|
||
| ```python | ||
| from PIL import Image | ||
|
|
||
| with Image.open("/workspace/input.png") as im: | ||
| if im.mode in ("RGBA", "P"): | ||
| im = im.convert("RGB") # JPEG has no alpha channel | ||
| im.save("/workspace/output.jpg", "JPEG", quality=90) | ||
| print("Converted /workspace/input.png -> /workspace/output.jpg (JPEG q=90)") | ||
| ``` | ||
|
|
||
| ### D. Basic Color / Brightness Analysis | ||
|
|
||
| Use to summarize an image without a full vision model. | ||
|
|
||
| ```python | ||
| import json | ||
| from PIL import Image, ImageStat | ||
|
|
||
| with Image.open("/workspace/input.png") as im: | ||
| stat = ImageStat.Stat(im.convert("RGB").resize((64, 64))) | ||
|
|
||
| avg_rgb = [round(c, 1) for c in stat.mean] # mean per R, G, B channel | ||
| mean_brightness = round(sum(stat.mean) / 3, 1) # 0-255 | ||
| print(json.dumps({"avg_rgb": avg_rgb, "mean_brightness_0_255": mean_brightness}, indent=2)) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Troubleshooting in the Sandbox | ||
|
|
||
| - Missing Pillow: If `from PIL import Image` fails, report that the sandbox image needs `pillow` installed. Do not describe image contents from assumption. | ||
| - No network: The sandbox cannot fetch image URLs. Stage bytes from `/shared` (via `read_file`) or accept inline base64. | ||
| - `/shared` access: Sandbox code cannot read or write `/shared/...`. Use `/workspace` inside `execute`, then return the text result in your `ResearchNotes` (the harness persists it) — do not `write_file`. | ||
| - Binary outputs: There is no durable binary-artifact capture, so transformed images live only in `/workspace`. Persist results as text (metadata/JSON/Markdown), plus a small base64 thumbnail when a preview is needed. | ||
| - JPEG and alpha: Convert `RGBA`/`P` images to `RGB` before saving as JPEG. | ||
| --- | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.