feat(deep-research): add image-processing reference skill - #285
feat(deep-research): add image-processing reference skill#285zhongxuanwang-nv wants to merge 2 commits into
Conversation
WalkthroughAdds a new research skill document for image-processing that defines sandbox execution requirements, supported Pillow operations, example scripts, and troubleshooting guidance. ChangesImage processing skill documentation
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
ce48bbe to
a6cb85a
Compare
a6cb85a to
4b40732
Compare
Add the image-processing reference skill under the deep researcher's research/ skill collection (Pillow: metadata, format conversion, resize/thumbnail, crop, rotate, grayscale, basic color/brightness), mirroring the existing research skills: process in /workspace, then RETURN text-survivable results (JSON/Markdown metadata, optional small base64 thumbnail) in the researcher's ResearchNotes, which run_research_batch persists to /shared. The researcher must not call write_file (research.py), so the skill uses the return-in-ResearchNotes pattern. Rebased onto develop after NVIDIA-AI-Blueprints#284 restructured the skills layout (research-sandbox -> research) and rewrote the runtime/test. Placed at skills/research/image-processing/; no test change needed -- the post-NVIDIA-AI-Blueprints#284 test enumerates only synthesis skills, so research skills (incl. this one) are auto-discovered. Covers SKILLS-REF-1's image-processing category. No durable binary-artifact capture on develop, so outputs are text. Validation: uv run pytest tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (18 passed). Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
4b40732 to
a665e3c
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md`:
- Around line 31-32: Clarify the retry logic in the image-processing skill
around execute: when `execute` fails because Pillow is missing, treat that as a
sandbox limitation and stop instead of retrying, while only retrying `execute`
for fixable problems like an incorrect file path or unreadable/decode errors.
Update the guidance in SKILL.md near the `execute` and `ResearchNotes`
instructions so the distinction is explicit and the retry advice only applies to
recoverable file/input issues.
- Around line 45-64: The image metadata example in the Inspect Image Metadata
section should account for EXIF orientation when reporting dimensions. Update
the snippet around Image.open in the metadata example to either apply
ImageOps.exif_transpose before reading width and height, or include the EXIF
orientation value alongside the raw dimensions from im. Preserve the existing
meta structure in the Image/ImageOps example and make sure the displayed size
cannot be misreported for orientation-tagged photos.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 881e45a1-7644-4c60-abee-e9a85987a472
📒 Files selected for processing (1)
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
📜 Review details
🧰 Additional context used
📓 Path-based instructions (2)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
Address CodeRabbit review on NVIDIA-AI-Blueprints#285: - Template A (metadata) and Template B (thumbnail preview): apply ImageOps.exif_transpose so width/height and the preview reflect the *displayed* image (orientation-tagged photos store pixels rotated). Read .format before the transpose, since exif_transpose returns a new image with .format == None. - Step 4 retry guidance: retry `execute` only for fixable errors (bad path, unreadable/decode); a missing library (e.g. Pillow) is reported as a sandbox limitation and stops -- matching the Troubleshooting bullet, no pointless retry. Verified locally (Pillow 12.2.0) against an EXIF Orientation=6 image: all four templates run, and metadata reports the displayed 200x300 (not the stored 300x200). Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md`:
- Around line 79-83: The PNG preview generation in
Image.open/ImageOps.exif_transpose currently converts everything to RGB, which
strips transparency from RGBA/P images. Update the image handling in this
snippet to preserve alpha when saving the inline PNG thumbnail, and only convert
to RGB in the JPEG path if needed, so the thumbnail preserves native
transparency correctly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 2459d8ee-84ae-4d6f-81c3-471ca348c809
📒 Files selected for processing (1)
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
📜 Review details
🧰 Additional context used
📓 Path-based instructions (2)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md
…ile (#286) The researcher worker is instructed (research.py format_research_request): "Do not call write_file or edit_file; run_research_batch will persist the returned ResearchNotes under /shared/." But the research skills told the agent to persist artifacts to /shared via write_file -- a contradicted instruction, since persistence is automatic from the returned ResearchNotes. Reword the persistence step in data-table-analysis, lightweight-calculation, and forecast-analysis to: include the result in your returned ResearchNotes (e.g. a ResearchFinding's evidence / narrative_notes); do not call write_file/edit_file. The separate, correct rule that sandbox code uses /workspace and cannot touch /shared is unchanged. (image-processing carries the same fix in PR #285; chart-generation in PR #280 needs it too -- flagged to its author.) Validation: uv run pytest tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (18 passed); grep confirms write_file now appears only as "do not call write_file". Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
|
Closed per discussion in internal slack channel |
1 participant
Overview
Adds the
image-processingreference skill under the deep researcher'sresearch/skill collection, completing SKILLS-REF-1's three named categories(data analysis =
data-table-analysis; chart generation =chart-generation, PR #280;image processing = this PR).
A Pillow skill for metadata (dimensions/mode/format/size), format conversion,
resize/thumbnail, crop, rotate, grayscale, and basic color/brightness. It mirrors the
existing
research/skills: run in/workspace, then return text-survivable results(JSON/Markdown metadata + an optional small base64 thumbnail) in the researcher's
ResearchNotes, whichrun_research_batchpersists to/shared. Perresearch.pytheresearcher must not call
write_file, so the skill uses the return-in-ResearchNotespattern (consistent with the companion fix in #286).
Rebased onto develop after #284 restructured the skills layout
(
research-sandbox/→research/) and rewrote the runtime/test. Placed atskills/research/image-processing/; no test change — the post-#284 test enumeratesonly synthesis skills, so research skills are auto-discovered.
The sandbox has no network and no durable binary-artifact capture on develop, so outputs
are text. When #280's artifact runtime lands, the base64 thumbnail can move to
.Validation
All code examples are verified locally (Pillow 12.2.0): each of the four templates was
extracted from this
SKILL.mdand executed against a real test image, producing thedocumented output — metadata, a 256px base64 thumbnail (aspect preserved), RGBA→JPEG
conversion, and color/brightness. The color template uses
ImageStatrather than the deprecatedgetdata()(removed inPillow 14), and the metadata/preview templates apply
ImageOps.exif_transposeso theyreflect displayed dimensions/orientation. Addresses the CodeRabbit review (EXIF
orientation; narrowed retry guidance so a missing library stops rather than retries).
git commit -s.Where should reviewers start?
src/aiq_agent/agents/deep_researcher/skills/research/image-processing/SKILL.md— it mirrorsresearch/data-table-analysis/SKILL.md(the/workspace→return-in-ResearchNotespattern).Related Issues
Summary by CodeRabbit