refactor(deep-research): single-source the chart contract via an on-demand visualization skill

CHANGELOG.md

@@ -10,6 +10,7 @@ release; the candidate will be stabilized before the final `v2.2.0` release.
 - Routed deep research now uses explicit source-router, structured planner, concurrent researcher, and writer roles, with bounded source-tool batching and no research-plan approval step
 - Report follow-up supports answers over a completed report, child-job cosmetic rewrites, and delta research that carries the parent report forward as context
 - Clarification is more targeted: it can search for context before asking the user to narrow scope or choose an output shape
+- The writer produces figures through an on-demand `chart-generation` skill in the new `visualization` skill collection, rendering a sandbox PNG artifact when a sandbox is available and an inline chart spec otherwise; the skill ships enabled only in the skills and sandbox example configs (`config_domain_routing_and_skills`, `config_openshell`), while every other config presents chart-worthy data as a Markdown table, and a writer that wants inline charts must be assigned the `visualization` collection (charts are not sandbox-gated)
 
 **Sources and integrations**
 

configs/config_domain_routing_and_skills.yml

@@ -179,7 +179,7 @@ functions:
     _type: deep_research_skills
     agents:
       researcher-agent: [research]
-      writer-agent: [synthesis]
+      writer-agent: [synthesis, visualization]
     require_sandbox:
       - research
 

configs/config_openshell.yml

@@ -148,7 +148,7 @@ functions:
     _type: deep_research_skills
     agents:
       researcher-agent: [research]
-      writer-agent: [synthesis, research]
+      writer-agent: [synthesis, research, visualization]
     require_sandbox:
       - research
 

docs/source/architecture/agents/deep-researcher.md

@@ -137,10 +137,11 @@ runtime dependencies, not additional agents:
 | Inference and source tools | LLM calls, source-tool calls, credentials, orchestration state, and `/shared/` remain in the AI-Q process. Only generated code and job-workspace files cross the sandbox boundary. |
 
 The shipped `config_domain_routing_and_skills.yml` profile assigns the
-`research` collection to researcher workers and the `synthesis` collection to
-the writer. The research collection currently includes chart generation,
-table analysis, forecast analysis, and lightweight calculations. The synthesis
-collection includes long-form and prediction report writers. A skill provides
+`research` collection to researcher workers and the `synthesis` and
+`visualization` collections to the writer. The research collection currently
+includes table analysis, forecast analysis, and lightweight calculations. The
+synthesis collection includes long-form and prediction report writers, and the
+visualization collection provides chart generation. A skill provides
 instructions; only skills that invoke `execute` require the optional sandbox.
 Modal and OpenShell implement the same provider-neutral job-scoped contract.
 

docs/source/customization/configuration-reference.md

@@ -540,6 +540,16 @@ ordered `preferred_tools` and `fallback_tools` guidance on each `ResearchQuery`;
 request-filtered callable set. Refer to [Tools and Sources](./tools-and-sources.md#automatic-source-routing) and the
 [`config_domain_routing_and_skills.yml`](../../../configs/config_domain_routing_and_skills.yml) reference profile.
 
+```{note}
+**Migration: `chart-generation` moved to the `visualization` collection.** The built-in
+`chart-generation` skill previously lived in the `research` collection; it now lives in its own
+`visualization` collection, and charts are no longer sandbox-gated. The `visualization` skill ships
+enabled only in the skills and sandbox example configs (`config_domain_routing_and_skills.yml` and
+`config_openshell.yml`); every other shipped config presents chart-worthy data as a Markdown table.
+A writer that wants inline charts must be assigned the `visualization` collection in its
+`deep_research_skills` assignment.
+```
+
 ---
 
 ## `workflow` Section

docs/source/examples/skills-sandbox/index.md

@@ -33,8 +33,9 @@ The built-in collections currently expose these role-oriented skills:
 
 | Collection | Default assignment | Skills |
 | ---------- | ------------------ | ------ |
-| `research` | `researcher-agent` | `chart-generation`, `data-table-analysis`, `forecast-analysis`, `lightweight-calculation` |
+| `research` | `researcher-agent` | `data-table-analysis`, `forecast-analysis`, `lightweight-calculation` |
 | `synthesis` | `writer-agent` | `long-form-report-writer`, `prediction-report-writer` |
+| `visualization` | `writer-agent` | `chart-generation` |
 
 The assignment is configurable. Skill definitions stay host-side and read-only;
 only workflows that invoke `execute` require a sandbox.
@@ -78,6 +79,7 @@ functions:
         - research
       writer-agent:
         - synthesis
+        - visualization
     require_sandbox:
       - research
 
@@ -104,7 +106,7 @@ functions:
     sandbox: deep_research_sandbox
 ```
 
-AI-Q validates the public skill collection names (`research`, `synthesis`) and resolves them to DeepAgents source paths internally. When skills are configured, AI-Q mounts the configured built-in skill collections into the DeepAgents virtual filesystem. When the sandbox ref is present, DeepAgents `execute` calls run in the configured provider. Modal creates a fresh sandbox named for the job.
+AI-Q validates the public skill collection names (`research`, `synthesis`, `visualization`) and resolves them to DeepAgents source paths internally. When skills are configured, AI-Q mounts the configured built-in skill collections into the DeepAgents virtual filesystem. When the sandbox ref is present, DeepAgents `execute` calls run in the configured provider. Modal creates a fresh sandbox named for the job.
 
 In the reference async API flow, artifact capture uses the job database configured by
 `general.front_end.db_url` (`NAT_JOB_STORE_DB_URL`) for metadata. Artifact bytes use SQL BLOB storage in the job database

src/aiq_agent/agents/deep_researcher/deepagents_runtime.py

@@ -46,6 +46,7 @@
 BUILTIN_SKILL_SOURCE = "/skills/"
 SHARED_ROUTE = "/shared/"
 SKILL_AGENT_NAMES = frozenset({"researcher-agent", "writer-agent"})
+CHART_SKILL_NAME = "chart-generation"
 DEFAULT_WORKDIR = "/workspace"
 
 
@@ -267,6 +268,18 @@ def skill_sources_for(self, agent_name: str) -> list[str] | None:
         sources = self._skill_sources_by_agent.get(agent_name)
         return list(sources) if sources else None
 
+    def agent_has_chart_skill(self, agent_name: str) -> bool:
+        """Return true when the agent's assigned collections provide the chart skill on disk."""
+        sources = self._skill_sources_by_agent.get(agent_name)
+        if not sources:
+            return False
+        collection_for_source = {source: name for name, source in discover_skill_collections().items()}
+        return any(
+            (BUILTIN_SKILLS_DIR / collection_for_source[source] / CHART_SKILL_NAME / "SKILL.md").exists()
+            for source in sources
+            if source in collection_for_source
+        )
+
     @property
     def workdir(self) -> str:
         """Job-scoped sandbox working directory (or the default when no sandbox)."""

src/aiq_agent/agents/deep_researcher/factory.py

@@ -532,7 +532,10 @@ def build_deep_research_subagents(context: DeepResearchGraphContext) -> list[dic
                 TodoSuppressionMiddleware(),
                 RequiredOutputFileMiddleware(tracker=context.final_report_tracker),
             ],
-            prompt_values={"parent_report_context_available": context.parent_report_context_available},
+            prompt_values={
+                "parent_report_context_available": context.parent_report_context_available,
+                "chart_skill_enabled": context.runtime.agent_has_chart_skill(WRITER_AGENT),
+            },
             skills=context.skill_sources(WRITER_AGENT),
         ),
     )

src/aiq_agent/agents/deep_researcher/prompts/writer.j2

@@ -75,15 +75,22 @@ When synthesizing:
 - Do not expose internal `evidence_judgment` scores or rationale in the final answer unless the user explicitly asks for methodology.
 - Err on the side of more useful information rather than less, while staying focused on the requested answer shape.
 
-{% if execution_enabled %}## Figures
+{% if chart_skill_enabled %}## Figures and Charts
 
-When `answer_strategy.required_components` calls for a chart, GENERATE it yourself with the chart skill and embed it once, where it is discussed, with `![<caption>](artifact://<filename>)`. Read the relevant chart `SKILL.md` first, then write the plotting script to `{{ sandbox_workdir }}/make_chart.py`. The script must accept its artifact directory as its first argument; run it exactly as `python3 {{ sandbox_workdir }}/make_chart.py {{ sandbox_artifact_dir }}` so it writes the image plus `manifest.json` under the only directory the runtime harvests. Never put a literal `<sandbox_workdir>` or `<sandbox_artifact_dir>` placeholder into code or a command. After execution, use the exact filename confirmed by the artifact checkpoint in the report reference.
+Charts are available for this answer. When `answer_strategy.required_components` calls for a chart (a ranking or top-N, a distribution across categories, a trend over an ordered axis, gains vs losses, or a single headline value), consult the `chart-generation` skill and read its `SKILL.md` first. The skill is the controlling procedure: it owns the full chart contract, both delivery modes, and the mode decision, so this base prompt carries no chart syntax or steps.
 
-Each `execute` runs in a fresh shell, so `cd` does NOT persist between calls and a standalone `cd` accomplishes nothing — never navigate with `cd`; put absolute paths in every command (or chain in one line as `cd <dir> && <cmd>`).
+A figure is earned: chart only source-anchored, reasonably complete data. If a series is mostly undisclosed or mixes metric definitions, present the table (which shows the gaps) and state the limitation in one sentence instead of a misleading chart.
 
-NEVER paste the plotting code, a sandbox file path (e.g. `{{ sandbox_artifact_dir }}/chart.png`), or base64 image data into the report as prose - the `artifact://` token is the ONLY way a figure renders. Do not call `read_file` on a generated PNG just to verify it; that injects the image's base64 bytes into model context. Verify the manifest and use the artifact-checkpoint response instead. A "plotting code" or "chart plan" section is not a substitute for the actual figure. Precede each embed with a one-sentence description of what it shows.
+Runtime context for the skill:
+{% if execution_enabled %}
+- A sandbox is available, so the skill renders a PNG artifact. Your per-job working directory is `{{ sandbox_workdir }}` and the harvested artifact directory is `{{ sandbox_artifact_dir }}`; give the skill these exact paths where it asks for `sandbox_workdir` and `sandbox_artifact_dir`.
+{% else %}
+- There is no sandbox, so the skill delivers the chart inline in the report.
+{% endif %}
+
+{% else %}## Figures and Charts
 
-A figure is earned: generate or embed one only when its data is source-anchored and reasonably complete. If a quantitative series is mostly undisclosed or mixes metric definitions, present the table (which shows the gaps) and state the limitation in one sentence instead of a misleading chart.
+When a required component calls for a chart, present the values as a compact Markdown table and state the takeaway in one sentence.
 
 {% endif %}## Citations
 - Citations are mandatory for any sourced final answer. Every material source-derived factual claim, number, date, quote, source-specific caveat, or source claim must have an inline citation like `[1]`.
@@ -102,37 +109,7 @@ A figure is earned: generate or embed one only when its data is source-anchored
 - Do not place bare URLs in the report body; URLs belong only in the `## Sources` section.
 - Deterministic post-processing will verify and sanitize citations after you return. Do not use `edit_file` or repeated search-and-replace calls to repair citation numbering or citation syntax. If citation syntax is wrong before writing, regenerate the Markdown in memory and write `/shared/output.md` once.
 
-{% if not execution_enabled %}## Presenting Data (Charts)
-When the report presents numbers that compare multiple things (a ranking or top-N across entities, a distribution or counts across categories, a trend over an ordered/time axis, or gains vs losses), present them as a chart, not just prose or a table. Lead with a one-sentence verdict, then the chart.
-
-- Emit the chart as a fenced code block tagged `chart` holding a SINGLE line of valid JSON (no comments, no trailing commas), right after the sentence that introduces it.
-- Chart to reveal the pattern and state the verdict in prose. Inline `chart` blocks render only in the web app, so ALSO place a compact markdown table of the same values immediately after each chart, keeping PDF, Markdown, API, and CLI exports readable.
-- Use the ACTUAL numbers from the evidence (top ~10 rows); never invent, pad, or round away data. If you show only the top rows of a larger set, say so.
-- At most 3 charts per section, and put each chart before any table.
-- A single value or a one-entity yes/no result is NOT a chart: emit a KPI-only block, a fenced `chart` block whose JSON has just `title` and `kpis`.
-
-Chart types: `bar` (category magnitudes), `hbar` (rankings with long text labels), `line`/`area` (a trend across an ordered axis), `grouped-bar` (2-4 series per category), `delta` (gains vs losses around zero).
-
-Spec fields: `type`; `title` (short) and optional `subtitle`; `x` = `{ "key": "<field in each row>", "label": "optional" }`; optional `y` = `{ "label": "optional unit", "format": "number | compact | percent | currency" }`; `series` = `[ { "key": "<numeric field>", "label": "optional", "color": "green | blue | amber | red" } ]`; `data` = rows as objects with raw numbers (fractions 0-1 for `percent`); optional `kpis` = `[ { "label": "...", "value": "preformatted", "tone": "accent | warn | alarm" } ]`.
-
-Example (ranking):
-```chart
-{"type":"hbar","title":"Top suppliers by late shipments","x":{"key":"supplier"},"y":{"format":"number"},"series":[{"key":"late","color":"amber"}],"data":[{"supplier":"Acme","late":42},{"supplier":"Globex","late":31},{"supplier":"Initech","late":19}]}
-```
-
-Example (single value, KPI-only):
-```chart
-{"title":"On-time delivery rate","kpis":[{"label":"On-time","value":"92.4%","tone":"accent"}]}
-```
-
-For several related trends over time, emit one fenced `chart-carousel` block holding a SINGLE line of JSON with at least two line-chart specs: `{ "title": "...", "charts": [ <line chart spec>, ... ] }`.
-
-Example (related trends, carousel):
-```chart-carousel
-{"title":"Quarterly delivery trends","charts":[{"type":"line","title":"On-time delivery rate","x":{"key":"quarter"},"y":{"format":"percent"},"series":[{"key":"rate","color":"green"}],"data":[{"quarter":"Q1","rate":0.88},{"quarter":"Q2","rate":0.90},{"quarter":"Q3","rate":0.93}]},{"type":"line","title":"Late shipments","x":{"key":"quarter"},"y":{"format":"number"},"series":[{"key":"late","color":"amber"}],"data":[{"quarter":"Q1","late":52},{"quarter":"Q2","late":41},{"quarter":"Q3","late":28}]}]}
-```
-
-{% endif %}## Final Output
+## Final Output
 
 Write the final Markdown to `/shared/output.md`.
 Then return only this short completion marker as your final response: `Wrote /shared/output.md`.