Skip to content
Closed
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ release; the candidate will be stabilized before the final `v2.2.0` release.

**Sandboxes, artifacts, and policy**

- DeepAgents execution uses a provider-neutral sandbox contract: Modal is fresh per job, while the experimental OpenShell profile uses one shared, pre-provisioned sandbox and is not a multi-tenant isolation boundary
- Opt-in durable artifact capture checkpoints manifest-declared files after successful sandbox `execute` calls, performs one final manifest-plus-directory scan on success/failure, and preserves earlier checkpoints without delaying cancellation when the provider is busy
- DeepAgents execution uses a provider-neutral sandbox contract: the Modal and experimental OpenShell profiles create one sandbox per job, while OpenShell shared attachment remains a debug-only, non-multi-tenant mode
- Opt-in durable artifact capture checkpoints writer manifest writes and successful writer chart executions, performs one final manifest-plus-directory scan on success/failure, and preserves earlier checkpoints without delaying cancellation when the provider is busy
- Captured files store metadata in SQL and bytes in SQL or S3-compatible storage, emit metadata-only `artifact.update` events for live and replayed Files-tab access, and remain available through job-scoped list/content endpoints that enforce ownership when `REQUIRE_AUTH=true`
- Opt-in NeMo Guardrails middleware covers selected workflow and agent input/output boundaries; defining middleware does not activate every boundary
- Opt-in content encryption protects final async output and selected artifact event content only; it is off by default, forward-only, and does not encrypt checkpoints or most job/event metadata
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ The NVIDIA AI-Q Blueprint is an enterprise-grade research agent built on the [NV
- **Report follow-up** — Ask questions about a completed report, create a child-job cosmetic rewrite, or run delta research with the parent report as context.
- **Workflow configuration** — YAML configs define agents, tools, LLMs, and routing behavior so you can tune workflows without code changes.
- **Modular workflows** — All agents (orchestration node, shallow researcher, deep researcher, clarifier) are composable; each can run standalone or as part of the full pipeline.
- **Skills, sandbox execution, and durable outputs** — Built-in research/synthesis skills run code through a provider-neutral sandbox contract. Modal is fresh per job; the experimental OpenShell profile uses one shared, pre-provisioned sandbox and is not a multi-tenant isolation boundary. Opt-in rich-file capture checkpoints manifest-declared files after successful sandbox commands, finalizes on success/failure, stores bytes in SQL or S3-compatible storage, and delivers metadata to the Files tab live and on replay.
- **Skills, sandbox execution, and durable outputs** — Built-in research/synthesis skills run code through a provider-neutral sandbox contract. The Modal and experimental OpenShell profiles create one sandbox per job; OpenShell shared attachment is debug-only. Opt-in rich-file capture checkpoints writer-published manifests and successful writer chart executions, finalizes on success/failure, stores bytes in SQL or S3-compatible storage, and delivers metadata to the Files tab live and on replay.
- **Portable Agent Skills** — `aiq-deploy` selects, starts, and validates an AI-Q deployment; `aiq-research` calls routed chat and async research from compatible coding harnesses.
- **Data source registry** — UI toggles and request payloads can select web, paper, enterprise, collaboration, and knowledge-layer sources per message.
- **Expanded sources** — Paper search supports Serper, SerpAPI, and SearchAPI; You.com adds web, contents, general-research, and finance-research tools; Nimble adds configurable web search; focused profiles demonstrate DuckDuckGo news, Polymarket, OpenSearch, and Azure AI Search knowledge retrieval.
Expand Down Expand Up @@ -285,7 +285,7 @@ The `configs/` directory holds YAML workflow configs that define agents, tools,
| `config_web_default_guardrails.yml` | GPT-OSS-120B; Nemotron 3 Super; Nemotron Mini summary | LlamaIndex profile with workflow Guardrails attached and async deep-agent Guardrails selected; shallow middleware is defined but not attached. |
| `config_web_frag_mcp_auth.yml` | Nemotron 3 Super | Foundational RAG plus an opt-in protected per-user OAuth MCP source example. Requires a real MCP endpoint and shared token store. |
| `config_domain_routing_and_skills.yml` | Nemotron 3 Super; Nemotron Mini summary | Direct deep-research profile with domain routing, DuckDuckGo news, Polymarket, enabled Serper paper search, LlamaIndex, built-in skills, and a fresh per-job Modal sandbox. |
| `config_openshell.yml` | GPT-OSS-120B; Nemotron 3 Super; Nemotron Mini summary | Experimental web/API skills profile with artifact capture over one shared, pre-provisioned OpenShell sandbox; trusted single-operator use only. |
| `config_openshell.yml` | GPT-OSS-120B; Nemotron 3 Super; Nemotron Mini summary | Experimental web/API skills profile with artifact capture in one policy-bound OpenShell sandbox per job; shared attachment is debug-only. |
| `config_mcp.yml` | Nemotron 3 Super 120B | Standalone MCP server. Public NIM + Tavily research with PostgreSQL-backed stateless submit/poll/report. Requires `NVIDIA_API_KEY`, `TAVILY_API_KEY`, and `AIQ_CHECKPOINT_DB`. |

## Ways to Run the Agents
Expand Down Expand Up @@ -461,9 +461,9 @@ indicate availability in a published release.
- [x] **[NeMo Guardrails](docs/source/customization/guardrails.md) Integration:** Opt-in middleware for selected workflow, shallow-researcher, and deep-researcher boundaries.
- [ ] **[NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) Integration:** Reduce latency via priority scheduling at scale.
- [x] **Per-user MCP OAuth:** Connect each signed-in user to protected MCP data sources through the UI.
- [x] **Skills & Sandboxing:** Support built-in deep-research skills through a provider-neutral contract. Modal is fresh per job; the experimental OpenShell profile is shared and not a multi-tenant isolation boundary.
- [x] **Skills & Sandboxing:** Support built-in deep-research skills through a provider-neutral contract. The Modal and experimental OpenShell profiles create one sandbox per job; OpenShell shared attachment is debug-only and not a multi-tenant isolation boundary.
- [x] **Report Follow-up and Rewriting:** Answer against a completed report, submit cosmetic child rewrites, and run delta research with parent-report context.
- [x] **Durable Sandbox Artifacts:** Opt-in manifest checkpoints after successful sandbox commands, terminal harvesting on success/failure, SQL or S3-compatible byte storage, and live/replayed Files-tab access.
- [x] **Durable Sandbox Artifacts:** Opt-in checkpoints after writer manifest writes and successful writer chart executions, terminal harvesting on success/failure, SQL or S3-compatible byte storage, and live/replayed Files-tab access.
- [ ] **Custom Skill Management:** Add UI and lifecycle controls for user-provided skill bundles.
- [ ] **Dynamic Model Routing:** Allow sub-agents to automatically select the optimal model per task.
- [ ] **Resource Management:** Implement configurable token caps and tool-call budgets.
Expand Down
1 change: 1 addition & 0 deletions configs/config_domain_routing_and_skills.yml
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ functions:
writer-agent: [synthesis]
require_sandbox:
- research
- synthesis

deep_research_sandbox:
_type: deep_research_sandbox
Expand Down
3 changes: 2 additions & 1 deletion configs/config_openshell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,9 +120,10 @@ functions:
_type: deep_research_skills
agents:
researcher-agent: [research]
writer-agent: [synthesis, research]
writer-agent: [synthesis]
require_sandbox:
- research
- synthesis

deep_research_sandbox:
_type: deep_research_sandbox
Expand Down
80 changes: 70 additions & 10 deletions docs/source/architecture/agents/deep-researcher.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,9 @@ for configuration details.
| `domain_catalog_path` | `str` or `None` | `None` | Optional YAML or JSON domain catalog used by the source router |
| `enable_source_router` | `bool` | `true` | Run the advisory source-router stage before planning |
| `max_research_concurrency` | `int` | `6` | Maximum `ResearchQuery` items accepted and run concurrently per batch call |
| `max_researcher_execute_attempts` | `int` or `None` | `None` | Optional physical code-execution limit per researcher worker, including generic retries |
| `max_writer_execute_attempts` | `int` or `None` | `None` | Optional physical writer chart-execution limit, including generic retries |
| `workflow_timeout_seconds` | `float` or `None` | `None` | Optional async workflow deadline; expiry fails with `deep_research_workflow_timeout` |
| `skills` | `FunctionRef`, inline `deep_research_skills`, or `None` | `None` | Optional built-in skill assignments by agent name |
| `sandbox` | `FunctionRef`, inline `deep_research_sandbox`, or `None` | `None` | Optional sandbox profile for DeepAgents `execute` support |
| `enable_citation_verification` | `bool` | `true` | Verify generated citations against captured sources after final report extraction |
Expand All @@ -228,6 +231,30 @@ functions:
- web_search_tool
```

### Optional convergence safeguards

Execution budgets and the async workflow deadline are disabled by default so an upgrade does not truncate an existing
long-running or multi-step workflow. Operators can enable them independently:

```yaml
functions:
deep_research_agent:
max_researcher_execute_attempts: 3
max_writer_execute_attempts: 3
workflow_timeout_seconds: 3600
```

`max_researcher_execute_attempts` applies separately to each researcher worker. Every physical `execute` call, including
a call made by generic retry middleware, consumes an attempt. After exhaustion, the worker is asked to return supported
notes from evidence already collected; an unvalidated quantitative dataset may be omitted.

`max_writer_execute_attempts` counts only canonical chart-rendering `execute` calls, not generic writer file publication.
After exhaustion, the writer preserves the report, Markdown table, and already-published CSV, but may finish without a
PNG. `workflow_timeout_seconds` covers the asynchronous workflow invocation. Expiry marks the job as failed with
`deep_research_workflow_timeout`, requests forced sandbox termination, and performs only best-effort artifact recovery.
Choose these values from observed workload duration and execution counts; an execution budget alone does not bound
model or source-tool loops.

```{note}
**Nemotron Super — Build Endpoint Availability:** Nemotron Super (`nvidia/nemotron-3-super-120b-a12b`) is compatible and tested with AIQ, but Build API endpoints have limited availability due to high demand (HTTP 429/503 responses). The default configs use Nemotron Super for the `researcher_llm` role. For production deployments requiring consistent throughput, self-hosting via a [Brev Launchable](https://brev.nvidia.com/launchable/deploy?launchableID=nvidia-official-nemotron-super-49b-v1) is recommended. Refer to [Troubleshooting](../../resources/troubleshooting.md#nemotron-super--build-endpoint-availability) for details.
```
Expand Down Expand Up @@ -279,28 +306,61 @@ query concurrently, bounded by `max_research_concurrency`. Each worker:
2. Is prompted to follow the query's preferred and fallback tool order while
remaining bound to the full request-filtered tool set
3. Returns validated `ResearchNotes` with findings, sources, gaps, and an
evidence judgment
evidence judgment. Quantitative notes may also carry bounded canonical CSV,
Markdown tables that the skill contract requires to come from the same DataFrames,
summaries, source IDs, and caveats. The runtime does not compare those tables with
the CSV text.

The batch tool returns the notes to the orchestrator and persists them under
`/shared/research_note_*.json`. If only part of a batch fails, successful
notes remain registered and persisted; only failed or missing queries are
eligible for another call.

Each researcher worker has a task-local physical `execute` budget, including
automatic retries. After exhaustion, it gets one correction turn to return
supported `ResearchNotes`; a repeated execution request ends that worker rather
than waiting for the workflow's global deadline.

Canonical CSV is parsed strictly without normalization or reserialization. AI-Q
computes its SHA-256 digest from the exact UTF-8 text, persists that value, and
registers its dataset ID, artifact path, and digest with the job-scoped artifact
manager before writer execution.
Within an artifact manifest, a ``.csv`` entry with ``kind: dataset`` is reserved
for this canonical publication contract and must include ``expected_sha256``.
Legacy or noncanonical CSV outputs remain supported through directory scanning
or a manifest entry with ``kind: table``.
Canonical publication rejects missing, unknown, or byte-mismatched identities with
the stable reasons ``canonical_dataset_digest_missing``,
``canonical_dataset_digest_unregistered``, and
``canonical_dataset_digest_mismatch``, respectively. A rejected canonical path
cannot re-enter through terminal directory scanning.
Invalid structured output receives at most two validation-feedback corrections.
If quantitative validation still fails but the textual `ResearchNotes` fields
remain valid, AI-Q drops the invalid datasets and preserves the textual evidence
instead of re-entering an unbounded repair loop.

### Phase 4: Writer-First Final Synthesis

The orchestrator delegates once the plan and research notes are available.
The writer reads `/shared/plan.json`, all research-note files, and the
captured source registry. It may also read parent-report context for a report
edit. The writer performs no new research, writes the complete final answer to
`/shared/output.md`, and returns a short completion marker. The runtime loads
the Markdown from that file.

This is the only synthesis contract. The runtime accepts only non-empty writer
output whose exact UTF-8 bytes match the digest recorded after a successful
writer mutation in the current run. After one bounded corrective turn, missing,
stale, or mismatched output fails closed with
``writer_output_not_committed``; inline orchestrator messages are not salvaged
as final reports.
`/shared/output.md`, and returns a short completion marker. For quantitative
outputs, the writer then publishes the researcher's exact CSV and may render a
chart only from that published file. Canonical dataset capture verifies the
registered digest against downloaded bytes. Chart execution is physically
bounded; failure preserves the report, Markdown table, and published CSV. The
runtime loads the Markdown from `/shared/output.md`.

The runtime first accepts only non-empty writer output whose exact UTF-8 bytes
match the digest recorded after a successful writer mutation in the current
run. Stale or mismatched output is never accepted, and inline orchestrator
messages are not salvaged as final reports. If no valid writer commit exists,
the runtime may instead build a bounded deterministic report from schema-valid
research notes. That fallback copies validated evidence, canonical Markdown
tables, summaries, caveats, and source references without rerunning analysis.
A run with neither a valid writer commit nor valid research notes fails closed
with ``writer_output_not_committed``.

``/shared/output.md`` is the sole writer-facing path. When
``CompositeBackend`` routes ``/shared/`` through ``StateBackend``, raw graph
Expand Down
11 changes: 7 additions & 4 deletions docs/source/architecture/agents/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,13 @@ and [Production Considerations](../../deployment/production.md#artifact-storage)
caps (`AIQ_MAX_SANDBOXES_PER_PRINCIPAL` / `AIQ_MAX_SANDBOXES_GLOBAL`, default-off) bound
concurrency and cost.
- Custom client-supplied job IDs must not be reused for a new job.
- Manifest checkpoints preserve completed artifacts after successful sandbox commands. The
terminal finalizer harvests once before cleanup on success/failure; cancellation harvests
only when the provider is idle and otherwise terminates immediately.
- The runtime closes provider sessions on success, failure, cancellation, and timeout.
- Writer manifest writes and successful writer chart executions checkpoint declared artifacts;
execution by other roles does not publish durable artifacts. The terminal finalizer harvests
once before cleanup on success/failure; cancellation harvests only when the provider is idle
and otherwise requests immediate termination without extending the worker's bounded wait.
- The runtime requests provider cleanup on success, failure, cancellation, and timeout. Provider
calls are bounded from the worker's perspective; a hung provider cannot be guaranteed to have
completed cleanup when that wait expires.
- Per-job OpenShell mode requires `delete_on_exit: true`. A persistent shared sandbox is
possible only through the explicit debug attachment settings.
- OpenShell provisioning and authenticated gateway lifecycle have separate owners. See the
Expand Down
23 changes: 13 additions & 10 deletions docs/source/architecture/data-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ sequenceDiagram
A->>ES: Store intermediate events
ES-->>SSE: Push events to client
SSE-->>C: SSE event data
opt Successful sandbox execute with artifact capture enabled
opt Writer publication checkpoint with artifact capture enabled
A->>AS: Checkpoint manifest-declared files
A->>ES: Store artifact.update metadata
ES-->>SSE: Push artifact.update
Expand All @@ -60,11 +60,13 @@ sequenceDiagram
SSE-->>C: job.status {status: SUCCESS}
```

Artifact checkpoints run after successful sandbox `execute` calls. On a terminal
success or failure path, the worker performs one idempotent manifest-plus-directory
scan before sandbox cleanup. Cancellation performs that scan only when the provider
operation lease is immediately available; otherwise the provider is terminated without
waiting and artifacts from earlier checkpoints remain durable.
Artifact checkpoints run after writer manifest writes and successful writer chart
executions. Researcher, planner, source-router, and orchestrator execution cannot publish
durable artifacts. On a terminal success or failure path, the worker performs one
idempotent manifest-plus-directory scan before sandbox cleanup. Cancellation performs that
scan only when the provider operation lease is immediately available; otherwise the
runtime requests provider termination without extending the bounded worker wait, and
artifacts from earlier checkpoints remain durable.

## Async Job States

Expand Down Expand Up @@ -182,10 +184,11 @@ events.
| `todo` | A research task tracked by `TodoListMiddleware` |

Durable sandbox artifacts are a separate persistence contract for generated files such as charts, CSVs,
notebooks, and documents. Capture is opt-in and best-effort. Successful `execute` calls checkpoint
manifest-declared files; success/failure terminal paths perform one final manifest-plus-directory scan, while a
busy cancellation skips that scan rather than waiting on the provider. The durable store, not the replayed event,
is authoritative for artifact records and bytes. Stored `artifact.update` events provide metadata-only live and
notebooks, and documents. Capture is opt-in and best-effort. Writer manifest writes and successful writer chart
executions checkpoint manifest-declared files; execution by other roles does not publish durable artifacts.
Success/failure terminal paths perform one final manifest-plus-directory scan, while a busy cancellation skips that
scan rather than waiting on the provider. The durable store, not the replayed event, is authoritative for artifact
records and bytes. Stored `artifact.update` events provide metadata-only live and
replayed delivery to clients, including the web UI Files tab. A rejected candidate instead emits top-level
`artifact.warning`. List authoritative metadata with `GET /v1/jobs/async/job/{job_id}/artifacts` and fetch content with
`GET /v1/jobs/async/job/{job_id}/artifacts/{artifact_id}/content`. Refer to the
Expand Down
Loading
Loading