Add trace-based behavioral tests with Monocle Test Tools - #328
imohammedansari wants to merge 1 commit into
Conversation
WalkthroughAdds a default OpenAI CLI research workflow and a Monocle test suite with offline trace replay, opt-in live execution, recorded traces, dependencies, documentation, and generated-artifact ignore rules. ChangesOpenAI CLI and Monocle testing
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Test as test_nvidiaaiq
participant Runner as run_nvidiaaiq
participant Workflow as chat_deepresearcher_agent
participant Search as web_search_tool
participant Assertions as Monocle trace assertions
Test->>Runner: Run stock-price query
Runner->>Workflow: Execute configured workflow
Workflow->>Search: Search rewritten stock query
Search-->>Workflow: Return market documents
Workflow-->>Runner: Return cited response
Runner-->>Test: Return workflow result
Test->>Assertions: Validate agent, tool, output, and duration
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/monocle/requirements.txt (1)
1-6: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winPin
tests/monocle/requirements.txt
monocle_test_toolsandpython-dotenvare floating here. Pin them so the Monocle test environment stays reproducible;monocle_test_toolscurrently resolves a broad transitive set (gitpython==3.1.45,sentence-transformers==3.3.0,transformers>=4.0.0, etc.), so leaving it unpinned can drift into different or vulnerable releases over time.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/monocle/requirements.txt` around lines 1 - 6, Pin both dependencies in tests/monocle/requirements.txt by assigning explicit, reviewed versions to monocle_test_tools and python-dotenv, using compatible versions for the existing Monocle test suite and its transitive requirements.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.json`:
- Line 18: Remove contributor-specific absolute filesystem paths from every
span_source entry in the referenced trace fixture, including all listed
occurrences. Replace the local path prefix with a neutral placeholder while
preserving the package/module and location suffix needed by the fixture.
---
Outside diff comments:
In `@tests/monocle/requirements.txt`:
- Around line 1-6: Pin both dependencies in tests/monocle/requirements.txt by
assigning explicit, reviewed versions to monocle_test_tools and python-dotenv,
using compatible versions for the existing Monocle test suite and its transitive
requirements.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d5cec655-dfc1-4161-9b44-726ade818001
📒 Files selected for processing (8)
.gitignoreconfigs/config_openai_cli.ymltests/monocle/README.mdtests/monocle/conftest.pytests/monocle/requirements.txttests/monocle/test_nvidiaaiq.pytests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.jsontests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.json
📜 Review details
🧰 Additional context used
📓 Path-based instructions (4)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
tests/monocle/requirements.txttests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.jsontests/monocle/README.mdtests/monocle/conftest.pytests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.jsonconfigs/config_openai_cli.ymltests/monocle/test_nvidiaaiq.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
tests/monocle/conftest.pytests/monocle/test_nvidiaaiq.py
**/*test*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run pytest for all behavior changes in Python code
Files:
tests/monocle/conftest.pytests/monocle/test_nvidiaaiq.py
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_openai_cli.yml
🪛 OSV Scanner (2.4.0)
tests/monocle/requirements.txt
[HIGH] 1-1: gitpython 3.1.45: GitPython reference APIs has a path traversal vulnerability that allows arbitrary file write and delete outside the repository
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer() section parameter bypasses CVE-2026-42215 patch, enabling RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython has Command Injection via Git options bypass
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer().set_value() enables RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython: Unsafe option check validates multi_options before shlex.split transformation
[HIGH] 1-1: torch 2.9.1: undefined
(PYSEC-2026-139)
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.lstm_cell function
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.jit.script function
[HIGH] 1-1: transformers 4.57.6: undefined
(PYSEC-2025-217)
[HIGH] 1-1: transformers 4.57.6: HuggingFace transformers vulnerable to remote code execution
[HIGH] 1-1: transformers 4.57.6: HuggingFace Transformers allows for arbitrary code execution in the Trainer class
🔇 Additional comments (8)
tests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.json (1)
18-18: 🔒 Security & Privacy | ⚡ Quick winSame local-path leak as the sibling trace fixture.
Same
span_sourcelocal-path leak (contributor username + directory layout) as flagged inmonocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_...json. Apply the same redaction here.Also applies to: 52-52, 113-113, 168-168, 221-221
configs/config_openai_cli.yml (1)
1-130: LGTM!tests/monocle/conftest.py (1)
7-57: LGTM! The double-import guard and lazy NAT imports are sensible for a mixed offline/live test scaffold.tests/monocle/test_nvidiaaiq.py (3)
27-30: 🩺 Stability & Availability | ⚖️ Poor tradeoffGeneric
conftestmodule name risk with manualsys.pathimport.Using a bare
import conftestaftersys.path.insertworks today because this is the only file doing it, but it's fragile: Python caches modules by name insys.modules, so if any other test directory in this repo (or added later) performs the same "insert own dir +import conftest" pattern, whichever import runs first wins and later imports silently reuse the wrong module. The double-instrumentation guard inconftest.py(Lines 22-25) already anticipates a second import of this file, but a name collision with an unrelatedconftest.pyelsewhere is a different failure mode it doesn't protect against.Consider importing via
importlib.util.spec_from_file_locationwith a unique module name, or extractingTRACES/run_nvidiaaiqinto a uniquely-named helper module instead of relying on pytest's ownconftest.pydiscovery file.
42-91: LGTM! Offline assertions match the recorded trace fixtures (durations, tool calls, outputs all line up).
93-104: 🎯 Functional CorrectnessMonocle fixture/API usage matches the suite.
> Likely an incorrect or invalid review comment.tests/monocle/README.md (1)
1-74: LGTM!.gitignore (1)
43-45: LGTM!
Trace-based tests under tests/monocle/ asserting against the agent's Monocle execution traces (agent, tools, output, duration budget). Signed-off-by: Mohammed Ansari <mohammed.ansari@okahu.ai>
efd4bb3 to
207f4b9
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/monocle/conftest.py`:
- Around line 49-52: Remove the broad exception suppression around
ContextState.get().conversation_id.set(...) in the fixture setup; allow failures
to propagate so the fixture fails visibly, or explicitly log the exception
before handling it.
In `@tests/monocle/README.md`:
- Around line 45-49: Update the live-test documentation near the
`RUN_LIVE_NVIDIAAIQ` instructions to require `EXA_API_KEY` only, matching the
active `exa_web_search` configuration in `configs/config_openai_cli.yml`; remove
`SERPER_API_KEY` as an alternative, or instead change the workflow and maintain
documentation parity for the configured search provider.
In `@tests/monocle/requirements.txt`:
- Around line 1-5: Pin monocle_test_tools and its monocle_apptrace dependency in
tests/monocle/requirements.txt, either with compatible explicit versions or a
constraints file referenced by the requirements. Preserve the existing pytest
and environment-loading dependencies while ensuring repeated installs use
versions compatible with the recorded trace fixtures.
In `@tests/monocle/test_nvidiaaiq.py`:
- Around line 87-90: The _LIVE skip marker currently checks only
RUN_LIVE_NVIDIAAIQ; update it to also skip unless OPENAI_API_KEY and the
configured EXA_API_KEY environment variables are present, without logging or
exposing secret values. Do not use SERPER_API_KEY as a substitute, and ensure
missing credentials result in a graceful skip rather than workflow or
tool-execution failure.
- Around line 101-103: The output assertion in the NVIDIA test is too broad and
can accept error messages. Update the assertion near
monocle_trace_asserter.called_agent("LangGraph") to require stable success
indicators, including a citation section and a currency/amount pattern, while
leaving the exact market value unconstrained; retain the existing agent and tool
assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 2ff52581-21f8-4425-8cb4-d60f27dcd2a4
📒 Files selected for processing (8)
.gitignoreconfigs/config_openai_cli.ymltests/monocle/README.mdtests/monocle/conftest.pytests/monocle/requirements.txttests/monocle/test_nvidiaaiq.pytests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.jsontests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.json
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: CodeRabbit / Review
🧰 Additional context used
📓 Path-based instructions (4)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
tests/monocle/requirements.txttests/monocle/README.mdconfigs/config_openai_cli.ymltests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.jsontests/monocle/conftest.pytests/monocle/test_nvidiaaiq.pytests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.json
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_openai_cli.yml
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
tests/monocle/conftest.pytests/monocle/test_nvidiaaiq.py
**/*test*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run pytest for all behavior changes in Python code
Files:
tests/monocle/conftest.pytests/monocle/test_nvidiaaiq.py
🪛 OSV Scanner (2.4.0)
tests/monocle/requirements.txt
[HIGH] 1-1: gitpython 3.1.45: GitPython reference APIs has a path traversal vulnerability that allows arbitrary file write and delete outside the repository
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer() section parameter bypasses CVE-2026-42215 patch, enabling RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython has Command Injection via Git options bypass
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer().set_value() enables RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython: Unsafe option check validates multi_options before shlex.split transformation
[HIGH] 1-1: torch 2.9.1: undefined
(PYSEC-2026-139)
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.lstm_cell function
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.jit.script function
[HIGH] 1-1: transformers 4.57.6: undefined
(PYSEC-2025-217)
[HIGH] 1-1: transformers 4.57.6: HuggingFace transformers vulnerable to remote code execution
[HIGH] 1-1: transformers 4.57.6: HuggingFace Transformers allows for arbitrary code execution in the Trainer class
🔇 Additional comments (8)
configs/config_openai_cli.yml (2)
68-76: 🩺 Stability & AvailabilityVerify graceful behavior when
EXA_API_KEYis absent.Both Exa tools are enabled by default, while the supplied validation contract only checks LLM keys. If the tool does not already stub or skip without
EXA_API_KEY, a normal CLI run can fail at tool invocation. Add an explicit guard/fallback or a test covering the missing-key path.As per coding guidelines, missing-secret paths must degrade gracefully (stub/skip), not crash. As per path instructions, config changes must be reviewed for secret separation and safe defaults.
Sources: Coding guidelines, Path instructions
1-67: LGTM!Also applies to: 77-130
tests/monocle/conftest.py (1)
1-48: LGTM!Also applies to: 53-58
tests/monocle/test_nvidiaaiq.py (1)
1-86: LGTM!Also applies to: 91-100, 104-110
tests/monocle/traces/monocle_trace_nvidia-aiq_e2961458302a780a27f91a0d86b290dd_2026-07-09_13.25.14.json (1)
1-547: LGTM!tests/monocle/README.md (1)
1-44: LGTM!Also applies to: 50-74
tests/monocle/traces/monocle_trace_nvidia-aiq_4a94876861e3ade549dc74fb785e94d9_2026-07-09_13.17.46.json (1)
1-241: LGTM!.gitignore (1)
43-45: 📐 Maintainability & Code Quality | ⚡ Quick winAlso isolate or ignore workflow checkpoint artifacts.
The documented live command uses
checkpoint_db: ./checkpoints.db, so it may leavecheckpoints.dband SQLite sidecar files in the checkout. This block ignores only.monocle/; use a temporary checkpoint path for live tests or add an ignore pattern such ascheckpoints.db*.As per path instructions, config changes must be reviewed for safe local-vs-production behavior.
Source: Path instructions
| try: | ||
| ContextState.get().conversation_id.set(str(uuid.uuid4())) | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== File context =="
sed -n '1,140p' tests/monocle/conftest.py
echo
echo "== Search for ContextState usages =="
rg -n "ContextState|get\(\)\.conversation_id|conversation_id\.set|conversation_id" tests src sources -g '!**/.venv/**' -g '!**/node_modules/**' || true
echo
echo "== Search for monocle fixtures/tests =="
rg -n "monocle|run_nvidiaaiq|conversation_id" tests -g '!**/.venv/**' || trueRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 23860
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Monocle README =="
sed -n '1,140p' tests/monocle/README.md
echo
echo "== Monocle test file =="
sed -n '1,180p' tests/monocle/test_nvidiaaiq.py
echo
echo "== Search for conversation-id docs in repo =="
rg -n "conversation-id|conversation_id|ContextState|checkpoint|state leakage|thread ID" tests/monocle docs/source src sources -g '!**/.venv/**' -g '!**/node_modules/**' || trueRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 20851
🌐 Web query:
NeMo Agent Toolkit ContextState.get optional-context exception conversation_id set documented failure mode
💡 Result:
In the NVIDIA NeMo Agent Toolkit, ContextState is implemented as a singleton that manages context variables using contextvars [1][2]. The method ContextState.get() is used to retrieve the global ContextState instance [3][2][4]. There is no documented "failure mode" specifically for ContextState.get() regarding an "optional-context" exception or conversation_id. Based on the toolkit's architecture: 1. Context Management: ContextState provides thread-local or asynchronous-task-local storage for metadata such as conversation_id, user_id, and workflow_run_id [5][3][1]. 2. Property Access: The conversation_id property typically calls self._context_state.conversation_id.get() [5]. If the conversation_id has not been set in the current execution context, contextvars.ContextVar.get() will return its default value (which is None unless otherwise specified) [5][1]. It does not inherently raise an exception [5]. 3. Error Handling: While there are no exceptions specifically tied to accessing an unset conversation_id, the toolkit has implemented rigorous error propagation patterns in other areas (such as agent registration and runner result logging) to avoid "swallowing" exceptions [6]. If you are encountering an unexpected failure, it may be related to how downstream logic expects a non-null conversation_id or how ContextState is initialized within your specific runtime environment (e.g., inside an async with session(...) block) [3][1]. If you are experiencing an error, verify that the conversation_id has been explicitly set using context_state.conversation_id.set("your-id") within your session scope [3][1]. If you are working with custom components, ensure your usage conforms to the standard ContextState access patterns found in the nat.builder.context module [7][5][2].
Citations:
- 1: https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/68ce9b1f/packages/nvidia_nat_core/tests/nat/runtime/test_session_manager.py
- 2: https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/99e07260/packages/nvidia_nat_core/src/nat/builder/context.py
- 3: https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/f33952bf/packages/nvidia_nat_core/src/nat/runtime/session.py
- 4: https://docs.nvidia.com/nemo/agent-toolkit/latest/extend/telemetry-exporters.html
- 5: https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/f33952bf/packages/nvidia_nat_core/src/nat/builder/context.py
- 6: Fix swallowing client errors bug NVIDIA/NeMo-Agent-Toolkit#841
- 7: https://docs.nvidia.com/nemo/agent-toolkit/1.2/api/nat/builder/context/index.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Live workflow config =="
sed -n '1,220p' configs/config_openai_cli.yml
echo
echo "== Search for checkpoint / conversation wiring in configs =="
rg -n "checkpoint|conversation-id|conversation_id|thread_id|ContextState" configs src tests -g '!**/.venv/**' -g '!**/node_modules/**' || trueRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 21651
Don't silence context setup failures here. If ContextState.get().conversation_id.set(...) raises, let the fixture fail or log it explicitly so the live trace doesn't silently lose its per-run conversation ID.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/monocle/conftest.py` around lines 49 - 52, Remove the broad exception
suppression around ContextState.get().conversation_id.set(...) in the fixture
setup; allow failures to propagate so the fixture fails visibly, or explicitly
log the exception before handling it.
| The live test is opt-in (`RUN_LIVE_NVIDIAAIQ=1`) and skipped by default. | ||
| NAT binds its async singletons to the first event loop, so only one in-process | ||
| live run works per process, and NAT leaves non-daemon threads that keep the | ||
| interpreter from exiting cleanly. So run the live test in its own process, with | ||
| keys in `deploy/.env` (`OPENAI_API_KEY` plus a search key, `EXA_API_KEY`/`SERPER_API_KEY`): |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Document the key required by the configured live workflow.
configs/config_openai_cli.yml enables _type: exa_web_search; SERPER_API_KEY is only associated with the commented paper-search configuration. Listing it as an alternative can lead users to run the live test without the required Exa credential. Document EXA_API_KEY only, or change the workflow to use Serper.
As per path instructions, config changes must preserve documentation parity.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/monocle/README.md` around lines 45 - 49, Update the live-test
documentation near the `RUN_LIVE_NVIDIAAIQ` instructions to require
`EXA_API_KEY` only, matching the active `exa_web_search` configuration in
`configs/config_openai_cli.yml`; remove `SERPER_API_KEY` as an alternative, or
instead change the workflow and maintain documentation parity for the configured
search provider.
Source: Path instructions
| # Installing monocle_test_tools pulls in everything this suite needs | ||
| # (pytest, pytest-asyncio, and monocle_apptrace come transitively). | ||
| monocle_test_tools | ||
| # Auto-loads the repo deploy/.env for the live tests. | ||
| python-dotenv |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== tests/monocle/requirements.txt ==\n'
cat -n tests/monocle/requirements.txt
printf '\n== Nearby monocle files ==\n'
git ls-files 'tests/monocle/*' | sed 's#^`#-` #'
printf '\n== Search for monocle_test_tools / monocle_apptrace / python-dotenv / constraints ==\n'
rg -n --hidden --glob '!**/.git/**' \
'monocle_test_tools|monocle_apptrace|python-dotenv|requirements\.txt|constraints|lockfile|uv\.lock|poetry\.lock|pip-tools|pip compile' \
tests sources src docs configs .agents skills frontends deploy .
printf '\n== Dependency files that may constrain tests/monocle ==\n'
fd -H -a 'requirements.*|uv\.lock|pyproject\.toml|constraints.*|lock.*' tests . | sed 's#^`#-` #'
printf '\n== tests/monocle tree ==\n'
find tests/monocle -maxdepth 2 -type f | sort | sed 's#^`#-` #'Repository: NVIDIA-AI-Blueprints/aiq
Length of output: 29858
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== tests/monocle/README.md ==\n'
cat -n tests/monocle/README.md
printf '\n== tests/monocle/conftest.py ==\n'
cat -n tests/monocle/conftest.py
printf '\n== tests/monocle/test_nvidiaaiq.py ==\n'
cat -n tests/monocle/test_nvidiaaiq.py
printf '\n== monocle-related entries in uv.lock ==\n'
python3 - <<'PY'
from pathlib import Path
text = Path("uv.lock").read_text()
for needle in ["monocle_test_tools", "monocle_apptrace", "python-dotenv"]:
print(f"\n-- {needle} --")
for i, line in enumerate(text.splitlines(), 1):
if needle in line:
print(f"{i}: {line}")
PYRepository: NVIDIA-AI-Blueprints/aiq
Length of output: 13807
Pin the Monocle test deps. requirements.txt is installed directly with pip install -r, so monocle_test_tools can drift and pull a different monocle_apptrace, breaking the recorded trace fixtures. Add version pins or a constraints file for this suite.
🧰 Tools
🪛 OSV Scanner (2.4.0)
[HIGH] 1-1: gitpython 3.1.45: GitPython reference APIs has a path traversal vulnerability that allows arbitrary file write and delete outside the repository
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer() section parameter bypasses CVE-2026-42215 patch, enabling RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython has Command Injection via Git options bypass
[HIGH] 1-1: gitpython 3.1.45: GitPython: Newline injection in config_writer().set_value() enables RCE via core.hooksPath
[HIGH] 1-1: gitpython 3.1.45: GitPython: Unsafe option check validates multi_options before shlex.split transformation
[HIGH] 1-1: torch 2.9.1: undefined
(PYSEC-2026-139)
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.lstm_cell function
[HIGH] 1-1: torch 2.9.1: PyTorch is vulnerable to memory corruption through its torch.jit.script function
[HIGH] 1-1: transformers 4.57.6: undefined
(PYSEC-2025-217)
[HIGH] 1-1: transformers 4.57.6: HuggingFace transformers vulnerable to remote code execution
[HIGH] 1-1: transformers 4.57.6: HuggingFace Transformers allows for arbitrary code execution in the Trainer class
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/monocle/requirements.txt` around lines 1 - 5, Pin monocle_test_tools
and its monocle_apptrace dependency in tests/monocle/requirements.txt, either
with compatible explicit versions or a constraints file referenced by the
requirements. Preserve the existing pytest and environment-loading dependencies
while ensuring repeated installs use versions compatible with the recorded trace
fixtures.
Source: Linters/SAST tools
| _LIVE = pytest.mark.skipif( | ||
| os.environ.get("RUN_LIVE_NVIDIAAIQ") != "1", | ||
| reason="opt-in live run (set RUN_LIVE_NVIDIAAIQ=1; run one live test per process -- see note above)", | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Skip the live test when required credentials are unavailable.
The marker checks only RUN_LIVE_NVIDIAAIQ. Enabling it without OPENAI_API_KEY or the configured EXA_API_KEY can make workflow loading or tool execution fail instead of skipping safely. Gate the test on those environment variables without exposing their values; SERPER_API_KEY is not a substitute for this Exa-backed configuration.
As per coding guidelines, missing-secret paths must degrade gracefully (stub/skip), not crash.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/monocle/test_nvidiaaiq.py` around lines 87 - 90, The _LIVE skip marker
currently checks only RUN_LIVE_NVIDIAAIQ; update it to also skip unless
OPENAI_API_KEY and the configured EXA_API_KEY environment variables are present,
without logging or exposing secret values. Do not use SERPER_API_KEY as a
substitute, and ensure missing credentials result in a graceful skip rather than
workflow or tool-execution failure.
Source: Coding guidelines
| monocle_trace_asserter.called_agent("LangGraph") | ||
| monocle_trace_asserter.contains_any_output("NVIDIA", "NVDA", "stock", "price") | ||
| monocle_trace_asserter.called_tool("web_search_tool", "LangGraph") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the live assertion reject error responses.
contains_any_output("NVIDIA", "NVDA", "stock", "price") can pass for an error such as “I couldn’t retrieve the stock price.” Assert stable success markers, such as a citation section plus a currency/amount pattern, while keeping the exact market value unconstrained.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/monocle/test_nvidiaaiq.py` around lines 101 - 103, The output assertion
in the NVIDIA test is too broad and can accept error messages. Update the
assertion near monocle_trace_asserter.called_agent("LangGraph") to require
stable success indicators, including a citation section and a currency/amount
pattern, while leaving the exact market value unconstrained; retain the existing
agent and tool assertions.
|
Thanks for the companion testing proposal. The goal of catching agent-behavior regressions from structured traces is valuable, but AIQ’s trace contract has changed since this was opened. The two offline tests here also replay fixed trace files, so they validate those fixtures rather than execute the current application. Only the opt-in live test exercises AIQ, and the suite is not integrated into the repository’s locked test environment or CI. If we add more assembled-agent behavioral coverage, it should be Relay-native: ideally a deterministic current-workflow run that generates ephemeral ATOF/OTEL and asserts semantic agent and tool behavior without adding a second instrumentation runtime. If Monocle Test Tools can consume Relay ATOF or OTLP/OpenInference directly, please point us to that support and we can evaluate a consumer-only approach. Given the current architecture, we’re going to close this PR alongside #327 as superseded. Thank you again for the contribution and for demonstrating the testing idea. |
2 participants
Summary
Adds a behavioral test suite that asserts against the AI-Q research agent's Monocle execution traces: which agent ran, which tools it called, what it returned, and its duration. Additive only, under
tests/monocle/, with no app-code changes.Why
The existing suite (100+ files under
tests/) is thorough at the unit layer but never runs the assembled agent to check what a real run actually did. This adds that behavioral coverage, so if a later prompt, tool, or route change alters routing, tool calls, or latency, a test catches it.How it works
It uses Monocle Test Tools. The offline tests load a recorded trace from file with
with_trace_source("file", trace_path=...), which is fast, needs no keys, and is deterministic. They then assert with the fluent API:called_agent,called_tool,contains_input/contains_output,under_duration. The live test drives the real NAT workflow (viaconfigs/config_openai_cli.yml, with an auto-approving HITL callback so a clarification or plan-approval interrupt does not block) and asserts on structure and budget only.The point of loading from file is that the same assertions run in both modes. You pin a check against a recorded trace while iterating (fast, no keys, deterministic), then run the identical check against a live run for real regression coverage.
NAT traces carry no token metadata (inference spans record only finish reasons), so budgets are duration-based. The live test is opt-in (
RUN_LIVE_NVIDIAAIQ=1) and runs one at a time, since NAT binds its asyncio singletons to the first event loop.Changes (all under
tests/monocle/)test_nvidiaaiq.py: 2 offline file-loaded tests (a capabilities intro answered directly; an NVIDIA stock-price lookup viaweb_search_tool), plus 1 live test (stock-price lookup, opt-in viaRUN_LIVE_NVIDIAAIQ=1).conftest.py: Monocle setup,.envloading,run_nvidiaaiq()(auto-approve HITL).traces/: 2 recorded traces.configs/config_openai_cli.yml,requirements.txt,README.md.Summary by CodeRabbit
New Features
Documentation
Tests
Chores
PS: if Monocle looks useful, a ⭐ helps the project (https://github.com/monocle2ai/monocle). The instrumentation that emits these traces is the companion PR (#327).