fix(studio): reduce scope of sample agent - #1135
Merged
Merged
Conversation
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Contributor
📝 WalkthroughWalkthroughThe email security analyst sample now exposes four capabilities: ChangesEmail security analyst capability reduction
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md`:
- Around line 34-35: Update the metric-coverage statement near “Two per family”
to say that every metric flavor is represented across the suite, replacing the
inaccurate “represented once” wording while preserving the listed metric
flavors.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 359f2b46-8d08-465a-b73c-49b051347cd5
📒 Files selected for processing (10)
plugins/nemo-agents/examples/email-security-analyst/pyproject.tomlplugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/email-security-analyst-agent.ymlplugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/prompt.pyplugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/register.pyplugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/utils.pyplugins/nemo-agents/examples/email-security-analyst/tests/test_extract_iocs.pyweb/packages/studio/public/sample-agents/email-security-analyst/agent.ymlweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.mdweb/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.jsonweb/packages/studio/src/constants/sampleAgents.ts
💤 Files with no reviewable changes (5)
- plugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/prompt.py
- plugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/utils.py
- plugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/register.py
- plugins/nemo-agents/examples/email-security-analyst/tests/test_extract_iocs.py
- plugins/nemo-agents/examples/email-security-analyst/src/nat_email_security_analyst/email-security-analyst-agent.yml
walston
approved these changes
Aug 6, 2026
nv-odrulea
enabled auto-merge
August 6, 2026 19:40
nv-odrulea
disabled auto-merge
August 6, 2026 21:55
nv-odrulea
enabled auto-merge
August 6, 2026 22:03
Contributor
|
walston
added a commit
that referenced
this pull request
Aug 11, 2026
…mple
The orchestrator now fans out to specialist subagents before the verdict, so the
analysis is a traced, tunable chain instead of one opaque classification:
- url-brand-analyst - lookalike / brand-impersonation judgement on the domains
extract_iocs harvested (tool -> subagent composition)
- attack-attributor - names the attack category (bec/credential/malware/spam/benign)
- header-auth-analyst - SPF/DKIM/DMARC reading, gated on the email actually
carrying Authentication-Results:/Received: headers
phishing-analyzer stays the verdict owner and folds the findings in, adding
`attack_type` and `impersonated_brand` to the YAML verdict.
Prompts are ported from the email-security-analyst NAT example (they were removed
from it in #1135 as more protocol than that sample needed). There they were LLM
capability tools behind a router; here each is a declarative subagent, so its
reasoning gets its own trace span and its prompt is tunable in agent.yaml. The
prompt-injection guardrails are ported with them.
analyze_headers is demo-only: the scored dataset has no SMTP headers, and
synthesizing them per row would leak the phishing/benign label into the input and
inflate the eval. README shows a header-bearing email to exercise it by hand.
Verified live (--mode subprocess, real NVIDIA_API_KEY): 3/3 invocations return the
correct schema - phishing (plain + header-bearing) true, benign false - with
attack_type and impersonated_brand populated. The runtime checkpointer shows all
four subagent tasks plus the extract_iocs tool call. AgentConfig validate +
Fabric translate OK; ruff clean; 8/8 tool tests.
Co-authored-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Nathan Walston <nwalston@nvidia.com>
ryana
pushed a commit
to ryana/nemo-platform
that referenced
this pull request
Aug 12, 2026
…y Triage (NVIDIA-NeMo#1222) * feat(nemo-agents): add specialist subagents to the email-phishing example The orchestrator now fans out to specialist subagents before the verdict, so the analysis is a traced, tunable chain instead of one opaque classification: - url-brand-analyst - lookalike / brand-impersonation judgement on the domains extract_iocs harvested (tool -> subagent composition) - attack-attributor - names the attack category (bec/credential/malware/spam/benign) - header-auth-analyst - SPF/DKIM/DMARC reading, gated on the email actually carrying Authentication-Results:/Received: headers phishing-analyzer stays the verdict owner and folds the findings in, adding `attack_type` and `impersonated_brand` to the YAML verdict. Prompts are ported from the email-security-analyst NAT example (they were removed from it in NVIDIA-NeMo#1135 as more protocol than that sample needed). There they were LLM capability tools behind a router; here each is a declarative subagent, so its reasoning gets its own trace span and its prompt is tunable in agent.yaml. The prompt-injection guardrails are ported with them. analyze_headers is demo-only: the scored dataset has no SMTP headers, and synthesizing them per row would leak the phishing/benign label into the input and inflate the eval. README shows a header-bearing email to exercise it by hand. Verified live (--mode subprocess, real NVIDIA_API_KEY): 3/3 invocations return the correct schema - phishing (plain + header-bearing) true, benign false - with attack_type and impersonated_brand populated. The runtime checkpointer shows all four subagent tasks plus the extract_iocs tool call. AgentConfig validate + Fabric translate OK; ruff clean; 8/8 tool tests. Co-authored-by: Octavian Drulea <odrulea@nvidia.com> Signed-off-by: Nathan Walston <nwalston@nvidia.com> * docs(nemo-agents): address CodeRabbit review on the specialist example - agent.yaml: require lowercase impersonated_brand in the phishing-analyzer verdict schema too (the orchestrator copy was already lowercase; the sub-agent copy wasn't - that mismatch is why live output flip-flopped PayPal vs paypal). - README Step 2: phishing-analyzer owns the verdict and emits every field; the attack_type/impersonated_brand values are specialist-informed (advisory), not produced by the specialists instead of the classifier. - README Step 3: fix a self-contradictory synthetic header example - it had spf=fail but dkim=pass/dmarc=pass, which actually authenticates From: paypal.com. Make all three fail and lead the explanation with dmarc=fail on header.from, the check tied to the visible From: domain. - README Step 4: traces carry a task per *applicable* specialist; header-auth-analyst appears only for header-bearing input (the Step 2 email has none). Co-authored-by: Octavian Drulea <odrulea@nvidia.com> Signed-off-by: Nathan Walston <nwalston@nvidia.com> * refactor(nemo-agents): rename example email-phishing-agent -> email-security-triage With the specialist sub-agents this example is broader than "phishing" — it's an email-security triage orchestrator — so rename it to match (display name "Email Security Triage", slug `email-security-triage`). Folded into this PR because the specialists are the reason for the rename. Full identifier sweep (collision-checked vs the still-present NAT `email-security-analyst` and `email-phishing-analyzer`): - example dir, agent `name`, telemetry project - package `nemo-agent-config-example-email-phishing` -> `...-email-security-triage` - console script / MCP url `email-phishing-iocs` -> `email-security-triage-iocs` (MCP server key stays `iocs`; module stays `mcps/iocs.py`) - eval config `email-phishing-eval.yml` -> `email-security-triage-eval.yml` - default deployment name in the README - Studio registry key/namePrefix/displayName/agentConfigPath + sample dir - README/CUSTOMIZE branding; re-synced the Studio sample copy to this branch's (specialist) agent.yaml Unchanged on purpose: the `is_likely_phishing` verdict schema, the `phishing-analyzer` sub-agent (it IS the phishing specialist), and domain references to phishing. Verified: AgentConfig validate + Fabric translate; ruff; 8/8 tool tests; Studio loader tests; zero stragglers; and live create -> deploy -> invoke under the new slug returns is_likely_phishing: true (renamed console script resolves). Signed-off-by: Nathan Walston <nwalston@nvidia.com> * docs(nemo-agents): name both delegated roles in triage goal Signed-off-by: Nathan Walston <nwalston@nvidia.com> --------- Signed-off-by: Nathan Walston <nwalston@nvidia.com> Co-authored-by: Octavian Drulea <odrulea@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
email-security-analystis Studio's showcase sample agent. It currently ships 11 tools and a 22-task eval suite.For a "hello world" demo that is too much surface. A newcomer opening register.py sees 346 lines and 11 near-identical registrars; a newcomer opening
eval-config.task-driven.jsonsees 1619 lines. The diversity that matters for the demo is metric diversity, not tool count, and all four metric flavors survive a much smaller tool set.Outcome: 4 tools, 10 tasks, 5 views. Pure deletion. Every surviving task and metric block copies over byte-for-byte. Nothing is authored from scratch. The dataset-driven eval path is untouched.
Related Issue
https://linear.app/nvidia/issue/ASTD-319
Summary by CodeRabbit
New Features
Changes
Documentation