feat(experiments): filter evaluations by agent name, agent version, or model name - #1160
Merged
shanaiabuggy merged 10 commits intoAug 10, 2026
Conversation
Add a debounced background worker (EvaluationDenormalizer) that materializes each evaluation's distinct agent_names/agent_versions/model_names from the ClickHouse rollup onto its Evaluation entity, so the list can filter by name against the entity store. Ingest marks (workspace, evaluation_id) dirty; the worker coalesces and drains on an interval, skips no-op writes, and drains gracefully on shutdown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
…el name Expose agent_name/agent_version/model_name filters on the evaluations list, rewriting each scalar equality into a $contains match over the denormalized list fields (mirroring experiment_id membership). Also self-heal on read: when a read's live rollup names differ from the stored fields, enqueue the evaluation for the denormalizer, backfilling pre-existing data as it is viewed with no migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
…ters Regenerated from the new EvaluationFilter fields via make refresh-openapi + make stainless. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Add free-text agent name, agent version, and model filter inputs to the evaluations list, mapped to the filter[agent_name]/[agent_version]/[model_name] API params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (20)
🚧 Files skipped from review as they are similar to previous changes (15)
📝 WalkthroughWalkthroughEvaluation entities now store observed agent and model facets. Intake refreshes these facets asynchronously from ClickHouse, API filters support them, and Studio exposes corresponding text filters. ChangesEvaluation facet denormalization
Sequence Diagram(s)sequenceDiagram
participant SpanIngest
participant EvaluationDenormalizer
participant ClickHouseRollup
participant EvaluationEntity
SpanIngest->>EvaluationDenormalizer: mark_dirty(workspace, evaluation_id)
EvaluationDenormalizer->>ClickHouseRollup: fetch workspace rollups
ClickHouseRollup-->>EvaluationDenormalizer: observed agent and model facets
EvaluationDenormalizer->>EvaluationEntity: update denormalized facet lists
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
services/intake/tests/test_evaluation_facet_filter.py (1)
90-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd endpoint coverage for
filter[agent_version].The rewrite unit test bypasses public query parsing. Add a persisted
agent_versionsvalue and an endpoint request forfilter[agent_version].🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@services/intake/tests/test_evaluation_facet_filter.py` around lines 90 - 103, Extend the evaluation fixture setup to persist an agent_versions value, then add a public client request using filter[agent_version] and assert it returns the expected evaluation. Keep the existing agent_name, model_name, and unmatched-filter assertions unchanged.services/intake/tests/test_evaluation_denormalizer.py (1)
6-6: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove
from __future__ import annotationsfrom all three test modules.Their annotations use imported, local, or built-in types and do not require postponement.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@services/intake/tests/test_evaluation_denormalizer.py` at line 6, Remove the from __future__ import annotations statement from services/intake/tests/test_evaluation_denormalizer.py:6, services/intake/tests/test_evaluation_denormalizer_self_heal.py:11, and services/intake/tests/test_evaluation_facet_filter.py:11; no other changes are needed.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@services/intake/tests/test_evaluation_denormalizer.py`:
- Line 6: Remove the from __future__ import annotations statement from
services/intake/tests/test_evaluation_denormalizer.py:6,
services/intake/tests/test_evaluation_denormalizer_self_heal.py:11, and
services/intake/tests/test_evaluation_facet_filter.py:11; no other changes are
needed.
In `@services/intake/tests/test_evaluation_facet_filter.py`:
- Around line 90-103: Extend the evaluation fixture setup to persist an
agent_versions value, then add a public client request using
filter[agent_version] and assert it returns the expected evaluation. Keep the
existing agent_name, model_name, and unmatched-filter assertions unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f53378f5-d393-49db-bcd6-f3260ed7a384
⛔ Files ignored due to path filters (3)
sdk/python/nemo-platform/.nmpcontext/openapi.yamlis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/types/evaluations/evaluation_filter_param.pyis excluded by!sdk/**sdk/python/nemo-platform/tests/api_resources/test_evaluations.pyis excluded by!sdk/**
📒 Files selected for processing (17)
openapi/ga/individual/platform.openapi.yamlopenapi/ga/openapi.yamlopenapi/openapi.yamlservices/intake/src/nmp/intake/api/v2/experiments/dependencies.pyservices/intake/src/nmp/intake/api/v2/experiments/endpoints.pyservices/intake/src/nmp/intake/api/v2/experiments/schemas.pyservices/intake/src/nmp/intake/config.pyservices/intake/src/nmp/intake/entities/experiments.pyservices/intake/src/nmp/intake/experiments/denormalizer.pyservices/intake/src/nmp/intake/service.pyservices/intake/src/nmp/intake/spans/api/dependencies.pyservices/intake/src/nmp/intake/spans/ingest/atif.pyservices/intake/src/nmp/intake/spans/ingest/chat_completions.pyservices/intake/tests/test_evaluation_denormalizer.pyservices/intake/tests/test_evaluation_denormalizer_self_heal.pyservices/intake/tests/test_evaluation_facet_filter.pyweb/packages/studio/src/components/dataViews/ExperimentDataView/index.tsx
shanaiabuggy
commented
Aug 7, 2026
…ame-agent-version-model-name Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
|
shanaiabuggy
commented
Aug 7, 2026
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@services/intake/src/nmp/intake/debounced_refresher.py`:
- Around line 54-62: The stop method’s final drain can leave keys queued by
flush processing. Update DebouncedRefresher.stop to continue draining after the
task completes, repeating flush while _dirty contains pending keys; if using a
bounded retry policy instead, report any keys remaining after the limit.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 814c1c3e-d267-4b0e-893a-9fb5594de492
📒 Files selected for processing (2)
services/intake/src/nmp/intake/debounced_refresher.pyservices/intake/src/nmp/intake/experiments/denormalizer.py
…ame-agent-version-model-name Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
BrianNewsom
reviewed
Aug 10, 2026
Signed-off-by: shanaiabuggy <59746633+shanaiabuggy@users.noreply.github.com>
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
BrianNewsom
approved these changes
Aug 10, 2026
shanaiabuggy
deleted the
sbuggy/ase-809-filter-evaluations-by-agent-name-agent-version-model-name
branch
August 10, 2026 23:18
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the ability to filter Evaluations by agent name, agent version, or model name. These values are ClickHouse-derived rollups that were only exposed for display; there was no way to filter the list by them. Filtering them live at workspace scale would mean scanning the whole session table on every request, so instead we denormalize the distinct name sets onto the Evaluation entity (in the entity store) and filter there with the store's existing
$containsoperator. The entity's numeric rollups (cost/latency/tokens/scores) are intentionally not denormalized — only the names, which have no formula and so can't go stale the way a computed aggregate can.Today the filter surfaces in Studio on the (experiment-scoped) evaluations list, but the API/SDK support it workspace-wide.
Related Issue
Relates to ASE-809 — https://linear.app/nvidia/issue/ASE-809 (Linear; auto-linked via branch name).
Changes
Four commits, each self-contained:
feat(intake): denormalize agent/model names onto Evaluation entities— a debounced background worker (EvaluationDenormalizer) that materializes each evaluation's distinctagent_names/agent_versions/model_namesfrom the ClickHouse rollup onto its entity. Ingest marks(workspace, evaluation_id)dirty; the worker coalesces, drains on an interval, skips no-op writes, and drains gracefully on shutdown.feat(intake): filter evaluations by agent name, agent version, or model name— newagent_name/agent_version/model_namefilters that rewrite a scalar equality into a$containsmatch over the denormalized list fields (mirroringexperiment_idmembership). Plus self-heal on read: when a read's live rollup names differ from the stored fields, the evaluation is enqueued for the denormalizer, so pre-existing data backfills as it's viewed with no migration to run on any instance.chore(sdk): regenerate OpenAPI and Python SDK— regenerated from the newEvaluationFilterfields.feat(studio): add agent/model name filters to the evaluations list— free-text filter inputs mapped to the new API params.Type of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation (only targeted checks were run locally, not the full pre-commit sweep):
uv run --frozen pytest services/intake/tests -k "denormalizer or facet or experiment or evaluation or ingest or crud"→ 231 passeduv run --frozen ty check <changed intake files>→ no new diagnostics (2 reported errors pre-exist on main, unrelated)uv run ruff check→ cleanpnpm --filter nemo-studio-ui typecheck && lint && test ExperimentDataView→ pass, 30 tests🤖 Generated with Claude Code
Summary by CodeRabbit