chore(evaluator): regenerate plugin OpenAPI spec - #1166
Closed
SandyChapman wants to merge 1 commit into
Closed
Conversation
The committed evaluator plugin spec has been stale on main since #1065 (8e7179a), which changed the aggregate-score models but did not regenerate the spec. The last regen was #1023 (c71ca67), which landed earlier the same day, so every regen since has reproduced the same gap and shown up as unrelated drift in other contributors' PRs. Regenerated with `make refresh-openapi`. The diff is entirely the aggregate-score schemas catching up to packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py: - add `median`, `sample_std_dev`, `sample_variance` to the three aggregate-score schemas - drop `count` from `required` (source is `count: int | None = None`) - add the `AggregateScalarScore` schema No source or behavior changes; generated output only. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Contributor
📝 WalkthroughWalkthroughThe OpenAPI schemas now support optional aggregate counts, median values, separate population and sample variance statistics, and standalone scalar scores in aggregated metric results. ChangesAggregate score schemas
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-evaluator/openapi/openapi.yaml`:
- Around line 2668-2673: Update the three count descriptions in
plugins/nemo-evaluator/openapi/openapi.yaml at lines 2668-2673, 2751-2756, and
2835-2841 to state “Omitted when unknown” instead of describing unknown values
as None, then regenerate the OpenAPI specification so all generated content
reflects the updated documentation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ffaf0f89-72da-4c6e-97d5-cb7733c08a45
📒 Files selected for processing (1)
plugins/nemo-evaluator/openapi/openapi.yaml
Comment on lines
2668
to
+2673
| title: Count | ||
| description: Number of samples evaluated (excluding NaN). | ||
| description: "Number of samples evaluated (excluding NaN). None when the\ | ||
| \ sample size is unknown \u2014 e.g. a figure imported from a backend\ | ||
| \ that reports statistics without the n behind them. Distinct from 0,\ | ||
| \ which asserts that nothing was evaluated." | ||
| type: integer |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect score construction and response serialization for None handling.
ast-grep outline packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py --items all --type class --match 'Aggregate.*Score'
rg -n -P -C 6 \
'\b(?:AggregateRangeScore|AggregateRubricScore|AggregateScalarScore)\s*\(|\bcount\s*=\s*None\b|\b(?:model_dump|model_dump_json|jsonable_encoder)\s*\(|response_model_exclude_none' \
packages/nemo_evaluator_sdk \
plugins/nemo-evaluator/srcRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50382
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- aggregate score models ---'
sed -n '270,415p' packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
printf '%s\n' '--- aggregate score schema declarations and route models ---'
rg -n -P -C 5 \
'Aggregate(?:Range|Rubric|Scalar)Score|AggregatedMetricResult|response_model=.*(?:Result|Evaluation)|response_model_exclude_none' \
plugins/nemo-evaluator/src/nemo_evaluator packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk \
-g '*.py' | head -n 500
printf '%s\n' '--- OpenAPI generation and policy targets ---'
rg -n -P -C 3 'openapi\.yaml|generate.*openapi|check-policy|build-policy' \
Makefile '**/Makefile' plugins/nemo-evaluator -g '*.py' -g '*.yaml' -g '*.yml' -g 'Makefile' 2>/dev/null | head -n 300Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- result route implementations ---'
sed -n '1,280p' plugins/nemo-evaluator/src/nemo_evaluator/api/v2/results.py
printf '%s\n' '--- result service serialization paths ---'
rg -n -P -C 8 \
'EvaluationResult|AggregatedMetricResult|aggregate_scores|scores|model_dump_json|model_dump\(' \
plugins/nemo-evaluator/src/nemo_evaluator/api plugins/nemo-evaluator/src/nemo_evaluator/sdk \
-g '*.py' | head -n 700
printf '%s\n' '--- OpenAPI count schemas and document version ---'
sed -n '1,30p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2645,2685p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2728,2770p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2815,2855p' plugins/nemo-evaluator/openapi/openapi.yaml
printf '%s\n' '--- all generated count declarations ---'
rg -n -P -B 8 -A 8 'title: Count|description: "Number of samples evaluated' \
plugins/nemo-evaluator/openapi/openapi.yamlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- exact OpenAPI score schemas ---'
sed -n '1,12p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2648,2695p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2731,2780p' plugins/nemo-evaluator/openapi/openapi.yaml
sed -n '2818,2865p' plugins/nemo-evaluator/openapi/openapi.yaml
printf '%s\n' '--- nearby required arrays and schema references ---'
python3 - <<'PY'
from pathlib import Path
p = Path("plugins/nemo-evaluator/openapi/openapi.yaml")
lines = p.read_text(encoding="utf-8").splitlines()
for i, line in enumerate(lines):
if "AggregateRangeScore:" in line or "AggregateRubricScore:" in line or "AggregateScalarScore:" in line:
start = max(0, i - 2)
end = min(len(lines), i + 105)
print(f"\n--- lines {start + 1}-{end} ---")
print("\n".join(f"{n + 1}: {lines[n]}" for n in range(start, end)))
PY
printf '%s\n' '--- route-level exclusion settings ---'
rg -n 'response_model_exclude_none|response_model=' \
plugins/nemo-evaluator/src/nemo_evaluator/api/v2/results.py \
plugins/nemo-evaluator/src/nemo_evaluator/api/schemas.pyRepository: NVIDIA-NeMo/nemo-platform
Length of output: 20156
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import ast
from pathlib import Path
import re
routes = Path("plugins/nemo-evaluator/src/nemo_evaluator/api/v2/results.py").read_text()
tree = ast.parse(routes)
route_rows = []
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
decorator_text = ast.get_source_segment(routes, node) or ""
if "response_model=" in decorator_text and "response_model_exclude_none=True" in decorator_text:
route_rows.append((node.name, "response_model_exclude_none=True" in decorator_text))
print("result routes with exclusion:")
for row in sorted(route_rows):
print(row)
model = Path("packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py").read_text()
count_match = re.search(
r"count:\s*int\s*\|\s*None\s*=\s*Field\(\s*default=None,\s*description=",
model,
re.S,
)
print("SDK count is optional with default None:", bool(count_match))
openapi = Path("plugins/nemo-evaluator/openapi/openapi.yaml").read_text()
for schema in ("AggregateRangeScore", "AggregateRubricScore", "AggregateScalarScore"):
block = openapi.split(f" {schema}:", 1)[1].split("\n ", 1)[0]
count_block = block.split(" count:", 1)[1].split(" nan_count:", 1)[0]
required_block = block.split(" required:", 1)[1].split(" title:", 1)[0]
print(
schema,
"count_has_null_type=", "null" in count_block,
"count_is_required=", "- count" in required_block,
"description_mentions_None=", "None" in count_block,
)
PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 349
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
routes = Path("plugins/nemo-evaluator/src/nemo_evaluator/api/v2/results.py").read_text()
decorated = re.findall(
r'(?ms)^@(?:agent_eval_results_router|evaluate_results_router)\.get\(.*?\n'
r'.*?response_model=(?P<model>[^,\n]+),.*?\n'
r'\s*response_model_exclude_none=(?P<exclude>True|False),',
routes,
)
print("result route response exclusion:")
for model, exclude in decorated:
print(model.strip(), exclude)
print("all result routes exclude None:", len(decorated) == 4 and all(x[1] == "True" for x in decorated))
sdk = Path("packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py").read_text()
print(
"SDK count is int|None with default None:",
bool(re.search(r"count:\s*int\s*\|\s*None\s*=\s*Field\(\s*default=None", sdk, re.S)),
)
openapi = Path("plugins/nemo-evaluator/openapi/openapi.yaml").read_text()
for schema in ("AggregateRangeScore", "AggregateRubricScore", "AggregateScalarScore"):
match = re.search(
rf"(?ms)^ {schema}:\n(?P<body>.*?)(?=^ [A-Za-z_][A-Za-z0-9_]*:|\Z)",
openapi,
)
if not match:
print(schema, "not found")
continue
body = match.group("body")
count = re.search(r"(?ms)^ count:\n(?P<body>.*?)(?=^ [A-Za-z_][A-Za-z0-9_]*:|\Z)", body)
required = re.search(r"(?ms)^ required:\n(?P<body>.*?)(?=^ title:)", body)
count_body = count.group("body") if count else ""
required_body = required.group("body") if required else ""
print(
schema,
"count_has_null=", bool(re.search(r"\bnull\b", count_body)),
"count_required=", bool(re.search(r"(?m)^ - count$", required_body)),
"description_mentions_None=", "None" in count_body,
)
PYRepository: NVIDIA-NeMo/nemo-platform
Length of output: 663
Document omitted unknown count values.
All result routes use response_model_exclude_none=True, so count=None is omitted rather than serialized as null. Update the three descriptions to say “Omitted when unknown,” then regenerate plugins/nemo-evaluator/openapi/openapi.yaml.
📍 Affects 1 file
plugins/nemo-evaluator/openapi/openapi.yaml#L2668-L2673(this comment)plugins/nemo-evaluator/openapi/openapi.yaml#L2751-L2756plugins/nemo-evaluator/openapi/openapi.yaml#L2835-L2841
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-evaluator/openapi/openapi.yaml` around lines 2668 - 2673, Update
the three count descriptions in plugins/nemo-evaluator/openapi/openapi.yaml at
lines 2668-2673, 2751-2756, and 2835-2841 to state “Omitted when unknown”
instead of describing unknown values as None, then regenerate the OpenAPI
specification so all generated content reflects the updated documentation.
Contributor
|
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The committed evaluator plugin OpenAPI spec has been stale on
mainsince #1065 (8e7179a141), which changed the aggregate-score models but did not regenerate the spec. The last regen was #1023 (c71ca675dc), which landed earlier the same day. Because the gap lives onmain, every regen since has faithfully reproduced it, and it has been surfacing as unexplainedAggregatedMetricResultdrift in unrelated contributors' PRs — repeatedly diagnosed as a codegen bug and reverted, which is why it kept coming back. This regenerates the spec so the diff stops re-appearing for everyone.Related Issue
Changes
Regenerated
plugins/nemo-evaluator/openapi/openapi.yamlwithmake refresh-openapi. The diff is entirely the aggregate-score schemas catching up topackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py:median,sample_std_dev, andsample_varianceto the three aggregate-score schemas.countfromrequired— the source field iscount: int | None = None, so it is genuinely optional.AggregateScalarScoreschema.No source or behavior changes; generated output only. Every line traces to #1065.
Note that the stale file is the plugin spec (
plugins/nemo-evaluator/openapi/openapi.yaml), not the rootopenapi/openapi.yaml. Grepping the root spec forAggregatedMetricResultreturns nothing, which is likely why this kept being read as phantom drift.Type of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
make refresh-openapi— succeeded; produced exactly this diff (150 insertions, 10 deletions).make refresh-openapiafter committing — working tree clean, so the committed spec now matches generator output exactly.tools/lint/lint-openapi.sh— exit 0.uv run pre-commit run -a— not fully green. Two hooks fail, both reproduced on a pristineorigin/maincheckout with zero changes applied, so neither is caused by this PR:Check for uv.lock drift— theuv lockhook rewritesuv.lockon a local macOS/arm64 machine, stripping ~412 lines of non-native platform wheels (armv7l,ppc64le,s390x,riscv64). Local uv is 0.9.30, which is inside the>=0.9.14,<0.10.0pin, so this looks like skew between the committed lock and the current uv resolver rather than an out-of-range toolchain.uv.lockis deliberately not included in this PR. Flagging separately; it needs its own fix.Run UI lint-staged— local mise toolchain is not provisioned (mise.tomluntrusted, nopnpmshim version set). Noweb/files are touched by this PR.ruff,ruff format,tytypechecks, config-reference doc, Helm docs, copyright headers, plugin-import check, merge-conflict check.Summary by CodeRabbit
New Features
Documentation