Skip to content

feat: support no-evals in perf changelog entries - #2935

Merged
cquil11 merged 1 commit into
mainfrom
codex-changelog-no-evals
Sep 9, 2026
Merged

cquil11 merged 1 commit into
mainfrom
codex-changelog-no-evals

Conversation

@cquil11

@cquil11 cquil11 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add opt-in no-evals: true to perf-changelog.yaml entries so PR sweeps can run throughput-only validation through the normal full-sweep-enabled workflow. This is needed by #2931 to validate Prometheus cache-source attribution without rerunning model-quality evals.

  • Defaults remain unchanged; benchmark duration, collection, and artifacts are unchanged.
  • Skip all four eval families for the opted-out entry. Other entries can still request evals, including for overlapping configurations.
  • Reject conflicting entry-level eval selectors and PR-wide eval modifiers.
  • Preserve the flag in changelog metadata; document that these runs do not provide model-evaluation evidence.

Validation

uv run --no-project --with pytest --with pydantic --with pyyaml --python 3.12 pytest utils/test_process_changelog.py utils/matrix_logic/test_validation.py utils/changelog_gate_tests/ -q

311 passed. Tests exercise throughput preservation, default behavior, entry isolation, metadata, and conflicting options. No GPU recipe changes in this PR. Prepared with AI assistance.


Note

Medium Risk
Changes PR sweep matrix composition and validation gates; misconfiguration could skip evals unintentionally, though the flag is opt-in and guarded by explicit conflict checks and tests.

Overview
Adds no-evals: true on perf-changelog.yaml entries so a normal full-sweep-enabled PR can run throughput-only for those entries without PR-wide evals-only / --no-evals CLI tricks.

process_changelog still generates benchmark rows with the existing --no-evals generator path, then skips all eval matrix generation for opted-out entries (same early-exit pattern as append-only). no-evals is per-entry: another changelog line can still schedule evals for overlapping configs. The flag is validated against conflicting entry fields (evals-only, all-evals, eval-min-prefill-ep) and against PR modifiers --all-evals / --evals-only.

ChangelogEntry in infx/matrix/validation.py defines the new field (default false). English and Chinese eval procedure docs describe throughput-only sweeps and that these runs are not model-evaluation evidence. Tests cover throughput preservation, metadata, entry isolation, and rejection of conflicting options.

Reviewed by Cursor Bugbot for commit a898b0d. Bugbot is set up for automated code reviews on this repo. Configure here.

@cquil11
cquil11 requested a review from a team September 9, 2026 19:21
@cquil11
cquil11 merged commit cf98870 into main Sep 9, 2026
8 checks passed
@cquil11
cquil11 deleted the codex-changelog-no-evals branch September 9, 2026 19:24

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment on lines +18 to +20
选择 eval;若需完全禁用,请在所有相关条目中设置该选项。条目中不能同时使用
`all-evals`、`evals-only` 或 `eval-min-prefill-ep`,PR 也不能同时使用两个 eval
modifier 中的任意一个。这类运行提供吞吐量证据,不提供模型评估证据。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The Chinese translation drops the "combined with no-evals" qualifier, so it reads as an absolute ban on using all-evals/evals-only/eval-min-prefill-ep together (or the two PR modifiers together) — which contradicts actual validator behavior (an entry may freely combine evals-only with eval-min-prefill-ep, or all-evals is just mutually exclusive with evals-only at CLI level, absent no-evals). Chinese-only readers get a false validation rule not present in the English page or the code. Fix: retranslate to explicitly tie the prohibition to no-evals, e.g. "与 no-evals 同时使用 all-evals/evals-only/eval-min-prefill-ep,或在 PR 上同时使用 no-evals 与任一 eval PR modifier,都会被拒绝", matching validation.py's validate_append_only_mode and process_changelog.py's no_evals/all_evals/evals_only check.

Extended reasoning...

English (docs/eval-agentx-procedures.md:20-21): "Combining no-evals with all-evals, evals-only, or eval-min-prefill-ep on the entry, or with either eval PR modifier, is rejected" — the ban is conditioned on no-evals, matching infx/matrix/validation.py's model_validator (raises only when self.no_evals is True) and process_changelog.py's main() (raises only when any entry.no_evals and args.all_evals/evals_only). The zh page (lines 18-20) renders this as "条目中不能同时使用 all-evals、evals-only 或 eval-min-prefill-ep,PR 也不能同时使用两个 eval modifier 中的任意一个" — omitting the no-evals connector entirely, so it states an entry can never combine those three fields, and a PR can never use both eval modifiers, neither of which the code actually enforces outside the no-evals case. A Chinese-reading operator following only this page would believe legal configurations (e.g. evals-only + eval-min-prefill-ep) are forbidden.

Verification: nit (docs-only, no runtime break, but introduces false content the base lacks). English docs/eval-agentx-procedures.md:18-20 conditions the rejection on no-evals: "Combining no-evals with all-evals, evals-only, or eval-min-prefill-ep on the entry, or with either eval PR modifier, is rejected." This mirrors infx/matrix/validation.py validate_append_only_mode, which raises ONLY when…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant