RFC 008 slice 0b: registries, report schema, and severity policy by zkwentz · Pull Request #1045 · huggingface/OpenEnv

zkwentz · 2026-08-04T19:11:14Z

Registries, report schema, and severity policy on top of #1044. Still no CLI.

Parser and grader registries, plus the provider protocol. Graders get picked from the manifest, not the signature (there's a test for that). Third-party graders go through the openenv.validation.graders entry point.

severity-v1.json has 45 check ids, 33 local and 12 hub. Local runs don't include hub checks. ERROR is a fail, SKIP is a warn, unknown check id is an internal error. Policy bounds are placeholders (0.1, 0.1, 0.2, 3600s). Cosign stays warn.

ValidationReport.manifest can be null when the schema check fails. Duplicate check ids in a policy are rejected. WELL_KNOWN_FILES is empty. detect_signature is in #1091.

PYTHONPATH=src:envs uv run pytest tests/test_validation/ -q # 64 passed
uv run python scripts/sync_validation_schemas.py --check

Next is #1091. Refs #778, #898.

Note

Medium Risk
Introduces the canonical verdict and lane policy contract that future CLI and hub runs will depend on; mistakes in apply_policy or policy data would affect pass/fail semantics repo-wide.

Overview
Adds the RFC 008 validation pipeline contracts on top of the manifest types: parser and grader registries, a validation-specific provider protocol, structured ValidationReport / CheckResult, and versioned severity policy — still no CLI or concrete graders.

Grader selection is manifest-driven (capabilities, level ceiling, applies_to); GraderRegistry supports third-party graders via the openenv.validation.graders entry point. Parsers map SignatureKind to NormalizedManifest; missing parsers raise UnsupportedPackageError (parser-not-implemented). severity-v1.json pins 45 check ids (local vs hub lanes); apply_policy is the only path that assigns severities and overall verdict (ERROR → fail, SKIP → warn, unknown ids → PolicyError).

Packaging and CI now ship policies/*.json and sync report.schema.json from the Pydantic models. Tests lock the policy table against drift and cover registries, report round-trips, and policy edge cases.

^{Reviewed by Cursor Bugbot for commit 6f98b03. Bugbot is set up for automated code reviews on this repo. Configure here.}

bot-ci-comment · 2026-08-04T19:13:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

…ign warn Review finding on #1045: the rollup sentence said "30 fail, 9 warn, 1 advisory" but the committed policy carries 34/10/1 over 45 check ids. Both countings are now stated explicitly, and the per-test tally is corrected to 32 fail (the prior 30 was an arithmetic slip — 30+9+1+2 != 44). Test #8's row now pins hub.cosign_signature at warn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pinned report version Bugbot finding: a grader CLASS loaded from an entry point already satisfies the runtime-checkable Grader protocol (class attributes exist on the class object), so the isinstance guard skipped instantiation and registered the class itself — instantiation now keys off isinstance(candidate, type). Registration also validates requires_capabilities names against CapabilitiesSpec fields so a typo'd capability errors clearly at register time instead of crashing at selection. report_schema_version is pinned to Literal["1"] for parity with the manifest. Regression tests for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review direction from Zach on #1045: WELL_KNOWN_FILES must not list formats whose parsers are not implemented in the build. Detection recognizes only supported formats (entries graduate in with their parsers: openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser); anything else is refused as unrecognized, exit 2. parser-not-implemented remains as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pinned report version Bugbot finding: a grader CLASS loaded from an entry point already satisfies the runtime-checkable Grader protocol (class attributes exist on the class object), so the isinstance guard skipped instantiation and registered the class itself — instantiation now keys off isinstance(candidate, type). Registration also validates requires_capabilities names against CapabilitiesSpec fields so a typo'd capability errors clearly at register time instead of crashing at selection. report_schema_version is pinned to Literal["1"] for parity with the manifest. Regression tests for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

WELL_KNOWN_FILES no longer pre-declares formats whose parsers do not exist in the build — entries are added alongside their parsers (openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser). The parser-not-implemented category is retained as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pinned report version Bugbot finding: a grader CLASS loaded from an entry point already satisfies the runtime-checkable Grader protocol (class attributes exist on the class object), so the isinstance guard skipped instantiation and registered the class itself — instantiation now keys off isinstance(candidate, type). Registration also validates requires_capabilities names against CapabilitiesSpec fields so a typo'd capability errors clearly at register time instead of crashing at selection. report_schema_version is pinned to Literal["1"] for parity with the manifest. Regression tests for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

WELL_KNOWN_FILES no longer pre-declares formats whose parsers do not exist in the build — entries are added alongside their parsers (openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser). The parser-not-implemented category is retained as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pinned report version Bugbot finding: a grader CLASS loaded from an entry point already satisfies the runtime-checkable Grader protocol (class attributes exist on the class object), so the isinstance guard skipped instantiation and registered the class itself — instantiation now keys off isinstance(candidate, type). Registration also validates requires_capabilities names against CapabilitiesSpec fields so a typo'd capability errors clearly at register time instead of crashing at selection. report_schema_version is pinned to Literal["1"] for parity with the manifest. Regression tests for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

WELL_KNOWN_FILES no longer pre-declares formats whose parsers do not exist in the build — entries are added alongside their parsers (openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser). The parser-not-implemented category is retained as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… 0b (#778, #898) Second half of the slice-0 contracts: signature detection rules and the unsupported-categories error contract; Parser/Provider/Grader protocols with working Parser and Grader registries (third-party graders via the openenv.validation.graders entry-point group); CheckResult/ValidationReport models with the committed report JSON Schema; the versioned severity policy (severity-v1.json, all 45 check ids including reserved hub/statistical ids) with lane filtering and fail-closed verdict application; and the conformance tests: policy completeness against the RFC table, protocol conformance against test-only fakes, report round-trips, and the accreting checkpoint suite. No runner, no CLI change, no real grader — those land per vertical slice. Checkpoint: PYTHONPATH=src:envs pytest tests/test_validation/ -v # 59 passed python scripts/sync_validation_schemas.py --check Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… pinned report version Bugbot finding: a grader CLASS loaded from an entry point already satisfies the runtime-checkable Grader protocol (class attributes exist on the class object), so the isinstance guard skipped instantiation and registered the class itself — instantiation now keys off isinstance(candidate, type). Registration also validates requires_capabilities names against CapabilitiesSpec fields so a typo'd capability errors clearly at register time instead of crashing at selection. report_schema_version is pinned to Literal["1"] for parity with the manifest. Regression tests for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

WELL_KNOWN_FILES no longer pre-declares formats whose parsers do not exist in the build — entries are added alongside their parsers (openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser). The parser-not-implemented category is retained as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Checkpoint tests were work-organization for the RFC 008 stack, not product scope. Fixture and policy coverage already lives in test_manifest and test_policy. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

NormalizedManifest now requires minLength 1 on type tags. The report schema embeds that model. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

static.manifest is the schema check. A package that fails that check has no NormalizedManifest, so the report field has to be optional. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

entries_for_lane last-write-wins on a dict. Duplicate ids in the JSON would silently drop an entry. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

This slice ships registries, the report, and the severity policy. detect_signature and the runner are later. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

WELL_KNOWN_FILES is empty in this slice. Status and severity both contain fail, so they are not disjoint vocabularies. Deleted the lane-uniqueness test that unique ids already implied. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

…ign warn Review finding on huggingface#1045: the rollup sentence said "30 fail, 9 warn, 1 advisory" but the committed policy carries 34/10/1 over 45 check ids. Both countings are now stated explicitly, and the per-test tally is corrected to 32 fail (the prior 30 was an arithmetic slip — 30+9+1+2 != 44). Test huggingface#8's row now pins hub.cosign_signature at warn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review direction from Zach on huggingface#1045: WELL_KNOWN_FILES must not list formats whose parsers are not implemented in the build. Detection recognizes only supported formats (entries graduate in with their parsers: openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser); anything else is refused as unrecognized, exit 2. parser-not-implemented remains as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 5ffa176. Configure here.}

burtenshaw added feature size: large Large pull request labels Aug 4, 2026 — with Cursor

cursor Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread src/openenv/validation/graders/__init__.py

github-code-quality Bot found potential problems Aug 4, 2026

View reviewed changes

cursor Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread src/openenv/validation/policies/severity-v1.json

Comment thread src/openenv/validation/report.py Outdated

zkwentz force-pushed the rfc-008/pr2-contracts branch from 51c6d01 to 85e9b73 Compare August 4, 2026 19:30

cursor Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread src/openenv/validation/graders/__init__.py Outdated

Comment thread src/openenv/validation/graders/__init__.py

zkwentz force-pushed the rfc-008/pr2-contracts branch from 85e9b73 to 81ab9ab Compare August 4, 2026 19:37

zkwentz force-pushed the rfc-008/pr2-contracts branch from 81ab9ab to e1a5f68 Compare August 4, 2026 19:48

zkwentz assigned Darktex and burtenshaw Aug 4, 2026

cursor Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread src/openenv/validation/policy.py

zkwentz mentioned this pull request Aug 4, 2026

RFC 008 slice 1: walking skeleton — openenv validate end to end #1051

Closed

This was referenced Aug 24, 2026

RFC 008 slice 1: walking skeleton — openenv validate end to end #1087

Closed

RFC 008 slice 1: walking skeleton for openenv validate #1091

Merged

zkwentz force-pushed the rfc-008/pr2-contracts branch from 8018afb to 934c3a3 Compare August 26, 2026 16:32

cursor Bot force-pushed the rfc-008/pr2-contracts branch from 934c3a3 to 5713036 Compare August 26, 2026 17:00

zkwentz and others added 6 commits August 26, 2026 10:02

fix: harden validation policy verdicts

8e331e8

chore(validation): regenerate report.schema.json for TypeSpec tags

21f7c6b

NormalizedManifest now requires minLength 1 on type tags. The report schema embeds that model. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

zkwentz force-pushed the rfc-008/pr2-contracts branch from 6e8ca96 to 21f7c6b Compare August 26, 2026 17:02

zkwentz mentioned this pull request Aug 26, 2026

RFC 008 slice 0a: core types, normalized manifest, and fixtures #1044

Merged

cursoragent and others added 4 commits August 26, 2026 17:47

fix(validation): allow a null ValidationReport.manifest

6a2eb27

static.manifest is the schema check. A package that fails that check has no NormalizedManifest, so the report field has to be optional. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

fix(validation): reject duplicate severity-policy check ids

0675969

entries_for_lane last-write-wins on a dict. Duplicate ids in the JSON would silently drop an entry. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

docs(validation): describe the 0b contract, not slice 0a

ae67cff

This slice ships registries, the report, and the severity policy. detect_signature and the runner are later. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>

burtenshaw added 2 commits September 1, 2026 12:13

chore: refresh validation stack

5ad69bc

chore: refresh validation stack

5ffa176

cursor Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread src/openenv/validation/graders/__init__.py Outdated

fix: support legacy entry points

6f98b03

burtenshaw merged commit 6f5e7c9 into main Sep 1, 2026
9 checks passed

This was referenced Sep 1, 2026

Revert RFC 008 validation stack #1108

Merged

Restore RFC 008 validation contracts #1111

Merged

Navigation Menu

Uh oh!

RFC 008 slice 0b: registries, report schema, and severity policy - #1045

RFC 008 slice 0b: registries, report schema, and severity policy#1045
burtenshaw merged 13 commits into
rfc-008/pr2a-manifestfrom
rfc-008/pr2-contracts

zkwentz commented Aug 4, 2026 •

edited by cursor Bot

Loading

Uh oh!

bot-ci-comment Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

zkwentz commented Aug 4, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bot-ci-comment Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zkwentz commented Aug 4, 2026 •

edited by cursor Bot

Loading