Skip to content

RFC 008 slice 0a: core types, normalized manifest, and fixtures - #1044

Merged
burtenshaw merged 9 commits into
mainfrom
rfc-008/pr2a-manifest
Sep 1, 2026
Merged

RFC 008 slice 0a: core types, normalized manifest, and fixtures#1044
burtenshaw merged 9 commits into
mainfrom
rfc-008/pr2a-manifest

Conversation

@zkwentz

@zkwentz zkwentz commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Types and NormalizedManifest for RFC 008. Parsers will produce this, graders will consume it. No runner or CLI yet.

Graders emit CheckStatus. Only the severity policy assigns Severity. Missing oracle is a valid parse (semantic.oracle_max fails it later). Unpinned LLM judge fails the pydantic model. Tags need at least one nonempty string. GPU fields are just resources.

Six fixtures: served_min_pass, no_oracle, harbor_task_min, posttrain_task_min, broken_manifest, unpinned_judge. Deleted four that were just served_min_pass with a different name.

JSON Schema is generated from the model (scripts/sync_validation_schemas.py). Some rules only exist as pydantic validators, so unpinned_judge can pass jsonschema and still fail NormalizedManifest.

Follows #1041 (merged). Next is #1045.

PYTHONPATH=src:envs uv run pytest tests/test_validation/ -q # 23 passed
uv run python scripts/sync_validation_schemas.py --check

Refs #778, #898.


Note

Low Risk
Additive schema/types and tests with a dependency pin; no changes to runtime auth, execution, or existing CLI behavior.

Overview
Introduces openenv.validation as the RFC 008 contract layer: shared enums (Level, CheckStatus, Severity, etc.) and a strict Pydantic NormalizedManifest that parsers will emit and graders will read (reward/oracle/verifier, resources, network, capabilities, type tags).

Cross-format fixtures cover openenv.yaml, Harbor task.toml, and task.md, plus negative cases (invalid reward range, unpinned LLM judge). Tests lock in invariants such as missing oracle parsing cleanly, llm_judged requiring judge pin + variance_tolerance, and injected_state requiring set_state.

Committed manifest.schema.json is exported from the model via new scripts/sync_validation_schemas.py (--check / --fix); packaging ships schemas/*.json. fastmcp is capped at <4.0.0. No validation runner or CLI in this slice.

Reviewed by Cursor Bugbot for commit 4e52afa. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Aug 4, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@burtenshaw burtenshaw added feature size: large Large pull request labels Aug 4, 2026 — with Cursor
Comment thread tests/test_validation/test_types.py Outdated
Comment thread src/openenv/validation/manifest.py
Comment thread pyproject.toml Outdated
zkwentz added a commit that referenced this pull request Aug 4, 2026
Review finding on #1044: the illustrative openenv.yaml omitted the verifier
binding, so the RFC's own example would fail its normative schema. The
verifier stays required — how "evaluate this state" is invoked is not
defaultable; a served env declares kind: reward_channel explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zkwentz added a commit that referenced this pull request Aug 4, 2026
Collapse the test_types import to one line (CI usort gate), and ship only the
schemas/*.json package-data glob here — the policies/*.json half moves to
slice 0b where the policies/ directory actually lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zkwentz
zkwentz force-pushed the rfc-008/pr2a-manifest branch from e798610 to b86a665 Compare August 4, 2026 19:29
zkwentz added a commit that referenced this pull request Aug 4, 2026
Collapse the test_types import to one line (CI usort gate), and ship only the
schemas/*.json package-data glob here — the policies/*.json half moves to
slice 0b where the policies/ directory actually lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zkwentz
zkwentz force-pushed the rfc-008/pr2a-manifest branch from b86a665 to 5cc099f Compare August 4, 2026 19:36
Comment thread tests/fixtures/validation/served_min_pass/normalized_manifest.json Outdated
zkwentz added a commit that referenced this pull request Aug 4, 2026
Collapse the test_types import to one line (CI usort gate), and ship only the
schemas/*.json package-data glob here — the policies/*.json half moves to
slice 0b where the policies/ directory actually lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zkwentz
zkwentz force-pushed the rfc-008/pr2a-manifest branch from 5cc099f to beb3616 Compare August 4, 2026 19:47

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e2e84a8. Configure here.

Comment thread src/openenv/validation/manifest.py
zkwentz added a commit that referenced this pull request Aug 26, 2026
Collapse the test_types import to one line (CI usort gate), and ship only the
schemas/*.json package-data glob here — the policies/*.json half moves to
slice 0b where the policies/ directory actually lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zkwentz
zkwentz force-pushed the rfc-008/pr2a-manifest branch from e2e84a8 to e7ab04d Compare August 26, 2026 16:32
Base automatically changed from rfc-008/pr1-rfc to main August 26, 2026 17:02
zkwentz and others added 6 commits August 26, 2026 10:02
First half of the slice-0 contracts: core enums (Level, Lane, CheckStatus,
Severity, Verdict, SignatureKind, ProviderCapability), the NormalizedManifest
pydantic models with all schema rules (judge pin iff llm_judged, set_state
required for injected-state oracles, verifier entry iff script, NetworkPolicy
per the Harbor task.toml 1.4 precedent, GPU resource declarations), the
committed manifest JSON Schema with its CI sync script, the ten golden/defect
fixture packages whose normalized_manifest.json doubles as parser golden
output, and the schema round-trip tests.

Registries, report/policy contracts, and conformance tests follow in slice 0b.

Checkpoint: PYTHONPATH=src:envs pytest tests/test_validation/ -v   # 26 passed
            python scripts/sync_validation_schemas.py --check

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collapse the test_types import to one line (CI usort gate), and ship only the
schemas/*.json package-data glob here — the policies/*.json half moves to
slice 0b where the policies/ directory actually lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
empty_solution_max_reward, leaky_observation, leaky_egress, and
nondeterministic matched served_min_pass except the name field.
Keep served_min_pass, no_oracle, harbor_task_min, posttrain_task_min,
broken_manifest, and unpinned_judge.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The package currently ships types and NormalizedManifest. Detection,
parsers, report, and policy are later slices.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Type tags select domain graders. An empty list selected none with no
signal. Field(min_length=1) on the list and on each string makes that
state unrepresentable.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
test_harbor_manifest_signature_is_task_toml asserts what the body does.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
@zkwentz
zkwentz force-pushed the rfc-008/pr2a-manifest branch from f2d63a3 to 57d7e04 Compare August 26, 2026 17:02
nblintao pushed a commit to nblintao/OpenEnv that referenced this pull request Aug 29, 2026
Review finding on huggingface#1044: the illustrative openenv.yaml omitted the verifier
binding, so the RFC's own example would fail its normative schema. The
verifier stays required — how "evaluate this state" is invoked is not
defaultable; a served env declares kind: reward_channel explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@burtenshaw

Copy link
Copy Markdown
Collaborator

Dependency blocker follow-up: fresh CI began resolving FastMCP 4.0.0, which breaks the existing HTTP and WebSocket MCP session-persistence tests on Python 3.11/3.12. This base branch now bounds FastMCP to <4.0.0; an isolated fresh resolve selected 3.4.7, and both blocking tests pass. The bound is propagated through #1045 and #1091.

@burtenshaw
burtenshaw merged commit 07a0744 into main Sep 1, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size: large Large pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants