RFC 008 slice 0a: core types, normalized manifest, and fixtures - #1044
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Review finding on #1044: the illustrative openenv.yaml omitted the verifier binding, so the RFC's own example would fail its normative schema. The verifier stays required — how "evaluate this state" is invoked is not defaultable; a served env declares kind: reward_channel explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collapse the test_types import to one line (CI usort gate), and ship only the schemas/*.json package-data glob here — the policies/*.json half moves to slice 0b where the policies/ directory actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e798610 to
b86a665
Compare
Collapse the test_types import to one line (CI usort gate), and ship only the schemas/*.json package-data glob here — the policies/*.json half moves to slice 0b where the policies/ directory actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b86a665 to
5cc099f
Compare
Collapse the test_types import to one line (CI usort gate), and ship only the schemas/*.json package-data glob here — the policies/*.json half moves to slice 0b where the policies/ directory actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5cc099f to
beb3616
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e2e84a8. Configure here.
Collapse the test_types import to one line (CI usort gate), and ship only the schemas/*.json package-data glob here — the policies/*.json half moves to slice 0b where the policies/ directory actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e2e84a8 to
e7ab04d
Compare
First half of the slice-0 contracts: core enums (Level, Lane, CheckStatus,
Severity, Verdict, SignatureKind, ProviderCapability), the NormalizedManifest
pydantic models with all schema rules (judge pin iff llm_judged, set_state
required for injected-state oracles, verifier entry iff script, NetworkPolicy
per the Harbor task.toml 1.4 precedent, GPU resource declarations), the
committed manifest JSON Schema with its CI sync script, the ten golden/defect
fixture packages whose normalized_manifest.json doubles as parser golden
output, and the schema round-trip tests.
Registries, report/policy contracts, and conformance tests follow in slice 0b.
Checkpoint: PYTHONPATH=src:envs pytest tests/test_validation/ -v # 26 passed
python scripts/sync_validation_schemas.py --check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collapse the test_types import to one line (CI usort gate), and ship only the schemas/*.json package-data glob here — the policies/*.json half moves to slice 0b where the policies/ directory actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
empty_solution_max_reward, leaky_observation, leaky_egress, and nondeterministic matched served_min_pass except the name field. Keep served_min_pass, no_oracle, harbor_task_min, posttrain_task_min, broken_manifest, and unpinned_judge. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The package currently ships types and NormalizedManifest. Detection, parsers, report, and policy are later slices. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Type tags select domain graders. An empty list selected none with no signal. Field(min_length=1) on the list and on each string makes that state unrepresentable. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
test_harbor_manifest_signature_is_task_toml asserts what the body does. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
f2d63a3 to
57d7e04
Compare
Review finding on huggingface#1044: the illustrative openenv.yaml omitted the verifier binding, so the RFC's own example would fail its normative schema. The verifier stays required — how "evaluate this state" is invoked is not defaultable; a served env declares kind: reward_channel explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Dependency blocker follow-up: fresh CI began resolving FastMCP 4.0.0, which breaks the existing HTTP and WebSocket MCP session-persistence tests on Python 3.11/3.12. This base branch now bounds FastMCP to |

Types and
NormalizedManifestfor RFC 008. Parsers will produce this, graders will consume it. No runner or CLI yet.Graders emit
CheckStatus. Only the severity policy assignsSeverity. Missing oracle is a valid parse (semantic.oracle_maxfails it later). Unpinned LLM judge fails the pydantic model. Tags need at least one nonempty string. GPU fields are just resources.Six fixtures:
served_min_pass,no_oracle,harbor_task_min,posttrain_task_min,broken_manifest,unpinned_judge. Deleted four that were justserved_min_passwith a differentname.JSON Schema is generated from the model (
scripts/sync_validation_schemas.py). Some rules only exist as pydantic validators, sounpinned_judgecan pass jsonschema and still failNormalizedManifest.Follows #1041 (merged). Next is #1045.
PYTHONPATH=src:envs uv run pytest tests/test_validation/ -q # 23 passed
uv run python scripts/sync_validation_schemas.py --check
Refs #778, #898.
Note
Low Risk
Additive schema/types and tests with a dependency pin; no changes to runtime auth, execution, or existing CLI behavior.
Overview
Introduces
openenv.validationas the RFC 008 contract layer: shared enums (Level,CheckStatus,Severity, etc.) and a strict PydanticNormalizedManifestthat parsers will emit and graders will read (reward/oracle/verifier, resources, network, capabilities, type tags).Cross-format fixtures cover
openenv.yaml, Harbortask.toml, andtask.md, plus negative cases (invalid reward range, unpinned LLM judge). Tests lock in invariants such as missing oracle parsing cleanly,llm_judgedrequiring judge pin +variance_tolerance, andinjected_staterequiringset_state.Committed
manifest.schema.jsonis exported from the model via newscripts/sync_validation_schemas.py(--check/--fix); packaging shipsschemas/*.json.fastmcpis capped at<4.0.0. No validation runner or CLI in this slice.Reviewed by Cursor Bugbot for commit 4e52afa. Bugbot is set up for automated code reviews on this repo. Configure here.