RFC 008 slice 1: walking skeleton for openenv validate - #1091
Conversation
|
Cursor Agent can help with this pull request. Just |
d8e53f8 to
6ab6c00
Compare
6ab6c00 to
3bf862d
Compare
3bf862d to
21247f3
Compare
21247f3 to
b529d28
Compare
Wire detect_signature, OpenEnvYamlParser, static.manifest, run_validation, policy application, and the 0/1/2/3 CLI exit-code contract so later slices only add parsers, graders, or providers. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Schema failures already have a validation: block; tell authors to fix it rather than add another one. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 1 has to prove signature detection, parse, static.manifest, policy, report, and the 0/1/2/3 exit codes. Later-slice fixture yamls and a second detect-and-parse wrapper were not part of that proof. - Delete `ParserRegistry.parse` and the unused `provider` argument on `run_validation`. - Register `StaticManifestGrader` inline and stop creating an outputs tempdir no grader writes. - Delete `openenv.yaml` from `leaky_*`, `no_oracle`, `nondeterministic`, and `empty_solution_max_reward`. - Point checkpoint and CLI tests at `python -m openenv.cli`, not CliRunner. - Map unexpected local-validate exceptions to exit code 3. `skip_build` stays on `run_validation` because the CLI flag is in the RFC contract. It does nothing until a build-dependent grader lands. `openenv validate` on a served package keeps the same 0/1/2 exits. A crash in `run_validation` now exits 3 instead of falling through `__main__` as 1. Pin before this commit: `envs/echo_env` exit 0, `broken_manifest` exit 1, `ambiguous_package` exit 2, 239 tests green. Real-CLI re-run follows on this commit. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Seed a project-local skill that drives `openenv validate` the way an author does, with a feature map starting at echo_env static validate. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 1 already proves the pipeline. Module essays, later-slice comments, and a second invented happy-path fixture made the walking skeleton heavier to read without changing the contract. - Trim module and helper prose in runner, signature, parser, StaticManifestGrader, and the validate CLI. - Delete the tmp-dir `--json` happy path from tests/test_cli/test_validate.py. served_min_pass and test_cli_validate already pin that. - Keep the missing-`validation:` CLI test. That path has no fixture. ParserRegistry, GraderRegistry, skip_build, and the verify-openenv-validate skill stay. They are the spine later slices register into, not leftover weight. openenv validate behavior is unchanged. Authors still get the same 0/1/2/3 exits and report text. Pre-change pin: echo_env exit 0, broken_manifest exit 1, ambiguous and unrecognized exit 2, harbor_task_min unrecognized, cosmic --level exit 3. Tests and the verify skill run after this commit. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Keep public API contract docstrings, SPDX headers, the PR2 unsupported-categories note, and the proven pydantic missing-vs-None omit in OpenEnvYamlParser.parse. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The comment-strip commit deleted the one-line module banners the rest of `openenv.validation` uses, and put back the omit-field sermon in OpenEnvYamlParser.parse. The omit is already proven by the schema-error tests. - Restore one-line module docstrings on runner, signature, parser, StaticManifestGrader, static graders package, and the validate command. - Delete the three-line omit-field comment in OpenEnvYamlParser.parse. No behavior change. ruff check and format on the touched files passed after the restore. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
## Why source_digest already excludes an outputs/ directory. The runner still handed graders the package root, so the first replay writer would mutate the tree and change the next digest. ## Scope run_validation now sets Subject.outputs_dir to target / "outputs". Slice 1 still does not create that directory. ## Blast Radius No grader reads or writes outputs_dir today. echo_env and fixture validates are unchanged. ## Verification Convention already pinned in tests/test_validation/test_protocols.py. Runner and CLI pins follow this commit. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
## Why Slice 1 should ship the CLI pipeline only. The Cursor skill can stack on the next PR. ## Scope Removes .cursor/skills/verify-openenv-validate/. Checkpoints and pytest still pin the CLI. ## Blast Radius Authors and CI are unchanged. Agents lose this skill until the follow-up lands. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Real-package pin: openenv validate envs/echo_env exits 0 with a static.manifest PASS. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 0a narrowed the contract docstring. This slice ships the pipeline. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The 0b pin was empty. This slice registers openenv.yaml. Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
b529d28 to
6c673a5
Compare
|
Addressed the latest review findings and stack blockers in
Focused validation on the bounded resolver: 235 passed ( |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c898cef. Configure here.

First actually runnable
openenv validatefor RFC 008. Stacked on #1045.Local path is detect signature → parse
openenv.yaml→static.manifest→ policy → report. Exit 0 for pass/warn, 1 for fail, 2 for unrecognized or unsupported, 3 for internal errors.That's all that runs.
--levelstill defaults to semantic, butlevels_runis just static. Harbortask.tomland PostTraintask.mdexit 2 until those parsers exist. Two well-known files in one directory also exit 2.--urlis still the old FastAPI probe (JSON withpassedandcriteria). It never callsrun_validation.--skip-buildis on the CLI and does nothing.Breaking:
openenv validate some/dirno longer runs the multi-mode deployment checks.Refs #778.