Skip to content

RFC 008 slice 1: walking skeleton for openenv validate - #1091

Merged
burtenshaw merged 17 commits into
rfc-008/pr2-contractsfrom
zkwentz/rfc-008-slice-1-walking-skeleton
Sep 1, 2026
Merged

RFC 008 slice 1: walking skeleton for openenv validate#1091
burtenshaw merged 17 commits into
rfc-008/pr2-contractsfrom
zkwentz/rfc-008-slice-1-walking-skeleton

Conversation

@zkwentz

@zkwentz zkwentz commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

First actually runnable openenv validate for RFC 008. Stacked on #1045.

Local path is detect signature → parse openenv.yamlstatic.manifest → policy → report. Exit 0 for pass/warn, 1 for fail, 2 for unrecognized or unsupported, 3 for internal errors.

That's all that runs. --level still defaults to semantic, but levels_run is just static. Harbor task.toml and PostTrain task.md exit 2 until those parsers exist. Two well-known files in one directory also exit 2.

--url is still the old FastAPI probe (JSON with passed and criteria). It never calls run_validation. --skip-build is on the CLI and does nothing.

Breaking: openenv validate some/dir no longer runs the multi-mode deployment checks.

uv run openenv validate envs/echo_env --level static --skip-build
# exit 0, Verdict: PASS

PYTHONPATH=src:envs uv run pytest tests/test_validation tests/test_cli -q
# 229 passed

Refs #778.

Screenshot 2026-08-26 at 10 40 42 AM Screenshot 2026-08-26 at 10 52 36 AM

@cursor

cursor Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
Learn more about Cursor Agents

@cursor
cursor Bot force-pushed the zkwentz/rfc-008-slice-1-walking-skeleton branch from d8e53f8 to 6ab6c00 Compare August 26, 2026 15:57
@zkwentz
zkwentz force-pushed the zkwentz/rfc-008-slice-1-walking-skeleton branch from 6ab6c00 to 3bf862d Compare August 26, 2026 16:32
@cursor
cursor Bot force-pushed the zkwentz/rfc-008-slice-1-walking-skeleton branch from 3bf862d to 21247f3 Compare August 26, 2026 17:01
@zkwentz
zkwentz force-pushed the zkwentz/rfc-008-slice-1-walking-skeleton branch from 21247f3 to b529d28 Compare August 26, 2026 17:02
cursoragent and others added 12 commits August 26, 2026 17:48
Wire detect_signature, OpenEnvYamlParser, static.manifest, run_validation,
policy application, and the 0/1/2/3 CLI exit-code contract so later slices
only add parsers, graders, or providers.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Schema failures already have a validation: block; tell authors to fix it
rather than add another one.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 1 has to prove signature detection, parse, static.manifest, policy, report, and the 0/1/2/3 exit codes. Later-slice fixture yamls and a second detect-and-parse wrapper were not part of that proof.

- Delete `ParserRegistry.parse` and the unused `provider` argument on `run_validation`.
- Register `StaticManifestGrader` inline and stop creating an outputs tempdir no grader writes.
- Delete `openenv.yaml` from `leaky_*`, `no_oracle`, `nondeterministic`, and `empty_solution_max_reward`.
- Point checkpoint and CLI tests at `python -m openenv.cli`, not CliRunner.
- Map unexpected local-validate exceptions to exit code 3.

`skip_build` stays on `run_validation` because the CLI flag is in the RFC contract. It does nothing until a build-dependent grader lands.

`openenv validate` on a served package keeps the same 0/1/2 exits. A crash in `run_validation` now exits 3 instead of falling through `__main__` as 1.

Pin before this commit: `envs/echo_env` exit 0, `broken_manifest` exit 1, `ambiguous_package` exit 2, 239 tests green. Real-CLI re-run follows on this commit.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Seed a project-local skill that drives `openenv validate` the way an
author does, with a feature map starting at echo_env static validate.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 1 already proves the pipeline. Module essays, later-slice comments, and a second invented happy-path fixture made the walking skeleton heavier to read without changing the contract.

- Trim module and helper prose in runner, signature, parser, StaticManifestGrader, and the validate CLI.
- Delete the tmp-dir `--json` happy path from tests/test_cli/test_validate.py. served_min_pass and test_cli_validate already pin that.
- Keep the missing-`validation:` CLI test. That path has no fixture.

ParserRegistry, GraderRegistry, skip_build, and the verify-openenv-validate skill stay. They are the spine later slices register into, not leftover weight.

openenv validate behavior is unchanged. Authors still get the same 0/1/2/3 exits and report text.

Pre-change pin: echo_env exit 0, broken_manifest exit 1, ambiguous and unrecognized exit 2, harbor_task_min unrecognized, cosmic --level exit 3. Tests and the verify skill run after this commit.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Keep public API contract docstrings, SPDX headers, the PR2 unsupported-categories note, and the proven pydantic missing-vs-None omit in OpenEnvYamlParser.parse.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The comment-strip commit deleted the one-line module banners the rest of `openenv.validation` uses, and put back the omit-field sermon in OpenEnvYamlParser.parse. The omit is already proven by the schema-error tests.

- Restore one-line module docstrings on runner, signature, parser, StaticManifestGrader, static graders package, and the validate command.
- Delete the three-line omit-field comment in OpenEnvYamlParser.parse.

No behavior change.

ruff check and format on the touched files passed after the restore.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
## Why
source_digest already excludes an outputs/ directory. The runner still handed graders the package root, so the first replay writer would mutate the tree and change the next digest.

## Scope
run_validation now sets Subject.outputs_dir to target / "outputs". Slice 1 still does not create that directory.

## Blast Radius
No grader reads or writes outputs_dir today. echo_env and fixture validates are unchanged.

## Verification
Convention already pinned in tests/test_validation/test_protocols.py. Runner and CLI pins follow this commit.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
## Why
Slice 1 should ship the CLI pipeline only. The Cursor skill can stack on the next PR.

## Scope
Removes .cursor/skills/verify-openenv-validate/. Checkpoints and pytest still pin the CLI.

## Blast Radius
Authors and CI are unchanged. Agents lose this skill until the follow-up lands.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Real-package pin: openenv validate envs/echo_env exits 0 with a
static.manifest PASS.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
Slice 0a narrowed the contract docstring. This slice ships the pipeline.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
The 0b pin was empty. This slice registers openenv.yaml.

Co-authored-by: Zach Wentz <zkwentz@users.noreply.github.com>
@cursor
cursor Bot force-pushed the zkwentz/rfc-008-slice-1-walking-skeleton branch from b529d28 to 6c673a5 Compare August 26, 2026 17:49
@zkwentz
zkwentz marked this pull request as ready for review August 26, 2026 18:14
@burtenshaw burtenshaw added feature size: large Large pull request labels Aug 26, 2026 — with Cursor
Comment thread src/openenv/validation/runner.py Outdated
Comment thread src/openenv/validation/signature.py
@burtenshaw

burtenshaw commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Addressed the latest review findings and stack blockers in 7ac4edaf plus the refreshed dependency/compatibility heads:

  • bounded FastMCP to the compatible 3.x line after fresh resolution of 4.0.0 broke both MCP session-persistence tests on Python 3.11/3.12;
  • made grader entry-point discovery work with both the selectable and legacy mapping APIs, verified on CPython 3.10.15;
  • canonicalized and canonically sorted source-digest paths for cross-platform stability, with a byte-level regression;
  • removed the downstream fixture version injections so RFC 008 slice 1: walking skeleton for openenv validate #1091 consumes RFC 008 slice 0a: core types, normalized manifest, and fixtures #1044’s corrected null goldens faithfully;
  • mapped --output write failures to the documented internal-error exit code 3, with CLI coverage;
  • made task.md signature probing tolerate invalid UTF-8 instead of escaping as an internal error;
  • tracked the intentional signature-ambiguity behavior versus RFC wording on RFC: 008 Environment auto validation #778.

Focused validation on the bounded resolver: 235 passed (tests/test_validation, tests/test_cli, and both previously failing MCP session tests).

@bot-ci-comment

bot-ci-comment Bot commented Sep 1, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c898cef. Configure here.

Comment thread src/openenv/validation/signature.py
@burtenshaw
burtenshaw merged commit 3d9eac1 into main Sep 1, 2026
9 checks passed
burtenshaw added a commit that referenced this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size: large Large pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants