Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
189 commits
Select commit Hold shift + click to select a range
83644b9
refactor!: remove the v1 framework (verifiers/v1) and its core re-exp…
mikasenghaas Jun 9, 2026
eca5304
refactor!: delete v1 example envs + v1 tests; strip v1 flags from v0 …
mikasenghaas Jun 9, 2026
e48abd6
feat: vf-nano is verifiers v1 (submodule + verifiers.v1 alias)
mikasenghaas Jun 9, 2026
d2189b8
chore: bump deps/vf-nano (legacy v0->Trace bridge)
mikasenghaas Jun 9, 2026
f8d6939
feat(eval): v0-only vf-eval + add vf-eval-v1 (nano); bump vf-nano (re…
mikasenghaas Jun 9, 2026
963db28
chore: bump deps/vf-nano (plugins drop vf-nano dep)
mikasenghaas Jun 9, 2026
37923e0
refactor(types): scrub v1 artifacts from the v0 State
mikasenghaas Jun 9, 2026
de1238b
chore: finish v1 hygiene (init.py scaffolding, v1 tests, byo-harness …
mikasenghaas Jun 9, 2026
ed0d49a
refactor(v1): vendor vf-nano as verifiers.v1, drop old v1 packages
mikasenghaas Jun 9, 2026
19432df
refactor(v1): eval/serve scripts, v1 deps to base, bundle shipped plu…
mikasenghaas Jun 9, 2026
a7bd5a4
fix(ci): green the test/semgrep jobs after the v1 vendor + drop py3.10
mikasenghaas Jun 9, 2026
732bcde
chore: retire the v1 semgrep policy
mikasenghaas Jun 9, 2026
00c7b77
feat(v1): Task.system_prompt + harness APPENDS_SYSTEM_PROMPT support
mikasenghaas Jun 9, 2026
557e706
fix: render with the base-model tokenizer, not the per-request model
mikasenghaas Jun 9, 2026
b8a0852
feat(v1): parse vLLM token ids + logprobs in the chat client (#1585)
mikasenghaas Jun 9, 2026
b2a2238
fix(v1): preserve rollout meta in the v0 -> v1 legacy bridge (#1586)
mikasenghaas Jun 9, 2026
caeb6d7
chore(v1): suffix example tasksets with -v1 to avoid name clashes (#1…
mikasenghaas Jun 9, 2026
babe469
feat(v1): native rollout retries (EnvConfig.retry) (#1588)
mikasenghaas Jun 9, 2026
fba4642
feat: per-rollout token limits (max_input/output/total_tokens) (#1591)
mikasenghaas Jun 9, 2026
a65445e
fix: free runtime resources on exit/interrupt (#1590)
mikasenghaas Jun 9, 2026
33bd392
feat: add textarena taskset with a framework-driven user simulator (#…
mikasenghaas Jun 9, 2026
677145c
feat: add alphabet-sort-v1 taskset (#1595)
mikasenghaas Jun 9, 2026
4535551
feat: name runtime resources after the rollout id (#1596)
mikasenghaas Jun 9, 2026
5dc196c
feat: add modal runtime (#1594)
mikasenghaas Jun 9, 2026
3997c62
feat: resolve taskset / harness / env ids from the hub (#1597)
mikasenghaas Jun 10, 2026
907dac6
feat: run v0 environments on the eval CLI (legacy bridge) (#1598)
mikasenghaas Jun 10, 2026
6c4499a
feat: first-class v1 harnesses/tasksets packages (+ README, serve/CLI…
mikasenghaas Jun 10, 2026
6f1f934
fix: gate task-tool tasksets on harness support (SUPPORTS_TASK_TOOLS)…
mikasenghaas Jun 10, 2026
b0e18a1
feat: enable labels for prime sandboxes + tunnels (#1604)
mikasenghaas Jun 10, 2026
0b3555c
feat: multiplex interception servers + tunnels across rollouts (#1605)
mikasenghaas Jun 10, 2026
17f7e02
feat: delta-native message-graph trajectory (#1606)
mikasenghaas Jun 10, 2026
81cf7a2
refactor: inline branch building into Trace.branches
mikasenghaas Jun 10, 2026
5268e28
chore: add v1 end-to-end eval test suite (#1609)
mikasenghaas Jun 10, 2026
0cece1e
fix: honor group scoring in the v0 legacy env server (#1612)
mikasenghaas Jun 10, 2026
1bf7937
chore: guard the v0 legacy bridge against non-renderer training clien…
mikasenghaas Jun 10, 2026
0960b60
fix: serve the eval split for eval-only v0 envs in the legacy bridge …
mikasenghaas Jun 10, 2026
a9427e2
fix: build a chat-completions client for v0 eval in the legacy bridge…
mikasenghaas Jun 10, 2026
14ea5b1
fix(v1): align message graph hashing with renderer prefixes (#1617)
eligotts Jun 11, 2026
fd2f865
feat: add scaleswe v1 taskset + per-task setup/workdir hooks (#1616)
mikasenghaas Jun 11, 2026
853f9fc
feat(v1): carry multimodal images through the message graph
mikasenghaas Jun 11, 2026
c114bac
feat(v1): colocate the user simulator in the agent runtime
mikasenghaas Jun 11, 2026
d668319
feat(v1): add the color-codeword-v1 taskset
mikasenghaas Jun 11, 2026
1cde7e1
Revert "feat(v1): add the color-codeword-v1 taskset"
mikasenghaas Jun 11, 2026
872cc8d
Revert "feat(v1): colocate the user simulator in the agent runtime"
mikasenghaas Jun 11, 2026
188db92
Revert "feat(v1): carry multimodal images through the message graph"
mikasenghaas Jun 11, 2026
ac21e16
chore: add v1 #1559-vs-#1576 comparison doc (#1619)
mikasenghaas Jun 11, 2026
ef61543
feat: multimodal images through the v1 message graph + color-codeword…
mikasenghaas Jun 11, 2026
22590ef
feat(v1): add r2e-gym-v1 taskset (#1624)
S1ro1 Jun 11, 2026
fd6a083
fix: treat overlong prompts as clean truncated rollouts (#1625)
mikasenghaas Jun 11, 2026
be76cbc
feat: env-server worker pool for v1 (router + N workers) (#1623)
mikasenghaas Jun 11, 2026
36261fe
fix(v1): configure logging in spawned env-server workers (#1626)
mikasenghaas Jun 11, 2026
955b6cd
fix(v1): fall back to provider usage for dashboard token counts (#1627)
mikasenghaas Jun 11, 2026
8e4ad73
fix(v1): clean env-server teardown on SIGTERM
mikasenghaas Jun 11, 2026
7f614e0
chore(deps): raise renderers floor to 0.1.8.dev40
mikasenghaas Jun 11, 2026
db82b38
fix: reap the whole subprocess tree when a runtime run is cancelled (…
mikasenghaas Jun 11, 2026
f404e97
feat: elastic env-server pool (static/elastic pool config) (#1629)
mikasenghaas Jun 11, 2026
6af7897
feat(v1): record per-rollout setup timing as a distinct phase (#1631)
mikasenghaas Jun 11, 2026
88e9bed
feat: per-call model + runtime retries (#1632)
mikasenghaas Jun 11, 2026
ead66d2
fix(v1): blank elapsed for not-yet-started rollouts in the dashboard …
mikasenghaas Jun 11, 2026
651bf13
fix(v1): drop every timing span's computed duration from to_wire (#1636)
mikasenghaas Jun 11, 2026
40a2e89
fix: tee eval logs to output dir; keep them off the rich dashboard (#…
mikasenghaas Jun 11, 2026
5dc084f
feat: add --resume for evals (re-run a previous run's missing/errored…
mikasenghaas Jun 11, 2026
9c76f6d
feat(v1): finalize hook + per-stage timeouts (#1640)
mikasenghaas Jun 11, 2026
472622b
feat: configurable creation-rate limiter for modal + prime runtimes (…
mikasenghaas Jun 11, 2026
7270e69
fix: retry the user-sim connect at high concurrency (capture failures…
mikasenghaas Jun 12, 2026
66c87d5
feat(v1): record sampled provenance on MessageNode (#1643)
eligotts Jun 12, 2026
ef45f72
fix: resume reads results.jsonl by newline, not splitlines (#1644)
mikasenghaas Jun 12, 2026
31800c8
fix: run alphabet-sort user simulator on the host (colocated=False) (…
mikasenghaas Jun 12, 2026
f8425c9
chore: bump modal creates_per_sec default 5 -> 40 (#1646)
mikasenghaas Jun 12, 2026
2d913cb
feat(v1): add swelego-v1 taskset (SWE-Lego on rlm harness / prime run…
hallerite Jun 12, 2026
fc2bb20
fix Harbor agent timeout mapping (#1647)
xeophon Jun 12, 2026
4ac4e30
feat: add v1 validate entrypoint + Taskset.validate hook (#1655)
mikasenghaas Jun 12, 2026
4b608ba
fix(v1): carry multimodal tensors across the env-server wire (#1653)
mikasenghaas Jun 12, 2026
8873a74
fix: route-detected wire dialects + 1:1 relay (chat/responses/anthrop…
mikasenghaas Jun 12, 2026
de9e07b
perf: run subprocess uv-scripts via a cached interpreter, not per-cal…
mikasenghaas Jun 12, 2026
79a3e12
feat(v1): codex built-in harness (on the Responses dialect + streamin…
mikasenghaas Jun 12, 2026
b70d934
docs: v1 user guide (#1662)
mikasenghaas Jun 13, 2026
a887827
feat: add free-form `info` field on Trace for taskset-scraped metadat…
mikasenghaas Jun 13, 2026
e66dc3e
Rename RLM ref to version (#1667)
xeophon Jun 13, 2026
47772fc
Add harness environment variables (#1668)
xeophon Jun 13, 2026
a7902db
Ignore Anthropic service tier metadata (#1671)
xeophon Jun 13, 2026
faf7ce1
Add Claude Code harness (#1669)
xeophon Jun 14, 2026
7a98b56
feat(v1): capture MoE router-replay routed_experts on Trace/Graph (#1…
S1ro1 Jun 14, 2026
1b7736a
Merge origin/main into feat/nano-as-v1
mikasenghaas Jun 14, 2026
fe68312
fix(v1): import TypedDict from typing_extensions for py3.11 schema build
mikasenghaas Jun 14, 2026
5fc0b29
fix(v1): run prime-sandbox programs for the sandbox lifetime + retry-…
mikasenghaas Jun 14, 2026
3af08c9
fix(scaleswe-v1): resolve task images via Prime Artifact Registry (#1…
mikasenghaas Jun 14, 2026
7862985
feat(scaleswe-v1): make Prime Artifact Registry opt-in (default off) …
mikasenghaas Jun 15, 2026
211faec
feat(v1): session-affinity routing header for cross-turn prefix reuse…
S1ro1 Jun 15, 2026
3f45f73
feat: filter unavailable scaleswe-v1 images at load (#1679)
mikasenghaas Jun 15, 2026
44da378
feat(r2e-gym-v1): make Prime Artifact Registry opt-in (default off) (…
mikasenghaas Jun 15, 2026
a723977
fix(scaleswe-v1): filter on Docker Hub image_url, not the resolved im…
mikasenghaas Jun 15, 2026
c4e1bc0
chore: remove the Claude Code harness (#1684)
mikasenghaas Jun 15, 2026
0c8d0aa
fix: harnesses install/run without root (pinned /tmp/vf-{harness}) (#…
mikasenghaas Jun 15, 2026
d4b054e
test: skip test_multi_turn on prime (colocated user-sim port exposure…
mikasenghaas Jun 15, 2026
e88bc4e
Add mini-swe-agent V1 harness (#1673)
xeophon Jun 15, 2026
1872b99
Fix mini-swe-agent E2E matrix (#1693)
xeophon Jun 15, 2026
3c2512c
Support images in v1 tool responses (#1694)
eligotts Jun 16, 2026
2822e23
map v1 reasoning effort by dialect (#1690)
xeophon Jun 16, 2026
e906fc1
Add Kimi Code V1 harness (#1675)
xeophon Jun 16, 2026
36faabd
Ignore unsupported Anthropic service tiers (#1689)
xeophon Jun 16, 2026
2cab1fe
Keep relayed SSE streams alive (#1688)
xeophon Jun 16, 2026
621bb52
Add Harbor task multipliers (#1700)
xeophon Jun 16, 2026
b1f827e
Extend the V1 bash tool timeout (#1701)
xeophon Jun 16, 2026
68aac71
Use Prime CLI config for v1 eval (#1703)
xeophon Jun 16, 2026
22b02e4
chore: flatten examples/ into a single environments/ section (#1695)
mikasenghaas Jun 16, 2026
b45d4b8
feat: vf-native Toolset/User class surface + per-server runtime place…
mikasenghaas Jun 16, 2026
a485dcf
chore: resolve v1 plugins via __all__ export, split into taskset.py/h…
mikasenghaas Jun 16, 2026
f4b9422
feat: allow user to open the conversation when a task has no prompt (…
mikasenghaas Jun 16, 2026
0d6cb90
feat: typed transient Trace.state shared across tool/user servers + s…
mikasenghaas Jun 16, 2026
af10c8c
fix(v1): relax MCP DNS-rebinding guard for tunneled host-bound server…
mikasenghaas Jun 16, 2026
bf30ae5
feat: add an `init` scaffolding command for new v1 environments (#1713)
mikasenghaas Jun 16, 2026
bd4939a
feat: default the harness runtime to subprocess (#1719)
mikasenghaas Jun 17, 2026
f7fa748
feat: train-client prefix bridging + token-based message-graph branch…
mikasenghaas Jun 17, 2026
73156a7
chore: run v1 tests with -n auto in a separate CI step (#1718)
mikasenghaas Jun 17, 2026
5c54387
test(envs): keep test_envs.py v0-only, add tests/v1/test_envs.py for …
mikasenghaas Jun 17, 2026
1766df6
Add per-harness tool disabling (#1723)
xeophon Jun 17, 2026
e9dccf2
Forward chat completion headers and consolidate clients (#1666)
xeophon Jun 17, 2026
bbfd564
warn on local subprocess harnesses (#1722)
xeophon Jun 17, 2026
d66aab7
build: trim runtime dependencies (#1702)
xeophon Jun 17, 2026
220f21d
feat(v1): per-rollout isolation for shared writable tool servers (#1727)
mikasenghaas Jun 17, 2026
e01b422
feat(v1): fork-per-rollout isolation for shared tool servers (#1729)
mikasenghaas Jun 17, 2026
caaf0ff
chore: general v1 cleanup (#1732)
mikasenghaas Jun 17, 2026
a2b0d88
fix: unified, robust env-server teardown (no leaked tunnels/sandboxes…
mikasenghaas Jun 18, 2026
04385ae
feat: general-agent solver v1 taskset (#1734)
mikasenghaas Jun 18, 2026
cc4afd5
fix: reject unsupported task system_prompt (#1735)
mikasenghaas Jun 18, 2026
bca4716
chore(v1): slow dashboard page rotation to 5s (#1737)
mikasenghaas Jun 18, 2026
d7f984c
fix: incomplete URL substring sanitization in quest pdf tool (CodeQL)…
mikasenghaas Jun 18, 2026
d5261a1
feat: swebench-verified harbor taskset (prime prebuilt images) (#1738)
mikasenghaas Jun 18, 2026
3c5f658
fix: don't retry the rollout's program exec (forked duplicate trace b…
mikasenghaas Jun 18, 2026
952b2a7
fix: keep the swebench/ org in swebench-verified-v1 prime-registry im…
mikasenghaas Jun 18, 2026
56a720c
fix(general-agent-v1): make the toolset work when colocated in a sand…
mikasenghaas Jun 18, 2026
6647fe0
fix: add backoff to per-call retries; let harness clients retry the r…
mikasenghaas Jun 18, 2026
0c6f3cb
Add Harbor Terminus 2 harness (#1747)
xeophon Jun 18, 2026
a630d3b
Track cached, reasoning, and cost usage in v1 (#1704)
xeophon Jun 18, 2026
7bb546c
fix(v1): reuse pool tunnel for colocated tool/user state channel (#1752)
mikasenghaas Jun 18, 2026
b8fad84
feat: show per-reward/metric running means under the eval progress ba…
mikasenghaas Jun 18, 2026
4a8fd0a
feat(v1): INFO interception up/down + debug-log every intercepted rou…
mikasenghaas Jun 18, 2026
cd205ee
feat: type v1 rollout errors by boundary (#1751)
mikasenghaas Jun 18, 2026
7ad99f7
fix(v1): swallow SIGTERM KeyboardInterrupt in spawned env-server work…
mikasenghaas Jun 19, 2026
2474b05
feat(v1): own retries in the SDKs; drop framework per-call model/runt…
mikasenghaas Jun 19, 2026
e5b5350
fix: keep interception endpoint out of agent-inheritable env (#1758)
mikasenghaas Jun 19, 2026
e002e80
fix: retry in-sandbox /state + /task channel fetches; name us in prim…
mikasenghaas Jun 19, 2026
8c9f284
fix(v1): ModalRuntime.read resolves relative paths; tail tool logs un…
mikasenghaas Jun 19, 2026
1914af6
test(v1): mark swebench_verified_v1 as a container-only env (#1762)
mikasenghaas Jun 19, 2026
89c9522
fix: feed malformed tool-call JSON back to the model instead of crash…
mikasenghaas Jun 19, 2026
94b7df1
fix(swebench-verified-v1): pin task workdir to /testbed (#1765)
mikasenghaas Jun 19, 2026
6ff2ac5
Install pinned Harbor CLI on demand (#1745)
xeophon Jun 19, 2026
047633b
Preserve provider reasoning state across turns (#1748)
xeophon Jun 19, 2026
ac79618
Migrate Tau2 benchmark to v1 (#1692)
xeophon Jun 19, 2026
6ac5a3b
Separate V1 sandbox lifetimes from rollout timeouts (#1749)
xeophon Jun 19, 2026
2c5e004
chore: final touches (#1770)
mikasenghaas Jun 19, 2026
de430c1
chore: re-lock after the 0.2.0 bump and reverse-text examples removal
mikasenghaas Jun 19, 2026
3cc6183
fix: green the v1 e2e suite and run it in CI (host + docker) (#1771)
mikasenghaas Jun 19, 2026
28de935
style: wrap the general_agent_v1 skip line (ruff format)
mikasenghaas Jun 20, 2026
361491e
ci: drop feat/nano-as-v1 from test triggers (avoid duplicate matrix r…
mikasenghaas Jun 20, 2026
fd964e7
ci: target only the default branch (main) in test triggers
mikasenghaas Jun 20, 2026
8ae30c4
fix: declare the verifiers dependency in code-golf-v1 and aime24-v1
mikasenghaas Jun 20, 2026
e902485
Reduce Prime background job polling latency (#1777)
xeophon Jun 20, 2026
50e0377
optimize v1 trace graph construction (#1780)
xeophon Jun 20, 2026
1edba1d
Optimize routed-expert graph memory (#1783)
xeophon Jun 20, 2026
e349d7a
Optimize V1 SSE stream handling (#1784)
xeophon Jun 20, 2026
af1a6e2
Optimize V1 MCP and state data paths (#1785)
xeophon Jun 20, 2026
17d3bf7
Optimize General Agent V1 scoring imports (#1779)
xeophon Jun 20, 2026
ed0f079
Optimize Color Codeword task rendering (#1781)
xeophon Jun 20, 2026
6aaad81
Optimize TextArena task generation (#1778)
xeophon Jun 20, 2026
22f9daf
Optimize Wikispeedia V1 loading (#1782)
xeophon Jun 20, 2026
824d87e
Build train completion usage directly (#1787)
xeophon Jun 21, 2026
a3b19e8
Skip unavailable malloc_trim offloads (#1788)
xeophon Jun 21, 2026
04131a5
Avoid multimodal paths for text-only turns (#1789)
xeophon Jun 21, 2026
e16a03d
Avoid duplicate bridge completion copy (#1790)
xeophon Jun 21, 2026
9d8ef41
Cache MCP source archives (#1792)
xeophon Jun 21, 2026
7fb71fc
Serialize intercepted completions directly to JSON bytes (#1795)
xeophon Jun 21, 2026
17ba1ae
Reuse dashboard branch materialization (#1796)
xeophon Jun 21, 2026
8db1017
Tail remote MCP logs before downloading (#1797)
xeophon Jun 21, 2026
92e8ffb
Use native JSON bytes for upstream eval traffic (#1798)
xeophon Jun 21, 2026
4117896
Revert "Use native JSON bytes for upstream eval traffic (#1798)" (#1800)
xeophon Jun 21, 2026
6e497bc
Serialize upstream eval JSON directly to bytes (#1802)
xeophon Jun 21, 2026
c05bd7d
Lazily prepare bridged train prompts (#1801)
xeophon Jun 21, 2026
34c34fc
Use zero-copy ZMQ response relay (#1803)
xeophon Jun 21, 2026
fd134b2
Transfer completion logprobs without copying (#1804)
xeophon Jun 21, 2026
dd97f28
Avoid env-server trace recopy (#1809)
xeophon Jun 21, 2026
5c067b7
Match EvalClient pool to V1 concurrency (#1811)
xeophon Jun 21, 2026
5964239
Avoid copying trusted training token lists (#1813)
xeophon Jun 21, 2026
3d49747
Avoid gather for zero or one scoring hook (#1810)
xeophon Jun 21, 2026
d7e1bb9
Persist large V1 traces off the event loop (#1814)
xeophon Jun 21, 2026
15359ac
Release intercepted request bodies before inference (#1818)
xeophon Jun 21, 2026
276ed74
Use stateless JSON for vf MCP servers (#1822)
xeophon Jun 22, 2026
6e8b3cc
fix: carry v0 truncation flag through legacy bridge + declare compact…
mikasenghaas Jun 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
14 changes: 4 additions & 10 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,16 @@ This directory contains automated workflows for the verifiers project.
## Workflows

### 1. Style (`style.yml`)
**Purpose**: Code style checking using ruff, ty, and Semgrep policy rules.
**Purpose**: Code style checking using ruff and ty.

**Triggers**:
- Pull requests (opened, synchronized, reopened)
- Pushes to `main` branch

**What it does**:
- Runs ruff for linting and formatting checks
- Runs ty type checks with `uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses`
- Runs Semgrep policy checks from the isolated `policy` dependency group.
- Uses configuration from `pyproject.toml`, `.pre-commit-config.yaml`, and `.semgrep/verifiers.yml`
- Runs ty type checks with `uv run ty check verifiers`
- Uses configuration from `pyproject.toml` and `.pre-commit-config.yaml`

### 2. Test (`test.yml`)
**Purpose**: Comprehensive testing with coverage reports.
Expand All @@ -36,8 +35,6 @@ This directory contains automated workflows for the verifiers project.

**Workflows**:
- `publish-verifiers.yml` publishes `verifiers` with trusted publishing. On every push to `main` it builds and publishes a pre-release (`skip-existing`), and on a pushed `vX.Y.Z` tag (or manual dispatch) it publishes that stable release and creates a GitHub release.
- `publish-tasksets.yml` publishes `tasksets` from `tasksets-v*` tags with trusted publishing. On `main`, it creates `tasksets-v<version>` when `packages/tasksets/tasksets/__init__.py` defines `__version__` and the matching remote tag does not already exist, then builds and publishes from that tag in the same workflow run.
- `publish-harnesses.yml` publishes `harnesses` from `harnesses-v*` tags with trusted publishing. On `main`, it creates `harnesses-v<version>` when `packages/harnesses/harnesses/__init__.py` defines `__version__` and the matching remote tag does not already exist, then builds and publishes from that tag in the same workflow run.
- `publish-verifiers-rl.yml` publishes `verifiers-rl` from `verifiers-rl-v*` tags.

## Setting Up
Expand All @@ -55,10 +52,7 @@ To run checks locally the same way they run in CI:

```bash
# Ty parity with CI (Python 3.13 target configured in `pyproject.toml`)
uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses

# Verifiers-specific policy lint
env PYTHONWARNINGS=ignore::SyntaxWarning uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet
uv run ty check verifiers

# Tests
uv sync
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/publish-envs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ jobs:
- name: Set matrix
id: set-matrix
run: |
# Get all environment directories
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | jq -R . | jq -sc .)
# Auto-publish only the classic (v0) envs for now; v1 envs (the *_v1 packages
# and the `compact` harness example) aren't hub-published yet.
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | { grep -vE '_v1$|^compact$' || true; } | jq -R . | jq -sc .)

if [ "$ENV_IDS" = "[]" ] || [ "$ENV_IDS" = "null" ]; then
echo "has_envs=false" >> $GITHUB_OUTPUT
Expand Down
24 changes: 13 additions & 11 deletions .github/workflows/publish-harnesses.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,15 @@ jobs:
echo "created=false" >> "$GITHUB_OUTPUT"

VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import re
import sys

match = re.search(r'__version__\s*=\s*"([^"]+)"', Path("packages/harnesses/harnesses/__init__.py").read_text())
if not match:
sys.exit("Could not find __version__ in packages/harnesses/harnesses/__init__.py")
print(match.group(1))
data = tomllib.loads(Path("packages/harnesses/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/harnesses/pyproject.toml")
print(version)
PY
)

Expand Down Expand Up @@ -158,19 +159,20 @@ jobs:

VERSION="${TAG#harnesses-v}"
FILE_VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import re
import sys

match = re.search(r'__version__\s*=\s*"([^"]+)"', Path("packages/harnesses/harnesses/__init__.py").read_text())
if not match:
sys.exit("Could not find __version__ in packages/harnesses/harnesses/__init__.py")
print(match.group(1))
data = tomllib.loads(Path("packages/harnesses/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/harnesses/pyproject.toml")
print(version)
PY
)

if [ "$FILE_VERSION" != "$VERSION" ]; then
echo "Version mismatch: tag requests '$VERSION' but packages/harnesses/harnesses/__init__.py defines '$FILE_VERSION'" >&2
echo "Version mismatch: tag requests '$VERSION' but packages/harnesses/pyproject.toml defines '$FILE_VERSION'" >&2
exit 1
fi

Expand Down
24 changes: 13 additions & 11 deletions .github/workflows/publish-tasksets.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,15 @@ jobs:
echo "created=false" >> "$GITHUB_OUTPUT"

VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import re
import sys

match = re.search(r'__version__\s*=\s*"([^"]+)"', Path("packages/tasksets/tasksets/__init__.py").read_text())
if not match:
sys.exit("Could not find __version__ in packages/tasksets/tasksets/__init__.py")
print(match.group(1))
data = tomllib.loads(Path("packages/tasksets/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/tasksets/pyproject.toml")
print(version)
PY
)

Expand Down Expand Up @@ -158,19 +159,20 @@ jobs:

VERSION="${TAG#tasksets-v}"
FILE_VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import re
import sys

match = re.search(r'__version__\s*=\s*"([^"]+)"', Path("packages/tasksets/tasksets/__init__.py").read_text())
if not match:
sys.exit("Could not find __version__ in packages/tasksets/tasksets/__init__.py")
print(match.group(1))
data = tomllib.loads(Path("packages/tasksets/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/tasksets/pyproject.toml")
print(version)
PY
)

if [ "$FILE_VERSION" != "$VERSION" ]; then
echo "Version mismatch: tag requests '$VERSION' but packages/tasksets/tasksets/__init__.py defines '$FILE_VERSION'" >&2
echo "Version mismatch: tag requests '$VERSION' but packages/tasksets/pyproject.toml defines '$FILE_VERSION'" >&2
exit 1
fi

Expand Down
20 changes: 1 addition & 19 deletions .github/workflows/style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,22 +41,4 @@ jobs:
- name: Install dependencies
run: uv sync
- name: Run ty
run: uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses
semgrep:
name: Semgrep
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Install dependencies
run: uv sync --no-dev --group policy
- name: Run Semgrep policy checks
run: env PYTHONWARNINGS=ignore::SyntaxWarning uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet
run: uv run ty check verifiers
17 changes: 6 additions & 11 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@ name: Test

on:
pull_request:
branches: [main, master]
branches: [main]
paths:
- "**.py"
- "pyproject.toml"
- "packages/**"
- ".github/workflows/test.yml"
push:
branches: [main, master]
branches: [main]
paths:
- "**.py"
- "pyproject.toml"
Expand All @@ -30,7 +30,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
python-version: ["3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -61,16 +61,11 @@ jobs:

- name: Run tests
run: |
uv run pytest tests/ -v --ignore=tests/test_envs.py -m "not prime_sandbox" --cov=verifiers --cov-report=xml --cov-report=term
uv run pytest tests/ -v --ignore=tests/test_envs.py --ignore=tests/v1 -m "not prime" --cov=verifiers --cov-report=xml --cov-report=term

- name: Run Prime sandbox tests
env:
PRIME_API_KEY: ${{ secrets.PRIME_API_KEY }}
PRIME_TEAM_ID: ${{ secrets.PRIME_TEAM_ID }}
- name: Run v1 tests
run: |
test -n "$PRIME_API_KEY"
test -n "$PRIME_TEAM_ID"
uv run pytest tests/test_v1_runtime_lifecycle.py -v -m prime_sandbox --cov=verifiers --cov-append --cov-report=xml --cov-report=term
uv run pytest tests/v1 -vv -n auto -m "not prime and not modal" --cov=verifiers --cov-append --cov-report=xml --cov-report=term
test-envs:
name: Environments
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
Expand Down
7 changes: 1 addition & 6 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,6 @@ repos:
entry: uv run ruff format
language: system
types_or: [python, pyi]
- id: semgrep-v1-policy
name: Semgrep v1 policy
entry: uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet
language: system
pass_filenames: false
- id: sync-agents-md
name: Check generated AGENTS.md / CLAUDE.md
entry: uv run python scripts/sync.py --check
Expand All @@ -28,7 +23,7 @@ repos:
pass_filenames: false
- id: ty
name: ty (ci parity)
entry: uv run --python 3.13 ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses
entry: uv run --python 3.13 ty check verifiers
language: system
pass_filenames: false
stages: [pre-push]
Loading
Loading