Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
167 commits
Select commit Hold shift + click to select a range
6873716
refactor!: set up nano-as-v1 (vendor vf-nano as verifiers.v1)
mikasenghaas Jun 9, 2026
d4a0b97
feat(v1): parse vLLM token ids + logprobs in the chat client (#1585)
mikasenghaas Jun 9, 2026
08c090b
fix(v1): preserve rollout meta in the v0 -> v1 legacy bridge (#1586)
mikasenghaas Jun 9, 2026
15761a8
chore(v1): suffix example tasksets with -v1 to avoid name clashes (#1…
mikasenghaas Jun 9, 2026
db1b522
feat(v1): native rollout retries (EnvConfig.retry) (#1588)
mikasenghaas Jun 9, 2026
70795d3
feat: per-rollout token limits (max_input/output/total_tokens) (#1591)
mikasenghaas Jun 9, 2026
02dcc77
fix: free runtime resources on exit/interrupt (#1590)
mikasenghaas Jun 9, 2026
ceb55a4
feat: add textarena taskset with a framework-driven user simulator (#…
mikasenghaas Jun 9, 2026
f4eedfe
feat: add alphabet-sort-v1 taskset (#1595)
mikasenghaas Jun 9, 2026
75c98ec
feat: name runtime resources after the rollout id (#1596)
mikasenghaas Jun 9, 2026
7ce3b57
feat: add modal runtime (#1594)
mikasenghaas Jun 9, 2026
3592e15
feat: resolve taskset / harness / env ids from the hub (#1597)
mikasenghaas Jun 10, 2026
a75af8c
feat: run v0 environments on the eval CLI (legacy bridge) (#1598)
mikasenghaas Jun 10, 2026
50d9876
feat: first-class v1 harnesses/tasksets packages (+ README, serve/CLI…
mikasenghaas Jun 10, 2026
b0d7f8e
fix: gate task-tool tasksets on harness support (SUPPORTS_TASK_TOOLS)…
mikasenghaas Jun 10, 2026
4a388ea
feat: enable labels for prime sandboxes + tunnels (#1604)
mikasenghaas Jun 10, 2026
940fedf
feat: multiplex interception servers + tunnels across rollouts (#1605)
mikasenghaas Jun 10, 2026
964bc22
feat: delta-native message-graph trajectory (#1606)
mikasenghaas Jun 10, 2026
64dde83
refactor: inline branch building into Trace.branches
mikasenghaas Jun 10, 2026
4110fa5
chore: add v1 end-to-end eval test suite (#1609)
mikasenghaas Jun 10, 2026
1f7d2c6
fix: honor group scoring in the v0 legacy env server (#1612)
mikasenghaas Jun 10, 2026
fd86bff
chore: guard the v0 legacy bridge against non-renderer training clien…
mikasenghaas Jun 10, 2026
d7b12e6
fix: serve the eval split for eval-only v0 envs in the legacy bridge …
mikasenghaas Jun 10, 2026
f72e7d9
fix: build a chat-completions client for v0 eval in the legacy bridge…
mikasenghaas Jun 10, 2026
a917bea
fix(v1): align message graph hashing with renderer prefixes (#1617)
eligotts Jun 11, 2026
98fd074
feat: add scaleswe v1 taskset + per-task setup/workdir hooks (#1616)
mikasenghaas Jun 11, 2026
b59e33f
chore: add v1 #1559-vs-#1576 comparison doc (#1619)
mikasenghaas Jun 11, 2026
adf4310
feat: multimodal images through the v1 message graph + color-codeword…
mikasenghaas Jun 11, 2026
cf97388
feat(v1): add r2e-gym-v1 taskset (#1624)
S1ro1 Jun 11, 2026
a969787
fix: treat overlong prompts as clean truncated rollouts (#1625)
mikasenghaas Jun 11, 2026
399cfef
feat: env-server worker pool for v1 (router + N workers) (#1623)
mikasenghaas Jun 11, 2026
7fed36a
fix(v1): configure logging in spawned env-server workers (#1626)
mikasenghaas Jun 11, 2026
c3cf536
fix(v1): fall back to provider usage for dashboard token counts (#1627)
mikasenghaas Jun 11, 2026
334595a
fix(v1): SIGTERM env-server teardown + raise renderers floor
mikasenghaas Jun 11, 2026
85f2d13
fix: reap the whole subprocess tree when a runtime run is cancelled (…
mikasenghaas Jun 11, 2026
7999268
feat: elastic env-server pool (static/elastic pool config) (#1629)
mikasenghaas Jun 11, 2026
5b8ede7
feat(v1): record per-rollout setup timing as a distinct phase (#1631)
mikasenghaas Jun 11, 2026
9303397
feat: per-call model + runtime retries (#1632)
mikasenghaas Jun 11, 2026
f580815
fix(v1): blank elapsed for not-yet-started rollouts in the dashboard …
mikasenghaas Jun 11, 2026
aa7bef4
fix(v1): drop every timing span's computed duration from to_wire (#1636)
mikasenghaas Jun 11, 2026
a69c544
fix: tee eval logs to output dir; keep them off the rich dashboard (#…
mikasenghaas Jun 11, 2026
0c48043
feat: add --resume for evals (re-run a previous run's missing/errored…
mikasenghaas Jun 11, 2026
75ccf5b
feat(v1): finalize hook + per-stage timeouts (#1640)
mikasenghaas Jun 11, 2026
7438b26
feat: configurable creation-rate limiter for modal + prime runtimes (…
mikasenghaas Jun 11, 2026
6c87f4b
fix: retry the user-sim connect at high concurrency (capture failures…
mikasenghaas Jun 12, 2026
f94ec7b
feat(v1): record sampled provenance on MessageNode (#1643)
eligotts Jun 12, 2026
b4faba2
fix: resume reads results.jsonl by newline, not splitlines (#1644)
mikasenghaas Jun 12, 2026
a06dd15
fix: run alphabet-sort user simulator on the host (colocated=False) (…
mikasenghaas Jun 12, 2026
02f15ae
chore: bump modal creates_per_sec default 5 -> 40 (#1646)
mikasenghaas Jun 12, 2026
d074dc9
feat(v1): add swelego-v1 taskset (SWE-Lego on rlm harness / prime run…
hallerite Jun 12, 2026
54d64b2
fix Harbor agent timeout mapping (#1647)
xeophon Jun 12, 2026
191a27d
feat: add v1 validate entrypoint + Taskset.validate hook (#1655)
mikasenghaas Jun 12, 2026
54343f9
fix(v1): carry multimodal tensors across the env-server wire (#1653)
mikasenghaas Jun 12, 2026
95c8f38
fix: route-detected wire dialects + 1:1 relay (chat/responses/anthrop…
mikasenghaas Jun 12, 2026
a850f6d
perf: run subprocess uv-scripts via a cached interpreter, not per-cal…
mikasenghaas Jun 12, 2026
b8f290a
feat(v1): codex built-in harness (on the Responses dialect + streamin…
mikasenghaas Jun 12, 2026
2c12489
docs: v1 user guide (#1662)
mikasenghaas Jun 13, 2026
95f973a
feat: add free-form `info` field on Trace for taskset-scraped metadat…
mikasenghaas Jun 13, 2026
d4a279c
Rename RLM ref to version (#1667)
xeophon Jun 13, 2026
24d7160
Add harness environment variables (#1668)
xeophon Jun 13, 2026
926b540
Ignore Anthropic service tier metadata (#1671)
xeophon Jun 13, 2026
f05ad41
Add Claude Code harness (#1669)
xeophon Jun 14, 2026
315e07d
feat(v1): capture MoE router-replay routed_experts on Trace/Graph (#1…
S1ro1 Jun 14, 2026
489072f
Merge origin/main into feat/nano-as-v1
mikasenghaas Jun 14, 2026
59d43db
fix(v1): import TypedDict from typing_extensions for py3.11 schema build
mikasenghaas Jun 14, 2026
d596deb
fix(v1): run prime-sandbox programs for the sandbox lifetime + retry-…
mikasenghaas Jun 14, 2026
b2d2955
fix(scaleswe-v1): resolve task images via Prime Artifact Registry (#1…
mikasenghaas Jun 14, 2026
1a07bbe
feat(scaleswe-v1): make Prime Artifact Registry opt-in (default off) …
mikasenghaas Jun 15, 2026
0ce2161
feat(v1): session-affinity routing header for cross-turn prefix reuse…
S1ro1 Jun 15, 2026
d186d03
feat: filter unavailable scaleswe-v1 images at load (#1679)
mikasenghaas Jun 15, 2026
87dab85
feat(r2e-gym-v1): make Prime Artifact Registry opt-in (default off) (…
mikasenghaas Jun 15, 2026
76dc1d4
fix(scaleswe-v1): filter on Docker Hub image_url, not the resolved im…
mikasenghaas Jun 15, 2026
a4aaeca
chore: remove the Claude Code harness (#1684)
mikasenghaas Jun 15, 2026
486440a
fix: harnesses install/run without root (pinned /tmp/vf-{harness}) (#…
mikasenghaas Jun 15, 2026
2f28f16
test: skip test_multi_turn on prime (colocated user-sim port exposure…
mikasenghaas Jun 15, 2026
dfc073d
Add mini-swe-agent V1 harness (#1673)
xeophon Jun 15, 2026
f172e9f
Fix mini-swe-agent E2E matrix (#1693)
xeophon Jun 15, 2026
5624ca9
Support images in v1 tool responses (#1694)
eligotts Jun 16, 2026
4a5b4a2
map v1 reasoning effort by dialect (#1690)
xeophon Jun 16, 2026
64e7cab
Add Kimi Code V1 harness (#1675)
xeophon Jun 16, 2026
da609dd
Ignore unsupported Anthropic service tiers (#1689)
xeophon Jun 16, 2026
3833c39
Keep relayed SSE streams alive (#1688)
xeophon Jun 16, 2026
f1568d5
Add Harbor task multipliers (#1700)
xeophon Jun 16, 2026
6424221
Extend the V1 bash tool timeout (#1701)
xeophon Jun 16, 2026
cabbbdf
Use Prime CLI config for v1 eval (#1703)
xeophon Jun 16, 2026
058cb54
chore: flatten examples/ into a single environments/ section (#1695)
mikasenghaas Jun 16, 2026
cf6f703
feat: vf-native Toolset/User class surface + per-server runtime place…
mikasenghaas Jun 16, 2026
8594115
chore: resolve v1 plugins via __all__ export, split into taskset.py/h…
mikasenghaas Jun 16, 2026
2aa27b6
feat: allow user to open the conversation when a task has no prompt (…
mikasenghaas Jun 16, 2026
1bfe4c4
feat: typed transient Trace.state shared across tool/user servers + s…
mikasenghaas Jun 16, 2026
0d7dee0
fix(v1): relax MCP DNS-rebinding guard for tunneled host-bound server…
mikasenghaas Jun 16, 2026
c3dd132
feat: add an `init` scaffolding command for new v1 environments (#1713)
mikasenghaas Jun 16, 2026
8a5d15e
feat: default the harness runtime to subprocess (#1719)
mikasenghaas Jun 17, 2026
1a529a3
feat: train-client prefix bridging + token-based message-graph branch…
mikasenghaas Jun 17, 2026
7e0dfc4
chore: run v1 tests with -n auto in a separate CI step (#1718)
mikasenghaas Jun 17, 2026
3aacebf
test(envs): keep test_envs.py v0-only, add tests/v1/test_envs.py for …
mikasenghaas Jun 17, 2026
2015214
Add per-harness tool disabling (#1723)
xeophon Jun 17, 2026
7e087c8
Forward chat completion headers and consolidate clients (#1666)
xeophon Jun 17, 2026
bb89b27
warn on local subprocess harnesses (#1722)
xeophon Jun 17, 2026
69e9100
build: trim runtime dependencies (#1702)
xeophon Jun 17, 2026
ac3edb8
feat(v1): per-rollout isolation for shared writable tool servers (#1727)
mikasenghaas Jun 17, 2026
8c07327
feat(v1): fork-per-rollout isolation for shared tool servers (#1729)
mikasenghaas Jun 17, 2026
75990f2
chore: general v1 cleanup (#1732)
mikasenghaas Jun 17, 2026
e7bbfc2
fix: unified, robust env-server teardown (no leaked tunnels/sandboxes…
mikasenghaas Jun 18, 2026
86d4824
feat: general-agent solver v1 taskset (#1734)
mikasenghaas Jun 18, 2026
6f46f3a
fix: reject unsupported task system_prompt (#1735)
mikasenghaas Jun 18, 2026
938c00c
chore(v1): slow dashboard page rotation to 5s (#1737)
mikasenghaas Jun 18, 2026
f6c755e
fix: incomplete URL substring sanitization in quest pdf tool (CodeQL)…
mikasenghaas Jun 18, 2026
d6f32a0
feat: swebench-verified harbor taskset (prime prebuilt images) (#1738)
mikasenghaas Jun 18, 2026
791147e
fix: don't retry the rollout's program exec (forked duplicate trace b…
mikasenghaas Jun 18, 2026
5930038
fix: keep the swebench/ org in swebench-verified-v1 prime-registry im…
mikasenghaas Jun 18, 2026
b44be5c
fix(general-agent-v1): make the toolset work when colocated in a sand…
mikasenghaas Jun 18, 2026
8a568ea
fix: add backoff to per-call retries; let harness clients retry the r…
mikasenghaas Jun 18, 2026
0ec11a6
Add Harbor Terminus 2 harness (#1747)
xeophon Jun 18, 2026
04326a7
Track cached, reasoning, and cost usage in v1 (#1704)
xeophon Jun 18, 2026
cd49303
fix(v1): reuse pool tunnel for colocated tool/user state channel (#1752)
mikasenghaas Jun 18, 2026
eda7aa7
feat: show per-reward/metric running means under the eval progress ba…
mikasenghaas Jun 18, 2026
af1b56e
feat(v1): INFO interception up/down + debug-log every intercepted rou…
mikasenghaas Jun 18, 2026
6a9002a
feat: type v1 rollout errors by boundary (#1751)
mikasenghaas Jun 18, 2026
b35e4f3
fix(v1): swallow SIGTERM KeyboardInterrupt in spawned env-server work…
mikasenghaas Jun 19, 2026
5d65505
feat(v1): own retries in the SDKs; drop framework per-call model/runt…
mikasenghaas Jun 19, 2026
9ed5fbf
fix: keep interception endpoint out of agent-inheritable env (#1758)
mikasenghaas Jun 19, 2026
7ec8f39
fix: retry in-sandbox /state + /task channel fetches; name us in prim…
mikasenghaas Jun 19, 2026
1edafdf
fix(v1): ModalRuntime.read resolves relative paths; tail tool logs un…
mikasenghaas Jun 19, 2026
cf73176
test(v1): mark swebench_verified_v1 as a container-only env (#1762)
mikasenghaas Jun 19, 2026
76f43ce
fix: feed malformed tool-call JSON back to the model instead of crash…
mikasenghaas Jun 19, 2026
3b060d8
fix(swebench-verified-v1): pin task workdir to /testbed (#1765)
mikasenghaas Jun 19, 2026
cc91026
Install pinned Harbor CLI on demand (#1745)
xeophon Jun 19, 2026
07fb7f1
Preserve provider reasoning state across turns (#1748)
xeophon Jun 19, 2026
d8fc64f
Migrate Tau2 benchmark to v1 (#1692)
xeophon Jun 19, 2026
18e1c4d
Separate V1 sandbox lifetimes from rollout timeouts (#1749)
xeophon Jun 19, 2026
e2715d7
chore: final touches (#1770)
mikasenghaas Jun 19, 2026
95b04ed
chore: re-lock after the 0.2.0 bump and reverse-text examples removal
mikasenghaas Jun 19, 2026
41882b1
fix: green the v1 e2e suite and run it in CI (host + docker) (#1771)
mikasenghaas Jun 19, 2026
e0924de
chore: CI triggers, deps, and formatting cleanup
mikasenghaas Jun 20, 2026
1a09d5d
Reduce Prime background job polling latency (#1777)
xeophon Jun 20, 2026
39e4e32
optimize v1 trace graph construction (#1780)
xeophon Jun 20, 2026
dee6afb
Optimize routed-expert graph memory (#1783)
xeophon Jun 20, 2026
5ce290e
Optimize V1 SSE stream handling (#1784)
xeophon Jun 20, 2026
468fc60
Optimize V1 MCP and state data paths (#1785)
xeophon Jun 20, 2026
7696f86
Optimize General Agent V1 scoring imports (#1779)
xeophon Jun 20, 2026
6353dbf
Optimize Color Codeword task rendering (#1781)
xeophon Jun 20, 2026
7fec272
Optimize TextArena task generation (#1778)
xeophon Jun 20, 2026
3f78889
Optimize Wikispeedia V1 loading (#1782)
xeophon Jun 20, 2026
d990cd4
Build train completion usage directly (#1787)
xeophon Jun 21, 2026
496be00
Skip unavailable malloc_trim offloads (#1788)
xeophon Jun 21, 2026
d008109
Avoid multimodal paths for text-only turns (#1789)
xeophon Jun 21, 2026
e0c450c
Avoid duplicate bridge completion copy (#1790)
xeophon Jun 21, 2026
cb8d25e
Cache MCP source archives (#1792)
xeophon Jun 21, 2026
6509c75
Serialize intercepted completions directly to JSON bytes (#1795)
xeophon Jun 21, 2026
0ab9943
Reuse dashboard branch materialization (#1796)
xeophon Jun 21, 2026
c7ef2f1
Tail remote MCP logs before downloading (#1797)
xeophon Jun 21, 2026
7aa77ae
Use native JSON bytes for upstream eval traffic (#1798)
xeophon Jun 21, 2026
28b8b9b
Revert "Use native JSON bytes for upstream eval traffic (#1798)" (#1800)
xeophon Jun 21, 2026
ca066bb
Serialize upstream eval JSON directly to bytes (#1802)
xeophon Jun 21, 2026
b4e85f6
Lazily prepare bridged train prompts (#1801)
xeophon Jun 21, 2026
83018e3
Use zero-copy ZMQ response relay (#1803)
xeophon Jun 21, 2026
7e5efe9
Transfer completion logprobs without copying (#1804)
xeophon Jun 21, 2026
ee7cd9c
Avoid env-server trace recopy (#1809)
xeophon Jun 21, 2026
de35dba
Match EvalClient pool to V1 concurrency (#1811)
xeophon Jun 21, 2026
5772f6e
Avoid copying trusted training token lists (#1813)
xeophon Jun 21, 2026
c11bbfa
Avoid gather for zero or one scoring hook (#1810)
xeophon Jun 21, 2026
576ed2c
Persist large V1 traces off the event loop (#1814)
xeophon Jun 21, 2026
1fe031f
Release intercepted request bodies before inference (#1818)
xeophon Jun 21, 2026
2adf5f1
Use stateless JSON for vf MCP servers (#1822)
xeophon Jun 22, 2026
e57c064
fix: carry v0 truncation flag through legacy bridge + declare compact…
mikasenghaas Jun 22, 2026
71c81bc
refactor: move built-in tasksets/harnesses into verifiers.v1 submodul…
mikasenghaas Jun 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
14 changes: 4 additions & 10 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,16 @@ This directory contains automated workflows for the verifiers project.
## Workflows

### 1. Style (`style.yml`)
**Purpose**: Code style checking using ruff, ty, and Semgrep policy rules.
**Purpose**: Code style checking using ruff and ty.

**Triggers**:
- Pull requests (opened, synchronized, reopened)
- Pushes to `main` branch

**What it does**:
- Runs ruff for linting and formatting checks
- Runs ty type checks with `uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses`
- Runs Semgrep policy checks from the isolated `policy` dependency group.
- Uses configuration from `pyproject.toml`, `.pre-commit-config.yaml`, and `.semgrep/verifiers.yml`
- Runs ty type checks with `uv run ty check verifiers`
- Uses configuration from `pyproject.toml` and `.pre-commit-config.yaml`

### 2. Test (`test.yml`)
**Purpose**: Comprehensive testing with coverage reports.
Expand All @@ -36,8 +35,6 @@ This directory contains automated workflows for the verifiers project.

**Workflows**:
- `publish-verifiers.yml` publishes `verifiers` with trusted publishing. On every push to `main` it builds and publishes a pre-release (`skip-existing`), and on a pushed `vX.Y.Z` tag (or manual dispatch) it publishes that stable release and creates a GitHub release.
- `publish-tasksets.yml` publishes `tasksets` from `tasksets-v*` tags with trusted publishing. On `main`, it creates `tasksets-v<version>` when `packages/tasksets/tasksets/__init__.py` defines `__version__` and the matching remote tag does not already exist, then builds and publishes from that tag in the same workflow run.
- `publish-harnesses.yml` publishes `harnesses` from `harnesses-v*` tags with trusted publishing. On `main`, it creates `harnesses-v<version>` when `packages/harnesses/harnesses/__init__.py` defines `__version__` and the matching remote tag does not already exist, then builds and publishes from that tag in the same workflow run.
- `publish-verifiers-rl.yml` publishes `verifiers-rl` from `verifiers-rl-v*` tags.

## Setting Up
Expand All @@ -55,10 +52,7 @@ To run checks locally the same way they run in CI:

```bash
# Ty parity with CI (Python 3.13 target configured in `pyproject.toml`)
uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses

# Verifiers-specific policy lint
env PYTHONWARNINGS=ignore::SyntaxWarning uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet
uv run ty check verifiers

# Tests
uv sync
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/publish-envs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ jobs:
- name: Set matrix
id: set-matrix
run: |
# Get all environment directories
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | jq -R . | jq -sc .)
# Auto-publish only the classic (v0) envs for now; v1 envs (the *_v1 packages
# and the `compact` harness example) aren't hub-published yet.
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | { grep -vE '_v1$|^compact$' || true; } | jq -R . | jq -sc .)

if [ "$ENV_IDS" = "[]" ] || [ "$ENV_IDS" = "null" ]; then
echo "has_envs=false" >> $GITHUB_OUTPUT
Expand Down
208 changes: 0 additions & 208 deletions .github/workflows/publish-harnesses.yml

This file was deleted.

Loading
Loading