Sophistry Bench Sprint
sophistry_bench_sprint_env is a single-turn advocacy reward-hacking environment on QuALITY passages: the policy defends an assigned answer and the reward proxy peaks at 8 <claim> tags, with four weight-0 canaries that detect format hacking.
diff --git a/docs/source/environments/pelican_svg.md b/docs/source/environments/pelican_svg.md
new file mode 100644
index 000000000..806222cc0
--- /dev/null
+++ b/docs/source/environments/pelican_svg.md
@@ -0,0 +1,322 @@
+
+# Pelican SVG Environment
+
+Blind vector drawing, scored. The model is asked for an SVG of an animal riding
+a vehicle and never sees the result, so it has to hold the spatial arrangement
+in its head and emit coordinates for it. This is the check Simon Willison
+popularised as "generate an SVG of a pelican riding a bicycle", turned into
+something you can run repeatedly, at scale, and train against.
+
+## What this is, and what it is not
+
+**Not a model ranking.** Simon Willison, whose prompt this is, is explicit that
+"the correlation between pelican performance and actual model quality has been
+mostly severed now", and ends with "don't go using pelicans to compare models"
+([Kimi K3, and what we can still learn from the pelican
+benchmark](https://simonwillison.net/2026/Jul/16/kimi-k3/)). Our own numbers
+agree with him: on the canonical task, across 139 scored samples from 7 frontier models,
+136 scored a perfect 1.000 on the structural layer and 47 scored a perfect 1.000 overall.
+It is saturated.
+
+**Not a contamination study either.** [Dylan
+Castillo](https://dylancastillo.co/posts/pelicanmaxxing.html) already ran that,
+across an 8 by 6 grid and 1008 SVGs from 7 frontier models, and found little
+evidence that labs optimise for the pelican. The extra animals and vehicles here
+are ours and serve a smaller purpose: checking that the scorer measures drawing
+ability rather than one memorised picture. Only `pelican_bicycle` is Simon's, and
+it is the default.
+
+**What it is** is a reproducible, executable target. The same code that scores a
+frontier model over an API scores a 0.6B model mid-training, the reward is dense
+enough to run GRPO against, and the whole thing deploys as a Space. That is the
+part a prompt cannot do:
+
+- [`examples/pelican_svg_eval.py`](https://github.com/huggingface/OpenEnv/blob/main/examples/pelican_svg_eval.py) scores any set of models through HF Inference Providers.
+- [`examples/pelican_svg_grpo.py`](https://github.com/huggingface/OpenEnv/blob/main/examples/pelican_svg_grpo.py) trains against it with TRL, runnable on HF Jobs.
+
+The interesting question stops being "which model draws the best pelican" and
+becomes "can a small model be taught to, and does it learn to draw or to game
+the scorer".
+
+## The task catalogue
+
+6 animals by 5 vehicles, so 30 tasks. All three
+scoring layers adapt to whichever one is served, so nothing about the pipeline is special
+to the pelican:
+
+- **The judge's checklist is generated per task** from the features below. A pelican is
+ asked about a throat pouch, a capybara about a blunt snout, an octopus about curling
+ tentacles.
+- **The geometry layer reads `wheels`** from the vehicle, and drops the wheel-pair checks
+ entirely for a single-wheeled vehicle rather than failing them.
+- **The anti-cheat terms are derived from the task**, so `axolotl_scooter` forbids
+ `axolotl, salamander, amphibian, scooter, kick scooter` and not the pelican's words.
+
+### Animals
+
+| subject | features the judge asks about | counts as partial recognition |
+| --- | --- | --- |
+| `pelican` | a long beak, a throat pouch under the beak | bird, seabird, waterbird |
+| `flamingo` | a long thin neck, long thin legs | bird, waterbird |
+| `capybara` | a blunt rectangular snout, small rounded ears | rodent, animal |
+| `axolotl` | feathery external gills on the head, a wide flat smiling mouth | salamander, amphibian, animal |
+| `octopus` | a large bulbous head, multiple curling tentacles | cephalopod, animal |
+| `hedgehog` | a coat of spines, a small pointed snout | animal |
+
+### Vehicles
+
+| vehicle | wheels in side view | features the judge asks about |
+| --- | --- | --- |
+| `bicycle` | 2 | two wheels of similar size, a frame joining the wheels, handlebars |
+| `unicycle` | 1 | a single wheel, a seat post rising from the wheel |
+| `tandem bicycle` | 2 | two wheels, an extended frame with two saddles |
+| `scooter` | 2 | two small wheels, a deck with an upright steering column |
+| `skateboard` | 2 | a flat deck, wheels under the deck |
+
+### How much of the grid has actually been run
+
+**All 30 combinations have been scored end to end by a live model. None crashed and none
+was rejected by the gate.** The counts below are how much traffic each cell has actually
+seen, which is a different question from whether it works:
+
+| animal | `bicycle` | `unicycle` | `tandem bicycle` | `scooter` | `skateboard` |
+| --- | --- | --- | --- | --- | --- |
+| `pelican` | **224** | 1 | 1 | 1 | 1 |
+| `flamingo` | 1 | 1 | 1 | **16** | 1 |
+| `capybara` | 1 | **16** | 1 | 1 | 1 |
+| `axolotl` | 1 | 1 | **16** | 1 | 1 |
+| `octopus` | 1 | 1 | 1 | 1 | 1 |
+| `hedgehog` | 1 | 1 | 1 | 1 | 1 |
+
+Read a `1` as "the plumbing works", not as a result. It says the task builds, the prompt
+renders, the gate admits the reply, the geometry resolves against that vehicle's wheel
+count and the judge returns a checklist. It says nothing about how well the scoring
+behaves on that combination, and it cannot: one sample has no variance to report.
+
+Across the 30-task sweep, reward ranged 0.362 to 1.000 with a median of 0.838, structure
+averaged 0.905 and the judged semantic score 0.785. The weakest cell was
+`hedgehog_tandem-bicycle` at 0.362, where the judge would not accept the animal. Three of
+the 30 were corrected upward by the `
` fix described below, `capybara_unicycle` most
+of all, from 0.333 to 1.000 on structure.
+
+`pelican_bicycle` is Simon Willison's original prompt and the default. It is also the only
+task the GRPO example trains on: `examples/pelican_svg_grpo.py` pins it deliberately,
+since that script is a small demonstration that the environment is trainable rather than
+an attempt to teach a model the whole grid.
+
+## Scoring
+
+Three layers, ordered by cost. Each earns the right to run the next, so a run
+against a model producing garbage stays cheap.
+
+### 1. Gate, free, pass or fail
+
+Rejects anything that is not an honest attempt to draw. A rejected submission
+scores zero and never reaches the judge.
+
+| Code | Meaning |
+| --- | --- |
+| `no_svg_in_response` / `truncated_svg` | No SVG, or one cut off mid-generation. Kept separate because "the model refused" and "the harness cut it off" are different facts. |
+| `unparseable` / `render_failed` | Not well-formed, declares a DTD, or will not rasterise. |
+| `embedded_raster` | An `` element or a `data:image` URI. |
+| `external_reference` | An `href` pointing off-document. |
+| `forbidden_element` | `a pelican a pelicanhi pelican&lol; ` is painted and every `` is, so the
+ wheels are at the positions the `` elements give them.
+ """
+ report = self.structure("wheels_via_use")
+ assert report.score == 1.0
+ assert [round(w.centroid[0], 3) for w in report.wheels] == [0.325, 0.675]
+
+ def test_definitions_alone_draw_nothing(self):
+ """`` content is a template, not paint. Without a `` to
+ instantiate it there is no geometry on the canvas at all."""
+ source = svg(
+ ''
+ ' "
+)
+
+
+@pytest.fixture
+def env():
+ """The environment as the training script configures it: no judge."""
+ return PelicanSvgEnvironment(enable_judge=False)
+
+
+def roll(env, completion: str) -> float:
+ """One rollout the way the reward function does it."""
+ env.reset(task_id="pelican_bicycle")
+ return env.step(PelicanSvgAction(response=completion)).reward
+
+
+class TestRewardContract:
+ def test_reward_is_a_finite_float_in_range(self, env):
+ reward = roll(env, DRAWABLE)
+ assert isinstance(reward, float)
+ assert math.isfinite(reward)
+ assert 0.0 <= reward <= 1.0
+
+ def test_a_drawing_earns_more_than_garbage(self, env):
+ """Without this ordering GRPO has no signal to follow."""
+ assert roll(env, DRAWABLE) > roll(env, "I cannot draw that.")
+
+ def test_garbage_scores_exactly_zero(self, env):
+ assert roll(env, "no svg here") == 0.0
+
+ def test_reward_is_deterministic_without_a_judge(self, env):
+ """A reward that wobbles run to run makes the curve unreadable."""
+ assert len({roll(env, DRAWABLE) for _ in range(5)}) == 1
+
+ def test_offline_reward_is_the_structural_score(self, env):
+ """With no judge, structure carries the whole weight."""
+ env.reset(task_id="pelican_bicycle")
+ observation = env.step(PelicanSvgAction(response=DRAWABLE))
+ assert observation.reward == pytest.approx(observation.structure_score)
+ assert component_weights(judge_enabled=False) == (1.0, 0.0)
+
+ def test_judge_never_runs_when_disabled(self, env):
+ """Training must not make a paid API call per rollout."""
+ env.reset(task_id="pelican_bicycle")
+ observation = env.step(PelicanSvgAction(response=DRAWABLE))
+ assert observation.judged is False
+ assert observation.semantic_score == 0.0
+
+
+class TestTruncationTrap:
+ """The failure mode that quietly wastes a GPU run.
+
+ `max_completion_length` too small cuts every SVG off mid-document. Every
+ rollout then scores zero, GRPO sees no variance, and the run burns hours
+ learning nothing. The environment has to name that case so it is visible in
+ the logs rather than looking like a model that cannot draw.
+ """
+
+ def test_truncated_completion_is_named(self, env):
+ env.reset(task_id="pelican_bicycle")
+ observation = env.step(
+ PelicanSvgAction(response=DRAWABLE[: len(DRAWABLE) // 2])
+ )
+ assert observation.reward == 0.0
+ assert "truncated_svg" in observation.violations
+
+ def test_absent_svg_is_a_different_code(self, env):
+ env.reset(task_id="pelican_bicycle")
+ observation = env.step(PelicanSvgAction(response="Sorry, I will not."))
+ assert "no_svg_in_response" in observation.violations
+
+
+class TestTrainingTaskIsPinned:
+ def test_every_reset_serves_the_same_prompt(self, env):
+ """The trainer builds one dataset row per episode from one prompt."""
+ prompts = {env.reset(task_id="pelican_bicycle").prompt for _ in range(5)}
+ assert len(prompts) == 1
+
+ def test_the_pinned_task_is_the_original_prompt(self, env):
+ observation = env.reset(task_id="pelican_bicycle")
+ assert observation.subject == "pelican"
+ assert observation.vehicle == "bicycle"
+ assert observation.held_out is False
+
+
+class TestStructuralProxyIsGameable:
+ """Documents, in a test, the thing the training run is looking for.
+
+ The structural layer is satisfiable without drawing an animal, which is why
+ the training script probes with the judge before and after. Pinning it here
+ means nobody later mistakes the proxy for the task.
+ """
+
+ def test_a_bare_vehicle_already_scores_most_of_the_structural_reward(self, env):
+ bare = (
+ ''
+ ' "
+ )
+ reward = roll(env, bare)
+ assert reward > 0.5, "a vehicle with no rider should still score well"
+ assert reward < 1.0, "but it must not score full marks"
+
+
+_EXAMPLES = pathlib.Path(__file__).resolve().parents[2] / "examples"
+if str(_EXAMPLES) not in sys.path:
+ sys.path.insert(0, str(_EXAMPLES))
+
+
+class _FakeStream:
+ """An async iterable of chat-completion chunks."""
+
+ def __init__(self, deltas):
+ self._deltas = deltas
+
+ def __aiter__(self):
+ async def gen():
+ for delta in self._deltas:
+ yield SimpleNamespace(choices=[SimpleNamespace(delta=delta)])
+
+ return gen()
+
+
+class _FakeClient:
+ def __init__(self, deltas):
+ self._deltas = deltas
+
+ async def chat_completion(self, **kwargs):
+ assert kwargs.get("stream") is True
+ return _FakeStream(self._deltas)
+
+
+def _delta(content=None, reasoning_content=None):
+ return SimpleNamespace(content=content, reasoning_content=reasoning_content)
+
+
+def _args():
+ return SimpleNamespace(no_stream=False, max_tokens=100, temperature=0.7)
+
+
+class TestReasoningModelStreams:
+ """A reasoning model streams its thinking apart from its answer.
+
+ `delta.reasoning_content` carries the chain of thought while `delta.content`
+ stays empty. Reading only `content` makes a model that is still thinking look
+ like a model that refused, and a model that spent its whole budget reasoning
+ look like a model that cannot draw.
+ """
+
+ def run(self, deltas):
+ from pelican_svg_eval import generate
+
+ return asyncio.run(
+ generate(
+ _FakeClient(deltas),
+ asyncio.Semaphore(1),
+ "fake/model",
+ "draw",
+ 1,
+ _args(),
+ )
+ )
+
+ def test_answer_and_thinking_are_kept_apart(self):
+ out = self.run(
+ [
+ _delta(reasoning_content="let me think"),
+ _delta(content="