[surajsharan]

Summary

The semantic half of the reward, weight 0.65, comes from the vision judge, and the judge could only reach Hugging Face Inference Providers: HFVisionClient never passed a base URL to AsyncInferenceClient, and _judge_from_env returned None without an HF token. Training with --judge-in-reward therefore paid for two judge calls per sample against a 72B model, and running offline dropped the component entirely, which the README already identifies as the thing that changes what training teaches.

• HFVisionClient takes a base_url, and _judge_from_env reads PELICAN_SVG_JUDGE_BASE_URL plus an optional PELICAN_SVG_JUDGE_API_KEY.
• A configured endpoint enables the judge with no Hugging Face token, and an ambient token is never forwarded to it: with a base URL the client sends a placeholder unless a key is given.
• PELICAN_SVG_JUDGE_MODEL is required alongside an endpoint, rather than defaulting to a 72B repository a local server is unlikely to serve. That follows the reasoning already in _judge_from_env: a judge whose every call fails is worse than no judge. server/app.py checks this once at startup, so a misconfigured server refuses to start with the fix in the message. Without that, the error is raised when each session builds its environment, and the server reports it to the client only as a generic FACTORY_ERROR (see Out of scope). The check reuses the private _judge_from_env from the sibling module rather than adding a public name; happy to rename it if you'd prefer.
• The hosted path is untouched: no base URL, token from the environment, same model default, same call shape.

This is a parameter passthrough rather than a new client. AsyncInferenceClient accepts base_url as an alias for model, and chat_completion routes to that URL while the per-call model name travels in the request body, which is what the judge already passes.

Closes #1216

Type of Change

• Bug fix
• New feature
• Documentation

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run bash .claude/hooks/lint.sh and tests and addressed all issues

RFC Status

• Not required (bug fix, docs, minor refactoring)

Env-local and additive: nothing in src/openenv/core/ changes and the hosted behaviour is identical.

Test Plan

Eleven new tests in tests/envs/test_pelican_svg_env.py, none of which touch the network:

• _judge_from_env: offline with no token and no endpoint; enabled by an endpoint without a token; PELICAN_SVG_DISABLE_JUDGE still wins; the hosted path still builds the same client; an endpoint without PELICAN_SVG_JUDGE_MODEL raises, and importing server.app with that configuration raises too, while importing it with a hosted HF_TOKEN configuration still succeeds.
• The Hugging Face token is not sent to an endpoint, and PELICAN_SVG_JUDGE_API_KEY is used when given.
• HFVisionClient request shape through a fake inference client: the hosted client gets no base URL, the endpoint client routes to the base URL while naming the model in the body, temperature stays 0.

pytest tests/envs/test_pelican_svg_env.py tests/envs/test_pelican_svg_training_contract.py → 166 passed with resvg-py installed. These files skip without it, in CI as well, which is how the environment already behaves on main. Full suite with CI's dependency set and GITHUB_ACTIONS=true: 2761 passed, 95 skipped, 0 failed. ruff check, ruff format --check, usort check and scripts/sync_env_docs.py --check pass.

End to end, through the server and client

Qwen/Qwen3.5-4B served by vLLM on one 24 GB GPU. The environment server was started with uvicorn envs.pelican_svg_env.server.app:app and configured only through PELICAN_SVG_JUDGE_BASE_URL and PELICAN_SVG_JUDGE_MODEL, plus a decoy HF_TOKEN. Episodes were driven with PelicanSvgEnv(...).sync() over the WebSocket, reset(task_id="pelican_bicycle") then step() with a repository fixture. Values are as reported on the observation:

fixture	reward	structure_score	semantic_score	judged
good_pelican_bike	0.919	1.000	0.875	true
bike_no_bird	0.336	0.714	0.132	true
bird_no_bike	0.056	0.000	0.087	true
bad_scribble	0.000	0.000	0.000	true

The blind captions were on target, for example "a cartoon-style illustration of a white bird with a yellow beak and feet" for the good drawing and "a starburst" shape for the scribble.

A recording proxy between the environment and vLLM captured every judge request in one log: 22 episodes on the final build (the four above, plus six drawings on each of pelican_bicycle, capybara_unicycle and flamingo_scooter), 44 judge calls, all returning 200:

• Every judge request carried Authorization: Bearer EMPTY; the decoy Hugging Face token appeared in none of them.
• Every request named Qwen/Qwen3.5-4B in the body, attached the image, and used temperature 0. Caption calls capped at 120 tokens, checklist calls at 400.
• 22 of 22 checklist replies contained every required key with a boolean value.

Failure paths, same setup:

• Endpoint down: reward 0.35 from structure alone, judged=False, feedback judge unavailable (ConnectError: ...), returned in 0.1 s. This is the existing, deliberate behaviour.
• No judge configured: structure carries the full weight (good_pelican_bike 1.000, bike_no_bird 0.714), unchanged.
• Endpoint without PELICAN_SVG_JUDGE_MODEL: uvicorn exits 1 at startup with ValueError: PELICAN_SVG_JUDGE_BASE_URL is set but PELICAN_SVG_JUDGE_MODEL is not. Set it to the model the endpoint serves.

Two things the run turned up, both now in the README:

• A reasoning model must have thinking turned off. The caption call caps at 120 tokens and the checklist at 400; with thinking on, the model spends the budget reasoning and returns an empty caption, scoring the semantic component zero on every sample. I hit exactly that before adding --default-chat-template-kwargs '{"enable_thinking": false}'.
• huggingface_hub sends the checklist schema to a custom endpoint in TGI's format, {"type": "json_object", "value": <schema>}. TGI enforces it; vLLM reads it as plain JSON mode. In the run above that made no difference (22/22 replies had every key), and a missing key already counts as false, so a drifting reply scores lower rather than failing.

Out of scope

• The default judge model stays Qwen/Qwen2.5-VL-72B-Instruct for the hosted path. Choosing a smaller default is a quality call for the environment's owner.
• No change to how the weights behave when a judge is configured but failing. That deliberate choice, and the judged flag, are untouched.
• In core, _create_session (src/openenv/core/env_server/http_server.py:410) replaces any exception from an environment's constructor with EnvironmentFactoryError(factory_name), whose message drops the cause and which is not logged. Every environment's configuration errors reach the client as the same generic FACTORY_ERROR with nothing in the server log. The startup check here works around it for this environment; including the cause is a small core fix I'm happy to send separately.
• Noticed while running these tests with resvg-py installed: tests/envs/test_pelican_svg_training_contract.py:150 puts examples/ on sys.path, which makes the repository's examples/thinkingbox directory satisfy pytest.importorskip("thinkingbox") in tests/envs/test_thinkingbox_env.py:28, and collection then fails on thinkingbox.common. It reproduces on main and stays hidden in CI only because resvg-py is not installed there. Narrowing that importorskip to thinkingbox.common.chat_types fixes it; happy to send it separately.

Claude Code Review

N/A

[k21993]

Reviewed exact head fb9087232985d0e230785618fa09d465666a5b16.

I ran the Pelican SVG environment and training contract tests in an isolated environment with resvg-py==0.3.3: 166 passed. Ruff, Ruff format, usort, docs sync, and git diff --check also passed for the changed files. The repository-wide hooks still report existing issues elsewhere in the tree, but none come from this PR.

I also inspected the request produced by huggingface_hub 1.3.3 for the custom URL path. It sends the request to /v1/chat/completions, keeps the requested model in the JSON body, uses Bearer EMPTY when no endpoint key is supplied, and preserves the JSON object response format. I did not run vLLM locally, so the end-to-end vLLM result remains author-reported.

I found no blocking issue or conflict with an OpenEnv principle, invariant, or active RFC.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[burtenshaw]

Reviewed 316830c after refreshing onto main at 060bd10. The refresh preserves session concurrency and both sets of tests. 168 Pelican environment/training-contract tests pass; actual huggingface_hub request construction preserves the endpoint URL, body model, and Bearer EMPTY credential boundary. Touched lint, formatting, import order, and docs sync pass. No remaining correctness or alignment blocker found.

[burtenshaw]

Refreshed onto main 07f26b0 after #1198. The Pelican app retains its custom-judge startup validation and now preserves PelicanSvgState for /schema and /state. Focused Pelican/training-contract and State-wiring validation passes 197 tests; touched-file lint, format, and import sorting pass. Current head: 2f60bcc. Fresh CI is running.

[burtenshaw]

Reviewed the #1198 merge resolution: PelicanSvgState wiring and judge-configuration startup validation are both preserved. Independent validation passed 197 tests, including startup rejection of an endpoint without a model. HTTP /schema and /state returned 200 and retained all four Pelican State fields; touched-file lint passed.

[burtenshaw]

Refreshed this branch with main 9b2c0bea348d0d0844e8ebef485a9f6ec3bce8b0 and validated head 9a4230268497f31dcde5f9801204c5b6ae1145cc. 149 scoped pelican SVG tests pass; touched-file lint/format checks also passed before the refresh. No new alignment issue found in the focused patch. Refreshed CI is pending; the coordinating maintainer will make the merge decision after checking this exact head.

[burtenshaw]

Independent review: 168 Pelican/training tests pass with resvg-py; changed-file lint, format, and docs sync pass. Endpoint credentials and startup validation preserved. No new blocker found; required CI remains queued.