Let the pelican_svg judge use a self-hosted endpoint

docs/source/environments/pelican_svg.md

@@ -264,9 +264,43 @@ PYTHONPATH=src:envs uv run uvicorn pelican_svg_env.server.app:app --port 8000
 
 Environment variables: `HF_TOKEN` for the judge, `PELICAN_SVG_JUDGE_MODEL` to
 change it from `Qwen/Qwen2.5-VL-72B-Instruct`, `PELICAN_SVG_DISABLE_JUDGE=1` to
-force offline scoring, `MAX_CONCURRENT_ENVS` for the maximum number of
+force offline scoring, and `PELICAN_SVG_JUDGE_BASE_URL` (with
+`PELICAN_SVG_JUDGE_API_KEY` where the endpoint wants one) to judge with a
+self-hosted model instead. `MAX_CONCURRENT_ENVS` sets the maximum number of
 concurrent WebSocket sessions (default `8`).
 
+### Judging with a local model
+
+The judge speaks the OpenAI-compatible chat API, so any server that does will
+do. Serving a vision model on one GPU keeps the judged component in the reward
+without paying per rollout:
+
+```bash
+vllm serve Qwen/Qwen2.5-VL-7B-Instruct --port 8000 --limit-mm-per-prompt '{"image":1}'
+
+export PELICAN_SVG_JUDGE_BASE_URL=http://localhost:8000/v1
+export PELICAN_SVG_JUDGE_MODEL=Qwen/Qwen2.5-VL-7B-Instruct
+```
+
+`PELICAN_SVG_JUDGE_MODEL` is required here, since the hosted default names a
+72B repository the local server is unlikely to be serving, and a judge whose
+every call fails is worse than no judge at all; the server refuses to start
+without it. No Hugging Face token is needed, and an ambient one is never sent
+to the endpoint.
+
+Turn thinking off if the model has it. The caption call is capped at 120
+tokens and the checklist at 400, and a reasoning model spends that budget
+thinking and returns an empty caption, which scores the semantic component
+zero on every sample. With vLLM that is
+`--default-chat-template-kwargs '{"enable_thinking": false}'`. Instruct-only
+vision models need nothing.
+
+`huggingface_hub` sends the checklist schema to a custom endpoint in TGI's
+format, `{"type": "json_object", "value": <schema>}`. TGI enforces the schema;
+vLLM reads it as plain JSON mode, so the reply is asked to be JSON but not held
+to the keys. The prompt lists them and a missing key counts as `false`, so a
+drifting reply scores lower rather than failing.
+
 With no judge configured, structure carries the full weight. A judge that is
 configured but *fails* is treated differently: the semantic component stays at
 zero and `judged` is set false on the observation, so a harness can drop or

envs/pelican_svg_env/README.md

@@ -275,9 +275,43 @@ PYTHONPATH=src:envs uv run uvicorn pelican_svg_env.server.app:app --port 8000
 
 Environment variables: `HF_TOKEN` for the judge, `PELICAN_SVG_JUDGE_MODEL` to
 change it from `Qwen/Qwen2.5-VL-72B-Instruct`, `PELICAN_SVG_DISABLE_JUDGE=1` to
-force offline scoring, `MAX_CONCURRENT_ENVS` for the maximum number of
+force offline scoring, and `PELICAN_SVG_JUDGE_BASE_URL` (with
+`PELICAN_SVG_JUDGE_API_KEY` where the endpoint wants one) to judge with a
+self-hosted model instead. `MAX_CONCURRENT_ENVS` sets the maximum number of
 concurrent WebSocket sessions (default `8`).
 
+### Judging with a local model
+
+The judge speaks the OpenAI-compatible chat API, so any server that does will
+do. Serving a vision model on one GPU keeps the judged component in the reward
+without paying per rollout:
+
+```bash
+vllm serve Qwen/Qwen2.5-VL-7B-Instruct --port 8000 --limit-mm-per-prompt '{"image":1}'
+
+export PELICAN_SVG_JUDGE_BASE_URL=http://localhost:8000/v1
+export PELICAN_SVG_JUDGE_MODEL=Qwen/Qwen2.5-VL-7B-Instruct
+```
+
+`PELICAN_SVG_JUDGE_MODEL` is required here, since the hosted default names a
+72B repository the local server is unlikely to be serving, and a judge whose
+every call fails is worse than no judge at all; the server refuses to start
+without it. No Hugging Face token is needed, and an ambient one is never sent
+to the endpoint.
+
+Turn thinking off if the model has it. The caption call is capped at 120
+tokens and the checklist at 400, and a reasoning model spends that budget
+thinking and returns an empty caption, which scores the semantic component
+zero on every sample. With vLLM that is
+`--default-chat-template-kwargs '{"enable_thinking": false}'`. Instruct-only
+vision models need nothing.
+
+`huggingface_hub` sends the checklist schema to a custom endpoint in TGI's
+format, `{"type": "json_object", "value": <schema>}`. TGI enforces the schema;
+vLLM reads it as plain JSON mode, so the reply is asked to be JSON but not held
+to the keys. The prompt lists them and a missing key counts as `false`, so a
+drifting reply scores lower rather than failing.
+
 With no judge configured, structure carries the full weight. A judge that is
 configured but *fails* is treated differently: the semantic component stays at
 zero and `judged` is set false on the observation, so a harness can drop or

envs/pelican_svg_env/server/app.py

@@ -7,7 +7,13 @@
 from openenv.core.env_server import create_app
 
 from ..models import PelicanSvgAction, PelicanSvgObservation, PelicanSvgState
-from .pelican_svg_environment import PelicanSvgEnvironment
+from .pelican_svg_environment import _judge_from_env, PelicanSvgEnvironment
+
+# Check the judge configuration once at startup. Environments are built per
+# session, and an error raised there reaches the client only as a generic
+# factory error, so a misconfigured judge would otherwise fail every session
+# without saying why.
+_judge_from_env()
 
 # The class is passed rather than an instance so each WebSocket session gets
 # its own environment and its own sampled task.

envs/pelican_svg_env/server/pelican_svg_environment.py

@@ -35,18 +35,45 @@ def _judge_from_env() -> VisionJudge | None:
     training loops want. It does mean no judge, though. Configuring one whose
     every call fails would keep the 0.35/0.65 weight split while `semantic`
     never scores, silently capping the reward at 0.35.
+
+    `PELICAN_SVG_JUDGE_BASE_URL` points the judge at an OpenAI-compatible
+    server instead, a local vLLM for example. It needs no Hugging Face token,
+    but it does need `PELICAN_SVG_JUDGE_MODEL`, since the default repository id
+    is a 72B model that such a server is unlikely to be serving. Raising beats
+    guessing here for the same reason the docstring above gives: a judge whose
+    every call fails is worse than no judge at all.
+
+    Raises:
+        ValueError: If `PELICAN_SVG_JUDGE_BASE_URL` is set without
+            `PELICAN_SVG_JUDGE_MODEL`.
     """
     if os.environ.get("PELICAN_SVG_DISABLE_JUDGE", "").lower() in {"1", "true", "yes"}:
         return None
+    model = os.environ.get("PELICAN_SVG_JUDGE_MODEL")
+    base_url = os.environ.get("PELICAN_SVG_JUDGE_BASE_URL")
+    if base_url:
+        if not model:
+            raise ValueError(
+                "PELICAN_SVG_JUDGE_BASE_URL is set but PELICAN_SVG_JUDGE_MODEL "
+                "is not. Set it to the model the endpoint serves."
+            )
+        return VisionJudge(
+            HFVisionClient(
+                model=model,
+                api_key=os.environ.get("PELICAN_SVG_JUDGE_API_KEY"),
+                base_url=base_url,
+            )
+        )
     token = (
         os.environ.get("HF_TOKEN")
         or os.environ.get("HUGGING_FACE_HUB_TOKEN")
         or get_token()
     )
     if not token:
         return None
-    model = os.environ.get("PELICAN_SVG_JUDGE_MODEL", DEFAULT_JUDGE_MODEL)
-    return VisionJudge(HFVisionClient(model=model, api_key=token))
+    return VisionJudge(
+        HFVisionClient(model=model or DEFAULT_JUDGE_MODEL, api_key=token)
+    )
 
 
 class PelicanSvgEnvironment(

envs/pelican_svg_env/server/vision_judge.py

@@ -28,6 +28,11 @@
 # Weight split between the blind caption and the feature checklist.
 BLIND_WEIGHT = 0.5
 
+# Sent as the bearer token to a `base_url` endpoint when none is configured.
+# Self-hosted servers rarely check it, and the ambient Hugging Face token must
+# not be forwarded to an endpoint the user pointed us at.
+_PLACEHOLDER_API_KEY = "EMPTY"
+
 _CAPTION_PROMPT = (
     "Describe this image in one short sentence. Name the things you can "
     "actually identify. If you cannot tell what it depicts, say so plainly. "
@@ -54,22 +59,36 @@ async def complete_with_image(
 
 
 class HFVisionClient:
-    """Vision client backed by Hugging Face Inference Providers.
+    """Vision client backed by Hugging Face Inference Providers, or any
+    OpenAI-compatible endpoint given a `base_url`.
 
     Args:
         model (`str`, *optional*, defaults to `"Qwen/Qwen2.5-VL-72B-Instruct"`):
-            Repository id of the judge model.
+            Repository id of the judge model, or the model name the endpoint at
+            `base_url` serves.
         api_key (`str`, *optional*):
             Token to authenticate with. Falls back to the ambient Hugging Face
-            token when omitted.
+            token, except with a `base_url`: the endpoint is someone else's
+            server, so the Hugging Face token is never sent to it and a
+            placeholder is used when no key is given.
         timeout (`float`, *optional*, defaults to `120.0`):
             Per-request timeout in seconds.
+        base_url (`str`, *optional*):
+            Base URL of an OpenAI-compatible server to call instead of
+            Inference Providers, for example a local vLLM at
+            `http://localhost:8000/v1`. The model name still travels in the
+            request body, so it must match what that server serves.
 
     Examples:
 
     ```python
     client = HFVisionClient(model="Qwen/Qwen3-VL-30B-A3B-Instruct")
     reply = await client.complete_with_image("What is this?", png)
+
+    local = HFVisionClient(
+        model="Qwen/Qwen2.5-VL-7B-Instruct",
+        base_url="http://localhost:8000/v1",
+    )
     ```
     """
 
@@ -78,11 +97,16 @@ def __init__(
         model: str = DEFAULT_JUDGE_MODEL,
         api_key: str | None = None,
         timeout: float = 120.0,
+        base_url: str | None = None,
     ):
         from huggingface_hub import get_token
 
         self.model = model
-        self._api_key = api_key or get_token()
+        self.base_url = base_url
+        if base_url:
+            self._api_key = api_key or _PLACEHOLDER_API_KEY
+        else:
+            self._api_key = api_key or get_token()
         self._timeout = timeout
 
     async def complete_with_image(
@@ -108,9 +132,16 @@ async def complete_with_image(
         # synchronous `step()` path runs each call under its own short-lived
         # loop, so a cached client fails with "Event loop is closed" on every
         # request after the first.
-        async with AsyncInferenceClient(
-            api_key=self._api_key, timeout=self._timeout
-        ) as client:
+        client_kwargs: dict[str, Any] = {
+            "api_key": self._api_key,
+            "timeout": self._timeout,
+        }
+        if self.base_url:
+            # `base_url` is the client's alias for `model`, so the two cannot
+            # both be set here. `chat_completion(model=...)` below still names
+            # the model in the request body, which is what the endpoint reads.
+            client_kwargs["base_url"] = self.base_url
+        async with AsyncInferenceClient(**client_kwargs) as client:
             response = await client.chat_completion(
                 model=self.model,
                 messages=[