Repository navigation
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7e94e3dfc2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
| else: | ||
| clients = InferenceClient(config, model_name=config.name) | ||
| clients = InferenceClient(config, model_name=config.name) |
There was a problem hiding this comment.
When an SFT sampling source uses a different tokenizer from the trainable policy—which FrozenModelConfig currently permits—this client returns the frozen server's token IDs, and those IDs are forwarded unchanged into policy TrainingSamples. The IDs can represent different text or exceed the policy vocabulary, causing corrupted CE targets or embedding-index failures. Preserve policy-side rendering/retokenization for frozen SFT responses, or explicitly require and validate tokenizer identity.
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 49f001ef66
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Match the policy chat template. The frozen generation server also owns its renderer. | ||
| [inference.renderer] | ||
| name = "prime-qwen3" |
There was a problem hiding this comment.
This block configures only the managed policy server, while the frozen teacher at localhost:8001 is started separately by tests/integration/test_reverse_text_rl_sft.py:56-66 without --renderer.name prime-qwen3. Because the teacher model’s -RL suffix does not resolve to the intended renderer, the new server-owned training-record path renders teacher prompts with the auto/default renderer rather than the policy-compatible renderer, so this SFT integration can produce mismatched CE targets. Fresh evidence beyond the existing tokenizer comment is that the equivalent debug launch now explicitly passes --renderer.name prime-qwen3, but the fixture launch was not updated.
Useful? React with 👍 / 👎.
49f001e to
b5618d1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5618d13d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async def capture(): | ||
| async for result in result_generator: | ||
| for output in result.outputs: | ||
| routing = serialize_routed_experts(output.routed_experts, request.routed_experts_prompt_start) | ||
| if routing is not None: |
There was a problem hiding this comment.
When routed-expert or sampling-mask capture is enabled and the run also performs evaluation, this wrapper runs for ordinary eval Chat Completions too: it serializes every server-wide routed-expert tensor, collects every sampling mask, and later attaches both to the response even though eval did not request training metadata. In the checked router-replay configurations with eval, such as examples/advanced/glm-4.5-air/swe.toml, long rollouts can therefore add tens of megabytes of base64 routing data—and potentially larger sampling masks—to every eval response, causing severe HTTP and memory overhead. Gate capture and attachment on return_training_metadata while continuing to clear native replay fields for ordinary chat.
Useful? React with 👍 / 👎.
b5618d1 to
8261e85
Compare
8261e85 to
7a978cb
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
7a978cb to
00c6de9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00c6de96df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| renderer: RendererConfig = AutoRendererConfig() | ||
| """Server-side prompt rendering and message/image attribution for training records.""" |
There was a problem hiding this comment.
When vllm.model is not present in MODEL_RENDERER_MAP and a training environment sends tools, this new default silently resolves to DefaultRenderer; _render_training_prompt then passes those tools to renderer.render, which rejects them unless a tool parser was explicitly configured. The removed orchestrator validator previously surfaced this during config parsing, so moving equivalent validation to InferenceConfig is necessary to prevent every rollout for an unmapped tool-using model from failing only at request time.
Useful? React with 👍 / 👎.
| tokenizer: TokenizerConfig = TokenizerConfig() | ||
|
|
||
| renderer: RendererConfig = AutoRendererConfig() | ||
| """Typed renderer config (``renderers.RendererConfig`` discriminated union), required — | ||
| training is renderer-only. Defaults to ``"auto"``, which resolves from | ||
| ``tokenizer.name_or_path`` via ``MODEL_RENDERER_MAP``. RL/OPD roll out through the renderer | ||
| client; SFT uses it to backfill tokens for its chat-completions teacher.""" | ||
|
|
||
| eval: RLOnlineEvalConfig | None = None |
There was a problem hiding this comment.
Removing the orchestrator renderer field leaves docs/training.md:190 instructing RL users to configure custom renderers under [orchestrator.renderer]; that section is no longer accepted or used, so following the current documentation cannot select the renderer. Update the remaining reference to [inference.renderer] alongside this configuration move.
AGENTS.md reference: AGENTS.md:L40-L42
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f52987511
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.training_tokenizer is None: | ||
| self.training_tokenizer = load_tokenizer(self.model_config.tokenizer) | ||
| self.training_renderer_config = _resolve_renderer_config( | ||
| self.training_tokenizer, self.training_renderer_config |
There was a problem hiding this comment.
When a run sets the shared tokenizer.chat_template or trainer.tokenizer.chat_template, config propagation places it in inference.vllm.chat_template, and the ordinary vLLM Chat path uses that override. This training path instead reloads the tokenizer from its model identifier and never applies the configured template, so training requests are rendered with the checkpoint's original template while evaluation and the trainer use the override; the resulting prompt tokens and attribution can silently train against a different conversation format. Apply the server's configured chat template, including resolving template file paths, before creating the renderer or reuse the already configured server tokenizer.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7449410e8c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if isinstance(self.config.source, FrozenModelConfig): | ||
| self.clients = await connect_frozen_client(self.config.source, renderer_config=self.renderer_config) | ||
| self.connected = self.clients | ||
| clients = await connect_frozen_client(self.config.source) |
There was a problem hiding this comment.
This vocabulary check runs only for frozen sources, but policy rollouts now also consume token IDs rendered by the inference server directly. The configuration intentionally permits trainer/orchestrator tokenizers to differ from the inference tokenizer (for example via inference.vllm.tokenizer or separate component model configuration), so an incompatible live endpoint can silently send IDs with different meanings and corrupt every RL sample. Perform the same /v1/tokenizer compatibility check for the policy inference engines before dispatching live rollouts.
Useful? React with 👍 / 👎.
7449410 to
4f5b801
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f5b801c2f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def _render_training_prompt(self, request: ChatCompletionRequest): | ||
| # Hold the lock in the worker thread even if the HTTP request is cancelled. | ||
| with self.training_renderer_lock: |
There was a problem hiding this comment.
When an API worker handles concurrent training rollouts, this lock remains held across the entire render path—including renderer.render and multimodal feature construction—so all prompt tokenization and image preprocessing is serialized; one slow image request can block every other rollout assigned to that worker. Restrict the lock to renderer initialization/cache updates, or use request-local/thread-local renderers so the work submitted with asyncio.to_thread can actually run concurrently.
Useful? React with 👍 / 👎.
| "return_tokens_as_token_ids": True, | ||
| "return_training_metadata": True, | ||
| "include_reasoning": True, |
There was a problem hiding this comment.
These flags move training rollouts to server-rendered Chat Completions, but docs/overview.md:11 still says the orchestrator drives rollouts through the token-in /inference/v1/generate route and presents Chat Completions as external-client-only. That now directs operators and contributors to the wrong request path when debugging routing, rendering, or caching; update the architecture overview to match the new data flow.
AGENTS.md reference: AGENTS.md:L40-L42
Useful? React with 👍 / 👎.
Removes duplicate and unused entries from `pyproject.toml` (rev-infra P9). The set of installed packages does not change for any extra combination (checked below). | Removed | Reason | |---|---| | second `orjson>=3.10` in `dependencies` | Duplicate of `orjson>=3.11.0` a few lines above. | | `pandas>=2.0` | Nothing imports it: no `import pandas` / `pd.` in `src/`, `tools/`, `scripts/`, `tests/`, `examples/`, `skills/`, `packages/`. It was added for the nano bridge (7089ba1). It stays installed because `datasets` depends on it. | | `pyarrow>=21.0.0` | No direct use anywhere in those directories. The one parquet writer, `tools/convert_traces_to_hf_dataset.py`, calls `datasets.Dataset.to_parquet`, and `datasets` depends on `pyarrow` itself. It stays installed. | | `nixl-cu13==0.10.1` in the `disagg` extra | `disagg` already lists `nixl`, and the `[[tool.uv.dependency-metadata]]` override for `nixl` 0.10.1 requires exactly `nixl-cu13==0.10.1`. The `gpu` extra keeps its explicit pin, because `transports/weights/nixl/agent.py` imports `nixl_cu13` directly. | | `mamba-ssm` dependency group | Duplicate of the `gpu` extra entry, which was added with the native Nemotron-H model (#3426). Every user of the group also syncs `--extra gpu` or `--all-extras`, so I dropped `--group mamba-ssm` from `Dockerfile.cuda` (2 places), `scripts/docker-entrypoint.sh` and the `pre_run_command` in `configs/advanced/nemotron-3-super/swe.toml`. The `[tool.uv.sources]` git pin and the build dependencies for mamba-ssm stay. | Behavior change: - None for installs: same locked packages and versions. - `uv sync --group mamba-ssm` no longer works; use `--extra gpu`, which every repo caller already passes. - `pandas` and `pyarrow` lose their direct lower bounds. They are still pulled in by `datasets`. ## Checks - `uv lock`: resolves 564 packages, the same set as before (568 `[[package]]` entries before and after). `uv lock --check` passes. The `uv.lock` diff only touches prime-rl's own metadata. I reverted the reordering of `[options.exclude-newer-package]` that my uv version (0.12.10) introduced. - I ran `uv export --frozen --all-packages` and compared the package lists before and after, ignoring the `# via` comment lines. They are identical for: no extras, `--extra disagg`, `--extra gpu`, `--extra dashboard`, `--all-extras`, and `--all-extras --all-groups`. `--extra disagg` still yields `nixl==0.10.1` and `nixl-cu13==0.10.1`, and `--extra gpu` still yields `mamba-ssm` at the pinned rev. - No Python changes, so no unit tests are affected. GPU: not needed. The Docker image build in CI covers the Dockerfile and entrypoint edits. ## Conflicts Text conflicts only in `pyproject.toml` / `uv.lock`, as with any dependency PR. None of the open PRs that touch these files (e.g. #3829, #3803, #3767, #3660) edit the lines removed here. A re-lock after rebase is enough. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Metadata and sync-flag cleanup only; lockfile exports are unchanged and runtime installs should match prior behavior. > > **Overview** > Cleans up **duplicate and redundant dependency declarations** in `pyproject.toml` and aligns `uv.lock`, Docker, and Slurm sync commands with the slimmer manifest. **No intended change to installed packages** for existing extra combinations (`gpu`, `disagg`, `--all-extras`, etc.). > > **Removed from core `dependencies`:** duplicate `orjson>=3.10`, unused direct pins on `pandas` and `pyarrow` (they still come in via `datasets`). **Removed from `disagg` extra:** explicit `nixl-cu13==0.10.1` because `nixl` metadata already pulls that version; the **`gpu` extra keeps** its explicit `nixl-cu13` pin for direct imports. **Dropped the `mamba-ssm` dependency group** as a duplicate of the `gpu` extra entry—`Dockerfile.cuda`, `scripts/docker-entrypoint.sh`, and the Nemotron SWE `pre_run_command` no longer pass `--group mamba-ssm`. > > **Workflow note:** `uv sync --group mamba-ssm` alone is no longer supported; use `--extra gpu` (or `--all-extras`), which callers already use. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f0f2ff6. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
4f5b801 to
4dd6f52
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4dd6f523bc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.training_tokenizer is None: | ||
| self.training_tokenizer = load_tokenizer(self.model_config.tokenizer) |
There was a problem hiding this comment.
When inference.vllm uses tokenizer-specific settings such as trust_remote_code, tokenizer_revision, or tokenizer_mode, vLLM can load and serve the model successfully, but this independent reload receives only the tokenizer identifier and therefore does not honor those settings. A remote-code tokenizer can fail on the first training request, while a pinned revision or alternate mode can produce IDs different from those accepted by the engine; /v1/tokenizer repeats the same reload. Reuse vLLM's configured tokenizer or pass through the complete tokenizer-loading configuration.
Useful? React with 👍 / 👎.
| response = await client.get("/v1/tokenizer", timeout=source.wait_for_ready_timeout) | ||
| if response.status_code != 200: | ||
| raise ValueError( | ||
| f"Frozen generation requires GET /v1/tokenizer; {client.base_url} returned " | ||
| f"HTTP {response.status_code}. If the router does not forward this endpoint, " | ||
| "set admin_base_url to the inference engine URLs." |
There was a problem hiding this comment.
This makes /v1/tokenizer a mandatory startup endpoint, while docs/training.md:98-110 still advertises SFT teachers as “any OpenAI-compatible endpoint” and shows a generic inference launch without explaining the tokenizer route or training-record response extensions. Following that documented contract now always fails here for an ordinary OpenAI provider, before any rollout is dispatched. Update the frozen-model documentation and FrozenModelConfig contract to state the PrimeRL-specific tokenizer and training-metadata requirements.
AGENTS.md reference: AGENTS.md:L40-L42
Useful? React with 👍 / 👎.
fca0862 to
29798be
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29798be82c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ``tokenizer.name_or_path`` via ``MODEL_RENDERER_MAP``. RL/OPD roll out through the renderer | ||
| client; SFT uses it to backfill tokens for its chat-completions teacher.""" | ||
|
|
||
| eval: RLOnlineEvalConfig | None = None |
There was a problem hiding this comment.
Removing this field leaves examples/advanced/glm-4.5-air/ipo-ablation/base.toml:69 with [orchestrator.renderer], even though the documented launch command composes that base for both the IPO and IcePop experiments. That key is now unknown to OrchestratorConfig, so these advertised ablation launches fail configuration validation before starting; migrate this remaining block to [inference.renderer] as was done for the sibling swe.toml configuration.
Useful? React with 👍 / 👎.
fc3c7ac to
78ef881
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 78ef881891
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| max_model_len = 8192 | ||
| data_parallel_size = 4 | ||
| [orchestrator.renderer] | ||
| [inference.renderer] |
There was a problem hiding this comment.
For the Hendrycks runs using Qwen/Qwen3-1.7B, migrating this block leaves reasoning_parser = "think" under inference.renderer, but PrimeRlServingChat now parses completions exclusively with self.parser_cls, which is configured by inference.vllm.reasoning_parser; auto-resolution returns no parser for this model. Consequently <think> output remains in message content instead of being separated as reasoning, changing reward inputs and subsequent multi-turn messages. Move this setting (and the identical one in examples/basic/hendrycks-sanity/rl.toml) to [inference.vllm], or explicitly connect renderer parser settings to the native response parser.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d58bc3c804
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| flash-linear-attention = { git = "https://github.com/fla-org/flash-linear-attention" } | ||
| mamba-ssm = { git = "https://github.com/state-spaces/mamba", rev = "f577286" } | ||
| flash_attn_3 = { index = "pytorch-cu130-test" } | ||
| prime-rl-configs = { workspace = true, editable = true } |
There was a problem hiding this comment.
editable from the workspace source
Every uv sync or uv run that parses this project fails with cannot specify both workspace and editable, preventing environment setup and all documented entrypoints from running. The uv workspace documentation specifies this source as { workspace = true } and notes that workspace dependencies are already editable, so remove the redundant editable key.
AGENTS.md reference: AGENTS.md:L32-L35
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24417aa4a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if len(rendered.token_ids) >= self.model_config.max_model_len: | ||
| raise ValueError( | ||
| f"Prompt length ({len(rendered.token_ids)}) exceeds maximum context length " | ||
| f"({self.model_config.max_model_len})" | ||
| ) |
There was a problem hiding this comment.
When a Chat Completions client sets truncate_prompt_tokens, this custom rendering path never applies it: prompts below the model limit retain more context than requested, while prompts at or above the limit fail here instead of being truncated. This affects both rollout and ordinary eval requests because every chat request now uses _render_prompt; vLLM's native path explicitly builds tokenization parameters from the request before rendering (vLLM 0.31 source). Apply the requested truncation while keeping message attribution aligned, or explicitly reject the unsupported option rather than silently changing its semantics.
Useful? React with 👍 / 👎.
24417aa to
5d1f3f0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d1f3f0405
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| enabled = true | ||
|
|
||
| [orchestrator.renderer] | ||
| [inference.renderer] |
There was a problem hiding this comment.
When the documented Dynamo example is launched, this section constructs a non-null RLConfig.inference, so rl_local starts a managed vLLM process and assigns the configured inference GPU to it. That contradicts examples/extra/dynamo/README.md, where the Dynamo worker is already running on GPU 1 and the trainer must retain GPU 0; the extra server will contend for the Dynamo GPU and port instead of using the external endpoint. Remove this managed [inference] section from the example and configure rendering on the external service through its supported mechanism.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5d1f3f0. Configure here.
| raise ValueError( | ||
| f"Frozen generation requires GET /v1/tokenizer; {client.base_url} returned " | ||
| f"HTTP {response.status_code}. If the router does not forward this endpoint, " | ||
| "set admin_base_url to the inference engine URLs." |
There was a problem hiding this comment.
Frozen vocab check hits the router
High Severity
Frozen generation now requires GET /v1/tokenizer, but the check uses setup_admin_clients(source), which falls back to base_url when admin_base_url is unset. Policy clients already get admin_base_url pointed at the engine because the default router does not serve custom routes. SFT and OPD teacher configs only set base_url on the client-facing port, so setup fails before distillation starts.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 5d1f3f0. Configure here.
Training and evaluation use one native Chat Completions client. The endpoint's
[inference.renderer]renders every prompt; vLLM handles generation, tools, reasoning, and streaming for both. Requesting training metadata does not select a different rendering or response-parsing path.return_training_metadata=truerequests completed training records and implies the supporting token-ID and reasoning flags. Logprobs remain independently selectable for frozen-model CE generation. The chat handler reuses the token endpoint's packed float32 logprobs, CSR sampling masks, and compact expert-routing capture. Native top-k and prompt-logprob requests retain their normal representation.Multimodal feature construction returns native vLLM objects directly. The server no longer serializes image tensors only to decode them immediately. Images stay inline in messages; the trainer receives image references and expanded token ranges and builds its own features.
InferenceClientowns its lazy prefill-scoring SDK client directly.Verifiers merges exact token prefixes and forks when a re-rendered history differs. ECHO retains observation weights across those branches while excluding sampled replies and repeated observations. Frozen generation checks every endpoint's token-to-ID vocabulary against the policy tokenizer. The server exposes
/v1/tokenizer; unavailable or mismatched vocabularies fail setup.Training metadata requires non-streaming requests; the shared Verifiers client obtains the completed record and can deliver SSE to the harness afterward. Endpoint rendering requires complete messages and rejects per-request template overrides, final-message continuation, and prompt truncation. Ordinary eval requests retain native vLLM streaming. The GLM IPO/IcePop ablation base config places its renderer under
[inference.renderer].Dependencies: Verifiers #2676, on #2674 → #2675, is pinned at
37bac61; Renderers #193 is pinned ata3c71b3. The package metadata pins these public revisions for standalone installs. The slim config package is a workspace member, so source checkouts keep using the editable submodules. Replace the Git pins with released versions when publishing to PyPI.Both Hendrycks configurations explicitly select the native vLLM qwen3 reasoning parser. The current main (d61df56) rollout-accounting, discarded-object release, batch-encoding, and CPU Qwen weight-reload changes are retained.
Validation: locked workspace sync and the standalone config-wheel build passed; wheel metadata pins the published Verifiers revision. The 398 focused client/config/parser/algorithm/accounting/transport/dashboard tests passed, including upstream discarded-rollout and byte-preserving batch-encoding tests. The configuration-file matrix was excluded from this focused local run. Ruff and the pinned pre-commit hooks passed. A CPU probe verifies prompt truncation is rejected explicitly while ordinary requests continue. No live GPU engine was run locally.
Note
High Risk
This is a core data-plane change: token identity, multi-turn merging, frozen-model distillation, and inference serving all move together; misconfiguration or vocab mismatch now fails at setup rather than silently drifting logprobs.
Overview
Moves prompt rendering from the orchestrator to the inference server by replacing
[orchestrator.renderer]with[inference.renderer]everywhere and wiring that config into vLLM startup (prime_renderer).Training and eval now share one Chat Completions path. Rollouts request
return_training_metadata=truefor exact prompt/completion token IDs, message attribution, optional logprobs, sampling masks, and routed experts. A newPrimeRlServingChatrenders prompts on the server (viarenderers) and attaches training metadata to responses;GET /v1/tokenizersupports frozen-teacher vocab checks at env setup.The orchestrator drops the renderer client, separate train/eval clients, and orchestrator-side auto-renderer validation. Verifiers merges multi-turn traces only on exact token-prefix matches; ECHO tracks message paths so observation CE survives re-rendered branches. Docs and distillation debug comments are updated accordingly.
Pins
verifiersandrenderersto specific git revisions;prime-rl-configsjoins the uv workspace. Hendrycks configs setreasoning_parser = "qwen3"under[inference.vllm]instead of on the old renderer block.Reviewed by Cursor Bugbot for commit 5d1f3f0. Bugbot is set up for automated code reviews on this repo. Configure here.