Skip to content

fix(runs): reach the platform's internal RFT router for hosted runs - #895

Merged
kcoopermiller merged 2 commits into
mainfrom
fix/prime-runs-internal-api-root
Sep 4, 2026
Merged

kcoopermiller merged 2 commits into
mainfrom
fix/prime-runs-internal-api-root

Conversation

@kcoopermiller

@kcoopermiller kcoopermiller commented Sep 4, 2026 •

Copy link
Copy Markdown
Member

Why

Since Sep 1, hosted managed RFT runs start their orchestrator with --monitors.prime.name, RUN_ID, a per-run token in PRIME_API_KEY and PRIME_API_BASE=<platform>/api/internal/rft. That internal router authenticates the run's token from the x-api-key header and serves the monitoring routes only (/rft/metrics, /rft/samples/presign, /rft/samples/confirm, /rft/finalize). The old prime-rl TrainRun used that root verbatim and sent both Authorization and x-api-key.

PlatformClient addressed everything under <base>/api/v1 with a Bearer header alone, so once prime-rl's monitor moves to this SDK (PrimeIntellect-ai/prime-rl#3459) an attached hosted run would 404 every upload and go silently unmonitored — the failure #4885 had just fixed. Against prod, unauthenticated, for the record:

POST https://api.primeintellect.ai/api/internal/rft/metrics          → 422 "header -> x-api-key: Field required"
POST https://api.primeintellect.ai/api/internal/api/v1/rft/metrics   → 404   (what 0.1.0 builds from that base URL)

What

  • PlatformClient recognises the internal root (…/api/internal, with or without the trailing /rft): requests go under <base>/api/internal and the token is repeated in x-api-key. The public root is unchanged; a …/api/v1/rft root (the RFT client's historical default) is normalised the same way. client.internal says which router is in use.
  • Auth headers are also sent per request, not only as the pool's defaults, so a caller-supplied httpx.Client authenticates like one the SDK opened.
  • Docs: init() and RftBackend.attach docstrings, README (training bullet + configuration). Registering a run or setting its status is not available on the internal root — attach only — which is exactly the hosted case.

No change for external runs or evals: same URLs, same headers.

Tests

tests/test_http.py: internal roots → /api/internal prefix + x-api-key (parametrised over the three spellings), public roots → /api/v1 and no x-api-key (four spellings), injected client gets the headers. tests/test_rft_backend.py: attach + finalize through the internal router with the run's token. 254 passed; ruff clean.

Release

Ships as prime-runs 0.1.1 (release PR follows, stacked on this one). prime-rl#3459 then bumps to prime-runs[train]>=0.1.1; its _base_url() already passes $PRIME_API_BASE minus /rft, which this accepts unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g


Note

Medium Risk
Changes core HTTP routing and auth for all platform calls when base_url includes an internal path; mistakes could break hosted monitoring or mis-route external runs, though public URLs are explicitly preserved and heavily tested.

Overview
PlatformClient now supports the internal RFT base URL that hosted training launchers inject (…/api/internal or …/api/internal/rft). When that root is detected via new split_api_root(), requests target /api/internal (not a doubled /api/v1 path) and the run token is sent in x-api-key in addition to Bearer auth; client.internal flags this mode. Public origins and historical …/api/v1/rft URLs keep the previous behavior.

Each HTTP call merges auth headers on the request so a caller-injected httpx.Client still authenticates like the SDK-owned pool.

README and init() / RftBackend.attach docs explain attaching hosted runs with $PRIME_API_BASE and that register/status APIs stay on the public API only. Tests cover internal vs public URL spellings, hostnames that look like path suffixes, per-request headers, and finalize through the internal router.

Reviewed by Cursor Bugbot for commit b1a26e6. Bugbot is set up for automated code reviews on this repo. Configure here.

Hosted managed RFT runs start their orchestrator with
`PRIME_API_BASE=<platform>/api/internal/rft` and a per-run token; that
router authenticates the token from the `x-api-key` header and serves the
monitoring routes (metrics, presign, confirm, finalize) only. `PlatformClient`
addressed everything under `<base>/api/v1` with a Bearer header alone, so an
attached hosted run would 404 every upload and go silently unmonitored.

`PlatformClient` now recognises the internal root (with or without the
trailing `/rft`): requests go under `<base>/api/internal` and the token is
repeated in `x-api-key`. The public root is unchanged, and a `/api/v1/rft`
root is accepted the same way. Headers are also sent per request, so a
caller-supplied `httpx.Client` authenticates like one the SDK opened.
Docs: `init()`, `RftBackend.attach`, README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-04T04:21:17.829304Z c9e1a2e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9e1a2eaf3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

`split_api_root` stripped `/rft` from the whole string, so a platform host
that is itself named `rft` (`http://rft`, a plain service name) collapsed to
`http:/` and every request went to a host called `api`; the same slip
existed for `/api/v1` on a host named `api` before this branch. Suffixes are
now matched on the parsed URL's path only, longest first, and `/rft` is
recognised only after an API path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
@kcoopermiller
kcoopermiller merged commit 22e3c57 into main Sep 4, 2026
34 of 36 checks passed
@kcoopermiller
kcoopermiller deleted the fix/prime-runs-internal-api-root branch September 4, 2026 16:52
kcoopermiller added a commit that referenced this pull request Sep 4, 2026
* fix(runs): reach the platform's internal RFT router for hosted runs

Hosted managed RFT runs start their orchestrator with
`PRIME_API_BASE=<platform>/api/internal/rft` and a per-run token; that
router authenticates the token from the `x-api-key` header and serves the
monitoring routes (metrics, presign, confirm, finalize) only. `PlatformClient`
addressed everything under `<base>/api/v1` with a Bearer header alone, so an
attached hosted run would 404 every upload and go silently unmonitored.

`PlatformClient` now recognises the internal root (with or without the
trailing `/rft`): requests go under `<base>/api/internal` and the token is
repeated in `x-api-key`. The public root is unchanged, and a `/api/v1/rft`
root is accepted the same way. Headers are also sent per request, so a
caller-supplied `httpx.Client` authenticates like one the SDK opened.
Docs: `init()`, `RftBackend.attach`, README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g

* release: prime-runs 0.1.1

Bumps prime-runs to 0.1.1 so release-runs.yml publishes the one change since
0.1.0: #895 (PlatformClient reaches the platform's internal RFT router for
hosted runs, with the run's token in x-api-key).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g

* fix(runs): normalise only the URL path when recognising an API root

`split_api_root` stripped `/rft` from the whole string, so a platform host
that is itself named `rft` (`http://rft`, a plain service name) collapsed to
`http:/` and every request went to a host called `api`; the same slip
existed for `/api/v1` on a host named `api` before this branch. Suffixes are
now matched on the parsed URL's path only, longest first, and `/rft` is
recognised only after an API path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
kcoopermiller added a commit to PrimeIntellect-ai/prime-rl that referenced this pull request Sep 9, 2026
Replaces `monitors/prime.py`'s`TrainRun` with `pr.init(kind="train",
...)`.

## What moved where

| was in prime-rl | now |
| --- | --- |
| `TrainRun.create` / `$RUN_ID` attach | `pr.init(kind="train", model=,
environments=, training=pr.TrainingSpec(...), config=, ...)` /
`pr.init(kind="train", id=$RUN_ID)` |
| `TrainRun.log_metrics` (fire-and-forget task per POST) |
`run.log_metrics(values, step=)` — a queue put; the SDK's uploader owns
retries, `Retry-After`, backpressure |
| `episodes_to_parquet_bytes` + `SAMPLE_SCHEMA` + presign→PUT→confirm |
`prime_runs.projection` + the SDK's training samples sink (same schema,
same flow) |
| the `step % 10` cadence in `log_episodes` | the sink
(`step_interval=10`), keyed on each episode's `run.work.step` — the
`TrainRunInfo` the dispatcher stamps |
| `finalize` → `POST /finalize`, fallback `set_status` | `run.finish()`
— drain, then an idempotent finalize; 409 on the failure path is treated
as already-closed |
| atexit `_mark_failed` | the SDK's atexit hook reports **crashed**
(with a message); an attached run's failure marking stays with the
launcher, as before |

The `kind == "train" and subset == "effective"` cohort filter stays
here. The monitor keeps `sanitize()` so dropped non-finite metric paths
are still named in the log (the SDK would drop them silently).
`$PRIME_API_BASE` still works — it historically points at
`…/api/v1/rft`, so the monitor strips what the SDK re-adds.

## Decisions

- **Step keying.** The samples sink keys each episode by the step it was
*dispatched* at (`run.work.step`), not the batch step; the platform
keeps one Parquet object per upload and the viewer unions every object
under a step, so an off-policy episode landing in a later batch adds an
object rather than replacing one, and each object numbers its
`sample_id`s in its own range (SDK side, `aba26790`). "Step N" in the
sample viewer therefore means dispatched at N, while the metrics at N
describe the batch trained at N.
- **Run identity.** The orchestrator keeps the launcher's `PRL_RUN_ID`
(or a local uuid) as the run identity for episodes, W&B and sandbox
labels, exactly as on `main`. The SDK keys every uploaded record to the
platform run (`run.id` / `run.type` on the uploaded copy, `work` kept),
so Prime Traces documents carry the id the dashboard shows without the
orchestrator learning it.

## Dependency

`prime-runs[train]>=0.1.1` from PyPI
([0.1.1](https://pypi.org/project/prime-runs/0.1.1/),
PrimeIntellect-ai/prime#896); the lock carries `prime-runs 0.1.1` +
`prime-traces 0.0.3` from the registry. The monitor's own unit tests run
in CI; there is no GPU-free end-to-end path for the training seam, so
the RFT contract was verified by reading the platform's `/api/v1/rft/*`
and `/api/internal/rft/*` routes against the SDK's calls.

## Hosted managed runs

Since PrimeIntellect-ai/platform#4885, hosted managed runs start the
orchestrator with `--monitors.prime.name`, `RUN_ID`, a per-run token in
`PRIME_API_KEY` and `PRIME_API_BASE=<platform>/api/internal/rft`.
prime-runs 0.1.1 (PrimeIntellect-ai/prime#895) addresses that internal
router and sends the run's token in `x-api-key`, the header it
authenticates on; 0.1.0 only spoke `/api/v1` with a Bearer header and
would have left those runs silently unmonitored. This monitor's
`_base_url()` passes `$PRIME_API_BASE` minus the trailing `/rft`, which
0.1.1 accepts, so the floor bump is the whole change. External runs
(prime-rl launched outside the platform) were never affected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes how training runs register and upload to the Prime platform
(including hosted `$RUN_ID` / internal RFT paths); misconfiguration now
fails launch instead of silently skipping monitoring, but orchestrator
run identity for W&B stays unchanged.
> 
> **Overview**
> **Replaces** the in-repo `TrainRun` HTTP/Parquet implementation in
`PrimeMonitor` with **`prime-runs[train]>=0.1.1`**, so run registration
(or `$RUN_ID` attach), metrics streaming, episode sample uploads,
finalize, and crash reporting are owned by the SDK uploader instead of
custom `httpx` + fire-and-forget tasks.
> 
> The monitor layer is now a thin adapter: `pr.init(kind="train", …)` on
startup, `run.log_metrics` / `run.log_episodes` via `asyncio.to_thread`,
and `run.finish()` with a **60s** drain timeout. Local behavior kept
where it matters: non-finite metrics are still **sanitized with named
warnings**, only **train / effective** episodes are forwarded, and
**`$PRIME_API_BASE`** is normalized for the SDK. Episode
**every-10th-step** upload cadence moves into the SDK sink (keyed on
dispatch step on `run.work`).
> 
> **Docs** in `training.md` describe the SDK, team allowlisting,
**`PRIME_RUNS_MODE=disabled`**, and **`RUN_ID`** attach.
**Lock/pyproject** add `prime-runs` and transitive **`prime-traces`**,
with fast-track resolver entries.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
aed9bbf. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants