[mikasenghaas]

Summary

Prime uploads keep up with the orchestrator, and logging to Prime never blocks it. Companion to PrimeIntellect-ai/prime#1136.

• SDK pin. prime-runs[train]>=0.1.6, which pulls prime-traces 0.0.12 from PyPI. These releases contain feat: non-blocking, concurrent run uploads in prime-runs and prime-traces prime#1136: gzip level 1, up to 8 upload requests in flight, an upload queue bounded by records whose submit never blocks, and lazily built upload records.
  • uv.lock is written with uv 0.12.0 (lockfile revision 3); only the two SDK packages change.
• No thread hop for Prime. PrimeMonitor calls run.log_metrics and run.log_episodes directly on the event loop, not through asyncio.to_thread. A queue put that never blocks needs no thread, and each hop had to win the GIL back from the uploader thread.
• A production-sized load test. tests/unit/monitors/test_monitor_load.py:
  • Episode sizes: episodes draw their size from a lognormal fitted to a production RL run (median 6.4 MB, p90 17 MB; it also matches the mean of 8.5 MB and the p99 of 35 MB), capped at 60 MB. 1,000 episodes is ~8 GB of JSON, close to one production step of 1,024 samples.
  • Memory: episodes share 24 traces at log-spaced sizes, so the test's own memory stays small.
  • Fake endpoint: the fake Traces endpoint gzips at the SDK's level and counts lines under a lock, because the client now sends several requests at once.
  • File monitor check: it streams every trace chunk, since 8 GB spans two 5 GiB chunks.
  • No expected failure: the prime monitor at 1,000 episodes is no longer marked as one.
  • A separate log() limit of 1 s. A log() call may await the monitor's own work: the file monitor awaits its write, which encodes the episode on a worker thread, while the event loop keeps serving other tasks. The lag limits still cover the loop.
  • The p99 lag limit rises from 50 to 100 ms. At production sizes the file monitor encodes each episode in one call, which stalls the event loop with a p99 of ~60 ms on CI. Main's code shows the same stall with this test. feat: encode episodes for the file and prime monitors one node at a time #4366 makes the encoder work in short steps and tightens the limits to 25 ms p99 and 100 ms max.

Verification

Headline: Prime monitor, 1,000 production-sized episodes (~8 GB of JSON), main against this branch. Main ran this branch's test, with the fake endpoint gzipping at main's level 9.

	main	This branch
Episodes delivered	725 of 1,000	1,000 of 1,000
Lost to a full upload queue	27	0
Lost when the 60 s drain ran out	248	0
Slowest log()	up to 5 s (blocked on the full queue)	0.6 ms
Event-loop lag p99 / max	not reached (failed on delivery)	14 / 17 ms
Test time	217 s	28 s

Scale: 10,000 episodes of 2.2 MB (~22 GB), from a local copy of the test with the count raised:

Setup	Result
No network time	0 dropped; log() max 1.4 ms; event-loop lag p99 14 ms, max 21 ms; 570 MB peak RSS
100 ms + 100 MB/s per request, 1 request in flight	2,595 delivered within the 60 s drain
Same, 4 in flight	9,822 delivered
Same, 8 in flight (the default)	10,000 delivered; event-loop lag p99 12 ms; 860 MB peak RSS

The upload rate is ~180 episodes/s (~400 MB/s of JSON). Encoding the records is the limit.

Full load-test grid at production sizes, this branch (dev box): 9 passed.

1,000 episodes	Event-loop lag p99 / max	log() max
prime	14 / 17 ms	0.6 ms
file	47 / 93 ms	177 ms
wandb	0.9 / 0.9 ms	0.6 ms

CI: on the CI runner, the file monitor at 1,000 episodes measured event-loop lag p99 60 ms and max 157 ms, within this PR's limits (100 / 250 ms), and log() up to 299 ms (limit 1 s). With #4366 the same case measures p99 6 ms and max 12 ms.

Other checks:

• pytest tests/unit/train/sft/test_prime_monitor.py tests/unit/utils/test_links.py tests/unit/orchestrator/test_episode_records.py tests/unit/train/rl -m "not gpu": 29 passed.
• uvx uv@0.12.0 lock --check passes.

Merge order

1. feat: non-blocking, concurrent run uploads in prime-runs and prime-traces prime#1136 and feat: upload an episode's own to_json when it names the run prime#1140: merged and released as prime-runs 0.1.6 and prime-traces 0.0.12.
2. feat: non-blocking, concurrent prime uploads #4363 (this PR): ready to merge. It does not need feat: encode episodes to JSON one node at a time and record runs on traces verifiers#3042.
3. feat: encode episodes to JSON one node at a time and record runs on traces verifiers#3042
4. feat: encode episodes for the file and prime monitors one node at a time #4366

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the hot path for RL training telemetry (episode/metrics enqueue) and bumps external SDK versions; misbehavior could drop or delay uploads under load, though load tests now enforce delivery and loop responsiveness.

Overview
Prime logging no longer blocks the orchestrator event loop by pinning prime-runs[train]>=0.1.6 (and prime-traces 0.0.12 in the lockfile), which uses a non-blocking upload queue, concurrent uploads, and lighter gzip.

PrimeMonitor now calls run.log_metrics and run.log_episodes directly instead of asyncio.to_thread, matching SDK behavior where enqueue never blocks.

The monitor load test is reworked to use production-like episode sizes (lognormal ~6.4 MB median), shared trace templates for memory, a thread-safe fake Traces API, and multi-chunk file delivery checks. The Prime monitor at 1,000 episodes is no longer expected to fail; responsiveness asserts split event-loop lag (p99 100 ms) from per-log() duration (max 1 s).

^{Reviewed by Cursor Bugbot for commit 11ae629. Bugbot is set up for automated code reviews on this repo. Configure here.}