[kcoopermiller]

Summary

A burst of long episodes filled the prime_runs upload queue. Each further log() then blocked for 5 s and dropped its episode. The uploader took everything queued (up to 256 episodes, many requests) into one long write while the producer filled the queue again. #1129 made uploads faster, but the drops remained.

• log() never waits. submit() appends to the queue, or drops at once when the queue is over budget.
• The queue is budgeted in bytes. The limit is about 4 GiB of encoded records, plus a cap of 65,536 records. Records are only encoded at upload, so the uploader measures them as it takes them and keeps a running estimate. The producer pays nothing, and each record encodes one extra time on the uploader thread.
• Writes are capped by bytes. One write takes at most about 30 MiB of what is queued, whatever mix of sizes is there, which is about one request. A failed write now costs that much rather than everything queued. An oversized batch goes out in order across writes, and a single record over the limit goes out alone.
• Drop warnings are limited to one per 10 s. run.dropped_records still has the full count.

UploadWorker's max_queue_size and put_timeout arguments are gone. pr.init never exposed them.

Behavior to check in review

• Sustained overload now drops rather than slows the producer. Before, a full queue held the producer back for up to 5 s per call. A producer that outruns the service for longer than the budget covers loses what doesn't fit.
• More can be queued at exit. finish() still drains for finish_timeout, and whatever is left is lost with a warning giving the count. A caller with a short timeout should size it for what it may have queued: 1000 long episodes took about 3 minutes here.
• The byte budget is an estimate. It multiplies the queued record count by a running average of encoded size, so a sudden shift to larger records can overshoot it for a while. It also measures encoded size, not memory: records holding plain Python lists use more memory than their encoded size, and numpy-backed ones use less.

Verification

1000 episodes of about 2.3 MiB, each with its own numpy arrays, submitted as fast as possible through the real UploadWorker, TracesSink and TracesClient against a dev Prime Traces deployment, from a laptop on about 320 Mbit/s of uplink:

	Stored	Dropped	Total time	Longest submit()	Peak memory
main (from an earlier run, episodes sharing arrays)	982	18	177 s	5.0 s	not measured
This PR	1000	0	178 s	0.2 ms	1.9 GiB

The stored count was read back from the service. Throughput is unchanged: uploads are still sequential, at about 5.6 episodes/s.

The prime-rl monitor load test, on this branch with its expected failure enforced: 9 passed. prime-1000 passed three more times alone, with log() at 0.3 ms p99 and about 26 ms at most, against about 3 s before.

packages/prime-runs/tests: 308 passed, 1 skipped. ruff check . at the locked version passes.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes backpressure and memory footprint on the hot path for trace logging during training runs; sustained overload drops data instead of slowing the producer, and large queues can hold substantially more than the old 256-slot cap.

Overview
Fixes training log() stalls when trace uploads lag: UploadWorker.submit() no longer blocks on a full queue (removed put_timeout / max_queue_size). Overload is handled by immediate drops once a record count (~65k) or estimated encoded-byte budget (~4 GiB) is exceeded, with drop warnings throttled to once per 10s.

The uploader thread now sizes records (JSON encode on the worker) to enforce those budgets without charging the producer, and _coalesce caps each sink write at max_write_bytes (~one HTTP batch) so a slow or failed upload does not drain the entire backlog in one call. flush() / close() enqueue barriers and stop via put_nowait instead of timing out behind a saturated queue. README documents that logging never waits for upload. Tests cover non-blocking drops, byte/write limits, ordering, and warning rate limits.

^{Reviewed by Cursor Bugbot for commit 012db5e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T19:06:41.432676Z	2b2f262	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[kcoopermiller]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b2f262b43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kcoopermiller]

Closing in favor of #1136, which also uploads concurrently and is about 4x faster on a real service. Results and the parts worth carrying over are in my comment there.