[NSagan271]

What does this PR do?

Final rust graph PR, benchmarking code.

benchmark/worker_phases wraps mstar serve with MSTAR_PHASE_TIMING set, parses the worker's periodic phase-timing lines and prints one table per steady-state segment. Clients for Orpheus, Qwen3-Omni and BAGEL drive it closed-loop.

The worker now reports the mean in-flight batch size with each flush (bs=), which is what the harness segments on: ramp-up, steady state and drain are separate regimes, and one mean across them describes none. It also times the pieces of postprocess an A/B between graph runtimes needs apart: the completion sync, the stop check, storing outputs, the routing call, registration, and building vs. making the send.

How was it tested?

(testing in progress)

Checklist

• ruff check . passes
• Added or updated tests / docs where relevant

[merceod]

Used it for the #328 A/B and it does the job. The parser and segmenter tests pass, the client scripts line up with the runner's flags, and on two real three pass closed loop runs it split each worker's log into batch 1, 7.5, 28 and 47 regimes with the defaults and no hand tuning. I checked the numbers by hand. For every segment where the worker was continuously busy the sum of iter_total over the segment matches the wall clock between its first and last flush within 3 percent, and the segments that do not match are the ones spanning a gap between two runner passes, which the README already explains. The postprocess split is what was needed to see where the Rust runtime saves its time (route, send, speculate) and what it does not touch (register_outputs, check_stop). Merge.

Two notes. Segmentation compares each record against the segment's running mean, so a slow ramp where each flush moves under 15 percent would stay in one segment. It never showed here because the ramps are fast, but comparing against the segment's first record or capping a segment's bs range would make it robust. And a line in the README saying that TTFT and ITL come from the runner's closed loop, not from the harness, would save a reader the question.

[Gaurav-Shah05]

Instrumentation side holds up. Two nits: the clients default to port 8000 while every server command here uses 8100, and qwen3omni_t2s.sh's ${CONC:-4} is dead because _common.sh already set CONC=16, so it runs 16 concurrent against 12 requests and never holds a steady batch.

[Gaurav-Shah05]

A server left over from an earlier run answers this, so SERVER READY can fire before the child is up. The proc.poll() check catches a child that died, not one that never owned the port.

[Gaurav-Shah05]

a["sum"] counts phases sampled on passes that never increment phase_iter, like worker.schedule on an idle poll, but request_steps only counts iterations that did. So us/1k reads high for the scheduler phases and right for the postprocess ones, with nothing in the table saying which.

[Gaurav-Shah05]

Nothing installs a SIGTERM or SIGHUP handler, so either signal skips the finally and orphans the workers with their GPU memory. Same failure the killpg below is there to prevent.

[Gaurav-Shah05]

segment compares against cur.records[0].bs now, not the running mean.