Repository navigation
Worker: step threads on one core's two hardware threads, API result threads on another (opt-in) - #919
Conversation
Each worker picks a CPU from its GPU's local list (sysfs), within the process mask, distinct per worker on the host; knob step_thread_core / env MSTAR_STEP_THREAD_CORE (auto, off or a CPU number).
They take the last CPU of the process mask; the workers leave it out of their table. Same knob (step_thread_core); the step-thread helpers move to mstar/utils so both processes use them.
mstar-review
Bot
added
area/api
area/conductor
area/engine
area/utils
area/worker
size/M
labels
…hot-core # Conflicts: # mstar/platforms/interface.py
![mstar-review[bot]](https://avatars.githubusercontent.com/in/5406546?s=60&v=4){kind=small}
Automated review🔴 2 P1 on Description
Questions
What the review checked
Advisory, not a merge gate. Rules: |
…hot-core # Conflicts: # mstar/platforms/interface.py
Gaurav-Shah05
left a comment
Gaurav-Shah05
left a comment
There was a problem hiding this comment.
auto slows decode on a host without deep C-states, so it should stay off by default until some placement of the three threads shows no slowdown on such a host.
On one H100 in a cloud VM with no deep C-states, server under a 40-CPU taskset, shipped configs, two server sessions per arm (Qwen3.5-0.8B at 256 output tokens, Qwen3-TTS 1.7B at c=1):
| main | #919 | Δ | #919 on the GPU-local core's two hardware threads | |
|---|---|---|---|---|
| 0.8B c=1 throughput | 525 tok/s | 486 tok/s | 1.08x ↓ | 434 tok/s |
| 0.8B c=1 ITL p50 | 2.19 ms | 2.20 ms | 1.0x | 2.19 ms |
| 0.8B c=64 throughput | 14455 tok/s | 12800 tok/s | 1.13x ↓ | 14040 tok/s |
| 0.8B c=64 TTFT p50 | 98 ms | 114 ms | 1.16x ↑ | 94 ms |
| Qwen3-TTS TTFA mean | 21.0 ms | 25.6 ms | 1.22x ↑ | 21.0 ms |
| Qwen3-TTS requests over 30 ms | 1 of 80 | 29 of 80 | 0 of 80 |
The description's handoff times, 130-175 µs to another core against 4-5 µs on the same one, are the woken core leaving a deep idle state, as the module docstring says. On this host, which has none, a handoff costs 22 µs between two cores and 11 µs on one CPU, so c=1 doesn't gain and only the queueing shows. The description should say the gain needs deep idle states.
| return StepCore(None, f"skipped: step_thread_core {first} leaves no allowed CPU " | ||
| f"for worker {slot} in {format_cpu_list(allowed)}") | ||
| return StepCore(upward[slot], f"step_thread_core {first}") | ||
| if not narrowed_mask(allowed, host_cpus): |
There was a problem hiding this comment.
Each server picks CPUs only among its own workers. In my run two 0.8B servers under one 40-CPU taskset, one per GPU, both put their step threads on CPU 14 and their result threads on CPU 45, and at c=16 they averaged 3515 tok/s each against 6262 on main. The step and result CPUs should come from the device's rank among the host's GPUs on its NUMA node, read from sysfs, which differs between servers on different GPUs without any coordination, and then narrowed_mask can go.
There was a problem hiding this comment.
Agreed: the step and result CPUs come from the device's rank among the node's GPUs in sysfs, so two servers on two GPUs never meet, and narrowed_mask goes.
| _process_cpus = frozenset(process_cpus) | ||
| os.sched_setaffinity(0, process_cpus) | ||
| torch.empty(torch.get_num_threads() * _INTRA_OP_GRAIN, dtype=torch.uint8).fill_(0) | ||
| os.sched_setaffinity(0, {cpu}) |
There was a problem hiding this comment.
On main the three threads use 1.1 CPUs between them at 0.8B c=1, so on one CPU they queue, and that is the Qwen3-TTS regression the description leaves open: pinned, the worker's main and GPU threads waited 4.5 s and 3.6 s for the CPU over 44 requests. The GPU-local core's two hardware threads removed that in my runs (the summary's last column) but cost c=1 on this H100, so can you run that variant on your B300s to see whether it keeps your c=1 gain?
There was a problem hiding this comment.
Will run the two-hardware-threads variant on the B300s at 0.8B/2B c=1/16/64 and Qwen3-TTS c=1, against the single CPU and main, and put the table here.
| os.sched_setaffinity(0, {cpu}) | ||
|
|
||
|
|
||
| def unpin_thread() -> None: |
There was a problem hiding this comment.
Every pool a pinned thread starts has to remember unpin_thread, and dropping it from any of the three call sites fails no test. One helper that starts every pool on the process mask, with ruff's banned-api refusing a bare ThreadPoolExecutor or threading.Thread under mstar/, makes it a rule nobody can forget.
There was a problem hiding this comment.
Will do: one helper that starts every pool and thread on the process mask, with a banned-api rule for bare ThreadPoolExecutor and threading.Thread under mstar/.
| """Whether the process mask leaves out some of the host's CPUs, i.e. the | ||
| server runs under ``taskset`` or a cpuset.""" | ||
| if host_cpus is None: | ||
| host_cpus = os.cpu_count() or len(allowed) |
There was a problem hiding this comment.
cpus.py usable_cpus already reads the same affinity, with the cgroup quota, to size the API server's torch pool and the encode pool, and the CPUs reserved here still count toward them. One CPU plan in cpus.py should hold the mask, the step and result CPUs and the pool sizes from the CPUs left, with cpu_list.py folded into it.
There was a problem hiding this comment.
Will fold it: one CPU plan in cpus.py (mask, step and result CPUs, pool sizes from what is left, cpu_list.py folded in), read by the step threads and the API server's pools alike.
| if setting != "auto": | ||
| first = int(setting) | ||
| upward = sorted(c for c in allowed if c >= first) | ||
| if first not in allowed or slot >= len(upward): |
There was a problem hiding this comment.
A CPU number outside the mask, or too near its top for every worker, leaves the workers unpinned with an INFO line while the result threads still pin through api_thread_cpu. It should fail at startup with this message instead.
There was a problem hiding this comment.
Will make it fail at startup with that message.
| stray.append(tid) | ||
| except OSError: # exited meanwhile | ||
| pass | ||
| assert stray == [] |
There was a problem hiding this comment.
CI runs this file with OMP_NUM_THREADS=1, where torch starts no pool threads, so this check still passes with the warm-up op in pin_step_thread deleted (it fails at 4 threads). The test should run with a pool of at least two threads.
There was a problem hiding this comment.
Will run that test under a pool of at least two threads so the warm-up op is covered.
|
Thanks for the H100 run; that settles the default. Plan: auto stays off until a placement shows no slowdown on a host without deep C-states, and the description says the gain needs them; the two-hardware-threads variant gets measured on the B300s next to the single CPU; CPUs come from the device's rank among the node's GPUs in sysfs; one CPU plan in cpus.py with the pools sized from what is left; a pool-start helper plus a banned-api rule; a CPU number the mask can't honour fails at boot; the pin test runs under a real pool. Push and numbers to follow. |
…ne CPU plan The step and result CPUs come from sysfs (the GPU's rank among the host's GPUs on its NUMA node), so servers on different GPUs pick different CPUs. mstar/utils/cpus.py holds the plan: mask, usable count, pinned CPUs and the pool sizes from what is left. A CPU number the mask can't honour fails at startup. Threads and pools start on the process mask through mstar/utils/threads.py, and ruff refuses bare ones under mstar/.
On one CPU the main, GPU and plan threads queue behind each other (Qwen3-TTS c=1 TTFA p50 23.5 -> 34.6 ms); on the core's two hardware threads TTFA matches main and the c=1 decode gain stays. Docs give the numbers for the B300 host and the H100 VM from the review.
|
Rework pushed (e58fb54, main merged through #869), description rewritten for the final state.
One finding on the side: the 9B late-token greedy flips come with main's per-process Inductor coordinate-descent picks (98 of 130 kernels differ between two main processes; two -O0 processes agree 32/32), not with this branch. |
Gaurav-Shah05
left a comment
Gaurav-Shah05
left a comment
There was a problem hiding this comment.
With off as the default, main's path runs as before, and what still needs fixing is in auto's placement and in the thread helper.
I ran Qwen3.5-0.8B with the Rust frontend and Qwen3-TTS 1.7B with the Python one on one H100 in a cloud VM with no C-states exposed, under a 32-CPU taskset. The 0.8B numbers are medians of three server sessions per arm, the TTS ones of two. auto costs decode throughput but not TTFA:
| main | off |
auto |
Δ auto vs off |
|
|---|---|---|---|---|
| 0.8B c=1 throughput | 517 tok/s | 529 tok/s | 419 tok/s | 1.26x ↓ |
| 0.8B c=1 ITL p95 | 2.38 ms | 2.30 ms | 3.28 ms | 1.43x ↑ |
| 0.8B c=16 throughput | 6410 tok/s | 6520 tok/s | 5526 tok/s | 1.18x ↓ |
| 0.8B c=64 throughput | 15720 tok/s | 16569 tok/s | 15803 tok/s | 1.05x ↓ |
| Qwen3-TTS c=1 TTFA p50 | 20.1 ms | 20.1 ms | 20.4 ms | 1.0x |
| """``(workers, result, why)``, or why nothing can be pinned.""" | ||
| if not cores or not addresses: | ||
| return "no CPU or no worker to place" | ||
| gpus = next((found for a in addresses if a and (found := host_gpus(a, pci_root))), []) |
There was a problem hiding this comment.
The plan depends only on the GPU's PCI address and the mask, so two servers on one GPU, two MIG instances included, get the same CPUs. Here auto should notice the second server and pin nothing there with a warning, for example through a lock file per GPU address that the first server holds.
There was a problem hiding this comment.
Done. Under auto the API server takes an flock on /tmp/mstar-gpu-<PCI address>.lock for every GPU of the deployment before anything starts (cpus.claim_gpus, called from entrypoint.deployment_cpu_plan). If one is already held it holds none of them, logs step_thread_core auto: nothing pinned, another server holds GPU 0000:fa:00.0 (...), and sets the knob's env var to off so the conductor and workers it spawns pin nothing either. The kernel drops the lock however the process exits, so a dead server doesn't block the next one, and the file is never unlinked (same reasoning as the communicator's deployment lock). MIG instances should report the parent GPU's PCI address, which puts two MIG servers on one lock; I couldn't check that here, since MIG isn't enabled on these B300s. Only auto locks: a CPU number is the operator's call. Tests: test_a_second_server_on_a_gpu_is_refused_until_the_first_exits, test_a_killed_server_does_not_block_the_next (the first holder is a real process, SIGKILLed), test_a_refused_claim_holds_nothing, test_the_first_server_pins_and_the_second_pins_nothing_with_a_warning (through deployment_cpu_plan and a real Worker._plan_step_cpus). On GPU 1 here: the first auto server took 97,225 (step) and 102 (result); a second one on the same GPU logged step_thread_core auto: nothing pinned, another server holds GPU 0000:fa:00.0 (its lock is /tmp/mstar-gpu-0000:fa:00.0.lock), its worker logged step threads not pinned (step_thread_core off) and every thread stayed on the process mask; once the first was stopped a third took 97,225 and 102 again.
| # workers of this deployment on one device (a split config) step by | ||
| # the GPU count, so the next GPU's first worker keeps its core | ||
| nth = sum(a == addresses[i] for a in addresses[:i]) | ||
| start = (rank + nth * gpus_here) % len(pool) |
There was a problem hiding this comment.
With fewer than two cores per GPU in a pool, the ranks wrap silently: on the test's fake host with its 32-39 mask, the servers on the first and fifth GPUs both get step CPU 34 and result CPU 37. In that case auto should pin nothing and warn, as it does for a mask that is too small.
There was a problem hiding this comment.
Done. The pools are now a _Pool(cores, rank, gpus, why) per GPU (cpus._gpu_pools), and _auto pins nothing and warns when any of the deployment's GPUs picks from a pool with fewer than 2 * gpus cores, rather than taking the rank modulo the pool. The extra workers of a split config step through the cores between the bottom gpus and the top gpus and also pin nothing once they run out, instead of falling back to "the first free core of the mask". GPUs whose node has no allowed CPU now rank among themselves rather than among all of the host's GPUs (6 of the 8 here and in the fake host, when the mask sits on one node), so they need 12 spare cores rather than 16. Your example: on the fake host with 32-39, every off-node GPU now gets not pinned: no allowed CPU on its NUMA node, so GPU 1 of 6 without one, ...: 4 core(s) there for 6 GPU(s), and each needs two; widen the mask or name a CPU. Tests: test_auto_pins_nothing_with_fewer_than_two_cores_per_gpu_in_a_pool (5 cases, including the first/fifth GPU pair you named and a node-local pool of 3 cores for 2 GPUs), test_two_cores_per_gpu_is_enough, test_no_two_gpus_of_the_host_share_a_cpu_under_one_mask (all 8 GPUs pairwise under 8 masks), test_workers_that_outgrow_their_pool_pin_nothing_rather_than_wrap. On the real host: a server on GPU 1 under 32-39 logged not pinned: no allowed CPU on its NUMA node, so GPU 6 of 6 without one, ...: 4 core(s) there for 6 GPU(s), and each needs two; widen the mask or name a CPU in the API server and the worker. One case this doesn't cover: a split config's second worker on a node-local GPU takes a core from the middle of the node's cores, which a server on a GPU without allowed CPUs under the same mask can also take (listed under Not covered in the description). That also means the A/B mask from the last round (32-39,160-167 for GPU 0) no longer pins under auto, so the re-measure uses 32-47,160-175.
| """A ``threading.Thread`` that runs on the process mask, whatever its | ||
| creator's. Pass ``target``; a subclass overriding ``run`` skips this.""" | ||
|
|
||
| def run(self) -> None: |
There was a problem hiding this comment.
A subclass that overrides run without calling super().run() starts on its creator's CPU, and ruff doesn't flag it. The Thread class should reset the mask before any subclass's run, for example by wrapping it in __init_subclass__.
There was a problem hiding this comment.
Done, without asking anything of the subclass: Thread.start swaps the instance's run (whatever it resolves to: a target, an override, a mixin's, an attribute assigned on the instance) for a wrapper that moves to the process mask and then calls it. I wrapped at start rather than in __init_subclass__ because the latter only sees a run defined on the subclass itself and misses a mixin's run ahead of Thread in the MRO; it also keeps a direct t.run() call on the caller's own thread and mask. The wrapper deletes itself after running, as Thread.run drops its target, so no cycle is left behind. Tests: test_a_thread_runs_on_the_process_mask_whatever_its_run_is (override without super().run(), a subclass of that, a mixin, an assigned run, and a direct run() call that stays pinned) and test_an_override_started_from_a_pinned_thread_is_on_the_process_mask; both fail with the old run override in place.
| groups = [g for g in config.get("node_groups") or [] if isinstance(g, dict)] | ||
| ranks = sorted({r for g in groups for r in g.get("ranks") or [] if isinstance(r, int)}) | ||
| rank_devices = {int(r): int(i) for r, i in (config.get("rank_devices") or {}).items()} | ||
| indices = [rank_devices.get(rank, rank) for rank in ranks] or [0] |
There was a problem hiding this comment.
deployment_cpu_plan rebuilds the workers' devices from node_groups and rank_devices beside the conductor's _sorted_ranks and worker_device. If the two lists ever differ, the API server's result CPU can land on a worker's step core, so both should come from one function that takes the config.
There was a problem hiding this comment.
Done. mstar/conductor/workers.py:deployment_workers(config, device_type) returns (ranks, devices): every rank node_groups names, on the device worker_device gives it through rank_devices. The conductor sets _sorted_ranks and _worker_devices from it, spawns each worker on its entry and passes the list as local_worker_devices; deployment_cpu_plan calls the same function, and both build the plan through cpus.devices_plan(setting, devices). The conductor used to take its ranks from the worker graphs instead, and the two can only differ for a rank that no walk's node runs on, so check_placement now refuses that (node_groups names ranks [1, 2], but no node of any graph walk runs on them.). The shipped-config sweep in test_startup_graph_validation.py asserts conductor ranks == worker-graph ranks for every config. test_the_api_server_and_the_conductors_workers_build_one_plan runs the real Conductor.__init__ and _launch_workers (spawn stubbed) on qwen3tts_1p7b_split.yaml (two ranks on cuda:0) and qwen3_5_27b_tp4.yaml (TP4), then the real deployment_cpu_plan and each spawned worker's real _plan_step_cpus from its spawn kwargs: same plan everywhere, and the result CPU on no worker's step core. It fails if the API side ignores rank_devices or if the conductor goes back to its own list.
|
Taking all four: a per-GPU-address lock so a second server on the same GPU (MIG included) pins nothing and warns; fewer than two cores per GPU in a pool pins nothing and warns instead of wrapping; |
- auto takes a lock per GPU address (/tmp/mstar-gpu-<addr>.lock, flock held for the API server's life). A second server on the GPU pins nothing and warns, and its workers see the knob off. - A pool with fewer than two cores per GPU that picks from it pins nothing and warns instead of wrapping. GPUs off their node rank among themselves. - threads.Thread wraps run at start, so an override that skips super().run() still starts on the process mask. - conductor/workers.py: deployment_workers(config) gives the ranks and devices to both the conductor and the API server's CPU plan; placement refuses a rank no walk runs on, so the conductor's list is that one. - test_run_moves...: the stray check reads the GPU thread by name (it could look before the GPU thread reported).
merceod
changed the title
|
Thanks, all four are done in 1924922, on top of 32b0756, which merges main 7eac23c (only import and arg-list conflicts with #849/#850/#941). Inline replies below. Re-measured on the B300 host, |
Under a umask of 077 the lock was created 0600, so every other user's `auto` server on that GPU failed to open it and pinned nothing until the file was deleted.
There was a problem hiding this comment.
LGTM. The points from my last review are done, and I pushed three commits on top for what was left:
- The lock file's mode.
os.openapplies the umask to the mode it's given, so under a umask of 077 the lock was created 0600, and every other user'sautoserver on that GPU then failed to open it and pinned nothing until the file was deleted._open_locknow creates the file withO_EXCL, sets 0644 withos.fchmod, and opens an existing one read-only withoutO_CREAT. - The docs row. It now quotes the 16-core run, since under the 8-core mask
autopins nothing for GPU 0, and it carries the two limits the description lists under Not covered: the lock only reaches servers that share/tmp, and a split config's second worker can take a core that a server on a GPU without allowed CPUs also takes.
… into perf/split-7-generation Main's #919 bans a bare ThreadPoolExecutor under mstar/ and sizes the process's pools from its CPU plan, so the GOP-parallel video decode pool is a ThreadPool sized from the CPUs the plan leaves unpinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A decode step hands work between the worker's main, GPU and plan threads five times, and every step's tokens pass from the API server's message thread to the Rust frontend's bridge. On a host whose idle cores drop into deep C-states, each handoff to a sleeping core waits for it to leave C6: 130-175 µs per handoff on the 2x B300 host (Xeon 6767P,
intel_idle, C6 latency 170 µs), 4-5 µs when the woken thread lands on a running core. The gain needs deep idle states. On an H100 cloud VM without them a handoff costs 22 µs to another core and 11 µs on one CPU, so pinning gains nothing there and costs decode throughput (review table below). Sostep_thread_coreis off by default; turn it on (knobs: {step_thread_core: auto}) where/sys/devices/system/cpu/cpu*/cpuidle/state*/latencyshows a C6-class state of 100 µs or more and the server is host-bound.What changes
autopins each worker's main, GPU and plan threads to one core's two hardware threads (both, where the mask has both), and the API server's message thread and Rust bridge to one CPU of another core. On a single CPU the three threads queue: at Qwen3-TTS c=1 they need about one CPU between them (the worker main thread alone ~0.55-0.7, mostly_wait_for_step's event synchronize, which spin-waits).mstar/utils/cpus.py:_gpu_pools,_auto). The host's GPUs are the PCI functions of the device's vendor and class in address order (all of the host's, not only the visible ones). A GPU with allowed CPUs on its NUMA node picks from those by its rank on the node: its worker takes the core at that rank from the bottom, its API server the core at that rank from the top, and more workers of one deployment on that GPU step by the node's GPU count. A GPU whose node has no CPU in the mask picks, by its rank among the GPUs in the same case, from the cores the node-local GPUs leave.autopins nothing and warns rather than wrapping onto another GPU's cores. On this host that means GPU 0 under the 8-core32-39,160-167pins nothing (4 spare cores for the 6 GPUs off their node) and needs32-47,160-175.autothe API server holds anflockon/tmp/mstar-gpu-<PCI address>.lockper GPU for its life (cpus.claim_gpus); a second server finds it held, warns (step_thread_core auto: nothing pinned, another server holds GPU 0000:fa:00.0 (its lock is /tmp/mstar-gpu-0000:fa:00.0.lock)) and pins nothing, here or in its workers. The kernel drops the lock however the process exits, and the file is never unlinked.mstar/conductor/workers.py:deployment_workers(config, device_type)gives the ranks and devices; the conductor spawns its workers from it and the API server builds its plan from it, both throughcpus.devices_plan, so the API server's result CPU can't land on a worker's step core. The conductor used to take its ranks from the worker graphs; the two differ only for a rank no walk runs on, whichcheck_placementnow refuses (no shipped config has one; the shipped-config sweep checks it).CpuPlaninmstar/utils/cpus.py,cpu_list.pyfolded in): the mask, the count the cgroup quota lets run, the pinned CPUs, and the pool sizes from what is left (the API server's torch pool, the data worker's encode pool and output-copy width, the PNG deflate threads).mstar: step_thread_core 5 is outside the process mask 96-103). Nothing pins partially.mstar.utils.threads.Thread/ThreadPool.Thread.startwraps whateverrunresolves to (a target, an override that never callssuper().run(), a mixin's), so a subclass needs no cooperation. Ruffbanned-api(TID251) refuses a barethreading.ThreadorThreadPoolExecutorundermstar/, except in the helper module and tests.step_thread_core: off | auto | <cpu>(envMSTAR_STEP_THREAD_CORE), defaultoff. Docs: environment_variables.rst, serving.rst.Review numbers (H100 VM, Gaurav-Shah05)
Qwen3.5-0.8B with the Rust frontend and Qwen3-TTS 1.7B with the Python one on one H100 in a cloud VM with no C-states exposed, under a 32-CPU
taskset; 0.8B medians of three server sessions per arm, TTS of two:offautoautovsoffoffruns main's path, and on that hostautocosts decode throughput, which is why it stays opt-in.Numbers on the B300 host
2x B300, torch 2.12.1+cu132, FlashInfer 0.7.2.post1. GPU 0, client on 64-67. Qwen3.5: llm1 cells (512-token prompts, 256 out),
--rust-frontend -O2,MSTAR_SHM_ARENA=1, knobs{kv_plan_cache: 1, ingraph_decode_rope: true, gc_gen0_threshold: 50000}, prefix cache off, two runs per session (5 and 1 warm-up waves), medians [min-max]. Qwen3-TTS: 200 Seed-TTS sentences per session at c=1, closed loop, WER by Whisper large-v3. 0 failed requests.This branch at its head (main
7eac23c10merged), servertaskset -c 32-47,160-175(16 node-1 cores and their siblings;autoputs GPU 0's step threads on 35,163 and the result threads on 44), one server session per arm for 0.8B and two per arm for TTS, alternated:offautoThe previous round, on main
a56ebe4cdundertaskset -c 32-39,160-167, two sessions per arm: "single" is all three threads on one CPU, "pair" the core's two hardware threads (the same step core 35,163 as above; under that 8-core maskautonow pins nothing for GPU 0).Per-thread CPU and run-queue wait in that round (schedstat, ms per s of wall; main / single / pair). Qwen3.5-0.8B c=1: worker main 197/0, 174/109, 195/6; GPU 257/0, 205/182, 242/6; plan 257/0, 247/132, 263/26. Qwen3-TTS c=1: worker main 546/1, 683/197, 704/7; GPU 264/5, 159/140, 191/8; plan 171/5, 107/103, 124/4 (the single CPU 95% busy): on one CPU the GPU and plan threads queue behind the main thread's spin-wait, and the pair removes the wait. worker_phases (0.8B,
MSTAR_PHASE_TIMING=100): c=16 iter_total 3.84 / 2.45 / 2.57 ms, c=64 5.17 / 3.79 / 3.86 ms; on the pair each thread runs ~10% slower sharing the core, so at c=16-64 the second hardware thread doesn't shorten the step.Kokoro, Whisper and FLUX.2-klein were not rerun with the pair; their single-CPU numbers from the first version of this PR were +26-99% (Kokoro), +5-29% (Whisper) and -6%/+3% (klein).
Late-token flips at 9B are per process, on main too
Greedy outputs at Qwen3.5-9B differ across server processes on main alone (three main processes agree 22-26/32, first splits at tokens 0-112). Each -O2 9B server re-runs Inductor's coordinate-descent tuning for the same 130 Triton kernels at startup, and 98 of the 130 picks differed between two main processes; two -O0 processes agree 32/32. So the flips come with main's per-process kernel picks, not with this PR (the causal link rests on one -O0 pair: strongly suggested, not proven).
Tests
test/modular/test_step_threads.py(61 tests): host GPUs from a fake sysfs, rank placement, two servers one per GPU under one taskset take disjoint CPUs (5 masks), all 8 GPUs pairwise disjoint under 8 masks, pools with fewer than two cores per GPU pin nothing (5 cases, including off-node GPUs 1 and 5 under32-39), split-config workers that outgrow the pool pin nothing, a VM without NUMA nodes, CPU numbers and their startup errors, pool sizes; the GPU lock against a real second process (refused while it runs, free after it exits or is SIGKILLed, a refused claim holds nothing, another user's lock file, an unopenable lock dir) and throughdeployment_cpu_planplus a realWorker._plan_step_cpus; the real conductor's worker list and spawn kwargs vs the API server's plan and each worker's plan forqwen3tts_1p7b_split.yamlandqwen3_5_27b_tp4.yaml;Threadwith an override that skipssuper().run(), a subclass of it, a mixin and an assignedrun;ThreadPoolvs a bare pool from a pinned thread; the intra-op warm-up; the realWorker.runwith 1 and 2 step CPUs; the ruff rule per path; the API message thread and bridge. Each new test fails with its fix taken out (checked one fix at a time). Alsotest_startup_graph_validation.py(the idle-rank refusal; the sweep checks conductor ranks == worker-graph ranks for every shipped config),test_knobs.py,test_preprocess_threads.py,test_data_worker_encode_pool.py,test_host_copy.py.OMP_NUM_THREADS=1and4:test_step_threads.pyplustest_knobs,test_preprocess_threads,test_data_worker_encode_pool,test_host_copy,test_worker_device_map,test_conductor_moe_prebuild: 126 passed at both in a CI-replica venv (torch 2.10.0+cpu); withtest_startup_graph_validation.py, 250 passed / 1 skipped at both under torch 2.12.1+cu132.scripts/ci/select_cpu_tests.pypaths, CI's flags:PYTEST_DISABLE_PLUGIN_AUTOLOAD=1,OMP_NUM_THREADS=1,-p xdist -n 12 --dist worksteal), GPUs idle: main7eac23c1041 failed / 8856 passed / 787 skipped, this branch 41 / 8924 / 787, the same 41 (32test_loop_output_ownershipon the Rust build's_ref_info, 5 libibverbs, 2test_eval_compare_multigpustack dumps, 1 prometheus_client, 1test_client_error_text);test_piecewise_config_signature.pyand the lockstep sim, run alone, 19 passed on both.ruff check .clean.auto,taskset -c 96-103,224-231): the first server on GPU 1 pinned main/GPU/plan to 97,225 and the result threads to 102; a second server on the same GPU logged the warning above and its worker loggedstep threads not pinned (step_thread_core off), every thread on the process mask; after the first stopped, a third took 97,225 and 102. A server on GPU 1 under32-39pinned nothing and warned... 4 core(s) there for 6 GPU(s), and each needs two; widen the mask or name a CPU. All served requests completed.Not covered
/tmp, so it only separates servers that share it (not containers with private/tmpon one GPU).