[dghosef]

Not sure if we actually want this, might close

Stacked on #1175. Pending: a two-lane run on a real server (the config I tried doesn't load on the test box, for a reason unrelated to lanes).

What this does

• A worker's GPU can run any number of lanes. Lane 0 is the main chain, and each other lane is a CUDA stream with one step in flight. lane: <id> on a node group puts that node on a lane, and every walk of the node runs there. It replaces side_stream: true (which now fails at startup and points to lane:), and models no longer declare side_stream_walks. Nodes with different ids overlap; nodes with the same id take turns. The colocated Qwen3-TTS configs put the Codec on lane: 1.

• A lane can hold any resource only its own nodes declare, not just a recurrent pool. Whether a resource can live on a lane is decided from what the resource reports, not from its kind.

• What it doesn't do:

  • Lanes are streams, not a split of the GPU. Kernels on different lanes overlap but share the SMs and memory bandwidth, so a heavy lane can still slow the chain. Real partitioning would come from green contexts (not built) or from a split worker with MPS (already ships).
  • Not every node can go on a lane. These are refused at startup with the reason: TP/SP > 1, a walk with other nodes or a loop in it, a stop check or output spans, and a resource shared with another lane. A paged KV cache (eviction, prefix caching) and a sampler (host rows) are refused too, because they rely on chain-only machinery.
  • Lanes don't speculate or pre-plan. Their steps admit and plan inline, so heavy autoregressive nodes stay on the chain.
  • There are no per-lane memory budgets or preemption. A lane's resources are sized like any other's.
  • It isn't free. On H200, putting the Qwen3-TTS Codec on a lane costs about 6% audio throughput at c=1, is flat at c=8 and gains 3–6% at c=32. Side lane admits a recurrent pool: Qwen3-TTS codec beside the chain again #1175's side lane measures the same, so the colocated configs keep lane: 1 as Side lane admits a recurrent pool: Qwen3-TTS codec beside the chain again #1175 has it.

  So it fits light, stream-fed nodes (codecs, vocoders) beside a heavy chain, at high concurrency.

• Fixes from Side lane admits a recurrent pool: Qwen3-TTS codec beside the chain again #1175: the teardown fence is gone, and teardown no longer re-enters the lane's stream from the main thread. A refused step on a lane now fails rows that failed preparing, where before they hung.

How it works

• EngineManager.build reads the node→lane map and refuses nodes a lane can't serve. The engine then makes one stream per lane at load. Everything a lane issues goes on that stream through Engine.on_lane: capture, collating inputs, the step, and every call into its resources from outside the step (ingest, readiness, teardown). on_lane sets the thread's current stream with the runtime's per-call context, never with stream:, so the main thread and the GPU thread can both be on a lane at once.
• Engine._lane_contexts gives each lane the resources only its nodes declare. It refuses a resource that is shared across lanes or workers, depends on another lane's, keeps a prefix cache, has state the main thread moves outside the step, or stages host rows. StepRunner gets one map, resource → lane context. It uses the map to put those calls on the lane's stream and to leave the chain's staged pre-plan alone.
• The worker runs a tuple of Lanes (mstar/worker/lanes.py). The scheduler maps node → lane. Every lane shares the chain's submit, refused-step tail (_settle_step_rows) and postprocess. With the chain idle, the loop finishes whichever lane lands first.
• A step now waits only on the pre-plans of nodes it shares a resource with, directly or through what the resources depend on, not on every node's. The codec lane stops waiting on the Talker's plan stream. This also applies to chain models, so it gets its own A/B below.

Tests

Test	Where	Result
ruff check .	CPU	pass
CPU Core suite vs main	CPU box	11194 passed, 0 failures only on this branch
test_lanes.py: two lanes; capture on the lane's stream; a lane taking a resource that isn't a recurrent pool and refusing shared, cross-lane-dependent, prefix-cached, host-row and evicting ones; every off-step call into a lane's resource running on its stream; the lane that lands first finishing first; side_stream refused; a refused step failing its failed rows	CPU	pass
test_preplan_error_paths.py: a step waits only on the pre-plans of nodes sharing a resource	CPU	pass
Qwen3-TTS lane on vs off, audio-s/s c=1/8/32, 2 sessions each, qwen3tts.yaml (0.6B)	H200, torch 2.12.1+cu132	off 26.2 / 118.2 / 305.0, on 24.5 / 119.6 / 324.3, #1175 24.3 / 120.4 / 328.0; failed 0, no runaways
Same, qwen3tts_1p7b.yaml	H200	off 22.6 / 110.4 / 295.4, on 21.3 / 110.1 / 305.2, #1175 21.2 / 110.8 / 313.5; failed 0, no runaways
c=32 soak, 20 min, clients cancelling mid-stream (~2,800 cancels)	H200	~41,600 completed, 0 failed, 0 runaways, no stream errors; #1175 the same
Main thread and GPU thread on one lane at once; a freed slot is zeroed before the next request writes it	H200	pass
Pre-plan ordering, Qwen3-Omni c=8/32, this branch vs #1175	H200	64 / 149 vs 63 / 149 audio-s/s
Seeded byte compare at c=8	H200	inconclusive: the same server resending the same seeds matches only 8–15 of 56 at c=8 (batching changes the output)
Two lanes on a real server	H200	pending: qwen3tts_base.yaml fails to load (qwen_tts encoder upsample setter error) with or without lanes

🤖 Generated with Claude Code