Repository navigation
Host-side overhead: per-row Python out of the decode step #368
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 109 commits
Commits
Show all changes
112 commits
Select commit
Hold shift + click to select a range
2430058
resources: publish snapshot hooks so a step can defer its publication
merceod 91cae50
runner: publish_snapshot and publish_from_snapshots sweeps
merceod 68d7c4d
kv: snapshot the publication per step and finish it on demand
merceod 18d8e17
rope: snapshot the publication per step and finish it on demand
merceod e84b7c2
engine: finalize_batch snapshots the publication, materialize_publish…
merceod 99593f1
worker: finish publications only for stopped rids and rids that send …
merceod 258e27a
test: lazy publication
merceod b36967b
submodule_base: a stop buffer that is a row output shares the row clone
merceod 52b171e
engine: collect_outputs fast path for row-only outputs
merceod 9272246
test: collect_outputs fast path
merceod c05f8bb
micro_scheduler: check backlog readiness only up to a step's worth
merceod 2191035
test: bounded backlog readiness checks
merceod f5a7327
resources: keys_prefix_chains says whether a prefix cache is open
merceod 9e38073
kv: report an open prefix cache through keys_prefix_chains
merceod f1b0be5
runner: prefix_chain_keys, and extend_prefix_chains over a given key …
merceod 5ef8de3
engine: skip the per-request prefix-chain sweep when no cache is open
merceod bdf055e
rust frames: RESULT_TOKENS, one frame per step for inline client values
merceod 77d32ec
rust runtime: route an inline client signal without staging, refs or …
merceod 77aab0c
graph runtime: RouteInput.inline_signal/inline_values and supports_in…
merceod db4def8
rust runtime wrapper: pass inline values, report supports_inline_emit
merceod 95006f7
api_server: ResultTokens message type
merceod 6661a71
wire: register result_tokens
merceod f6e5939
api_server: dispatch result_tokens frames to the data worker
merceod 27322e4
data worker: detokenize inline tokens in arrival order, no read and n…
merceod 810cf24
submodule_base: inline_client_signals hook
merceod 1761cde
qwen3_5: the decode loop's client token rides inline
merceod 02a6af9
engine: inline_client_signals passthrough
merceod d57681c
worker: hand the stop check's host tokens to the route as inline values
merceod 21fb696
test: inline token path (wire, data worker, worker)
merceod a31c952
test: rust runtime inline emit
merceod b61b1b5
qwen3_5: decode capture buckets up to 128 rows
merceod 51edd0c
api_server: wake the streaming handler on events instead of polling p…
merceod 8d21448
test: a chunk from the message thread wakes the stream
merceod 85c5367
submodule_base: step_is_input_free hook
merceod ba56be0
qwen3_5: decode steps declare from spans alone
merceod 2ce4dac
engine: run the pre-plan's declared step for input-free walks
merceod 6557266
docs: MSTAR_LAZY_PUBLISH
merceod 080065c
worker: build and pre-plan N+2 while N+1 runs, between N's routing an…
merceod 65ffc1a
docs: MSTAR_EARLY_SPEC
merceod 7b21494
sampler: skip the static gather for a steady batch while other reques…
merceod 6be4dd1
kv plan: build the paged indptr tensors through numpy
merceod ab3c97a
test: static gather skip with queued requests
merceod 1025fd5
ci: list the new cpu tests
merceod 7359e67
batched output: carry the row views the collect fast path built
merceod 13e7943
engine: hand the row views to the collected output
merceod 5a057b6
tensor store: batch the ownership maps, dtype index helper
merceod cb3d7e6
tensors: row store for a step whose outputs are row views of one batc…
merceod f19248a
worker: store decode row outputs through the row store
merceod 129311f
test: merge_per_rid returns the row views
merceod c104c99
test: row store against the general path
merceod 28d739d
ci: list the row store test
merceod 760bde4
worker: build the stop check's per-rid host dicts only when read
merceod 4e50fb7
engine: skip the unpack pass when nothing is packed
merceod 9163255
test: lazy per-rid host dicts
merceod 5e2b57c
ci: list the lazy per-rid test
merceod 083cc0f
qwen3_5: MSTAR_QWEN35_DECODE_BUCKETS overrides the decode capture ladder
merceod 5025706
docs: MSTAR_QWEN35_DECODE_BUCKETS
merceod a025fac
step context: constant-time padding and real-row tests
merceod b3a01b6
kv: batched publish snapshot under one lock, real-row test in the pag…
merceod 91e4d28
resources: publish_snapshot_batch hook
merceod b2f60e6
runner: snapshot the publication per resource for the whole batch
merceod cb6bed1
sampler: one graph sampler per bucket and slot
merceod 1dce205
test: step context row membership
merceod 6cb5601
test: batched publish snapshot
merceod d2dcb38
test: sampler_for cache
merceod 9f222e4
ci: list the step, snapshot and sampler tests
merceod fb04771
kv admit: skip the allocation call for rows whose pages already cover…
merceod a2d1155
engine: hoist the per-row constants out of prepare_inputs, look up th…
merceod dbbc572
arena: batched cleanup of collectable tensors
merceod 50baf43
worker: send only the request entries that carry counts or stream con…
merceod 5404251
engine: direct properties for the forward-pass wrapper's hot attributes
merceod bcfe673
test: batched stream consumption report
merceod 75f5ea4
ci: list the send prep test
merceod c72d6af
kv commit: per-row constants hoisted, index offer only with a chain
merceod 3a31d8b
api_server: wake the streaming handlers once per message pass
merceod b861dcf
test: batched handler wake-up
merceod f5f5303
rust runtime: env-gated timing of send_outputs (body vs GIL reacquire)
merceod 6de4423
worker: MSTAR_SPEC_YIELD_HOLD_MS holds the decode loop's yield so arr…
merceod 2b422b9
docs: MSTAR_SPEC_YIELD_HOLD_MS
merceod 720587d
rust runtime: flat export of the dynamic loop iterations
merceod d47c22f
graph runtime: update_dynamic_loop_iters hook
merceod 880cdbe
rust runtime wrapper: refresh loop iterations from the flat export
merceod cb0b9ce
worker: refresh loop iterations through the runtime hook
merceod aa5c467
test: update_dynamic_loop_iters matches the per-rid form
merceod 92bed84
kv: batched prefix-chain extension off the stop check's host rows
merceod cb338d8
runner: batched chain extension; sweep resources that override the ho…
merceod da6df5b
resources: document when the chain hook is swept
merceod 61ea439
engine: extend prefix chains from the host rows
merceod f66cb9b
worker: hand the host rows to the prefix-chain extension
merceod a490f5b
test: batched prefix-chain extension
merceod b369e75
ci: list the prefix batch test
merceod 60c0ab2
engine: read the keyed walks leniently in prepare_inputs
merceod d35c66b
zmq communicator: non-blocking try_send
merceod 22278c5
rust runtime: send frames with the GIL held, release it only for a pe…
merceod 04a8f36
docs: MSTAR_RUST_SEND_TIMING
merceod 5db372f
data worker: sort the import block
merceod 5912708
rust runtime: the inline signal keys of a route input are optional
merceod 4406b62
test: row store tests skip without the columnar tensor store
merceod e3c54f5
engine: a reset pre-plan drops its kept step
merceod f6274b1
test: a partly dropped batch does not keep its pre-planned step
merceod eef4b85
kv: a snapshot of a stream reset since publishes the stream as it stands
merceod 57ebad9
test: publish from a snapshot after a reset
merceod b3e3b55
worker: drop the spec yield hold, note why two slots suffice, note th…
merceod 734972e
docs: MSTAR_SPEC_YIELD_HOLD_MS removed, MSTAR_DECODE_BUCKETS added
merceod 5b54fc7
api: APIServerMessageBody names the message body union
merceod 5838168
tensor store: ColumnarTensorInfo.add_row
merceod 37f9613
tensors: the row fast path fills the columns through add_row
merceod 3a814eb
qwen3_5: decode buckets from MSTAR_DECODE_BUCKETS, the model knob ove…
merceod cef920e
nvtx had _postprocess_batch_stops being closed at start of _postproce…
stephen-dwq e7d109f
revert N+2 speculation
stephen-dwq 7729610
CI fix + merge
stephen-dwq b759735
Merge remote-tracking branch 'origin/main' into perf/host-overhead
stephen-dwq File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: this environment variable is framed as a diagnostic for this host-overhead work (aka this PR); if you want to keep the switch in production, can you frame it as a being for generic performance debugging?