Repository navigation

[Feat] beam: Add native Beam serving support - #44747

Draft

Jiminator wants to merge 347 commits into

mainfrom

beam

Draft

[Feat] beam: Add native Beam serving support#44747
Jiminator wants to merge 347 commits into
mainfrom
beam

This pull request is big! We're only showing the most recent 250 commits

Commits on Oct 3, 2026
- Budget Beam collective CI for cold compilation on H200
  thibautbar
  committed
  5d24e5d
- Merge pull request #63 from radixark/perf/beam-qk-prologue
  Jiminator
  authored
  bbc426a
- cleanup
  AymenKallala
  committed
  743bd1d
- Merge dev into aymen/beam-fp8
  AymenKallala
  committed
  f9869f5
- Merge pull request #61 from radixark/iaroslav/oss-1429-beam-reference
  Jiminator
  authored
  74545ea
- Merge pull request #58 from radixark/iaroslav/oss-1042-beam-container
  Jiminator
  authored
  672ff55
- Merge pull request #62 from radixark/iaroslav/beam-parallel-request-cleanup
  Jiminator
  authored
  9ef403d
Commits on Oct 4, 2026
- Optimize Beam FP32 router for small decode batches
  thibautbar
  committed
  dca9f43
- Keep larger Beam decode batches on the cuBLAS router
  thibautbar
  committed
  d46960a
- Limit router dispatch to the validated BF16 MoE runner
  thibautbar
  committed
  ecbb996
Commits on Oct 5, 2026
- Merge dev and retain fused QK prologue with packed projections
  thibautbar
  committed
  1fa3efe
- fix(beam): preserve separate attention projections with LoRA
  thibautbar
  committed
  214d056
- perf(beam): read strided attention gates without copying
  thibautbar
  committed
  d91ee82
- Remove added SWA prefix reuse tests
  Jiminator
  committed
  ed44674
- Remove added PD logprob tests and shorten handoff comment
  Jiminator
  committed
  80d213e
- Merge pull request #52 from radixark/neil/beam-pd-swa-page-retention-20260928
  Jiminator
  authored
  6e2199c
- Merge pull request #53 from radixark/neil/beam-pd-stream-logprobs-20260928
  Jiminator
  authored
  03c88f1
- Extend tiny GEMM to FP32 weights for router comparison
  thibautbar
  committed
  732512e
- Merge dev into beam-allreduce-norm
  thibautbar
  committed
  3124a24
- Remove standalone Beam all-reduce benchmark from PR
  thibautbar
  committed
  48bf862
- Remove manual packed projection benchmark
  thibautbar
  committed
  8d2ef16
- Add a reproducible three-way FP32 router comparison
  thibautbar
  committed
  51f86a2
- Reuse tiny GEMM instead of maintaining a separate FP32 router kernel
  thibautbar
  committed
  08c04de
- Adapt Beam prologue numerical test to packed projections
  thibautbar
  committed
  f7bcb70
- Extend Beam BF16 collective fusion to attention and measured TP shapes
  thibautbar
  committed
  0233232
- Limit Hopper TP8 attention fusion to the measured model crossover
  thibautbar
  committed
  f25a177
- Make Beam's default response template match the chat template exactly
  
  yonigozlan
  and
  cursoragent
  authored
  c90ff61
- Share setup across Beam's template framing tests
  
  yonigozlan
  and
  cursoragent
  authored
  13dac5c
Commits on Oct 6, 2026
- Fix Beam numerical test projection mock for deferred reduction
  thibautbar
  committed
  998e18f
- Clarify Beam fusion dispatch and communicator capacity
  thibautbar
  committed
  4b70ae3
- Keep regions that output cut before the next message start
  
  yonigozlan
  and
  cursoragent
  committed
  eef66ba
- Preserve unquantized expert dispatch during compilation
  Jiminator
  committed
  9689f85
- Trim comments and added test coverage
  Jiminator
  committed
  6462aa1
- Merge pull request #47 from radixark/aymen/beam-fp8
  Jiminator
  authored
  5159890
Commits on Oct 7, 2026
- Merge pull request #75 from radixark/kai/fix-trtllm-mha-unified-graph-seqlens
  Qiaolin-Yu
  authored
  a92b882
- Merge pull request #72 from radixark/yoni/beam-strict-response-template
  Jiminator
  authored
  01d2452
- Constrain every Beam turn to the response template
  
  authored and
  Jiminator
  committed
  796b993
- Drop comments and churn the turn grammar does not need
  
  yonigozlan
  authored and
  Jiminator
  committed
  62d2ffd
- Require visible reasoning when the template grammar requires reasoning
  
  authored and
  Jiminator
  committed
  b0f94e9
- Check the schema shape of strict tools only and ignore `$schema`
  
  authored and
  Jiminator
  committed
  9b058a6
- Let strict tools the template cannot hold take any arguments
  
  authored and
  Jiminator
  committed
  d6cad35
- Hold strict string formats and patterns in native tool calls
  
  authored and
  Jiminator
  committed
  fc114d9
- Take the turn grammar only with the xgrammar backend
  
  authored and
  Jiminator
  committed
  f7645f1
- Fix whitespace escaping in response-template reasoning grammar
  Jiminator
  committed
  1bfca85
- Remove newline closer regression test
  Jiminator
  committed
  07aa12f
- Merge pull request #73 from radixark/yoni/default-turn-grammar
  Jiminator
  authored
  12fe984
- Merge pull request #45 from radixark/iaroslav/oss-1389-beam-moe-tuner
  Jiminator
  authored
  1de5ea6
- Derive response-template grammars only for Beam
  
  yonigozlan
  authored and
  Jiminator
  committed
  6133aec
- Allow at most a newline between template tool-call parameters
  
  yonigozlan
  authored and
  Jiminator
  committed
  b3256b5
- Let any first character that starts no closer open required reasoning
  
  yonigozlan
  authored and
  Jiminator
  committed
  bf21d28
- Keep closer starts out of the first required reasoning character
  Jiminator
  committed
  436ecf3
- Derive Beam's JSON answers with the tool-call grammar
  
  yonigozlan
  and
  cursoragent
  committed
  92f81c9
- Merge pull request #96 from radixark/yoni/beam-only-template-grammar
  Jiminator
  authored
  d86d110
- Let automatic Beam calls name tools that were not declared
  
  yonigozlan
  authored and
  Jiminator
  committed
  0658e34
- Admit undeclared Beam tool names only where they are served and allowed
  Jiminator
  committed
  2207e13
- Merge pull request #97 from radixark/yoni/beam-undeclared-tool-calls
  Jiminator
  authored
  72f23d9
- fix(beam): scope MLP reduction before DP reduce-scatter
  Qiaolin-Yu
  committed
  da09ad5
- Merge dev into Beam DP attention and adapt fused MLP path
  Qiaolin-Yu
  committed
  faca5ed
- Merge pull request #59 from radixark/feat/beam-dp-attention
  Qiaolin-Yu
  authored
  cc9173c
Commits on Oct 8, 2026
- Merge commit 'refs/review/20261008-78' into bbuf/beam-gb300-8k1k-20261008
  BBuf
  committed
  a4d6902
- Merge commit 'refs/review/20261008-82' into bbuf/beam-gb300-8k1k-20261008
  BBuf
  committed
  1810fd5
- Merge commit 'refs/review/20261008-84' into bbuf/beam-gb300-8k1k-20261008
  BBuf
  committed
  770af64
- fix(nccl): disable graph buffer registration for pausable graph pools (#40648)
  
  authored and
  zhihengy
  committed
  0d7899b
- perf(beam): tune GB300 BF16 MoE and add opt-in decode GEMV
  
  BBuf
  and
  thibautbar
  committed
  0e15e90
- [Messages] Preserve response-template reasoning history
  neilkg
  committed
  03f0c43
- [Messages] Forward strict tools and parallel-call controls
  neilkg
  committed
  18bbeed
- [Messages] Apply JSON output schemas to constrained decoding
  neilkg
  committed
  c152da8
- Remove redundant pausable graph helper docstring
  Jimmy Shong
  committed
  3fb861d
- [responses] preserve Beam output order (OSS-1001)
  neilkg
  authored
  4be38a7
- [responses] align Beam streaming order and cutoff output (OSS-1006)
  neilkg
  authored
  23dde52
- Support Responses namespaces through shared tool adapters
  neilkg
  authored
  d514e81
- Fix Responses namespace replay and Harmony custom tools
  Jiminator
  authored
  0db0b7e
- Adapt Beam bash calls to client-executed Responses shell
  
  neilkg
  authored and
  Jiminator
  committed
  29cdb2f
- Remove fixture provenance README
  
  neilkg
  authored and
  Jiminator
  committed
  c5cc7d6
- Simplify shell tests and use bare native tool names
  
  neilkg
  authored and
  Jiminator
  committed
  d207394
- Keep Responses stream events typed until serialization
  
  neilkg
  authored and
  Jiminator
  committed
  5120420
- Preserve adapted tool choices outside the request schema
  
  neilkg
  authored and
  Jiminator
  committed
  13bece9
- Consolidate Responses output item types
  
  neilkg
  authored and
  Jiminator
  committed
  96d2461
- Upgrade OpenAI SDK to 2.8.0 for Responses shell types
  
  neilkg
  authored and
  Jiminator
  committed
  29c9eab
- Reuse SDK shell schemas and validate replay in the adapter
  
  neilkg
  authored and
  Jiminator
  committed
  fa45fcb
- Restore inline Responses output unions
  
  neilkg
  authored and
  Jiminator
  committed
  007966a
- Remove special-case legacy local_shell rejection
  
  neilkg
  authored and
  Jiminator
  committed
  b391330
- Align shell test output with current Beam parameter format
  
  neilkg
  authored and
  Jiminator
  committed
  81e232f
- Fix Responses custom tools and registered test entry points
  Jiminator
  committed
  b569cae
- Keep Responses tool rendering internal
  Jiminator
  committed
  9f9bfc0
- Merge pull request #89 from radixark/neil/responses-beam-shell
  Jiminator
  authored
  c78cf74
- Streamed LoRA updates, upsert and deferred publication (#27268, #31759, #30913, #30912, #33299, #36384, #38165)
  
  gongyisheng
  authored and
  zhihengy
  committed
  4d0fc4b
- Adapt streamed-LoRA unit tests to this base
  
  zhihengy
  and
  claude
  committed
  4a0510b
- fix(nccl): disable graph buffer registration whenever a decode graph can capture a collective
  
  zhihengy
  and
  claude
  committed
  035ee0f
- Support CUDA DFlash DP attention with deferred verify metadata
  Qiaolin-Yu
  committed
  19f6506
- test(nccl): cover the generalized graph-register guard
  
  zhihengy
  and
  claude
  committed
  32c5218
- Merge pull request #108 from radixark/codex/beam-dflash2-dp-attention
  Qiaolin-Yu
  authored
  4becab2
- Merge the tool-controls fix into the JSON-output branch
  Jiminator
  committed
  067710a
- [Messages] Keep inline thinking for templates without reasoning_content
  Jiminator
  committed
  4293087
- Merge the reasoning-history fix into the tool-controls branch
  Jiminator
  committed
  d7747e7
- [Messages] Forward output_config to count_tokens
  Jiminator
  committed
  74578f0
- [Tests] Document the Messages reasoning-history regression
  Jiminator
  committed
  f20ad88
- [Messages] Return 400 for requests the Chat pipeline rejects
  Jiminator
  committed
  32ad281
- [Tests] Document the Messages JSON-output regressions
  Jiminator
  committed
  73daab7
- Simplify reasoning-history regression coverage
  Jiminator
  committed
  c465f0e
- Merge reasoning and tool test cleanup from parent PR
  Jiminator
  committed
  483a9a8
- Trim tool-control tests and redundant comments
  Jiminator
  committed
  80247fe
- Merge reasoning and tool test cleanup from parent PR
  Jiminator
  committed
  a4a5216
- Isolate invalid output-format coverage
  Jiminator
  committed
  e915e7b
Commits on Oct 9, 2026
Commits on Oct 10, 2026
- Merge pull request #115 from radixark/jimmy/reopen-pr-101
  Jiminator
  authored
  188210f
- Merge main into dev
  Jiminator
  committed
  6ba64c0
- Revert "Merge main into dev"
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  90c70a5
- Restore main integration for review
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  0ea7852
Commits on Oct 11, 2026
- beam: support the FlashInfer MegaMOE all-to-all under attention DP
  
  nujoug
  and
  claude
  committed
  9c50e85
Commits on Oct 12, 2026
- Let Beam parameter values end before the next opener
  
  yonigozlan
  authored and
  Jimmy Shong
  committed
  5eacd3a
- Let automatic Beam calls name undeclared tools beside strict tools
  
  yonigozlan
  authored and
  Jimmy Shong
  committed
  902ebf2
- Hold Beam's declared arguments at every strict level
  
  yonigozlan
  authored and
  Jimmy Shong
  committed
  bb3e9e5
- Factor optional parameter grammar prefixes
  
  Jiminator
  authored and
  Jimmy Shong
  committed
  72ffe45
- Preserve tool arguments held by native response grammars
  
  Jiminator
  authored and
  Jimmy Shong
  committed
  8e22b89
- beam: faster TP all-reduce + norm boundaries on GB300
  
  nujoug
  and
  claude
  committed
  e59a3f4
- [Beam] Pass parallel_group="attn_tp" to attention linear layers
  
  kaiwu-reflection
  authored and
  Jimmy Shong
  committed
  15f4d2d
- [Beam] Fix DP-attention hang: select embedding group by attn_dp_enabled
  
  kaiwu-reflection
  authored and
  Jimmy Shong
  committed
  310a276
- [Beam] Update CPU fixtures for parallel-group API
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  f825639
- Fix explicit chat-formatting options in tokenization
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  2e7ae1f
- beam: format-independent attention fusion, FlashInfer naming, one MLP path
  
  nujoug
  and
  claude
  committed
  1f90e16
- beam: fuse the summed MLP output like attention's
  
  nujoug
  and
  claude
  committed
  a256e14
- Merge pull request #112 from radixark/yoni/beam-lenient-parameter-end
  Jiminator
  authored
  558932b
- Merge pull request #117 from radixark/fix/beam-parallel-group
  Jiminator
  authored
  63cb445
- beam: name boundary all-reduce replacements by boundary and transport
  
  nujoug
  and
  claude
  committed
  cfec8e1
- beam: skip FlashInfer all-reduce setup during meta-device construction
  
  nujoug
  and
  claude
  committed
  f60df76
- beam: one boundary helper, lazy FlashInfer setup, stricter checks
  
  nujoug
  and
  claude
  committed
  59d0c8a
- beam: test boundary ownership, FlashInfer setup lifecycle and graph replay
  
  nujoug
  and
  claude
  committed
  4a53db5
Commits on Oct 13, 2026
- Merge pull request #100 from radixark/zhihengy/nccl-graph-register-pausable
  Jiminator
  authored
  ae82fea
- Merge pull request #107 from radixark/zhihengy/lora-stream-update
  Jiminator
  authored
  c0418fa
- Trim tokenization formatting comments and redundant tests
  Jiminator
  committed
  65ae4cc
- Add /v1/parse to turn output token IDs into an assistant message
  Jiminator
  committed
  ec1cfc0
- Render Beam with SGLang's template and read compact tool parameters
  Jiminator
  committed
  b0025af
- Relax only Beam's own parameter rule when parsing
  Jiminator
  committed
  b720f9e
- Name the Beam template's source by its hash
  Jiminator
  committed
  c47df3c
- Trim the PR to what its behavior needs
  Jiminator
  committed
  89c1116
- Reuse the tool-call detector and test literal omission
  Jiminator
  committed
  d19307b
- Remove the /v1/parse endpoint from this PR
  Jiminator
  committed
  20834ae
- beam: honor --enforce-disable-flashinfer-allreduce-fusion in the FlashInfer band
  
  nujoug
  and
  claude
  committed
  50bda02
- Merge dev c0418fa4 (PR100, PR107) into the main integration
  Jiminator
  committed
  c4b7605
- Merge main 5bcaf0f8 into the main integration
  Jiminator
  committed
  d66cd32
- Fix parallel-sampling request states and LoRARef server-info fixture
  Jiminator
  committed
  abd0560
- Merge pull request #116 from radixark/mloh/beam-dp4-prefill-megamoe
  Qiaolin-Yu
  authored
  9ac1c65
- flashinfer_megamoe: opt-in decode-only workspace (experiment)
  
  authored and
  Qiaolin-Yu
  committed
  a7a8a08
- flashinfer_megamoe: log each workspace capacity as it is created
  
  authored and
  Qiaolin-Yu
  committed
  86d613c
- Merge dev 9ac1c65b (PR116) into the main integration
  Jiminator
  committed
  fc695c0
- Adapt five unit tests to the integrated interfaces
  Jiminator
  committed
  4c22197
- Complete the abort-prefix scheduler fake for the decode waiting path
  Jiminator
  committed
  4bccb7d
Commits on Oct 14, 2026
- Fix TRT-LLM metadata fixtures and Docker gRPC tools pin
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  caf87b0
- Pin a real Dynamo release in the Docker dependency step
  
  authored and
  Jimmy Shong
  committed
  f2ca3d0
- Constrain the gateway wheel reinstall to the frozen pins
  
  authored and
  Jimmy Shong
  committed
  27240b4
- Merge pull request #118 from radixark/integrate/main-into-dev-20261010
  Jiminator
  authored
  b7f6b54
- Merge pull request #123 from radixark/mloh/beam-ar-norm-fusion
  Jiminator
  authored
  d4be2f8
- Accept whole-number floats for integer tool arguments
  
  yonigozlan
  authored and
  Jimmy Shong
  committed
  54ddc15
- Backport chat parsing event fixes from upstream #40478
  Jiminator
  committed
  e377bdf
- Merge dev d4be2f8ca0 into mloh/megamoe-split-buffer
  Jiminator
  committed
  a4295c7
- Remove merge-resolution regression test
  Jiminator
  committed
  7b65291
- Merge pull request #127 from radixark/mloh/megamoe-split-buffer
  Qiaolin-Yu
  authored
  bbb84c7
- Merge pull request #132 from radixark/codex/cherry-pick-43941-integer-tool-args
  Jiminator
  authored
  cde0b6e
- perf(kernels): add a BF16x3 tensor-core GEMM for FP32 router gates
  
  authored and
  jinzex
  committed
  934e6a3
- perf(kernels): run the router GEMV on CUDA for FP32 gates
  
  authored and
  jinzex
  committed
  3f911a9
- perf(beam): compute router logits with the GEMV and BF16x3 GEMM
  
  authored and
  jinzex
  committed
  5b7802c
- test(beam): set the fast-router flag in the MegaMoE layer test
  
  authored and
  jinzex
  committed
  78dc9c3
- test(beam): widen the long-decode logprob tolerance in the cache test
  
  authored and
  jinzex
  committed
  401e9d6
- Update python/sglang/srt/models/beam.py
  
  fgbelidji
  and
  jinzex
  committed
  bdb8378
- Complete cuBLAS router GEMM rename
  jinzex
  committed
  8926dfe
- Keep non-BF16 Beam router inputs on the linear fallback
  jinzex
  committed
  c40b27d
- Add prepared-weight cuTile BF16x3 router GEMM
  jinzex
  committed
  050c0b0
- Integrate cuTile router with refreshable prepared weights
  jinzex
  committed
  c3802a9
- Reduce medium-batch cuTile router latency
  jinzex
  committed
  302977b
- Report FP64 error alongside router GEMM latency
  jinzex
  committed
  a8b933f
- Add Hopper launch configs for the cuTile BF16x3 router GEMM
  jinzex
  committed
  8b7c802
- Use cuTile as the default Beam router backend
  jinzex
  committed
  c29fcea
- Simplify router dispatch tests and clarify cuTile naming
  jinzex
  committed
  0730533
- Clarify cuTile router kernel names and documentation
  jinzex
  committed
  693504f
- Clarify Beam router dispatch order
  jinzex
  committed
  b4fdeaf
- Align cuTile router attribute with kernel name
  jinzex
  committed
  a8c468e
- Update router dispatch test for cuBLAS function name
  jinzex
  committed
  733e5b4
- Fix upstream PDL synchronization in cuTile router GEMM
  jinzex
  committed
  6f51e4a
- Use the shared Beam router fallback test
  jinzex
  committed
  0ea0fca
- Merge pull request #126 from radixark/jinzex/cutile-bf16x3
  Jiminator
  authored
  afb239c
- [unified-memory] Lift the trtllm_mha decode-graph refusal
  
  authored and
  Jimmy Shong
  committed
  1c1a5bf
- [unified-memory] Zero MHA page envelopes on hand-out
  
  authored and
  Jimmy Shong
  committed
  1e0df66
- [unified-memory] Require packed page tables on trtllm_mha's graphs-off path
  
  authored and
  Jimmy Shong
  committed
  94a2e0d
- [unified-memory] Keep the slot-0 sink out of the debug poison fill
  
  authored and
  Jimmy Shong
  committed
  95c3e76
- [trtllm_mha] Keep fmha_v2 prefill out of XQA's workspace semaphores
  
  authored and
  Jimmy Shong
  committed
  4ab710a
- [trtllm_mha] Copy only the batch's pages for the fmha_v2 prefill
  
  authored and
  Jimmy Shong
  committed
  54f3979
- feat(unified-memory): admit trtllm_mha on the translated MHA rails
  
  authored and
  Jimmy Shong
  committed
  805a03c
- [SWA] Keep a whole window behind the page an insert can end on
  
  authored and
  Jimmy Shong
  committed
  9e19b26
- [unified-memory] Stop allocation eviction on the joint FULL/SWA fit
  
  authored and
  Jimmy Shong
  committed
  f15a3d6
- [unified-memory] Free SWA segments without a host sync
  
  authored and
  Jimmy Shong
  committed
  3ca2455
- Merge branch 'dev' into fix/beam-tokenize-formatting
  Jiminator
  committed
  76e1a69
- Drop the bundled Beam chat template
  Jiminator
  committed
  7b4a06d
- Extend mixed-batch decode splitting to FP8 KV
  jinzex
  committed
  be757e8
- Measure mixed attention with cold L2 between graph replays
  jinzex
  committed
  46202cc
- [Test] Set finite capacity in allocation eviction fixture
  Jiminator
  committed
  472393e
- Merge pull request #121 from radixark/fix/beam-tokenize-formatting
  Jiminator
  authored
  8b7a307
- Merge pull request #125 from radixark/jinzex/trtllm-mha-mixed-decode
  Jiminator
  authored
  7c2ed55
- Merge pull request #133 from radixark/codex/cherry-pick-unified-memory-swa-20261014
  Jiminator
  authored
  8f94384
Commits on Oct 15, 2026
- Allow DFlash attention DP with FlashInfer MegaMoE
  
  Jimmy Shong
  authored and
  Jimmy Shong
  committed
  1f80bbd
- Merge pull request #135 from radixark/codex/dflash-flashinfer-megamoe-20261014
  Jiminator
  authored
  f747367

Commits

Commits on Oct 3, 2026

Commits on Oct 4, 2026

Commits on Oct 5, 2026

Commits on Oct 6, 2026

Commits on Oct 7, 2026

Commits on Oct 8, 2026

Commits on Oct 9, 2026