Inference: Optimized triton kernels to extract mamba states in prefix caching - #5866

Merged

sidsingh-nvidia merged 5 commits into

NVIDIA:mainfrom

sidsingh-nvidia:siddharth/mamba-fused-intermediate-extraction

Jul 25, 2026

sidsingh-nvidia commented Jul 17, 2026

Contributor

I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

When your PR is ready, click Ready for Review.
An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
- Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.


          use optimized triton kernels to extract mamba states in prefix caching

f0f549e

sidsingh-nvidia requested review from a team as code owners

July 17, 2026 19:19

svcnvidia-nemo-ci marked this pull request as draft

July 17, 2026 19:19

github-actions Bot commented Jul 17, 2026

Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

Add the oncall reviewer (optional reviewer)
Add required review teams based on your changes

See the contribution guide for more details.

copy-pr-bot Bot commented Jul 17, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:20

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:23

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:23

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:23

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:23

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:24

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 19:33

Inactive


          lint

6ae9ded

sidsingh-nvidia marked this pull request as ready for review

July 17, 2026 20:09


          Merge branch 'main' into siddharth/mamba-fused-intermediate-extraction

cc78b0d

sidsingh-nvidia changed the title ~~Infernce: Optimized triton kernels to extract mamba states in prefix caching~~ Inference: Optimized triton kernels to extract mamba states in prefix caching

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:09

Inactive

svcnvidia-nemo-ci added the complexity: medium label

copy-pr-bot Bot temporarily deployed to test

July 17, 2026 20:10

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:13

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:13

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:13

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:13

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:13

Inactive

copy-pr-bot Bot temporarily deployed to public

July 17, 2026 20:23

Inactive

sidsingh-nvidia added a commit to sidsingh-nvidia/Megatron-LM that referenced this pull request


          Inference: Optimized triton kernels to extract mamba states in prefix…

1967c80

… caching

Cherry-picked from NVIDIA#5866.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Siddharth Singh <sidsingh@nvidia.com>

lmcafee-nvidia reviewed

View reviewed changes

megatron/core/ssm/ops/intermediate_extraction.py

lmcafee-nvidia approved these changes

View reviewed changes

shanmugamr1992 approved these changes

View reviewed changes

sidsingh-nvidia enabled auto-merge

July 23, 2026 18:57

sidsingh-nvidia disabled auto-merge

July 23, 2026 18:57

sidsingh-nvidia enabled auto-merge

July 23, 2026 18:57

svcnvidia-nemo-ci added Approved and removed Final Review labels

sidsingh-nvidia added this pull request to the merge queue

svcnvidia-nemo-ci commented Jul 23, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30040189728

svcnvidia-nemo-ci commented Jul 23, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30045775203

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks

sidsingh-nvidia added this pull request to the merge queue

svcnvidia-nemo-ci commented Jul 23, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30052417543

svcnvidia-nemo-ci commented Jul 23, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30052580976

svcnvidia-nemo-ci commented Jul 24, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30055539081

svcnvidia-nemo-ci commented Jul 24, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30071619240

svcnvidia-nemo-ci commented Jul 24, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30092278345

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks

sidsingh-nvidia added this pull request to the merge queue

svcnvidia-nemo-ci commented Jul 24, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30134351193

svcnvidia-nemo-ci commented Jul 25, 2026

Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30136327103

Merged via the queue into NVIDIA:main with commit 648bc01

92 of 95 checks passed

sidsingh-nvidia deleted the siddharth/mamba-fused-intermediate-extraction branch

July 25, 2026 04:07

Connor-XY mentioned this pull request

[feat] HybridStack grouped syntax + checkpoint compat + EP-overlap (2/4 of #4798) #4942

Open

3 tasks

shjwudp pushed a commit to shjwudp/Megatron-LM that referenced this pull request


          Inference: Optimized triton kernels to extract mamba states in prefix…

947ff5e

… caching (NVIDIA#5866)

terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request


          Inference: Optimized triton kernels to extract mamba states in prefix…

a13fb2a

… caching (NVIDIA#5866)

svcnvidia-nemo-ci pushed a commit to dimapihtar/Megatron-LM that referenced this pull request


          Inference: Optimized triton kernels to extract mamba states in prefix…

b83b05e

… caching (NVIDIA#5866)

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

This was referenced Aug 17, 2026

2026年7月 Megatron-LM 工作汇总 lhb8125/Megatron-LM#56

Open

Megatron-LM Monthly Activity Report — July 2026 #6607

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved complexity: medium