Enable CUDA graph capture for CUDA EP to improve decode throughput

[Akshay Sonawane (apsonawane)]

Enable CUDA graph capture for CUDA EP to improve decode throughput

Summary

Enable CUDA graph capture for the CUDA execution provider, which was previously blocked with a hard exception. This eliminates per-token kernel launch overhead during the decode phase, improving throughput by 20-45% across Qwen3.5 model sizes (0.8B through 9B).

Changes

1. src/config.cpp

Remove the exception that blocked enable_cuda_graph=1 for CUDA EP. Instead of throwing, return true to enable graph capture — matching the behavior already implemented for DML and NvTensorRtRtx providers.

2. src/cuda/session_options.cpp

Make CUDA's AppendExecutionProvider respect the disable_graph_capture parameter (previously ignored). When disable_graph_capture=true (used for vision and embedding sessions), force enable_cuda_graph to "0" in the provider options. This ensures CUDA graph is only enabled on the decoder session, preventing crashes from dynamic-shape vision/embedding models.

3. src/models/recurrent_state.cpp

Fix RecurrentState::Update() for CUDA graph compatibility. The original code used std::swap to alternate past/present buffers each decode step, which changes memory addresses and invalidates the captured CUDA graph. When graph capture is enabled, copy present→past in-place instead, preserving the stable pointers that the CUDA graph expects.

Performance Results (A100, Qwen3.5 INT4, 256 tokens, greedy)

Model	Before	After	Improvement
0.8B	166 t/s	244 t/s	+47%
2B	141 t/s	195 t/s	+38%
4B	94 t/s	120 t/s	+28%
9B	76 t/s	93 t/s	+22%

Testing

• Text-only generation: all 4 model sizes pass
• VLM with images: tested 10 different image sizes (224×224 to 800×656) on 0.8B — all pass without CUDA errors
• Vision and embedding sessions correctly run without graph capture
• RecurrentState (DeltaNet layers) pointer stability verified across multiple generations

[Copilot]

Pull request overview

Enables CUDA graph capture for the CUDA Execution Provider to reduce per-token kernel launch overhead during decoding, while adding safeguards for non-decoder sessions and ensuring recurrent-state updates keep stable device pointers needed by captured graphs.

Changes:

• Remove the hard failure blocking enable_cuda_graph=1 for the CUDA provider and treat it as an enabled graph-capture configuration.
• Make CUDA EP registration honor disable_graph_capture by overriding enable_cuda_graph to "0" for non-decoder sessions.
• Update RecurrentState::Update() to avoid pointer swapping under graph capture by copying present→past in-place to preserve stable addresses.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/config.cpp	Allows CUDA graph capture to be enabled via enable_cuda_graph=1 (no longer throws).
src/cuda/session_options.cpp	Applies disable_graph_capture by overriding CUDA provider options for non-decoder sessions.
src/models/recurrent_state.cpp	Ensures recurrent-state buffers remain pointer-stable under graph capture by copying instead of swapping.