Skip to content

Add caller-owned workspace sizing helper - #5

Closed
lesj0610 wants to merge 9 commits into
mainfrom
lesj/flashinfer-workspace-sizing-helper-20260626
Closed

lesj0610 wants to merge 9 commits into
mainfrom
lesj/flashinfer-workspace-sizing-helper-20260626

Conversation

@lesj0610

@lesj0610 lesj0610 commented Jun 26, 2026

Copy link
Copy Markdown
Owner

📌 Description

Add a caller-owned workspace sizing helper so integrations can allocate the FlashInfer float workspace to the actual planned requirement instead of always reserving the default workspace size.

This PR:

  • Adds count-only support to AlignedAllocator so plan allocation offsets can be reused for sizing without touching workspace memory.
  • Adds batch decode and batch prefill workspace_size FFI functions that return required float and int workspace bytes.
  • Exposes workspace_size on generated and JIT batch decode/prefill Python modules, with None for TRTLLM-gen where the helper is not available.
  • Adds CUDA tests that verify decode and FA2 prefill can plan with buffers sized exactly from workspace_size.
  • Leaves plan_info layout and runtime run paths unchanged.

🔍 Related Issues

N/A

🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

✅ Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit (or used your preferred method).
  • I have installed the hooks with pre-commit install.
  • I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

If you are unsure about how to set up pre-commit, see the pre-commit documentation.

🧪 Tests

  • Tests have been added or updated as needed.
  • All tests are passing (unittest, etc.).

Validation performed:

  • pre-commit run --all-files
  • FLASHINFER_DISABLE_VERSION_CHECK=1 MAX_JOBS=4 FLASHINFER_NVCC_THREADS=2 python -m pytest tests/attention/test_workspace_size.py -q -s -x --tb=short

Not yet verified:

  • Full CUDA/JIT regression matrix across all dtype, mask, sliding-window, positional-encoding, and backend combinations.

Reviewer Notes

This PR is independent against main.

Summary by CodeRabbit

  • New Features
    • Added workspace-size reporting for paged-KV batched decode and KV-cache prefill.
    • Planning modules/wrappers now expose workspace_size to let callers size buffers before running.
  • Bug Fixes
    • Improved workspace sizing/allocation behavior for more accurate buffer requirements.
    • Clarified that required float workspace buffers must be 16-byte aligned.
  • Tests
    • Added CUDA-only tests validating decode and prefill workspace sizing and planning with exact-sized buffers.

@lesj0610
lesj0610 force-pushed the lesj/nvfp4-slot-mapping-kv-cache-20260624 branch from 247c465 to ba80aff Compare June 26, 2026 10:48
@lesj0610
lesj0610 force-pushed the lesj/flashinfer-workspace-sizing-helper-20260626 branch from bba4a75 to 9038328 Compare June 26, 2026 10:48
@lesj0610
lesj0610 changed the base branch from lesj/nvfp4-slot-mapping-kv-cache-20260624 to main June 26, 2026 10:48
@lesj0610
lesj0610 marked this pull request as ready for review June 26, 2026 10:51
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@lesj0610
lesj0610 marked this pull request as draft June 26, 2026 10:52
@lesj0610
lesj0610 marked this pull request as ready for review June 26, 2026 11:41
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@lesj0610
lesj0610 force-pushed the lesj/flashinfer-workspace-sizing-helper-20260626 branch from 51281b2 to f83d21a Compare June 29, 2026 03:47
@lesj0610

lesj0610 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Upstream merged.

@lesj0610 lesj0610 closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant