[V1][TPU] TPU-optimized top-p implementation (avoids scattering). by hyeygit · Pull Request #15736 · vllm-project/vllm

[V1][TPU] Add a TPU-optimized top-p algorithm.

root

authored and

hyeygit

committed

Bugfix: Make sure uniform sampling happens on XLA not CPU.

hyeygit

committed

Update tests.

hyeygit

committed

Change tie-breaking behavior

hyeygit

committed

Update tests to perform assertion on CPU.

hyeygit

committed

Update tests to use explicit .cpu() calls.

hyeygit

committed

Fix merge mistake

hyeygit

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[V1][TPU] TPU-optimized top-p implementation (avoids scattering).#15736

[V1][TPU] TPU-optimized top-p implementation (avoids scattering).#15736
robertgshaw2-redhat merged 7 commits intovllm-project:mainfrom
hyeygit:tpu_topp

Commits on Apr 2, 2025

[V1][TPU] Add a TPU-optimized top-p algorithm.

Bugfix: Make sure uniform sampling happens on XLA not CPU.

Update tests.

Change tie-breaking behavior

Update tests to perform assertion on CPU.

Update tests to use explicit .cpu() calls.

Fix merge mistake

Uh oh!

[V1][TPU] TPU-optimized top-p implementation (avoids scattering).#15736robertgshaw2-redhat merged 7 commits intovllm-project:mainvllm-project/vllm:mainfrom hyeygit:tpu_topphyeygit/vllm:tpu_toppCopy head branch name to clipboard

Commits

Commits on Apr 2, 2025

[V1][TPU] Add a TPU-optimized top-p algorithm.

Bugfix: Make sure uniform sampling happens on XLA not CPU.

Update tests.

Change tie-breaking behavior

Update tests to perform assertion on CPU.

Update tests to use explicit .cpu() calls.

Fix merge mistake

[V1][TPU] TPU-optimized top-p implementation (avoids scattering).#15736
robertgshaw2-redhat merged 7 commits intovllm-project:mainfrom
hyeygit:tpu_topp