[Meta] Unshift eagle prefill support with potential KV sharing from base to draft

[morgendave]

Purpose

Eagle Draft Prefill Unshift the tokens + KV sharing support, generally removes the condition of shifting in prefill which is a better case for calculating prefix caching and supports MM

Pending work needs to be followed
Under auto batch size for lm_eval, MM might crash due to some scheduling issues causing extra/unexpected placeholder tokens leaked into input tokens. Roger might fix this using mm positions

Test Plan

Local test with MT Bench
Unshift only - TEST A

 CUDA_VISIBLE_DEVICES=4,5,6,7 VLLM_USE_V1=1 python examples/offline_inference/spec_decode.py  --num_spec_tokens 3 --num_prompts 80 --method eagle --model_dir /home/zhiweiz/local/models/scout_base_HF_20250605_201140 --eagle_dir /home/zhiweiz/local/models/scout_draft_HF_20250605_202942 --tp 4 --dataset-name hf --dataset-path philschmid/mt-bench --no-prefill-token-shift

shift original - Baseline

 CUDA_VISIBLE_DEVICES=4,5,6,7 VLLM_USE_V1=1 python examples/offline_inference/spec_decode.py  --num_spec_tokens 3 --num_prompts 80 --method eagle --model_dir /home/zhiweiz/local/models/scout_base_HF_20250605_201140 --eagle_dir /home/zhiweiz/local/models/scout_draft_HF_20250605_202942 --tp 4 --dataset-name hf --dataset-path philschmid/mt-bench

Test Result

Baseline

total_num_output_tokens: 17757
num_drafts: 6348
num_draft_tokens: 19044
num_accepted_tokens: 11413
mean acceptance length: 2.80
--------------------------------------------------
acceptance at token 0: 0.79
acceptance at token 1: 0.58
acceptance at token 2: 0.42

TEST A

total_num_output_tokens: 17697
num_drafts: 6375
num_draft_tokens: 19125
num_accepted_tokens: 11329
mean acceptance length: 2.78
--------------------------------------------------
acceptance at token 0: 0.78
acceptance at token 1: 0.58
acceptance at token 2: 0.42

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀