[BugFix] Fix spec decode + structured outputs + preemption edge case

[njhill]

Fix an edge case that can be triggered when using spec decode with structured outputs.

There is a sequence of preemption and drafting being skipped that can result in the scheduler's request.spec_token_ids being stale which can then fail bitmask generation because they will be out of sync with the grammar.

When triggered, vLLM crashes with an error like this:

(EngineCore_DP0 pid=1549249)   File "/home/nickhill/workspace/vllm2/vllm/vllm/v1/engine/core.py", line 920, in _process_engine_step
(EngineCore_DP0 pid=1549249)     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=1549249)                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1549249)   File "/home/nickhill/workspace/vllm2/vllm/vllm/v1/engine/core.py", line 344, in step
(EngineCore_DP0 pid=1549249)     grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
(EngineCore_DP0 pid=1549249)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1549249)   File "/home/nickhill/workspace/vllm2/vllm/vllm/v1/core/sched/scheduler.py", line 1022, in get_grammar_bitmask
(EngineCore_DP0 pid=1549249)     bitmask = self.structured_output_manager.grammar_bitmask(
(EngineCore_DP0 pid=1549249)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1549249)   File "/home/nickhill/workspace/vllm2/vllm/vllm/v1/structured_output/__init__.py", line 277, in grammar_bitmask
(EngineCore_DP0 pid=1549249)     assert accepted, (token, req_id, scheduled_spec_decode_tokens)
(EngineCore_DP0 pid=1549249)            ^^^^^^^^
(EngineCore_DP0 pid=1549249) AssertionError: (220, '80', {'80': [330, 220]})

This PR fixes that case by ensuring that request.spec_token_ids is cleared when drafting is skipped for a given step, rather than potentially leaving set to a prior step's draft tokens.

It will covered by tests once we extend test_async_scheduling.py test matrix in #29821.

Note that this bug/fix doesn't actually apply to the async scheduling case.

[gemini-code-assist]

Code Review

This pull request addresses an edge case in speculative decoding when used with structured outputs. The issue arises when drafting is skipped, which could leave stale spec_token_ids in the scheduler, leading to failures in bitmask generation for structured outputs. The fix correctly modifies take_draft_token_ids to return an empty DraftTokenIds object when no new draft tokens are proposed, ensuring the scheduler's state is properly cleared. The change is well-targeted and effectively resolves the bug. The code looks good.