[Frontend] Omit absent fields from /inference/v1/generate stream chunks

[aoshen02]

Purpose

Follow-up to #53187. Python /inference/v1/generate stream chunks serialized absent optional fields as null, except for the two prompt-metadata keys that #53187 excluded per-field. The Rust frontend omits every absent field (skip_serializing_none), and the Python final usage chunk already uses exclude_none=True.

Serialize all Python data chunks with exclude_none=True so:

• both frontends emit identical stream JSON (absent usage, logprobs, finish_reason, prompt_token_ids, mm_placeholders are omitted, not null);
• the per-field exclude special case from [Frontend] Return prompt metadata from /inference/v1/generate #53187 goes away;
• Python data chunks and the final usage chunk follow the same rule.

Non-streaming responses are unchanged on both sides. Clients that read chunk["finish_reason"] on non-final Python chunks must use .get(); the OpenAI-style clients and prime-rl's generate client already do.

Why this is not a duplicate

Searched open PRs for exclude_none, skip_serializing_none, generate stream and token_in_token_out. #47933 changes abort finish_reason on this path but not serialization; no overlap. Independent of #57265 (empty-chunk skip rule).

Test Plan

• Python: new test_stream_chunks_omit_absent_fields; existing key-absence assertions for the metadata fields unchanged.
• Rust: new stream_chunk_omits_absent_fields unit test locking the existing skip_serializing_none contract.

Test Result

.venv/bin/python -m pytest tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py \
  tests/entrypoints/scale_out/token_in_token_out/test_protocol.py -q
# 33 passed
cargo test --manifest-path rust/Cargo.toml -p vllm-server --no-default-features
# 413 passed
cargo clippy --manifest-path rust/Cargo.toml -p vllm-server --tests --no-default-features -- -D warnings
# clean; cargo fmt --check clean

Model evaluation is not applicable: wire-format-only change, generated tokens are unchanged.

Developed with AI assistance (Claude Code); the human submitter has reviewed every changed line.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

/ci run

[github-actions]

❌ This PR is 16 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89535 for commit 306ff57636af.

[aoshen02]

Sorry cyrus, it's not ready yet.