fix: Use streaming detokenizer for UTF-8-safe incremental decode

[janhilgard]

Summary

• Replace per-token tokenizer.decode([token]) with NaiveStreamingDetokenizer (or BPEStreamingDetokenizer when available) for UTF-8-safe incremental decoding during SSE streaming
• Fix corrupted multi-byte characters (e.g. Czech ď → ��, emoji → ���) in streaming responses
• Both LLM scheduler and MLLM scheduler are fixed

Problem

When a tokenizer produces byte-level tokens, multi-byte UTF-8 characters (like ď = 0xC4 0x8F) are split across two tokens. Decoding each byte-token individually via tokenizer.decode([single_token]) produces invalid UTF-8 (replacement character U+FFFD), which then gets sent to clients in SSE chunks:

chunk 1: {"content":" \ufffd"}     ← 0xC4 decoded alone → invalid
chunk 2: {"content":"\ufffdáb"}    ← 0x8F decoded alone → invalid

The word "ďábelské" appears as "��ábelské" in the client.

Fix

Use mlx_lm's streaming detokenizer which buffers byte-tokens until a complete UTF-8 character boundary is reached:

# Before (broken):
new_text = tokenizer.decode([response.token])

# After (UTF-8 safe):
detok = self._get_detokenizer(request_id)
detok.add_token(response.token)
new_text = detok.last_segment

A per-request detokenizer pool is maintained and cleaned up when requests finish.

Changes

File	Change
vllm_mlx/scheduler.py	Add _detokenizer_pool, _get_detokenizer(), _cleanup_detokenizer(); use streaming detokenizer in _process_batch_responses()
vllm_mlx/mllm_scheduler.py	Same fix for the MLLM code path

Test plan

• uvx black --check vllm_mlx/ passes
• Existing tests/test_streaming_detokenizer.py validates the detokenizer logic
• Stream a response containing multi-byte characters (e.g. Příliš žluťoučký kůň úpěl ďábelské ódy) and verify no � in SSE chunks
• Non-streaming responses still work correctly (uses detok.finalize() + detok.text for full output)

🤖 Generated with Claude Code

[waybarrios]

Pushed a cleanup commit to fix detokenizer pool leaks in all exit paths:

scheduler.py:

• _do_abort_request now calls _cleanup_detokenizer
• _recover_from_generation_error clears the pool after aborting all requests
• reset() clears the pool

mllm_scheduler.py:

• abort_request now pops from _detokenizer_pool
• reset() clears the pool

Also closed #195 as duplicate since this PR covers both schedulers.