[Bugfix] Lazy tokenizer init to prevent semaphore leak in multiprocess mode

[kitaekatt]

Summary

Supersedes #30409 (rebased and rewritten against current main).

Defers tokenizer initialization in StructuredOutputManager from __init__ to first access via a @property. This prevents semaphore exhaustion when GGUF models are loaded in multiprocess mode.

Problem

When StructuredOutputManager.__init__ eagerly calls cached_tokenizer_from_config(), the tokenizer builds BPE merges using multiprocessing primitives. In forked subprocesses (multiprocess model loading for GGUF), these primitives leak POSIX semaphores that aren't cleaned up, eventually exhausting the system limit (/proc/sys/kernel/sem) and causing the server to hang.

Changes

• Replace eager self.tokenizer = cached_tokenizer_from_config(...) with a @property that initializes on first access
• Reasoning parser init is also deferred (depends on tokenizer)
• ThreadPoolExecutor for grammar compilation is still created eagerly (no multiprocessing primitives)
• skip_tokenizer_init check moved to the property, raising a clear error if structured output is used without a tokenizer

Testing

Tested with repeated GGUF model load/unload cycles in multiprocess mode — no semaphore exhaustion.

[gemini-code-assist]

Code Review

This pull request correctly addresses a semaphore leak that occurs during GGUF model loading in multiprocess mode. By deferring the tokenizer initialization in StructuredOutputManager to a lazily-loaded @property, the eager creation of multiprocessing primitives within __init__ is avoided, which was the source of the leak. The reasoning parser's initialization is also correctly deferred, as it depends on the tokenizer. The implementation is clean, follows standard Python patterns for lazy initialization, and includes appropriate error handling for cases where structured output is attempted without a tokenizer. The changes are well-contained and effectively resolve the described issue.

[kitaekatt]

Closing — not pursuing this approach. The lazy tokenizer init idea isn't the right shape for fixing the semaphore leak; will explore alternatives separately if needed.