[BugFix] Work-around incremental detokenization edge case error (#19449)

[kdamaszk]

Cherry-pick of vllm-project#19449 which resolves the issue with tokenizers package

[darekkreft]

@kdamaszk I tested your cherry-pick.
Issue with Exception: Invalid prefix encountered no longer exists
but in logs I got for e.x:
mixtral_8x7b_bf16_1x_perf_apc_ibm

WARNING 07-01 10:29:36 [detokenizer.py:233] Encountered invalid prefix detokenization error for request 19, resetting decode stream.
WARNING 07-01 10:29:46 [detokenizer.py:233] Encountered invalid prefix detokenization error for request 55, resetting decode stream.
FMWORK REP 1 / 3 : 1751354936.625164722 1751355041.486498489 104.861 102.4 1132.8
WARNING 07-01 10:31:10 [detokenizer.py:233] Encountered invalid prefix detokenization error for request 228, resetting decode stream.

perf 1454.7
TPOT 79.7

full log

@kdamaszk Could look on it.
Image to reproduction:
artifactory-kfs.habana-labs.com/docker-developers/users/qauser/pt_nightly_vllm_sw_23302/1.22.0/ubuntu22.04/pt_2.7.1:1.22.0_447

[kdamaszk]

@darekkreft warning is expected -- this is the way how broken tokens are handled

[kdamaszk]

/run-gaudi-tests

[kdamaszk]

/run-gaudi-tests