Skip to content

fix: guard against runaway/looping generations - #11

Merged
jdrean merged 1 commit into
mainfrom
fix/runaway-generation-guards
Sep 17, 2026
Merged

jdrean merged 1 commit into
mainfrom
fix/runaway-generation-guards

Conversation

@jdrean

@jdrean jdrean commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

GLM-5.3-Flash can degrade into repetition loops in long tool-calling sessions (vllm#54337, HF discussion #23). With no max_tokens, a looping request generates until the 1M context and holds a --max-num-seqs slot the whole time. The upstream fix (vllm#54368) is unmerged and not in the pinned base.

Three independent guards:

  1. --chat-template-content-format=string — stops content: null + tool_calls assistant turns from rendering a literal None into the GLM template, the trigger identified upstream. Present in the GLM-5 recipe; missing from the 5.3-Flash one.
  2. max_new_tokens: 131072 in --override-generation-config — vLLM's fallback max_tokens for requests that omit one; explicit client values still win (get_max_tokens() in entrypoints/serve/utils/api_utils.py). Matches OpenAI's GPT-5.x output limit.
  3. patches/0003 — arms vLLM's built-in repetition detector (SamplingParams.repetition_detection → finish_reason=repetition) for requests that don't set one. Explicit values, including all zeros, are preserved as the opt-out. Default: a ≥4-token pattern repeated 16× back-to-back, patterns up to 64 tokens — catches any loop within ≤1024 tokens while letting wide markdown tables, repeated code lines and rulers through. Tunable at runtime via container env TINFOIL_REPETITION_DETECTION="max,min,count" ("0" disables), so no rebuild is needed to adjust.

Verification: patch applies to pristine vllm/sampling_params.py at main@385dce36 with patch -p1 --fuzz=0 and compiles; thresholds exercised against upstream's check_sequence_repetition; config parses.

Not included: a nightly bump (patches 0001/0002 need rebasing on current main) and any tool_choice: "required" handling (vllm#50399).

GLM-5.3-Flash can degrade into repetition loops in long tool-calling
sessions (vllm#54337); with no max_tokens a looping request holds a
--max-num-seqs slot until the 1M context.

- --chat-template-content-format=string so content:null tool-call turns
  do not render a literal "None" into the template.
- max_new_tokens=131072 as the fallback max_tokens for requests that omit
  one; explicit values still win.
- patches/0003 arms vLLM's built-in repetition detector for requests that
  do not set repetition_detection (>=4-token pattern x16, up to 64
  tokens); env TINFOIL_REPETITION_DETECTION overrides, "0" disables.
@jdrean
jdrean force-pushed the fix/runaway-generation-guards branch from a7fa9e6 to a98d3d7 Compare September 17, 2026 15:33
@jdrean jdrean changed the title fix: guard against runaway/looping generations (content-format flag, max_new_tokens fallback, repetition detector) fix: guard against runaway/looping generations Sep 17, 2026
@jdrean
jdrean requested a review from lothan September 17, 2026 15:35
@jdrean
jdrean merged commit a98d3d7 into main Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant