reasoning-budget: opt-in soft wrap-up hint and bounded grace with hard termination cap

[masterjaso]

Overview

This is based on the work in draft PR ggml-org/llama.cpp#25961 (reasoning-budget: Implementing thinking-budget mechanism to control thought process) by @laurencehardman, updated for the current tree and with a slightly different approach to what happens near the end of the reasoning budget.

It adds two optional behaviors to the existing reasoning-budget sampler:

1. Soft wrap-up hint — when reasoning reaches a configurable fraction of the budget (--reasoning-budget-soft-ratio N), a hint is injected at the next newline boundary. It fires at most once per reasoning block and re-arms for later blocks. The text can be changed with --reasoning-budget-soft-message MESSAGE.

2. Grace tokens — once the reasoning budget is exhausted, the model gets up to --reasoning-budget-grace-tokens N additional tokens to finish naturally. If it still has not ended reasoning by then, the end sequence is forced.

The result is still bounded at budget + grace_tokens, but avoids cutting the model off immediately when it reaches the configured budget.

I did not carry over the intro-budget message from #25961. That PR can inject a message at the start of reasoning telling the model how much budget it has. This version only injects the wrap-up hint near the end.

Differences from #25961

Behavior	#25961	This PR
Intro message at reasoning start	Yes	No
Soft warning	Configurable fraction of budget	Configurable fraction, fired once at a newline boundary
After budget is reached	Hard cutoff	Bounded grace period, then forced end
Natural reasoning end	—	Always takes precedence over forcing
Multiple reasoning blocks	—	Soft hint and grace state re-arm per block
Default behavior	Opt-in	Opt-in; unchanged when the new options are not used

I left out the intro message because it is not necessary to enforce the limit, and injecting text at the beginning of every reasoning block can influence the model before it has started working on the problem. The soft hint and grace window are enough for the intended behavior while keeping the default path close to upstream.

Files changed

• common/arg.cpp — CLI options and environment-variable handling.
• common/common.h — new common_params_sampling fields.
• common/reasoning-budget.h — adds SOFT_PENDING, SOFT_FORCING, and HARD_PENDING; extends the init signature while preserving existing defaults.
• common/reasoning-budget.cpp — soft threshold handling, grace window, natural-end precedence, and per-block reset.
• common/sampling.cpp — tokenizes the optional soft message with common_tokenize(..., false, true) and passes it into the sampler.
• tests/test-reasoning-budget.cpp — 7 additional unit tests.

CLI / environment variables

--reasoning-budget-soft-ratio N         LLAMA_ARG_THINK_BUDGET_SOFT_RATIO
--reasoning-budget-soft-message MESSAGE LLAMA_ARG_THINK_BUDGET_SOFT_MESSAGE
--reasoning-budget-grace-tokens N       LLAMA_ARG_THINK_BUDGET_GRACE_TOKENS

These only have an effect when reasoning is enabled and a reasoning-budget is active.

Invalid values disable the corresponding behavior. For example, a soft ratio outside (0, 1] disables the soft hint.

Tests

Added 7 tests to tests/test-reasoning-budget.cpp:

• test_soft_fires_once_at_threshold — fires the soft hint once after the threshold is crossed and a newline is reached.
• test_soft_does_not_fire_before_threshold — does not fire early.
• test_soft_disabled_on_invalid_ratio — ratios <= 0 or > 1 disable the feature.
• test_grace_exhaustion_forces_end — forces the end sequence once grace tokens are exhausted.
• test_grace_natural_close_wins — a normal reasoning close during grace is allowed through.
• test_grace_bounds_total_reasoning — reasoning cannot exceed budget + grace_tokens.
• test_soft_and_grace_rearm_per_block — both mechanisms reset correctly for later reasoning blocks.

Validation

On base a298422da (b10548):

• tests/test-reasoning-budget.cpp: 19/19 passing (12 existing + 7 new)
• GNU 14.2.0 / Linux x86_64
• CUDA llama-server built from this branch using .devops/cuda.Dockerfile

I also ran a live server smoke test with:

--reasoning on
--reasoning-budget 8192
--reasoning-budget-soft-ratio 0.65
--reasoning-budget-grace-tokens 1536

The server accepted the new settings and streamed normal reasoning_content.

There was no intro text at the start of reasoning, as intended. The test generation also ended before reaching the 65% soft threshold, and no soft message was injected.

With the new options omitted, the existing reasoning-budget behavior is unchanged.

Motivation

The main use case is reasoning models that sometimes get stuck in repetitive or increasingly unproductive thought loops, particularly at lower temperatures.

A hard token cutoff solves the upper-bound problem, but it can also stop the model in the middle of a useful thought. The soft hint gives it a chance to start wrapping up before the limit, and the small grace window lets it finish naturally without turning the reasoning budget into an unbounded suggestion.

Based on: #25961.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: Yes. An AI coding agent assisted with the sampler state-machine implementation, unit tests, build work, and validation.

[ggml-gh-bot]

Hi @masterjaso, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[masterjaso]

Updated Title and Description to be less 'ai' - but overall content is correct - and hope this contribution is helpful.

(Using it locally on my box - and seems to be a really nice fix to 'thought hungry' models like Qwen 3.8 27B and Ornith 1.5 35B)