Studio: queue local GGUF OpenAI-compatible requests before llama-server

[Apoze]

Summary

This adds a Studio-side admission queue for local GGUF OpenAI-compatible requests before they reach llama-server.

The queue is scoped to local GGUF /v1/chat/completions and /v1/responses paths, and uses the effective llama.cpp parallel slot count so Studio does not forward more concurrent upstream requests than the running backend can service.

Problem

When Studio starts llama.cpp with a small --parallel value, clients can still send multiple OpenAI-compatible requests at once. Before this change, excess requests were forwarded to llama-server, where they waited behind the active request. For streaming clients this can look like a stalled or terminated stream because no SSE bytes are emitted while the request is stuck upstream.

Changes

• Add a framework-free llama_admission queue with bounded capacity, timeout, cancellation cleanup, and async release handling.
• Track LlamaCppBackend.effective_parallel_slots after a healthy load, and reset it on unload/cleanup.
• Gate local GGUF /v1/chat/completions and /v1/responses requests before forwarding to llama.cpp.
• Emit SSE keepalive comments while queued streaming requests wait for a Studio admission slot.
• Return OpenAI/Responses-style overload or timeout errors when queued requests cannot be admitted.
• Keep the rollback/tuning surface in env vars only:
  • UNSLOTH_OPENAI_COMPAT_ADMISSION_CONTROL
  • UNSLOTH_OPENAI_COMPAT_ADMISSION_QUEUE_TIMEOUT
  • UNSLOTH_OPENAI_COMPAT_ADMISSION_KEEPALIVE_INTERVAL
  • UNSLOTH_OPENAI_COMPAT_ADMISSION_MAX_QUEUE

This intentionally does not add UI settings and does not gate /v1/completions in this PR.

Validation

Automated checks:

python -m py_compile studio/backend/core/inference/llama_admission.py studio/backend/core/inference/llama_cpp.py studio/backend/routes/inference.py
git diff --check upstream/main..HEAD
python -m pytest studio/backend/tests/test_llama_admission.py studio/backend/tests/test_llama_cpp_effective_parallel_slots.py -q --tb=short
# 18 passed
python -m pytest studio/backend/tests/test_openai_tool_passthrough.py -q --tb=short
# 247 passed, 1 warning
python -m pytest studio/backend/tests/test_responses_tool_passthrough.py studio/backend/tests/test_responses_api.py -q --tb=short
# 126 passed

Source-clone LAN API checks:

Patched clone: 0.0.0.0:8899, --parallel 1
Model: unsloth/Qwen3.6-27B-MTP-GGUF, UD-Q4_K_XL
llama.cpp: b9923, stale=false

Direct API checks:

/v1/chat/completions non-stream: status 200, object=chat.completion
/v1/chat/completions stream: status 200, [DONE]=true, JSON parsing failed=false
/v1/responses stream: response.completed=true, response.failed=false
Concurrent chat stream behind long request: first SSE line was ': keep-alive', [DONE]=true, JSON parsing failed=false
llama-server stats during the concurrent test: running=1, waiting=0

Real client checks from a LAN Ubuntu VM against http://192.168.10.113:8899/v1:

OpenCode CLI 1.17.18, temporary XDG config/data dirs, @ai-sdk/openai-compatible:
RC1=0, text=OPENCODE_PR2_OK
RC2=0 with --continue, text=OPENCODE_PR2_TOUR2_OK
No terminated output and no JSON parsing failure.

Codex CLI 0.142.5, temporary custom provider using wire_api="responses":
RC=0, text=CODEX_PR2_OK
usage: input_tokens=8354, output_tokens=20

For the Codex CLI check, the GGUF was loaded with a 16k context because Codex's default prompt envelope is larger than an 8k context window.

Notes

This builds on the streaming hardening merged in #6950. It handles a different layer: request admission before llama.cpp receives the request, so streaming clients get an immediate SSE response and keepalives while waiting instead of a silent upstream wait.

[gemini-code-assist]

Code Review

This pull request introduces an admission control mechanism for local llama-server generation requests, managing active upstream requests and queuing excess ones using a cancellable FIFO queue. It integrates this control across various streaming and non-streaming inference routes and updates the llama-cpp backend to track effective parallel slots. The reviewer identified a critical issue where a cancelled waiter future could cause waiting loops to spin infinitely and consume 100% CPU; they suggested adding and checking an is_cancelled property on reservations to prevent this. Additionally, the reviewer pointed out a potential double-exit of the tracker context manager if a stream generator fails before yielding, suggesting that stream_started be set to True immediately upon iterator creation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Apoze]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 3085340691

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[oobabooga]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 11397bdc1f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[oobabooga]

I wanted to confirm this fixes a real dead stream and not just a slow first byte, so before merging I reproduced it. One long request held a single-slot llama-server (--parallel 1), a short one queued behind it, and I watched the second request's raw SSE bytes:

• Admission off (its kill-switch, which is main's behavior today): a single chunk at 2ms, then about 7.6s of complete silence with no keepalives before any content.
• Admission on (your PR): : keep-alive every interval until the slot frees, then the content.

Studio flushes the 200 and SSE headers before the body, so with admission off that silence sits on an already-open stream, which is what a client can time out on. Holding the request in the queue and keeping it alive is the right fix, since once the response has started you can't hold it back. CI is green across the fork matrix, and the Gemini "critical" notes describe code your commit already has.

Follow-up I pushed: the admission registry keys on the llama-server base_url, which carries a fresh ephemeral port every load, so it gained a dead entry per model load. It now drops idle queues from prior loads (anything in-flight is kept), plus two dead-code removals. Your admission path is untouched.

Merging.