Skip to content

fix(backend): add structure-aware chat admission - #8296

Merged
diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.49from
RaviTharuma:fix/8276-structure-aware-admission
Jul 23, 2026
Merged

diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.49from
RaviTharuma:fix/8276-structure-aware-admission

Conversation

@RaviTharuma

Copy link
Copy Markdown
Contributor

Summary

  • promote byte-light chat requests into the existing heavyweight lease when message, tool, or conservative token-cost thresholds are crossed
  • reject pathological histories above a configurable hard message cap with a structured compact-required 413
  • bound structural inspection by nodes and depth, conservatively treating exhausted inspection as heavyweight
  • preserve the existing actual-byte ingestion and exactly-once lease lifecycle

Scope

This is the smallest native structure-aware admission slice for the release branch. It deliberately reuses the existing immediate retryable 503 behavior rather than retaining parsed requests in a new in-process queue. Queueing, cgroup recovery gating, and metrics remain follow-up scheduler work.

Tests

  • node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --test tests/unit/chat-body-admission.test.ts tests/unit/chat-completions-parse-once-7847.test.ts — 23 passed
  • git diff --check — passed
  • npm run test:coverage — blocked before tests by the shared Node 26 dependency install (yargs CommonJS entry throws require is not defined)
  • npm run typecheck:core — blocked by missing installed modules (safe-regex, @toon-format/toon, omniglyph, sql.js, ioredis, js-tiktoken)
  • npm run check:docs-all — changed env vars are synchronized; repository baseline still reports unrelated missing GROK_AUTH_PATH / OMNIROUTE_VNC_CONTAINER_VNC_PORT entries

Changed test file: tests/unit/chat-body-admission.test.ts.

Coverage result: unavailable because the coverage runner fails during c8 startup under the current Node 26/shared dependency environment, before test execution.

Closes #8276

Co-Authored-By: Claude <noreply@anthropic.com>
@RaviTharuma
RaviTharuma force-pushed the fix/8276-structure-aware-admission branch from 797c952 to 00af177 Compare July 23, 2026 14:44
@RaviTharuma

Copy link
Copy Markdown
Contributor Author

Rebased the branch onto the latest release/v3.8.49 after the base-drift warning. Verification immediately before push:

upstream...HEAD: 0 behind / 1 ahead
5 files changed, 309 insertions(+), 2 deletions(-)
.env.example
docs/reference/ENVIRONMENT.md
src/app/api/v1/chat/completions/route.ts
src/shared/middleware/chatBodyAdmission.ts
tests/unit/chat-body-admission.test.ts

The diff contains only the focused #8276 implementation/tests/docs. Also incorporated review hardening: bounded object iteration, fail-heavy inspection exhaustion, schema-key accounting, and conservative non-ASCII token estimation. Focused suite: 26 passed.

@RaviTharuma

Copy link
Copy Markdown
Contributor Author

CI note: the failing fast-path shards include repository-wide baseline failures outside this PR’s changed files (for example DashScope video assertions appear on multiple unrelated PRs). The focused tests listed in this PR body pass locally. I am still checking for any failure attributable to this patch.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks, @RaviTharuma! 🙏 Merged — structure-aware chat admission middleware (validates request body shape before dispatch). Rebased onto the current release tip; 22/22 admission tests green.

@diegosouzapw
diegosouzapw merged commit 08a21bc into cheaper-inference:release/v3.8.49 Jul 23, 2026
5 checks passed
@diegosouzapw diegosouzapw mentioned this pull request Jul 28, 2026
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
)

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
)

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(backend): structure- and cgroup-aware long-context admission queue

2 participants