Skip to content

fix(logging): raise chat-log array truncation cap to 1000 and unify duplicate implementations - #11499

Merged
diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.51from
hartmark:fix/chat-log-array-cap-bump
Aug 25, 2026
Merged

diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.51from
hartmark:fix/chat-log-array-cap-bump

Conversation

@hartmark

Copy link
Copy Markdown
Contributor

What Problem This Solves

resolvePreviousResponseState (added for previous_response_id server-side
continuation) reads the same call-log artifact that cloneBoundedChatLogPayload
and cloneBoundedForLog bound for storage. Once a stored conversation's
input/output array crossed the truncation cap, continuation hit the
_omniroute_truncated_array sentinel and the reconstructed request became
unresolvable — worse than a diagnosability gap, it broke the call outright
(see the companion fix in #11473).

Separately, open-sse/utils/requestLogger.ts's cloneBoundedForLog had its
own independent hardcoded MAX_LOG_ARRAY_ITEMS = 24, drifted from and
duplicating open-sse/handlers/chatCore/logTruncation.ts's
cloneBoundedChatLogPayload, which already read the configurable
getChatLogArrayTailItems() (previously defaulted to 128).

Why This Change Was Made

  • A multi-round agentic tool-calling turn routinely logs well over a hundred
    input items (reasoning/function_call/function_call_output triples per
    round), so even the existing 128 default was tight for real usage.
  • CALL_LOG_RETENTION_DAYS (default 7 days) already bounds total on-disk
    storage independent of this per-item cap — raising the cap doesn't change
    the storage ceiling, it only changes how much of a single long conversation
    stays reconstructable within the retention window. Measured ~2.1GB for 7
    days of call-log artifacts on a live instance; the per-item cap was never
    the actual storage constraint.
  • Unified both truncation implementations onto the same
    getChatLogArrayTailItems() (env-configurable via
    CHAT_LOG_ARRAY_TAIL_ITEMS) so the two bounding passes over the same
    artifact data can't drift again.

User Impact

  • previous_response_id continuation stays resolvable for much longer
    real-world conversations before falling back to full history.
  • No storage/retention behavior change — only the per-array truncation
    threshold, which is a debug/reconstruction fidelity concern, not a size
    control (size is controlled by retention days/row count).
  • Operators can still override via CHAT_LOG_ARRAY_TAIL_ITEMS if 1000 is
    too high or too low for their traffic pattern.

Evidence

  • Reproduced the failure live: resolvePreviousResponseState returned a
    valid reconstruction when called directly against on-disk data, but the
    live dev server (omniroute-dev, running 14+ hours through many live
    HMR edits) returned a stale null for the identical lookup — traced this
    to that dev instance being stale, not a code bug (fixed by restart,
    unrelated to this PR).
  • 46/46 unit tests pass locally (node --import tsx --test tests/unit/chat-log-array-tail-items-default.test.ts tests/unit/chatcore-log-truncation.test.ts tests/unit/request-logger-bounded-clone.test.ts tests/unit/request-logger-bounded-idempotence.test.ts tests/unit/repro-7847-bound-client-raw-request.test.ts tests/unit/responses-continuation-store.test.ts), including three test
    files updated to exercise truncation above the new 1000-item cap instead
    of the old 24/128-item fixtures.
  • Live end-to-end verification post-change against a running instance:
    two-call continuation test (previous_response_id set on call 2) logged
    1 msgs and returned 200 for both calls.

…uplicate implementations

CHAT_LOG_ARRAY_TAIL_ITEMS default 128 -> 1000: a multi-round agentic
tool-calling turn routinely logs well over a hundred input items, and
resolvePreviousResponseState reads this same bounded artifact back to
reconstruct previous_response_id history server-side -- once a stored
conversation's input/output array crossed the cap, continuation failed
outright on the truncation sentinel, not just a diagnosability gap.
Retention (CALL_LOG_RETENTION_DAYS) already bounds total on-disk size
independent of this per-item cap.

requestLogger.ts's cloneBoundedForLog had its own separate hardcoded
MAX_LOG_ARRAY_ITEMS = 24, independent of and drifting from
chatCore/logTruncation.ts's cloneBoundedChatLogPayload, which already
read the configurable getChatLogArrayTailItems(). Pointed both at the
same value so the two bounding passes over the same artifact data stay
consistent.
@hartmark
hartmark requested a review from diegosouzapw as a code owner August 25, 2026 10:31
@diegosouzapw
diegosouzapw merged commit 3f6a881 into cheaper-inference:release/v3.8.51 Aug 25, 2026
7 of 16 checks passed
@hartmark
hartmark deleted the fix/chat-log-array-cap-bump branch August 26, 2026 20:17
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…uplicate implementations (cheaper-inference#11499)

Validated in a combined 2-PR batch worktree off release/v3.8.51 tip (companion fix to cheaper-inference#11473, merged first).
- Focused tests: chat-log-array-tail-items-default, chatcore-log-truncation, request-logger-bounded-clone, request-logger-bounded-idempotence, repro-7847-bound-client-raw-request — part of batch's 46/46 node:test run
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity — all OK
- Full-repo lint: 228 pre-existing dashboard react-hooks/* findings, unrelated to this diff

Thanks for unifying the two independently-drifted truncation caps onto one configurable source — measured evidence that the storage ceiling comes from retention days, not the per-item cap, makes the 128→1000 raise a clear correctness improvement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants