Skip to content

fix(sse): auto-calibrate heap-pressure threshold to the V8 heap ceiling - #3052

Merged
diegosouzapw merged 1 commit into
release/v3.8.8from
fix/heap-pressure-autocalibrate
Jun 2, 2026
Merged

diegosouzapw merged 1 commit into
release/v3.8.8from
fix/heap-pressure-autocalibrate

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Problem

The chatCore memory-pressure guard returned 503 Service temporarily unavailable due to resource pressure once heapUsed passed a hardcoded 200MB. The app's own working set is ~260MB, so on the default 512MB heap — and on the no-cap PM2 deploy (V8 ceiling ~2240MB) — the guard rejected every request after warmup. Observed live on the Local VPS: all connections failing, surfaced to clients as [anthropic/claude-haiku-4.5] Service temporarily unavailable....

Fix

Derive the threshold from the live V8 heap ceiling (v8.getHeapStatistics().heap_size_limit) at 85%, floored at 400MB, so it tracks --max-old-space-size across 1GB / 2GB / large VPS and always clears the runtime baseline. Explicit HEAP_PRESSURE_THRESHOLD_MB still overrides.

VPS OMNIROUTE_MEMORY_MB heap ceiling auto threshold
1GB/1core 512 (default) ~704 598
2GB 1024 ~1216 1034
4GB 2048 ~2240 1904
no-cap (PM2) — 2240 1904

Changes

  • open-sse/utils/heapPressure.ts — pure computeHeapPressureThresholdMb + resolved constant
  • chatCore.ts — import the constant instead of the inline parse-or-200
  • .env.example — document auto-calibration + per-tier OMNIROUTE_MEMORY_MB guidance (incl. the container/cgroup caveat)
  • tests — heap-pressure.test.ts (full matrix) + replace the obsolete "defaults to 200" assertion with one tied to the live production constant

13/13 heap tests pass; typecheck / lint / cycles clean.

The chatCore memory-pressure guard rejected requests with a 503 "Service
temporarily unavailable due to resource pressure" once heapUsed passed a
hardcoded 200MB. The app's own working set is ~260MB, so on the default
512MB heap — and on the no-cap PM2 deploy (heap ceiling ~2240MB) — the guard
503'd EVERY request after warmup, a full outage.

Derive the threshold from the live V8 heap ceiling
(v8.getHeapStatistics().heap_size_limit) at 85%, floored at 400MB, so it
tracks --max-old-space-size across 1GB / 2GB / large VPS and always clears the
runtime baseline. An explicit HEAP_PRESSURE_THRESHOLD_MB still overrides.

- open-sse/utils/heapPressure.ts: pure computeHeapPressureThresholdMb + resolved constant
- chatCore.ts: import the constant instead of the inline parse-or-200
- .env.example: document the auto-calibration + per-tier OMNIROUTE_MEMORY_MB guidance
- tests: heap-pressure.test.ts (1GB/2GB/no-cap/floor/override matrix) + replace the
  obsolete "defaults to 200" assertion with one tied to the live production constant
@diegosouzapw
diegosouzapw merged commit a5f3c99 into release/v3.8.8 Jun 2, 2026
2 checks passed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces the fixed 200MB heap-pressure threshold with an auto-calibrating mechanism that dynamically sets the threshold to 85% of the V8 heap ceiling (with a 400MB floor), preventing premature 503 rejections. The changes include a new utility 'heapPressure.ts', updates to 'chatCore.ts', and corresponding unit tests. The review feedback correctly points out that using 'Number(override)' to parse the environment variable is less robust than 'parseInt' because it fails on values with unit suffixes like '512mb', and suggests a backward-compatible fix.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +27 to +28
const explicit = Number(override);
if (Number.isFinite(explicit) && explicit > 0) return Math.floor(explicit);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using Number(override) to parse the environment variable is less robust than parseInt and changes the previous behavior in chatCore.ts (which used parseInt). Specifically, Number will fail (returning NaN and falling back to auto-calibration) on values with common unit suffixes like "400MB" or "512mb". Using parseInt with a fallback to typeof override === 'number' maintains backward compatibility and gracefully handles trailing non-numeric characters.

Suggested change
const explicit = Number(override);
if (Number.isFinite(explicit) && explicit > 0) return Math.floor(explicit);
const explicit = typeof override === "number" ? override : parseInt(String(override || ""), 10);
if (Number.isFinite(explicit) && explicit > 0) return Math.floor(explicit);

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a514420ce9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// "guard never fires" rather than "guard rejects everything".
const SHED_RATIO = 0.85;
const FLOOR_MB = 400;
return Math.max(Math.round(heapSizeLimitMb * SHED_RATIO), FLOOR_MB);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the auto threshold below the heap ceiling

For supported low-memory settings, this floor puts the shed threshold above the process's actual V8 ceiling: the launchers accept OMNIROUTE_MEMORY_MB down to 64 MB (bin/cli/commands/serve.mjs and scripts/build/runtime-env.mjs), and e.g. --max-old-space-size=128 reports a heap limit of about 320 MB. In that case computeHeapPressureThresholdMb() returns 400, so chatCore's heapUsed > HEAP_PRESSURE_THRESHOLD_MB check cannot trip before V8 reaches its own limit and hard-OOMs, regressing the guard specifically for low-memory deployments.

Useful? React with 👍 / 👎.

diegosouzapw added a commit that referenced this pull request Jun 2, 2026
…add contributor hall

Audited all 687 commits / 60 release/v3.8.8 PRs since v3.8.7 against the CHANGELOG:

- Added 5 missing Fixed entries: #3052 (heap-pressure auto-calibration),
  #3051/#3048 (proxy fail-closed + registry assignments, @terence71-glitch),
  #3049/#3046 (session-pool fingerprint rotation + claude-web cf_clearance, @oyi77).
- Credited previously-uncredited contributors: @branben (#2958 scope fix, #2959
  Notion context source) and @JxnLexn (per-API-key stream default mode).
- Added an Added entry for the per-API-key stream default mode feature.
- Added the "🏆 Contributors" hall (24 contributors), matching the v3.8.6 format.

Maintainer fix-PRs (#2966–#3030) are intentionally referenced by their original
issue numbers in the body rather than the fix-PR number; @diegosouzapw is in the hall.
@diegosouzapw diegosouzapw mentioned this pull request Jun 2, 2026
@diegosouzapw
diegosouzapw deleted the fix/heap-pressure-autocalibrate branch June 2, 2026 21:17
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…ng (cheaper-inference#3052)

Threshold now derives from the live V8 heap ceiling (85%, floor 400MB) instead of a fixed 200MB that sat below the ~260MB app baseline and 503'd every request. Tracks --max-old-space-size across 1GB/2GB/large VPS.
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…add contributor hall

Audited all 687 commits / 60 release/v3.8.8 PRs since v3.8.7 against the CHANGELOG:

- Added 5 missing Fixed entries: cheaper-inference#3052 (heap-pressure auto-calibration),
  cheaper-inference#3051/cheaper-inference#3048 (proxy fail-closed + registry assignments, @terence71-glitch),
  cheaper-inference#3049/cheaper-inference#3046 (session-pool fingerprint rotation + claude-web cf_clearance, @oyi77).
- Credited previously-uncredited contributors: @branben (cheaper-inference#2958 scope fix, cheaper-inference#2959
  Notion context source) and @JxnLexn (per-API-key stream default mode).
- Added an Added entry for the per-API-key stream default mode feature.
- Added the "🏆 Contributors" hall (24 contributors), matching the v3.8.6 format.

Maintainer fix-PRs (cheaper-inference#2966–cheaper-inference#3030) are intentionally referenced by their original
issue numbers in the body rather than the fix-PR number; @diegosouzapw is in the hall.
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
…ng (cheaper-inference#3052)

Threshold now derives from the live V8 heap ceiling (85%, floor 400MB) instead of a fixed 200MB that sat below the ~260MB app baseline and 503'd every request. Tracks --max-old-space-size across 1GB/2GB/large VPS.
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
…add contributor hall

Audited all 687 commits / 60 release/v3.8.8 PRs since v3.8.7 against the CHANGELOG:

- Added 5 missing Fixed entries: cheaper-inference#3052 (heap-pressure auto-calibration),
  cheaper-inference#3051/cheaper-inference#3048 (proxy fail-closed + registry assignments, @terence71-glitch),
  cheaper-inference#3049/cheaper-inference#3046 (session-pool fingerprint rotation + claude-web cf_clearance, @oyi77).
- Credited previously-uncredited contributors: @branben (cheaper-inference#2958 scope fix, cheaper-inference#2959
  Notion context source) and @JxnLexn (per-API-key stream default mode).
- Added an Added entry for the per-API-key stream default mode feature.
- Added the "🏆 Contributors" hall (24 contributors), matching the v3.8.6 format.

Maintainer fix-PRs (cheaper-inference#2966–cheaper-inference#3030) are intentionally referenced by their original
issue numbers in the body rather than the fix-PR number; @diegosouzapw is in the hall.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ng (cheaper-inference#3052)

Threshold now derives from the live V8 heap ceiling (85%, floor 400MB) instead of a fixed 200MB that sat below the ~260MB app baseline and 503'd every request. Tracks --max-old-space-size across 1GB/2GB/large VPS.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…add contributor hall

Audited all 687 commits / 60 release/v3.8.8 PRs since v3.8.7 against the CHANGELOG:

- Added 5 missing Fixed entries: cheaper-inference#3052 (heap-pressure auto-calibration),
  cheaper-inference#3051/cheaper-inference#3048 (proxy fail-closed + registry assignments, @terence71-glitch),
  cheaper-inference#3049/cheaper-inference#3046 (session-pool fingerprint rotation + claude-web cf_clearance, @oyi77).
- Credited previously-uncredited contributors: @branben (cheaper-inference#2958 scope fix, cheaper-inference#2959
  Notion context source) and @JxnLexn (per-API-key stream default mode).
- Added an Added entry for the per-API-key stream default mode feature.
- Added the "🏆 Contributors" hall (24 contributors), matching the v3.8.6 format.

Maintainer fix-PRs (cheaper-inference#2966–cheaper-inference#3030) are intentionally referenced by their original
issue numbers in the body rather than the fix-PR number; @diegosouzapw is in the hall.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant