Skip to content

fix(next): do not start Chromium under an open-file limit it would run out of - #29377

Draft
lawrencecchen wants to merge 4 commits into
feat-cmux-nextfrom
cp/fix-cef-nofile-floor
Draft

lawrencecchen wants to merge 4 commits into
feat-cmux-nextfrom
cp/fix-cef-nofile-floor

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Under a hard RLIMIT_NOFILE below 1024 (a launch from a shell after ulimit -n 256, which sets both limits), the app cannot raise its open-file limit, and Chromium aborted the whole app about 6 s after launch. Its child launch treats a failed pipe() as fatal: sandbox/mac/seatbelt_exec.cc:86-87, PFatal("SeatbeltExecClient: pipe failed"), EMFILE.

Found by the restore/chaos fuzz (fuzz catalog row h, finding h1): run h2-fd seed 2012, cmux21, pid 23845, 2026-10-31 10:15:04Z, feat d1f0e63 Debug. Symbolicated with the fork's unstripped engine cef-154.0.28-cmux.28-macos-arm64-debug.tar.xz (the engine UUID matches the crash report). The crashing thread: ChildProcessLauncherHelper::BeforeLaunchOnLauncherThread -> sandbox::SeatbeltExecClient::SeatbeltExecClient() -> sandbox::logging::PFatal -> abort_report_np. The app's stderr shows FILE_ERROR_TOO_MANY_OPENED in the same second. This is not memory pressure: the hog had ended 47 s earlier, and with the 15 GB hog the same build relaunched 82 times with 0 deaths.

Change

  • CEFRuntime.loadLibrary refuses to start Chromium, before the framework is mapped, when the soft limit after the app's own raise is below OpenFileLimit.chromiumFloor (1024). So no Chromium thread runs.
  • The failure is typed (CEFUnavailableReason.openFileLimit). Chromium tabs wait in place (WebKit would open them signed out) with a localized notice that names the limit and the way out (start from Finder or the Dock, or raise ulimit -n). An explicit Chromium request gets the same text.
  • Agent chats in Chromium (agent.chromium): ChromiumPageHost.isAvailable now reads the refused start, so new chats open in WebKit. A chat that asked before the refusal reopens in WebKit at once.
  • It is logged once, and the start is not tried again. No retry, no relaunch loop, no catch of the abort. Terminals, the daemon and WebKit pages do not change.
  • A launch from Finder, the Dock or open always raises to at least 10240, so only a lowered hard limit is refused.
  • The floor value is a judgement for now. A measurement on a mini (hard limits 512, 1024, 2048 with the restored session and 5 Chromium tabs) is queued; the value and its numbers go into the code comment when it is done.

Tests (red first)

  • Red 3f4f7507fa3: OpenFileLimitTests (aHardLimitOf256KeepsChromiumFromStarting and 3 more), CEFOpenFileFloorTests.aHardLimitOf256RefusesTheLoad.
  • At the head also: CEFOpenFileFloorTests.theRefusalNamesTheLimitAndKeepsTabsOutOfWebKit, aRefusedStartIsUnavailableWithItsReason, PageWebViewChromiumTests.anAgentChatInChromiumFallsBackToWebKitWhenChromiumIsRefused.

Observed / Not run

  • Observed: fleet compile-check e41127457b18b29d053c48f0 at 632d52ad8e2: CmuxNext, app and cmux-tui ok. prepush-gate.py: no new violations, app_lines +0. localization_catalog.py check: no finding for the new key (21 locales).
  • Not run: the red and green Mac test runs. proof-step.py refuses Mac steps for this branch until it is on the Mac lane allow list (faster-2.md item 4); I asked the lead.
  • Not run: a real launch under ulimit -n 256 with the fix (part of the queued floor measurement).

No capture: the notice shows only under a hard open-file limit below 1024; a real capture comes with the floor measurement.

Changelog

Fixed: cmux no longer quits when it starts with a very low open-file limit; web pages show a notice that names the limit instead.

perf claim: none


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the app aborting about 6 seconds after launch when started under a hard open-file limit below 1024 — Chromium treated a failed pipe() on its child-launch path as fatal. The app now refuses to start Chromium before its framework is loaded when the limit is too low, and shows a notice instead of crashing.

  • Refusal is typed (openFileLimit): Chromium tabs wait in place with a localized notice that names the limit and the way out, instead of opening signed out in WebKit.
  • Agent chats in Chromium open in WebKit when Chromium is refused; a chat already asking reopens in WebKit at once.
  • A refused start is logged once and never retried.
  • The 1024 floor is backed by a cmux21 measurement: the app process peaked at about 270 open descriptors, and hard limits of 512, 1024, and 2048 produced no aborts.

Written for commit a40015e. Summary will update on new commits.

View guided diff Turn on auto-fix

… run out of

Red: OpenFileLimit.chromiumShortfall() and CEFRuntime.loadLibrary's open-file
gate are stubs here. Under a hard RLIMIT_NOFILE of 256 the app cannot raise its
limit, and Chromium aborted the app 6 s after launch in its child launch
(sandbox/mac/seatbelt_exec.cc:86-87, PFatal "SeatbeltExecClient: pipe failed",
EMFILE; chaos fuzz row h, cmux21, 2026-10-31, symbolicated with the fork's
cef-154.0.28-cmux.28 engine).
…n out of

Under a hard RLIMIT_NOFILE below 1024 (a launch from a shell after
`ulimit -n 256`, which sets both limits) the app cannot raise its limit, and
Chromium aborted the whole app 6 s after launch: its child launch treats a
failed pipe() as fatal (sandbox/mac/seatbelt_exec.cc:86-87,
"SeatbeltExecClient: pipe failed", EMFILE). Chaos fuzz row h, cmux21,
2026-10-31, run h2-fd seed 2012, symbolicated with the fork's
cef-154.0.28-cmux.28 engine.

CEFRuntime.loadLibrary now refuses before the framework is mapped when the
soft limit (after the app's raise) is below OpenFileLimit.chromiumFloor, so no
Chromium thread runs. The failure is typed (CEFUnavailableReason.openFileLimit):
Chromium tabs wait in place with a localized notice that names the limit and
the way out (start from Finder or the Dock, or raise ulimit -n), and an
explicit Chromium request is refused with the same text. It is logged once; a
failed start is never tried again. Terminals, the daemon and WebKit pages are
unchanged. Chrome's own model: it raises its limit at start and cannot run
below its needs either; cmux refuses visibly instead of aborting.
…efused; CEFEngine reason mapping is pure and tested
@lawrencecchen lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Default changes

This pull request changes 1 default at a40015eac3a8 (what everyone gets without configuring anything):

  • tunable-source.CmuxNextApp/ChromiumFallbackLog.swift:refusal: "784865d5a93078b896e9bf00884807e5648958e549ab27f5f9620f33bcac246c" -> "315ee4b63d9ef78d96631fdc4f03a77465952ecfd0102fb64f8d74664fade5c9"

Needs a review. It passes with an independent review receipt on this commit: a comment or review whose first line is Review (<reviewer>): SHIP at a40015eac3a851b1be11d89a9f0c7ef524731e4f, from a member, not a self-review (the newest verdict naming the head decides; a later BLOCK beats an earlier SHIP), or with an approving review on this commit from @austinywang or @teamleaderleo (the call owners in .github/defaults-call-owners.json, never the author). A receipt posted as a review re-runs this check; after a plain comment, re-run it. The needs a call label alone does not pass, and a new push needs a new receipt.

Regenerate the snapshot with python3 scripts/ci/defaults_guard.py snapshot.

@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…nt page from a test root; floor comment gets the cmux21 numbers
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Red proof (fleet step 511c65ab7749550617a4a903, red commit 3f4f750, cmux18-perf, compiled-at 632d52a): red as intended. Each new test that checks the refusal fails on the stubs:

  • CEFOpenFileFloorTests.aHardLimitOf256RefusesTheLoad: fails (CEFOpenFileFloorTests.swift:28, "Chromium's load went on under an open-file limit of 256").
  • OpenFileLimitTests.aHardLimitOf256KeepsChromiumFromStarting, launchdsDefaultLetsChromiumStartAfterTheRaise, theFloorItselfIsEnough: fail (chromiumShortfall() returns nil).
  • The control tests pass on the red commit as they must: aRaisedLimitDoesNotRefuse, anUnreadableLimitLetsChromiumStart and the 6 existing OpenFileLimitTests.

Green at 632d52a (step 38b800283e82968fef401c4b, cmux22-perf): OpenFileLimitTests and CEFOpenFileFloorTests pass. PageWebViewChromiumTests had 1 failure, in the new agent-chat test only: it built the reserved cmux.agent page without testRoot: true and got nil (a test fault). Fixed in a40015e, which also puts the cmux21 floor numbers in the chromiumFloor comment. The green run at a40015e follows its compile-check (8391dfd6a733cba4a215cd40).

This branch was previously deployed

1 inactive deployment
ci-routing-agent — a40015ea Deployed Oct 31, 2026 by lawrencecchen via cmux-tui tree preflight #89939
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant