Skip to content

fix(runtime): keep catalog probes async and release aborted translator budgets - #6796

Merged
lidge-jun merged 1 commit into
devfrom
codex/l7-catalog-async-budget
Oct 9, 2026
Merged

lidge-jun merged 1 commit into
devfrom
codex/l7-catalog-async-budget

Conversation

@lidge-jun

Copy link
Copy Markdown
Owner

Summary

Carries the runtime part of #6723 by @mashfromband so it runs under repository CI. Refs #6671 (Windows memory growth); this does not claim to resolve it.

Two problems, both seen on Windows:

  • Catalog probes blocked the event loop. Resolving the Codex version and bundled model list spawned synchronous probes. With three slow Windows candidates, model-list output stalled for about 23 s and /healthz for about 20 s in the source PR's measurement. Catalog reads now serve the same selection's last confirmed catalog while an async refresh runs. The refresh is single-flight with a 45 s deadline and bounded backoff, rejects a late result whose input or selection epoch changed, and keeps caller cancellation separate from the shared refresh. Explicit synchronous CLI calls are unchanged, and the observed-catalog effort clamp reads the runtime snapshot instead of spawning.
  • Aborted requests kept their translator budget. When a client disconnected, including before reading the body, the HTTP owner's translator budget stayed live. Responses, Chat Completions and Messages now pass their request signal to one shared finalizer. Abort stays a stream error, accounting is disposed before awaiting the producer, response markers are kept, and late callbacks cannot revive disposed accounting.

Change beyond the source PR, from review: the async executor no longer relies on execFile's own abort/timeout. A reaper independent of promise settlement sends SIGTERM, escalates to SIGKILL after 2 s, and waits for exit, so an aborted or timed-out probe cannot leave a child running. The caller still gets a prompt rejection.

src/server/responses/core.ts stays at its 210-line cap. The Rust contract crate and the dedicated workflow from #6723 are not part of this PR; this PR carries Bun regressions instead.

Verification

  • New tests/adapters/translator-budget-lifetime.test.ts and tests/codex-integration/catalog-async-lifetime.test.ts (registered in both layout tables). On the unchanged source: budget 5 pass / 20 fail, catalog 0 pass / 21 fail. With the change: all pass. The child-reaping cases use READY-synchronized children that ignore SIGTERM and assert SIGKILL and PID exit (about 2.01 s).
  • Focused suites (runtime, catalog, scheduler, slug uniqueness, translator budget, Responses/Chat/Messages native, catalog sync, tests/lab/core-lab-boundary.test.ts, layout): 472 pass / 0 fail.
  • bun run typecheck, bun run privacy:scan, bun run structure:check, git diff --check: pass.
  • Independent gpt-6.1-sol review: FAIL (child not reaped after SIGTERM), fixed; re-review PASS at 9dcc26366e.
  • Not run locally: the full suite and native Windows. fix(runtime): keep catalog probes async and release aborted budgets #6723's fork CI had six failing jobs at its head; hosted Cross-platform CI on this head is the evidence for those platforms.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. (structure catalog/runtime/transport docs; troubleshooting/windows-memory.md wording.)
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (No credential or workflow changes.)

Co-authored-by: mashfromband matsumoto.yukuhashi@gmail.com

Carry the runtime portion of #6723: cancellable bounded Codex probes,
same-selection snapshots, guarded persistence, shared flights and retry cooldown.
Release owned translator accounting on request abort without changing prepared
transport verdicts or awaiting producer cancellation.

Own subprocess abort/deadline handling rather than execFile's built-in observer.
Reject the caller promptly while the existing bounded-subprocess helper reaps
actual exit, escalating SIGTERM to SIGKILL after the shared 2-second grace.
Readiness-synchronized Bun children prove termination even when SIGTERM is
ignored; Windows keeps forceful termination and still asserts exit.

Keep Windows removal ownership, Rust diagnostics and workflow changes in their
separate lanes. Refs #6671; this does not establish the Windows leak is resolved.

Co-authored-by: mashfromband <matsumoto.yukuhashi@gmail.com>
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner October 9, 2026 02:01
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T02:05:38.946161Z 9dcc263 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 27 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 3d182125-4af4-4930-8f86-98f0e02d0a68
📥 Commits

Reviewing files that changed from the base of the PR and between c15037b and 9dcc263.

📒 Files selected for processing (22)
  • docs-site/src/content/docs/troubleshooting/windows-memory.md
  • scripts/test-layout/layout.json
  • src/codex/catalog-auto-refresh-sources.ts
  • src/codex/catalog/bundled.ts
  • src/codex/catalog/effort.ts
  • src/codex/catalog/retained-sync.ts
  • src/codex/runtime.ts
  • src/lib/translator-budget.ts
  • src/server/chat-completions.ts
  • src/server/claude-messages.ts
  • src/server/responses/core-lifetime.ts
  • src/server/responses/core.ts
  • structure/catalog.md
  • structure/data-planes/inbound-compat.md
  • structure/runtime.md
  • structure/transports/byte-accounting.md
  • structure/transports/responses.md
  • tests/adapters/translator-budget-lifetime.test.ts
  • tests/codex-integration/catalog-async-lifetime.test.ts
  • tests/codex-integration/catalog-auto-refresh-scheduler.test.ts
  • tests/codex-integration/catalog-slug-uniqueness-boundary.test.ts
  • tests/fixtures/test-layout-expected.json
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the bug Something isn't working label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9dcc26366e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/codex/runtime.ts
Comment on lines +528 to +529
void waitForSubprocessExit({ exited, kill: () => child.kill("SIGKILL") }, SUBPROCESS_KILL_GRACE_MS);
finish(error);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reap the version probe before cleaning its sandbox

When a version probe times out or is aborted, stop() rejects the caller immediately while waitForSubprocessExit continues detached. This lets probeVersionAsync enter its finally and call removeProbeHome() while the Codex child may remain alive for the two-second grace period; on Windows, a child holding files under its temporary CODEX_HOME can make that best-effort removal fail with EBUSY, permanently leaking an ocx-codex-probe-* directory on every stalled refresh. Keep prompt caller rejection if needed, but defer sandbox removal until the reaper confirms actual exit.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant