Skip to content

feat(lifecycle): allow an explicit restart drain grace (carry #6703) - #6724

Merged
lidge-jun merged 6 commits into
devfrom
codex/carry-6703-restart-drain-grace
Oct 8, 2026
Merged

lidge-jun merged 6 commits into
devfrom
codex/carry-6703-restart-drain-grace

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Carries #6703 by @hulkbig with the one fix its review asked for. Closes #6643.

POST /api/system/restart accepts an optional JSON body such as {"drainGraceMs":2000} to shorten how long the restart waits for active turns. Omitting the body or sending {} keeps the existing 60-second default. The value must be an integer from 1 to 60000; malformed JSON and invalid values are rejected with 400 before any drain starts. Only a management session or the admin token may set it — the target-bound local restart capability gets 403. The first accepted grace is fixed for repeated requests, the 60 s cleanup watchdog and 70 s replacement readiness budget are unchanged, and an interrupted turn is never replayed automatically. Dashboard, CLI, tray, join and automatic restarts keep their default; there is no gui/ change. The English and seven localized API pages document the option.

The fix on top of #6703: the drain always started 200 ms after acceptance so the 202 response could flush, so a grace below 200 ms (for example {"drainGraceMs":1}) cancelled active turns at about 200 ms instead of at its accepted deadline. Shortening that delay would have let drainAndShutdown's listener stop (server.stop(true)) or a supervised exit cut the 202 before it flushed. Instead, a grace below 200 ms now arms a separate timer at acceptance + grace that cancels and releases admitted turns only. Cleanup, listener stop and process exit still wait the full 200 ms, and a pending restart veto cancels the timer. Grace of 200 ms or more, including the default, arms nothing new.

Co-authored-by: Hulk happyhls@gmail.com

Verification

On this head (c7ebc4f142), macOS arm64, Bun 1.4.0:

  • New regressions in tests/server/system-restart-admission.test.ts: grace 1 ms and 199 ms cancel the active turn by the accepted deadline on both the standalone and supervised paths, with no drain, listener stop or exit before 200 ms; a veto cancels the early cut; grace 200, 60000 and the default arm no early cut. On the unmodified feat(lifecycle): allow an explicit restart drain grace #6703 head cc7cae71b5 these fail (29 pass / 5 fail); with the fix, 34 pass / 0 fail.
  • bun test tests/server/system-restart-admission.test.ts tests/server/system-restart.test.ts tests/service/shutdown-drain.test.ts tests/lib/system-restart-contract-security.test.ts — 87 pass.
  • bun run typecheck, layout tests (18 pass), tests/ci-workflows/file-size-ratchet.test.ts (9 pass), tests/lab/core-lab-boundary.test.ts (25 pass), bun run structure:check and git diff --check — pass.
  • feat(lifecycle): allow an explicit restart drain grace #6703's own wider verification (25-file restart/lifecycle/auth regression set, docs build, privacy scan) was recorded on its head cc7cae71b5; it is that head's evidence, not a rerun here. The full suite was not run locally; required hosted CI covers this head.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • New Features
    • Restart requests can now optionally set a drain grace period from 1 to 60,000 ms. Requests without this setting retain the 60-second default.
    • The first accepted restart request determines the grace period; subsequent requests do not change it.
    • Invalid values return an error without starting a restart, and local restart capabilities cannot set a custom grace period.
  • Documentation
    • Clarified restart timing and noted that an interrupted request may have already taken effect; automatic retries are not performed.

hulkbig and others added 6 commits October 6, 2026 23:22
Separate the proposed two-second active/scoped-drain grace from the existing sixty-second cleanup watchdog and seventy-second replacement-readiness budget. Keep API and dashboard timing aligned, cover lifecycle timing and ownership with synthetic clocks, and document interrupted-execution uncertainty.

Proposal for owner review of the shared lifecycle default; does not authorize automatic replay or guarantee a two-second recovery.

Refs #6643
Signed-off-by: BigHulk <happyhls@gmail.com>
Signed-off-by: BigHulk <happyhls@gmail.com>
Signed-off-by: BigHulk <happyhls@gmail.com>
…ning the flush window

The restart drain always started 200ms after acceptance so the 202 response
could flush, which meant an explicit drainGraceMs below 200 cancelled active
turns late. Arm a separate early-cut timer at acceptance + grace for those
values; cleanup, listener stop and process exit still wait for the 200ms
flush window, and a pending veto cancels the timer. Default and >=200ms grace
arm nothing new.

Co-authored-by: Hulk <happyhls@gmail.com>
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner October 8, 2026 00:50
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T00:54:07.973330Z c7ebc4f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 8, 2026
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 1e36bb15-138b-423d-afd6-3767c4b19782
📥 Commits

Reviewing files that changed from the base of the PR and between bbab02a and c7ebc4f.

📒 Files selected for processing (15)
  • docs-site/src/content/docs/fr/reference/management-api.md
  • docs-site/src/content/docs/ja/reference/management-api.md
  • docs-site/src/content/docs/ko/reference/management-api.md
  • docs-site/src/content/docs/reference/management-api.md
  • docs-site/src/content/docs/ru/reference/management-api.md
  • docs-site/src/content/docs/tr/reference/management-api.md
  • docs-site/src/content/docs/zh-cn/reference/management-api.md
  • docs-site/src/content/docs/zh-tw/reference/management-api.md
  • scripts/test-layout/layout.json
  • src/server/lifecycle.ts
  • src/server/management/system-restart.ts
  • src/server/management/system-routes.ts
  • structure/ops/service-and-sidecars.md
  • tests/fixtures/test-layout-expected.json
  • tests/server/system-restart-admission.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The restart endpoint now accepts an optional drainGraceMs value. Restart handling uses the accepted grace for draining active turns and reports it as drainTimeoutMs. Tests and documentation cover validation, cancellation timing, and the separate cleanup and replacement-readiness budgets.

Changes

Restart drain grace

Layer / File(s) Summary
Request contract and admission
src/server/management/system-routes.ts, src/server/management/system-restart.ts, tests/server/system-restart-admission.test.ts, tests/fixtures/test-layout-expected.json, scripts/test-layout/layout.json, docs-site/src/content/docs/*/reference/management-api.md, docs-site/src/content/docs/reference/management-api.md
The restart route accepts an optional integer drainGraceMs from 1 to 60,000 ms. Invalid input returns 400, and target-bound restart capabilities cannot set the option. The first accepted grace is reported as drainTimeoutMs. API tests and localized endpoint documentation cover the request contract.
Drain and shutdown timing
src/server/management/system-restart.ts, src/server/lifecycle.ts, tests/server/system-restart-admission.test.ts, structure/ops/service-and-sidecars.md
Restart handling stores the accepted grace and passes the remaining time to the drain. Grace periods under 200 ms schedule active-turn cancellation without stopping the listener before response flushing. The restart cleanup watchdog and replacement-readiness budgets remain separate. Tests cover deadlines, cancellation, vetoes, cleanup, and replacement readiness.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant ManagementClient
  participant SystemRestartRoute
  participant acceptSystemRestart
  participant drainAndShutdown
  participant ActiveTurns
  ManagementClient->>SystemRestartRoute: POST /api/system/restart with optional drainGraceMs
  SystemRestartRoute->>acceptSystemRestart: validated restart options
  acceptSystemRestart->>drainAndShutdown: accepted grace and remaining drain time
  drainAndShutdown->>ActiveTurns: wait until completion or grace expiry
  drainAndShutdown->>ActiveTurns: abort active turns if short grace expires
Loading

Merge Risk: ⚪ Minimal · up to c7ebc

The restart endpoint gains an optional, validated drain-grace setting while the 60-second default is kept for everything else. No merge-blocking risk was identified.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning Issue #6643 describes a failure when the restart fence remains at 60 seconds. The implementation keeps that behavior for omitted bodies and existing restart callers. In `src/server/management/system-r… Apply the #6643 handoff bound to the affected default restart path, including omitted-body and service restart callers, while keeping drain cleanup separate. Add regression tests that reproduce the original default-call failure and verify a…
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 4 files. (11 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: adding an explicit restart drain grace option. This matches the implementation in src/server/management/system-restart.ts and src/server/management/system…
Out of Scope Changes check ✅ Passed The changes stay connected to the restart-grace behavior. src/server/management/system-restart.ts and src/server/management/system-routes.ts implement the option and its authorization. `src/server…
Full details: Linked Issues check

Explanation

Issue #6643 describes a failure when the restart fence remains at 60 seconds. The implementation keeps that behavior for omitted bodies and existing restart callers. In src/server/management/system-restart.ts, acceptSystemRestart selects MEMORY_DRAIN_RESTART_MS when drainGraceMs is omitted. In src/server/management/system-routes.ts, the omitted-body path passes no grace option. tests/server/system-restart-admission.test.ts confirms that the default remains 60,000 ms. An explicit {"drainGraceMs":2000} addresses the case only when the caller uses the new option. The dashboard, CLI, and other default callers described by #6643 can still refuse new requests for the full 60 seconds.

Resolution

Apply the #6643 handoff bound to the affected default restart path, including omitted-body and service restart callers, while keeping drain cleanup separate. Add regression tests that reproduce the original default-call failure and verify a refusal or handoff bound of no more than 2,000 ms. If the intended requirement is only opt-in behavior, update #6643 before merging so its coding requirement matches that contract.

Full details: Docstring Coverage

Explanation

Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 4 files. (11 skipped: 11 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7ebc4f142

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +190 to +193
if (!body || typeof body !== "object" || Array.isArray(body)) {
return jsonResponse({ success: false, error: "Invalid restart JSON body." }, 400, req, config);
}
const drainGraceMs = (body as Record<string, unknown>).drainGraceMs;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject unknown restart body fields

Reject keys other than drainGraceMs before accepting the restart. Currently a typo such as {"drainGraceMS":2000} passes the object check, is treated like {}, and immediately starts a restart with the 60-second default instead of returning the documented 400; because this is a mutating endpoint, silently ignoring the requested option can leave the caller waiting through an unintended drain. Add an explicit key whitelist and cover this case in the route tests so the implementation matches the public API contract.

AGENTS.md reference: docs-site/AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

@lidge-jun
lidge-jun merged commit a12bb1f into dev Oct 8, 2026
35 checks passed
@lidge-jun
lidge-jun deleted the codex/carry-6703-restart-drain-grace branch October 8, 2026 01:01
lidge-jun added a commit that referenced this pull request Oct 8, 2026
…atchet (#6733)

#6721 and #6724 each added one explicit registration and together took scripts/test-layout/layout.json to 2000 lines, over the 1999-line NEW_OVERSIZED ratchet, so file-size ratchet: repository fails on dev and on every PR's merge ref. Fold forty single-entry lines into four-per-line rows (the file's existing compact form). The parsed mapping is unchanged; the file drops to 1970 lines.
@lidge-jun lidge-jun mentioned this pull request Oct 8, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants