Skip to content

docs: update production shutdown sequence with new lifecycle hooks #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ankurs merged 3 commits into from
Apr 28, 2026
Merged

docs: update production shutdown sequence with new lifecycle hooks #82

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6c6e6c3
docs: update production shutdown sequence with new lifecycle hooks
ankurs Apr 27, 2026
4b7a049
fix: address review comments
ankurs Apr 27, 2026
d7f3073
fix: distinguish ColdBrew CBPreStopper from Kubernetes lifecycle.preStop
ankurs Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions howto/production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,17 +174,21 @@ Set `terminationGracePeriodSeconds` to at least `SHUTDOWN_DURATION_IN_SECONDS` t

## Graceful shutdown tuning

ColdBrew's shutdown sequence (bounded by `SHUTDOWN_DURATION_IN_SECONDS`, default 15s):
When pod termination begins, Kubernetes runs any configured `lifecycle.preStop` hook, then the kubelet sends `SIGTERM`. ColdBrew's in-process shutdown sequence then begins, bounded by `SHUTDOWN_DURATION_IN_SECONDS` (default 15s). Note: the `PreStop(ctx)` hook below refers to ColdBrew's [CBPreStopper] interface, not Kubernetes' `lifecycle.preStop`:

1. Receive SIGTERM from Kubernetes
1. `PreStop(ctx)` on `CBPreStopper` services — deregister from service discovery, flush buffers
2. `FailCheck(true)` on `CBGracefulStopper` services — `/readycheck` starts failing
3. Wait `GRPC_GRACEFUL_DURATION_IN_SECONDS` (default: 7s, included in shutdown timeout) for the load balancer to drain
4. Shutdown admin server if configured (`ADMIN_PORT`)
5. Shutdown HTTP server (stop accepting new requests)
6. `GracefulStop()` gRPC server (finish in-flight RPCs, reject new ones)
7. Force-stop gRPC server if graceful shutdown didn't complete in time
8. Call `Stop()` on `CBStopper` services — close database pools, flush metrics, drain message producers
9. Exit
4. Cancel worker context, wait for workers to exit
5. Shutdown admin server if configured (`ADMIN_PORT`)
6. Shutdown HTTP server (stop accepting new requests)
7. `GracefulStop()` gRPC server (finish in-flight RPCs, reject new ones)
8. Force-stop gRPC server if graceful shutdown didn't complete in time
9. Call `Stop()` on `CBStopper` services — close database pools, flush metrics, drain message producers
10. `PostStop(ctx)` on `CBPostStopper` services — final cleanup, audit log close
11. Exit
Comment thread
ankurs marked this conversation as resolved.

See [Shutdown Lifecycle](/howto/signals) for the full interface table and [Readiness Patterns](/howto/readiness) for combining workers with health checks.

Tune these values based on your service:

Expand Down Expand Up @@ -633,3 +637,4 @@ These are your responsibility to handle at the infrastructure level:
- [Workers](/howto/workers) — background goroutine management with restart and metrics

[ColdBrew cookiecutter]: /getting-started
[CBPreStopper]: https://pkg.go.dev/github.com/go-coldbrew/core#CBPreStopper
Loading