[PauloFH]

Summary

The repo ships docker-compose.yml and fly.toml, and DOCKER_GUIDE.md already warns that "stock Docker / Kubernetes OmniRoute is one Node process + one SQLite writer" — but there are no manifests. Anyone running this on k3s or k8s writes them from scratch, against a topology whose constraints are invisible until they bite:

• a second replica against the same volume corrupts the database;
• an ingress that buffers responses withholds the SSE stream until the provider turn ends, which reads as a hung request, not as broken streaming;
• a liveness probe pointed at /api/monitoring/health does real DB work on every kubelet tick and restarts the only pod under load.

So the topology is encoded, not documented. replicas: 1, Recreate and ReadWriteOnce are fixed in the base, in the chart and in the generator — not exposed as options anywhere. And the chart and the spec schema refuse to render a configuration that fails at runtime, naming the field rather than emitting something that breaks later:

Rejected	Because
replicaCount > 1	two pods on one volume corrupt the SQLite file
access mode ≠ ReadWriteOnce	a shared-write volume invites the same second writer
heap ceiling ≥ memory limit	the kernel OOM-kills the pod instead of Node raising a recoverable heap error (#7849)
shutdownTimeoutMs ≥ grace − preStop	the kubelet SIGKILLs mid-drain, so in-flight SSE is cut rather than drained
TLS with no Ingress	terminating TLS at something that does not exist

Four artifacts, one topology:

• deploy/kubernetes/ — Kustomize base plus k3s (Traefik + local-path) and generic k8s (ingress-nginx) overlays.
• deploy/helm/omniroute/ — the same topology, values-driven.
• omniroute deploy k8s — prints manifests built from a running instance's settings, so ports, origins and flags are not transcribed by hand. It writes to stdout and touches nothing, so it composes: omniroute deploy k8s --target local | kubectl apply -f -.
• docs/ops/KUBERNETES_DEPLOYMENT_GUIDE.md — 416 lines covering deploy, ingress/SSE, probes, upgrades, storage/backups, sizing and troubleshooting.

local and vps differ only in defaults. local assumes k3s on the same machine. vps assumes a hostname in front of a cluster other people can reach, so TLS is on, REQUIRE_API_KEY=true, the cluster's default StorageClass binds, and LIVE_WS_ALLOWED_ORIGINS is derived from the host — without that, the dashboard's live view is rejected by the origin check on the very host it is served from.

Probes follow docs/ops/MONITORING_GUIDE.md ("Kubernetes probe recommendations", merged in #10297): /healthz for startup and readiness, /livez for liveness, never /api/monitoring/health. The nginx overlay disables response buffering; the Traefik idle-timeout caveat is documented for k3s, since that one is an entrypoint setting and not an Ingress annotation.

Provisioning is advisory on purpose. omniroute deploy k8s provision prints the k3s commands and never runs them or connects to a server. Doing it directly would mean shipping an SSH client and holding a private key — a standing surface for a one-off task — so it follows the shape contrib/vps already uses for a VPS install. The VPS plan covers the two steps easiest to get wrong by hand: --tls-san, without which the cluster is only reachable as 127.0.0.1 and a remote kubectl fails TLS verification, and rewriting the 127.0.0.1 server address in the kubeconfig k3s writes. Values rendered into those commands are constrained to hostname, version, username and context shapes, so a printed command cannot mean something other than what it looks like.

No new API route and no new dashboard surface. An earlier draft put this behind /api/deploy/* with a UI tab; it was dropped before landing. Applying manifests by clicking is at odds with GitOps — the desired state should be versioned, not produced by a form — and the routes existed only to serve the button, spawning kubectl with the host kubeconfig, which is a full cluster credential. The CLI reaches the same result without that surface, so routeGuard.ts is untouched.

Two fixes to the manifests themselves, both surfaced by the test that pins the generator against deploy/kubernetes/base:

• Dropped API_PORT from the base ConfigMap. Setting it moves the API to a separate port (src/lib/runtime/ports.ts:27) that neither the container nor the Service exposes, leaving that listener unreachable. Unset, the API and dashboard share PORT — which is what the manifests actually expose.
• Made the container port protocol explicit, so the committed manifest and the generator agree byte for byte.

deploy also joins --mcp in reserving stdout for data (bin/omniroute.mjs). Env-loading notices there made the YAML unparseable — control characters are not allowed — which broke the pipe the command exists for.

Related Issues

• Related to docs(ops): k8s probe recommendations (TCP liveness, HTTP /healthz readiness) #10297 — the probe recommendations these manifests implement
• Related to fix(backend): long-context compression can exhaust V8 heap and terminate OmniRoute #7849 — the V8-heap-vs-cgroup ceiling the spec schema now enforces
• Related to feat(backend): pluggable external durable-state DB backend (PostgreSQL + MySQL) — unified design & demand thread #8075 — multi-writer HA; until it ships, single-replica is the documented ceiling and this PR encodes it

Validation

• Change type: build-deploy (plus CLI)
• Focused tests and category gates from the golden path
• npm run lint
• Reconciled with the current active release base; focused checks rerun afterward
• Production-code changes include a new or updated automated test in this PR

Applied to a live cluster, not just a dry run. Single-node k3s v1.36.4+k3s1:

Checked	Result
PVC on local-path	Bound, 10Gi, RWO
Pod	1/1 Running, 0 restarts
Rollout	completed, Available=True
/healthz and /livez through the Service	HTTP 200 both
/v1/models	answers application JSON, so the app serves — not just the probes
Probe failure events	none recorded
Helm chart, --dry-run=server	accepted by the real API server

That run corrected two documentation errors a client-side dry run cannot catch: the health check used wget, and the published image ships neither wget nor curl (node is the only HTTP client in it); and an Ingress naming an IngressClass the cluster does not run is accepted by the API server and then silently routes nothing. Both are now in the guide, the second as a troubleshooting row.

Red first, per invariant: each manifest guard was verified to fail under a deliberate mutation of the file it protects — flipping replicas to 2, the access mode to RWX, proxy-buffering to on, liveness to /api/monitoring/health, or the shutdown budget past the grace period each turns exactly one test red, and only one. The generator/CLI suites found three real defects while being written: a fail placed at the top level of a _*.tpl file (Helm parses and discards it, so the replica guard was a silent no-op), context-name validation that rejected legitimate EKS contexts, and the stdout pollution above.

The automated review then found four more, fixed in the second commit: an address pattern that accepted a port or a bare IPv6 literal and rendered https://host:123:6443; a context pattern that rejected the ARN-shaped contexts managed clusters produce; the context being interpolated into a temp path, which the previous item would have turned into a directory traversal; and a quantity pattern that accepted K while rejecting k, the lowercase decimal thousand Kubernetes actually uses, so 512k was refused. Each has a regression case.

Rebased onto release/v3.8.51 and the full set rerun there: 54 tests passing, npm run typecheck:core clean, both overlays render with zero kustomize warnings, helm lint clean, check:docs-all / check:tracked-artifacts / check:cli-i18n passing, and omniroute deploy k8s | kubectl apply --dry-run=client -f - still creates all six objects.

npm run lint reports zero errors but exits 2, on this branch and on an untouched checkout of the base alike — suppressions left that do not occur anymore, unrelated to any file here.

Tests Added Or Updated

• tests/unit/k8s-deploy-manifests.test.ts — 12 cases over the committed manifests: single replica, Recreate, RWO, probe targets, drain budget, unbuffered SSE, secret hygiene, and that the chart's replica guard is actually reachable rather than discarded.
• tests/unit/deploy/kubernetes-manifest-generator.test.ts — 13 cases: generation invariants, target profiles, storage-class semantics, and parity between the generator and deploy/kubernetes/base.
• tests/unit/deploy/kubernetes-provision-plan.test.ts — 19 cases: the provisioning plan's correctness (including --tls-san and the kubeconfig rewrite), the deploy-spec guard rails, and the five review findings below.
• tests/unit/deploy/kubernetes-cli-command.test.ts — 10 cases driving the real binary: the stdout contract, exit codes, and argument rejection.

Coverage Notes

New production code is src/lib/deploy/kubernetes/ (three pure modules, 830 lines) and bin/cli/commands/deploy.mjs (250 lines).

The generator and the spec schema are pure — no cluster, no filesystem, no clock — and are covered directly by the two unit suites. The CLI is covered end to end by a suite that executes bin/omniroute.mjs and parses its stdout, which is what pinned the stdout-pollution bug; a mocked test would not have caught it. bin/omniroute.mjs gains one condition, exercised by that same suite.

Nothing existing loses coverage. Beyond the new files, the only changes are the two ConfigMap/protocol fixes, that one condition, a .prettierignore entry for Helm's Go templates (which Prettier's YAML parser cannot read), and two doc cross-references.

Reviewer Notes

• The chart's guard rails live in a define that deployment.yaml includes. Helm parses and then discards top-level code in a _*.tpl file, so a bare fail there is a silent no-op — the first version of this had exactly that bug, and replicaCount=2 rendered happily. A test now pins the include.
• storageClassName is omitted, never set to "". An empty string disables dynamic provisioning entirely; omitting the key binds the cluster default. The two look interchangeable, so a test covers the distinction.
• The k3s overlay assumes the Traefik that k3s ships by default. A cluster installed with --disable=traefik has no such IngressClass, and the API server accepts the Ingress anyway — it is created and routes nothing. Documented in troubleshooting rather than worked around, since the fix is to name the class the cluster actually runs.
• Context names accept : and /. Managed clusters produce them (an EKS context is arn:aws:eks:<region>:<account>:cluster/<name>). Safe because nothing here goes through a shell, the leading-alphanumeric requirement prevents a value being read as an option, and — the part that had to change for this to hold — the temp filename in the fetch-kubeconfig step is fixed rather than derived from the context, so a / cannot turn it into a path.
• Recreate means every upgrade is a short full outage — in-flight SSE and dashboard sessions drop, and the empty-endpoint window returns a controller 502. That is the SQLite topology, not a probe misconfiguration, and the guide says so where an operator will hit it.
• The --tls-san step matters more than it looks: omit it and the cluster works perfectly from the VPS itself while every remote kubectl fails TLS verification, which is a confusing way to lose an afternoon.
• No migration, no feature flag, no change to the request path, no new environment variable.

---

Status update — 2026-09-24 (head d905e453b)

⚠️ base-red inherited: #14547

The pre-merge question is answered

npm run lint's "zero errors but exit 2" is pre-existing, not introduced here. Verified on an untouched checkout of the base tip 30f7088c7 — not one commit of this branch applied: exit 2, zero errors, byte-identical message (There are suppressions left that do not occur anymore).

Every red on this PR is inherited, and Fast Quality Gates now proves it

Ref	Fast Quality Gates
base release/v3.8.51 alone (8feaafcf, run 35967005359)	8 gates: deps file-size mutation-test-coverage complexity-ratchets dead-code open-sse-typecheck typecheck:core ts7-diagnostics-ratchet
this PR (d905e453b, run 35980453742)	5 gates: deps file-size mutation-test-coverage open-sse-typecheck typecheck:core

The PR's five are a strict subset of the base's eight. The branch additionally passes complexity-ratchets, dead-code and ts7-diagnostics-ratchet, which the base fails — new-code mode scores only what this diff adds. file-size joined the list once the base was merged in: the drift it reports is the base's own chat.ts, which is why the same gate fails on the base alone.

The other jobs trace to #14547 / #14496 the same way: API Route Typecheck is auggie.ts + projectCombo.ts, the four Unit Tests fast-path shards are the provider-catalog drift (reproduced locally: provider-node-reserved-prefix asserts 415 against an expected 414), and Docs Gates is the stale migration count in README.md / AGENTS.md / llm.txt — the exact hard failure #14547 was opened for. None is fixed here: a base-red fix is its own freeze-gated PR.

What changed in this branch since the review

Probe budgets re-derived (5b6e50c4f). The startup-budget guard pinned its rationale to db.exec("VACUUM") inside startCleanupScheduler. #12821 — filed from this PR's own review thread — replaced that with bounded incremental reclamation, so the pin stopped matching and the test went red on rebase. That is the pin working: the manifest's reason moved, so the manifest was re-argued rather than the assertion relaxed.

Cold start still blocks, for reasons that hold on this base: opening the database runs the legacy call-log offload, which checkpoints the WAL and then rebuilds the file with a full VACUUM (src/lib/db/core.ts), and the startup cleanup pass reclaims freed pages on the same synchronous connection. The blocking full VACUUM also still runs on a live, healthy process from vacuumScheduler at the configured hour — which is what the tcpSocket liveness default exists for, so the liveness guard pins it there now. Manifest, chart, generator and guide comments were updated to name the offload and the reclamation pass instead of the cleanup VACUUM. No manifest value changed: replicas, strategy, access mode, probe targets, budgets and annotations are untouched.

Docs frontmatter (0f6bfb1ba). source.config.ts now compiles every Markdown file under docs/ with fumadocs-mdx, and check:docs-frontmatter fails Docs Gates for any compiled doc without a title. The guide predates that gate on this base — this was the one Docs Gates failure that belonged to this PR, and it is fixed.

Verified at 5b6e50c4f against the then-current base 30f7088c7: 57/57 tests across the four suites, typecheck:core clean, check:complexity-ratchets green in absolute mode (complexity 2910 vs baseline 3218; cognitive 1322 vs 1437).

The base was merged into this branch at d905e453b (60 commits), so the branch is level with release/v3.8.51 and carries its reds directly. That is why the gate count moved from four to five between runs — nothing in this diff changed.

[Copilot]

🟡 Changes recommended

The provisioning plan’s input validation and command rendering have correctness issues that can generate broken commands or reject valid Kubernetes quantity/context inputs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds first-class Kubernetes deployment support to OmniRoute by introducing committed Kustomize manifests (base + k3s/k8s overlays), a Helm chart, and a new omniroute deploy k8s CLI generator/provision-plan, plus an ops guide and unit tests that pin the single-writer + SSE/probe invariants.

Changes:

• Add deploy/kubernetes/ (Kustomize base + k3s/k8s overlays) and deploy/helm/omniroute/ (Helm chart) that encode single-replica + RWO + Recreate + SSE/probe guardrails.
• Add pure TS deploy-spec validation + manifest generator + provisioning plan under src/lib/deploy/kubernetes/, surfaced via omniroute deploy k8s CLI command.
• Add docs (docs/ops/KUBERNETES_DEPLOYMENT_GUIDE.md) and tests to prevent drift/regressions; adjust CLI stdout handling so deploy output is pipe-safe.

Validation / Coverage (custom)

• Commands run (reviewer): not run in this review (see PR description for author-run results).
• Tests added/updated: tests/unit/k8s-deploy-manifests.test.ts, tests/unit/deploy/kubernetes-*.test.ts (3 files).
• Coverage result (custom): not measured in this review.

File summaries

File	Description
tests/unit/k8s-deploy-manifests.test.ts	Pins invariants across committed Kustomize + Helm artifacts (replicas/strategy/RWO/probes/SSE/secret hygiene).
tests/unit/deploy/kubernetes-provision-plan.test.ts	Verifies provision-plan output and deploy-spec guardrails (TLS SAN, kubeconfig rewrite, input safety).
tests/unit/deploy/kubernetes-manifest-generator.test.ts	Validates manifest generation invariants + parity vs deploy/kubernetes/base.
tests/unit/deploy/kubernetes-cli-command.test.ts	End-to-end CLI tests for stdout/data contract and deploy-spec validation errors.
src/lib/deploy/kubernetes/types.ts	Zod schema for deploy spec + memory-limit parsing helper used for guardrails.
src/lib/deploy/kubernetes/provisionPlan.ts	Pure “print-only” k3s provisioning plan builder + input validators.
src/lib/deploy/kubernetes/manifestGenerator.ts	Pure manifest object builder + YAML renderer + target-specific defaults.
docs/ops/meta.json	Registers the new Kubernetes ops guide in ops docs metadata.
docs/ops/KUBERNETES_DEPLOYMENT_GUIDE.md	End-to-end Kubernetes deployment guide (k3s + k8s), covering SSE/Ingress and probes.
docs/guides/DOCKER_GUIDE.md	Cross-links to Kubernetes deployment guide from the Docker guide.
deploy/kubernetes/base/namespace.yaml	Adds base Namespace manifest.
deploy/kubernetes/base/configmap.yaml	Adds base ConfigMap (non-secret runtime config) with API_PORT explicitly unset.
deploy/kubernetes/base/pvc.yaml	Adds base PVC pinned to ReadWriteOnce.
deploy/kubernetes/base/deployment.yaml	Adds base Deployment pinned to replicas=1 and strategy=Recreate, with probe/lifecycle defaults.
deploy/kubernetes/base/service.yaml	Adds base Service exposing http + live-ws ports with explicit TCP protocols.
deploy/kubernetes/base/secret.example.yaml	Adds Secret template with placeholders only (not referenced by kustomization).
deploy/kubernetes/base/kustomization.yaml	Adds base kustomization wiring (intentionally excludes secret.example.yaml).
deploy/kubernetes/overlays/k3s/kustomization.yaml	k3s overlay resources + patches (storageClass + resources trim).
deploy/kubernetes/overlays/k3s/ingress.yaml	Traefik ingress overlay with streaming caveats documented.
deploy/kubernetes/overlays/k3s/pvc-storageclass.yaml	k3s overlay pinning StorageClass to local-path.
deploy/kubernetes/overlays/k3s/resources.yaml	k3s overlay resource sizing patch.
deploy/kubernetes/overlays/k8s/kustomization.yaml	Generic k8s overlay resources + PVC patch.
deploy/kubernetes/overlays/k8s/ingress.yaml	ingress-nginx ingress overlay with buffering/timeouts configured for SSE.
deploy/kubernetes/overlays/k8s/pvc-storageclass.yaml	Generic k8s PVC patch leaving storageClassName unset (bind default).
deploy/kubernetes/overlays/k8s/liveness-tcp.yaml	Optional TCP liveness patch for event-loop stall resilience.
deploy/helm/omniroute/Chart.yaml	Introduces the Helm chart definition.
deploy/helm/omniroute/values.yaml	Chart values encoding single-writer constraints and runtime config knobs.
deploy/helm/omniroute/.helmignore	Adds Helm ignore rules.
deploy/helm/omniroute/templates/_helpers.tpl	Adds chart guardrails + common naming/label helpers.
deploy/helm/omniroute/templates/deployment.yaml	Adds Deployment template mirroring k8s invariants and probes.
deploy/helm/omniroute/templates/service.yaml	Adds Service template exposing http + live-ws.
deploy/helm/omniroute/templates/configmap.yaml	Adds ConfigMap template (non-secret env) matching runtime expectations.
deploy/helm/omniroute/templates/secret.yaml	Adds optional Secret template with guardrails to avoid conflicting secret configs.
deploy/helm/omniroute/templates/pvc.yaml	Adds PVC template with helm.sh/resource-policy: keep.
deploy/helm/omniroute/templates/ingress.yaml	Adds Ingress template controlled by values.
deploy/helm/omniroute/templates/serviceaccount.yaml	Adds ServiceAccount template with automount disabled.
deploy/helm/omniroute/templates/NOTES.txt	Adds post-install notes highlighting outage semantics and key settings.
changelog.d/features/kubernetes-deploy-manifests.md	Adds changelog entry describing Kubernetes support artifacts.
bin/omniroute.mjs	Treats deploy as a “stdout is data” mode (redirects logs to stderr).
bin/cli/commands/registry.mjs	Registers the new deploy command in the CLI registry.
bin/cli/commands/deploy.mjs	Implements omniroute deploy k8s + provision command surface and spec validation.
.prettierignore	Excludes Helm templates from Prettier YAML parsing.

Review details

Suppressed comments (1)

src/lib/deploy/kubernetes/types.ts:146

• parseMemoryToMib() mirrors the same K vs k issue (it parses K but not k). If a user supplies a valid k-suffixed quantity for limitsMemory, the heap-vs-cgroup validation will silently skip (match fails) or miscompute. Parse lowercase k instead.

• Files reviewed: 39/42 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[insoln]

Notes from a production deployment running exactly this topology (single replica, Recreate, RWO PVC on Longhorn, ingress-nginx) — three things that bit us and may be worth folding in:

1. Consider splitting the API and dashboard into two Ingresses. The single-Ingress layout exposes the dashboard with the same (lack of) auth as /v1, while the image boots with INITIAL_PASSWORD=CHANGEME unless overridden. We run /v1 + /api/v1 on one Ingress (API-key auth, buffering off, long timeouts) and the dashboard on a separate Ingress behind oauth2-proxy — that split is impossible to express with one Ingress resource, and it's the difference between "dashboard leaked" and "dashboard behind SSO" for anyone applying these manifests on a reachable cluster. Even just a commented-out second Ingress + a line in the guide would help.

2. proxy-body-size: 32m is below the app's own default body cap. OMNIROUTE_CHAT_HARD_MAX_BODY_BYTES defaults to 50 MB, so nginx starts returning 413s the application itself would have accepted (and the app's structured 413 handling — compact-required semantics — never gets a chance to run). Suggest ≥ 50m, or a comment tying the annotation to that env var.

3. Sizing guide: mention the chat-admission knobs for agentic workloads. With parallel coding-agent sessions (Claude Code with MCP tools), effectively every request classifies as heavyweight (>64 tools / >32k estimated tokens), and the OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT=1 default turns a second concurrent session into a stream of retryable 503s (chat_admission_busy, queue_timeout). We run MAX_HEAVY_IN_FLIGHT=4 / ADMISSION_QUEUE_MS=20000 / ADMISSION_MAX_QUEUED_BYTES=32M on a 4Gi-limit pod without issue. A row in the sizing table would save k8s adopters a confusing first day.

Also, for the storage section: PVC sizing currently has to budget for unbounded telemetry growth (conversation_turn_nodes, ~190 MB/day under agentic load, no retention path) — tracked in #12453.

[PauloFH]

Thanks — this is the kind of feedback that only comes from actually running it. Items 1, 2 and 4 are in as of b6087dc47.

proxy-body-size was a real defect: 32m against a 50 MB CHAT_HARD_MAX_BODY_BYTES default meant nginx rejected bodies the app would have taken, and the app's compact-required 413 never ran. Fixed in both the overlay and the generator, with a test that reads the default out of chatBodyAdmission.ts rather than restating it, so raising one side alone now fails.

The Ingress split is now overlays/k8s/ingress-split-example.yaml plus a guide section — /v1 + /api/v1 on an API host with REQUIRE_API_KEY, dashboard on its own host behind an external IdP. Not wired into the kustomization, same convention as the liveness-tcp patch. You are right that one Ingress cannot express it; the INITIAL_PASSWORD=CHANGEME default makes that worth calling out explicitly rather than leaving implicit.

Telemetry sizing now says the PVC is sized by retention window, citing #12453 and your ~190 MB/day figure.

On item 3, I think the behaviour changed under you. On release/v3.8.51:

function resolveLegacyCountCap(): number {
  const raw = process.env.OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT;
  if (raw === undefined || raw.trim() === "") return Number.MAX_SAFE_INTEGER;
  return CHAT_MAX_HEAVY_IN_FLIGHT;
}

Left unset, the legacy count cap resolves to unlimited and the auto-derived ingest byte budget is what gates — the source comment says as much ("the default on every deployment that produced the multi-subagent 503 storm ... resolves to effectively unlimited"). So MAX_HEAVY_IN_FLIGHT=4 enables the count cap at 4 rather than relaxing anything; unset is the looser setting. Your report mentions Hub 3.8.49, which predates that.

I documented both knobs in the sizing section with that caveat, since the variable name reads like a ceiling you raise. If chat_admission_busy is still showing up on a current build with it unset, that is the byte budget binding and a different conversation — worth its own issue, and I would be interested in the numbers.

[insoln]

Confirmed: we're on 3.8.50, where the bundle still reads o(process.env.OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT, 1) — a hard default of 1. So both hold: on ≤3.8.50 =4 relaxes the cap, on ≥3.8.51 it enables one. We'll drop the var on upgrade. Thanks for landing 1/2/4.

[insoln]

Ran this exact topology in production today (generic k8s, ingress-nginx, Longhorn, ~1 GB storage.sqlite, agentic load) and hit the failure the liveness comment hints at — hard enough that I think two defaults should flip. Sequence: HTTP liveness (3 s timeout) killed the pod mid-startup-VACUUM → each restart grew the WAL (it reached 999 MB) → every checkpoint then stalled the event loop for ~a minute → readiness flapped → nginx 503s. tcpSocket liveness plus a bigger startup budget broke the loop. Three inline suggestions with the numbers.

[diegosouzapw]

Sweep-reds babysit (round 5 Fase B)

No push. HEAD remains b6087dc471. Did not merge origin/release/v3.8.51 (base-red #12581).

Remaining reds are inherited — not this PR's fault

• No new ESLint warnings: lint itself was eslintWarnings=0, eslintErrors=0. The job failed on CodeQL ratchet 12 > baseline 11 (repo-wide open alerts, not this k8s-deploy diff). Same class of inherited ESLint-job red as the tip.
• Fast Quality Gates: 1 gate(s) failed: complexity-ratchets. CI new-code mode walks the merge commit vs the current release/v3.8.51 tip (file-size chat.ts drift / 🔴 Release branch not green: release/v3.8.51 #12581). Locally, this PR's own 6 in-scope files vs merge-base 7f25d67d076b are 0/0 (complexityNewCode=0, cognitiveComplexityNewCode=0).

Not merging. Needs the base-red drain, not a k8s rewrite.

[PauloFH]

Both defaults flipped, plus a third that fell out of testing. Pushed as e3e4960b1.

I went looking for the mechanism in the repo before changing anything, and it is there in two places: src/lib/db/cleanup.ts runs db.exec("VACUUM") from the startup cleanup scheduler, and closeDbInstance() in src/lib/db/core.ts runs wal_checkpoint(TRUNCATE) on the way out. better-sqlite3 is synchronous, so both block the event loop for as long as they take — /livez included, since it shares that loop. The crashloop you hit is not an unlucky timeout; it is the expected outcome of pointing an HTTP probe at a process that legitimately stops answering HTTP.

Liveness is now tcpSocket. The asymmetry argument is the one that decided it, and it is in the manifest as a comment so the next person to "fix" it back has to read the reason first. overlays/k8s/liveness-tcp.yaml is inverted into liveness-http.yaml rather than deleted — HTTP does catch a wedged server that TCP cannot, so it stays as an opt-in with the condition under which it is defensible spelled out (small database, fast local storage, verified against your own PRAGMA wal_checkpoint(TRUNCATE) timings).

Startup budget 150s → 10 min, for the reason you give: the failure is self-amplifying, and readiness holds traffic back for exactly as long as it takes, so the budget costs nothing.

terminationGracePeriodSeconds 90 → 300. Your point about the close-time checkpoint is confirmed by the ordering in gracefulShutdown.ts: waitForDrain() is what SHUTDOWN_TIMEOUT_MS bounds, and cleanup() runs after it with no timeout of its own. So the drain budget never covered the checkpoint at all. The ConfigMap comment is retouched to say what the headroom is actually for.

A third one, found by running it. On a real cluster the startup probe logged Get "http://…/healthz": context deadline exceeded on a trivial boot with an empty database — the kubelet's probe timeout defaults to 1s, and /healthz shares the same event loop as everything else. Startup now sets timeoutSeconds: 3 explicitly.

Verified on k3s v1.36.4+k3s1, not just rendered: pod 1/1 Running, 0 restarts, /healthz and /livez 200 through the Service. And because a 300s grace period reads like a five-minute upgrade stall, I timed a real delete — the pod terminated in 25s. It is a ceiling, not a delay; the guide now says so, together with the Recreate interaction for the case where a pod really does hang.

The tests pin each default to the source fact it derives from rather than to the number: one greps the VACUUM call out of cleanup.ts, another asserts the drain-then-cleanup ordering in gracefulShutdown.ts, so moving the code without moving the manifest fails. Checked by mutation — each of the five reverts fails at least one test.

Credited you as co-author on the commit; the tcpSocket rationale is yours.

(For anyone tracking CI here: HEAD moved from b6087dc471 to e3e4960b1 with this change.)

[diegosouzapw]

This is an exceptionally well-executed PR — the self-review notes (the silently-discarded
Helm fail, the directory-traversal fix on the kubeconfig context path, the deliberate
decision to skip a dashboard button to avoid exposing the host kubeconfig via HTTP) show real
rigor. Ran all 4 new test files live against this branch's head: 57/57 pass. Only thing to
double-check before merge is that npm run lint's "zero errors but exit 2" (stale
suppressions) is confirmed pre-existing on the current release tip and not something this
branch introduced. Otherwise this looks ready.

[diegosouzapw]

Re-homed to release/v3.8.52: v3.8.51 entered its release freeze, so the branch now belongs to the release captain and development continues on the next cycle. Nothing is wrong with this PR — it just needed a live base. No action needed from you; CI will re-run against the new base.

[insoln]

Concrete event-loop stall relevant to the TCP-liveness default

Filed separately as #16039: clicking USD cost on a provider card calls /api/usage/provider-window-costs, whose detailed cost-matching fallback repeatedly scans the recorded-cost array from the beginning. The synthetic reproduction uses the actual matcher: 8,000 rows produce 32,011,999 candidate visits; 16,000 produce 128,023,999. The synchronous work runs on the HTTP-serving event loop, including when invoked through Promise.all.

This is an application defect, not introduced by this deployment PR, and is separate from #16036/#16037. Linking it here because it makes the recovery tradeoff in this PR concrete:

1. A dashboard analytics operation can monopolize the event loop without a slow disk, checkpoint or VACUUM.
2. HTTP readiness fails and removes the sole replica from Service endpoints.
3. A tcpSocket liveness probe can still succeed while the HTTP event loop is unresponsive: a successful TCP handshake is not evidence that the application can serve requests.
4. Consequently, the default offers no bounded automatic recovery from this class of hung-but-listening server. The outage lasts until the calculation finishes or an operator intervenes.

I checked the current head 326b884373464e829d164acb58a4b00fbfafde92: the base manifest explicitly acknowledges this tradeoff, Helm defaults probes.livenessType to tcp, and the optional HTTP overlay is not enabled by default. This is not a claim that the limitation was undocumented. However, the initial PR summary still says /livez is used for liveness, while the later status update/current files say TCP; please align that summary so operators do not infer automatic HTTP-stall recovery from it.

Suggested follow-up for the deployment policy:

• Document this CPU-bound analytics example alongside the SQLite maintenance rationale: fast local storage does not eliminate application event-loop stalls.
• Make the recovery choice and its unbounded-outage consequence explicit. For deployments opting into HTTP liveness, use the lightweight /livez endpoint, with startup/runtime budgets justified by measured blocking work. Do not switch to the DB-heavy /api/monitoring/health endpoint or blindly shorten timeouts.
• Add a fault-injection acceptance case: deliberately block the application event loop after startup, observe readiness dropping, and verify the intended recovery behavior for each probe mode. Keep the separate long-startup/maintenance case so the solution does not introduce restart loops.

The primary fix belongs in the analytics algorithm (#16039); probes only determine what happens when such a defect slips through. This comment intentionally contains only public-code analysis and synthetic evidence, no production logs or private deployment details.

[diegosouzapw]

Hi @PauloFH — closing this pull request to clear the open queue. The branch feat/k8s-deploy is kept, so the commits stay. Comment here if it should be reopened.

[diegosouzapw]

Hi @PauloFH — reopening this pull request. The close was a mistake, and it is back in the merge queue. The branch feat/k8s-deploy was kept.

[diegosouzapw]

Re-homed to release/v3.8.53: v3.8.52 entered its release freeze, so the branch now belongs to the release captain and development continues on the next cycle. Nothing is wrong with this PR — it just needed a live base. No action needed from you; CI will re-run against the new base.