Skip to content

Live-reload passthrough-watcher healthPath instead of requiring a restart - #58

Merged
yuki-ariheber merged 4 commits into
mainfrom
fix/passthrough-watcher-config-checksum
Aug 30, 2026
Merged

Live-reload passthrough-watcher healthPath instead of requiring a restart#58
yuki-ariheber merged 4 commits into
mainfrom
fix/passthrough-watcher-config-checksum

Conversation

@yuki-ariheber

@yuki-ariheber yuki-ariheber commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • HEALTH_URL in the passthrough-watcher's watch.sh was set once at process start from the ConfigMap-baked script text, so a ConfigMap update (e.g. Repoint bypass watcher healthPath to /health/passthrough #56's /health/live/health/passthrough fix) never reached an already-running watcher pod.
  • Confirmed in prod: every tenant's watcher pod except two (restarted manually during investigation) is still polling /health/live, which always returns 200 regardless of dependency health or the bypass toggle — so both the manual bypass toggle and automatic failover-on-outage have been silent no-ops fleet-wide since before Repoint bypass watcher healthPath to /health/passthrough #56 shipped.
  • Fix: healthPath is now re-read from a file each loop, exactly like mode already is (see the existing "Re-read each loop iteration, so a values change applies live" comment). No restart, no checksum annotation — a ConfigMap change (from a values change, or a chart upgrade like this one) takes effect within one poll interval everywhere, automatically.

Test plan

  • helm template renders cleanly
  • Manually verified in prod (jsjvnsg-prod-aws-n-virginia, aejupnv-consumer-1-aws-us-east-2): after this ConfigMap update rolls out, watcher correctly picks up /health/passthrough within one poll interval, no pod restart — confirmed it flips the Service selector to passthrough for an account with the bypass toggle enabled, and reverts correctly

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Passthrough watcher deployments now automatically restart when their configuration changes, ensuring updated settings take effect.

watch.sh reads mode from a file each loop, but HEALTH_URL is a plain
shell variable set once at process start from the ConfigMap-baked
script text. A ConfigMap update (e.g. the healthPath fix in #56) never
reaches an already-running watcher pod without a restart, so any
tenant whose pod predated that fix was silently stuck polling
/health/live — which always returns 200 — making both the manual
bypass toggle and automatic dependency-outage failover no-ops.

Add a checksum/config annotation on the pod template so config changes
trigger a real rollout.
@coderabbitai

coderabbitai Bot commented Aug 30, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4db984f5-c0ba-4563-b151-815b3431df8e

📥 Commits

Reviewing files that changed from the base of the PR and between da1b52f and fb41122.

📒 Files selected for processing (1)
  • charts/yuki/templates/passthrough-watcher-deployment.yaml

📝 Walkthrough

Walkthrough

The passthrough watcher pod template now includes a checksum annotation for its ConfigMap. Changes to the annotation trigger a Deployment rollout.

Changes

Watcher configuration rollout

Layer / File(s) Summary
Trigger rollout on configuration changes
charts/yuki/templates/passthrough-watcher-deployment.yaml
The watcher pod template includes the passthrough watcher ConfigMap checksum annotation.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to da1b5

The watcher now applies health-path ConfigMap changes without requiring a pod restart; no actionable merge-blocking risk remains after normal checks and review.

Poem

A rabbit checks the watcher chart,
A checksum marks the changing part.
When settings shift, pods wake anew,
The rollout follows the ConfigMap through.
Soft paws approve the tidy start.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title claims that healthPath is live-reloaded without a restart. The changeset only adds a ConfigMap checksum annotation that triggers a Deployment rollout when configuration changes. Change the title to describe the checksum annotation and automatic watcher Deployment rollout, or update the changeset to implement live re-reading of healthPath.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI

Comment @coderabbitai help to get the list of available commands.

HEALTH_URL was baked in once at process start, so a ConfigMap change
(like #56's /health/live -> /health/passthrough fix) never reached an
already-running watcher pod. Re-read healthPath from a file each loop,
the same way mode already is, so a values change (or this ConfigMap
already having drifted ahead of a stale pod) takes effect within one
poll interval — no restart, no checksum annotation needed.
@yuki-ariheber yuki-ariheber changed the title Roll passthrough-watcher on ConfigMap change (checksum annotation) Live-reload passthrough-watcher healthPath instead of requiring a restart Aug 30, 2026
@yuki-ariheber
yuki-ariheber merged commit 60f6ec0 into main Aug 30, 2026
1 check was pending
@yuki-ariheber
yuki-ariheber deleted the fix/passthrough-watcher-config-checksum branch August 30, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant