Skip to content

Record the #284 host rollout on the WSL workstation and the early expiry of silence 87eabf8a - #287

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/paper-alert-host-rollout-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/paper-alert-host-rollout-20260925

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

What

This records #284's post-merge host steps on the WSL workstation (nativestack-5975wx-20260925) in observability/backends/paper-alert-file-sd-host-20260925.json, as promised in #284. It also adds a pointer to the receipt from the backends README.

How

Surgical apply, not a configure.py re-render. The live WSL2 config is a host-adapted rendering with shifted loopback ports: collector 28889/28888, qdrant 26333, vllm 18231, httpcheck 24333. All WSL 2 distributions share one network namespace (adoption/platforms/linux-wsl2.md), and the renderer has no host-override input, so a re-render would reset those ports.

Staging and diff. Main's templates were rendered into a scratch directory and diffed against the live files.

  • The staged rules differ from the live file only at the two descriptions, the expr and the summary.
  • The staged prometheus.yml differs only by the 5-line adaptive-paper file_sd job.
  • The live file had no adaptive-paper job at all. That is why absent(paper_trial_active) fired permanently here.

Install and reload.

  1. promtool 3.14.0 checked both staged files (14 rules; config valid).
  2. Both were installed by atomic rename, and <config-root>/adaptive-paper-targets.json was created as [].
  3. A SIGHUP reload followed: lastConfigTime moved to 21:01:32Z with reloadConfigSuccess true.

Read-back through the Prometheus API. The rule carries the new expr, is inactive and ok with 0 alerts, and was last evaluated at 21:01:52Z. The adaptive-paper job has 0 active and 0 dropped targets. The expression result and ALERTS{alertname="EquitiesPaperMetricsMissing"} are both empty.

Silence 87eabf8a. It matched alertname=EquitiesPaperMetricsMissing and was due to end 2026-10-02. It was expired through the Alertmanager v2 API: the DELETE returned 200 and the silence is expired as of 21:02:18Z. Alertmanager then holds 0 alerts for that alert name.

Limits

Checks

validate.py, evidence_manifest.py --check and validate_convergence.py --all-recorded pass. The full silence UUID is omitted, because the publication validator treats a UUID as a possible local session identifier.

🤖 Generated with Claude Code

…iry of silence 87eabf8a

Adds observability/backends/paper-alert-file-sd-host-20260925.json.

- #284's lines were applied surgically to the live config, and
  configure.py was not re-run. The WSL2 rendering carries shifted loopback
  ports (collector 28889/28888, qdrant 26333, vllm 18231, httpcheck 24333),
  and the renderer cannot override them.
- Both staged files passed promtool 3.14.0 before install. After a SIGHUP,
  reloadConfigSuccess was true.
- The Prometheus API read back the new expr, inactive, with 0 adaptive-paper
  targets, and an empty ALERTS query.
- Silence 87eabf8a (alertname=EquitiesPaperMetricsMissing, due to end
  2026-10-02) was expired early; Alertmanager then held 0 alerts for it.

The live firing path still waits for a registered exporter, since the
paper keys return HTTP 401.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement label Sep 25, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Foundation ACK from native-agent-stack-9a, for head bbae57f3. The rollout record is sanitized (no host paths, and the silence id is shortened to a prefix). It separates the surgical apply from a re-render, keeps the host port adaptations, and has read-backs from the Prometheus API (rule inactive/ok, 0 adaptive-paper targets, empty expression and ALERTS results) and from Alertmanager (silence 87eabf8a expired at 21:02:18Z, 0 alerts). Its limitations say the firing path isn't exercised live, since nothing is registered while the paper keys return 401, and that the renderer has no host-override input. I'm taking that renderer gap as a foundation follow-up.

@seathatflowsinourveins
seathatflowsinourveins merged commit ea710ec into main Sep 25, 2026
23 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/paper-alert-host-rollout-20260925 branch September 25, 2026 21:30
seathatflowsinourveins added a commit that referenced this pull request Sep 25, 2026
…e-render (#290)

* Observability renderer: host loopback port overrides that survive a re-render

All WSL 2 distributions share one network namespace, so the workstation
moved five scraped services off their template ports (collector 18889 and
18888, Qdrant 16333, vLLM 8231, the 14333 HTTP check). configure.py had no
input for that, so the host's live config was hand-adapted and any re-render
would reset those ports; the #284 rollout (#287) had to be applied
surgically for that reason.

configure.py --port-overrides FILE takes a JSON object of template port to
host port and rewrites every 127.0.0.1:<port> address in the configs and
units in one pass. It keeps the map as <config-root>/port-overrides.json,
which later renders reuse. It refuses the whole render (exit 2, nothing
written) for a stale override, for a port that also appears outside a
127.0.0.1:<port> address (Grafana's http_port, Loki's listen ports), for a
host port that collides with another rendered address, and for invalid
input. The map is validated against the full render before any file is
written.

Rendering main with the workstation's map into a scratch root reproduces
the live files: 14 of 15 byte-identical, and ecosystem-prometheus.yml equal
as parsed YAML.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Port overrides: address the independent review

- Medium: the bare-port check scanned the caller's root paths, so a data
  root such as .../18888-data refused an 18888 override with a misleading
  cause. The checks now ignore the four root paths, and the refusal names
  the file that has the bare port.
- Low: a non-ASCII digit key (such as U+00B2) passed str.isdigit() and
  crashed int() with exit 1. Keys must now be ASCII digits (exit 2).
- Low: the README lists malformed input among the refusals.
- Tests: each template output without overrides equals the template with
  only its path tokens replaced; a root path containing the port digits
  renders; a mixed valid and stale map is refused; a non-ASCII digit key
  is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant