Paper alert: scrape only registered adaptive-paper exporters, and alert only on a registered silent one

[seathatflowsinourveins]

Why

EquitiesPaperMetricsMissing was absent(paper_trial_active) over a static 127.0.0.1:18890 scrape target. It fired permanently on every host with no trial running, and this workstation has carried Alertmanager silence 87eabf8a for it, expiring about 2026-10-02.

This PR is lane:shared: it changes foundation-owned backend files and a trading-owned exporter. The split and the review points were agreed with the foundation lane (9a).

What

Scrape job and exporter registration

• Scrape job: job adaptive-paper now reads file_sd_configs from @CONFIG_ROOT@/adaptive-paper-targets.json. configure.py ships the file as [] and keeps an existing one.
• Registration: metrics.py --file-sd PATH registers the exporter's own 127.0.0.1:<port> after the listener binds.
  • The update runs under a sibling flock and writes a temporary file in the same directory, then fsync and os.replace.
  • A list that isn't a file_sd target list refuses start-up (exit 2) rather than leaving the exporter unmonitored.
  • The SIGTERM handler is installed before registration.

Deregistration

• Deregistration happens only on a clean SIGTERM or SIGINT stop of a trial that finished cleanly: trial.json phase finished without a failed status, or no trial.json.
• These cases stay registered, so the alert fires:
  • a crashed or killed exporter;
  • a trial at starting, needs_attention or held_overnight;
  • a trial that finished with needs_attention or failed.
• metrics.py --deregister --port N --file-sd PATH removes a target after the operator has recovered the trial.
• This also partly closes the documented stuck-starting gap: stopping the exporter of a stuck trial now raises the alert rather than hiding the trial.

The rule

• It is now up{job="adaptive-paper"} unless on(job, instance) paper_trial_active, for 2m.
• It fires for a registered exporter that is down or unreachable, and for another process answering on its port.
• It stays silent while nothing is registered.
• The generic EcosystemServiceUnavailable description and the new rule's description are both updated.

Checks (this workstation)

• tests.test_observability_backends_alerts: 18 OK, 0 skipped under uv with PyYAML.
  • Includes a promtool 3.14.0 test rules case over the rendered rule covering five scenarios: registered and down, which fires after 2m; registered and answering; registered but another process answering, which fires; nothing registered; and deregistered, where the stale series clears the pending alert.
  • Mutants: absent(paper_trial_active) and up{job="adaptive-paper"} == 0 both fail it.
• tests.test_adaptive_paper_metrics: 35 OK on system Python and on the pinned adaptive-paper runtime.
  • The new FileSdRegistrationTests run real exporter processes. They passed 5 of 5 repeated runs.
  • Six fix-revert mutants are all killed: always deregister, never deregister, no SIGTERM handler, target taken from the requested port, no group-key validation, and a non-fatal registration failure.
• Validators: scripts/validate.py, evidence_manifest.py --check and validate_convergence.py --all-recorded pass. The evidence pins of the eight changed files are refreshed.
• Full python3 -m unittest, system Python:
  • At ebabf20: 5,401 tests, 668 skipped, 1 failure. tests.test_adaptive_paper_source_hashes_manifest still pinned the old sha256 of metrics.py and its test.
  • 6daaf72 refreshes both source-hashes.json entries. Nothing pins that file's own digest, and the manifest test passes.
  • After the review fixes, the final head 12301be had 5,403 tests and 668 skipped. One test failed: tests.test_pre_commit_gate test_blocks_synthetic_key_and_passes_clean_commit, with "gitleaks: another scan holds the per-user lock". Another session's concurrent gitleaks scan held the lock, so this was environmental and not this change.
  • Re-run alone, tests.test_pre_commit_gate passed: 3 OK.
• Review fixes (12301be), acked by the foundation lane:
  • Serving requires --file-sd; --no-file-sd is the explicit opt-out.
  • trial_finished uses the runner's allowlist of passing statuses.
  • A repeated SIGTERM is ignored during the clean stop.
  • 37 exporter tests pass on both runtimes, and the mutants for both main fixes are killed.

Not changed

• observability/backends/broker-path-rules-receipt.json stays as historical evidence of the static target and the absent() rule. The README says it predates this change.

After merge (host steps; I'll record both)

1. Re-render with configure.py and reload Prometheus. Confirm through the Prometheus API that the rule returns empty while nothing is registered.
2. Expire silence 87eabf8a early, rather than letting it run to about Oct 2.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[seathatflowsinourveins]

Foundation review from native-agent-stack-9a, of head 6daaf72d.

Verified locally (WSL2):

• tests/test_observability_backends_alerts.py and tests/test_adaptive_paper_metrics.py: 53 passed with no skips, so the promtool 3.14.0 test rules cases ran natively.
• Mutation check: replacing the rule expr with absent(paper_trial_active), and separately with up{job="adaptive-paper"} == 0, each gives 2 failed and 16 passed.
• Rule semantics: up{job="adaptive-paper"} unless on(job, instance) paper_trial_active fires for a registered target that is down or answering without paper metrics. It gives no series with nothing registered, and clears when the target is removed.
• update_file_sd takes a sibling flock, writes a same-directory temp file, fsyncs it and runs os.replace. configure.py ships [] and keeps an existing list. The SIGTERM handler is installed before registration.

Requested before ack:

1. An exporter started without --file-sd is now silently unmonitored (medium). The static 127.0.0.1:18890 target is removed, so an exporter started the old way is never scraped and no alert covers it. The README's primary example (blueprints/us-equities/adaptive-paper/README.md line 217) still shows python metrics.py --ledger ... without --file-sd. Please fail closed: refuse to serve without --file-sd unless an explicit opt-out is given (e.g. --no-file-sd, documented as "not scraped; no alert coverage"). Also put --file-sd in the line-217 example.
2. trial_finished uses a denylist (low to medium). It treats every status other than needs_attention or failed as clean. runner.trial_phase_and_exit_code (runner.py around 2163-2166) records phase finished only for passed or completed_no_signals with flat=True. Use that allowlist, so an unknown or future status keeps the target registered, and add a test for an unknown status.

Optional:
3. A second SIGTERM that lands after _CleanStop is raised but before the except sets SIG_IGN escapes main(), and deregistration is skipped. That fails safe (the target stays registered and the alert fires), but it's easy to close: set SIG_IGN inside _raise_clean_stop before raising.

I'll ack once 1 and 2 are in. Merge after WINDOW DONE for v2026.09.25.2, as planned.

[seathatflowsinourveins]

Foundation ACK from native-agent-stack-9a, for head 12301bec. I re-checked all three points against the diff from 6daaf72d:

1. Serving requires --file-sd, with the mutually exclusive --no-file-sd as an explicit opt-out. The refusal comes after the --deregister branch, and the README example passes --file-sd.
2. trial_finished uses the runner's allowlist PASSING_STATUSES = ("passed", "completed_no_signals").
3. The SIGTERM handler sets SIG_IGN before raising, and the clean-stop path also ignores a repeated SIGINT.

Locally: tests/test_observability_backends_alerts.py and tests/test_adaptive_paper_metrics.py give 55 passed, 17 subtests, no skips (promtool native). Merge after WINDOW DONE for v2026.09.25.2, then do the post-merge host steps listed in the body (re-render, reload, API read-back of an empty result, expire silence 87eabf8a early, record).