feat(alerts): complete alerting system overhaul — retry limits, HMAC signing, severity, resolution notifications

[AbdulmalikAlayande]

Summary

Complete overhaul of the alerting system to address 10 identified reliability, security, and usability gaps. The previous alerting system had critical issues: infinite retry loops for failed deliveries, no webhook authentication, silent email channel acceptance that never delivered, missing resolution notifications, and no way to verify channel connectivity or view alert history.

This PR transforms the alerting system from a basic notification mechanism into a production-grade, fault-tolerant alerting pipeline.

What Changed

Critical Fixes (P0)

• Retry limits with tracking — Failed alert deliveries now increment a retry_count column on alerts_fired. After 5 failed attempts (MAX_RETRY_COUNT), the alert is excluded from future delivery cycles. Previously, failed alerts retried every daemon cycle forever with no limit or backoff. DeliveryResult now reports abandoned count for observability.

• Slack token wired from config — SentinelConfig had a slackToken field that was never used. The Slack handler always read SENTINEL_SLACK_TOKEN from env directly, creating a confusing disconnect. Now resolveSlackToken() checks env first, then falls back to config.slackToken from ~/.soroban-sentinel/config.yaml. Error message now mentions both configuration methods.

• Email channel blocked — sentinel alerts add --type email now exits immediately with "Email alerting is not yet implemented. Use 'webhook' or 'slack'." Previously, email configs were silently accepted and stored in the database, counted as "attempted" during delivery, but never actually sent. The --email CLI option has been removed. Schema CHECK constraint updated to only allow slack and webhook.

Important Improvements (P1)

• Resolution notifications — When a monitored entry's TTL recovers above the alert threshold (e.g., after a successful extension), the monitor now sends alert_resolved events to all channels that received the original threshold_crossed alert. Previously, resolveAlerts() only flipped resolved=1 in the database — users were told something was wrong but never told when it recovered. Resolution delivery is fire-and-forget (best-effort, failures logged but never block the cycle).

• Webhook HMAC-SHA256 signing — Webhook payloads now include an X-Sentinel-Signature: sha256=<hex> header when a secret is configured. Receivers can recompute the HMAC to verify payload authenticity. Follows the same pattern used by GitHub, Stripe, and Slack webhooks. Secrets are auto-generated (32 random bytes, hex-encoded) when adding webhook configs and displayed once to the user. Can be overridden via --secret flag.

• Alert severity levels — AlertEvent now includes a severity field (critical | warning | info). Critical = remaining TTL is at or below 25% of threshold or expired. Warning = below threshold but above 25%. Info = resolution events. Slack Block Kit formatting adapts: red circle emoji for critical, warning emoji for warning, green check for resolved. Previously all alerts looked identical regardless of urgency.

New CLI Commands (P2)

• sentinel alerts test --id <configId> — Sends a synthetic test alert to verify channel connectivity. Validates that webhook URLs are reachable and Slack tokens have correct permissions. Reports success/failure immediately. Essential for production verification before relying on alerts.

• sentinel alerts history --contract <id> [--limit N] — Shows chronological alert history with status icons (resolved/active, delivered/undelivered), TTL at fire time, channel info, retry count, and resolution timestamps. Previously alert data existed in the DB but was inaccessible to users.

Schema Changes

Table	Column	Type	Purpose
alerts_fired	retry_count	INTEGER NOT NULL DEFAULT 0	Track delivery attempts per alert
alert_configs	webhook_secret	TEXT (nullable)	HMAC signing secret per webhook config
alert_configs	channel_type CHECK	Removed 'email'	Block unsupported channel type at DB level

All schema changes include live migrations in database.ts using ALTER TABLE with try/catch for idempotent upgrades of existing sentinel.db files.

Files Changed (15 files, +685/-155 lines)

Source (9 files)

• src/db/schema.sql — Schema changes (retry_count, webhook_secret, email removal)
• src/db/database.ts — Live migration statements for existing databases
• src/db/repositories.ts — New functions (incrementRetryCount, getAlertConfigById, getAlertHistory), updated queries (retry cap, webhook_secret join), resolveAlerts now returns config IDs
• src/alerts/types.ts — AlertSeverity type, computeSeverity(), severity in AlertEvent
• src/alerts/webhook.ts — HMAC-SHA256 signing with X-Sentinel-Signature header
• src/alerts/slack.ts — Token resolution from config, severity-aware Block Kit formatting
• src/alerts/dispatcher.ts — Retry limits, incrementRetryCount on failure, deliverSingleAlert(), email channel removal
• src/core/monitor.ts — Resolution notification delivery via deliverSingleAlert()
• src/commands/alerts.ts — Email blocking, alerts test command, alerts history command, webhook secret generation

Tests (6 files)

• tests/alerts/dispatcher.test.ts — Retry limits suite, severity assertions, webhook secret verification
• tests/alerts/webhook.test.ts — HMAC signing suite (4 tests), severity in body structure
• tests/alerts/slack.test.ts — Severity field, updated token error message pattern
• tests/commands/alerts.test.ts — Email rejection test replaces email success test
• tests/db/alert_delivery.test.ts — retryCount and webhookSecret field assertions

Commit Strategy

9 granular commits, each self-contained and individually reviewable:

1. feat(db): add retry_count and webhook_secret columns to alert schema
2. feat(db): add retry tracking, alert history, and webhook secret to repositories
3. feat(alerts): add severity levels to AlertEvent
4. feat(alerts): add HMAC-SHA256 signing to webhook deliveries
5. feat(alerts): wire Slack token from config and add severity formatting
6. feat(alerts): add retry limits and resolution delivery to dispatcher
7. feat(core): send resolution notifications when alerts recover
8. feat(cli): overhaul alerts command with test, history, and email blocking
9. test(alerts): update test suites for alerting system overhaul

Test Plan

• All 227 tests pass (0 failures, 5 skipped)
• TypeScript compiles cleanly with tsc --noEmit
• Verify sentinel alerts add --type webhook --contract <id> --url <url> --threshold 10000 auto-generates and displays HMAC secret
• Verify sentinel alerts add --type email is rejected with clear error
• Verify sentinel alerts test --id <configId> delivers test alert to webhook/Slack
• Verify sentinel alerts history --contract <id> displays alert records
• Verify failed webhook deliveries increment retry_count and stop after 5 attempts
• Verify resolution notifications are sent when TTL recovers above threshold
• Verify Slack token resolves from config.yaml when env var is not set

Breaking Changes

• Email alert configs — Existing email alert configs in the database will no longer match the updated CHECK constraint on new databases. On existing databases, the constraint is preserved (SQLite limitation), but email configs will never be delivered since the email branch was removed from the dispatcher. Users should remove email configs and reconfigure as webhook or Slack.
• AlertEvent payload — Now includes a severity field. Webhook consumers parsing the JSON body will see the new field. This is additive and should not break existing consumers.
• DeliveryResult — Now includes an abandoned field (number). Code that destructures DeliveryResult will need to account for this.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 7953a13c-d4e5-44e3-93ca-471876c2ac1b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/alerting-system-overhaul

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}