feat(api): add queue_health monitoring endpoint by riderx · Pull Request #2756 · Cap-go/capgo.app

riderx · 2026-07-25T14:59:56Z

Summary (AI generated)

Added GET /queue_health to audit every pgmq queue and return 200/ok or 503/ko, matching the /replication monitoring pattern.
Health criteria cover unread stale messages (read_ct=0 past cron-derived threshold), stuck retries (read_ct > 5), queue depth, stale archives (>2 days), recent archive ramp-up, and missing queue/archive tables.
Response includes thresholds, human-readable healthy/unhealthy criteria, per-queue metrics, reasons, and reason details for each failure.
Wired through Supabase function queue_health, Cloudflare API route, and unit tests for evaluation helpers.

Motivation (AI generated)

Operators need a single monitorable endpoint that proves queues are being consumed and archives are not silently growing. Without this, stuck unread work and archive ramp-up are hard to catch before they impact processing latency or database size.

Business Impact (AI generated)

Faster detection of queue consumer failures reduces update/webhook/cron processing delays and avoids archive/storage growth incidents that can pressure production Postgres.

Test Plan (AI generated)

Unit tests for queue health criteria and cron interval derivation (tests/queue-health.unit.test.ts)
Call GET /queue_health with apisecret against a healthy environment and expect HTTP 200 + status: "ok"
Seed a stale unread message / stuck read_ct / old archive row and expect HTTP 503 with matching reasons and reason_details
Confirm non-admin JWT gets 403 and missing auth gets 401

Generated with AI

Summary by CodeRabbit

New Features
- Added a new GET /queue_health endpoint for authorized administrators to check the health of message queues.
- Returns an overall status summary (healthy vs. unhealthy) with per-queue findings across backlog, unread staleness, stuck reads, and archive delays.
- Includes configurable access behavior to reach the endpoint without JWT verification.
Tests
- Added unit tests covering queue name validation, schedule/interval parsing, threshold handling, and health evaluation scenarios.

Expose GET /queue_health that audits every pgmq queue for unread stale messages, stuck retries, backlog depth, and archive ramp-up, returning 503 with per-queue reasons when unhealthy (same pattern as /replication). Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

coderabbitai · 2026-07-25T15:00:04Z

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d9a3d983-ae5b-49ee-92cf-b89d218d4064

📥 Commits

Reviewing files that changed from the base of the PR and between 21dac4b and 7fef484.

📒 Files selected for processing (4)

supabase/functions/_backend/public/queue_health.ts
supabase/functions/_backend/public/replication.ts
supabase/functions/_backend/utils/platform_admin_access.ts
tests/queue-health.unit.test.ts

📝 Walkthrough

Walkthrough

Adds queue health computation for PGMQ queues, including cron-aware thresholds, metrics, authorization, JSON reporting, Supabase Edge Function wiring, and a Cloudflare /queue_health route.

Changes

Queue health monitoring

Layer / File(s)	Summary
Health contracts and scheduling helpers `supabase/functions/_backend/public/queue_health.ts`, `tests/queue-health.unit.test.ts`	Defines health thresholds and result types, validates queue names, parses queue targets and cron intervals, resolves stale-read thresholds, and tests helper behavior.
Metrics and health evaluation `supabase/functions/_backend/public/queue_health.ts`, `tests/queue-health.unit.test.ts`	Loads queue/archive metrics, evaluates status and failure reasons, builds criteria metadata, and tests healthy and unhealthy scenarios.
Authorized endpoint and runtime wiring `supabase/functions/_backend/public/queue_health.ts`, `supabase/functions/queue_health/index.ts`, `supabase/config.toml`	Validates admin access, returns aggregated health reports, configures the Edge Function entrypoint, and adds deployment settings.
Cloudflare route registration `cloudflare_workers/api/index.ts`	Registers the queue health handler at `/queue_health` and relocates existing worker imports.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant CloudflareAPI
  participant QueueHealthApp
  participant AdminRPC
  participant Postgres
  Caller->>CloudflareAPI: GET /queue_health
  CloudflareAPI->>QueueHealthApp: Route request
  QueueHealthApp->>AdminRPC: Check is_platform_admin
  AdminRPC-->>QueueHealthApp: Authorization result
  QueueHealthApp->>Postgres: Load queues, schedules, and queue metrics
  Postgres-->>QueueHealthApp: Queue data and metrics
  QueueHealthApp-->>CloudflareAPI: JSON health report
  CloudflareAPI-->>Caller: HTTP response

Suggested labels: codex

Suggested reviewers: dalanir

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: adding a queue_health monitoring endpoint.
Description check	✅ Passed	It includes Summary, Motivation, Business Impact, and Test Plan, but the template's Checklist section is missing.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{Comment @coderabbitai help to get the list of available commands.}

codspeed-hq · 2026-07-25T15:02:10Z

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks¹

_{Comparing cursor/queue-health-endpoint-388e (7fef484) with main (c4b31f0)²}

2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on main (a0132cd) during the generation of this report, so c4b31f0 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

cursor · 2026-07-25T15:05:10Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_42d13ce5-b35c-4ef3-873d-71235f2605b4)

cursor

Stale comment

Risk: medium. Not approving because Cursor Bugbot did not complete successfully (usage limit reached / skipped). Human review is needed before merge; reviewers assigned.

_{Sent by Cursor Approval Agent: Pull Request Approver}

cursor

Stale comment

Risk: medium. Cursor Bugbot did not complete (check skipped; usage limit reached), so automated review signal is unavailable. Human review is needed for this new admin queue-health endpoint (~740 lines, operational SQL).

_{Sent by Cursor Approval Agent: Pull Request Approver External}

coderabbitai

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

supabase/config.toml (1)
443-449: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Remove the duplicate [functions.queue_health] table

supabase/config.toml defines [functions.queue_health] twice. TOML treats duplicate table headers as invalid, so this should be left as a single entry and the duplicate block removed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@supabase/config.toml` around lines 443 - 449, Remove the duplicate
[functions.queue_health] table from supabase/config.toml, keeping one definition
with its existing verify_jwt and import_map settings unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@supabase/functions/_backend/public/queue_health.ts`:
- Around line 390-454: Reduce repeated database load in the queue health
aggregation around the queue and archive count queries by adding a short-TTL
cache or equivalent sampling strategy for health results, while preserving
current values and threshold behavior. Ensure queue enumeration is bounded or
paged if the registered queue count can grow without limit, and apply the same
protection to both the queue scan and archive scan paths.
- Around line 499-501: Update the overall status calculation near unhealthy and
overallStatus so an empty queues list is not classified as unhealthy or returned
as a 503; preserve the existing failure behavior when unhealthy queues exist,
and represent the no-queues condition with a distinct status or reason using the
surrounding queue health response flow.

In `@tests/queue-health.unit.test.ts`:
- Around line 102-165: Add a dedicated test within the evaluateQueueHealth suite
for the queue_depth_exceeded branch by providing metrics with queue_count
greater than thresholds.queue_depth_threshold. Assert the result is unhealthy,
includes queue_depth_exceeded in reasons, and verifies the corresponding reason
detail if the existing branch exposes one.

---

Outside diff comments:
In `@supabase/config.toml`:
- Around line 443-449: Remove the duplicate [functions.queue_health] table from
supabase/config.toml, keeping one definition with its existing verify_jwt and
import_map settings unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5ca9e34b-dd15-4246-9801-d6f700edbc2b

📥 Commits

Reviewing files that changed from the base of the PR and between a0132cd and f925508.

📒 Files selected for processing (5)

cloudflare_workers/api/index.ts
supabase/config.toml
supabase/functions/_backend/public/queue_health.ts
supabase/functions/queue_health/index.ts
tests/queue-health.unit.test.ts

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

Cap-go/capacitor-updater (manual)

Treat an empty pgmq registry as healthy with no_queues_registered, and add unit coverage for the queue_depth_exceeded criterion. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor · 2026-07-25T15:14:12Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_cc9cc345-4cf6-4af2-b6ee-076560673101)

cursor

Stale comment

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), so the required automated review signal is unavailable. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver}

cursor

Stale comment

Risk: medium. Not approving because Cursor Bugbot did not complete successfully (check skipped; usage limit reached). Human review is needed before merge; reviewers are already assigned.

_{Sent by Cursor Approval Agent: Pull Request Approver External}

cubic-dev-ai

All reported issues were addressed

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 2 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor · 2026-07-25T15:25:27Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_ea75370e-9112-4dd6-981e-2ea66e8b230b)

cursor

Stale comment

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), so the required automated review signal is unavailable. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver}

cursor

Stale comment

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), and this new admin monitoring endpoint exceeds the low-risk approval threshold. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver External}

Extract shared validatePlatformAdminOrApiSecret used by /replication and /queue_health to clear SonarCloud duplicated-lines gate, and alias SQL count union types. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor · 2026-07-25T15:26:33Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_04ea234f-3139-4d0c-bb21-943d9f49770b)

cursor

Stale comment

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), and this new admin monitoring endpoint exceeds the low-risk approval threshold. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver External}

cursor

Stale comment

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), so the required automated review signal is unavailable. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver}

Validate queue names inside fetchQueueMetrics, honor weekly/monthly cron cadence markers, give archive cleanup a 1-day grace window, and mark unknown registration state as null on error responses. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor · 2026-07-25T15:37:30Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_0bae6f48-23a9-4fe0-a9d4-060542d2bfdc)

cursor

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), and this new admin monitoring endpoint exceeds the low-risk approval threshold. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver External}

cursor

Risk: medium. Not approving: Cursor Bugbot did not complete successfully (check skipped; usage limit reached), so the required automated review signal is unavailable. Human review is still needed; Dalanir and WcaleNieWolny are already assigned as reviewers.

_{Sent by Cursor Approval Agent: Pull Request Approver}

sonarqubecloud · 2026-07-25T15:44:37Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

riderx temporarily deployed to deepsec-pr July 25, 2026 15:00 — with GitHub Actions Inactive

cursor Bot marked this pull request as ready for review July 25, 2026 15:04

cursor Bot requested review from Dalanir and WcaleNieWolny July 25, 2026 15:05

cursor Bot reviewed Jul 25, 2026

View reviewed changes

coderabbitai Bot added the codex label Jul 25, 2026

coderabbitai Bot requested changes Jul 25, 2026

View reviewed changes

Comment thread supabase/functions/_backend/public/queue_health.ts

Comment thread supabase/functions/_backend/public/queue_health.ts

Comment thread tests/queue-health.unit.test.ts

fix(api): address queue_health review feedback

21dac4b

Treat an empty pgmq registry as healthy with no_queues_registered, and add unit coverage for the queue_depth_exceeded criterion. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor Bot temporarily deployed to deepsec-pr July 25, 2026 15:14 Inactive

cursor Bot reviewed Jul 25, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread supabase/functions/_backend/public/queue_health.ts

Comment thread supabase/functions/_backend/public/queue_health.ts

Comment thread supabase/functions/_backend/public/queue_health.ts Outdated

Comment thread tests/queue-health.unit.test.ts

coderabbitai Bot approved these changes Jul 25, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Jul 25, 2026

View reviewed changes

Comment thread supabase/functions/_backend/public/queue_health.ts Outdated

chore(ci): retrigger checks after cancelled run

c6fe000

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor Bot temporarily deployed to deepsec-pr July 25, 2026 15:25 Inactive

cursor Bot reviewed Jul 25, 2026

View reviewed changes

fix(api): dedupe platform-admin auth for queue_health

68abbe8

Extract shared validatePlatformAdminOrApiSecret used by /replication and /queue_health to clear SonarCloud duplicated-lines gate, and alias SQL count union types. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>

cursor Bot had a problem deploying to deepsec-pr July 25, 2026 15:26 Failure

cursor Bot reviewed Jul 25, 2026

View reviewed changes

cursor Bot temporarily deployed to deepsec-pr July 25, 2026 15:37 Inactive

cursor Bot reviewed Jul 25, 2026

View reviewed changes

riderx merged commit dd4c6d5 into main Jul 25, 2026
55 of 75 checks passed

riderx deleted the cursor/queue-health-endpoint-388e branch July 25, 2026 16:22

coderabbitai Bot mentioned this pull request Jul 26, 2026

fix(queue): enforce max 5 retries and make queues healthy #2765

Merged

5 tasks

Uh oh!

Uh oh!

Conversation

riderx commented Jul 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary (AI generated)

Motivation (AI generated)

Business Impact (AI generated)

Test Plan (AI generated)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

codspeed-hq Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

cursor Bot commented Jul 25, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot commented Jul 25, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot commented Jul 25, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot commented Jul 25, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

riderx commented Jul 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 25, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 25, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading