feat(cloud-agent-next): emit point-in-time Cloud Agent open stock metric

[eshurakov]

What

Emit a new point-in-time observability metric, cloud_agent_open_stock, from the cloud-agent-next Worker on the existing 3-minute cron (*/3 * * * *), as a separate log record alongside cloud_agent_outcome_aggregate.

Why

cloud_agent_outcome_aggregate counts only turns whose terminal_at falls in a rolling window, so a tick with no settlement reports settled: 0. That value is truthful but cannot distinguish an idle system from one with work in flight. Nothing read the non-terminal (queued / accepted) population.

This is stock as of now, not a stall detector and not a sandbox-liveness probe. accepted means dispatched, not sandbox-alive. Age is time in that row state, not "stuck".

Shape

One grouped statement over cloud_agent_session_runs filtered to status in ('queued','accepted') and terminal_at is null, joined to retained cloud_agent_sessions (same 90-day cutoff as outcomes), split by generation (workspace_ prefix = control).

Per generation the record carries queuedTurns, queuedSessions, acceptedTurns, acceptedSessions, oldestQueuedAgeMs, oldestAcceptedAgeMs, plus counts of rows missing the age timestamp (so absent timestamps stay visible and no age is invented). Ages are epoch-millisecond differences computed in TypeScript; no timestamp text parsing.

Envelope: metric/logTag = cloud_agent_open_stock, collectionTimestamp = now (no reporting delay), coverage shared with outcomes, expectedGenerations: ['legacy','control'], limitations, and collectionStatus: complete | failed. A database failure emits one failed record and never zeros.

It reuses generationExpression, COVERAGE_LABEL, EXPECTED_GENERATIONS and retentionCutoff from the existing code rather than re-deriving them. Dispatch runs in finally after the outcome collection, so a query outage in one metric cannot blank the other. Admission, retry, cancellation, cleanup and terminalization are untouched. No new cron and no wrangler change.

Verification

• Real local Worker runtime: wrangler dev --test-scheduled with Hyperdrive pointed at this worktree's Postgres; the actual scheduled tick emitted the cloud_agent_open_stock record with collectionStatus: complete, both generations, a null age for the all-null control group, and the terminal/pre-cutoff rows contributing nothing. The same tick still emitted the three outcome records first.
• Fault path: injected select failure produces one failed record with no generations; getPgDb throwing produces the same.
• Tests: unit 223 files / 6691 passed / 3 skipped; postgres 2 files / 27 passed (null and all-null timestamps, terminal exclusion where a non-null terminal_at on a queued/accepted row makes the predicate load-bearing, generation split, retention boundary, distinct sessions, raw driver-boundary types).
• Gates: lint 0 warnings / 0 errors; typecheck clean; format:check clean.
• EXPLAIN cost gate: on a representative 125,132-run / 5,012-session dataset (ANALYZE first), EXPLAIN (ANALYZE, BUFFERS) uses IDX_cloud_agent_session_runs_status_terminal with Index Cond: status = ANY('{queued,accepted}') AND terminal_at IS NULL, examines 12 rows (0.0096% of history), loops bounded to the 12 open rows via the sessions primary key, no full-history scan and no sessions-first traversal. Planning 0.837 ms, execution 0.194 ms.

Honest limitations

• Verification ran on local workerd, not the Cloudflare edge or production Hyperdrive. Post-merge, Logpush lines with metric = cloud_agent_open_stock (dataset cloudflare-logpush, ScriptName == cloud-agent-next, EventType == scheduled) must still be confirmed after a normal deploy.
• The EXPLAIN dataset is synthetic and capped at ~125k rows, not production data.
• Age is not clamped; cross-host clock skew can make it slightly negative.
• The failure-path test is fault injection at the reader's select boundary, not an observed database outage.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental commit (8748e10c1) fixes the previously reported test-isolation defect by moving fixtures into a reserved post-2095 range excluded by the reader's own retention predicate; no new findings in the changed lines.

Files Reviewed (1 file)

• services/cloud-agent-next/test/postgres/open-stock.postgres.test.ts

Resolved in this increment

The prior WARNING (whole-table aggregation in the Postgres open-stock suite) is addressed: FIXTURE_RETENTION_CUTOFF = 2095-01-01 and FIXTURE_CREATED_AT = 2096-02-01 now exclude pre-existing rows through readOpenStock's created_at > retentionCutoff predicate, and the retention case derives its own cutoff around the fixture base. Exact counts and the toEqual([]) case remain discriminating. The residual assumptions (no other writer in the reserved range, no concurrent runs sharing the database) are documented in the fixture comment.

Previous Review Summary (commit 1784fa7)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 1784fa7)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
services/cloud-agent-next/test/postgres/open-stock.postgres.test.ts	67	Postgres tests read every open run in the table instead of only their own fixture rows, so exact counts and toEqual([]) fail against any database with pre-existing open runs.

Files Reviewed (6 files)

• services/cloud-agent-next/src/server.ts
• services/cloud-agent-next/src/server.test.ts
• services/cloud-agent-next/src/telemetry/open-stock.ts
• services/cloud-agent-next/src/telemetry/open-stock.test.ts
• services/cloud-agent-next/src/telemetry/outcome-aggregate.ts
• services/cloud-agent-next/test/postgres/open-stock.postgres.test.ts - 1 issue

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0}

_{Review guidance: REVIEW.md from base branch main}