Skip to content

feat(admin): add container metrics to session traces - #4935

Merged
pandemicsyn merged 5 commits into
mainfrom
feat/session-container-metrics
Jul 31, 2026
Merged

feat(admin): add container metrics to session traces#4935
pandemicsyn merged 5 commits into
mainfrom
feat/session-container-metrics

Conversation

@pandemicsyn

@pandemicsyn pandemicsyn commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add Cloudflare container diagnostics to the admin session trace viewer, including the Durable Object instance ID, billing SKU, memory and CPU trends, and exact minute samples.
  • Resolve root and scoped Cloud Agent sessions through metered container intervals, clip provider queries to each container lifetime and a 10-minute padded session window, and keep shared-container and application ambiguity explicit.
  • Add a server-only containersMetricsAdaptiveGroups client with account capability checks, retention/window limits, partial-result handling, and provider-safe errors.
  • Snapshot provisioned container capacity into new usage records and centralize the existing service capacity mapping so historical sessions can draw an accurate memory limit when available.

Verification

  • Opened /admin/session-traces?sessionId=ses_048a82950fffQ60eeZmoxxETi6 locally with a representative Cloud Agent usage interval and confirmed Session Details displayed the container instance and SKU.
  • Checked the Session Details and container diagnostics layout at desktop and 390px viewport widths.
  • Confirmed the metrics section displays an isolated, non-blocking unavailable state when Cloudflare Analytics is not configured in development.
  • Add any additional manual verification details.

Visual Changes

Before After
Session trace viewer without container identity, SKU, or workload diagnostics.
Session Details includes container identity and SKU, followed by memory/CPU graphs and a metric samples table.

Reviewer Notes

  • Cloudflare workload Analytics is production-only in the current development setup; provider behavior is covered by normalized response fixtures, while local browser verification exercised the unavailable state.
  • Memory receives a capacity reference line only when every plotted interval has the same known capacity. CPU intentionally has no capacity line.
  • Review focus: GraphQL retention/error handling, shared-session padded time clipping, and per-placement chart separation.

@pandemicsyn
pandemicsyn marked this pull request as ready for review July 31, 2026 17:24
},
];
});
if (windows.length === 0) return { available: false, reason: 'no_provider_identity' };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: no_provider_identity is reported for causes other than missing provider identity

windows can also end up empty because the clipping above dropped every interval via if (Date.parse(end) <= Date.parse(start)) return [];. That happens whenever a matched interval lies outside [windowStartAt, windowEndAt] — which is reachable, because the session_id branch of intervalIdentityCondition (line 178) has no time bound, so an interval that starts after the session's updated_at is selected and then clipped to an empty range.

In that case the UI renders "No Cloudflare instance identity was recorded for these container intervals" (SessionContainerTelemetry.tsx:206), which is factually wrong and sends the operator looking for missing durable_object_id metadata. Consider distinguishing "no interval had a cloudflareInstanceId" from "every interval clipped to an empty window" so the diagnostic stays accurate.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

eq(container_usage_interval.cloud_billing_sku_id, cloud_billing_sku.id)
)
.where(
and(like(container_usage_interval.service, 'cloud-agent-next%'), intervalIdentityCondition)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: This filter cannot use any existing index on container_usage_interval, and the whole function runs twice per metrics request

container_usage_interval only defines (status, last_seen_at), (subject_type, subject_id, started_at), and a partial unique (service, instance_id) WHERE status = 'open' (packages/db/src/schema.ts:10118-10126). This predicate filters on service LIKE 'cloud-agent-next%' plus session_id / instance_id, so none of those indexes apply and Postgres will sequentially scan the full platform-wide billing interval table on every admin session-trace view.

Two amplifiers:

  • getSessionContainerMetrics (line 278) calls getSessionContainerInfo again, so getContainerMetrics re-executes all three queries that getContainerInfo already ran for the same session.
  • The cloud_agent_session_runs query (line 155) and the runs array it populates are never read by SessionContainerTelemetry.tsx or SessionTraceViewer.tsx, so that query and payload are currently pure overhead.

Consider an index supporting session_id (and instance_id), passing the already-fetched SessionContainerInfo into getSessionContainerMetrics, and dropping the unused runs selection.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

sandboxId?.startsWith('crv-') ||
sandboxId?.startsWith('dind-')
? 'isolated'
: 'shared',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Unknown sandbox identity is reported as a shared container

cloud_agent_sessions.sandbox_id is nullable (packages/db/src/schema.ts:5639) and sandboxId is also null when no cloud_agent_sessions row exists, so every optional-chained startsWith is undefined and scope falls through to 'shared'. The UI then renders the "Isolated container"/"Shared container" badge and the assertive banner "This container is shared. Workload samples may include activity from other sessions." (SessionContainerTelemetry.tsx:182-188) for sessions where the sandbox identity is simply unknown.

A third 'unknown' scope (or gating the banner on sandboxId !== null) would avoid asserting sharing that was never observed.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

for (const [index, plan] of batch.entries()) {
const alias = `m${index}`;
const aliasErrors = errors.filter(
error => error.path?.includes(alias) || error.path === undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Unscoped GraphQL errors are silently dropped for single-window batches

Line 451 only throws on unscopedErrors when batch.length > 1. For a single-plan batch the intent is clearly to downgrade those errors to partial, but this filter only re-captures errors with path === undefined. An unscoped error whose path is defined yet contains no m<n> segment (for example ["viewer", "accounts"]) matches neither branch: it is not thrown, not added to issues, and does not set partial.

If account and groups are still present, the result is returned as { partial: false, issues: [] } even though Cloudflare reported an error, so the admin UI presents incomplete data as complete. Attributing unscopedErrors to the single plan (rather than only path-less errors) would close the gap.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

<div className="grid gap-6 xl:grid-cols-2">
<figure
className="space-y-2"
role="img"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: role="img" hides the chart's heading and figure semantics from assistive technology

role="img" makes the element a leaf node in the accessibility tree, so every descendant is pruned — including the <h3>Memory</h3> on line 257 (and <h3>CPU Utilization</h3> on line 344). Screen-reader users lose both section headings, the document heading outline breaks, and the figure role is overridden so figure/figcaption semantics no longer apply.

Since the same data is already exposed in the "Metric samples" table, the accessible label belongs on the inner chart container (<div className="h-64 w-full">) rather than on the figure wrapping the heading.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

label:
placementId && placementId.length > 12
? `${placementId.slice(0, 8)}...`
: (placementId ?? 'unknown'),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Series labels drop the windowKey, so per-interval separation is invisible in the legend

The refactor deliberately keys series by (windowKey, placementId), but label only encodes placementId. When the same placement appears across two container intervals, two distinct Line series render with identical legend/tooltip names ("placem... max" twice), which defeats the separation this change introduced. Including a short window discriminator in label would keep them distinguishable.

Also, placementId now comes from Set<string> (line 109), so it is non-nullable — both the placementId && guard and the ?? 'unknown' fallback are unreachable leftovers from the previous split('\0') implementation.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

@kilo-code-bot

kilo-code-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

All six previously reported findings are fixed at HEAD; the remaining risk is that folding container info and Cloudflare metrics into one tRPC query makes an analytics outage or missing analytics config hide the Postgres-derived container interval, SKU, and instance data entirely.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
apps/web/src/routers/admin-router.ts 2223 ContainerMetricsAnalyticsError (including missing_config and every transient BAD_GATEWAY) now rejects the combined query, so the interval/SKU table and the summary card's Container:/SKU: rows disappear instead of just the metrics section

SUGGESTION

File Line Issue
apps/web/src/routers/admin/session-container-telemetry.ts 337 getSessionContainerMetrics (and the queryMetrics injection point) is now reachable only from admin-session-traces.test.ts; admin.sessionTraces.getContainerInfo likewise has no UI consumer left
Files Reviewed in this incremental pass (9 files)
  • apps/web/src/routers/admin-router.ts - 1 issue
  • apps/web/src/routers/admin/session-container-telemetry.ts - 1 issue
  • apps/web/src/lib/cloudflare/container-metrics-analytics.ts - 0 issues
  • apps/web/src/app/admin/components/SessionContainerTelemetry.tsx - 0 issues
  • apps/web/src/app/admin/components/SessionTraceViewer.tsx - 0 issues
  • apps/web/src/app/admin/api/session-traces/hooks.ts - 0 issues
  • apps/web/src/routers/admin-session-traces.test.ts - 0 issues
  • apps/web/src/app/admin/components/SessionContainerTelemetry.test.ts - 0 issues
  • apps/web/src/lib/cloudflare/container-metrics-analytics.test.ts - 0 issues
Previously reported findings verified as fixed
  • Unindexed, doubly-executed interval scan: the query now filters subject_type/subject_id plus started_at < windowEndAt, which matches IDX_container_usage_interval_subject_started, and getSessionContainerInfo runs once per request because the router passes info into getSessionContainerMetricsForInfo.
  • no_provider_identity overloading: intervals are pre-filtered on cloudflareInstanceId !== null, and empty clipped windows now return the new no_overlapping_intervals reason, which the UI renders with its own copy.
  • Unscoped GraphQL errors dropped: single-window batches now attribute unscopedErrors to the sole alias, so they surface as partial: true with issues; multi-window batches still throw. Covered by the new reports single-window unscoped GraphQL errors as partial results test.
  • role="img" pruning the chart heading: role/aria-label moved to the inner chart div, leaving figure semantics and the <h3> in the accessibility tree.
  • Null sandbox_id asserted as shared: scope is now 'unknown', the badge reads Unknown container scope, and the shared-container warning alert is suppressed.
  • Legend labels colliding across intervals: containerMetricSeriesLabel(windowKey, placementId) includes both, and the unreachable placementId && / ?? 'unknown' fallbacks are gone.
Assumptions
  • No tests were executed (read-only review). Correctness of the new subject predicate was verified against packages/db/src/schema.ts (subject_type/subject_id are notNull text; cli_sessions_v2.kilo_user_id is notNull, so organizationId ?? kiloUserId is always a string).
  • The subject narrowing is read as intended tenant scoping. Note it silently yields zero intervals in the rare cases where the recorded billing subject diverges from the session row: organization_id is ON DELETE SET NULL (only a dev route hard-deletes orgs), and container_usage_interval.subject_id stores the caller-supplied org UUID verbatim while cli_sessions_v2.organization_id is a uuid column Postgres renders lowercase, so a mixed-case org id from a client would not match. No test covers an org-owned session or a mismatched-subject interval.
  • Moving overlapsSessionWindow into the top-level WHERE also constrains the session_id branch. Given the 10-minute padding and the started_at < end AND last_seen_at > start shape, only intervals entirely outside the padded window are dropped, which matches the documented intent.

Fix these issues in Kilo Cloud

Previous Review Summaries (2 snapshots, latest commit 35092d8)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 35092d8)

Status: 6 Issues Found | Recommendation: Address before merge

Executive Summary

Incremental review of the pad container metrics around sessions commit found no new defects; the six previously reported findings (led by the unindexed, doubly-executed container_usage_interval scan) are still present at current HEAD.

Overview

Severity Count
CRITICAL 0
WARNING 4
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
apps/web/src/routers/admin/session-container-telemetry.ts 213 service LIKE + session_id/instance_id predicate matches no existing index, so every admin session-trace view sequentially scans the platform-wide billing interval table; getSessionContainerMetrics re-runs the whole resolver
apps/web/src/routers/admin/session-container-telemetry.ts 303 no_provider_identity is also returned when every interval clipped to an empty window, so the UI claims no Cloudflare instance identity was recorded
apps/web/src/lib/cloudflare/container-metrics-analytics.ts 466 Unscoped GraphQL errors with a defined non-alias path are neither thrown (guarded by batch.length > 1) nor captured as alias errors, so incomplete results are returned as partial: false
apps/web/src/app/admin/components/SessionContainerTelemetry.tsx 254 role="img" on the figure prunes descendants from the accessibility tree, removing the chart <h3> heading and overriding figure semantics

SUGGESTION

File Line Issue
apps/web/src/routers/admin/session-container-telemetry.ts 252 Null sandbox_id falls through to scope: 'shared', asserting cross-session sharing that was never observed
apps/web/src/app/admin/components/SessionContainerTelemetry.tsx 118 Series label omits windowKey, producing identical legend names for the same placement across intervals; the placementId && guard and ?? 'unknown' fallback are unreachable after the Set<string> refactor
Files Reviewed in this incremental pass (2 files)
  • apps/web/src/routers/admin/session-container-telemetry.ts - 0 new issues (changed lines 24, 174-177)
  • apps/web/src/routers/admin-session-traces.test.ts - 0 new issues
Verified as correct in the new commit
  • SESSION_METRICS_PADDING_MS (10 min) widens both the window bounds and the provider query clipping symmetrically; metrics windows stay clamped to each interval's startedAt/stoppedAt ?? lastSeenAt, so padding cannot request metrics outside a container's observed lifetime.
  • Updated admin-session-traces.test.ts fixtures match the new arithmetic: session 08:43:36.040Z/08:55:07.000Z pads to 08:33:36.040Z-09:05:07.000Z, which is the clip of the interval 08:20:00Z-09:30:00Z and exactly the asserted provider window.
  • Padding cannot break queryContainerMetricsAnalytics limits: +20 min at most adds one splitWindow part per window, well below MAX_WINDOWS, and a future-dated datetime_lt bound is harmless.
  • windowStartAt/windowEndAt are not rendered in the UI, so the padded bounds are not surfaced as the session's own time range.
Assumptions
  • No tests were executed (read-only review); the padded-window arithmetic was verified by hand against the updated fixtures.
  • Widening the shared-container instance_id overlap predicate by 10 minutes can now match adjacent intervals of the same sandbox from neighbouring sessions. This is read as the intended "10-minute padded session window" behavior described in the PR body rather than a defect.

Fix these issues in Kilo Cloud

Previous review (commit f8ad783)

Status: 6 Issues Found | Recommendation: Address before merge

Executive Summary

Highest risk is the unindexed, doubly-executed container_usage_interval scan behind the new admin getContainerInfo/getContainerMetrics endpoints; the remaining findings are diagnostic accuracy in the telemetry resolver, silently-dropped Cloudflare GraphQL errors, and an accessibility regression in the chart markup.

Overview

Severity Count
CRITICAL 0
WARNING 4
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
apps/web/src/routers/admin/session-container-telemetry.ts 209 service LIKE + session_id/instance_id predicate matches no existing index, so every admin session-trace view sequentially scans the platform-wide billing interval table; getSessionContainerMetrics re-runs all three queries, and the cloud_agent_session_runs query/runs payload is never consumed by the UI
apps/web/src/routers/admin/session-container-telemetry.ts 299 no_provider_identity is also returned when every interval clipped to an empty window, so the UI claims no Cloudflare instance identity was recorded
apps/web/src/lib/cloudflare/container-metrics-analytics.ts 466 Unscoped GraphQL errors with a defined non-alias path are neither thrown (guarded by batch.length > 1) nor captured as alias errors, so incomplete results are returned as partial: false
apps/web/src/app/admin/components/SessionContainerTelemetry.tsx 254 role="img" on the figure prunes descendants from the accessibility tree, removing both chart <h3> headings and overriding figure semantics

SUGGESTION

File Line Issue
apps/web/src/routers/admin/session-container-telemetry.ts 248 Null sandbox_id falls through to scope: 'shared', asserting cross-session sharing that was never observed
apps/web/src/app/admin/components/SessionContainerTelemetry.tsx 118 Series label omits windowKey, producing identical legend names for the same placement across intervals; the placementId && guard and ?? 'unknown' fallback are unreachable after the Set<string> refactor
Files Reviewed (16 files)
  • apps/web/src/app/admin/api/session-traces/hooks.ts - 0 issues
  • apps/web/src/app/admin/components/SessionContainerTelemetry.test.ts - 0 issues
  • apps/web/src/app/admin/components/SessionContainerTelemetry.tsx - 2 issues
  • apps/web/src/app/admin/components/SessionTraceViewer.tsx - 0 issues
  • apps/web/src/lib/cloudflare/container-capacity.ts - 0 issues
  • apps/web/src/lib/cloudflare/container-metrics-analytics.test.ts - 0 issues
  • apps/web/src/lib/cloudflare/container-metrics-analytics.ts - 1 issue
  • apps/web/src/routers/admin-router.ts - 0 issues
  • apps/web/src/routers/admin-session-traces.test.ts - 0 issues
  • apps/web/src/routers/admin/cloud-billing-skus-router.ts - 0 issues
  • apps/web/src/routers/admin/session-container-telemetry.ts - 3 issues
  • services/cloud-agent-next/src/container-capacity-parity.test.ts - 0 issues
  • services/cloud-agent-next/src/container-usage-context.test.ts - 0 issues
  • services/cloud-agent-next/src/container-usage-context.ts - 0 issues
  • services/cloud-agent-next/src/container-usage.test.ts - 0 issues
  • services/cloud-agent-next/src/container-usage.ts - 0 issues
Verified as correct (no action needed)
  • CLOUDFLARE_ACCOUNT_ID / CLOUDFLARE_ANALYTICS_API_TOKEN are exported point-of-use from config.server.ts; no token or response body appears in any error message.
  • recharts@3.8.1 is already an apps/web dependency and renderToStaticMarkup .test.ts files are an established pattern in apps/admin/components.
  • jsonc-parser is declared in services/cloud-agent-next/package.json, the service's default vitest project is the Node pool, and fs.readFileSync(path.join(process.cwd(), 'wrangler.jsonc')) matches the existing queue-config.test.ts precedent.
  • container-capacity-parity.test.ts reads the top-level containers block (7 entries, matching SANDBOX_CAPACITIES); the differing instance types at lines 466-535 are inside env.dev and correctly excluded.
  • sharedContainerCapacity adding a vcpu equality check is behavior-preserving for cloud-billing-skus-router.ts, since every service sharing a memory/disk tier also shares its vCPU count.
  • sessionViewerProcedure enforces admin + can_view_sessions + not-blocked against fresh primary-DB state for both new procedures.
  • .passthrough() and z.string().datetime() are deprecated but functional in zod 4.4.3 and already used in the sibling container-usage-analytics.ts.
Assumptions
  • Cloudflare's containersMetricsAdaptiveGroups cpuUtilization is a 0-1 fraction (the code multiplies by 100) and max_diskUsage/avg_memory are byte-valued. These could not be verified from the repository or a live account; the normalized-response fixtures encode the same assumption.
  • No tests were executed (read-only review), so runtime behavior of the cross-package ../../../apps/web/src/lib/cloudflare/container-capacity.js import in the parity test is inferred from Vite's TS .js-specifier resolution and the fact that the imported module has no @/ imports.

Fix these issues in Kilo Cloud


Reviewed by claude-opus-5 · Input: 66 · Output: 21.2K · Cached: 2.5M

Review guidance: REVIEW.md from base branch main

@pandemicsyn
pandemicsyn merged commit 9834de0 into main Jul 31, 2026
21 checks passed
@pandemicsyn
pandemicsyn deleted the feat/session-container-metrics branch July 31, 2026 18:09
try {
const info = await getSessionContainerInfo(input.session_id);
const metrics = info
? await getSessionContainerMetricsForInfo(info)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A Cloudflare analytics failure now hides the DB-derived container info too

Merging info and metrics into one procedure means any ContainerMetricsAnalyticsError thrown by getSessionContainerMetricsForInfo rejects the whole query, so containerTelemetryQuery.data stays undefined. SessionTraceViewer then renders neither the interval/SKU/capacity table (SessionContainerTelemetryContent, gated on data?.info) nor the Container:/SKU: rows in the summary card (latestContainerInterval, SessionTraceViewer.tsx:239) — only a destructive alert.

This is not an exotic path: missing_config is thrown whenever CLOUDFLARE_ACCOUNT_ID/CLOUDFLARE_ANALYTICS_API_TOKEN are absent (container-metrics-analytics.ts:330), and BAD_GATEWAY covers every transient Cloudflare GraphQL failure. Before this change the separate getContainerInfo query kept the interval table visible and confined the metrics error to an inline alert inside the card.

Consider catching ContainerMetricsAnalyticsError here and returning { info, metrics: { available: false, ... } } (or an error field the card can render) so the Postgres-derived interval data survives analytics outages.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

return { available: true, ...metrics };
}

export async function getSessionContainerMetrics(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: getSessionContainerMetrics is now only reachable from tests

After admin-router.ts switched to getSessionContainerInfo + getSessionContainerMetricsForInfo, this wrapper has no production caller; its only remaining use is admin-session-traces.test.ts:264. The queryMetrics parameter on the new ForInfo function is likewise only injected from tests. The same coverage can be obtained by composing the two exported functions in the test, which also keeps the tested path identical to the router path.

Related: admin.sessionTraces.getContainerInfo no longer has a UI consumer now that useAdminSessionContainerInfo is gone — worth deciding whether it stays as a debugging endpoint or is removed with the wrapper.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants