Skip to content

feat(billing): record Cloud Agent container usage - #4732

Merged
pandemicsyn merged 11 commits into
mainfrom
feat/cloud-agent-usage-shadow
Jul 27, 2026
Merged

feat(billing): record Cloud Agent container usage#4732
pandemicsyn merged 11 commits into
mainfrom
feat/cloud-agent-usage-shadow

Conversation

@pandemicsyn

@pandemicsyn pandemicsyn commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Wire every cloud-agent-next Sandbox class to the record-only ContainerUsageMeter using explicit immutable SKUs and trusted user, organization, bot, and origin attribution. Usage generations follow physical container processes, not logical Cloud Agent or Sandbox sessions.
  • Use the exact Cloudflare Sandbox ID (org-*, usr-*, bot-*, ubt-*, ses-*, dind-*, or crv-*) as the recorder instanceId, allowing reconciliation to query containersUsageAdaptiveGroups directly with the value shown in the Cloudflare Containers dashboard.
  • Add generation-scoped durable state for idempotent starts, approximately five-minute heartbeat segments, confirmed physical stops, lost-acknowledgement recovery, rollout adoption, and stale-container reconciliation.
  • Isolate shadow telemetry from the real Sandbox control path: acquisition, physical start, physical stop, idle expiry, and live-wrapper lookup do not wait for meter RPCs or shadow storage. Durable Object waitUntil tasks retain best-effort recording and structured failure logs.
  • Harden the service boundary with strict runtime validation, class-to-instance-prefix checks, trusted fallback routing, organization membership checks, producer-owned Sandbox/SKU selection, and allowlisted origin metadata.
  • Bound missed-stop recovery: wait 15 minutes for authoritative onStop, bill only through the first observed stop, and abandon local retries after one hour while PostgreSQL stale reconciliation remains the database safety net.
  • Keep shadow mode financially inert: no wallet debits, budget enforcement, or customer-facing charging behavior are introduced.

Verification

  • Ran the real local Worker + Durable Object + Sandbox + fake-LLM cold-hot lifecycle. Two completed logical turns reported sameContainers=true and produced exactly one ses-* usage interval. The interval emitted five-minute segments (299s, then 300s segments), closed on physical idle shutdown with activity_expired, and recorded 1,227 confirmed seconds through the final observed stop. The general harness still reported its unrelated warm-path noPrepare=false assertion.
  • Ran the existing kill-mid-flight hang lifecycle. The harness killed the active SandboxSmall container and proxy while the fake LLM request was pending, surfaced cloud.message.failed, and the recorder immediately closed the single interval with close_reason=exit, exit_code=137, and a 10-second final segment. No stale-open interval or duplicate generation remained.
  • Confirmed recorder identity and attribution in PostgreSQL: instance_id matched the exact Cloudflare ses-* instance, the Small SKU was selected, user subject/actor and session were retained, and metadata contained trusted origin, container class, and Durable Object ID.
  • Fault-injection tests used meter RPCs and shadow storage operations that never resolve; Sandbox acquisition, physical start, physical stop, and idle expiry still completed without waiting for shadow telemetry.
Screenshot 2026-07-24 at 2 24 09 PM Screenshot 2026-07-24 at 2 24 31 PM

Visual Changes

N/A

Reviewer Notes

  • Focus on the physical-generation state machine in container-usage.ts: repeated logical sessions and duplicate SDK onStart callbacks reuse one generation, while a replacement physical process receives a new startEpochMs under the same Cloudflare instanceId.
  • Shadow work is intentionally best-effort and runs through Durable Object waitUntil; it must never stop, reset, delay, or prevent reaping a customer container.
  • Example DIND start payload sent to the recorder:
recordStart({
  service: 'cloud-agent-next-sandbox-dind',
  instanceId: 'dind-abcdef',
  startEpochMs,
  sku: 'cloud-agent-dind-2026-07',
  subject: { type: 'org', id: 'org_1' },
  actor: { type: 'user', id: 'user_1' },
  sessionId: 'agent_1',
  metadata: {
    container_class: 'SandboxDIND',
    durable_object_id: this.ctx.id.toString(),
    origin: 'cloud-agent',
  },
});
  • Recorder service is namespaced by concrete Sandbox class (for example, cloud-agent-next-sandbox-dind) so separate Cloudflare applications/namespaces cannot collide while sharing the same raw instance name.
  • instanceId is the exact Cloudflare dashboard/GraphQL lookup key. durable_object_id is retained only for namespace-specific diagnostics; user, organization, and bot attribution remain typed fields rather than metadata.
  • Before rollout, verify all four Cloud Agent SKUs exist and accept usage in the target environment. Run record-only shadow for at least one week and reconcile recorded seconds and estimated cents against Cloudflare data.

Use ctx.container.running as the authoritative signal for adopting or reusing a usage generation. Persisted Sandbox health can remain stale after the physical process exits, which could otherwise create phantom intervals or carry one interval across two container instances.

Mark heartbeat measurement active only after Container scheduling succeeds. This keeps scheduling failures recoverable instead of leaving a running generation permanently marked as metered without heartbeat segments.
Treat exhausted meter retries as telemetry delivery failures: retain durable generation state, arm heartbeat recovery, and warn without rejecting sandbox acquisition or physical container start. This preserves the record-only shadow contract while keeping local attribution validation strict.

Bound deferred stopped-state observation to the existing 15-minute stale grace. If authoritative onStop never arrives, close the interval at the first observed stopped timestamp so alarms do not wake forever and the grace period is not billed.
Run acquisition and Sandbox lifecycle telemetry outside the real container control path. Start, stop, idle expiry, and wrapper lookup now complete without waiting for meter RPCs or shadow storage, while Durable Object waitUntil tasks retain best-effort recording and failure logs.

Bound missed-stop retries, preserve the first observed stop cutoff, use trusted fallback routing metadata, validate origins at the Sandbox boundary, and remove the organization-handler import cycle. Shadow failures can no longer stop, reset, delay, or prevent reaping a customer container.
Record the exact Cloudflare Sandbox ID as the usage instanceId so reconciliation can pass recorder values directly to containersUsageAdaptiveGroups and match the Containers dashboard.

Keep the opaque Durable Object ID as diagnostic metadata, retain trusted origin for isolated workloads, and remove allocation and repository-provider metadata that are not needed for billing reconciliation.
Shared container billing now carries the Cloudflare sandbox ID directly and intentionally omits allocation metadata. Keep the containment test aligned with that recorder contract while preserving its ordering assertions for billing dispatch, containment activation, and workspace probing.
@pandemicsyn
pandemicsyn marked this pull request as ready for review July 24, 2026 19:24
Comment thread services/cloud-agent-next/src/container-usage.ts
Comment thread services/cloud-agent-next/src/container-usage.ts
Comment thread services/cloud-agent-next/src/container-usage-context.ts Outdated
@kilo-code-bot

kilo-code-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The latest commit (386dee0ef) tightens generation-currency checks around the billing heartbeat's schedule/reschedule/abandon paths and fixes stoppedAtFromState to only trust lastChange when the container is actually in a terminal stopped/stopped_with_code state, both covered by new regression tests; no new issues were found.

Files Reviewed (4 files)
  • packages/container-usage/src/heartbeat.ts - 0 issues
  • packages/container-usage/src/heartbeat.test.ts - 0 issues (new regression tests added)
  • services/cloud-agent-next/src/container-usage.ts - 0 issues
  • services/cloud-agent-next/src/container-usage.test.ts - 0 issues (new regression test added)
Previous Review Summaries (3 snapshots, latest commit 87f30b7)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 87f30b7)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The new commit (87f30b7d1) reworks heartbeat/stop-segment accounting to acknowledge a pending heartbeat before computing the final stop segment, captures onStop timestamps before super.onStop(), derives a per-Sandbox-class recorder service name via a getter (fixing a constructor-ordering issue that a plain class field would have caused), adds generation-replacement recovery via onGenerationClosed, and removes a redundant/inconsistent second sandbox-ID derivation in session-registration.ts; each change is covered by new regression tests and no new issues were found.

Files Reviewed (7 files)
  • packages/container-usage/src/heartbeat.ts - 0 issues
  • packages/container-usage/src/heartbeat.test.ts - 0 issues (new regression tests added)
  • services/cloud-agent-next/src/container-usage.ts - 0 issues
  • services/cloud-agent-next/src/container-usage.test.ts - 0 issues (new regression tests added)
  • services/cloud-agent-next/src/sandbox-outbound.ts - 0 issues
  • services/cloud-agent-next/src/session/session-registration.ts - 0 issues (removes a redundant sandbox-id derivation that ignored createdOnPlatform)
  • services/cloud-agent-next/src/session-prepare.test.ts - 0 issues (updated for the session-registration change)

Previous review (commit 0cd4ce2)

Status: No Issues Found | Recommendation: Merge

Executive Summary

All three previously flagged WARNING issues in container-usage.ts and container-usage-context.ts are fixed at the current commit (0cd4ce2fe), each with a new regression test; no new issues were found in the fix commit.

Files Reviewed (4 files)
  • services/cloud-agent-next/src/container-usage.ts - 0 issues (previously flagged issues fixed)
  • services/cloud-agent-next/src/container-usage-context.ts - 0 issues (previously flagged issue fixed)
  • services/cloud-agent-next/src/container-usage.test.ts - 0 issues (new regression tests added)
  • services/cloud-agent-next/src/container-usage-context.test.ts - 0 issues (new regression test added)

Previous review (commit fdf4075)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The highest-risk issue is a missing return in container-usage.ts's configureBilling recovery path that can start a new billing generation while a prior stop is still unresolved, silently dropping a usage segment.

Overview

Severity Count
CRITICAL 0
WARNING 3
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/cloud-agent-next/src/container-usage.ts 137 Missing return after deferred billing delivery in unmeasured stop recovery branch can drop a billing segment
services/cloud-agent-next/src/container-usage.ts 192 activityExpiryRequested in-memory flag can leak across billing generations when onStop returns early with no context
services/cloud-agent-next/src/container-usage-context.ts 126 Legacy __ fallback lets an isolated-prefixed sandboxId pass the shared-sandbox check, weakening the class-to-instance-prefix invariant
Files Reviewed (23 files)
  • dev/local/services.ts - 0 issues
  • packages/container-usage/src/context.ts - 0 issues
  • packages/container-usage/src/heartbeat.ts - 0 issues
  • packages/container-usage/src/heartbeat.test.ts - 0 issues
  • services/cloud-agent-next/package.json - 0 issues
  • services/cloud-agent-next/src/agent-sandbox/cloudflare/cloudflare-agent-sandbox.ts - 0 issues
  • services/cloud-agent-next/src/agent-sandbox/cloudflare/cloudflare-agent-sandbox.test.ts - 0 issues
  • services/cloud-agent-next/src/container-usage-context.ts - 1 issue
  • services/cloud-agent-next/src/container-usage-context.test.ts - 0 issues
  • services/cloud-agent-next/src/container-usage.ts - 2 issues
  • services/cloud-agent-next/src/container-usage.test.ts - 0 issues
  • services/cloud-agent-next/src/kilo-facade/session-proxy.ts - 0 issues
  • services/cloud-agent-next/src/persistence/session-metadata.ts - 0 issues
  • services/cloud-agent-next/src/router.test.ts - 0 issues
  • services/cloud-agent-next/src/router/handlers/organization-membership.ts - 0 issues
  • services/cloud-agent-next/src/router/handlers/session-prepare.ts - 0 issues
  • services/cloud-agent-next/src/router/handlers/session-start.ts - 0 issues
  • services/cloud-agent-next/src/sandbox-outbound.ts - 0 issues
  • services/cloud-agent-next/src/sandbox-outbound.test.ts - 0 issues
  • services/cloud-agent-next/src/session-prepare.test.ts - 0 issues
  • services/cloud-agent-next/src/session/session-registration.ts - 0 issues
  • services/cloud-agent-next/src/types.ts - 0 issues
  • services/cloud-agent-next/wrangler.jsonc - 0 issues

pnpm-lock.yaml and worker-configuration.d.ts were excluded as generated files.

Fix these issues in Kilo Cloud


Reviewed by claude-sonnet-5 · Input: 40 · Output: 10.2K · Cached: 1.1M

Review guidance: REVIEW.md from base branch main

Stop failed prior-generation recovery from falling through into replacement generation creation, consume activity-expiry intent at the physical stop boundary, and reject isolated-prefixed legacy IDs for shared sandbox classes.
Namespace recorder services by concrete Sandbox class while retaining raw Cloudflare instance IDs, and preserve authoritative isolated routing targets for code-review and DIND sessions.

Capture physical stop cutoffs, deliver pending heartbeats before final stop remainders, use container transition timestamps for missed stops, bound authoritative stop retries, and start already-running replacements after prior generations close.
@pandemicsyn
pandemicsyn merged commit 0011ea8 into main Jul 27, 2026
68 checks passed
@pandemicsyn
pandemicsyn deleted the feat/cloud-agent-usage-shadow branch July 27, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants