Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 18 additions & 37 deletions docs/contributing/architecture/run-records.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,7 +367,7 @@ read `RunLog`.
Run records are **excluded from the `storage_bytes` entitlement**. They are
observability history, not user content.

## RunLog authority and deploy sequencing
## RunLog authority

The per-user `RunLog` DO is the **sole runtime authority** for:

Expand All @@ -378,34 +378,16 @@ The per-user `RunLog` DO is the **sole runtime authority** for:
`activation_milestones`).

Runtime paths read and write those tables only inside `RunLog`. Workflow
lifecycle code does not touch D1 `workflow_runs`. Job finalization and
activation increments do not lazy-seed dedicated RunLog rows from D1 on read or
finish. The completed admin seed/parity MCP capability is absent from the admin
domain.

**Application deploy (code-only, PR #1205):** ships RunLog-only workflow,
job-observability, and activation behavior; bounded content-free RunLog point
reads for admin workflow/activation insights; and schema-v8
`job_run_observability.legacy_seeded` kept inert for rollback compatibility.

**Production parity gate (2026-08-04 02:45–02:46 UTC):** 41/41 non-deleting
users matched across nine keyset pages with zero failed, missing, or undercount
rows before the application deploy.

**Post-cutover production smoke:** `workflow_run_list` succeeded after the
application deploy.

**Destructive follow-up deploy (separate PR, migration `0137`):** drops the
retired D1 projection tables `workflow_runs`, `user_package_run_successes`, and
`user_activation_milestones`; removes them from account deletion/export
inventory and retires the hourly D1 `workflow_runs` retention lane.
Pre-migration D1 Time Travel bookmark for database
`8c1014d1-6b41-4695-a0a2-159071f0f919`:
lifecycle code does not use D1 workflow projections. Job finalization and
activation increments do not seed dedicated RunLog rows from D1. D1 `jobs`
retains schedule metadata and retention anchors, but job outcome aggregates come
from `job_run_observability`.

The pre-drop D1 Time Travel bookmark for database
`8c1014d1-6b41-4695-a0a2-159071f0f919` is:
`0000116d-000000d2-000050bd-c7ecd5892a189df7cda145af746bc9c9`. Restore from that
bookmark (or a fresher snapshot taken immediately before apply) if the
destructive migration must be rolled back — do not reintroduce dual-write paths.
Application cutover evidence:
https://github.com/kentcdodds/kody/actions/runs/30876391371

## Dedicated RunLog state (outside run-history caps)

Expand All @@ -429,17 +411,16 @@ cursors remain valid.

### Admin insights RunLog reads

The role-gated `/admin/insights` dashboard loads workflow status totals and
activation funnel/latency from **bounded, content-free per-user RunLog point
reads** (`getAdminInsightsSnapshot`; concurrency capped by
`adminInsightsRunLogConcurrency`). Each snapshot returns aggregate workflow
status counts and activation milestone timestamps/ids only — never workflow
names, errors, logs, or other user-authored content. The page exposes
`runLogCompleteness` (`usersAttempted`, `usersLoaded`, `complete`) so partial
fanout degrades run-derived charts with an explicit warning instead of failing
the whole dashboard. Legacy D1 `workflow_runs`, `user_activation_milestones`,
and `user_package_run_successes` were retired by migration `0137` and are not
queried on that path.
The role-gated `/admin/insights` dashboard loads workflow status totals and job
success/error totals, plus activation funnel/latency, from **bounded,
content-free per-user RunLog point reads** (`getAdminInsightsSnapshot`;
concurrency capped by `adminInsightsRunLogConcurrency`). Each snapshot returns
aggregate workflow statuses, job outcome counts, and activation milestone
timestamps/ids only — never workflow or job names, errors, logs, or other
user-authored content. The page exposes `runLogCompleteness` (`usersAttempted`,
`usersLoaded`, `complete`) so partial fanout degrades run-derived charts with an
explicit warning instead of failing the whole dashboard. D1 supplies only job
schedule totals (`totalJobs` and `enabledJobs`) on that path.

## Related

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- RunLog is the sole authority for job error, duration, and outcome counters.
-- Keep the D1 scheduling/retention anchors last_run_at and last_run_status.

ALTER TABLE jobs DROP COLUMN last_run_error;
ALTER TABLE jobs DROP COLUMN last_duration_ms;
ALTER TABLE jobs DROP COLUMN run_count;
ALTER TABLE jobs DROP COLUMN success_count;
ALTER TABLE jobs DROP COLUMN error_count;
30 changes: 27 additions & 3 deletions packages/worker/src/app/admin-insights-data.node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,11 @@ const defaultInsightsUsers = [
]

function emptySnapshot(): RunLogAdminInsightsSnapshot {
return { workflowStatusCounts: [], activationMilestones: [] }
return {
workflowStatusCounts: [],
activationMilestones: [],
jobRunCounts: { success: 0, error: 0 },
}
}

function createInsightsTestDb(
Expand All @@ -203,8 +207,6 @@ function createInsightsTestDb(
return {
total: 4,
enabled: 3,
success_runs: 20,
error_runs: 2,
} as T
}
if (normalizedQuery.includes('from users where created_at < ?')) {
Expand Down Expand Up @@ -370,6 +372,7 @@ test('loadAdminInsightsData assembles the dashboard payload from D1 plus RunLog
{ status: 'complete', count: 5 },
{ status: 'errored', count: 1 },
],
jobRunCounts: { success: 12, error: 2 },
activationMilestones: [
{
milestone: 'package_run_succeeded',
Expand All @@ -385,6 +388,7 @@ test('loadAdminInsightsData assembles the dashboard payload from D1 plus RunLog
},
'user-b': {
workflowStatusCounts: [{ status: 'complete', count: 3 }],
jobRunCounts: { success: 8, error: 0 },
activationMilestones: [
{
milestone: 'package_run_succeeded',
Expand Down Expand Up @@ -465,6 +469,12 @@ test('loadAdminInsightsData assembles the dashboard payload from D1 plus RunLog
successRuns: 20,
errorRuns: 2,
})
const jobStatsQuery = db.seenQueries.find((query) =>
normalizeQuery(query).includes('from jobs'),
)
expect(normalizeQuery(jobStatsQuery ?? '')).toBe(
'select count(*) as total, sum(enabled) as enabled from jobs',
)
expect(data.activation.steps).toEqual([
{ step: 'signed_up', users: 8 },
{ step: 'email_verified', users: 5 },
Expand Down Expand Up @@ -503,6 +513,7 @@ test('multi-user RunLog aggregation sums workflow statuses and milestone users',
{ status: 'running', count: 2 },
{ status: 'complete', count: 1 },
],
jobRunCounts: { success: 3, error: 1 },
activationMilestones: [
{
milestone: 'package_run_succeeded',
Expand All @@ -524,6 +535,7 @@ test('multi-user RunLog aggregation sums workflow statuses and milestone users',
},
snapshot: {
workflowStatusCounts: [{ status: 'complete', count: 4 }],
jobRunCounts: { success: 5, error: 2 },
activationMilestones: [
{
milestone: 'package_run_succeeded',
Expand All @@ -547,6 +559,8 @@ test('multi-user RunLog aggregation sums workflow statuses and milestone users',
{ status: 'complete', count: 5 },
{ status: 'running', count: 2 },
])
expect(folded.jobSuccessRuns).toBe(8)
expect(folded.jobErrorRuns).toBe(3)
expect(folded.packageRunSucceededUsers).toBe(2)
expect(folded.packageActivatedUsers).toBe(1)
expect(folded.medianHoursToActivation).toBe(12)
Expand Down Expand Up @@ -596,6 +610,7 @@ test('activation latency uses package_activated reachedAt against D1 email_verif
stubSnapshotsByUser({
'user-a': {
workflowStatusCounts: [],
jobRunCounts: { success: 0, error: 0 },
activationMilestones: [
{
milestone: 'package_activated',
Expand All @@ -606,6 +621,7 @@ test('activation latency uses package_activated reachedAt against D1 email_verif
},
'user-b': {
workflowStatusCounts: [],
jobRunCounts: { success: 0, error: 0 },
activationMilestones: [
{
milestone: 'package_activated',
Expand All @@ -617,6 +633,7 @@ test('activation latency uses package_activated reachedAt against D1 email_verif
},
'user-c': {
workflowStatusCounts: [],
jobRunCounts: { success: 0, error: 0 },
activationMilestones: [
{
milestone: 'package_activated',
Expand Down Expand Up @@ -694,6 +711,12 @@ test('degraded RunLog snapshots warn and zero run-derived charts without failing
expect(data.ok).toBe(true)
expect(data.totals.workflowRuns).toBe(0)
expect(data.workflowStatuses).toEqual([])
expect(data.jobHealth).toEqual({
totalJobs: 4,
enabledJobs: 3,
successRuns: 0,
errorRuns: 0,
})
expect(data.activation.steps).toEqual([
{ step: 'signed_up', users: 8 },
{ step: 'email_verified', users: 5 },
Expand Down Expand Up @@ -733,6 +756,7 @@ test('admin insights uses content-free getAdminInsightsSnapshot point reads only
consoleWarn.mockImplementation(() => {})
const snapshot: RunLogAdminInsightsSnapshot = {
workflowStatusCounts: [{ status: 'complete', count: 1 }],
jobRunCounts: { success: 2, error: 1 },
activationMilestones: [
{
milestone: 'package_run_succeeded',
Expand Down
24 changes: 14 additions & 10 deletions packages/worker/src/app/admin-insights-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,6 @@ type HeatmapRow = { day: string; hour: string; n: number }
type JobStatsRow = {
total: number
enabled: number | null
success_runs: number | null
error_runs: number | null
}
type ForkActorRow = { actor: string | null; n: number }
type InsightsUserRow = {
Expand All @@ -79,6 +77,8 @@ type InsightsUserRow = {
type AggregatedRunLogInsights = AdminInsightsRunLogCompleteness & {
workflowStatuses: Array<AdminInsightsWorkflowStatus>
workflowRuns: number
jobSuccessRuns: number
jobErrorRuns: number
packageRunSucceededUsers: number
packageActivatedUsers: number
medianHoursToActivation: number | null
Expand All @@ -101,7 +101,7 @@ export async function loadAdminInsightsData(
: null
if (!cache) return await queryAdminInsights(env, now)
return await cachified({
key: 'admin-insights:v5',
key: 'admin-insights:v6',
cache,
ttl: insightsCacheTtlMs,
getFreshValue: () => queryAdminInsights(env, now),
Expand Down Expand Up @@ -146,11 +146,7 @@ async function queryAdminInsights(
] = await Promise.all([
queryTotals(db),
db
.prepare(
`SELECT COUNT(*) AS total, SUM(enabled) AS enabled,
SUM(success_count) AS success_runs, SUM(error_count) AS error_runs
FROM jobs`,
)
.prepare(`SELECT COUNT(*) AS total, SUM(enabled) AS enabled FROM jobs`)
.first<JobStatsRow>(),
db
.prepare(
Expand Down Expand Up @@ -226,8 +222,8 @@ async function queryAdminInsights(
const jobHealth: AdminInsightsJobHealth = {
totalJobs: Number(jobStats?.total ?? 0),
enabledJobs: Number(jobStats?.enabled ?? 0),
successRuns: Number(jobStats?.success_runs ?? 0),
errorRuns: Number(jobStats?.error_runs ?? 0),
successRuns: runLogInsights.jobSuccessRuns,
errorRuns: runLogInsights.jobErrorRuns,
}

return {
Expand Down Expand Up @@ -494,6 +490,8 @@ async function aggregateRunLogInsights(input: {
...completeRunLogInsights,
workflowStatuses: [],
workflowRuns: 0,
jobSuccessRuns: 0,
jobErrorRuns: 0,
packageRunSucceededUsers: 0,
packageActivatedUsers: 0,
medianHoursToActivation: null,
Expand Down Expand Up @@ -535,6 +533,8 @@ export function foldRunLogSnapshots(
let usersLoaded = 0
const statusCounts = new Map<string, number>()
let workflowRuns = 0
let jobSuccessRuns = 0
let jobErrorRuns = 0
let packageRunSucceededUsers = 0
let packageActivatedUsers = 0
const activationHours: Array<number> = []
Expand All @@ -549,6 +549,8 @@ export function foldRunLogSnapshots(
statusCounts.set(status, (statusCounts.get(status) ?? 0) + count)
workflowRuns += count
}
jobSuccessRuns += Math.max(0, Number(snapshot.jobRunCounts.success) || 0)
jobErrorRuns += Math.max(0, Number(snapshot.jobRunCounts.error) || 0)
let hasRunSucceeded = false
let activatedReachedAt: string | null = null
for (const milestone of snapshot.activationMilestones) {
Expand Down Expand Up @@ -582,6 +584,8 @@ export function foldRunLogSnapshots(
right.count - left.count || left.status.localeCompare(right.status),
),
workflowRuns,
jobSuccessRuns,
jobErrorRuns,
packageRunSucceededUsers,
packageActivatedUsers,
medianHoursToActivation: medianOf(activationHours),
Expand Down
7 changes: 1 addition & 6 deletions packages/worker/src/community/community-flow-test-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -177,12 +177,7 @@ export async function ensureCommunityFlowSchema(db: D1Database) {
updated_at TEXT NOT NULL,
last_run_at TEXT,
last_run_status TEXT,
last_run_error TEXT,
last_duration_ms INTEGER,
next_run_at TEXT NOT NULL,
run_count INTEGER NOT NULL DEFAULT 0,
success_count INTEGER NOT NULL DEFAULT 0,
error_count INTEGER NOT NULL DEFAULT 0
next_run_at TEXT NOT NULL
)`,
`CREATE TABLE IF NOT EXISTS published_bundle_artifacts (
id TEXT PRIMARY KEY,
Expand Down
1 change: 0 additions & 1 deletion packages/worker/src/entitlements/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,6 @@ export async function calculateUserD1StorageBytes(input: {
'timezone',
'caller_context_json',
'last_run_status',
'last_run_error',
'source_id',
'published_commit',
'storage_id',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,30 @@ test('job view hydrate overlays RunLog outcomes for point and batch reads', asyn
successCount: 3,
errorCount: 1,
})
expect(applyJobRunObservabilityToJobView(job, null)).toBe(job)
expect(
applyJobRunObservabilityToJobView(
createJobView({
lastRunAt: '2026-04-20T09:00:00.000Z',
lastRunStatus: 'success',
lastRunError: 'stale D1 error',
lastDurationMs: 999,
runCount: 9,
successCount: 8,
errorCount: 1,
}),
null,
),
).toEqual(
expect.objectContaining({
lastRunAt: '2026-04-20T09:00:00.000Z',
lastRunStatus: 'success',
lastRunError: undefined,
lastDurationMs: undefined,
runCount: 0,
successCount: 0,
errorCount: 0,
}),
)

const env = {} as Env
getJobRunObservability.mockResolvedValue({
Expand Down Expand Up @@ -116,7 +139,14 @@ test('job view hydrate overlays RunLog outcomes for point and batch reads', asyn
userId: 'user-1',
jobs: [
createJobView({ id: 'job-a' }),
createJobView({ id: 'job-b', runCount: 0 }),
createJobView({
id: 'job-b',
lastRunError: 'stale D1 error',
lastDurationMs: 999,
runCount: 9,
successCount: 8,
errorCount: 1,
}),
],
})
expect(getJobRunObservabilityBatch).toHaveBeenCalledWith({
Expand All @@ -133,5 +163,9 @@ test('job view hydrate overlays RunLog outcomes for point and batch reads', asyn
expect(batch[1]).toMatchObject({
id: 'job-b',
runCount: 0,
successCount: 0,
errorCount: 0,
lastRunError: undefined,
lastDurationMs: undefined,
})
})
15 changes: 7 additions & 8 deletions packages/worker/src/jobs/job-run-observability-hydrate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,15 @@ export function applyJobRunObservabilityToJobView(
job: JobView,
observability: JobRunObservabilityRecord | null | undefined,
): JobView {
if (!observability) return job
return {
...job,
lastRunAt: observability.lastRunAt ?? job.lastRunAt,
lastRunStatus: observability.lastRunStatus ?? job.lastRunStatus,
lastRunError: observability.lastRunError ?? undefined,
lastDurationMs: observability.lastDurationMs ?? undefined,
runCount: observability.runCount,
successCount: observability.successCount,
errorCount: observability.errorCount,
lastRunAt: observability?.lastRunAt ?? job.lastRunAt,
lastRunStatus: observability?.lastRunStatus ?? job.lastRunStatus,
lastRunError: observability?.lastRunError ?? undefined,
lastDurationMs: observability?.lastDurationMs ?? undefined,
runCount: observability?.runCount ?? 0,
successCount: observability?.successCount ?? 0,
errorCount: observability?.errorCount ?? 0,
}
}

Expand Down
Loading
Loading