Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -686,7 +686,7 @@ jobs:
needs.sha-guard.outputs.deploy_status_worker == 'true'
environment:
name: production
url: https://status.heykody.dev
url: https://status.kody.codes
steps:
- name: 📦 Checkout
uses: actions/checkout@v6.0.2
Expand Down Expand Up @@ -749,19 +749,26 @@ jobs:
EXPECTED_COMMIT_SHA: ${{ needs.sha-guard.outputs.deploy_sha }}
run: |
set -euo pipefail
HEALTHCHECK_URL="https://status.heykody.dev/health"
echo "Healthcheck URL: $HEALTHCHECK_URL"
# Canonical host first. Fall back to the legacy host so a 1016 on
# status.kody.codes (DNS not attached yet) does not fail the
# deploy. /health is sticky on .dev and is not 308'd. Other
# component probes never use these hostnames.
CANONICAL_HEALTH="https://status.kody.codes/health"
LEGACY_HEALTH="https://status.heykody.dev/health"

attempts=20
delay_seconds=3
for i in $(seq 1 "$attempts"); do
echo "Attempt $i/$attempts"
if curl --fail --silent --show-error --location --max-time 10 \
--header "Accept: application/json" \
"$HEALTHCHECK_URL" > status-health.json && \
node -e "const fs = require('node:fs'); const json = JSON.parse(fs.readFileSync('status-health.json','utf8')); const expected = process.env.EXPECTED_COMMIT_SHA; if (json?.ok !== true) { console.error('status-healthcheck-unexpected-response', json); process.exit(1); } if (json?.commit !== expected) { console.error('status-healthcheck-unexpected-commit', { expected, actual: json?.commit }); process.exit(1); } console.log('status-healthcheck-ok', json);"; then
exit 0
fi
for HEALTHCHECK_URL in "$CANONICAL_HEALTH" "$LEGACY_HEALTH"; do
echo "Healthcheck URL: $HEALTHCHECK_URL"
if curl --fail --silent --show-error --max-time 10 \
--header "Accept: application/json" \
"$HEALTHCHECK_URL" > status-health.json && \
node -e "const fs = require('node:fs'); const json = JSON.parse(fs.readFileSync('status-health.json','utf8')); const expected = process.env.EXPECTED_COMMIT_SHA; if (json?.ok !== true) { console.error('status-healthcheck-unexpected-response', json); process.exit(1); } if (json?.commit !== expected) { console.error('status-healthcheck-unexpected-commit', { expected, actual: json?.commit }); process.exit(1); } console.log('status-healthcheck-ok', json);"; then
exit 0
fi
done
sleep "$delay_seconds"
done

Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/architecture/primitives.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ primitives:
group: surfaces
name: Public status page
summary:
Independently deployed status worker (status.heykody.dev) probing public
Independently deployed status worker (status.kody.codes) probing public
endpoints every minute, storing history in its own Durable Object, and
emailing operator alerts under a capped policy.
code:
Expand Down
11 changes: 7 additions & 4 deletions docs/contributing/decisions/0004-status-page-separate-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ self-contained need; the repo already had a second-worker precedent in
## Decision

The status page is an independently deployed worker (`packages/status/`,
`status.heykody.dev`) that observes the product strictly from the outside via
public endpoints, and stores probe history, incidents, and notification state in
its own Durable Object — never in `APP_DB`. The main worker exposes
`status.kody.codes`) that observes the product strictly from the outside via
public endpoints (and a jobs-worker service binding, not a public jobs
hostname), and stores probe history, incidents, and notification state in its
own Durable Object — never in `APP_DB`. The main worker exposes
`GET /health/components` (cheap per-binding checks) so the prober can report
storage subsystems individually. Operator alert email goes through the
Cloudflare Email REST API under a strict policy: one email per outage episode,
Expand All @@ -38,4 +39,6 @@ top; do not move the status page into the main worker. The status worker
duplicates a small amount of email-sending code rather than importing from
`packages/worker` (import boundaries keep it dependency-free). Incident records
are probe-derived only; manually posted incident narratives are a possible later
addition, not built now.
addition, not built now. `status.heykody.dev` stays a worker custom domain for
legacy links and 308s to `status.kody.codes` except `/health`, which remains
reachable on the legacy host if the canonical hostname is not attached yet.
16 changes: 11 additions & 5 deletions docs/contributing/setup-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,12 +289,18 @@ production-identity, and restore-baseline registries.

### Status page worker

The public status page (`packages/status/`, served at `status.heykody.dev` via a
wrangler custom domain on the production zone) is an independently deployed
The public status page (`packages/status/`, served at `status.kody.codes` via a
wrangler custom domain on the `kody.codes` zone) is an independently deployed
Worker with a cron trigger and one `StatusStore` Durable Object (SQLite). It
probes public endpoints on the main worker and `kody.run` every minute and never
touches `APP_DB` (see decision record
[0004](./decisions/0004-status-page-separate-worker.md)).
probes public endpoints on the main worker and package-runtime liveness on
`kody.run`, and probes the jobs worker over a service binding — never through
the main app and never via a public jobs hostname. It never touches `APP_DB`
(see decision record [0004](./decisions/0004-status-page-separate-worker.md)).
`status.heykody.dev` remains attached as a legacy custom domain: GET/HEAD other
than `/health` 308 to `status.kody.codes`. `/health` stays sticky on the legacy
host so deploys can still probe the worker if the canonical hostname returns
Cloudflare 1016 until DNS exists. Component probes do not use the status
hostname.

Code deploys are automated by the production deploy workflow
(`.github/workflows/deploy.yml` job `deploy-status-worker`) when a `main` push
Expand Down
89 changes: 89 additions & 0 deletions packages/jobs-worker/src/health.node.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
import { expect, test } from 'vitest'
import { consoleWarn } from '#worker/test-support/console-spies.ts'
import {
collectJobsHealthComponents,
handleJobsHealthRequest,
} from './health.ts'

function healthyDb() {
return {
prepare: () => ({ first: async () => ({ 1: 1 }) }),
} as unknown as D1Database
}

test('jobs health reports liveness and treats JOBS_DB failure as components-down', async () => {
const liveness = await handleJobsHealthRequest(
new Request('https://kody-jobs.example/health'),
{ APP_COMMIT_SHA: 'abc123def456' },
)
expect(liveness?.status).toBe(200)
await expect(liveness?.json()).resolves.toEqual({
ok: true,
commit: 'abc123def456',
})

const healthy = await collectJobsHealthComponents({
APP_COMMIT_SHA: 'abc123def456',
JOBS_DB: healthyDb(),
})
expect(healthy.ok).toBe(true)
expect(healthy.commit).toBe('abc123def456')
expect(healthy.components).toEqual([
expect.objectContaining({ id: 'jobs_db', ok: true }),
])

const healthyResponse = await handleJobsHealthRequest(
new Request('https://kody-jobs.example/health/components'),
{ APP_COMMIT_SHA: 'abc123def456', JOBS_DB: healthyDb() },
)
expect(healthyResponse?.status).toBe(200)
await expect(healthyResponse?.json()).resolves.toMatchObject({
ok: true,
commit: 'abc123def456',
components: [expect.objectContaining({ id: 'jobs_db', ok: true })],
})

consoleWarn.mockImplementation(() => {})
const failedResponse = await handleJobsHealthRequest(
new Request('https://kody-jobs.example/health/components'),
{
APP_COMMIT_SHA: 'abc123def456',
JOBS_DB: {
prepare: () => ({
first: async () => {
throw new Error('database is unavailable')
},
}),
} as unknown as D1Database,
},
)
expect(failedResponse?.status).toBe(503)
await expect(failedResponse?.json()).resolves.toMatchObject({
ok: false,
components: [
expect.objectContaining({ id: 'jobs_db', ok: false, error: 'error' }),
],
})
expect(consoleWarn).toHaveBeenCalledWith(
'jobs-health-component-failed',
expect.any(String),
)

const missing = await collectJobsHealthComponents({
APP_COMMIT_SHA: undefined,
})
expect(missing.ok).toBe(false)
expect(missing.commit).toBeNull()
expect(missing.components[0]).toMatchObject({
id: 'jobs_db',
ok: false,
error: 'unavailable',
})

expect(
await handleJobsHealthRequest(
new Request('https://kody-jobs.example/other'),
{ JOBS_DB: healthyDb() },
),
).toBeNull()
})
111 changes: 111 additions & 0 deletions packages/jobs-worker/src/health.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
import { runD1WithRetry } from '@kody-internal/shared/d1-retry.ts'

/**
* Liveness and JOBS_DB checks for the jobs worker. The public status page
* probes these over a service binding (no public jobs hostname). JOBS_DB
* rides on the same Jobs component — there is no separate storage card.
*/

const componentCheckTimeoutMs = 3_000
const d1CheckRetryOptions = { maxAttempts: 3 } as const
const noStoreHeaders = { 'Cache-Control': 'no-store' } as const

export type JobsHealthComponentId = 'jobs_db'

export type JobsHealthComponentResult = {
id: JobsHealthComponentId
ok: boolean
latencyMs: number
error?: 'timeout' | 'unavailable' | 'error'
}

export type JobsHealthComponentsReport = {
ok: boolean
commit: string | null
checkedAt: string
components: Array<JobsHealthComponentResult>
}

type JobsHealthEnv = {
JOBS_DB?: D1Database
APP_COMMIT_SHA?: string
}

async function checkJobsDb(
db: D1Database | undefined,
): Promise<JobsHealthComponentResult> {
const startedAt = Date.now()
if (!db) {
return { id: 'jobs_db', ok: false, latencyMs: 0, error: 'unavailable' }
}
let timeoutHandle: ReturnType<typeof setTimeout> | undefined
const timeout = new Promise<'timeout'>((resolve) => {
timeoutHandle = setTimeout(
() => resolve('timeout'),
componentCheckTimeoutMs,
)
})
try {
const outcome = await Promise.race([
runD1WithRetry(
() => db.prepare('SELECT 1').first(),
d1CheckRetryOptions,
).then(() => 'ok' as const),
timeout,
])
const latencyMs = Date.now() - startedAt
if (outcome === 'timeout') {
console.warn(
'jobs-health-component-timeout',
JSON.stringify({ id: 'jobs_db' }),
)
return { id: 'jobs_db', ok: false, latencyMs, error: 'timeout' }
}
return { id: 'jobs_db', ok: true, latencyMs }
} catch (error) {
const latencyMs = Date.now() - startedAt
console.warn(
'jobs-health-component-failed',
JSON.stringify({
id: 'jobs_db',
message: error instanceof Error ? error.message : String(error),
}),
)
return { id: 'jobs_db', ok: false, latencyMs, error: 'error' }
} finally {
clearTimeout(timeoutHandle)
}
}

export async function collectJobsHealthComponents(
env: JobsHealthEnv,
): Promise<JobsHealthComponentsReport> {
const jobsDb = await checkJobsDb(env.JOBS_DB)
return {
ok: jobsDb.ok,
commit: env.APP_COMMIT_SHA ?? null,
checkedAt: new Date().toISOString(),
components: [jobsDb],
}
}

export async function handleJobsHealthRequest(
request: Request,
env: JobsHealthEnv,
): Promise<Response | null> {
const url = new URL(request.url)
if (url.pathname === '/health') {
return Response.json(
{ ok: true, commit: env.APP_COMMIT_SHA ?? null },
{ headers: noStoreHeaders },
)
}
if (url.pathname === '/health/components') {
const report = await collectJobsHealthComponents(env)
return Response.json(report, {
status: report.ok ? 200 : 503,
headers: noStoreHeaders,
})
}
return null
}
10 changes: 3 additions & 7 deletions packages/jobs-worker/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import * as Sentry from '@sentry/cloudflare'
import { type JobsWorkerEnv } from './env.ts'
import { handleJobsHealthRequest } from './health.ts'
import { JobManager } from './manager-do.ts'
import {
dispatchScheduledLanes,
Expand All @@ -12,13 +13,8 @@ export { JobManager, JobsService }

const handler = {
async fetch(request: Request, env: JobsWorkerEnv) {
const url = new URL(request.url)
if (url.pathname === '/health') {
return Response.json(
{ ok: true, commit: env.APP_COMMIT_SHA ?? null },
{ headers: { 'Cache-Control': 'no-store' } },
)
}
const healthResponse = await handleJobsHealthRequest(request, env)
if (healthResponse) return healthResponse
return new Response('Not found', { status: 404 })
},
async scheduled(
Expand Down
14 changes: 7 additions & 7 deletions packages/status/email-policy.node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,18 +97,18 @@ test('composed emails carry component names, status page link, and escape html',
const content = composeStatusEmail({
kind,
openIncidents: [incident],
statusPageUrl: 'https://status.heykody.dev',
statusPageUrl: 'https://status.kody.codes',
now: baseNow,
})
expect(content.subject).toContain('[kody status]')
expect(content.text).toContain('https://status.heykody.dev')
expect(content.html).toContain('https://status.heykody.dev')
expect(content.text).toContain('https://status.kody.codes')
expect(content.html).toContain('https://status.kody.codes')
expect(content.html).not.toContain('<script>')
}
const opened = composeStatusEmail({
kind: 'incident_opened',
openIncidents: [incident],
statusPageUrl: 'https://status.heykody.dev',
statusPageUrl: 'https://status.kody.codes',
now: baseNow,
})
expect(opened.subject).toContain('App & API')
Expand All @@ -129,7 +129,7 @@ test('outage emails annotate active relevant Cloudflare incidents', () => {
const opened = composeStatusEmail({
kind: 'incident_opened',
openIncidents: [openIncident()],
statusPageUrl: 'https://status.heykody.dev',
statusPageUrl: 'https://status.kody.codes',
now: baseNow,
providerIncidents,
})
Expand All @@ -141,7 +141,7 @@ test('outage emails annotate active relevant Cloudflare incidents', () => {
const reminder = composeStatusEmail({
kind: 'daily_reminder',
openIncidents: [openIncident()],
statusPageUrl: 'https://status.heykody.dev',
statusPageUrl: 'https://status.kody.codes',
now: baseNow,
providerIncidents,
})
Expand All @@ -152,7 +152,7 @@ test('outage emails annotate active relevant Cloudflare incidents', () => {
const allClear = composeStatusEmail({
kind: 'all_clear',
openIncidents: [],
statusPageUrl: 'https://status.heykody.dev',
statusPageUrl: 'https://status.kody.codes',
now: baseNow,
providerIncidents,
})
Expand Down
Loading
Loading