Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/actions/setup-validate/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,29 @@ runs:
restore-keys: |
${{ runner.os }}-playwright-

- name: 🧊 Cache Nx
uses: actions/cache@v5.0.3
with:
path: .nx/cache
key:
${{ runner.os }}-nx-${{ hashFiles('nx.json', 'package-lock.json')
}}-${{ github.sha }}-${{ github.job }}
restore-keys: |
${{ runner.os }}-nx-${{ hashFiles('nx.json', 'package-lock.json') }}-

- name: 🧊 Enable Nx remote cache
if: env.NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN != ''
shell: bash
run: |
server='https://nx-cache.kody.codes'
if curl -fsS --max-time 5 "$server/health" >/dev/null; then
echo "NX_SELF_HOSTED_REMOTE_CACHE_SERVER=$server" >> "$GITHUB_ENV"
echo "Nx remote cache enabled ($server)"
else
echo "Nx remote cache skipped: $server/health is unreachable."
echo "Validate will use local .nx + actions/cache until the worker is deployed."
fi

- name: 📥 Install Dependencies
shell: bash
run: npm ci
Expand Down
134 changes: 134 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ jobs:
deploy_backup_control_plane:
${{ steps.sha_guard.outputs.deploy_backup_control_plane }}
deploy_status_worker: ${{ steps.sha_guard.outputs.deploy_status_worker }}
deploy_nx_cache_worker:
${{ steps.sha_guard.outputs.deploy_nx_cache_worker }}
steps:
- name: 📦 Checkout
uses: actions/checkout@v6.0.2
Expand Down Expand Up @@ -70,6 +72,7 @@ jobs:
echo "should_deploy=false" >> "$GITHUB_OUTPUT"
echo "deploy_backup_control_plane=false" >> "$GITHUB_OUTPUT"
echo "deploy_status_worker=false" >> "$GITHUB_OUTPUT"
echo "deploy_nx_cache_worker=false" >> "$GITHUB_OUTPUT"
echo "Skipping deploy: target SHA is not current main HEAD."
echo "target=$DEPLOY_SHA"
echo "head=$HEAD_SHA"
Expand All @@ -83,8 +86,10 @@ jobs:
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
echo "deploy_backup_control_plane=true" >> "$GITHUB_OUTPUT"
echo "deploy_status_worker=true" >> "$GITHUB_OUTPUT"
echo "deploy_nx_cache_worker=true" >> "$GITHUB_OUTPUT"
echo "Backup control plane deploy: forced (workflow_dispatch)."
echo "Status worker deploy: forced (workflow_dispatch)."
echo "Nx cache worker deploy: forced (workflow_dispatch)."
exit 0
fi

Expand All @@ -106,6 +111,15 @@ jobs:
echo "Status worker deploy: skipped (no relevant path changes)."
fi

if git diff --name-only "${DEPLOY_SHA}^" "${DEPLOY_SHA}" | grep -E \
'^(\.github/workflows/deploy\.yml|tools/ci/nx-cache-resources|packages/nx-cache/)'; then
echo "deploy_nx_cache_worker=true" >> "$GITHUB_OUTPUT"
echo "Nx cache worker deploy: path filter matched."
else
echo "deploy_nx_cache_worker=false" >> "$GITHUB_OUTPUT"
echo "Nx cache worker deploy: skipped (no relevant path changes)."
fi

# The jobs worker deploys before the main worker on every production
# deploy: its JobManager script-transfer migration must run while the main
# worker still owns the class, and the main worker's JOBS service binding
Expand Down Expand Up @@ -857,3 +871,123 @@ jobs:
sleep "$deploy_delay_seconds"
deploy_delay_seconds=$((deploy_delay_seconds * 2))
done

deploy-nx-cache-worker:
runs-on: ubuntu-latest
name: 🧊 Deploy Nx cache worker
timeout-minutes: 8
needs: sha-guard
if: >-
needs.sha-guard.outputs.should_deploy == 'true' &&
needs.sha-guard.outputs.deploy_nx_cache_worker == 'true'
environment:
name: production
url: https://nx-cache.kody.codes
steps:
- name: 📦 Checkout
uses: actions/checkout@v6.0.2
with:
ref: ${{ needs.sha-guard.outputs.deploy_sha }}
fetch-depth: 0

- name: 🟢 Setup Node
uses: actions/setup-node@v6
with:
node-version: 26
cache: npm

- name: 📥 Install Dependencies
run: npm ci

- name: 🧱 Ensure Nx cache R2 bucket and lifecycle
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
run: node tools/ci/nx-cache-resources.ts

- name: ☁️ Deploy Nx cache worker
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
DEPLOY_COMMIT_SHA: ${{ needs.sha-guard.outputs.deploy_sha }}
CACHE_ACCESS_TOKEN:
${{ secrets.NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN }}
run: |
set -euo pipefail
if [ -z "${CACHE_ACCESS_TOKEN:-}" ]; then
echo "Skipping Nx cache worker deploy: repository secret" >&2
echo "NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN is not set." >&2
echo "Generate one with: openssl rand -hex 32" >&2
exit 0
fi

deploy_attempts=3
deploy_delay_seconds=10
for attempt in $(seq 1 "$deploy_attempts"); do
echo "Nx cache worker deploy attempt $attempt/$deploy_attempts"
set +e
npm run nx-cache:deploy -- \
--var "BUILD_COMMIT:${DEPLOY_COMMIT_SHA}" 2>&1 | tee nx-cache-deploy.log
deploy_status="${PIPESTATUS[0]}"
set -e
if [ "$deploy_status" -eq 0 ]; then
break
fi
if [ "$attempt" -eq "$deploy_attempts" ]; then
echo "Nx cache worker deploy failed after $deploy_attempts attempts." >&2
exit "$deploy_status"
fi
echo "Retrying in ${deploy_delay_seconds}s."
sleep "$deploy_delay_seconds"
deploy_delay_seconds=$((deploy_delay_seconds * 2))
done

- name: 🔐 Sync Nx cache access token
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
CACHE_ACCESS_TOKEN:
${{ secrets.NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN }}
run: |
set -euo pipefail
if [ -z "${CACHE_ACCESS_TOKEN:-}" ]; then
echo "Skipping Nx cache secret sync: token is not set."
exit 0
fi
node tools/ci/sync-worker-secrets.ts --env "" --config \
packages/nx-cache/wrangler.jsonc --name kody-nx-cache \
--set-from-env CACHE_ACCESS_TOKEN
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

- name: 🩺 Healthcheck (Nx cache worker)
shell: bash
env:
EXPECTED_COMMIT_SHA: ${{ needs.sha-guard.outputs.deploy_sha }}
CACHE_ACCESS_TOKEN:
${{ secrets.NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN }}
run: |
set -euo pipefail
if [ -z "${CACHE_ACCESS_TOKEN:-}" ]; then
echo "Skipping Nx cache healthcheck: token is not set."
exit 0
fi
HEALTHCHECK_URL="https://nx-cache.kody.codes/health"
attempts=20
delay_seconds=3
for i in $(seq 1 "$attempts"); do
echo "Attempt $i/$attempts"
echo "Healthcheck URL: $HEALTHCHECK_URL"
if curl --fail --silent --show-error --max-time 10 \
--header "Accept: application/json" \
"$HEALTHCHECK_URL" > nx-cache-health.json && \
node -e "const fs = require('node:fs'); const json = JSON.parse(fs.readFileSync('nx-cache-health.json','utf8')); const expected = process.env.EXPECTED_COMMIT_SHA; if (json?.ok !== true) { console.error('nx-cache-healthcheck-unexpected-response', json); process.exit(1); } if (json?.commit !== expected) { console.error('nx-cache-healthcheck-unexpected-commit', { expected, actual: json?.commit }); process.exit(1); } console.log('nx-cache-healthcheck-ok', json);"; then
exit 0
fi
sleep "$delay_seconds"
done

echo "Nx cache worker healthcheck failed after ${attempts} attempts." >&2
if [ -f nx-cache-health.json ]; then
echo "Last response body:" >&2
cat nx-cache-health.json >&2
fi
exit 1
13 changes: 13 additions & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@ on:
permissions:
contents: read

# Pin CI=1 (not GitHub's CI=true) so Nx cache hashes match local
# `validate` / `test:push`. Remote cache is used only when the access
# token secret is present and setup-validate can reach /health; Nx
# treats an unreachable self-hosted cache as a fatal error.
env:
CI: '1'
NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN:
${{ secrets.NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN }}

concurrency:
group: >-
validate-${{ github.event_name }}-${{ github.event.pull_request.number ||
Expand Down Expand Up @@ -69,6 +78,10 @@ jobs:
if: ${{ !cancelled() }}
run: npm run status:build

- name: 🧊 Nx cache build
if: ${{ !cancelled() }}
run: npm run nx-cache:build

- name: 🗺️ Primitives
if: ${{ !cancelled() }}
run: npm run primitives:check
Expand Down
15 changes: 15 additions & 0 deletions docs/contributing/cloud-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,21 @@ manually with native `unzip`:
headless via the separate headless-shell binary, so both are required).
4. `chmod +x` the `chrome` and `chrome-headless-shell` binaries.

## Nx remote cache

Validate and `test:push` write Nx task artifacts. Those stay local unless the
self-hosted cache is configured. To share hits with GitHub Actions, set both in
this VM (same values as the GitHub Actions secret / Worker secret):

```bash
export NX_SELF_HOSTED_REMOTE_CACHE_SERVER=https://nx-cache.kody.codes
export NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN="$NX_CACHE_TOKEN"
```

Use `CI=1` on cached test commands (the repo scripts already do). Leave the
variables unset to run without remote cache. See
[`packages/nx-cache/readme.md`](../packages/nx-cache/readme.md).

## Quick commands

| Task | Command |
Expand Down
35 changes: 35 additions & 0 deletions docs/contributing/decisions/0019-self-hosted-nx-remote-cache.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# 0019: Self-hosted Nx remote cache (not Nx Cloud)

- **Status:** accepted
- **Date:** 2026-08-16

## Context

Agents already run the same Nx targets CI runs (`test:node`, `test:workers`,
`test:e2e`, `test:mcp`, `typecheck`). Nx was only writing a local `.nx/cache`
(gitignored, `neverConnectToCloud: true`), so GitHub Actions always started
cold. Official S3/GCS/Azure cache packages are deprecated (CVE-2025-36852). Nx
Cloud would share artifacts, but it is a third-party control plane this repo
does not want to prompt for or depend on.

## Decision

Keep `neverConnectToCloud: true`. Share cache through a small Cloudflare Worker
(`packages/nx-cache`, `nx-cache.kody.codes`) that implements the Nx HTTP OpenAPI
spec and stores artifacts in R2. Clients set
`NX_SELF_HOSTED_REMOTE_CACHE_SERVER` and
`NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN`. Validate jobs also restore `.nx` via
`actions/cache` as a same-runner L1. Test hashes include `{env:CI}` and CI pins
`CI=1` so agent `validate` / `test:push` and GitHub Actions share keys.

## Consequences

Remote hits require the access token in GitHub Actions and in Cursor Cloud Agent
environments; without it, tasks run locally as before. Validate jobs probe
`GET /health` before setting the server URL because Nx treats an unreachable
self-hosted cache as a fatal error. First PUT wins (`409` on overwrite) so a
poisoned key cannot replace a stored artifact. R2 expires `v1/` objects 14 days
after write (Age, not last access — GET does not refresh mtime) and aborts
incomplete multipart uploads after 1 day. The cache worker is contributor infra,
not a product primitive, and holds no user data. Revisit only if Nx Cloud
becomes mandatory for a feature we need, or if the HTTP spec changes.
1 change: 1 addition & 0 deletions docs/contributing/decisions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,4 @@ behavior (see [documentation principles](../documentation.md)).
- [0016 — Extract the package runtime and jobs lanes into separate workers](./0016-mono-worker-extraction.md)
- [0017 — Per-user subdomains for hosted package apps](./0017-per-user-package-app-subdomains.md)
- [0018 — Inbound CLA for external contributions to this repository](./0018-inbound-cla.md)
- [0019 — Self-hosted Nx remote cache (not Nx Cloud)](./0019-self-hosted-nx-remote-cache.md)
2 changes: 2 additions & 0 deletions docs/contributing/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ style, tests, MCP capabilities, and runtime architecture.
- [Manual PR preview testing](./preview-manual-testing.md)
- [Optional Cloudflare offerings](./cloudflare-offerings.md)
- [Cursor Cloud Agent notes](./cloud-agents.md)
- [Nx remote cache](../../packages/nx-cache/readme.md) (self-hosted HTTP cache
shared by agents and CI)
- [Harness engineering](./harness-engineering.md) (agent-first loop, promoting
lessons into enforcement)

Expand Down
20 changes: 20 additions & 0 deletions docs/contributing/setup-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ This project uses the following resources:

- D1 database
- `database_name`: `<app-name>`
- R2 bucket for the self-hosted Nx remote cache
- `bucket_name`: `kody-nx-cache`
- Worker: `kody-nx-cache` at `nx-cache.kody.codes` (`packages/nx-cache`)
- Production deploy ensures the bucket and a 14-day `v1/` lifecycle (1-day
incomplete multipart abort) when the cache worker path changes
- KV namespace for OAuth/session storage
- `binding`: `OAUTH_KV`
- `title`: `<app-name>-oauth`
Expand Down Expand Up @@ -511,6 +516,15 @@ Configure these GitHub Actions secrets and variables for workflows:
- `SENTRY_AUTH_TOKEN` (optional GitHub **secret**; Sentry auth token with
`project:releases` / source map upload permissions — used only by CI to run
`npm run sentry:upload-sourcemaps` after deploy)
- `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN` (optional GitHub **secret**; bearer
token for the Nx HTTP cache worker at `https://nx-cache.kody.codes`. Generate
with `openssl rand -hex 32`. Validate jobs enable remote cache when this is
set; production deploy syncs it to the worker as `CACHE_ACCESS_TOKEN` and
creates the `kody-nx-cache` R2 bucket with a 14-day object lifecycle. Use the
same value in Cursor Cloud Agent environments so agent `validate` /
`test:push` can populate CI hits. Validate jobs probe `/health` before setting
the server URL so an undeployed worker does not fail Nx. Leave unset to run CI
with local `.nx` + `actions/cache` only.)
- **Repository variables** `SENTRY_ORG` and `SENTRY_PROJECT` (optional; Sentry
organization and project **slugs** for source map upload — same values as in
the Sentry wizard’s `--org` / `--project` flags)
Expand Down Expand Up @@ -594,6 +608,12 @@ How to get/set each value:
- In Sentry: **Settings → Auth Tokens** (or Organization settings), create a
token that can upload releases/source maps, and store it as the
`SENTRY_AUTH_TOKEN` repository secret.
- `NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN` (optional)
- Generate locally: `openssl rand -hex 32`
- Store the exact value as the repository secret
`NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN`, and use the same value in Cursor
Cloud Agent environments. Production deploy syncs it to the `kody-nx-cache`
Worker as `CACHE_ACCESS_TOKEN`.
- `SENTRY_ORG` / `SENTRY_PROJECT` (optional)
- In GitHub: **Settings → Secrets and variables → Actions → Variables**, add
`SENTRY_ORG` and `SENTRY_PROJECT` with your Sentry slugs (for example from
Expand Down
38 changes: 24 additions & 14 deletions docs/contributing/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,14 @@ Quick notes for getting a local kody environment running.
for the repo, and runs `npm run migrations:check` before the commit is
created.
- `git push` runs the Husky `pre-push` hook, which executes `npm run test:push`
(`CI=1` worker Vitest + Playwright E2E) so pushes are blocked when those
suites fail. Vitest's default `testTimeout` is 20s so the workers pool's first
Durable Object RPC in a file (~10s) does not fail the default budget (see
(`CI=1` `test:node` + `test:workers` + Playwright E2E) so pushes are blocked
when those suites fail. Those are the same Nx targets GitHub Actions runs, so
a remote-cache hit is possible after push. Vitest's default `testTimeout` is
20s so the workers pool's first Durable Object RPC in a file (~10s) does not
fail the default budget (see
[decision 0011](./decisions/0011-workers-unit-pool-harness.md)); the push gate
also sets `CI=1` so worker count and Playwright retries match GitHub Actions.
also sets `CI=1` so worker count, Playwright retries, and Nx cache hashes
match GitHub Actions.
- Because the commit hook already enforces formatting, lint fixes, and
typechecking, agents do not need to run those checks separately before every
commit unless they want earlier feedback or are validating a larger change set
Expand All @@ -85,23 +88,30 @@ Quick notes for getting a local kody environment running.
repo-wide format checks remain explicit checks because they are heavier than
the push gate.
- `npm run validate` is the single authoritative local gate. It is read-only and
executes `format:check`, `lint`, `typecheck`, unit tests, Playwright E2E, MCP
E2E, `backup:build`, `primitives:check`, and `migrations:check` in parallel,
plus `deploy-guardrails:check` and `docs:check-temporal`, reporting every
failure (sibling checks are not aborted on the first failure). The unit-test
and Playwright legs set `CI=1` so timeouts and worker limits match the
contended parallel layout used in GitHub Actions. CI runs the same checks as
parallel jobs (🧹 Static, 🧪 Node, ☁️ Workers, 🔌 MCP, 🎭 E2E, aggregated by
✅ Validate). If `npm run validate` passes locally, CI will pass.
executes `format:check`, `lint`, `typecheck`, `test:node`, `test:workers`,
Playwright E2E, MCP E2E, `backup:build`, `status:build`, `nx-cache:build`,
`primitives:check`, and `migrations:check` in parallel, plus
`deploy-guardrails:check` and `docs:check-temporal`, reporting every failure
(sibling checks are not aborted on the first failure). The unit-test and
Playwright legs set `CI=1` so timeouts, worker limits, and Nx cache hashes
match the contended parallel layout used in GitHub Actions. CI runs the same
checks as parallel jobs (🧹 Static, 🧪 Node, ☁️ Workers, 🔌 MCP, 🎭 E2E,
aggregated by ✅ Validate). If `npm run validate` passes locally, CI will
pass. When `NX_SELF_HOSTED_REMOTE_CACHE_SERVER` and
`NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN` are set, Nx uploads those task
artifacts to `https://nx-cache.kody.codes` so CI can reuse them (see
[decision 0019](./decisions/0019-self-hosted-nx-remote-cache.md) and
[`packages/nx-cache/readme.md`](../../packages/nx-cache/readme.md)).
- `npm run deploy-guardrails:check` protects reviewed Durable Object migration
history and bindings in both Wrangler configs, requires exact allowlisting for
class deletion, and rejects destructive Cloudflare CLI operations in
automatically triggered GitHub Actions jobs.
- `npm run validate:fix` runs `format` + `lint:fix` and is the explicit opt-in
for mutating auto-fixes. It is never required to pass `validate`.
- `npm run format` applies formatting updates on its own.
- `npm run test:push` runs the same worker tests and Playwright E2E suite
enforced by the Husky `pre-push` hook.
- `npm run test:push` runs the same `test:node`, `test:workers`, and Playwright
E2E suites enforced by the Husky `pre-push` hook and by the CI Node / Workers
/ E2E jobs.
- `npm run test:e2e:run` ensures Playwright Chromium is installed before the
suite starts, so `npm run validate` and `npm run test:push` self-heal on a
fresh machine.
Expand Down
Loading
Loading