fix(tenant-image): remove node user so canvas uid 1000 can be created - #1445
Merged
Merged
Conversation
…free teardown
Dedicated CI/CD lane that exercises the whole SaaS cross-EC2 shape end to
end, against live staging:
1. Accept terms / create org (POST /cp/orgs) — catches ToS gate, slug
validation, billing/quota, member insert regressions.
2. Wait for tenant EC2 + cloudflared tunnel + TLS propagation (up to
15 min cold).
3. Provision a parent + child workspace via the tenant URL.
4. Wait both online (exercises the SaaS register + token bootstrap
flow fixed in #1364).
5. A2A round-trip on parent — validates the full LLM loop (MCP tools,
provider auth, JSON-RPC response shape, proxy SSRF gate).
6. HMA memory write + read — validates awareness namespace + scope
routing.
7. Peers + activity smoke — route-registration regression guard.
8. Teardown via DELETE /cp/admin/tenants/:slug + leak assertion — a
leaked org at teardown fails CI with exit 4.
Why a dedicated workflow (not folded into ci.yml):
- ~20 min wall clock per run (EC2 boot is the long pole). Too slow
for every PR push.
- Needs its own concurrency group (staging has an org-create quota
and two overlapping runs would race on slug prefix).
- Distinct secret surface (session cookie + admin bearer) — keep it
off PR jobs that don't need them.
Triggers: push to main (provisioning-critical paths only), PRs on the
same paths, manual workflow_dispatch (with runtime + keep_org inputs),
and 07:00 UTC nightly cron for drift detection.
Belt-and-braces teardown: the script installs an EXIT trap, and the
workflow has an always()-step that greps e2e-YYYYMMDD-* orgs created
today and force-deletes them via the idempotent admin endpoint. Covers
the case where GH cancels the runner before the trap fires.
Docs: tests/e2e/STAGING_SAAS_E2E.md — what's covered, how to provision
the two required secrets, local-dev notes, cost (~$0.007/run), known
gaps (canvas UI + delegation + claude-code).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three additions on top of 187a9bf: 1. Canary (.github/workflows/canary-staging.yml) 30-min cron that runs the full-SaaS harness in E2E_MODE=canary: one hermes workspace + one A2A PONG + teardown. ~8-min wall clock vs ~20-min for the full run. Alerting is self-contained: opens a single 'Canary failing' issue on first failure, comments on subsequent failures (no issue spam), auto-closes the issue on the next green run. Labels: canary-staging, bug. Safety-net teardown step sweeps e2e-YYYYMMDD-canary-* orgs tagged today so a runner cancel can't leak EC2. 2. Canvas Playwright (canvas/e2e/staging-*.ts + playwright.staging.config.ts + .github/workflows/e2e-staging-canvas.yml) staging-setup.ts provisions a fresh org + hermes workspace (same lifecycle as the bash harness, just in TypeScript). staging-tabs.spec.ts clicks through all 13 workspace-panel tabs (chat, activity, details, skills, terminal, config, schedule, channels, files, memory, traces, events, audit) and asserts each renders without crashing and without 'Failed to load' error toasts. Known SaaS gaps (Files empty, Terminal disconnects, Peers 401) are documented in #1369 and whitelisted so they don't fail the test — the gate is 'no hard crash', not 'no issues'. staging-teardown.ts deletes the org via DELETE /cp/admin/tenants/:slug. playwright.staging.config.ts separates staging from local tests so pnpm test in dev doesn't try to provision against staging. Retries=2 and timeouts are longer; workers=1 because the setup provisions one shared workspace. Workflow uploads HTML report + screenshots on failure for 14 days. 3. Delegation mechanics (tests/e2e/test_staging_full_saas.sh section 10) Parent → child proxy test: POST /workspaces/CHILD/a2a with X-Source-Workspace-Id=PARENT and verify the child responds + child activity log captures PARENT as source. Intentionally LLM-free: the mechanics regression is what matters; prompt-driven delegation correctness belongs in canvas-driven tests. Also reorders teardown step to 11/11 since delegation is 10/11. Mode gating: E2E_MODE=canary -> skips child workspace, HMA memory, peers, activity, delegation (steps 6, 9, 10 no-op). Full-lifecycle still runs every piece. Validated both paths via 'bash -n' syntax check after each edit. Secrets requirement unchanged (same two secrets as 187a9bf): MOLECULE_STAGING_SESSION_COOKIE, MOLECULE_STAGING_ADMIN_TOKEN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rkflow
Reduces required secret surface from 2 (session cookie + admin token)
to 1 (admin token). Pairs with molecule-controlplane#202 which adds:
- POST /cp/admin/orgs — server-to-server org creation
- GET /cp/admin/orgs/:slug/admin-token — per-tenant bearer fetch
With those endpoints live, CI doesn't need to scrape a browser WorkOS
session cookie. CP admin bearer (Railway CP_ADMIN_API_TOKEN) drives
provision + tenant-token retrieval + teardown through a single
credential.
Changes
-------
test_staging_full_saas.sh: admin bearer for provision/teardown,
fetched per-tenant token drives all tenant API calls. Added
E2E_INTENTIONAL_FAILURE=1 toggle that poisons the tenant token
after provisioning so the teardown path gets exercised when the
happy-path isn't.
canvas/e2e/staging-setup.ts: same pivot; exports STAGING_TENANT_TOKEN
instead of STAGING_SESSION_COOKIE.
canvas/e2e/staging-tabs.spec.ts: context.setExtraHTTPHeaders with
Authorization: Bearer on every page request, no cookie handling.
All three workflows (e2e-staging-saas, canary-staging,
e2e-staging-canvas): drop MOLECULE_STAGING_SESSION_COOKIE env +
verification step. One secret to set.
NEW e2e-staging-sanity.yml: weekly Mon 06:00 UTC. Runs the harness
with E2E_INTENTIONAL_FAILURE=1 and inverts the pass condition —
rc=1 is green, rc=0 (unexpected success) or rc=4 (leak) open a
priority-high issue labelled e2e-safety-net. This is the
answer to 'how do we know the teardown path still works when
nothing else has failed recently.'
STAGING_SAAS_E2E.md refreshed: single-secret setup, sanity workflow
documented, canvas workflow added to the coverage matrix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rm_token' Verified against live staging: the admin endpoint returns 400 'confirm field must equal the URL slug' when the body key is 'confirm_token'. Every workflow's safety-net teardown step + the main harness + the Playwright teardown all had the wrong key. Fixed all six call sites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/cp/admin/orgs exposes `instance_status` (COALESCE'd from org_instances.status), NOT a top-level `status` field. The harness polled the wrong field and always read empty → timed out at 15min on a tenant that had actually provisioned successfully (confirmed 2026-04-21T14:22Z: EC2 launched, canary ok, but harness never saw status=running). No code change to the admin API — the field has never been named `status`. The harness just had a typo that happened to type-check (the Go struct hasn't changed, only the sh/py polling was wrong). Now the harness correctly reads `instance_status` and the main provision poll loop terminates on the expected transition. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous hardcode `$SLUG.moleculesai.app` only matched prod. Staging tenants live at `$SLUG.staging.moleculesai.app`, so the harness hit DNS for a nonexistent host and timed out at section 4 even after provisioning succeeded. Derive from CP URL: api.X → X, staging-api.X → staging.X. Override via MOLECULE_TENANT_DOMAIN for self-hosted setups. Confirmed gap on manual run 2026-04-21T14:40Z: section 2 passed in 2min but section 4 timed out at 3min on the wrong hostname. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TenantGuard middleware on the tenant platform returns 404 (not 403, by design — avoid leaking tenant existence to org scanners) when requests lack X-Molecule-Org-Id matching MOLECULE_ORG_ID. Harness hit this on POST /workspaces (section 5) despite having a valid Authorization bearer. - Capture org_id from admin-create response - Send X-Molecule-Org-Id on every tenant_call Confirmed via manual repro 2026-04-21T14:56Z: curl with Bearer but no org-id header → 404; with both headers → expected route reached. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously matched every e2e-YYYYMMDD-* slug, which stomped parallel
CI runs AND manual dev probes against staging. Incident 2026-04-21
15:02Z: this workflow's safety net deleted an unrelated manual tenant
1s after it hit 'running', timing out the dev run at 15min.
Scope to f'e2e-{today}-{GITHUB_RUN_ID}-' so each run only cleans its
own leftovers. Empty run_id (local invocation) keeps the old broader
behaviour so dev safety-nets still sweep.
Also fix: the previous filter used o.get('status') which doesn't exist
on the admin API response. Now reads instance_status (the real field).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
node:20-alpine ships with a `node` user at uid/gid 1000. The Dockerfile tried `addgroup -g 1000 canvas` which fails with exit 1 because 1000 is already taken. Publish-workspace-server-image workflow has been red for hours — tenant image :latest stuck on a digest that predates the X-Molecule-Admin-Token CPProvisioner fix. Staging workspace provisioning 401'd because the stale tenant binary never sent the admin header. Delete node user+group first (tolerant of future base-image changes that might not ship it), then create canvas at 1000/1000 as before. Mounted volumes continue to expect uid 1000. Repro: publish-workspace-server-image workflow run 24731870797: "process addgroup -g 1000 canvas && adduser... exit code: 1". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dockerfile.tenant addgroup -g 1000 canvas fails because node:20-alpine ships with
nodeuser/group at 1000. Publish workflow has been red for hours, meaning :latest is stale and missing the X-Molecule-Admin-Token CPProvisioner fix (commit 3c25211). Delete node user first, then recreate canvas at 1000.