fix(e2e): 9 follow-ups to make staging E2E actually green end-to-end - #1491
Conversation
Workspace runtimes (hermes, langgraph, etc.) crash at boot with 'No provider API key found' when no ANTHROPIC_API_KEY / OPENAI_API_KEY / etc. is set. Harness previously sent no secrets → workspace sat in provisioning for 10 min → harness timed out. Console log from staging run 2026-04-21T17:08Z showed the exact crash: ValueError: No Hermes provider API key found. Set any one of: ANTHROPIC_API_KEY, HERMES_API_KEY, NOUS_API_KEY, OPENROUTER_API_KEY, OPENAI_API_KEY, ... Read E2E_OPENAI_API_KEY from env and inject into both parent and child workspace POST bodies via the secrets field (persists as workspace_secret, materialises into container env). Empty key falls through — dev can still run smoke tests, workspace just won't reach online. For CI, a new repo secret MOLECULE_STAGING_OPENAI_KEY needs to be added and passed as E2E_OPENAI_API_KEY in the workflow env. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hermes's provider resolver checks ANTHROPIC_API_KEY first (resolution order puts anthropic before openai). Without MODEL_PROVIDER=openai explicitly set, Hermes defaults to claude-sonnet-4-6 against the OpenAI endpoint and 404s with model_not_found. Staging E2E run 2026-04-21T17:24Z hit this after every earlier fix landed (workspace online, A2A ready) — last remaining blocker for the happy path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
workspace/config.py:258 reads MODEL_PROVIDER as the full model string (format 'provider:model', e.g. 'anthropic:claude-opus-4-7'). My prior 'openai' alone got parsed as the model name → 404 model_not_found. Use 'openai:gpt-4o' and also set OPENAI_BASE_URL to api.openai.com (default was openrouter.ai which takes different key format). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Section 10's delegation call is a raw curl (not tenant_call, because it carries an additional X-Source-Workspace-Id). It was missing X-Molecule-Org-Id, which TenantGuard requires — so the tenant 404'd every delegation probe despite section 8's A2A call (via tenant_call) working correctly. Repro: staging run 2026-04-21T17:40Z had section 8 green (PONG) and section 10 red (rc=22) on the same workspace. Only difference was the missing header. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness needs E2E_OPENAI_API_KEY set for Hermes workspaces to boot — without it the runtime crashes with "No provider API key found" and workspaces never hit online. Preflight step fails fast with a clear error if the repo secret is missing, so CI doesn't burn 10 minutes on a foregone conclusion. Repo secret to add: Settings → Secrets → Actions → MOLECULE_STAGING_OPENAI_KEY. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Code review from Technical Writer (docs focus): LGTM — CI fixes are sound, BASE refactoring is cleaner. BASE detection fix — LGTMSimplifying the BASE="${GITHUB_BASE_REF:-${{ github.event.before }}}"
if [ "${{ github.event_name }}" = "pull_request" ] && [ -n "${{ github.event.pull_request.base.sha }}" ]; then
...
fito: if [ "${{ github.event_name }}" = "pull_request" ]; then
# GITHUB_BASE_REF is the base branch name (e.g. main/staging)
else
BASE="${{ github.event.before }}"This is cleaner. MOLECULE_STAGING_OPENAI_KEY — necessaryAdding Other files
No blocking issues. LGTM. (Note: cannot formally approve — GH_TOKEN is org bot account, same as PR author.) |
PR #1395 was merged with the scaffolding but the harness still had several gaps that only surfaced when exercised against real staging. This PR bundles the 9 fix commits pushed after #1395 merged — all validated by manual run 18 (2026-04-21T17:47Z) which hit all 11/11 sections green (tenant provision → workspace online → A2A PONG → HMA memory → peer discovery → delegation CHILD_PONG → clean teardown).
Fixes
a510573— pollinstance_status(the actual field name) notstatus37a02d6— derive tenant domain from CP URL so staging (staging-api.moleculesai.app→*.staging.moleculesai.app) works without overriding enve9d111d— sendX-Molecule-Org-Idon all tenant calls (TenantGuard 404s without it — returns 404 by design, not 403)81c4c02— scope safety-net teardown toGITHUB_RUN_ID, not all today'se2e-*orgs (incident: this workflow deleted an unrelated manual run's tenant 1s after it hit running)5be20ac— injectOPENAI_API_KEYvia workspacesecretsfield so runtime can actually boot392282c— setMODEL_PROVIDER=openai(later superseded)b8b3d5c—MODEL_PROVIDERisprovider:modelslug, not just provider; useopenai:gpt-4o+OPENAI_BASE_URL=api.openai.com5e130b7— delegation raw-curl was missingX-Molecule-Org-Idheader (caught because section 10 was the only raw curl; everything usingtenant_callhelper already got the fix from commit 3)bd020d8— wireMOLECULE_STAGING_OPENAI_KEYrepo secret into workflow env + preflight verifyManual verification
Run 18 (2026-04-21T17:41Z → 17:47Z, 5:52 total):
Required repo secrets
MOLECULE_STAGING_ADMIN_TOKEN(already set) — CP admin bearerMOLECULE_STAGING_OPENAI_KEY(already set as of this PR) — OpenAI key for workspace runtimes