Skip to content

fix(e2e): 9 follow-ups to make staging E2E actually green end-to-end - #1491

Merged
HongmingWang-Rabbit merged 5 commits into
mainfrom
feat/e2e-staging-saas-cicd
Apr 21, 2026
Merged

HongmingWang-Rabbit merged 5 commits into
mainfrom
feat/e2e-staging-saas-cicd

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

PR #1395 was merged with the scaffolding but the harness still had several gaps that only surfaced when exercised against real staging. This PR bundles the 9 fix commits pushed after #1395 merged — all validated by manual run 18 (2026-04-21T17:47Z) which hit all 11/11 sections green (tenant provision → workspace online → A2A PONG → HMA memory → peer discovery → delegation CHILD_PONG → clean teardown).

Fixes

  1. a510573 — poll instance_status (the actual field name) not status
  2. 37a02d6 — derive tenant domain from CP URL so staging (staging-api.moleculesai.app*.staging.moleculesai.app) works without overriding env
  3. e9d111d — send X-Molecule-Org-Id on all tenant calls (TenantGuard 404s without it — returns 404 by design, not 403)
  4. 81c4c02 — scope safety-net teardown to GITHUB_RUN_ID, not all today's e2e-* orgs (incident: this workflow deleted an unrelated manual run's tenant 1s after it hit running)
  5. 5be20ac — inject OPENAI_API_KEY via workspace secrets field so runtime can actually boot
  6. 392282c — set MODEL_PROVIDER=openai (later superseded)
  7. b8b3d5cMODEL_PROVIDER is provider:model slug, not just provider; use openai:gpt-4o + OPENAI_BASE_URL=api.openai.com
  8. 5e130b7 — delegation raw-curl was missing X-Molecule-Org-Id header (caught because section 10 was the only raw curl; everything using tenant_call helper already got the fix from commit 3)
  9. bd020d8 — wire MOLECULE_STAGING_OPENAI_KEY repo secret into workflow env + preflight verify

Manual verification

Run 18 (2026-04-21T17:41Z → 17:47Z, 5:52 total):

  • 2/11 Tenant provisioning (2:20)
  • 5-7/11 Workspace provision + online (3:25)
  • 8/11 A2A PONG
  • 9/11 HMA memory write/read
  • 9b Peer discovery
  • 10/11 Delegation CHILD_PONG
  • 11/11 Clean teardown, zero leaks

Required repo secrets

  • MOLECULE_STAGING_ADMIN_TOKEN (already set) — CP admin bearer
  • MOLECULE_STAGING_OPENAI_KEY (already set as of this PR) — OpenAI key for workspace runtimes

Hongming Wang and others added 5 commits April 21, 2026 10:18
Workspace runtimes (hermes, langgraph, etc.) crash at boot with
'No provider API key found' when no ANTHROPIC_API_KEY / OPENAI_API_KEY /
etc. is set. Harness previously sent no secrets → workspace sat in
provisioning for 10 min → harness timed out.

Console log from staging run 2026-04-21T17:08Z showed the exact crash:
  ValueError: No Hermes provider API key found. Set any one of:
  ANTHROPIC_API_KEY, HERMES_API_KEY, NOUS_API_KEY, OPENROUTER_API_KEY,
  OPENAI_API_KEY, ...

Read E2E_OPENAI_API_KEY from env and inject into both parent and
child workspace POST bodies via the secrets field (persists as
workspace_secret, materialises into container env). Empty key
falls through — dev can still run smoke tests, workspace just
won't reach online.

For CI, a new repo secret MOLECULE_STAGING_OPENAI_KEY needs to be
added and passed as E2E_OPENAI_API_KEY in the workflow env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hermes's provider resolver checks ANTHROPIC_API_KEY first (resolution
order puts anthropic before openai). Without MODEL_PROVIDER=openai
explicitly set, Hermes defaults to claude-sonnet-4-6 against the
OpenAI endpoint and 404s with model_not_found.

Staging E2E run 2026-04-21T17:24Z hit this after every earlier fix
landed (workspace online, A2A ready) — last remaining blocker for
the happy path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
workspace/config.py:258 reads MODEL_PROVIDER as the full model string
(format 'provider:model', e.g. 'anthropic:claude-opus-4-7'). My prior
'openai' alone got parsed as the model name → 404 model_not_found.

Use 'openai:gpt-4o' and also set OPENAI_BASE_URL to api.openai.com
(default was openrouter.ai which takes different key format).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Section 10's delegation call is a raw curl (not tenant_call, because
it carries an additional X-Source-Workspace-Id). It was missing
X-Molecule-Org-Id, which TenantGuard requires — so the tenant 404'd
every delegation probe despite section 8's A2A call (via tenant_call)
working correctly.

Repro: staging run 2026-04-21T17:40Z had section 8 green (PONG)
and section 10 red (rc=22) on the same workspace. Only difference
was the missing header.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness needs E2E_OPENAI_API_KEY set for Hermes workspaces to
boot — without it the runtime crashes with "No provider API key
found" and workspaces never hit online. Preflight step fails fast
with a clear error if the repo secret is missing, so CI doesn't
burn 10 minutes on a foregone conclusion.

Repo secret to add: Settings → Secrets → Actions →
MOLECULE_STAGING_OPENAI_KEY.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@molecule-ai

molecule-ai Bot commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Code review from Technical Writer (docs focus): LGTM — CI fixes are sound, BASE refactoring is cleaner.

BASE detection fix — LGTM

Simplifying the BASE detection from:

BASE="${GITHUB_BASE_REF:-${{ github.event.before }}}"
if [ "${{ github.event_name }}" = "pull_request" ] && [ -n "${{ github.event.pull_request.base.sha }}" ]; then
  ...
fi

to:

if [ "${{ github.event_name }}" = "pull_request" ]; then
  # GITHUB_BASE_REF is the base branch name (e.g. main/staging)
else
  BASE="${{ github.event.before }}"

This is cleaner. GITHUB_BASE_REF is set by GitHub for all PR events (it's the base branch name, not the SHA — so using it directly as BASE is the right move). No complex fallbacks needed.

MOLECULE_STAGING_OPENAI_KEY — necessary

Adding MOLECULE_STAGING_OPENAI_KEY to the e2e job env is correct. Section 8 (A2A) requires the workspace runtime to make LLM calls; without an OpenAI key, the Hermes runtime crashes at boot. The key must be in the workflow env (injected by secrets.MOLECULE_STAGING_OPENAI_KEY), not in GITHUB_ENV.

Other files

e2e/ test harness changes (MODEL_PROVIDER format, delegation header fix, CI re-trigger comment) — these are operational fixes for the e2e staging harness. They match the PR description (all 11/11 sections green in manual run 18). No docs impact.

No blocking issues. LGTM.

(Note: cannot formally approve — GH_TOKEN is org bot account, same as PR author.)

@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit 2133e56 into main Apr 21, 2026
10 of 11 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the feat/e2e-staging-saas-cicd branch April 24, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant