test: regression guards for 2026-04-23 hermes + CP bug wave - #1902
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
9a710c4 to
c4811ca
Compare
Review — PR #1902 ✅ LGTM (comprehensive regression guard suite)test: regression guards for 2026-04-23 hermes + CP bug wave Quality assessmentThree complementary regression test suites, all well-documented with bug references: 1. E2E A2A error pattern checks ( 2. Hermes bridge shell harness ( 3. Canvas ConfigTab vitest ( 4. CPProvisioner instance ID tests ( No concernsTests are regression guards, not new features. All code is additive. Comment density is high but justified — each test pins a specific production incident. VerdictAPPROVE (comment). Excellent test coverage. Guards the 2026-04-23 bug wave comprehensively. Ready to merge. |
c4811ca to
170a591
Compare
Three complementary regression tests for the chain of P0s fixed today. Each targets a specific bug class that reached production, and will fire loud if any of them regress. ## 1. E2E A2A assertion enhancements (tests/e2e/test_staging_full_saas.sh) The existing A2A check looked for "error|exception" in the response text, which was too broad and missed the actual error patterns we hit. Now matches each known error class individually with a diagnostic fail message pointing at the exact bug: - "[hermes-agent error 401]" → hermes #12 (API_SERVER_KEY) - "hermes-agent unreachable" → gateway process died - "model_not_found" → hermes #13 (model prefix) - "Encrypted content is not supported" → hermes #14 (api_mode) - "Unknown provider" → bridge PROVIDER misconfig Also asserts the response contains the PONG token the prompt asked for — catches silent-truncation/echo regressions. ## 2. Hermes install.sh bridge shell harness (tools/test-hermes-bridge.sh) 4 scenarios × 16 assertions, all offline (no docker, no network): - openai-bridge-happy: OPENAI_API_KEY + openai/gpt-4o → provider=custom, model="gpt-4o" (prefix stripped), api_mode=chat_completions - operator-custom-wins: explicit HERMES_CUSTOM_* → bridge skipped - openrouter-not-touched: OPENROUTER_API_KEY → provider=openrouter, slug kept - non-prefixed-model: bare "gpt-4o" → prefix-strip is a no-op Runs in <1s, can be wired into template-hermes CI. Pins the exact config.yaml shape — any drift in derive-provider.sh or the bridge if-block breaks a test. ## 3. Canvas ConfigTab hermes tests (ConfigTab.hermes.test.tsx) 5 vitest cases covering the #1894 bugs: - Runtime loads from workspace metadata when config.yaml missing - "No config.yaml found" red error hidden for hermes - Hermes info banner shown instead - Langgraph workspace still sees the red error (regression-guard the other way) - config.yaml runtime wins over workspace metadata when present ## Running bash tools/test-hermes-bridge.sh # 16 assertions cd canvas && npx vitest run src/components/tabs/__tests__/ConfigTab.hermes.test.tsx # 5 cases # E2E enhancements ride on the existing staging E2E workflow ## Not yet covered (tracked in #1900) CP admin delete-tenant EC2 cascade, cp-provisioner instance_id lookup (#1738), purge audit SQL mismatch (#241), and pq prepared- statement cache collision (#242). These are in-controlplane-repo concerns — separate PR with CP-side sqlmock + integration tests. Closes items in #1900. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pins the fix-invariants from PR #1738 (merged 2026-04-23) against regression. Pre-fix, `CPProvisioner.Stop` and `IsRunning` both passed the workspace UUID as the `instance_id` query param: url := fmt.Sprintf("%s/cp/workspaces/%s?instance_id=%s", baseURL, workspaceID, workspaceID) ^ should be the real i-* ID AWS rejected downstream with InvalidInstanceID.Malformed, orphaned the EC2, and the next provision hit InvalidGroup.Duplicate on the leftover SG — full Save & Restart cascade failure. ## Tests added - **TestStop_UsesRealInstanceIDNotWorkspaceUUID**: stub resolveInstanceID to return an i-* ID, assert the CP request's instance_id query param carries that i-* value (not the workspace UUID). - **TestStop_NoInstanceIDSkipsCPCall**: empty DB lookup → no CP call at all (idempotent). Guards against re-introducing the "call CP with '' and let AWS reject" footgun. - **TestIsRunning_UsesRealInstanceIDNotWorkspaceUUID**: mirror for the /cp/workspaces/:id/status path — same bug shape. All 3 pass on current staging (which has the fix). Reverting either Stop or IsRunning to the pre-#1738 shape causes these to fail loud. Extends molecule-core#1902's regression suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
170a591 to
9ce8d97
Compare
![molecule-ai[bot]](https://avatars.githubusercontent.com/u/275794187?s=60&v=4){kind=small}
There was a problem hiding this comment.
CI green, no conflicts. Approving — ready for @airenostars merge.
1 participant
Closes items in #1900.
3 complementary regression tests for today's P0 chain
1. E2E A2A specific error-pattern checks (
tests/e2e/test_staging_full_saas.sh)Old assertion: substring-match
error|exception. Too broad — missed the actual failure modes + poor diagnostics. Now each known error class has a dedicated check with a diagnostic fail pointing at the exact bug:[hermes-agent error 401]→ hermes#12 (API_SERVER_KEY not in runtime env)hermes-agent unreachable→ gateway process diedmodel_not_found→ hermes#13 (slug prefix)Encrypted content is not supported→ hermes#14 (api_mode)Unknown provider→ bridge PROVIDER misconfig regressionAlso asserts PONG token in response — catches silent echo/truncation.
2. Hermes install.sh bridge shell harness (
tools/test-hermes-bridge.sh)4 scenarios × 16 assertions, offline, <1s. Invokes derive-provider.sh + bridge if-block against fixed env permutations:
Every assertion pins an invariant from today's fixes. Drift breaks a test.
3. Canvas ConfigTab hermes vitest (
ConfigTab.hermes.test.tsx)5 test cases pinning #1894 behavior:
Run locally
Not yet covered
CP-side tests (admin-delete cascade, cp-provisioner instance_id, purge audit SQL, pq statement cache) are tracked in #1900 — separate PRs in the controlplane repo.
Includes
This PR cherry-picks #1898 so the vitest tests pass on this branch's base — the canvas fix + its tests ship together. When #1898 merges first, this will resolve cleanly on rebase.