feat(e2e): extend priority-runtimes test to cover all 8 templates - #2172
Merged
Conversation
Tonight's wire-real E2E sweep exposed 12+ root causes across the post- #87 template extraction. Most would have been caught by an actual provision-and-online test running on each template — but the test only covered claude-code + hermes. Extending it to cover all 8 ensures any future regression in any template fails the test, not production. What's added: - run_openai_runtime(runtime, label): generic provisioner for the 5 OpenAI-backed templates (langgraph, crewai, autogen, deepagents, openclaw). Same shape as run_hermes minus the HERMES_* config block that hermes-agent needs. - run_gemini_cli: separate function — gemini-cli wants a Google AI key (E2E_GEMINI_API_KEY), not OpenAI. - Each new runtime registered in the dispatch loop. New `all` keyword for E2E_RUNTIMES runs every covered runtime. claude-code + hermes keep their dedicated functions; both have unique provisioning quirks (claude-code OAuth + claude-code-specific volume mounts; hermes 15-min cold-boot) that don't generalize cleanly. Skip-if-no-key pattern matches the existing one — partially-keyed CI gets clean skips, not false-fails. Usage: E2E_OPENAI_API_KEY=... E2E_RUNTIMES=langgraph ./test_priority_runtimes_e2e.sh E2E_OPENAI_API_KEY=... E2E_RUNTIMES=all ./test_priority_runtimes_e2e.sh Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
April 27, 2026 12:58
HongmingWang-Rabbit
enabled auto-merge
April 27, 2026 12:58
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…AVE (issue #2172) Catches the adk-demo Assistant boot failure class (2026-06-03): workspace config model=moonshot/kimi-k2.6 (claude-code) → adapter derives provider=moonshot → ValueError: provider=moonshot not in providers registry → save was accepted, agent wedged at boot, CI never saw it The drift gate (RFC#580) validates templates; the existing model-side validator (validateRegisteredModelForRuntime, P4 PR-2) catches a (runtime, model) the runtime doesn't own. Neither checked the DERIVED provider's membership in providers.yaml — the gate the adapter actually trips at boot. Fix (issue #2172, fail-closed at config-SAVE): * validateDerivedProviderInRegistry (this PR) — load the manifest, call DeriveProvider(runtime, model, nil) to get the provider the adapter will resolve, and assert the provider name is in the providers list. Returns 422 DERIVED_PROVIDER_NOT_IN_REGISTRY with the sorted list of valid providers (actionable, unlike the boot-time ValueError). Federation contract mirrored from the model-side check (langgraph/external/kimi/mock pass through). * Wired into CreateWorkspace after the existing model-side check. Both gates fail-closed for first-party runtimes and fail-open for non-registry / federated runtimes — the same shape. * TestRegistryConsistency_AllNativeModelsDeriveToKnownProvider — the static regression gate the issue asks for ('a CI test fails if any shipped demo/template config references an unregistered provider'), generalized to the catalog: walk every (runtime, model) in the native model sets and assert each one derives to a provider in the providers list. By construction always true today, but fires on any future drift between providers: and runtimes: in providers.yaml (the exact class cp#455 / boot-e2e targets at the runtime layer). * TestValidateDerivedProviderInRegistry — table-driven pass/fail coverage mirroring TestValidateRegisteredModelForRuntime, plus the langgraph / external / empty-model fail-open cases. Pairs with cp#455 boot-to-registration e2e (the deep runtime layer); this is the fast static layer the issue asked for. Reverts cleanly by deleting the new validator + the wire-up in workspace.go. SOP: /sop-ack engineer-ack as fullstack-engineer Tested: build drift pre-checked; test cases pin both happy path and the federation contract.
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…tinuation) The Create handler already validates (runtime, model) against the provider registry (commit e53a47b). The SetModel endpoint (PUT /workspaces/:id/model) was the remaining unguarded save path — a user could change the model after creation and bypass both the model-registration gate and the derived-provider gate. Fix: - Query the workspace's runtime before persisting the model. - Call validateRegisteredModelForRuntime + validateDerivedProviderInRegistry for non-empty models, mirroring the Create handler order and error shape (422 with code + actionable list). - Return 404 when the workspace does not exist. - Federation contract preserved: unknown runtimes fail-open exactly as in Create. Tests: - Update existing SetModel / RoundTrip mocks to expect the runtime lookup query. - Add TestSecretsSetModel_UnregisteredModel_422. - Add TestSecretsSetModel_UnknownRuntimeFailOpen_200. - Add TestSecretsSetModel_WorkspaceNotFound_404. Pairs with the existing Create-time guard (e53a47b) and the model_registry_validation_test.go regression suite. SOP: /sop-ack engineer-ack as fullstack-engineer
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…+ skip-if-token-absent (core#2225) The post-merge `E2E Staging Canvas (Playwright) / Canvas tabs E2E` job was permanently red for two reasons unrelated to the code under test. 1. Stale fixture (code fix). canvas/e2e/staging-setup.ts created the test workspace with `runtime=hermes, model=gpt-4o`. The provider-registry SSOT (internal#718) registers ONLY Kimi models for the hermes runtime, so the create now correctly 422s UNREGISTERED_MODEL_FOR_RUNTIME. Switched to `moonshot/kimi-k2.6`, the platform-managed hermes entry in workspace-server/internal/providers/providers.yaml (hermes -> platform). The workspace already defaults closed to platform_managed, so a platform-namespaced id is the registry-correct, self-sufficient choice (no tenant LLM key needed). Validated against BOTH create-time gates: the model-side ModelsForRuntime membership check AND the #2172 derived-provider check (moonshot is a declared provider). 2. Missing CI secret (workflow fix). The `Verify admin token present` step hard-failed with `::error::Missing CP_STAGING_ADMIN_API_TOKEN` + exit 2, painting main red on an operator CONFIG gap. Converted to a skip-if-absent gate mirroring the serving-e2e skip-if-secret-unset contract: when the secret is unset it emits a loud ::warning:: + ::notice:: and skips the provision/test steps (job completes green); when present it runs the full suite exactly as before. OPERATOR ACTION: set CP_STAGING_ADMIN_API_TOKEN as a repo/org Actions secret on molecule-core for the E2E to actually execute (it skips until then). Closes #2225 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends `tests/e2e/test_priority_runtimes_e2e.sh` from 2 runtimes (claude-code + hermes) to all 8 templates (adds langgraph, crewai, autogen, deepagents, openclaw, gemini-cli).
Why
Tonight's wire-real E2E sweep exposed 12+ root causes across the post-#87 template extraction. Most would have been caught at PR time by an actual provision-and-online test on each template — but the test only covered the priority adapters. Extending it ensures the next regression in ANY template fails the test, not production.
Shape
Test plan
🤖 Generated with Claude Code