harness: SaaS routing + provider-agnostic config for RFC #2251 measurement - #2261
Merged
hongmingwang-moleculeai merged 3 commits intoApr 29, 2026
Merged
Conversation
…measurement
The original measure-coordinator-task-bounds.sh was hardcoded for
local-dev (workspace-server on :8080) with claude-code/langgraph
templates and OPENROUTER_API_KEY. Running it against staging requires
both auth-chain plumbing (per-tenant ADMIN_TOKEN + X-Molecule-Org-Id
TenantGuard header + tenant subdomain routing) and template/secret
flexibility (e.g. Hermes/MiniMax for Token Plan keys).
This adds:
* `measure-coordinator-task-bounds-runner.sh` — separate runner that
wraps the same workspace-server API calls but takes everything as
env-var inputs. Two MODE values:
- `local` → direct workspace-server (no auth/tenant scoping)
- `saas` → tenant subdomain + per-tenant ADMIN_TOKEN bearer +
X-Molecule-Org-Id TenantGuard header. Auto-fetches
tenant token via CP /cp/admin/orgs/<slug>/admin-token
given ORG_SLUG + CP_ADMIN_API_TOKEN, OR accepts a
pre-resolved TENANT_ADMIN_TOKEN.
* Configurable PM_TEMPLATE / CHILD_TEMPLATE / MODEL / SECRET_NAME /
SECRET_VALUE — defaults match the original (claude-code-default +
langgraph + OpenRouter). Hermes/MiniMax example documented in the
header.
* Per-poll status_change events during wait_online, so a workspace
that never reaches online surfaces its last status (provisioning,
failed, etc.) instead of a bare timeout.
* WAIT_ONLINE_SECS knob (default 180s; SaaS cold-start needs ~420s
for first hermes-image pull on a freshly-provisioned EC2 tenant).
* `${args[@]+...}` guard on the api() helper — avoids `set -u`
exploding on an empty header array (the local-dev hot-path).
The original script also gained a SECRET_VALUE block earlier in the
session — that change (separately staged) makes the secret-name
configurable without forcing every operator through the new runner.
V1.0 gate #1 (RFC #2251, Issue 4 repro) measurement results posted
as a separate comment on molecule-core#2256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
April 29, 2026 05:06
31 tasks
Two docs covering load-bearing patterns from today's work that weren't previously discoverable: 1. workspace/platform_tools/README.md — explains the ToolSpec single-source-of-truth pattern (#2240), the CLI-block alignment gap that hand-maintained generation can't close (#2258), the snapshot golden files + LF-pinning (#2260), and the add/rename/ remove playbook. The next reader who lands in workspace/platform_tools/ now has the design rationale + the safe-edit procedure colocated with the code. 2. scripts/README.md — disambiguates the three measure-coordinator- task-bounds.sh files that now exist across two repos: - scripts/measure-coordinator-task-bounds.sh (canonical OSS, this repo) - scripts/measure-coordinator-task-bounds-runner.sh (Hermes/MiniMax variant, this repo) - scripts/measure-coordinator-task-bounds.sh (production-shape, in molecule-controlplane) Cross-references reference_harness_pair_pattern (auto-memory) for the cross-repo design rationale. Documents the common safety pattern (cleanup trap, DRY_RUN, non-target guard, cleanup_*_failed events) and the heartbeat-trace caveat. Refs: #2240, #2254, #2257, #2258, #2259, #2260; molecule-controlplane#321. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…+ UUID/slug validation Three review-driven fixes to the runner before #2261 merges: 1. `WAIT_ONLINE_SECS / 3` truncated; an operator passing 200 actually waited 198s. Round up so 200 → 67 polls × 3s = 201s ≥ requested. 2. The heartbeat-history endpoint isn't on tenant workspace-servers — the platform's :8080 fallback proxies unmatched paths to the canvas Next.js, so the SaaS run captured 28KB of HTML in the `heartbeat_trace` event log. Skip the fetch in MODE=saas; emit an explicit `<skipped: ...>` placeholder. Local mode behaviour unchanged. 3. ORG_ID and ORG_SLUG had no client-side format check, so a typo'd value got swallowed by TenantGuard's intentionally-opaque 404 (which doesn't tell the operator whether slug, UUID, or auth was wrong). Validate UUID and slug shape up front; matching errors are actionable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…ire→WS upgrade→real frame)' (#2275) from feat/2261-gap1-takecontrol-e2e into main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
V1.0 of the harness was hardcoded for local-dev (
PLATFORM=http://localhost:8080,claude-code-default/langgraphtemplates,OPENROUTER_API_KEY). Running it against staging surfaced two gaps that this PR closes — without touching the canonical script's contract.Changes
scripts/measure-coordinator-task-bounds-runner.sh— separate runner alongside the canonical harness. TwoMODEvalues:local(default) — direct workspace-server, no auth/tenant scopingsaas— tenant subdomain + per-tenant ADMIN_TOKEN bearer +X-Molecule-Org-IdTenantGuard header. Auto-fetches the tenant token viaGET /cp/admin/orgs/<slug>/admin-tokenwhen givenORG_SLUG+CP_ADMIN_API_TOKEN, or accepts a pre-resolvedTENANT_ADMIN_TOKEN.PM_TEMPLATE,CHILD_TEMPLATE,SECRET_NAME,SECRET_VALUE,MODEL. Defaults match the original. Hermes/MiniMax-Token-Plan example in the header docstring.status_changeevents duringwait_onlineso a stuck-in-provisioning workspace surfaces its last status (instead of a bareonline_timeout).WAIT_ONLINE_SECSknob (default 180s; SaaS cold-start needs ~420s on the first hermes image pull).${args[@]+...}guard on the api() helper — defendsset -uagainst an empty header array on the local-dev hot path.The canonical
scripts/measure-coordinator-task-bounds.shkeeps its v1 shape (no template/secret overrides) — this is intentional. The new runner lives next to it for any provider-combo scenarios.Test plan
bash -npasses on both filesDRY_RUN=1 SECRET_VALUE=dummyexits cleanly with the expected provisioning summaryonline, A2A POST returns immediately. (Hermes-agent gateway returns 500 for theminimax/MiniMax-M2.7-highspeedmodel — separate routing bug, see Tracking: V1.0 implementation — operator ceiling + instrumentation + cancellation propagation (RFC #2251) #2256 measurement comment.)rfc2251-repro-*provisioned + torn down via CP admin API): runner provisions both workspaces, PM came online after 7min cold-start (first hermes pull on the EC2), child after 3s. Same hermes-agent 500 on the LLM call. Tenant teardown viaDELETE /cp/admin/tenants/:slugsucceeded.Linked