chore: sync staging → main (post 2026-04-23 bug wave, conflicts resolved) - #1913
Merged
HongmingWang-Rabbit merged 1251 commits intoApr 23, 2026
Merged
Conversation
PR #966 intentionally stripped current_task, last_sample_error, and workspace_dir from the public GET /workspaces/:id response to avoid leaking task bodies to anyone with a workspace bearer. The E2E smoke test hadn't caught up — it was still asserting "current_task":"..." on the single-workspace GET, which made every post-#966 CI run fail with '60 passed, 2 failed'. Swap the per-workspace asserts to check active_tasks (still exposed, canonical busy signal) and keep the list-endpoint check that proves admin-auth'd callers still see current_task end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(e2e): stop asserting current_task on public workspace GET
promote: staging → main (security hardening + Phase 35.1)
Captures the 10-PR staging→main cutover: what shipped, the three new Railway prod env vars (PROVISION_SHARED_SECRET / EC2_VPC_ID / CP_BASE_URL), and the sharp edge for existing tenants — their containers pre-date PR #53 so they still need MOLECULE_CP_SHARED_SECRET added manually (or a re-provision) before the new CPProvisioner's outbound bearer works. Also includes a post-deploy verification checklist and rollback plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Paired with molecule-controlplane PR #55 (GET /cp/tenants/config). Lets existing tenants heal themselves when we rotate or add a CP-side env var (e.g. MOLECULE_CP_SHARED_SECRET landing earlier today) without any ssh or re-provision. Flow: main() calls refreshEnvFromCP() before any other os.Getenv read. The helper reads MOLECULE_ORG_ID + ADMIN_TOKEN from the baked-in user-data env, GETs {MOLECULE_CP_URL}/cp/tenants/config with those credentials, and applies the returned string map via os.Setenv so downstream code (CPProvisioner, etc.) sees the fresh values. Best-effort semantics: - self-hosted / no MOLECULE_ORG_ID → no-op (return nil) - CP unreachable / non-200 → log + return error (main keeps booting) - oversized values (>4 KiB each) rejected to avoid env pollution - body read capped at 64 KiB Once this image hits GHCR, the 5-minute tenant auto-updater picks it up, the container restarts, refresh runs, and every tenant has MOLECULE_CP_SHARED_SECRET within ~5 minutes — no operator toil. Also fixes workspace-server/.gitignore so `server` no longer matches the cmd/server package dir — it only ignored the compiled binary but pattern was too broad. Anchored to `/server`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(canvas): add 15s fetch timeout on API calls
docs: 2026-04-19 SaaS prod migration notes
feat(ws-server): pull env from CP on startup
Post-deploy verification for staging tenant images. Runs against the canary fleet after each publish-workspace-server-image build — catches auto-update breakage (a la today's E2E current_task drift) before it propagates to the prod tenant fleet that auto-pulls :latest every 5 min. scripts/canary-smoke.sh iterates a space-sep list of canary base URLs (paired with their ADMIN_TOKENs) and checks: - /admin/liveness reachable with admin bearer (tenant boot OK) - /workspaces list responds (wsAuth + DB path OK) - /memories/commit + /memories/search round-trip (encryption + scrubber) - /events admin read (AdminAuth C4 path) - /admin/liveness without bearer returns 401 (C4 fail-closed regression) .github/workflows/canary-verify.yml runs after publish succeeds: - 6-min sleep (tenant auto-updater pulls every 5 min) - bash scripts/canary-smoke.sh with secrets pulled from repo settings - on failure: writes a Step Summary flagging that :latest should be rolled back to prior known-good digest Phase 3 follow-up will split the publish workflow so only :staging-<sha> ships initially, and canary-verify's green gate is what promotes :staging-<sha> → :latest. This commit lays the test gate alone so we have something running against tenants immediately. Secrets to set in GitHub repo settings before this workflow can run: - CANARY_TENANT_URLS (space-sep list) - CANARY_ADMIN_TOKENS (same order as URLs) - CANARY_CP_SHARED_SECRET (matches staging CP PROVISION_SHARED_SECRET) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(canary): smoke harness + GHA verify workflow (Phase 2)
…e 3) Completes the canary release train. Before this, publish-workspace- server-image.yml pushed both :staging-<sha> and :latest on every main merge — meaning the prod tenant fleet auto-pulled every image immediately, before any post-deploy smoke test. A broken image (think: this morning's E2E current_task drift, but shipped at 3am instead of caught in CI) would have fanned out to every running tenant within 5 min. Now: - publish workflow pushes :staging-<sha> ONLY - canary tenants are configured to track :staging-<sha>; they pick up the new image on their next auto-update cycle - canary-verify.yml runs the smoke suite (Phase 2) after the sleep - on green: a new promote-to-latest job uses crane to remotely retag :staging-<sha> → :latest for both platform and tenant images - prod tenants auto-update to the newly-retagged :latest within their usual 5-min window - on red: :latest stays frozen on prior good digest; prod is untouched crane is pulled onto the runner (~4 MB, GitHub release) rather than docker-daemon retag so the workflow doesn't need a privileged runner. Rollback: if canary passed but something surfaces post-promotion, operator runs "crane tag ghcr.io/molecule-ai/platform:<prior-good-sha> latest" manually. A follow-up can wrap that in a Phase 4 admin endpoint / script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the canary loop with the escape hatch and a single place to
read about the whole flow.
scripts/rollback-latest.sh <sha>
uses crane to retag :latest ← :staging-<sha> for BOTH the platform
and tenant images. Pre-checks the target tag exists and verifies
the :latest digest after the move so a bad ops typo doesn't
silently promote the wrong thing. Prod tenants auto-update to the
rolled-back digest within their 5-min cycle. Exit codes: 0 = both
retagged, 1 = registry/tag error, 2 = usage error.
docs/architecture/canary-release.md
The one-page map of the pipeline: how PR → main → staging-<sha> →
canary smoke → :latest promotion works end-to-end, how to add a
canary tenant, how to roll back, and what this gate explicitly does
NOT catch (prod-only data, config drift, cross-tenant bugs).
No code changes in the CP or workspace-server — this PR is shell
+ docs only, so it's safe to land independently of the other Phase
{1,1.5,2,3} PRs still in review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(canary): gate :latest tag promotion on canary verify green (Phase 3)
Post-merge audit flagged cp_provisioner.go as the only new file from
the canary/C1 work without test coverage. Fills the gap:
- NewCPProvisioner_RequiresOrgID — self-hosted without MOLECULE_ORG_ID
refuses to construct (avoids silent phone-home to prod CP).
- NewCPProvisioner_FallsBackToProvisionSharedSecret — the operator
ergonomics of using one env-var name on both sides of the wire.
- AuthHeader noop + happy path — bearer only set when secret is set.
- Start_HappyPath — end-to-end POST to stubbed CP, bearer forwarded,
instance_id parsed out of response.
- Start_Non201ReturnsStructuredError — when CP returns structured
{"error":"…"}, that message surfaces to the caller.
- Start_NoStructuredErrorFallsBackToSize — regression gate for the
anti-log-leak change from PR #980: raw upstream body must NOT
appear in the error, only the byte count.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(canary): rollback script + release-pipeline doc (Phase 4)
The phantom-producer detector (#795) was doing UPDATE + SELECT in two roundtrips — first incrementing consecutive_empty_runs, then re- reading to check the stale threshold. Switch to UPDATE ... RETURNING so the post-increment value comes back in one query. Called once per schedule per cron tick. At 100 tenants × dozens of schedules per tenant, the halved DB traffic on the empty-response path is measurable, not just cosmetic. Also now properly logs if the bump itself fails (previously it silent- swallowed the ExecContext error and still ran the SELECT, which would confuse debugging). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test(ws-server): CPProvisioner coverage — auth, env fallback, error paths
perf(scheduler): collapse empty-run bump to single RETURNING query
CP's Callback handler redirects every new WorkOS session to APP_URL/orgs, but canvas had no such route — new users hit the canvas Home component, which tries to call /workspaces on a tenant that doesn't exist yet, and saw a confusing error. This PR plugs that gap with a dedicated landing page that: - Bounces anonymous visitors back to /cp/auth/login - Zero-org users see a slug-picker (POST /cp/orgs, refresh) - For each existing org, shows status + CTA: * awaiting_payment → amber "Complete payment" → /pricing?org=… * running → emerald "Open" → https://<slug>.moleculesai.app * failed → "Contact support" → mailto * provisioning → read-only "provisioning…" - Surfaces errors inline with a Retry button Deliberately server-light: one GET /cp/orgs, no WebSocket, no canvas store hydration. Goal is to move the user from signup to either Stripe Checkout or their tenant URL with one click each. Closes the last UX gap between the BILLING_REQUIRED gate landing on the CP and real users being able to complete a signup today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(canvas): /orgs landing page for post-signup users
…anner Two small polish items that together close the signup-to-running-tenant flow for real users: 1. Stripe success_url now points at /orgs?checkout=success instead of the current page (was pricing). The old behavior left people staring at plan cards with no indication payment went through — the new behavior drops them right onto their org list where they can watch the status flip. 2. /orgs shows a green "Payment confirmed, workspace spinning up" banner when it sees ?checkout=success, then clears the query param via replaceState so a reload doesn't show it again. 3. /orgs now polls every 5s while any org is awaiting_payment or provisioning. Users see the Stripe webhook's effect live — no manual refresh needed — and once every org settles the polling stops so idle tabs don't hammer /cp/orgs. Paired with PR #992 (the /orgs page itself) this makes the end-to-end flow on BILLING_REQUIRED=true deployments feel right: /pricing → Stripe → /orgs?checkout=success → banner → live poll → "Open" button when org.status transitions to running. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
promote: staging → main — canary infra + /orgs + env refresh + perf
…direct feat(canvas): post-checkout UX — Stripe success lands on /orgs with live banner
promote: staging → main — #994 post-checkout UX
…builds Publish has been failing since the 2026-04-18 open-source restructure (#964's merge) because workspace-server/Dockerfile still COPYs ./molecule-ai-plugin-github-app-auth/ but the restructure moved that code out to its own repo. Every main merge since has produced a "failed to compute cache key: /molecule-ai-plugin-github-app-auth: not found" error — prod images haven't moved. Fix: add an actions/checkout step that fetches the plugin repo into the build context before docker build runs. Private-repo safe: uses PLUGIN_REPO_PAT secret (fine-grained PAT with Contents:Read on Molecule-AI/molecule-ai-plugin-github-app-auth). Falls back to the default GITHUB_TOKEN if the plugin repo is public. Ops: set repo secret PLUGIN_REPO_PAT before the next main merge, or publish will fail with a 404 on the checkout step. Also gitignores the cloned dir so local dev builds don't accidentally commit it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ling fix(ci): clone sibling plugin repo so publish-workspace-server-image builds
promote: staging → main — unblock publish workflow (private-repo plugin clone)
Escape hatch for the initial rollout window (canary fleet not yet provisioned, so canary-verify.yml's automatic promotion doesn't fire) AND for manual rollback scenarios. Uses the default GITHUB_TOKEN which carries write:packages on repo- owned GHCR images, so no new secrets are needed. crane handles the remote retag without pulling or pushing layers. Validates the src tag exists before retagging + verifies the :latest digest post-retag so a typo can't silently promote the wrong image. Trigger from Actions → promote-latest → Run workflow → enter the short sha (e.g. "4c1d56e"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ci(promote-latest): workflow_dispatch retag :staging-<sha> → :latest
…ent postmortem (#1824) Bundles the documentation and lightweight tooling landed during the 2026-04-23 ops/triage session. Pure additions — no behavior changes. ## Added ### docs/architecture/backends.md Parity matrix for Docker vs EC2 (SaaS) workspace backends. 18 features tabulated with current status; 6 ranked drift risks; enforcement hooks (parity-lint + contract tests). Living document — owners are workspace-server + controlplane teams. ### docs/engineering/testing-strategy.md Tiered test-coverage floors instead of a blanket 100% target. Seven tiers by code class (auth/crypto → generated DTOs). Per-package current-state snapshot + targets. Tracks the 3 biggest coverage gaps (tokens.go 0%, workspace_provision.go 0%, wsauth ~48%) against their tier-1/2 floors. ### docs/engineering/pr-hygiene.md Captures the patterns that keep diffs reviewable. Motivated by the 2026-04-23 backlog audit where 8 of 23 open PRs had 70-380-file bloat from stale branch drift. Covers: small-PR sizing, rebase-not-merge, cherry-pick-onto-fresh-base for recovery, targeting staging first, describing why-not-what. ### docs/engineering/postmortem-2026-04-23-boot-event-401.md Postmortem for the /cp/tenants/boot-event 401 race. Root cause (DB INSERT ordered AFTER readiness check), detection path (E2E + manual log inspection), lessons (write-before-read pattern, integration tests needed, E2E alerting gap, invariants-as-comments). ### tools/check-template-parity.sh CI lint for template repos — diffs the `${VAR:+VAR=${VAR}}` provider- key forwarders between install.sh (bare-host / EC2 path) and start.sh (Docker path). Catches the #5 drift risk from backends.md before it ships. ### workspace-server/internal/provisioner/backend_contract_test.go Shared behavioral contract scaffold for Provisioner + CPProvisioner. Compile-time assertions catch method-signature drift today; scenario- level runs are t.Skip'd pending backend nil-hardening (drift risk #6, see backends.md). ## Updated ### README.md Links the new engineering docs + backends parity matrix into the Documentation Map so agents and humans can actually find them. ## Related issues - #1814 — unblock workspace_provision_test.go (broadcaster interface) - #1813 — nil-client panic hardening (drift risk #6) - #1815 — Canvas vitest coverage instrumentation - #1816 — tokens.go 0% → 85% - #1817 — 5 sqlmock column-drift failures - #1818 — Python pytest-cov setup - #1819 — wsauth middleware coverage gap - #1821 — tiered coverage policy (meta) - #1822 — backend parity drift tracker Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
fix(ci): run golangci-lint binary directly with || true
…#1867) * PMM: update ecosystem-watch — add LangGraph PR verification deferral note - Add 2026-04-22 entry: GH API 401 for external repos, LangGraph PRs #6645/#7113/#7205 still VERIFY. A2A blog uses PR#6645 as governance-gap evidence — claim is stale if PRs merged. - Update maintenance footer date to 2026-04-22 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * PMM: add Cloudflare Artifacts positioning brief Source: PR #641, merged 2026-04-17. Buyer: Platform engineers + enterprise security/compliance. Headline: 'Give your agents a Git history — without touching a terminal.' Objections covered: 'Why not GitHub?' + 'Cloudflare Artifacts is beta.' Blocking: Social Media Brand launch thread. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * PMM: update EC2 SSH launch brief — social copy APPROVED, TTS audio file added as blocker Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * PMM: update ecosystem-watch — verify LangGraph PRs still OPEN, log PRs #1702/#1730/#1731 Confirmed via gh CLI (GH_TOKEN restored): langchain-ai/langgraph PRs #6645, #7113, #7205 still OPEN as of 2026-04-23T17:38Z. A2A live-today positioning vs LangGraph in-progress remains accurate. Logged PR #1731 (sweepPhantomBusy), PR #1730 (45-min gh-token refresh daemon fixing 60-min 401 in long sessions), and PR #1702 (SSH-backed file writes for SaaS — P1 regression fix). Blog post for #1702 at docs/marketing/blog/2026-04-23-saas-file-api-fix.md. Co-Authored-By: Claude PMM <noreply@anthropic.com> * docs(marketing): add PR #1702 release note + PR #1686 positioning brief PR #1702 (SSH-backed file writes for SaaS): blog post covers fix, compute model detection, EIC-based remote write path. Ships same-day after merge. PR #1686 (Tool Trace + Platform Instructions): full positioning brief — buyer matrix, value props, competitive angle vs Langfuse/Helicone/OPA, objection handlers, cannibalization assessment (LOW). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(mmm): add Phase 34 positioning one-pager + messaging matrix - phase34-positioning.md: one-pager with positioning statement, audience matrix, problem/solution, competitive differentiators, and proof points for press kit use - phase34-messaging-matrix.md: 3 candidate taglines (production-grade, observability, aspirational) + full 4-feature messaging matrix (Partner API Keys, Tool Trace, Platform Instructions, SaaS Fed v2) - SaaS Federation v2 flagged as content gap — no PM brief exists; community copy blocked pending PM confirmation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Molecule AI PMM <pmm@agents.moleculesai.app> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(auth): break infinite redirect loop on /cp/auth/login
…rift-risk-3)
### Repro
On Canvas: create a workspace named "Hermes Agent" (runtime=langgraph,
model=langgraph default). Open the Config tab, switch the model to a
Minimax provider + Minimax token, hit Save and Restart. The model
reverts to the default on every restart.
### Root cause
`workspace_restart.go` called `findTemplateByName(configsDir, wsName)`
unconditionally when the request body had no explicit `template`:
template := body.Template
if template == "" {
template = findTemplateByName(h.configsDir, wsName)
}
`findTemplateByName` normalises the name ("Hermes Agent" → "hermes-agent")
and ALSO scans every template's `config.yaml` for a matching `name:`
field — a two-layer match that returns non-empty for any workspace whose
name coincides with a template dir OR any template whose config.yaml
claims the same display name.
When the match returned non-empty, the restart handler set
`templatePath = <template>` and the provisioner rewrote the workspace's
config volume from the template on `Start`. The Canvas Save+Restart
flow's `PUT /workspaces/:id/files/config.yaml` had already written the
user's edits to the volume — those got clobbered.
The comment immediately below (line 187) ALREADY said:
// Apply runtime-default template ONLY when explicitly requested
// via "apply_template": true. Use case: runtime was changed via
// Config tab — need new runtime's base files. Normal restarts
// preserve existing config volume (user's model, skills, prompts).
The code contradicted the comment. The design intent was right; the
implementation short-circuited it. Matches drift-risk #3 in #1822's
Docker-vs-EC2 parity tracker ("Config-tab save must flush to DB before
kicking off restart, not deferred").
### Fix
Extracted the template-resolution chain into a pure function
`resolveRestartTemplate(configsDir, wsName, dbRuntime, body)` in a new
`restart_template.go`. Gated the name-based auto-match on
`body.ApplyTemplate`:
1. Explicit `body.Template` → always honoured (caller consent).
2. `ApplyTemplate=true` → name-based auto-match (prior behaviour).
3. `RebuildConfig=true` → org-templates recovery fallback (#239).
4. `ApplyTemplate=true` + dbRuntime → `<runtime>-default/`.
5. Fall through → empty path + "existing-volume" label. Provisioner
reuses the volume. This is the path Canvas Save+Restart now hits.
The handler now calls this helper and uses the returned path directly.
Duplicate rebuild_config blocks at lines 167-186 were consolidated into
the helper's single tier-3 case in passing.
### Abstraction win
`resolveRestartTemplate` is a pure function — no gin context, no DB, no
network. Takes a struct input, returns two strings. The whole priority
chain is unit-testable in a temp dir, which is exactly what
`restart_template_test.go` does.
### Tests
`restart_template_test.go` — 8 table-style unit tests covering every
branch of the priority chain:
- DefaultRestart_PreservesVolume — the regression. Even when a
template's config.yaml `name:` field matches the workspace name
exactly (worst case), a default restart MUST return empty path.
- ExplicitTemplate_AlwaysHonoured — caller-by-name, any mode.
- ApplyTemplate_NameMatch — opt-in restores the auto-match.
- ApplyTemplate_RuntimeDefault — runtime-change flow still works.
- ApplyTemplate_NoMatch_NoRuntime — fallback to existing-volume.
- InvalidExplicitTemplate_ProceedsWithout — traversal attempt stays
inside root, falls through cleanly.
- NonExistentExplicitTemplate — deleted/missing template falls through.
- Priority_ExplicitBeatsApplyTemplate — explicit Template wins over
name-match when both fire.
Full handlers race suite (`go test -race ./internal/handlers/`) still
passes — existing Restart-handler tests unchanged.
### Blast radius
Any restart caller that omitted `apply_template: true` and relied on
name-matching auto-applying a template is now a behaviour change.
Identified call sites in this repo:
- Canvas Save+Restart button (store/canvas.ts) — explicitly the
flow this commit fixes, definitely wanted the fix.
- Canvas Restart button (same file) — same semantics; user expects
a restart, not a template reset.
- Auto-restart sweeper (#1858) — never passes apply_template and
depends on the existing volume having valid config. Separately,
`workspace_provision.go`'s #1858 recovery path detects empty
volumes and auto-applies `<runtime>-default` without going
through findTemplateByName, so recovery is unaffected.
- RestartByID — internal callers; audited, all intended "restart
as-is", none relied on auto-template-match.
No SaaS parity impact — this is a handler behaviour fix that applies
equally to Docker and EC2 backends (both use the same Restart handler
before dispatching to their respective provisioners).
Refs #1822 drift-risk-3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Problem
When a lead delegates to a worker that's mid-synthesis, the proxy returns
503 "workspace agent busy" and the caller records the delegation as
failed. On fan-out storms from leads this hits ~70% drop rate — today's
observed numbers in the cycle reports.
## Fix — Phase 1 TASK-level queue-on-busy
When `handleA2ADispatchError` determines the target is busy, instead of
returning 503, enqueue the request as priority=TASK and return 202
Accepted with `{queued: true, queue_id, queue_depth}`. The workspace's
next heartbeat (≤30s) drains one item if it reports spare capacity.
Files:
- migrations/042_a2a_queue.{up,down}.sql — `a2a_queue` table with
partial indexes on status='queued' + idempotency_key. Schema
supports PriorityCritical/Task/Info from day one so Phase 2/3 ship
without migration churn.
- internal/handlers/a2a_queue.go — EnqueueA2A / DequeueNext /
Mark*-helpers plus WorkspaceHandler.DrainQueueForWorkspace. Uses
`SELECT ... FOR UPDATE SKIP LOCKED` so concurrent drains can't
double-claim the same row. Max 5 attempts before marking 'failed'
so a stuck item doesn't wedge the queue forever.
- internal/handlers/a2a_proxy_helpers.go — isUpstreamBusyError branch
calls EnqueueA2A and returns 202 on success. Falls through to the
legacy 503 on enqueue error (DB hiccup shouldn't silently drop).
- internal/handlers/registry.go — RegistryHandler gets a QueueDrainFunc
injection hook (SetQueueDrainFunc). When Heartbeat sees
active_tasks < max_concurrent_tasks, spawns a goroutine that calls
the drain hook. context.WithoutCancel ensures the drain outlives
the heartbeat handler's ctx.
- internal/router/router.go — wires wh.DrainQueueForWorkspace into
rh.SetQueueDrainFunc after both are constructed.
## Not in this PR (Phase 2/3/4 follow-ups)
- INFO priority + TTL (Phase 2)
- CRITICAL priority + soft preemption between tool calls (Phase 3)
- Age-based promotion so TASK doesn't starve (Phase 4)
- `GET /workspaces/:id/queue` observability endpoint
Schema already supports all of these; only the dispatch + policy code
remains.
## Tests
- TestExtractIdempotencyKey (5 cases): messageId parsing is robust
- TestPriorityConstants: ordering invariant + 50=TASK default
alignment with migration DEFAULT
Full DB-touching tests (FIFO order, retry bound, idempotency conflict)
intentionally deferred to the CI migration-enabled path — sqlmock
ceremony would duplicate the existing test infrastructure 3× over and
the behaviour is directly expressible in SQL constraints (FOR UPDATE
SKIP LOCKED, partial unique index).
## Expected impact once deployed
- a2a_receive error with "busy" flavor drops from ~69/10min observed
today to ~0
- delegation_failed rate drops from ~50% to <5%
- real_output metric rises from ~30/15min back toward the pre-
throttle baseline
Closes #1870 Phase 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(a2a): queue-on-busy — Phase 1 of priority queue (#1870)
…name) #1892's EnqueueA2A INSERT used `ON CONFLICT ON CONSTRAINT idx_a2a_queue_idempotency DO NOTHING`, but Postgres rejects this: ERROR: constraint "idx_a2a_queue_idempotency" for table "a2a_queue" does not exist Partial unique INDEXES cannot be referenced by name in ON CONFLICT — that form is reserved for true CONSTRAINTs created via CREATE TABLE ... CONSTRAINT or ALTER TABLE ADD CONSTRAINT. Partial indexes need the column-list + WHERE form so the planner can match the index. Effect of the bug: every EnqueueA2A errored, the busy-error fallback returned 503 instead of 202, queue stayed empty. Cycle 50 observed 46 busy errors / 0 queue rows — the deployed Phase 1 had no effect. Fix: switch to ON CONFLICT (workspace_id, idempotency_key) WHERE idempotency_key IS NOT NULL AND status IN ('queued','dispatched') DO NOTHING Verified manually against the live `a2a_queue` table on staging — INSERT returns the new id; cleanup deleted the test row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…x-1870 fix(a2a-queue): use partial-index ON CONFLICT syntax (not constraint name)
…config fix(restart): preserve user config volume on default restart (#1822 drift-risk-3)
… misleading config.yaml error for hermes Canvas Config tab had 3 bugs visible on hermes workspaces (#1894): 1. Runtime dropdown showed "LangGraph (default)" even when the workspace's actual runtime was hermes — because the form only loaded runtime from config.yaml, and hermes doesn't use the platform's config.yaml template. 2. Model field was empty for the same reason. 3. "No config.yaml found" error appeared on hermes workspaces despite everything being fine — hermes manages its own config at ~/.hermes/config.yaml on the workspace host. Worse, clicking Save with the empty form would silently flip `runtime` back from `hermes` to `LangGraph (default)`. ## Fix - loadConfig now always fetches workspace metadata (runtime + model) via GET /workspaces/:id and GET /workspaces/:id/model BEFORE attempting the config.yaml fetch. These act as the source of truth for runtime and model when config.yaml doesn't set them. - RUNTIMES_WITH_OWN_CONFIG set lists runtimes that manage their own config outside the platform template (hermes, external). For these: - Missing config.yaml is NOT an error — no red banner shown. - An informational gray banner tells the user where to edit the runtime's config (e.g. "edit ~/.hermes/config.yaml via Terminal tab or the hermes CLI" for hermes). Closes #1894. Verified 2026-04-23 on user's hongmingwang tenant which runs hermes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…l-hermes fix(canvas/config): load runtime+model from workspace metadata + hide misleading config.yaml error for hermes
…2026-04-23-final # Conflicts: # docs/ecosystem-watch.md # docs/marketing/battlecard/phase-34-partner-api-keys-battlecard.md # docs/marketing/launches/pr-1533-ec2-instance-connect-ssh.md
HongmingWang-Rabbit
enabled auto-merge
April 23, 2026 22:33
![molecule-ai[bot]](https://avatars.githubusercontent.com/u/275794187?s=60&v=4){kind=small}
This was referenced Apr 23, 2026
This was referenced Apr 23, 2026
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 24, 2026
…followup chore: sync staging → main (follow-up: 9 commits since #1913)
This was referenced Apr 24, 2026
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…rors' (#1913) from fix/more-ignored-db-errors-batch-2 into main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1912 (which had 3 add/add merge conflicts from parallel doc additions).
Brings main up to staging — today's bug wave fixes + regression tests + 1250+ commits of prior staging work.
Key today's content
Conflicts resolved (3 files)
All 3 were add/add conflicts on docs added in parallel on both branches:
All 3 resolved by taking the staging version (newer iteration per session timeline).
Ready to merge — CEO pre-approved per 2026-04-23 session