fix(ci): move changes job off self-hosted runner + add workflow concurrency - #1194
Conversation
…rrency
Two changes to relieve macOS arm64 runner contention:
1. `changes` job: runs on `ubuntu-latest` instead of
`[self-hosted, macos, arm64]`. This job does a plain `git diff`
— it has zero macOS dependencies. Moving it off the runner frees
the slot immediately on every workflow trigger.
2. Add workflow-level concurrency to `ci.yml`:
`concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true`
Without this, every new push to a PR or main queues a full new
workflow run, each competing for the same single runner. With
`cancel-in-progress: true`, stale in-flight CI runs are cancelled
when a newer commit arrives — the runner always runs the latest
state, not a backlog of old ones.
Context: the self-hosted macOS arm64 runner is shared by ci.yml,
e2e-api.yml, canary-verify.yml, and publish-*.yml. The combination of
(1) the `changes` job holding the runner during `fetch-depth: 0`
checkout on every trigger, and (2) no workflow-level cancellation
caused 100+ queued runs with 0 in-progress.
Follow-up candidates (need verification before changing):
- platform-build: Go build may work on ubuntu-latest (no macOS deps)
- canvas-build: Next.js build may work on ubuntu-latest
- python-lint: needs `setup-python` instead of Homebrew Python
There was a problem hiding this comment.
PR Review: #1194 — fix(ci): move changes job off self-hosted runner + add workflow concurrency
Approve ✅ — clean fix for the runner saturation problem.
What this PR does well
- Correct root cause: The
changesjob holding the macOS runner duringfetch-depth: 0before any real build work was the dominant contributor to the Apr 20 saturation event. Freeing it immediately is the right move. - Concurrency group by
github.ref: Correct scope — PRs cancel their own stale runs, pushes to main don't interfere with staging runs. cancel-in-progress: true: Correct behavior.
Minor suggestions (non-blocking)
-
canvas-deploy-reminderstill runs on[self-hosted, macos, arm64]: Considerruns-on: ubuntu-latest— thegh apicall works fine there and would avoid runner slot usage entirely. -
fetch-depth: 1for the new ubuntu-latestchangesjob: Only the single base commit is needed, not full history. Faster checkout. -
Consider adding
run-name: CIto the workflow — whencancel-in-progresskills a run, the new run inherits the old run name in the Actions UI, which is confusing.
These are nits. Core changes are correct and well-documented.
There was a problem hiding this comment.
QA Review — PR #1194 (CI: move changes job to ubuntu-latest + workflow concurrency)
✅ LGTM — recommend merge once CI passes
Two targeted changes:
changesjob moved from[self-hosted, macos, arm64]→ubuntu-latest— no macOS dependencies, frees the runner immediately.concurrencyblock — cancels in-progress runs on same ref, prevents stale run queuing.
Standard pattern, correct implementation. +11/-3, low risk.
Note: cannot approve as this appears to be from the same bot account as the reviewer token.
…#1236) * fix(security): call redactSecrets before seeding workspace memories (F1085) seedInitialMemories() in workspace_provision.go was inserting template/config memories directly into agent_memories without scrubbing credential patterns. A workspace provisioned from a template containing API keys, tokens, or other secrets would store them in plain text — the same class of issue as #838. Fix: call redactSecrets(workspaceID, content) on the truncated memory content before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400) is preserved — redaction runs after truncation so the size limit still applies. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(workspace_provision): add seedInitialMemories coverage for #1208 Cover the truncate-at-100k boundary (PR #1167, CWE-400) and the redactSecrets call (F1085 / #1132), both identified as untested in #1208. - TestSeedInitialMemories_TruncatesOversizedContent: boundary at exactly 100k, 1 byte over, far over, and well under. Verifies INSERT receives exactly maxMemoryContentLength bytes. - TestSeedInitialMemories_RedactsSecrets: verifies redactSecrets runs before INSERT, regression test for F1085. - TestSeedInitialMemories_InvalidScopeSkipped: invalid scope is silently skipped, no INSERT called. - TestSeedInitialMemories_EmptyMemoriesNil: nil slice is handled without DB calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(marketing): Discord adapter launch visual assets (#1209) Squash-merge: Discord adapter launch visual assets (3 PNGs) + social copy. Acceptance: assets on staging. * fix(ci): golangci-lint errcheck failures on staging Suppress errcheck warnings for calls where the return value is safely ignored: - resp.Body.Close() (artifacts/client.go): deferred cleanup — failure to close a response body is non-critical; the defer itself is what matters for connection reuse. - rows.Close() (bundle/exporter.go): deferred cleanup in a loop where rows.Err() already handles query errors. - filepath.Walk (bundle/exporter.go): top-level walk call; errors in sub-directory traversal are handled by the inner callback (which returns nil for err != nil). - broadcaster.RecordAndBroadcast (bundle/importer.go): fire-and-forget event broadcast; errors are logged internally by the broadcaster. - db.DB.ExecContext (bundle/importer.go): best-effort runtime column update; non-critical auxiliary data that the provisioner re-extracts if needed. Fixes: #1143 * test(artifacts): suppress w.Write return values to satisfy errcheck All httptest.ResponseWriter.Write calls in client_test.go now discard the byte count and error return with _, _ = prefix. The Write method is safe to discard in test handlers — httptest.ResponseWriter.Write never returns an error for in-memory buffers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(CI): move changes job off self-hosted runner + add workflow concurrency Cherry-pick from staging PR #1194 for main. Two changes to relieve macOS arm64 runner saturation: 1. `changes` job: runs on ubuntu-latest instead of [self-hosted, macos, arm64]. This job does a plain `git diff` with zero macOS dependencies — moving it off the runner frees a slot immediately on every workflow trigger. 2. Add workflow-level concurrency: concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true Prevents multiple stale in-flight CI runs from queuing on the same ref when new commits arrive. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(security): call redactSecrets before seeding workspace memories (F1085) (#1203) seedInitialMemories() in workspace_provision.go was inserting template/config memories directly into agent_memories without scrubbing credential patterns. A workspace provisioned from a template containing API keys, tokens, or other secrets would store them in plain text — the same class of issue as #838. Fix: call redactSecrets(workspaceID, content) on the truncated memory content before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400) is preserved — redaction runs after truncation so the size limit still applies. Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * tick: 2026-04-21 ~03:40Z — CI stalled 59+ min, GH_TOKEN 4th rotation, PR reviews done * fix(tenant-guard): allowlist /registry/register + /registry/heartbeat Final layer of today's stuck-provisioning saga. With the private-IP platform_url fix and the intra-VPC :8080 SG rule in place, workspace EC2s finally reached the tenant on the right port — only to have every POST bounced with a synthetic 404 by TenantGuard. TenantGuard is the SaaS hook that rejects cross-tenant routing. It demands X-Molecule-Org-Id on every request, but CP's workspace user- data doesn't export MOLECULE_ORG_ID (only WORKSPACE_ID, PLATFORM_URL, RUNTIME, PORT), so the runtime can't attach the header. Net effect: every workspace's first heartbeat to /registry/heartbeat was a silent 404, and the workspace sat in 'provisioning' until the platform sweeper timed it out. Allowlist the two workspace-boot paths: - /registry/register — one-shot at runtime startup - /registry/heartbeat — every 30s Both are still gated by wsauth.HasAnyLiveToken (workspaces with a token on file must present it; legacy tokenless workspaces are grandfathered). And the tenant SG already scopes :8080 to the VPC CIDR, so only intra-VPC callers can reach these paths in the first place. The allowlist bypasses cross-org routing, not auth. Follow-up: passing MOLECULE_ORG_ID into the workspace env would let the runtime attach the header and drop this allowlist entry. Tracked separately; not urgent since the multi-layer auth above is already adequate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Molecule AI Infra-SRE <infra-sre@agents.moleculesai.app> Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com> Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app> Co-authored-by: Molecule AI Core-UIUX <core-uiux@agents.moleculesai.app> Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
…#1236) * fix(security): call redactSecrets before seeding workspace memories (F1085) seedInitialMemories() in workspace_provision.go was inserting template/config memories directly into agent_memories without scrubbing credential patterns. A workspace provisioned from a template containing API keys, tokens, or other secrets would store them in plain text — the same class of issue as #838. Fix: call redactSecrets(workspaceID, content) on the truncated memory content before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400) is preserved — redaction runs after truncation so the size limit still applies. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(workspace_provision): add seedInitialMemories coverage for #1208 Cover the truncate-at-100k boundary (PR #1167, CWE-400) and the redactSecrets call (F1085 / #1132), both identified as untested in #1208. - TestSeedInitialMemories_TruncatesOversizedContent: boundary at exactly 100k, 1 byte over, far over, and well under. Verifies INSERT receives exactly maxMemoryContentLength bytes. - TestSeedInitialMemories_RedactsSecrets: verifies redactSecrets runs before INSERT, regression test for F1085. - TestSeedInitialMemories_InvalidScopeSkipped: invalid scope is silently skipped, no INSERT called. - TestSeedInitialMemories_EmptyMemoriesNil: nil slice is handled without DB calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(marketing): Discord adapter launch visual assets (#1209) Squash-merge: Discord adapter launch visual assets (3 PNGs) + social copy. Acceptance: assets on staging. * fix(ci): golangci-lint errcheck failures on staging Suppress errcheck warnings for calls where the return value is safely ignored: - resp.Body.Close() (artifacts/client.go): deferred cleanup — failure to close a response body is non-critical; the defer itself is what matters for connection reuse. - rows.Close() (bundle/exporter.go): deferred cleanup in a loop where rows.Err() already handles query errors. - filepath.Walk (bundle/exporter.go): top-level walk call; errors in sub-directory traversal are handled by the inner callback (which returns nil for err != nil). - broadcaster.RecordAndBroadcast (bundle/importer.go): fire-and-forget event broadcast; errors are logged internally by the broadcaster. - db.DB.ExecContext (bundle/importer.go): best-effort runtime column update; non-critical auxiliary data that the provisioner re-extracts if needed. Fixes: #1143 * test(artifacts): suppress w.Write return values to satisfy errcheck All httptest.ResponseWriter.Write calls in client_test.go now discard the byte count and error return with _, _ = prefix. The Write method is safe to discard in test handlers — httptest.ResponseWriter.Write never returns an error for in-memory buffers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(CI): move changes job off self-hosted runner + add workflow concurrency Cherry-pick from staging PR #1194 for main. Two changes to relieve macOS arm64 runner saturation: 1. `changes` job: runs on ubuntu-latest instead of [self-hosted, macos, arm64]. This job does a plain `git diff` with zero macOS dependencies — moving it off the runner frees a slot immediately on every workflow trigger. 2. Add workflow-level concurrency: concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true Prevents multiple stale in-flight CI runs from queuing on the same ref when new commits arrive. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(security): call redactSecrets before seeding workspace memories (F1085) (#1203) seedInitialMemories() in workspace_provision.go was inserting template/config memories directly into agent_memories without scrubbing credential patterns. A workspace provisioned from a template containing API keys, tokens, or other secrets would store them in plain text — the same class of issue as #838. Fix: call redactSecrets(workspaceID, content) on the truncated memory content before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400) is preserved — redaction runs after truncation so the size limit still applies. Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * tick: 2026-04-21 ~03:40Z — CI stalled 59+ min, GH_TOKEN 4th rotation, PR reviews done * fix(tenant-guard): allowlist /registry/register + /registry/heartbeat Final layer of today's stuck-provisioning saga. With the private-IP platform_url fix and the intra-VPC :8080 SG rule in place, workspace EC2s finally reached the tenant on the right port — only to have every POST bounced with a synthetic 404 by TenantGuard. TenantGuard is the SaaS hook that rejects cross-tenant routing. It demands X-Molecule-Org-Id on every request, but CP's workspace user- data doesn't export MOLECULE_ORG_ID (only WORKSPACE_ID, PLATFORM_URL, RUNTIME, PORT), so the runtime can't attach the header. Net effect: every workspace's first heartbeat to /registry/heartbeat was a silent 404, and the workspace sat in 'provisioning' until the platform sweeper timed it out. Allowlist the two workspace-boot paths: - /registry/register — one-shot at runtime startup - /registry/heartbeat — every 30s Both are still gated by wsauth.HasAnyLiveToken (workspaces with a token on file must present it; legacy tokenless workspaces are grandfathered). And the tenant SG already scopes :8080 to the VPC CIDR, so only intra-VPC callers can reach these paths in the first place. The allowlist bypasses cross-org routing, not auth. Follow-up: passing MOLECULE_ORG_ID into the workspace env would let the runtime attach the header and drop this allowlist entry. Tracked separately; not urgent since the multi-layer auth above is already adequate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Molecule AI Infra-SRE <infra-sre@agents.moleculesai.app> Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com> Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app> Co-authored-by: Molecule AI Core-UIUX <core-uiux@agents.moleculesai.app> Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
…rrency Cherry-pick from staging PR #1194 for main. Two changes to relieve macOS arm64 runner saturation: 1. `changes` job: runs on ubuntu-latest instead of [self-hosted, macos, arm64]. This job does a plain `git diff` with zero macOS dependencies — moving it off the runner frees a slot immediately on every workflow trigger. 2. Add workflow-level concurrency: concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true Prevents multiple stale in-flight CI runs from queuing on the same ref when new commits arrive. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
Relieves macOS arm64 self-hosted runner contention with two targeted changes:
1. Move
changesjob →ubuntu-latestThe
changesjob does a plaingit diff— it has zero macOS-specific dependencies. Moving it off the[self-hosted, macos, arm64]runner frees the slot immediately on every workflow trigger. Previously it held the runner duringfetch-depth: 0checkout before any actual build work started.2. Add workflow-level concurrency to
ci.ymlWithout this, every new push to a PR or main queues a full new workflow run, each competing for the same single runner. With
cancel-in-progress: true, stale in-flight CI runs are cancelled when a newer commit arrives — the runner always runs the latest state instead of a backlog of old ones.Context
The self-hosted macOS arm64 runner is shared by
ci.yml,e2e-api.yml(15-min timeout,cancel-in-progress: false),canary-verify.yml,codeql.yml, and thepublish-*.ymlworkflows. The combination of (1) thechangesjob holding the runner duringfetch-depth: 0on every trigger, and (2) no workflow-level cancellation, caused 100+ queued runs with 0 in-progress during the Apr 20 saturation event.Test plan
changesjob on ubuntu-latest doesgit diff— platform/canvas/python/scripts output booleans unchangedplatform-build,canvas-build,python-lint,shellcheck) all correctlyneeds: changesand use[self-hosted, macos, arm64]as before