Skip to content

fix(ci): move changes job off self-hosted runner + add workflow concurrency - #1194

Merged
molecule-ai[bot] merged 1 commit into
stagingfrom
fix/ci-runner-contention
Apr 21, 2026
Merged

molecule-ai[bot] merged 1 commit into
stagingfrom
fix/ci-runner-contention

Conversation

@molecule-ai

@molecule-ai molecule-ai Bot commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Relieves macOS arm64 self-hosted runner contention with two targeted changes:

1. Move changes job → ubuntu-latest

The changes job does a plain git diff — it has zero macOS-specific dependencies. Moving it off the [self-hosted, macos, arm64] runner frees the slot immediately on every workflow trigger. Previously it held the runner during fetch-depth: 0 checkout before any actual build work started.

2. Add workflow-level concurrency to ci.yml

concurrency:
  group: ci-\${{ github.ref }}
  cancel-in-progress: true

Without this, every new push to a PR or main queues a full new workflow run, each competing for the same single runner. With cancel-in-progress: true, stale in-flight CI runs are cancelled when a newer commit arrives — the runner always runs the latest state instead of a backlog of old ones.

Context

The self-hosted macOS arm64 runner is shared by ci.yml, e2e-api.yml (15-min timeout, cancel-in-progress: false), canary-verify.yml, codeql.yml, and the publish-*.yml workflows. The combination of (1) the changes job holding the runner during fetch-depth: 0 on every trigger, and (2) no workflow-level cancellation, caused 100+ queued runs with 0 in-progress during the Apr 20 saturation event.

Test plan

  • YAML syntax validated
  • Logic: changes job on ubuntu-latest does git diff — platform/canvas/python/scripts output booleans unchanged
  • Downstream jobs (platform-build, canvas-build, python-lint, shellcheck) all correctly needs: changes and use [self-hosted, macos, arm64] as before

…rrency

Two changes to relieve macOS arm64 runner contention:

1. `changes` job: runs on `ubuntu-latest` instead of
   `[self-hosted, macos, arm64]`. This job does a plain `git diff`
   — it has zero macOS dependencies. Moving it off the runner frees
   the slot immediately on every workflow trigger.

2. Add workflow-level concurrency to `ci.yml`:
   `concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true`

   Without this, every new push to a PR or main queues a full new
   workflow run, each competing for the same single runner. With
   `cancel-in-progress: true`, stale in-flight CI runs are cancelled
   when a newer commit arrives — the runner always runs the latest
   state, not a backlog of old ones.

Context: the self-hosted macOS arm64 runner is shared by ci.yml,
e2e-api.yml, canary-verify.yml, and publish-*.yml. The combination of
(1) the `changes` job holding the runner during `fetch-depth: 0`
checkout on every trigger, and (2) no workflow-level cancellation
caused 100+ queued runs with 0 in-progress.

Follow-up candidates (need verification before changing):
- platform-build: Go build may work on ubuntu-latest (no macOS deps)
- canvas-build: Next.js build may work on ubuntu-latest
- python-lint: needs `setup-python` instead of Homebrew Python

@molecule-ai molecule-ai Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: #1194 — fix(ci): move changes job off self-hosted runner + add workflow concurrency

Approve ✅ — clean fix for the runner saturation problem.

What this PR does well

  • Correct root cause: The changes job holding the macOS runner during fetch-depth: 0 before any real build work was the dominant contributor to the Apr 20 saturation event. Freeing it immediately is the right move.
  • Concurrency group by github.ref: Correct scope — PRs cancel their own stale runs, pushes to main don't interfere with staging runs.
  • cancel-in-progress: true: Correct behavior.

Minor suggestions (non-blocking)

  1. canvas-deploy-reminder still runs on [self-hosted, macos, arm64]: Consider runs-on: ubuntu-latest — the gh api call works fine there and would avoid runner slot usage entirely.

  2. fetch-depth: 1 for the new ubuntu-latest changes job: Only the single base commit is needed, not full history. Faster checkout.

  3. Consider adding run-name: CI to the workflow — when cancel-in-progress kills a run, the new run inherits the old run name in the Actions UI, which is confusing.

These are nits. Core changes are correct and well-documented.

@molecule-ai molecule-ai Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA Review — PR #1194 (CI: move changes job to ubuntu-latest + workflow concurrency)

LGTM — recommend merge once CI passes

Two targeted changes:

  1. changes job moved from [self-hosted, macos, arm64]ubuntu-latest — no macOS dependencies, frees the runner immediately.
  2. concurrency block — cancels in-progress runs on same ref, prevents stale run queuing.

Standard pattern, correct implementation. +11/-3, low risk.

Note: cannot approve as this appears to be from the same bot account as the reviewer token.

@molecule-ai
molecule-ai Bot merged commit 6f48e85 into staging Apr 21, 2026
1 of 11 checks passed
@molecule-ai
molecule-ai Bot deleted the fix/ci-runner-contention branch April 21, 2026 01:44
molecule-ai Bot added a commit that referenced this pull request Apr 21, 2026
…#1236)

* fix(security): call redactSecrets before seeding workspace memories (F1085)

seedInitialMemories() in workspace_provision.go was inserting template/config
memories directly into agent_memories without scrubbing credential patterns.
A workspace provisioned from a template containing API keys, tokens, or other
secrets would store them in plain text — the same class of issue as #838.

Fix: call redactSecrets(workspaceID, content) on the truncated memory content
before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400)
is preserved — redaction runs after truncation so the size limit still applies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(workspace_provision): add seedInitialMemories coverage for #1208

Cover the truncate-at-100k boundary (PR #1167, CWE-400) and the
redactSecrets call (F1085 / #1132), both identified as untested in #1208.

- TestSeedInitialMemories_TruncatesOversizedContent: boundary at exactly
  100k, 1 byte over, far over, and well under. Verifies INSERT receives
  exactly maxMemoryContentLength bytes.
- TestSeedInitialMemories_RedactsSecrets: verifies redactSecrets runs
  before INSERT, regression test for F1085.
- TestSeedInitialMemories_InvalidScopeSkipped: invalid scope is silently
  skipped, no INSERT called.
- TestSeedInitialMemories_EmptyMemoriesNil: nil slice is handled without
  DB calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(marketing): Discord adapter launch visual assets (#1209)

Squash-merge: Discord adapter launch visual assets (3 PNGs) + social copy. Acceptance: assets on staging.

* fix(ci): golangci-lint errcheck failures on staging

Suppress errcheck warnings for calls where the return value is safely
ignored:
  - resp.Body.Close() (artifacts/client.go): deferred cleanup — failure
    to close a response body is non-critical; the defer itself is what
    matters for connection reuse.
  - rows.Close() (bundle/exporter.go): deferred cleanup in a loop where
    rows.Err() already handles query errors.
  - filepath.Walk (bundle/exporter.go): top-level walk call; errors in
    sub-directory traversal are handled by the inner callback (which
    returns nil for err != nil).
  - broadcaster.RecordAndBroadcast (bundle/importer.go): fire-and-forget
    event broadcast; errors are logged internally by the broadcaster.
  - db.DB.ExecContext (bundle/importer.go): best-effort runtime column
    update; non-critical auxiliary data that the provisioner re-extracts
    if needed.

Fixes: #1143

* test(artifacts): suppress w.Write return values to satisfy errcheck

All httptest.ResponseWriter.Write calls in client_test.go now discard
the byte count and error return with _, _ = prefix. The Write method
is safe to discard in test handlers — httptest.ResponseWriter.Write
never returns an error for in-memory buffers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(CI): move changes job off self-hosted runner + add workflow concurrency

Cherry-pick from staging PR #1194 for main. Two changes to relieve
macOS arm64 runner saturation:

1. `changes` job: runs on ubuntu-latest instead of
   [self-hosted, macos, arm64]. This job does a plain `git diff`
   with zero macOS dependencies — moving it off the runner frees
   a slot immediately on every workflow trigger.

2. Add workflow-level concurrency:
   concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true

   Prevents multiple stale in-flight CI runs from queuing on the
   same ref when new commits arrive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(security): call redactSecrets before seeding workspace memories (F1085) (#1203)

seedInitialMemories() in workspace_provision.go was inserting template/config
memories directly into agent_memories without scrubbing credential patterns.
A workspace provisioned from a template containing API keys, tokens, or other
secrets would store them in plain text — the same class of issue as #838.

Fix: call redactSecrets(workspaceID, content) on the truncated memory content
before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400)
is preserved — redaction runs after truncation so the size limit still applies.

Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* tick: 2026-04-21 ~03:40Z — CI stalled 59+ min, GH_TOKEN 4th rotation, PR reviews done

* fix(tenant-guard): allowlist /registry/register + /registry/heartbeat

Final layer of today's stuck-provisioning saga. With the private-IP
platform_url fix and the intra-VPC :8080 SG rule in place, workspace
EC2s finally reached the tenant on the right port — only to have every
POST bounced with a synthetic 404 by TenantGuard.

TenantGuard is the SaaS hook that rejects cross-tenant routing. It
demands X-Molecule-Org-Id on every request, but CP's workspace user-
data doesn't export MOLECULE_ORG_ID (only WORKSPACE_ID, PLATFORM_URL,
RUNTIME, PORT), so the runtime can't attach the header. Net effect:
every workspace's first heartbeat to /registry/heartbeat was a silent
404, and the workspace sat in 'provisioning' until the platform
sweeper timed it out.

Allowlist the two workspace-boot paths:
  - /registry/register  — one-shot at runtime startup
  - /registry/heartbeat — every 30s

Both are still gated by wsauth.HasAnyLiveToken (workspaces with a
token on file must present it; legacy tokenless workspaces are
grandfathered). And the tenant SG already scopes :8080 to the VPC
CIDR, so only intra-VPC callers can reach these paths in the first
place. The allowlist bypasses cross-org routing, not auth.

Follow-up: passing MOLECULE_ORG_ID into the workspace env would let
the runtime attach the header and drop this allowlist entry. Tracked
separately; not urgent since the multi-layer auth above is already
adequate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Molecule AI Infra-SRE <infra-sre@agents.moleculesai.app>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-UIUX <core-uiux@agents.moleculesai.app>
Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
molecule-ai Bot added a commit that referenced this pull request Apr 21, 2026
…#1236)

* fix(security): call redactSecrets before seeding workspace memories (F1085)

seedInitialMemories() in workspace_provision.go was inserting template/config
memories directly into agent_memories without scrubbing credential patterns.
A workspace provisioned from a template containing API keys, tokens, or other
secrets would store them in plain text — the same class of issue as #838.

Fix: call redactSecrets(workspaceID, content) on the truncated memory content
before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400)
is preserved — redaction runs after truncation so the size limit still applies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(workspace_provision): add seedInitialMemories coverage for #1208

Cover the truncate-at-100k boundary (PR #1167, CWE-400) and the
redactSecrets call (F1085 / #1132), both identified as untested in #1208.

- TestSeedInitialMemories_TruncatesOversizedContent: boundary at exactly
  100k, 1 byte over, far over, and well under. Verifies INSERT receives
  exactly maxMemoryContentLength bytes.
- TestSeedInitialMemories_RedactsSecrets: verifies redactSecrets runs
  before INSERT, regression test for F1085.
- TestSeedInitialMemories_InvalidScopeSkipped: invalid scope is silently
  skipped, no INSERT called.
- TestSeedInitialMemories_EmptyMemoriesNil: nil slice is handled without
  DB calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(marketing): Discord adapter launch visual assets (#1209)

Squash-merge: Discord adapter launch visual assets (3 PNGs) + social copy. Acceptance: assets on staging.

* fix(ci): golangci-lint errcheck failures on staging

Suppress errcheck warnings for calls where the return value is safely
ignored:
  - resp.Body.Close() (artifacts/client.go): deferred cleanup — failure
    to close a response body is non-critical; the defer itself is what
    matters for connection reuse.
  - rows.Close() (bundle/exporter.go): deferred cleanup in a loop where
    rows.Err() already handles query errors.
  - filepath.Walk (bundle/exporter.go): top-level walk call; errors in
    sub-directory traversal are handled by the inner callback (which
    returns nil for err != nil).
  - broadcaster.RecordAndBroadcast (bundle/importer.go): fire-and-forget
    event broadcast; errors are logged internally by the broadcaster.
  - db.DB.ExecContext (bundle/importer.go): best-effort runtime column
    update; non-critical auxiliary data that the provisioner re-extracts
    if needed.

Fixes: #1143

* test(artifacts): suppress w.Write return values to satisfy errcheck

All httptest.ResponseWriter.Write calls in client_test.go now discard
the byte count and error return with _, _ = prefix. The Write method
is safe to discard in test handlers — httptest.ResponseWriter.Write
never returns an error for in-memory buffers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(CI): move changes job off self-hosted runner + add workflow concurrency

Cherry-pick from staging PR #1194 for main. Two changes to relieve
macOS arm64 runner saturation:

1. `changes` job: runs on ubuntu-latest instead of
   [self-hosted, macos, arm64]. This job does a plain `git diff`
   with zero macOS dependencies — moving it off the runner frees
   a slot immediately on every workflow trigger.

2. Add workflow-level concurrency:
   concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true

   Prevents multiple stale in-flight CI runs from queuing on the
   same ref when new commits arrive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(security): call redactSecrets before seeding workspace memories (F1085) (#1203)

seedInitialMemories() in workspace_provision.go was inserting template/config
memories directly into agent_memories without scrubbing credential patterns.
A workspace provisioned from a template containing API keys, tokens, or other
secrets would store them in plain text — the same class of issue as #838.

Fix: call redactSecrets(workspaceID, content) on the truncated memory content
before the INSERT. The truncation (maxMemoryContentLength = 100 KiB, CWE-400)
is preserved — redaction runs after truncation so the size limit still applies.

Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* tick: 2026-04-21 ~03:40Z — CI stalled 59+ min, GH_TOKEN 4th rotation, PR reviews done

* fix(tenant-guard): allowlist /registry/register + /registry/heartbeat

Final layer of today's stuck-provisioning saga. With the private-IP
platform_url fix and the intra-VPC :8080 SG rule in place, workspace
EC2s finally reached the tenant on the right port — only to have every
POST bounced with a synthetic 404 by TenantGuard.

TenantGuard is the SaaS hook that rejects cross-tenant routing. It
demands X-Molecule-Org-Id on every request, but CP's workspace user-
data doesn't export MOLECULE_ORG_ID (only WORKSPACE_ID, PLATFORM_URL,
RUNTIME, PORT), so the runtime can't attach the header. Net effect:
every workspace's first heartbeat to /registry/heartbeat was a silent
404, and the workspace sat in 'provisioning' until the platform
sweeper timed it out.

Allowlist the two workspace-boot paths:
  - /registry/register  — one-shot at runtime startup
  - /registry/heartbeat — every 30s

Both are still gated by wsauth.HasAnyLiveToken (workspaces with a
token on file must present it; legacy tokenless workspaces are
grandfathered). And the tenant SG already scopes :8080 to the VPC
CIDR, so only intra-VPC callers can reach these paths in the first
place. The allowlist bypasses cross-org routing, not auth.

Follow-up: passing MOLECULE_ORG_ID into the workspace env would let
the runtime attach the header and drop this allowlist entry. Tracked
separately; not urgent since the multi-layer auth above is already
adequate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Molecule AI Infra-SRE <infra-sre@agents.moleculesai.app>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
Co-authored-by: Molecule AI Core-DevOps <core-devops@agents.moleculesai.app>
Co-authored-by: Molecule AI Core-UIUX <core-uiux@agents.moleculesai.app>
Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
…rrency

Cherry-pick from staging PR #1194 for main. Two changes to relieve
macOS arm64 runner saturation:

1. `changes` job: runs on ubuntu-latest instead of
   [self-hosted, macos, arm64]. This job does a plain `git diff`
   with zero macOS dependencies — moving it off the runner frees
   a slot immediately on every workflow trigger.

2. Add workflow-level concurrency:
   concurrency: group: ci-${{ github.ref }}; cancel-in-progress: true

   Prevents multiple stale in-flight CI runs from queuing on the
   same ref when new commits arrive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants