Skip to content

chore(ci): migrate all jobs to self-hosted macOS arm64 runner - #186

Merged
HongmingWang-Rabbit merged 2 commits into
mainfrom
chore/ci-self-hosted-macos
Apr 15, 2026
Merged

chore(ci): migrate all jobs to self-hosted macOS arm64 runner#186
HongmingWang-Rabbit merged 2 commits into
mainfrom
chore/ci-self-hosted-macos

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

Migrates every CI job in this repo from GitHub-hosted ubuntu-latest to a self-hosted Apple-silicon runner ([self-hosted, macos, arm64], registered at the Molecule-AI org level) to sidestep GitHub-hosted minute rate limits.

What changed

.github/workflows/ci.yml — all 7 jobs migrated:

Job Adaptation
platform-build runs-on swap only (setup-go has native darwin-arm64)
canvas-build runs-on swap only
mcp-server-build runs-on swap only
e2e-api Non-trivial: replaced services: postgres/redis with inline docker run startup steps (GHA services: is Linux-only on self-hosted runners). Ports remapped to 15432/16379 to avoid collision with whatever the host exposes on the standard ports. Containers named molecule-ci-postgres / molecule-ci-redis, torn down in if: always(). Postgres readiness still gated on pg_isready via docker exec.
shellcheck Non-trivial: ludeeus/action-shellcheck is a Docker action (Linux-only). Replaced with a native shellcheck --severity=warning invocation against find tests/e2e -name '*.sh'.
canvas-deploy-reminder runs-on swap only (uses gh api, pre-installed on runner)
python-lint runs-on swap only

.github/workflows/publish-platform-image.yml — platform image publisher:

  • Added docker/setup-qemu-action@v3 with platforms: linux/amd64.
  • Added platforms: linux/amd64 to both docker/build-push-action@v5 calls. The self-hosted runner is arm64 but Fly tenant machines pull linux/amd64, so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to behavior unchanged.
  • ⚠️ Expect slower builds — amd64 emulation under QEMU on an 8GB M1 is measurably slower than a native ubuntu-latest amd64 build. Monitor the first run closely.

Runner host prereqs (already set up on hongming-m1-mini)

  • Docker Desktop installed + running (needed by e2e-api and publish-platform-image)
  • docker symlinked into /opt/homebrew/bin
  • shellcheck v0.11.0 installed into /opt/homebrew/bin
  • gh, Go, Node, Python all on PATH; setup-* actions handle version pinning per job

Risks & things to watch on the first run

  1. Docker Desktop must be running — if it crashes/quits, e2e-api and publish-platform-image fail. No graceful fallback.
  2. QEMU cross-arch image builds are slow + RAM-heavy on the 8GB M1. If publish-platform-image OOMs, the mitigation is to split Dockerfile builds into multi-stage with a native Go cross-compile (no emulation needed for the Go build stage) — can be a follow-up PR.
  3. Self-hosted runner has persistent filesystem state. Cached node_modules, Go build cache, etc. will grow across runs. Add a workspace-cleanup step or monitor _work/ disk usage.
  4. Concurrency — this one runner processes jobs serially. Multiple PRs will queue.
  5. Port collisions — CI postgres/redis use 15432/16379. If something else on the host takes those later, e2e-api will fail loudly.
  6. services: removal means migrations check uses the known container name ($PG_CONTAINER env) instead of docker ps --filter ancestor=postgres:16, which is actually more reliable.

Test plan

  • Watch the first run on this PR end-to-end — every job must turn green
  • Confirm e2e-api postgres/redis containers are cleaned up after both success and failure runs
  • Confirm publish-platform-image pushes a working linux/amd64 image to GHCR + Fly registry (manual workflow_dispatch on main after merge is a safer way to validate than waiting for the next platform-touching merge)
  • Watch runner disk usage after a few runs — add cleanup step if _work/ grows unbounded

HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 15, 2026
#167)

PR #167 gated /events and /bundles/export/:id behind AdminAuth. The e2e
script's 3 calls to these routes were unauthenticated and broke when the
runner picked them up for the first time on PR #186 (self-hosted runner
migration). Same admin-gate contract, same fix pattern as the #99/#110
e2e hotfixes.

POST /bundles/import is left unauthenticated because by that point in
the script both workspaces have been deleted and #110 revoked their
tokens, so HasAnyLiveTokenGlobal=0 and AdminAuth fails-open.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.

Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):

- e2e-api: `services: postgres/redis` replaced with inline `docker run`
  steps. Ports remapped to 15432/16379 to avoid collision with anything
  the host may already expose on the standard ports. Containers are named
  (`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
  `if: always()` step. Postgres readiness is still gated on pg_isready
  via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
  Replaced with a direct `shellcheck` invocation (pre-installed on the
  runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
  explicit `platforms: linux/amd64` on both `docker/build-push-action`
  invocations. The runner is arm64 but Fly tenant machines pull amd64,
  so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
  behavior is unchanged.

Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job
@HongmingWang-Rabbit
HongmingWang-Rabbit force-pushed the chore/ci-self-hosted-macos branch from f66233d to 434fc9a Compare April 15, 2026 17:40
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 15, 2026
…runner

PR #186 put python-lint on the Mac mini self-hosted runner. setup-python@v5
hardcodes /Users/runner/hostedtoolcache which doesn't exist (runner user is
hongming-claw), causing:
  ##[error]mkdir: /Users/runner: Permission denied

The runner's ~/.env RUNNER_TOOL_CACHE override didn't propagate into the job
env (unclear why — possibly service restart didn't re-read .env). Setting
AGENT_TOOLSDIRECTORY + RUNNER_TOOL_CACHE at the job level via env: is the
belt-and-suspenders fix that works regardless of runner config state.

setup-python@v5 explicitly honours AGENT_TOOLSDIRECTORY per its README.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.

Fixes the only failing job in CI run 24469156329 on PR #186.
@HongmingWang-Rabbit
HongmingWang-Rabbit marked this pull request as ready for review April 15, 2026 17:48
@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit aa41947 into main Apr 15, 2026
6 of 7 checks passed
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 15, 2026
…ash-merge)

#186's squash-merge commit (aa41947) took 15e15a2 (AGENT_TOOLSDIRECTORY
override) but missed a6cfc5f (bypass setup-python entirely) which was
pushed to the PR branch after the merge was initiated. The merge
commit still has the old setup-python@v5 job config.

Applies a6cfc5f's ci.yml verbatim via git checkout. Restores the
Homebrew-python3.11 bypass path that the user prototyped. No other
changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit added a commit that referenced this pull request Apr 15, 2026
…ython

fix(ci): apply bypass-setup-python to main (missed in #186 squash)
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 15, 2026
…ht sweep

Captures ~27 PRs merged across both repos this session: security
hardening cluster (#94/#99/#106/#110/#119/#162/#155/#167/#185/#200/#203/
#209/#233), data-integrity fixes (#212/#224/#236), CI runner migration
(#186), platform/scheduler reliability (#95/#149/#207/#206), workspace
runtime features (#205/#208/#198/#216/#225/#235/#231), code-review
follow-ups (#228/#232).

Updated counts: 816 Go (+70), 1180 Python (+40), 453 vitest (unchanged
— UI/a11y patches), 97 jest (unchanged).

CLAUDE.md additions:
- Idle Loop section (#205) under Architectural Patterns
- Admin auth middleware variants section linking docs/runbooks/admin-auth.md
- Migration runner section explaining the .down.sql filter (#212)
- Per-route auth notes in the API table (PATCH field-whitelist, CanvasOrBearer
  on PUT /canvas/viewport, AdminAuth on bundles/events/templates-import/
  approvals-pending/admin-liveness)
- Database section updated with workspace_auth_tokens auto-revoke (#110),
  scheduler.error_detail surfacing (#206), workspace_schedules.last_status
  'skipped' state (#207)

PLAN.md additions:
- New Recently launched (overnight sweep) section with full PR/issue index
- Phase status updated (B–G now complete, H partial)
- Live infrastructure deltas (migration fix, token rotation, legal pages)
- Outstanding items consolidated

Edit-history file expanded from the tick-9 stub to a full session record
covering malware cleanup, CI runner migration, security cluster, data
integrity, infra/feature/code-review batches, and outstanding user
actions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
#167)

PR #167 gated /events and /bundles/export/:id behind AdminAuth. The e2e
script's 3 calls to these routes were unauthenticated and broke when the
runner picked them up for the first time on PR #186 (self-hosted runner
migration). Same admin-gate contract, same fix pattern as the #99/#110
e2e hotfixes.

POST /bundles/import is left unauthenticated because by that point in
the script both workspaces have been deleted and #110 revoked their
tokens, so HasAnyLiveTokenGlobal=0 and AdminAuth fails-open.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
* chore(ci): migrate all jobs to self-hosted macOS arm64 runner

Switches every job in `ci.yml` and `publish-platform-image.yml` from
`ubuntu-latest` to `[self-hosted, macos, arm64]` to avoid GitHub-hosted
minute rate limits. All jobs run on a single Apple-silicon self-hosted
runner registered at the Molecule-AI org level.

Notable non-trivial adaptations (macOS runners can't use `services:` and
some GHA marketplace actions are Linux-only):

- e2e-api: `services: postgres/redis` replaced with inline `docker run`
  steps. Ports remapped to 15432/16379 to avoid collision with anything
  the host may already expose on the standard ports. Containers are named
  (`molecule-ci-postgres` / `molecule-ci-redis`) and torn down in an
  `if: always()` step. Postgres readiness is still gated on pg_isready
  via `docker exec`.
- shellcheck: `ludeeus/action-shellcheck` is a Docker action, Linux-only.
  Replaced with a direct `shellcheck` invocation (pre-installed on the
  runner) that scans `tests/e2e/*.sh` with `--severity=warning`.
- publish-platform-image: added `docker/setup-qemu-action@v3` and an
  explicit `platforms: linux/amd64` on both `docker/build-push-action`
  invocations. The runner is arm64 but Fly tenant machines pull amd64,
  so QEMU-emulated cross-arch builds are required. GHA cache-from/cache-to
  behavior is unchanged.

Runner prereqs (one-time host setup):
- Docker Desktop installed and running (for e2e-api + image publish)
- `shellcheck` on PATH
- `docker` on PATH
- Go / Node / gh / Python are installed via setup-* actions per job

* fix(ci): set AGENT_TOOLSDIRECTORY for python-lint on self-hosted runner

setup-python@v5 defaults to /Users/runner/hostedtoolcache which doesn't
exist on the hongming-claw self-hosted runner. AGENT_TOOLSDIRECTORY tells
the action to use a writable path under the runner user's home directory.

Fixes the only failing job in CI run 24469156329 on PR #186.

---------

Co-authored-by: Hongming Wang <HongmingWang-Rabbit@users.noreply.github.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
…ash-merge)

#186's squash-merge commit (3ff40c4) took 15e15a2 (AGENT_TOOLSDIRECTORY
override) but missed a6cfc5f (bypass setup-python entirely) which was
pushed to the PR branch after the merge was initiated. The merge
commit still has the old setup-python@v5 job config.

Applies a6cfc5f's ci.yml verbatim via git checkout. Restores the
Homebrew-python3.11 bypass path that the user prototyped. No other
changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
…ython

fix(ci): apply bypass-setup-python to main (missed in #186 squash)
molecule-ai Bot pushed a commit that referenced this pull request Apr 21, 2026
…ht sweep

Captures ~27 PRs merged across both repos this session: security
hardening cluster (#94/#99/#106/#110/#119/#162/#155/#167/#185/#200/#203/
#209/#233), data-integrity fixes (#212/#224/#236), CI runner migration
(#186), platform/scheduler reliability (#95/#149/#207/#206), workspace
runtime features (#205/#208/#198/#216/#225/#235/#231), code-review
follow-ups (#228/#232).

Updated counts: 816 Go (+70), 1180 Python (+40), 453 vitest (unchanged
— UI/a11y patches), 97 jest (unchanged).

CLAUDE.md additions:
- Idle Loop section (#205) under Architectural Patterns
- Admin auth middleware variants section linking docs/runbooks/admin-auth.md
- Migration runner section explaining the .down.sql filter (#212)
- Per-route auth notes in the API table (PATCH field-whitelist, CanvasOrBearer
  on PUT /canvas/viewport, AdminAuth on bundles/events/templates-import/
  approvals-pending/admin-liveness)
- Database section updated with workspace_auth_tokens auto-revoke (#110),
  scheduler.error_detail surfacing (#206), workspace_schedules.last_status
  'skipped' state (#207)

PLAN.md additions:
- New Recently launched (overnight sweep) section with full PR/issue index
- Phase status updated (B–G now complete, H partial)
- Live infrastructure deltas (migration fix, token rotation, legal pages)
- Outstanding items consolidated

Edit-history file expanded from the tick-9 stub to a full session record
covering malware cleanup, CI runner migration, security cluster, data
integrity, infra/feature/code-review batches, and outstanding user
actions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant