ci: route cmux-tui artifact jobs through live fleet by teamleaderleo · Pull Request #30622 · manaflow-ai/cmux

teamleaderleo · 2026-11-01T09:12:36Z

The cmux-tui artifact tree publisher could queue its Linux control and daemon jobs on a stale Blacksmith 4-vCPU label while the live fleet was unavailable. That leaves the tree manifest and daemon marker missing, so nightly-next cannot consume an otherwise built source.

This change carries the approved live GCP runner-group reader and adds a Linux-only placement step to tree preflight. It emits cmux-fleet-linux-x64 only when the authenticated runner inventory proves an online member; otherwise the daemon and other heavy work use measured Blacksmith overflow. The reusable package workflow receives that live output, and stale Blacksmith values cannot pin the selector. The picker reuses the workflow's existing routing App token under its existing builder-gated condition, with its existing administration/read and organization-self-hosted-runners/read permissions; no new App, secret, or permission is introduced. The active owner run 41289457009 is left untouched.

Validation:

python3 -m unittest tests.test_ci_portable_job_placement tests.test_cmux_tui_artifacts_linux_runner_set tests.test_ci_pr_runner_pool.OwnedPools (86 passed)
YAML parsing, Python compilation and git diff --check pass
The broader failover suites still report only their known base reds in cmux-next-chromium-currency.yml and platform-dev-secrets.yml

Agent session: agent session

github-actions · 2026-11-01T09:12:53Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 23c34a4: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30622 -f ref=lane/tree-availability-repair -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4141626810501 (a7f48bb820)
cmux NEXT nightly 1.0.0-nightly.4141702888901 (666ead5781)
cmux NEXT nightly 1.0.0-nightly.4141861940901 (ca2867968a)
cmux NEXT nightly 1.0.0-nightly.4141950554301 (badcc41cad)
cmux NEXT nightly 1.0.0-nightly.4142181714801 (2a518bac7f)

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/41415363947 at c3cb7c4 failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at b38a6b8. This pull request is one of 2 merged since: #30622, #32009.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

teamleaderleo · 2026-11-01T09:15:35Z

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 53cbf4c

BLOCK at 53cbf4c
Visual verdict: no images; Leo tries the build

[P1] [blocking] .github/workflows/cmux-tui-artifacts.yml:1375: The new selector breaks test_cmux_tui_daemon_tests_gate.py and test_ci_tui_routes_avoid_nightly_label.py. Their shared evaluator drops .linux, returning '{}'. The new default also conflicts with the 8-vCPU default still declared in runners.json. Both checks pass against the pre-change workflow and run in ci-guards.yml; update their evaluator and reconcile the declared fallback ordering.
[P1] [blocking] .github/workflows/cmux-tui-build-package.yml:164: Adding the pool branch here and in LINUX_BUILD_RUNNER introduces four failures in test_ci_linux_compile_runners.py. Its fallback and fork-routing assertions still require the previous expressions. This suite passes against the pre-change workflow and runs in ci-guards.yml. Update these guards to validate the new pool precedence while retaining fallback and fork checks.

cubic-dev-ai

2 issues found across 3 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/cmux-tui-build-package.yml">

<violation number="1" location=".github/workflows/cmux-tui-build-package.yml:165">
P1: This selector still honors a `blacksmith-*` `inputs.linux_runner`, so the binary matrix bypasses the new live-pool routing and can remain pinned to the stale 4/8-vCPU runner. Apply the same non-Blacksmith guard used by `LINUX_RUNNER` before accepting this input.</violation>

<violation number="2" location=".github/workflows/cmux-tui-build-package.yml:165">
P2: The new pool fallback changes this selector’s exact contract, but `test_ci_linux_compile_runners.py` still expects the pre-change expression and will fail when the CI compile-runner suite runs. Update that test’s expected prefix with the new fallback and input guard.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-01T09:16:21Z

-          LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
-          LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}
+          LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
+          LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}


P1: This selector still honors a blacksmith-* inputs.linux_runner, so the binary matrix bypasses the new live-pool routing and can remain pinned to the stale 4/8-vCPU runner. Apply the same non-Blacksmith guard used by LINUX_RUNNER before accepting this input.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/cmux-tui-build-package.yml, line 165: <comment>This selector still honors a `blacksmith-*` `inputs.linux_runner`, so the binary matrix bypasses the new live-pool routing and can remain pinned to the stale 4/8-vCPU runner. Apply the same non-Blacksmith guard used by `LINUX_RUNNER` before accepting this input.</comment> <file context> @@ -161,10 +161,10 @@ jobs: - LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }} - LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }} + LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }} + LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }} MACOS_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || (inputs.macos_runner != '' && inputs.macos_runner || vars.CI_PAID_MACOS_OVERFLOW == '1' && vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') }} - LINUX_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404') }} </file context>

Suggested change

LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}

LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && !startsWith(inputs.linux_runner, 'blacksmith-') && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}

cubic-dev-ai · 2026-11-01T09:16:21Z

-          LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
-          LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}
+          LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }}
+          LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }}


P2: The new pool fallback changes this selector’s exact contract, but test_ci_linux_compile_runners.py still expects the pre-change expression and will fail when the CI compile-runner suite runs. Update that test’s expected prefix with the new fallback and input guard.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/cmux-tui-build-package.yml, line 165: <comment>The new pool fallback changes this selector’s exact contract, but `test_ci_linux_compile_runners.py` still expects the pre-change expression and will fail when the CI compile-runner suite runs. Update that test’s expected prefix with the new fallback and input guard.</comment> <file context> @@ -161,10 +161,10 @@ jobs: - LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }} - LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }} + LINUX_DARWIN_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.LINUX_RUNNER_DARWIN_CROSS || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404' }} + LINUX_BUILD_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_BUILD_RUNNER || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-16vcpu-ubuntu-2404') }} MACOS_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || (inputs.macos_runner != '' && inputs.macos_runner || vars.CI_PAID_MACOS_OVERFLOW == '1' && vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') }} - LINUX_RUNNER: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || (inputs.linux_runner != '' && inputs.linux_runner || vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404') }} </file context>

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4ffe4a6a3e7fada96257 (cherry picked from commit 0829eb0)

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

cursor · 2026-11-01T09:30:31Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

cubic-dev-ai

3 issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/ci/portable_job_placement.py">

<violation number="1" location="scripts/ci/portable_job_placement.py:159">
P2: The live check counts an online but busy fleet runner as capacity. The daemon job then queues on that label with no Blacksmith fallback, which can delay artifact publication. Consider counting only idle runners for this gate, or document that busy capacity is accepted.</violation>
</file>

<file name="scripts/ci/pr_runner_pool.py">

<violation number="1" location="scripts/ci/pr_runner_pool.py:2332">
P2: A transient read failure on the optional `cmux-fleet-linux` group now aborts the whole `runners()` listing. Callers that need only glaeda-minis or owned Mac capacity then lose live data, and nightly and signing picks drop to Blacksmith. For an optional Linux group, skip the group on HTTP errors instead of raising, and keep raising only for `RUNNER_GROUP`.</violation>
</file>

<file name=".github/workflows/cmux-tui-artifacts.yml">

<violation number="1" location=".github/workflows/cmux-tui-artifacts.yml:1401">
P1: The daemon fallback sizes are reversed: normal runs resolve to 4 vCPU and overflow runs to 8 vCPU, contrary to the measured size/failover pair. Swap these two Blacksmith labels so the normal route uses 8 vCPU and `CI_CLOUD_OVERFLOW_SAVED` uses 4 vCPU.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-01T09:37:46Z

    # ci-routing-agent is the one the other glaeda-route readers use.
    environment: ci-routing-agent
-    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-4vcpu-ubuntu-2404' || 'blacksmith-8vcpu-ubuntu-2404' }}
+    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-4vcpu-ubuntu-2404' }}


P1: The daemon fallback sizes are reversed: normal runs resolve to 4 vCPU and overflow runs to 8 vCPU, contrary to the measured size/failover pair. Swap these two Blacksmith labels so the normal route uses 8 vCPU and CI_CLOUD_OVERFLOW_SAVED uses 4 vCPU.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/cmux-tui-artifacts.yml, line 1401: <comment>The daemon fallback sizes are reversed: normal runs resolve to 4 vCPU and overflow runs to 8 vCPU, contrary to the measured size/failover pair. Swap these two Blacksmith labels so the normal route uses 8 vCPU and `CI_CLOUD_OVERFLOW_SAVED` uses 4 vCPU.</comment> <file context> @@ -1372,7 +1398,7 @@ jobs: # ci-routing-agent is the one the other glaeda-route readers use. environment: ci-routing-agent - runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || fromJSON(vars.RUNNER_POOLS || '{}').linux || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-4vcpu-ubuntu-2404' }} + runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-4vcpu-ubuntu-2404' }} timeout-minutes: 60 env: </file context>

Suggested change

runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' || 'blacksmith-4vcpu-ubuntu-2404' }}

runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner || vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-4vcpu-ubuntu-2404' || 'blacksmith-8vcpu-ubuntu-2404' }}

cubic-dev-ai · 2026-11-01T09:37:46Z

+        # A configured label is only a candidate. Emit it for this run when a
+        # live runner listing proves at least one online member; otherwise the
+        # caller's measured Blacksmith branch remains the bounded fallback.
+        if not runners or pool.live_online(runners, [linux]).get(linux, 0) == 0:


P2: The live check counts an online but busy fleet runner as capacity. The daemon job then queues on that label with no Blacksmith fallback, which can delay artifact publication. Consider counting only idle runners for this gate, or document that busy capacity is accepted.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/portable_job_placement.py, line 159: <comment>The live check counts an online but busy fleet runner as capacity. The daemon job then queues on that label with no Blacksmith fallback, which can delay artifact publication. Consider counting only idle runners for this gate, or document that busy capacity is accepted.</comment> <file context> @@ -144,10 +151,20 @@ def decide(env: Mapping[str, str], runners: Sequence[Mapping[str, Any]] | None, + # A configured label is only a candidate. Emit it for this run when a + # live runner listing proves at least one online member; otherwise the + # caller's measured Blacksmith branch remains the bounded fallback. + if not runners or pool.live_online(runners, [linux]).get(linux, 0) == 0: + live_linux_missing = True + linux = "" </file context>

cubic-dev-ai · 2026-11-01T09:37:46Z

+                    continue
+                if not isinstance(group, Mapping) or not isinstance(group.get("id"), int):
+                    raise RuntimeError(f"no runner group {group_name} is visible to {self.repo}")
+                org.extend(self._runner_pages(f"/orgs/{owner}/actions/runner-groups/{group.get('id')}/runners"))


P2: A transient read failure on the optional cmux-fleet-linux group now aborts the whole runners() listing. Callers that need only glaeda-minis or owned Mac capacity then lose live data, and nightly and signing picks drop to Blacksmith. For an optional Linux group, skip the group on HTTP errors instead of raising, and keep raising only for RUNNER_GROUP.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/pr_runner_pool.py, line 2332: <comment>A transient read failure on the optional `cmux-fleet-linux` group now aborts the whole `runners()` listing. Callers that need only glaeda-minis or owned Mac capacity then lose live data, and nightly and signing picks drop to Blacksmith. For an optional Linux group, skip the group on HTTP errors instead of raising, and keep raising only for `RUNNER_GROUP`.</comment> <file context> @@ -2293,28 +2296,42 @@ class GitHub: + continue + if not isinstance(group, Mapping) or not isinstance(group.get("id"), int): + raise RuntimeError(f"no runner group {group_name} is visible to {self.repo}") + org.extend(self._runner_pages(f"/orgs/{owner}/actions/runner-groups/{group.get('id')}/runners")) except urllib.error.HTTPError as error: - raise RuntimeError(f"org runner group {RUNNER_GROUP} unreadable (HTTP {error.code}); the routing " </file context>

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

teamleaderleo · 2026-11-01T09:48:44Z

Review (codex gpt-6-astra high, router-review): BLOCK at 55d49ba

BLOCK at 55d49ba
Visual verdict: no images; Leo tries the build

[P1] [blocking] .github/workflows/cmux-tui-artifacts.yml:1401 and .github/workflows/cmux-tui-build-package.yml:164: This breaks existing CI guards. The daemon gate test fails because the default changes from 8 to 4 vCPU; two compile-routing tests also fail on the added pool expressions. All three tests pass against reconstructed base workflows and run in ci-guards.yml. Update the routing contracts and their tests together.
[P2] [follow-up] .github/workflows/cmux-tui-build-package.yml:164: The new RUNNER_POOLS.linux fallback bypasses the Blacksmith filter and outage switch. With empty live output, RUNNER_POOLS.linux=blacksmith-4vcpu-ubuntu-2404, and overflow active, the selector chooses 8 vCPU but compilation, verification, and packaging still select 4 vCPU. Apply the same filtering consistently so stale values cannot strand downstream jobs.
[P2] [follow-up] .github/workflows/cmux-tui-artifacts.yml:323: Linux placement depends on a token minted only when darwin_builder == 'linux'. With the supported mac setting, the fallback read token lacks runner-administration permissions, and github.token cannot read organization runners. Consequently, live Linux placement always falls back despite available fleet capacity. Mint the routing token independently of the Darwin builder.

Follow-ups:

Filter stale Blacksmith pool values consistently.
Decouple Linux routing-token creation from the Darwin builder setting.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

cubic-dev-ai

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/cmux-tui-artifacts.yml">

<violation number="1" location=".github/workflows/cmux-tui-artifacts.yml:323">
P2: The daemon picker now depends on `route-token`, but that token is skipped when `darwin_builder` is `mac`; the remaining tokens lack organization self-hosted-runner read permission, so live fleet routing silently falls back to Blacksmith for every mac-builder run. Mint a runner-readable token for the Linux picker independently, or remove the builder condition from the shared token step.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-01T09:55:08Z

+          GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }}
+          GITHUB_TRIGGERING_ACTOR: ${{ github.triggering_actor }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+          ROUTE_TOKEN: ${{ steps.route-token.outputs.token || steps.read-token.outputs.token }}


P2: The daemon picker now depends on route-token, but that token is skipped when darwin_builder is mac; the remaining tokens lack organization self-hosted-runner read permission, so live fleet routing silently falls back to Blacksmith for every mac-builder run. Mint a runner-readable token for the Linux picker independently, or remove the builder condition from the shared token step.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/cmux-tui-artifacts.yml, line 323: <comment>The daemon picker now depends on `route-token`, but that token is skipped when `darwin_builder` is `mac`; the remaining tokens lack organization self-hosted-runner read permission, so live fleet routing silently falls back to Blacksmith for every mac-builder run. Mint a runner-readable token for the Linux picker independently, or remove the builder condition from the shared token step.</comment> <file context> @@ -332,7 +320,7 @@ jobs: GITHUB_TRIGGERING_ACTOR: ${{ github.triggering_actor }} GITHUB_REPOSITORY: ${{ github.repository }} - ROUTE_TOKEN: ${{ steps.linux-route-token.outputs.token || steps.read-token.outputs.token }} + ROUTE_TOKEN: ${{ steps.route-token.outputs.token || steps.read-token.outputs.token }} ROUTE_TOKEN_FALLBACK: ${{ github.token }} SNAPSHOT_TOKEN: ${{ steps.read-token.outputs.token || github.token }} </file context>

teamleaderleo · 2026-11-01T09:58:50Z

Review (codex gpt-6-astra high, router-review): BLOCK at abf86a4

BLOCK at abf86a4

Visual verdict: no images; Leo tries the build

[P1][blocking] .github/workflows/cmux-tui-build-package.yml:164-165 adds fromJSON(vars.RUNNER_POOLS || '{}').linux before the Blacksmith fallback, but the required ci-guards test tests/test_ci_linux_compile_runners.py still rejects that selector. Running the test produces four failures for these two expressions (outage-switch and fork-routing assertions), so the CI guard fails on this PR. Update the guard/tests or preserve the enforced selector structure.

Follow-ups: none

blacksmith-sh · 2026-11-01T10:11:28Z

Found 37 test failures on Blacksmith runners:

Failures

Test	View Logs
`1 runner guard check(s) failed:/1 runner guard check(s) failed:`	View Logs
`1 runner guard check(s) failed:/1 runner guard check(s) failed:`	View Logs
`test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='big-red')/` `test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='big-red')`	View Logs
`test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='macos-26')/` `test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='macos-26')`	View Logs
`test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='ubuntu-24.04')/` `test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='ubuntu-24.04')`	View Logs
`test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='ubuntu-latest')/` `test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails (main.Workf` `lows.test_a_mutated_workflow_with_a_hosted_label_or_a_personal_machine_fails) (workflow` `='cmux-tui-artifacts.yml', label='ubuntu-latest')`	View Logs
`test_big_red (main.FleetRuleLints.test_big_red)/` `test_big_red (main.FleetRuleLints.test_big_red)`	View Logs
`test_daemon_tests_never_take_a_mac:/test_daemon_tests_never_take_a_mac:`	View Logs
`test_each_compile_job_follows_the_blacksmith_outage_switch (main.LinuxCompileRunner` `s.test_each_compile_job_follows_the_blacksmith_outage_switch) (workflow='cmux-tui-build` `-package.yml', job='plan-build', where='env:targets:LINUX_BUILD_RUNNER')/` `test_each_compile_job_follows_the_blacksmith_outage_switch (main.LinuxCompileRunner` `s.test_each_compile_job_follows_the_blacksmith_outage_switch) (workflow='cmux-tui-build` `-package.yml', job='plan-build', where='env:targets:LINUX_BUILD_RUNNER')`	View Logs
`test_each_compile_job_follows_the_blacksmith_outage_switch (main.LinuxCompileRunner` `s.test_each_compile_job_follows_the_blacksmith_outage_switch) (workflow='cmux-tui-build` `-package.yml', job='plan-build', where='env:targets:LINUX_DARWIN_RUNNER')/` `test_each_compile_job_follows_the_blacksmith_outage_switch (main.LinuxCompileRunner` `s.test_each_compile_job_follows_the_blacksmith_outage_switch) (workflow='cmux-tui-build` `-package.yml', job='plan-build', where='env:targets:LINUX_DARWIN_RUNNER')`	View Logs

...and 27 more test failures. View all on Blacksmith: run 41294832454, run 41294832453

^{Need help on this PR? Tag @codesmith-bot with what you need.}

teamleaderleo · 2026-11-01T11:49:56Z

For awareness: #30810 (merged as 76f23d1) already moves this workflow's 8 vCPU jobs, including cmux_next_ daemon tests and the two DAEMON_POOL mirrors, off the stalled 4 vCPU overflow failover to 16 vCPU. That covers the immediate artifacts unblock in this PR's cmux-tui-artifacts.yml part. The portable fleet placement here is still the better long-term route, but this branch will conflict with those runs-on lines and needs a base merge before it can land. #30837 does the same failover change for the remaining workflows.

teamleaderleo · 2026-11-02T00:27:33Z

Orphaned: no activity for 12 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in .github/workflows/cmux-tui-artifacts.yml, scripts/ci/portable_job_placement.py, tests/test_ci_pr_runner_pool.py. Listed for a lane to resolve.

teamleaderleo · 2026-11-02T08:06:54Z

Owned by an agent session · session

Keep base's stricter live-capacity check and shared runner state; place the daemon picker after the macOS pick so it reads the runner state document instead of minting routing App reads, and teach the daemon runs-on guard about the fleet Linux label the picker can emit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-496b6634de62976cea5a

The daemon gate's runs-on can now resolve to cmux-fleet-linux-x64 through tree-preflight's live placement output. Teach the runs-on label guard that output's only value, keep DAEMON_POOL as the Blacksmith route, and drop the RUNNER_POOLS property reads the guard's expression parser cannot read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-496b6634de62976cea5a

The placement counts room for one job, so passing its label to both build-package calls queued the musl, plan and package legs on one idle member. Only the daemon tests take it; the build keeps its own routes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-496b6634de62976cea5a

teamleaderleo · 2026-11-02T08:29:25Z

Review (subagent opus-5.5): SHIP at 23c34a4

Checked: no job can resolve to an empty runs-on; the live fleet Linux label goes only to the daemon tests (placement counts room for that one job); eligible() change only affects Linux-only pickers; no new secrets; the publisher deferral cannot loop. Placement, workflow-security, publisher-testing, linux-runner-set and runs-on label suites pass.

Follow-ups:

[follow-up] cmux-tui-build-package.yml LINUX_BUILD_RUNNER still accepts a Blacksmith-pinned inputs.linux_runner (no caller passes it now).

) #30622 wrapped the plan-build and package runs-on chains in parentheses that end after the literal Blacksmith label, so the repository-variable defaults guard no longer saw the fallback and went red on base. && binds tighter than ||, so dropping the parentheses keeps the same routing. Agent-Session: https://cmux.com/deeplink/agent?id=aid-496b6634de62976cea5a Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo requested a review from lawrencecchen as a code owner November 1, 2026 09:12

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:12 — with GitHub Actions Inactive

github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026

cubic-dev-ai Bot reviewed Nov 1, 2026

View reviewed changes

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:28 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:29 — with GitHub Actions Inactive

teamleaderleo added 3 commits November 1, 2026 09:30

ci: route cmux-tui artifact jobs through live fleet

07e7273

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

ci: include approved GCP runners in live pool reads

ef6f519

Agent-Session: https://cmux.com/deeplink/agent?id=aid-4ffe4a6a3e7fada96257 (cherry picked from commit 0829eb0)

ci: prove live Linux capacity before artifact routing

081fc49

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

teamleaderleo force-pushed the lane/tree-availability-repair branch from 4457db0 to 081fc49 Compare November 1, 2026 09:30

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:30 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:31 — with GitHub Actions Inactive

cubic-dev-ai Bot reviewed Nov 1, 2026

View reviewed changes

ci: keep artifact selector on measured Linux overflow

1a89951

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:38 — with GitHub Actions Inactive

teamleaderleo added 2 commits November 1, 2026 09:40

ci: scope Linux fleet token to artifact picker

dcc31a6

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

ci: reuse existing artifact routing token

55d49ba

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:46 — with GitHub Actions Inactive

test: cover Linux picker without route credentials

abf86a4

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fd07c55f543f6171a26c

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 09:50 — with GitHub Actions Inactive

cubic-dev-ai Bot reviewed Nov 1, 2026

View reviewed changes

This was referenced Nov 1, 2026

ci: guard cmux-tui Linux pool selectors #30724

Closed

ci(cmux-tui artifacts): 8 vCPU jobs fail over to 16 vCPU, not the stalled 4 #30810

Merged

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 12:04 — with GitHub Actions Inactive

teamleaderleo mentioned this pull request Nov 1, 2026

ci: secretless Linux jobs read RUNNER_POOLS.linux, not a hard-coded Blacksmith size #30878

Merged

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 12:24 — with GitHub Actions Inactive

teamleaderleo mentioned this pull request Nov 2, 2026

ci(cmux-tui): pick a mini only when the macOS targets may build #31824

Merged

teamleaderleo and others added 2 commits November 2, 2026 08:16

teamleaderleo temporarily deployed to ci-routing-agent November 2, 2026 08:20 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent November 2, 2026 08:28 — with GitHub Actions Inactive

cmux-bot Bot merged commit 008514f into feat-cmux-next Nov 2, 2026
162 of 164 checks passed

cmux-bot Bot deleted the lane/tree-availability-repair branch November 2, 2026 08:45

teamleaderleo mentioned this pull request Nov 2, 2026

ci: fix the cmux-tui package selectors' literal fallback (base red from #30622) #32035

Merged

github-actions Bot mentioned this pull request Nov 2, 2026

perf(cmux-tui): a respawn rebuilds its previous screen with no registry lock held; a test for the 4 MiB writer wake #32009

Merged

	runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' \|\| contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner \|\| vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-ubuntu-2404' \|\| 'blacksmith-4vcpu-ubuntu-2404' }}
	runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' \|\| contains(needs.tree-preflight.outputs.linux_jobs, ' cmux-next-daemon-tests ') && needs.tree-preflight.outputs.linux_runner \|\| vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-4vcpu-ubuntu-2404' \|\| 'blacksmith-8vcpu-ubuntu-2404' }}

Conversation

teamleaderleo commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot commented Nov 1, 2026

Bugbot needs on-demand usage enabled

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

blacksmith-sh Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

teamleaderleo commented Nov 2, 2026

Uh oh!

teamleaderleo commented Nov 2, 2026

Uh oh!

teamleaderleo commented Nov 2, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Nov 1, 2026 •

edited

Loading

github-actions Bot commented Nov 1, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

blacksmith-sh Bot commented Nov 1, 2026 •

edited

Loading