ci(overflow): the watch that holds the variable-write key keeps a literal ephemeral label

.github/workflows/ci-cloud-overflow-probe.yml

@@ -87,9 +87,14 @@ jobs:
     # every later scheduled run was evicted with no jobs and nothing decided.
     # The probes ask for the Blacksmith Linux sizes, so the watch sits on
     # Blacksmith macOS: it never queues behind its own probes and can put
-    # overflow back however the Linux sizes fare. CI_CONTROL_RUNNER, when set,
-    # moves it (an owned Mac label, for one).
-    runs-on: ${{ vars.CI_CONTROL_RUNNER || 'blacksmith-6vcpu-macos-26' }}
+    # overflow back however the Linux sizes fare.
+    # A literal ephemeral label, never a variable (not CI_CONTROL_RUNNER, as the
+    # nightly control jobs take): this job holds the App key that writes
+    # repository variables, so a variable must not move it to a reused machine,
+    # where the key could outlive the job, nor may it write its own runner.
+    # Hosted runners are refused (check_no_github_hosted_runners), so in an
+    # outage of all Blacksmith this waits and the values are set by hand.
+    runs-on: blacksmith-6vcpu-macos-26
     timeout-minutes: 25
     permissions:
       actions: write

docs/ci-runners.md

@@ -1056,8 +1056,14 @@ minutes its `probe` jobs ask for each Blacksmith Linux size compiles run on
 its `watch` job waits for them. Never one label: on 2026-10-28 the 4 vCPU
 label alone stalled four times while the other sizes started jobs, and each
 stall flipped the whole fleet. The watch never runs on GitHub-hosted Ubuntu
-or on a size the probes ask for: it takes `blacksmith-6vcpu-macos-26` (or
-`CI_CONTROL_RUNNER` when set), so it can always put overflow back. On
+or on a size the probes ask for: it takes `blacksmith-6vcpu-macos-26`, so it
+can put overflow back while Blacksmith macOS serves. That label is literal:
+the watch holds the `GLAEDA_ROUTE_APP_KEY` token that writes repository
+variables, so no variable (not `CI_CONTROL_RUNNER`) may move it to a reused
+machine, where the key could outlive the job, and it may not write its own
+runner. GitHub-hosted runners are refused, so when all of Blacksmith stops the
+watch waits, and an operator points the variables at their failovers by hand,
+as on 2026-09-29. On
 2026-10-27 a watch on hosted Ubuntu queued 94 minutes behind the switch's own
 failover and held the workflow's concurrency group, so every later run was
 evicted with nothing decided.
@@ -1072,12 +1078,15 @@ every Blacksmith Linux size stopped at once, and those two jobs, pinned to
 `blacksmith-2vcpu-ubuntu-2404`, had nowhere else to run. Set it to a label
 that starts while Blacksmith Linux does not, for example
 `blacksmith-6vcpu-macos-26`; delete it to put them back on
-`blacksmith-2vcpu-ubuntu-2404`. The overflow watch has run on that macOS
+`blacksmith-2vcpu-ubuntu-2404`. The overflow watch runs on that macOS
 label; the tick and feed watch are plain bash, Python and gh (the tick hashes
 with `shasum` where `sha256sum` is missing) but had not run on macOS when this
 was added, so read the first run after setting it. `promote-nightly-next`
-stays on its literal Blacksmith Linux label: it holds the release App
-credentials, which only a literal ephemeral label may carry. This moves only
+stays on its literal Blacksmith Linux label, and the overflow watch on its
+literal macOS label: they hold App credentials, so `CI_CONTROL_RUNNER`, which
+may name a reused machine, must not move them
+(`tests/test_ci_cloud_overflow_switch.py` refuses a secret in any job it
+moves). This moves only
 control jobs: a nightly's Linux cross-builds still wait for Blacksmith Linux
 (or the break-glass `LINUX_RUNNER` change below).
 

tests/test_ci_cloud_overflow_switch.py

@@ -825,9 +825,8 @@ def test_watch_runs_off_blacksmith_and_passes_every_switched_variable(self) -> N
         self.assertNotIn("ubuntu-24.04", runs_on)
         self.assertNotIn("ubuntu-latest", runs_on)
         self.assertNotIn("github-hosted-required", runs_on)
-        # Blacksmith macOS (or CI_CONTROL_RUNNER when set): never a size the probes ask for, so it never
-        # queues behind them.
-        self.assertEqual(runs_on, "${{ vars.CI_CONTROL_RUNNER || 'blacksmith-6vcpu-macos-26' }}")
+        # Blacksmith macOS: never a size the probes ask for, so it never queues behind them.
+        self.assertEqual(runs_on, "blacksmith-6vcpu-macos-26")
         self.assertNotIn("ubuntu", runs_on)
         names = set(switch.SWITCHED_VARIABLES) | {switch.RECORD_VARIABLE, switch.OVERRIDES_VARIABLE,
                                                   "CMUX_CI_XCODE_APP_PR"}
@@ -837,6 +836,30 @@ def test_watch_runs_off_blacksmith_and_passes_every_switched_variable(self) -> N
             self.assertTrue(passed, name)
         self.assertIn("force-cancel", self.text)
 
+    def test_the_job_with_the_variable_write_key_runs_on_a_literal_ephemeral_label(self) -> None:
+        # The watch mints the GLAEDA_ROUTE_APP_KEY token that writes repository variables. #27022 let
+        # CI_CONTROL_RUNNER pick its runner, an owned (reused) Mac for one, so a variable could put that
+        # key on a machine where it outlives the job, and the job could write the variable that picks
+        # its own runner. Only a literal ephemeral label may carry it, as for promote-nightly-next.
+        for name, job in yaml.safe_load(self.text)["jobs"].items():
+            if "secrets." not in json.dumps(job) and "create-github-app-token" not in json.dumps(job):
+                continue
+            self.assertNotIn("${{", job["runs-on"], name)
+            self.assertTrue(job["runs-on"].startswith(switch.CLOUD_PREFIX), name)
+
+    def test_no_job_that_ci_control_runner_moves_holds_a_secret(self) -> None:
+        # CI_CONTROL_RUNNER may name a reused machine, so the control jobs it moves hold only their
+        # job-scoped GITHUB_TOKEN: no secret, and no App token minted from one.
+        moved = []
+        for path in sorted(WORKFLOW.parent.glob("*.y*ml")):
+            for name, job in (yaml.safe_load(path.read_text(encoding="utf-8")).get("jobs") or {}).items():
+                if "CI_CONTROL_RUNNER" not in str(job.get("runs-on", "")):
+                    continue
+                moved.append(f"{path.name}:{name}")
+                self.assertNotIn("secrets.", json.dumps(job), f"{path.name}:{name}")
+                self.assertNotIn("create-github-app-token", json.dumps(job), f"{path.name}:{name}")
+        self.assertIn("nightly-next-tick.yml:request", moved)
+
     def test_one_fixed_group_so_two_watches_never_write_at_once(self) -> None:
         # The switch's variable writes and record delete are separate calls; a group keyed by anything a
         # run can change (a variable, an input) would let two watches interleave them.