Skip to content

ci: expose nightly tree dependencies and publisher ownership - #30221

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-tree-source-marker
Nov 1, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-tree-source-marker

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

The nightly status cannot identify which artifact-tree publisher blocks a build. The request was: “The nightly-stale line doesn't name what blocks it” and “A tree owner queued more than 3 min should show up as the nightly's blocker.”

Publish small observational artifacts from the existing decisions, before the heavy jobs queue:

  • A pending nightly dependency names its canonical v2 tree key, selected build SHA and run attempt. It deliberately does not substitute the workflow-control SHA for the build source.
  • A publisher records ownership only when the existing preflight or takeover decision says it owns an unpublished tree. The fixed artifact name supports exact-key lookup; its JSON contains the run ID and attempt, which readers must match against the current attempt.
  • Uploads are advisory, capped at one minute and retained for three days. Missing, expired, ambiguous or old-attempt evidence means unknown. No claims, workflow permissions, runner selection or release gates change.

Contract: nightly-tree-dependency-KEY-BUILDSHA-attempt-N contains tree-dependency.json; cmux-tui-tree-owner-KEY contains tree-owner.json. JSON fields are schema_version:1, kind, key, run_id, run_attempt, plus build_sha only for a dependency. Owner markers use overwrite for a rerun; consumers must still reject an old attempt. Runs created before this producer lands cannot be retroactively identified from these markers.

Validation: fleet run 41278321960 passed all 33 Python tests, the nightly-resolver check and the 231-workflow permission scan. Its terminal receipt binds tested_sha=3d4b174746b5ed17b2f91f5aceead2f11df999f4 to snapshot 4913d3a2dd08b2789b6c9a2c4d0bb70e34d72545, with exit 0 in 24.709 s on 16 CPUs. Existing warm test snapshot path is reused; no app compile or runner routing change is needed.

Independent Opus 5.5 medium review: SHIP at the same full head. Nonblocking follow-ups: #30227.

Prior art: preserves #28197's tree-at-install behavior, #26800's distinction between publication and daemon testing, #26781's bounded owner election and #28764's compatibility-tree resolution. The new evidence uses the canonical key, not a compatible published alias. The overlap helper found no open PR for this change.

Agent session: agent session

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 3d4b174: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30221 -f ref=lane/nightly-tree-source-marker -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on c9b1473 to compare; do not merge until green; see lane-rules

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/cmux-tui-artifacts.yml">

<violation number="1" location=".github/workflows/cmux-tui-artifacts.yml:1099">
P3: This observation step runs a script that is only checked out when `owner_run_id` is set. When `owner_run_id` is empty, the script is missing and the step fails on every run, showing red in an otherwise green job. Gate the step on `owner_run_id != ''`, which is the only case where the owner marker can be produced.</violation>
</file>

<file name="scripts/ci/nightly_tree_source.py">

<violation number="1" location="scripts/ci/nightly_tree_source.py:40">
P2: The docstring tells readers to match only `run_attempt`, but the owner name has no run ID, so exact-name lookups return markers from earlier runs. Those markers pass an attempt-only check whenever the attempt numbers match. Readers must also match JSON `run_id` against the current run, and the docstring should say so.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

value["build_sha"] = sha
name = f"nightly-tree-dependency-{key}-{sha}-attempt-{value['run_attempt']}"
else:
name = f"cmux-tui-tree-owner-{key}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The docstring tells readers to match only run_attempt, but the owner name has no run ID, so exact-name lookups return markers from earlier runs. Those markers pass an attempt-only check whenever the attempt numbers match. Readers must also match JSON run_id against the current run, and the docstring should say so.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly_tree_source.py, line 40:

<comment>The docstring tells readers to match only `run_attempt`, but the owner name has no run ID, so exact-name lookups return markers from earlier runs. Those markers pass an attempt-only check whenever the attempt numbers match. Readers must also match JSON `run_id` against the current run, and the docstring should say so.</comment>

<file context>
@@ -0,0 +1,56 @@
+        value["build_sha"] = sha
+        name = f"nightly-tree-dependency-{key}-{sha}-attempt-{value['run_attempt']}"
+    else:
+        name = f"cmux-tui-tree-owner-{key}"
+    return name, value
+
</file context>

continue-on-error: true
env:
TREE_KEY: ${{ needs.tree-preflight.outputs.key }}
OWNS_KEY: ${{ steps.recheck.outputs.takeover == 'true' && 'true' || steps.wait.outputs.takeover }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This observation step runs a script that is only checked out when owner_run_id is set. When owner_run_id is empty, the script is missing and the step fails on every run, showing red in an otherwise green job. Gate the step on owner_run_id != '', which is the only case where the owner marker can be produced.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/cmux-tui-artifacts.yml, line 1099:

<comment>This observation step runs a script that is only checked out when `owner_run_id` is set. When `owner_run_id` is empty, the script is missing and the step fails on every run, showing red in an otherwise green job. Gate the step on `owner_run_id != ''`, which is the only case where the owner marker can be produced.</comment>

<file context>
@@ -1068,6 +1090,27 @@ jobs:
+        continue-on-error: true
+        env:
+          TREE_KEY: ${{ needs.tree-preflight.outputs.key }}
+          OWNS_KEY: ${{ steps.recheck.outputs.takeover == 'true' && 'true' || steps.wait.outputs.takeover }}
+          TREE_READY: ${{ needs.tree-preflight.outputs.tree_ready }}
+        run: python3 scripts/ci/nightly_tree_source.py owner
</file context>

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (claude-opus-5-5 medium): SHIP at 3d4b174

Scope: c9b1473..3d4b174 (5 files, +221). Read-only review of the diff and the surrounding workflow control paths via git show; nothing run.

Checked:

  • Owner gate, tree-preflight: OWNS_KEY = steps.inspect.outputs.owns_key, TREE_READY = steps.inspect.outputs.tree_ready, and the script publishes only when owns_key == "true" AND tree_ready == "false". That is the same owns_key the build/build-windows gates read (needs.tree-preflight.outputs.owns_key).
  • Owner gate, tree-owner-wait: OWNS_KEY uses the same expression as the job's takeover output (steps.recheck.outputs.takeover == 'true' && 'true' || steps.wait.outputs.takeover), which is what the build gates read. TREE_READY comes from needs.tree-preflight.outputs.tree_ready. The two branches can't both fire: owner-wait is skipped when preflight owns_key == 'true'.
  • Pending dependency: TREE_KEY = steps.pending.outputs.key (written only when tree_pending=true, validated as 40 hex). BUILD_SHA = steps.build.outputs.build_sha, the commit the build chose, not github.sha. If BUILD_SHA is missing, the script refuses to publish rather than falling back (tested). The step sits right after "Choose the build commit" in resolve-nightly-cmux-tui-client, before chief-key/superseded and before every downstream build job.
  • Payload/naming: schema_version 1, kind, key, run_id, run_attempt (positive ints, validated), plus build_sha for a dependency. Names are cmux-tui-tree-owner-<KEY> -> tree-owner.json (a stable name, overwrite: true, so a rerun replaces the earlier attempt's artifact; readers compare run_attempt against the run's current attempt) and nightly-tree-dependency-<KEY>-<BUILDSHA>-attempt-<N> -> tree-dependency.json. Key and SHA are regex-validated before they go into GITHUB_OUTPUT, so no newline or output injection (tested with "KEY\nname=other").
  • Advisory behaviour: every record step has continue-on-error. The upload runs only when a name was emitted, and has continue-on-error and timeout-minutes: 1. A failure leaves no marker, which the consumer reads as unknown. It adds no gate.
  • Script availability: tree-preflight does a full checkout (fetch-depth 1). tree-owner-wait's sparse checkout adds scripts/ci/nightly_tree_source.py. The nightly resolve job checks out the full tip at depth 64. python3 is already used by each of these jobs.
  • No changes to permissions, secrets, triggers, claim authority (cmux-tui-tree-claim.sh untouched) or concurrency. No contents API or REST log reads. The upload-artifact SHA pin (v7.0.1, 043fb46d…) is already used in the repo's nightly notarization workflow.
  • Tests: tests/test_nightly_tree_source_marker.py checks the dependency binding to BUILD_SHA, the owner payload and stable name across attempts, that nothing is published for not-owner, unknown or ready, key/SHA/run-id/attempt validation, and the injection case. It is registered in tests/test-execution.toml as lane "manual" with a reason, so no new CI gate.

Follow-ups (non-blocking):

  1. [follow-up] The tests exercise the script only. Nothing pins the workflow wiring, such as the owner-wait OWNS_KEY expression staying equal to the job's takeover output, or the inspect/pending step ids. A small YAML-contract assertion would catch drift if those gates change.
  2. [follow-up] cmux-tui-artifacts.yml now uses upload-artifact v7.0.1 for these markers next to v4.6.2 for the existing feat-builder/deferred markers. That is harmless, since v4+ artifacts interoperate and these steps are advisory, but it's worth aligning on the next touch. If any runner in the tree-preflight/owner-wait pools lacks the Node runtime v7 needs, the marker silently goes missing ("unknown"), so NOW's consumer should show that state.
  3. [follow-up] When tree_pending is true but build.already_published is also true, a dependency marker is still written. That's harmless as an observation, but the consumer may want to ignore dependencies of runs that built nothing.

No blocking findings.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo
teamleaderleo merged commit 5bdd4c9 into feat-cmux-next Nov 1, 2026
183 of 206 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-tree-source-marker branch November 1, 2026 05:52
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
… (highlights, capabilities, step coverage, arm64 recorder) (#30624)

* base red: tooling-tests, register test_ci_blacksmith_linux_sizes.py as manual and the unregistered tests

tooling-tests failed on every feat-cmux-next pull request at
"validate_test_execution_registry.py --pr-lanes feat-cmux-next":
  tests/test_ci_blacksmith_linux_sizes.py: linux-guard lane is not run by any workflow
#30360 (d57f2fc) registered the test as linux-guard but added no
ci-guards.yml step that runs it. CLAUDE.md forbids new CI tests (Lawrence
2026-10-23), so it becomes a manual entry with how to run it.

The registry also warned about six tests with no entry. test_nightly_dedicated_mini.py
is already run by ci-guards.yml, so it takes linux-guard; the other five are run
by no workflow and enter as manual, each with its own run command.

Red at 03423de:
  python3 scripts/ci/validate_test_execution_registry.py --pr-lanes feat-cmux-next --base-sha HEAD^1   exit 1
Green here: same command, exit 0 (594 tests, manual=31).

* base red: Checks job structure, the preflight tests read the preflight step by id

tests/test_cmux_next_checks_workflow.py read the preflight script as the last
step of cmux-tui-artifacts.yml's tree-preflight job. #30221 (5bdd4c9) added
"Record tree owner observation" and an upload step after it, so 14 tests failed
with KeyError: 'run' on every pull request. They now take the step by its id
(push-head-preflight's `head`, tree-preflight's `inspect`); the assertions do
not change.

Red at 03423de: python3 tests/test_cmux_next_checks_workflow.py   FAILED (errors=14)
Green here: same command, 165 tests OK.

* base red: Tier routing, an agent pane on a dev server URL loads its own document

test_no_suite_runs_page_javascript_on_a_built_bundle flagged
AgentPageHibernationTests.swift after #30312 (b16b20c) made it call
waitUntilLoaded. That suite's agent pane loads AgentPaneSource.devURLVariable,
a loopback port nobody serves (Debug test builds honor it), not the built
bundle, so it reads no bundle content. The check now counts
AgentPaneSource.devURLVariable as a suite's own document, beside
loadHTMLString and the fixture roots.

Red at 03423de: python3 tests/test_cmux_next_route.py   FAILED (failures=1)
Green here: same command, 112 tests OK.

* base red: daemon capabilities export lists agent-session-start-no-cwd-v1

#30363 (efd30f5) added the optional capability agent-session-start-no-cwd-v1
to DaemonEndpoint.swift without the export, which reddened "Daemon capabilities
export is fresh" and the daemon-capabilities-fresh.test.sh script test.
Regenerated with the check's own renderer (--write), as base-autofix does.

Red at 03423de: python3 scripts/cmux-next/check-daemon-capabilities-fresh.py   exit 1
Green here: same command, exit 0; bash scripts/cmux-next/tests/daemon-capabilities-fresh.test.sh ok.

* base red: String tables, store.updates.available has English's plural forms in every language

#30288 (e6bfd26) made store.updates.available a plural with `one` and
`other`, but ja, km, ko, th, vi, zh-Hans and zh-Hant got only `other`, and
check-l10n.sh requires English's forms in every language. Their `one` form is
their `other` text, as home.stopAll.couldNotStop does.

Also adds the 12 languages #27883 (16b7cc2) left out of
action.sidebar.compactList.toggle (style debt CI only warns about), marked
needs_review.

Red at 03423de: scripts/cmux-next/check-l10n.sh .   7 errors (19 without L10N_STYLE_WARN_MAX)
Green here: same command, 0 errors.

* base red: release-notary, three highlights link their pull requests

scripts/whats-new/test_digest.py needs a try-it action, docs or a pr link on
every highlight. chat-title-plain (#29920), cloud-live-previews (#30334) and
working-harness-tab-icons (#27905) had none; each now links its pull request,
as #29939 did for priority-view-drag.

Red at 03423de: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: same command, OK.

* base red: release-notary, the static capability check skips agent-session-start-no-cwd-v1

#30363 (efd30f5) added agent-session-start-no-cwd-v1, which cmux-tui's
identify_capabilities pushes at runtime (capabilities.rs), so the bundled static
manifest never lists it. tests/test_cmux_next_static_capability_runtime_only.py
(release-notary, "Validate nightly push throttle") failed, and every nightly's
check-daemon-capabilities.sh --static-only would refuse the build. It joins
runtime_only, as #30447 did for agent-session-settings-v1.

Red at 03423de: python3 tests/test_cmux_next_static_capability_runtime_only.py   FAILED (failures=1)
Green here: same command, OK.

* l10n: compact workspace list uses this table's word for workspace (it, pl, pt-BR, uk, th)

* base red: release-notary, nightly step coverage follows #30200's visual verdict steps

#30200 (86fd376) removed find-nightly-next-recovery's "Read the visual
gate's verdicts" and renamed visual-gate-nightly-verdict's "Fail with the gate
when it holds the build" to "Record failed advisory verdict" (same script,
visual-gate-fail-when-held.sh). .github/nightly-step-coverage.json kept the old
names, so tests/test_nightly_step_coverage.py failed. The stale entry goes and
the renamed step keeps its plumbing entry.

Red at 03423de: python3 tests/test_nightly_step_coverage.py   FAILED (failures=1)
Green here: same command, OK.

* base red: release-notary, the nightly arm64-only recorder has the cache classifier the Release step sources

#30107 (2379dd9) made nightly.yml's "Build nightly app (Release)" source
scripts/cmux-next/lib/compilation-cache.sh. tests/test_nightly_next_arm64_only.py
runs that step in a scratch tree with recorder executables, and the tree had no
such file, so the step exited 1 before xcodebuild's recorder and the test failed
with CalledProcessError. The scratch tree now gets the real file, as it does
scripts/lib/rust-targets.sh. Every assertion is the same.

Red at 03423de: GITHUB_STEP_SUMMARY=/tmp/s python3 tests/test_nightly_next_arm64_only.py   CalledProcessError
Green here: same command, PASS: nightly-next arm64 only.

* base red: release-notary, nightly-next identity reads a YAML-anchored job env as its value

#30278 (fe4577e) turned the publish jobs' NIGHTLY_FEEDS into a YAML anchor
(`&pub-feeds ${{ needs.decide.outputs.feeds }}`) and aliases (`*pub-feeds`).
tests/test_nightly_next_identity.py reads job env as raw text, so
test_each_app_reads_the_feed_its_publisher_writes failed on the anchor prefix.
job_env now resolves an alias to its one anchor and drops an anchor name, as
YAML does; the asserted values are the same.

Red at 03423de: python3 tests/test_nightly_next_identity.py   7/8 passed
Green here: same command, 8/8 passed.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 3d4b1747 Deployed Nov 1, 2026 by teamleaderleo via wait for same-key tree publisher #93989
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant