Skip to content

ci(cmux-next): the DEV REF capture retries a rate-limited lookup and compiles instead of reusing - #25866

Merged
teamleaderleo merged 9 commits into
feat-cmux-nextfrom
darling-capture-fix
Oct 27, 2026
Merged

teamleaderleo merged 9 commits into
feat-cmux-nextfrom
darling-capture-fix

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

DEV run 40428066457 at c28f2f1 took no REF capture, so the fleet whole-app proof has no hybrid-ref-12 to build from. The capture-decision step's gh api lookup hit "API rate limit exceeded for installation" at 02:11:03Z and exited 1 before it wrote capture. Because the step is continue-on-error, the run stayed green. A rerun would not have fixed it: the product was a reuse hit, so the rerun would have compiled nothing.

  • hybrid_ref_capture.py have: the artifact lookup, moved to Python. It waits out a rate limit (Retry-After or x-ratelimit-reset) and backs off on 5xx and network errors, for 600 s at most, then fails with the last error. A 403 that is not a rate limit fails at once.
  • The decide step now runs before the product-reuse lookup. A key with no complete capture sets its new compile output, which SKIP_REASON reads, so that build really compiles and captures. Most landings share a key that already has a capture and still reuse.
  • An incomplete capture's manifest is now named hybrid-ref-capture-incomplete-KEY (it was hybrid-ref-capture-KEY-incomplete-RUNID, which no exact-name lookup finds). have counts the ones from the last day, and while one exists the key does not skip reuse. A key whose capture always comes out incomplete therefore forces at most one extra compile a day, not one on every landing. hq's hybrid-ref-run.sh reads only complete manifests, so the rename doesn't affect it.
  • A lookup that still fails sets capture=true compile=true, prints an ::error:: annotation and fails the step. The job is still not failed, so DEV publication is never held, but the run shows the failure and the build compiles and captures.
  • The decide step's timeout goes from 1 to 12 minutes. A capture still starts only before job minute 40, unchanged.
  • No new token: this job has no App token, and minting one here would be a secrets change. It stays on github.token.

Not touched: the nightly's capture in #25812, which has the same gh api lookup and can switch to have after this lands.

Tests

python3 tests/test_cmux_next_hybrid_ref_capture.py: 9 red at base cdc41df (first commit); 20 of 20 pass at the head, eight runs in a row. test_the_decision runs the decide step's real shell on four have answers. tests/test_dev_release_product_reuse.py passes. One of the 9, test_off_by_default_and_never_in_the_way, was already red on base: the job's timeout-minutes became an expression. It now reads the mini branch, 60. actionlint gives the same 4 findings as base, all pre-existing. prepush-gate is clean.

Observed / Not run

Ran: the test file above (base and head), actionlint and prepush-gate. Read the 40428066457 DEV compile job log (job 121939218995) for the failure.
Not run: a real DEV daily with capture on. That is the next step: a DEV daily dispatched at the tip after this lands.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session

teamleaderleo and others added 3 commits October 26, 2026 19:37
…nd a reused product

Red at cdc41df: the Have tests and the two new Workflow tests fail
(no have subcommand, the lookup runs after the product lookup and a failed
lookup sets no capture). The job-timeout assertion was already red on base
(the job's timeout-minutes is an expression); it now reads its mini branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
…compiles instead of reusing

DEV run 40428066457 (c28f2f1) took no capture: the decide step's gh api
lookup hit "API rate limit exceeded for installation", exited 1 before it
wrote capture, and continue-on-error kept the run green. A rerun would not
have helped: the product was a reuse hit, so it would have compiled nothing.

- hybrid_ref_capture.py have: the artifact lookup in Python, waiting out a
  rate limit (Retry-After or the reset) and backing off on server errors,
  for 600 s at most; then it fails with the last error.
- The decide step runs before the product lookup, and a key with no capture
  skips reuse (SKIP_REASON), so that build really compiles and captures.
- A lookup that still fails sets capture=true, prints an ::error:: and fails
  the step (continue-on-error), so the run shows it and the capture is taken.

tests/test_cmux_next_hybrid_ref_capture.py: 18 of 18 pass (9 red at base).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 27, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review cce480c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25866 -f ref=darling-capture-fix -f platform=macos -f backend_mode=local

web and size checks: green

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-7qnoi created for this PR by the cmux coordinator.

…nce a day, not on every landing

An incomplete capture was uploaded as hybrid-ref-capture-KEY-incomplete-RUNID,
which no exact-name lookup finds, so with reuse skipped for a missing capture
a key that never captures whole would have compiled on every landing. Its
manifest is now hybrid-ref-capture-incomplete-KEY; have also counts those of
the last day, and the decide step's new compile output (what SKIP_REASON reads)
is true only when there is neither. A failed lookup still captures a build
that compiles, without forcing one. tests: the decide step's shell runs on
four have answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
@cursor

cursor Bot commented Oct 27, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@blacksmith-sh

blacksmith-sh Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…t lost to a reused product

Codex review of 012949b: with compile=false a failed lookup could still
reuse the product, and a reuse builds nothing to capture (40428066457 again).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
teamleaderleo and others added 4 commits October 26, 2026 20:26
…n guards

The polling guard scans hybrid_ref_capture.py whole, so have's bounded
rate-limit retry flags both steps that call it; the retention allowlist is
keyed on the manifest's artifact name, which this PR changed. The manifest is
small JSON kept 14 days as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
…ture-fix

# Conflicts:
#	.github/workflows/cmux-next.yml
…lookup, and a truncated error body

Codex review round 2 of e3297c9:
- The capture step's own jobs read (for the 40-minute rule) has no retry; a
  rate limit still active after the lookup's 10 minutes made it skip the
  capture. It now falls back to the decide step's start, written before the
  lookup (that step runs early in the job).
- An HTTPError whose body read fails (IncompleteRead) escaped the retry; it now
  counts as its status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-a1351026211d556f5a03
…ture-fix

# Conflicts:
#	tests/test_cmux_next_hybrid_ref_capture.py
@teamleaderleo
teamleaderleo merged commit 9198239 into feat-cmux-next Oct 27, 2026
122 of 130 checks passed
@teamleaderleo
teamleaderleo deleted the darling-capture-fix branch October 27, 2026 03:52
teamleaderleo added a commit that referenced this pull request Oct 27, 2026
…e.py have

Takes #25866's lookup: have counts complete and recent incomplete captures,
waits out a rate limit (30 s here, before the nightly compile) and exits 1 on a
lookup it cannot answer. The nightly decide step compiles and captures anyway
and fails red then, as the DEV one does. The artifact names follow the profile
(hybrid-ref-capture-nightly-next-[incomplete-]KEY; DEV names unchanged), the
upload writes the same names have reads, and the product lookup misses on
compile=true. An unknown job start moves the capture's deadline earlier
instead of skipping. The cleanup tests reap their own orphans (subreaper) and
check the helper ended by SIGKILL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f6b9335f650995bf635b
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

3 inactive (2 outdated) deployments
ci-routing-agent — cce480cf Deployed Oct 27, 2026 by teamleaderleo via wait for same-key tree publisher #62035
artifacts — e3297c9a Deployed Oct 27, 2026 by teamleaderleo via publish to R2 #61888
sentry-dsyms — e3297c9a Deployed Oct 27, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #61888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant