Skip to content

fix(ci-overflow): admit hosted jobs at measured p90 estimate + headroom M; release runner-less cancels (t_f459aa52) - #1216

Merged
ang-fleet-lander[bot] merged 3 commits into
mainfrom
daedalus/t_f459aa52-admit-at-estimate
Sep 26, 2026
Merged

ang-fleet-lander[bot] merged 3 commits into
mainfrom
daedalus/t_f459aa52-admit-at-estimate

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #1125 (base = its branch; retargets to main when #1125 merges).

Apollo D4 ruling (option 1b) on t_f459aa52. The day drained to remaining_allowance=1 at 02:54Z because 13-14 concurrent merge_group attempts each held 300 reserved minutes, while real billed use was about 55-73 min per run.

  • Ledger(estimate_headroom=M): reserve() prices each hosted job at E, the p90 of the ledger's billed_samples[slice|e2e]. Until 20 samples exist, E falls back to the ceiling of 35/20. A job is admitted while consumed + E + M <= limit. None keeps legacy D4 ceiling mode, and an invalid M admits no cloud.
  • Reconcile charges min(ceil(billed), ceiling), including billed above E. An unmeasured executed job is charged the ceiling. Measured jobs are recorded as samples, capped at 200 per kind.
  • 5.3a ride-along: a job with status=completed, conclusion=cancelled, empty runner_name and steps=[] is released.
  • Validator: reservation must be 1..ceiling, hosted_minutes 1..ceiling (it may exceed the reservation), and billed_samples is optional and validated.
  • Mutation gate: 36/36 mutants killed, including 10 new D4a arms.

The controller side (config admission_headroom, passing conclusion+steps, vendor repin) is a separate fleet-ops-scripts PR.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Base automatically changed from daedalus-opus/t_38a419e0-hosted-actual to main September 26, 2026 04:37
…droom M; release runner-less cancels (t_f459aa52)

D4 amendment (Apollo ruling 2026-09-25 21:2x PT, option 1b). Ledger(estimate_headroom=M):
reserve() prices each hosted job at E = p90 of the ledger's billed_samples for its kind
(slice/e2e; ceiling 35/20 until >=20 samples) and admits while consumed + E + M <= limit.
estimate_headroom=None keeps the legacy ceiling mode. Reconcile charges min(ceil(billed),
ceiling) (above E too), unmeasured executed jobs the ceiling, records samples (cap 200),
and releases a cancelled job with no runner and steps=[] (5.3a). Validator: reservation
1..ceiling, hosted_minutes 1..ceiling, optional billed_samples validated.

Verified: tests/test_ci_overflow_ledger.py + plan + contract + preflight pass (py3.14);
scripts/ci_overflow_mutation_gate.py 36/36 mutants killed (10 new D4a arms).
@Kyzcreig
Kyzcreig force-pushed the daedalus/t_f459aa52-admit-at-estimate branch from 5c5bd39 to 78401d4 Compare September 26, 2026 05:26
@ang-fleet-lander

Copy link
Copy Markdown

🤖 merged-by: apollo · lane: kanban-t_f459aa52 · gate: BYPASS: FR PAUSED by Ace ruling 2026-09-22 (state/fleetreview-pause-20260922.md); Apollo-reviewed lands via bypass · why: Apollo review PASS (D4 option 1b): p90-per-kind admission estimate with MIN_SAMPLES ceiling fallback, headroom M, reconcile charges max(billed,estimate)+records samples, runner-less cancel release w/ negative control; rebased over #1125 (0 conflicts), ledger suite 92/92 via test-gate; the one red slice (test_mcp_serve startup-baseline) is unrelated to the PR's files and passed on rerun

@blacksmith-sh

This comment has been minimized.

@ang-fleet-lander

Copy link
Copy Markdown

🤖 merged-by: apollo · lane: review-1216 · gate: BYPASS: FR PAUSED by Ace ruling 2026-09-22 (state/fleetreview-pause-20260922.md); Apollo-reviewed lands via bypass · why: Apollo review PASS (D4 option 1b; ledger suite 92/92 on rebased head); rebased on #1232 e2b9ba8, checks green

@ang-fleet-lander
ang-fleet-lander Bot added this pull request to the merge queue Sep 26, 2026
Merged via the queue into main with commit fb4d122 Sep 26, 2026
56 checks passed
@ang-fleet-lander
ang-fleet-lander Bot deleted the daedalus/t_f459aa52-admit-at-estimate branch September 26, 2026 07:38
@ang-prism

ang-prism Bot commented Sep 27, 2026

Copy link
Copy Markdown

FleetReview

Review: post-merge · head fb4d12216f31 · duration 17m 33s
Profile: light (merit: default light: lines 243<800, files 3<1000000, hunks 14<1000000, no hot path) · policy: below-size-and-path-gates
Roster: B-assert-ctx → gpt-6-sol (openai), B-state → gpt-6-sol (openai), F → gpt-6-sol (openai), G → grok-4.6 (xai)

Post-merge review (fleetreview:post-merge override): this reviewed the merge commit against its first parent — the bytes that already shipped. It is not a pre-merge gate pass.

profile: light (rule: default light: lines 243<800, files 3<1000000, hunks 14<1000000, no hot path) · round 0 · members: B-assert-ctx, B-state, F, G · families: openai,xai

Confidence: 2/5

Findings

  • P1 scripts/ci_overflow_ledger.py:359 — Premature charge · agreed: B-assert-ctx,B-state,G (openai, xai)
  • P1 scripts/ci_overflow_ledger.py:286 — Estimated admission cannot recover jobs the planner already put on the local pool · agreed: F (openai)

FleetReview provenance · models: B=gpt-6-sol, D=grok-4.6, F=gpt-6-sol · cost: $0.91 · duration: 17m 30s · rounds: 1 · files examined: 3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant