feat(scripts): queue heavy gates behind a machine-wide slot lock

[mateo-berri]

TLDR

Problem this solves:

• N parallel make check runs across worktrees thrash one machine
• Every run slows down and long commands start timing out flaky

How it solves it:

• New scripts/gate_slot_lock.py: machine-wide fcntl slot lock, 2 slots
• make check/bootstrap/lint and the budget gates queue through it
• make lint takes its slot first, so even its dependency setup queues
• Nested invocations reenter via an env marker, so no deadlocks
• LITELLM_GATE_SLOTS overrides the count; 0 disables; errors fail open

User Flow

Before: a developer running several agent sessions has three worktrees run make check at once, and the machine becomes unusable

1. Three sessions each run make check in their own litellm worktree at the same time
2. Three basedpyright/prettier/pytest storms execute simultaneously, CPU and RAM peg, and every run crawls
3. Unrelated commands on the machine slow down or time out while all three grind on

After: the same three runs queue two at a time and all pass

1. Three sessions each run make check in their own litellm worktree at the same time
2. Two runs proceed immediately; the third prints gate_slot_lock: all 2 machine-wide slots are busy; queueing (set LITELLM_GATE_SLOTS=0 to disable) and waits
3. As soon as one of the first two finishes, the queued run starts on its own and ends in check: PASS like the others
4. A developer who wants the old behavior sets LITELLM_GATE_SLOTS=0 (or another slot count) and the lock steps aside

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before, at the merge base 870a8cf: three parallel make check runs across three bootstrapped worktrees all execute at once, nothing queues or throttles them

$ for i in 1 2 3; do bash -c "echo \"run $i launched \$(date +%T)\"; make -C $W/qa36988-b$i check > /tmp/qa36988_before_$i.log 2>&1; echo \"run $i exit=\$? finished \$(date +%T)\"" & done; wait
run 1 launched 21:18:25
run 2 launched 21:18:25
run 3 launched 21:18:25
run 3 exit=0 finished 21:18:27
run 1 exit=0 finished 21:18:27
run 2 exit=0 finished 21:18:27

$ grep -c queueing /tmp/qa36988_before_*.log
/tmp/qa36988_before_1.log:0
/tmp/qa36988_before_2.log:0
/tmp/qa36988_before_3.log:0

$ for i in 1 2 3; do echo "log $i: $(grep -o 'check: PASS' /tmp/qa36988_before_$i.log)"; done
log 1: check: PASS
log 2: check: PASS
log 3: check: PASS

After, at this PR's tip 17f5c90: the same three runs queue two at a time, the third prints the queueing notice, and all three pass

$ for i in 1 2 3; do bash -c "echo \"run $i launched \$(date +%T)\"; make -C $W/qa36988-a$i check > /tmp/qa36988_after_$i.log 2>&1; echo \"run $i exit=\$? finished \$(date +%T)\"" & done; wait
run 1 launched 21:19:12
run 2 launched 21:19:12
run 3 launched 21:19:12
run 1 exit=0 finished 21:19:14
run 2 exit=0 finished 21:19:14
run 3 exit=0 finished 21:19:16

$ grep -l queueing /tmp/qa36988_after_*.log
/tmp/qa36988_after_3.log
$ grep -h queueing /tmp/qa36988_after_3.log
gate_slot_lock: all 2 machine-wide slots are busy; queueing (set LITELLM_GATE_SLOTS=0 to disable)

$ grep -H "check: PASS" /tmp/qa36988_after_*.log
/tmp/qa36988_after_1.log:check: PASS
/tmp/qa36988_after_2.log:check: PASS
/tmp/qa36988_after_3.log:check: PASS

Direct make lint at 17f5c90 joins the queue before doing any setup: with both slots held by long-running checks, its first output is the queueing notice, and uv sync/git fetch only run once a slot frees (at the parent commit the setup deps ran before the queue was joined); the run then passes

$ head -3 /tmp/qa36988_lint.log
gate_slot_lock: all 2 machine-wide slots are busy; queueing (set LITELLM_GATE_SLOTS=0 to disable)
uv sync --inexact --frozen --group proxy-dev --group e2e-dev
Audited 198 packages in 21ms

lint exit=0 finished 21:30:27

The header-documented hook flow still works at the tip. Linked worktrees resolve .git/hooks to the shared main checkout, so this demo pins the same script via core.hooksPath; a fresh-clone ln -s install goes through the identical entrypoint and is pinned by a regression test

$ git -c core.hooksPath=.qa-hooks commit -m "chore: demo commit through the hook"
check: logging full output to .../worktrees/qa36988-a1/pre_commit_lint.log
check: summary
    skipped: Python lint (make lint) (no litellm/ Python files in scope)
    ...
check: PASS
[detached HEAD 5ec947e0b1] chore: demo commit through the hook
 1 file changed, 1 insertion(+)

Notes from the QA run:

• Clean-tree checks short-circuit to "nothing to check" (pre-existing, unchanged)
• make lint always runs full checks; only make check scopes (unchanged)

Type

🚄 Infrastructure

Caveats (if any)

• make -jN check sub-make may warn "jobserver unavailable"; inner targets are serial anyway
• Queueing is roughly FIFO; a fresh arrival can occasionally beat a waiter to a freed slot
• CI takes the instant-acquire path (one job per machine) and any lock error fails open

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR adds a machine-wide slot queue around heavy developer gates to limit concurrent resource usage across worktrees.

• Adds an fcntl-based slot-lock wrapper with configurable capacity, nested invocation handling, and fail-open behavior.
• Moves lint, bootstrap, and check work behind wrapper and inner targets.
• Wraps standalone budget gates and the pre-commit script in the same locking mechanism.
• Adds concurrency, lifecycle, target-ordering, exit-code, and hook-install regression tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
Makefile	Heavy public targets now acquire a slot before delegating to inner targets; the prior lint setup bypass is fixed.
scripts/gate_slot_lock.py	Implements configurable machine-wide file-lock slots, inherited re-entry markers, command execution, and fail-open filesystem handling.
scripts/pre_commit_lint.sh	Re-executes through the slot wrapper before running any lint work while preserving symlink-hook operation.
scripts/ruff_strict_gate.py	Runs the Ruff budget gate within the shared slot context.
scripts/type_check_gate.py	Runs environment preparation and type-check budget processing within the shared slot context.
scripts/type_discipline_gate.py	Runs the type-discipline budget gate within the shared slot context.
tests/test_litellm/test_gate_slot_lock.py	Exercises capacity limits, queueing, re-entry, cleanup, exit propagation, fail-open behavior, and lint target ordering.
tests/test_litellm/test_pre_commit_lint.py	Verifies pre-commit locking, held-slot re-entry, and symlink-based hook installation.

_{Reviews (2): Last reviewed commit: "fix(make): acquire the gate slot before ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit eafddaa. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 17f5c90. Configure here.}