test: deflake JWT tamper, fuzzy picker, tag routing, liveliness, redis stall burst, and pre-commit interrupt tests

[devin-ai-integration]

TLDR

Problem this solves:

• Twenty unit test node ids flaked in CI across the 96h window ending 2026-09-04 09:17 UTC, from seven root causes
• Redis loop-stall breaker test needed the event loop to reach its stall in under 1ms (fixed on staging by 7d3b03d while this PR was open)
• Pre-commit interrupt test caught a real leak: the eslint report file survived Ctrl-C
• JWT tamper tests could leave the signature bytes unchanged and still expect rejection
• Fuzzy picker tests sent keys after fixed sleeps, racing the widget on slow workers
• Tag routing tests sampled ten random picks and expected both deployments to show up
• Liveliness test timed the first request through a fresh app against a 100ms budget

How it solves it:

• The Redis test rewrite this PR carried was dropped in favor of staging's 7d3b03d, so test_redis_cache.py no longer differs from staging
• pre_commit_lint.sh removes the eslint report from an EXIT trap in the dashboard subshell
• Tamper helper flips a bit in the decoded signature, not in base64url padding
• Picker driver waits for the widget to highlight the expected choice before the next key
• Tag routing tests route through acompletion until both deployments are seen, capped at 100 picks
• Liveliness test warms up once, then times five polls and checks the median
• No proxy runtime code changes, no reruns, skips, or widened assertions added

User Flow

Before: a developer presses Ctrl-C while make check is running the dashboard lint

1. They run make check in a checkout with a dashboard file staged and press Ctrl-C while the eslint step is still running
2. The command exits with status 130 and the background lint jobs die
3. ls $TMPDIR shows a stray tmp.XXXXXXXXXX file left over from the eslint report, and one more appears every time this happens

After: the same Ctrl-C leaves nothing behind

1. They run make check with a dashboard file staged and press Ctrl-C while the eslint step is still running
2. The command exits with status 130 and the background lint jobs die
3. ls $TMPDIR shows no leftover file

Everything else in this PR only edits unit tests, so requests against a proxy behave identically before and after. The affected workflows (MCP outbound session tokens, the litellm autoroute configure wizard, tag based routing, and the liveliness probe) are covered by the tests below and are unchanged

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Flake evidence and fixes

This is the rolling deflake PR. Each dated section below is one daily run. Evidence is either the pytest-rerunfailures RERUN then PASSED pair for the same node id inside one job on one commit, or the same job on the same commit failing on attempt 1 and passing on attempt 2, read from the job logs of every Unit Tests run in that run's 24h window. Counts are the number of jobs where that happened

2026-09-01

tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_session_token.py::test_rotation_window_still_enforces_expiry_and_tamper_on_the_previous_key, 5 jobs in the window, 3 on this date

• run 33521083413 attempt 1, job 99900558601 proxy-infra, commit 6e388b1
• run 33538698074 attempt 1, job 99959269612 proxy-infra, staging f2a4172
• run 33564822247 attempt 1, job 100045475477 proxy-infra, staging 8469003

tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_session_token.py::test_rs256_tampered_signature_is_bad_signature, 2 jobs

• run 33549069174 attempt 1, job 99993728819 proxy-infra, staging 0cf236b
• run 33557266115 attempt 1, job 100021043687 proxy-infra, staging aab9abd

tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_session_credentials.py::test_resolve_fails_tampered_token_closed_without_expiry_flag, 1 job

• run 33549552638 attempt 1, job 99995356647 proxy-infra, staging 695d943

Mechanism (all three, unseeded randomness): every mint gets a fresh jti from secrets.token_urlsafe, so the signature differs each run. The tests tampered by rewriting the last two base64url characters of the signature. A 256 byte RS256 signature encodes to 342 characters where the final character carries only 2 real bits, and a 32 byte HS256 signature to 43 characters where the final one carries 2 bits. When the second to last character was already a and the last character shared its leading bits with a, the rewrite only changed padding bits and the decoded signature was identical, so jwt.decode verified it and the test got a valid token instead of SessionBadSignature. Measured on 3000 minted tokens: 14 RS256 and 4 HS256 tokens decoded to the same bytes after the old tamper. With several hundred proxy-infra jobs a day that matches the observed rate

Fix: _corrupt_signature base64url-decodes the signature, XORs the low bit of its first byte, and re-encodes it. Every call now changes the bytes the verifier checks. Mutation check: setting verify_signature: False in open_session_token makes all four tamper tests fail, reverting restores them

tests/test_litellm/proxy/client/cli/autoroute/test_wizard.py::TestFuzzyPickWidget::test_single_select_filters_and_returns_highlighted_match, 2 jobs

• run 33541414371 attempt 1, job 99968313038 proxy-auth, staging 720c1fa
• run 33562722747 attempt 1, job 100038764876 proxy-auth, staging b52b5d9

Mechanism (wall-clock dependence): _drive_fuzzy_pick slept 50ms after starting the prompt in a worker thread, then sent each key and slept a fixed 0.3s or 0.1s before the next one. InquirerPy does not filter synchronously on the keystroke: _on_text_changed schedules _filter_choices, which sleeps a 50ms debounce (_calculate_wait_time for 20 choices) and then awaits the async fuzzy_match before the callback moves the highlight. Enter is handled synchronously and accepts whatever is highlighted at that instant. If the debounce plus filter plus callback take longer than the 0.3s gap, Enter accepts the default model-0. Probing the real widget with the old driver locally shows the outcome is decided purely by that gap: a 0.05s gap returns model-0 5 of 5 times, a 0.1s gap or more returns model-13 5 of 5 times. In CI the failing attempt took about 2.1s versus 0.45s for the passing rerun on the same worker, so the worker was running 4 to 5 times slower than normal at that moment and the filter overran the 0.3s gap

Fix: each key event names the choice the widget must have highlighted before the next key is sent. The driver reads the live InquirerPyFuzzyControl.selection from the app session and polls it (10ms, 5s cap) instead of sleeping. Mutation checks: flipping multiselect in _fuzzy_pick and uppercasing the choice values both make the three widget tests fail, reverting restores them

2026-09-02

test_rotation_window_still_enforces_expiry_and_tamper_on_the_previous_key, 2 more jobs, same mechanism and fix as above

• run 33577643641 attempt 1, job 100085001786 proxy-infra, staging 48dd06e
• run 33589323758 attempt 1, job 100119902936 proxy-infra, commit c071329

2026-09-03

Window: 2026-09-02 09:17 UTC to 2026-09-03 09:17 UTC. The JWT and fuzzy picker tests above kept flaking on branches that do not carry this PR yet (test_resolve_fails_tampered_token_closed_without_expiry_flag 2 jobs, TestFuzzyPickWidget::test_multiselect_can_pick_more_than_one_across_filters 2 jobs, test_single_select_filters_and_returns_highlighted_match 1 job), same mechanism and fix as above. Three new flakes were fixed

Streaming guardrail block tests

tests/test_litellm/proxy/guardrails/guardrail_hooks/unified_guardrails/test_openai_streaming_block.py (test_chat_pre_stream_block_emits_standalone_completion, test_chat_mid_stream_block_continues_the_completion, test_chat_end_of_stream_block_terminates_cleanly), test_anthropic_streaming_block.py (test_mid_stream_block_emits_clean_anthropic_sse, test_mid_stream_block_after_prior_chunks_continues_message, test_end_of_stream_block_emits_clean_anthropic_sse, test_end_of_stream_only_block_does_not_append_after_message_stop) and test_streaming_buffer_until_moderated.py::test_buffered_block_withholds_original_content, 4 jobs. Every failure was no block SSE chunks were emitted and every job failed the whole group together, on both attempts inside the job, then passed on the next attempt of the same run or in the sibling job of the same run

• run 33725266627 commit 3794ba9: attempt 1 job 100552738397 proxy-endpoints failed the three OpenAI tests, attempt 2 job 100556314016 passed
• run 33716535482 commit 7804574: attempt 1 job 100526789355 proxy-endpoints failed the four Anthropic tests plus the buffer test, attempt 2 job 100528875321 passed
• run 33705504391 commit e03eb69: the Python 3.10 job 100493634924 failed the four Anthropic tests while the 3.11, 3.13, 3.14 and default proxy-endpoints jobs on the same commit passed them
• run 33718670728 staging 66a3d24, attempt 1 job 100533114611 failed the three OpenAI tests; every other staging run in the window passed them

The same mechanism also hit tests/test_litellm/proxy/guardrails/guardrail_hooks/openai/test_openai_moderation_streaming.py (all five tests, Expected 2 moderation calls ... got 0) in run 33728475666 job 100563167873 on litellm_heuristic_v2_license commit c9a6c11, whose neighbouring commits 43c56ab and a6ede47 on the same branch passed the same job with no guardrail files changed between them

Mechanism (shared module-level state between tests, made visible by test ordering): unified_guardrail.py keeps its call type to translation handler table in the module global endpoint_guardrail_translation_mappings, lazily filled by load_guardrail_translation_mappings() when it is None. test_unified_guardrail.py had a module-level autouse fixture that installed a reduced table and reset the global to None on teardown, while TestStreamingScanDedup swapped in the full table through monkeypatch.setattr. Because monkeypatch is set up before the module fixture and torn down after it, the module fixture set the global to None and then monkeypatch's undo put the reduced table back. Once a worker had run that class, the reduced table stayed installed for the rest of the process, and any later module on the same worker that streams through the unified guardrail found no chat handler and emitted no block chunks. The proxy-endpoints shard runs -n 4 --dist=loadscope, so the outcome depended on which modules landed on which worker

Fix: the 2026-09-03 run rewrote _use_real_mappings to assign the full table directly and reset the global on teardown. Staging has since landed the same repair in a different shape (_patch_translation_mappings(monkeypatch, mappings) used by every fixture in the module), so after merging staging on 2026-09-04 this PR no longer carries a diff in that file. The mutation checks from that run still hold on the tip

Tag routing tests

tests/test_litellm/router_strategy/test_router_tag_routing.py::test_negation_regex_pattern_treated_as_literal, 3 jobs

• run 33686765342 attempt 1, job 100436175129 enterprise-routing (Python 3.10), commit cfb0b59
• run 33691663975 attempt 1, job 100451513248 enterprise-routing, commit 09baa93
• run 33706398863 attempt 1, job 100496319966 enterprise-routing, commit ada4ce9

tests/test_litellm/router_strategy/test_router_tag_routing.py::test_chain_enable_tag_filtering_false_overrides_router_level_true, 1 job

• run 33688465979 attempt 1, job 100441611383 enterprise-routing, commit 4c2d537

Mechanism (unseeded randomness): both tests build a router with two deployments and no routing strategy, so simple_shuffle picks with random.choice. They sent ten requests and asserted the set of picked model ids equalled both deployments. The chance that ten fair picks all land on the same deployment is 2 in 1024, about 0.2 percent per test per job, which matches four flakes across the several hundred enterprise-routing jobs in the window

Fix: the tests keep routing real acompletion calls (mocked response, real router path) until both deployment ids have been seen, capped at 100 picks, and assert the seen set equals exactly the two deployments. A disallowed deployment fails the equality, a never-seen one fails at the cap, and the chance that 100 fair picks never land on one of two deployments is 2 in 2^100. An earlier revision called get_deployments_for_tag directly, which Greptile pointed out skipped the router's own routing path, so the tests now go through it. Mutation checks: making the negation branch match the tag as a regex fails test_negation_regex_pattern_treated_as_literal (every deployment is excluded and the router raises Not allowed to access model due to tags configuration), and ignoring the deployment-level override in _chain_tag_filtering_override fails test_chain_enable_tag_filtering_false_overrides_router_level_true, reverting restores both

Liveliness latency test

tests/test_litellm/proxy/health_endpoints/test_health_endpoints.py::test_health_liveliness_endpoint, 3 jobs

• run 33697474979 attempt 1, job 100469384247 proxy-endpoints, commit 95da998
• run 33706134311 attempt 2, job 100499937678 proxy-endpoints, commit f0f5f1e
• run 33718977024 attempt 1, job 100534022560 proxy-endpoints, commit 30d6516

Mechanism (wall-clock dependence): the test sent one request to /health/liveliness through a freshly built TestClient app and asserted it took under 100ms. The first request through a fresh Starlette app pays one-time costs (router compilation, middleware chain, anyio portal start, JSON encoder warm up) that later requests do not: locally the first request takes 11 to 39ms and requests two to five take about 1ms. On a proxy-endpoints shard running four xdist workers, with the worker also spinning up the app, that first request crossed 100ms

Fix: the test makes one warm-up request (asserting 200), then times five polls, asserts every poll returned 200 with "I'm alive!", and asserts the median of the five stayed under 100ms (a Greptile round asked for more than the fastest poll; the median still rides out a starved CI worker on two of five polls while failing on an endpoint that is slow on most). Mutation check: adding await asyncio.sleep(0.15) to health_liveliness fails the test, reverting restores it

2026-09-04

Window: 2026-09-03 09:17 UTC to 2026-09-04 09:17 UTC, 9904 Run tests job logs across 726 responses-caching-types, 650 misc and the other shards. Tests already carried by this PR kept flaking on branches that do not have it yet, same mechanism and fix as above: test_rotation_window_still_enforces_expiry_and_tamper_on_the_previous_key 3 jobs, test_health_liveliness_endpoint 3 jobs, test_negation_regex_pattern_treated_as_literal 2 jobs, TestFuzzyPickWidget::test_single_select_filters_and_returns_highlighted_match 2 jobs, TestFuzzyPickWidget::test_multiselect_can_pick_more_than_one_across_filters 2 jobs, test_rs256_tampered_signature_is_bad_signature 2 jobs, and one job each for test_multiselect_requires_tab_to_toggle_before_enter, test_resolve_fails_tampered_token_closed_without_expiry_flag, test_chain_enable_tag_filtering_false_overrides_router_level_true and test_tampered_signature_is_bad_signature. Two new flakes were investigated: the pre-commit one is fixed here, the Redis one landed on staging first

Redis loop-stall burst test

tests/test_litellm/caching/test_redis_cache.py::test_event_loop_stall_timeout_burst_keeps_breaker_closed, 32 jobs, all in the eleven hours after the test landed on staging in #38999 (942a46f, 2026-09-03 22:18 UTC), so roughly one in ten of the responses-caching-types jobs that ran it. First six:

• run 33816992136 attempt 1, job 100851212303 responses-caching-types, staging a53c550
• run 33826161322 attempt 1, job 100879167497 responses-caching-types, commit 82b325d
• run 33819699444 attempt 1, job 100859485552 responses-caching-types, commit 1481115
• run 33817643840 attempt 1, job 100853201642 responses-caching-types, commit d25cbdd
• run 33814735785 attempt 1, job 100844440062 responses-caching-types, commit c1bd39a
• run 33825573014 attempt 1, job 100877504992 responses-caching-types, commit 3377919

Mechanism (event-loop scheduling race, wall-clock dependence): the test gathered eight fake Redis calls, each asyncio.wait_for(asyncio.sleep(0.001), timeout=0.05), together with a task that did await asyncio.sleep(0) and then blocked the loop with time.sleep(0.2). It then asserted at least three of the eight calls timed out. That only holds if the loop gets from scheduling the eight sleep(0.001) timers to running the blocking task in under 1ms. When the loop iteration takes longer (a GC pause, four xdist workers sharing a two-core runner), the 1ms timers fire first, the eight calls return ok before the stall starts, there are zero timeouts, and the assertion fails. Reproduced locally by adding one 2ms delay to the loop iteration before the stall: 8 timeouts with no delay, 0 timeouts with the delay, see Proof of Fix

Fix: not carried by this PR any more. Staging fixed the same test in 7d3b03d (test(caching): drive the redis stall burst off the clock, not asyncio.wait_for): the fake call checks a monotonic client deadline and raises the redis TimeoutError itself instead of racing a 1ms sleep inside asyncio.wait_for, so a slow loop iteration cannot make the calls finish before the stall. The event-based rewrite this PR carried was dropped for staging's version when staging was merged in (822c862), so test_redis_cache.py is no longer part of this PR

Pre-commit interrupt test, and a real leak in scripts/pre_commit_lint.sh

tests/test_litellm/test_pre_commit_lint.py::test_interrupt_kills_background_jobs_and_removes_logs, 4 jobs (plus 1 job in the 2026-09-03 window that was left alone then)

• run 33761716750 attempt 1, job 100669406966 misc, commit f54a4c5
• run 33782213387 attempt 1, job 100738381147 misc, commit 466b44e
• run 33797980895 attempt 1, job 100790198664 misc, commit b38516d
• run 33743028798 attempt 1, job 100609098382 misc, commit 3abed5f

Mechanism (temp files plus a process race, and the flake exposed a real bug in the script): the test starts the real scripts/pre_commit_lint.sh with stubbed make, npx, npm and uv, waits for the Python lint stub to report it is running, sends SIGINT to the process group, and asserts the exit code, that the hung make is gone, and that $TMPDIR is empty. on_interrupt in the script removes the three job log files it knows about and kills the job process groups. But lint_dashboard runs in a background subshell and creates a fourth temp file, report=$(mktemp), for the whole-folder eslint JSON output, and only removed it on the happy path after check-lint-budgets.mjs. If SIGINT lands while npx eslint . -f json is still running, the subshell is killed by on_interrupt, the report file stays behind, and list(tmp_dir.iterdir()) == [] fails. Locally the stubs finish in a few milliseconds so the window is almost never hit; slowing the eslint stub by one second reproduces it every time (Proof of Fix). On a busy misc shard the dashboard job is slower than the Python job often enough to hit it four times a day. Outside the test, this is the same leak a developer hits when they Ctrl-C make check during the dashboard lint

Fix: the lint_dashboard subshell sets trap 'exit 143' TERM and trap 'rm -f "${report:-}"' EXIT, so the report is removed on the happy path, on failure and on interrupt, and the explicit rm -f "$report" at the end goes away. The test now also makes the npx eslint . -f json stub hang like make does when STUB_HANG_DIR is set, and waits for it to start before sending SIGINT, so the interrupt always lands inside the window that flaked, and the cleanup assertion waits on the condition (5s cap) instead of sampling the directory once, since on_interrupt does not wait for the job process groups it kills. Mutation checks: reverting the script's trap change fails the new test every time with the leftover tmp.* file in the assertion message, and replacing the rm -f of the three job logs in on_interrupt with a no-op fails it too, reverting restores both

Not fixed here

Real breakages (failed on every attempt in every job that ran them, so not flakes): tests/litellm-proxy-extras/test_litellm_proxy_extras_utils.py::TestMigrateDeployAttemptAccounting::* (four tests, 100 jobs across 40 branches, every branch that has not merged the proxy-extras fix yet), tests/test_litellm/router_strategy/test_complexity_router.py::test_every_routing_decision_field_is_classified (6 jobs on litellm_auto_setup), tests/test_litellm/test_claude_fable_5_config.py::* and tests/test_litellm/test_utils.py::test_aaamodel_prices_and_context_window_json_is_valid (5 jobs each on litellm_opencode_provider and litellm_registry_audit_2026_09_02), plus single-branch failures in test_spend_counter_reseed.py, test_cloudzero.py, test_mcp_server.py::test_mcp_manager_merges_public_and_restricted_servers and test_router_model_cost_isolation.py. The SAP transformation tests on litellm_sap_* fail with NameError: name 'model_serializer' is not defined on every attempt

Not reproduced: tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py::TestSpendLogsPayload::test_spend_logs_payload_e2e rerun to pass in 3 jobs, all on litellm_internal_copy_38013 (run 33822663839, run 33821535262, run 33824187901, proxy-infra). That branch does not touch spend tracking, but 20 local repeats of the node id passed and the rerun hides the traceback, so the mechanism is not pinned down. Seen once each and left alone: test_streaming_helpers.py::test_run_thread_pings_while_the_assistants_run_is_still_silent, test_io_token_rate_limits.py::test_itpm_estimate_failure_reserves_minimal_not_full_limit, test_spend_log_error_logger.py::test_spend_log_error_includes_traceback_by_default, tests/local_testing/test_caching_handler.py::test_async_get_cache_defers_streaming_completion_hit_callbacks, tests/local_testing/test_unit_test_caching.py::test_add_namespace_to_cache_key, and test_health_endpoints.py::test_health_liveness_endpoint on a branch that renamed the liveliness test

Infrastructure: PR #39644 had a check fail then pass on the same SHA because the Rust native wheel smoke test (tests/test_litellm/rust_bridge/native_route_wheel_test.py) timed out against its local 127.0.0.1 server with a BrokenPipeError. That is a harness timeout, not a pytest node flake. Validate PR title flipped on #39300 and #39299

Rolling PR bookkeeping: this PR was the only open litellm_deflake_ PR at the start of the 2026-09-04 run, so nothing was absorbed or dropped. Staging was merged in before adding the new fixes; the merge took staging's _patch_translation_mappings version of the guardrail fixture over this PR's earlier version, see above

Screenshots / Proof of Fix

Nothing in the proxy runtime changed, so there is no proxy request to curl. Before shows each mechanism on the merge base, After shows it gone at the tip, using the same commands. The tag routing case runs the test body from a script because the trigger (ten coin flips landing the same way) has to be repeated to show up on a fast laptop

Before (c8635ec)

Ctrl-C during make check leaks the eslint report

The User Flow above, run for real in a worktree at the merge base: an unformatted layout.tsx staged, TMPDIR pointed at an empty directory, make check started in a pty, and ^C typed once npx eslint . -f json shows up in pgrep

$ git rev-parse --short HEAD
c8635ecc67
$ git diff --cached --name-only
ui/litellm-dashboard/src/app/layout.tsx
$ ls -A $TMPDIR
$ make check
...
bootstrap: done
./scripts/pre_commit_lint.sh
check: logging full output to .../.git/worktrees/qa_base/pre_commit_lint.log
^Cmake[1]: *** [Makefile:277: check-inner] Error 130
make: *** [Makefile:274: check] Error 130
$ pgrep -f 'eslint . -f json' || echo 'no eslint left running'
no eslint left running
$ ls -A $TMPDIR
tmp.eZ44AG4GIP

CI job 100669406966 log: RERUN ...test_interrupt_kills_background_jobs_and_removes_logs followed by PASSED for the same node id. The new regression test against the merge-base script: 1 failed, assert _wait_until(lambda: not any(tmp_dir.iterdir()), 5) with the tmp.* file in the message

Tag routing depends on ten random picks

1. Import the old test module and run test_negation_regex_pattern_treated_as_literal and test_chain_enable_tag_filtering_false_overrides_router_level_true 1000 times each, counting AssertionError
2. Observed: runs per test: 1000 failures: {'test_negation_regex_pattern_treated_as_literal': 6, 'test_chain_enable_tag_filtering_false_overrides_router_level_true': 5}

Liveliness test times the cold first request

1. Build the same TestClient the test uses and time five consecutive GET /health/liveliness calls
2. Observed: req 0: 22.52ms, req 1: 1.74ms, req 2: 1.32ms, req 3: 0.99ms, req 4: 1.14ms. Only the first request carries the startup cost the old test measured against a 100ms budget

After (03a8282)

Commits since 03a8282: 822c862 merges staging and only drops this PR's test_redis_cache.py rewrite for staging's 7d3b03d, leaving the other files byte-identical, and 5bd4da0 changes only the liveliness assertion (median instead of fastest), so that leg is re-run at 5bd4da0 below; 41c8969 rewrites the two tag routing tests only, so that leg is re-run at 41c8969 below, and the other runs stand; b6a3cba merges staging (#39848's integrations / Run tests fix) with no diff change

Ctrl-C during make check leaks the eslint report

Same steps in a worktree at the tip

$ git rev-parse --short HEAD
03a82823fc
$ git diff --cached --name-only
ui/litellm-dashboard/src/app/layout.tsx
$ ls -A $TMPDIR
$ make check
...
bootstrap: done
./scripts/pre_commit_lint.sh
check: logging full output to .../.git/worktrees/qa_tip/pre_commit_lint.log
^Cmake[1]: *** [Makefile:277: check-inner] Error 130
make: *** [Makefile:274: check] Error 130
$ pgrep -f 'eslint . -f json' || echo 'no eslint left running'
no eslint left running
$ ls -A $TMPDIR

The uninterrupted paths through the same subshell still report correctly at the tip: make check with the unformatted layout.tsx staged prints check: FAIL and exits 2, and with a prettier-clean edit staged prints check: PASS and exits 0, with ls -A $TMPDIR empty after both. 20 repeats of the node id: 0 failures, and tests/test_litellm/test_pre_commit_lint.py as a whole passes

Tag routing depends on ten random picks (re-run at 41c8969)

1. Same 1000 iteration loop against the new test bodies
2. Observed: runs per test: 1000 failures: {'test_negation_regex_pattern_treated_as_literal': 0, 'test_chain_enable_tag_filtering_false_overrides_router_level_true': 0}
3. 20 repeats of the two node ids at 41c8969: 0 failures

Liveliness test times the cold first request (re-run at 5bd4da0)

1. Same five timed requests; the new test warms up once and then scores the median of five polls
2. Observed: req 0: 23.70ms, req 1: 7.13ms, req 2: 4.25ms, req 3: 2.32ms, req 4: 2.59ms. The first request is no longer scored, and the median of the warm polls is 4.25ms
3. 20 repeats of the node id at 5bd4da0: 0 failures

The 2026-09-01 and 2026-09-03 fixes were re-validated at 03a8282: 20 repeats of test_session_token.py, test_session_credentials.py, test_wizard.py, the two tag routing node ids and the liveliness node id together, 0 failures. Since then only the liveliness test (5bd4da0) and the two tag routing tests (41c8969) changed, and both are re-run above

• b6a3cba passes /live-pr-risk (822c862, 5bd4da0 and 41c8969 change tests only, no runtime code; b6a3cba is a staging merge with no diff change, and the one CircleCI red is the staging-wide proxy_store_model_in_db_tests failure listed under Caveats)

Type

Test
Bug Fix

Caveats (if any)

Low

• The liveliness test now scores warm requests only; a slow cold start would not fail it
• pre_commit_lint.sh's interrupt handler still does not wait for the killed jobs, so the test waits up to 5s for cleanup
• test_health_backlog_includes_admission_control_stats fails locally with a 401 on staging too; this PR does not touch it
• Staging's dashboard lint is one over the no-large-inline-object-arg budget (556 > 555), so make check on the merge commit reports that pre-existing red; no dashboard file changes here
• proxy_store_model_in_db_tests is red at b6a3cba on test_chat_completion_bad_model_with_spend_logs (model_group comes back empty), the same single failure on every staging pipeline since 2026-09-04 12:09 (89087 through 89149); not a file this PR touches

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Changes are mostly test hardening plus a developer-script temp-file cleanup fix; no production proxy routing or auth logic is modified.

Overview
This PR tightens flaky unit tests and fixes a real cleanup bug in local make check dashboard linting.

scripts/pre_commit_lint.sh — The dashboard lint subshell now uses EXIT/TERM traps so the eslint JSON mktemp report is removed on success, failure, and Ctrl-C; previously interrupting during npx eslint . -f json could leave stray files in $TMPDIR.

Tests (no proxy runtime behavior changes):

• MCP session token tests use _corrupt_signature (decode → flip a signature bit → re-encode) so tamper cases always fail verification instead of occasionally matching valid JWTs when only base64 padding changed.
• Autoroute wizard fuzzy-picker tests wait until the expected choice is highlighted before sending the next key, replacing fixed sleeps that raced InquirerPy’s debounced filter on slow CI workers.
• Router tag-routing tests route via acompletion until both deployment IDs are seen (capped at 100 picks) instead of asserting after ten random shuffles.
• /health/liveliness latency test warms up once, then asserts median of five polls stays under 100ms rather than timing the cold first request.
• Pre-commit interrupt regression test stubs a hanging whole-repo eslint JSON step and polls until $TMPDIR is empty after SIGINT.

^{Reviewed by Cursor Bugbot for commit b6a3cba. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/41e8c32159854ba790cdd93622ac60ca
Open in Devin Desktop: https://app.devin.ai/desktop/session/41e8c32159854ba790cdd93622ac60ca?variant=devin
Requested by: @mateo-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

PR #39306 has no labels (missing required enterprise label), so it is out of scope. No GitHub or Linear changes made; no risk label applied.

[greptile-apps]

Greptile Summary

This PR hardens several flaky tests and ensures the dashboard lint report is removed when pre-commit linting is interrupted.

• Corrupts decoded JWT signature bytes deterministically in tamper tests.
• Synchronizes fuzzy-picker input with the widget’s selected state.
• Exercises tag routing through acompletion until both expected deployments are observed.
• Measures median warm liveliness latency rather than cold startup or the fastest request.
• Adds reliable interrupt coverage for dashboard lint temporary-file cleanup.

Confidence Score: 5/5

The PR appears safe to merge; no outstanding or newly introduced actionable failures were identified.

Both previous findings are manually resolved and fully addressed by the current code: liveliness now checks the median rather than the fastest poll, and tag-routing tests once again exercise router.acompletion. The remaining changes strengthen test determinism or correctly clean up the temporary eslint report on interruption.

Important Files Changed

Filename	Overview
scripts/pre_commit_lint.sh	Adds TERM and EXIT handling so the temporary eslint report is removed on completion, failure, or interruption.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_session_credentials.py	Makes the session-credential tamper test deterministically modify decoded signature bytes.
tests/test_litellm/proxy/_experimental/mcp_server/outbound_credentials/test_session_token.py	Replaces padding-sensitive JWT mutations with deterministic signature-byte corruption.
tests/test_litellm/proxy/client/cli/autoroute/test_wizard.py	Replaces fixed input delays with synchronization against the fuzzy control’s highlighted choice.
tests/test_litellm/proxy/health_endpoints/test_health_endpoints.py	Warms the client and checks representative steady-state latency using the median of five validated responses.
tests/test_litellm/router_strategy/test_router_tag_routing.py	Retains public routing-path coverage while making randomized deployment sampling effectively deterministic.
tests/test_litellm/test_pre_commit_lint.py	Forces interruption during eslint report generation and waits for asynchronous temporary-file cleanup.

_{Reviews (7): Last reviewed commit: "Merge branch 'litellm_internal_staging' ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b6a3cba. Configure here.}

[mateo-berri]

LGTM