test(callbacks): harden flaky proxy callback-leak detector

[yuneng-berri]

What

tests/test_callbacks_on_proxy.py::test_check_num_callbacks_on_lowest_latency (and its sibling test_check_num_callbacks) was flaky in CI — e.g. assert abs(num_callbacks_1 - num_callbacks_2) <= 4 failing as assert 10 <= 4 (85 → 95).

Root cause

These tests assert on a process-global callback count (sum of 7 litellm.* callback lists) read from /active/callbacks, on a proxy that the build_and_test job shares across 4 parallel xdist workers. The 85 → 95 jump is a bounded, one-time registration the test triggers on itself: switching the router to latency-based-routing via /config/update registers the latency-strategy handler and (from the same payload) Slack alerting. Diagnostic sampling showed the count steps once and is then flat at 95 for 2.5 min under full parallel load — not a leak. The old fixed ±4 threshold over a 30s window was flaky purely because the two samples sometimes straddled that self-inflicted step.

Change

Rework both tests into a robust leak detector that keeps the original intent (no unbounded callback growth):

• Settle past the deliberate /config/update registration before baselining.
• Sample N times; flag only sustained, monotonic per-type growth (a real leak), not one-time pollution from other workers on the shared proxy.
• Normalize callback str() (strip instance addresses) so N leaked instances collapse to one rising counter; name the offending type in the failure message.
• @pytest.mark.flaky(reruns=2) for residual transient tolerance.

No production code changed.

Test plan

build_and_test job, commit 913c722202, 3 independent runs (pipelines 77855 / 77887 / 77892) — all green:

Run	test_check_num_callbacks	test_check_num_callbacks_on_lowest_latency
77855	✅ 124.5s	✅ 120.7s
77887	✅ 126.1s	✅ 120.3s
77892	✅ 126.6s	✅ 120.6s

Detector logic additionally unit-verified locally against the original failure series (correctly classified as no-leak) and against a synthetic sustained climb (correctly flagged).

[greptile-apps]

Greptile Summary

This PR hardens two previously flaky proxy callback-leak detector tests by replacing a simple abs(n1 - n2) <= 4 assertion with a multi-sample, per-type monotonic-growth detector that tolerates bounded one-time registrations from parallel xdist workers.

• Introduces _detect_leaks (sustained monotonic growth across ≥ 2 intervals, net ≥ 5) and _terminal_suspects / _detect_leaks_confirmed (an extra confirmation sample for burst-at-end growth), directly addressing the two blind spots noted in prior review threads.
• Adds @pytest.mark.flaky(reruns=2) and a finally block to ensure routing strategy is restored even when assertions fail.
• Normalizes callback str() representations to collapse multiple instances of the same type into one rising counter, producing named failure messages instead of opaque counts.

Confidence Score: 5/5

Test-only change; no production code touched. The new leak detector is strictly more precise than the old fixed-delta check and correctly handles the one-time registration step that caused CI flakiness.

The rework is carefully reasoned and well-documented. The _terminal_suspects confirmation path directly closes the burst-at-end blind spot from prior review. The only rough edge is a single hardcoded magic number (30) that should use the SETTLE_SECONDS constant, but this has no functional impact since the values are equal.

No files require special attention beyond the minor constant inconsistency in test_check_num_callbacks_on_lowest_latency.

Important Files Changed

Filename	Overview
tests/test_callbacks_on_proxy.py	Replaces the fragile fixed-delta assertion with a robust multi-sample monotonic-growth leak detector; adds _terminal_suspects + _detect_leaks_confirmed to handle burst-at-end growth. One minor inconsistency: hardcoded 30 vs SETTLE_SECONDS in test_check_num_callbacks_on_lowest_latency.

_{Reviews (2): Last reviewed commit: "test(callbacks): address review - drop r..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptile

[mateo-berri]

LGTM