Skip to content

test(gateway): de-flake concurrent-compression lock test with a barrier - #41971

Merged
teknium1 merged 1 commit into
mainfrom
fix/compression-concurrency-test-flake
Jun 8, 2026
Merged

test(gateway): de-flake concurrent-compression lock test with a barrier#41971
teknium1 merged 1 commit into
mainfrom
fix/compression-concurrency-test-flake

Conversation

@teknium1

@teknium1 teknium1 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

tests/gateway/test_compression_concurrent_sessions.py::test_concurrent_compressions_same_session_serialize no longer flakes under CI load. It was failing on shard test (1) with "Expected exactly one agent to compress, got 2" — including on main HEAD, independent of any feature PR.

Root cause (test-only, not a lock bug): the test relied on a time.sleep(0.25) inside the stubbed compressor to make the two threads overlap inside the per-session lock window. Under CI CPU starvation that sleep is insufficient — one thread can acquire → compress → rotate → release the lock before the other thread even reaches try_acquire. Both then acquire on the shared session_id and both compress. The real per-session lock (#34351) is correct; the test's overlap was a timing accident.

Changes

  • tests/gateway/test_compression_concurrent_sessions.py: wrap the shared db's try_acquire_compression_lock in a threading.Barrier(2) so both threads rendezvous immediately before the real atomic acquire — guaranteeing genuine simultaneous contention regardless of scheduling. The real lock logic is untouched; it still picks exactly one winner. The wrapper is restored after join() so the post-join lock-leak assertion hits the unwrapped method.

Validation

Old (sleep-overlap) New (barrier)
20× plain runs flaked in CI 20/20 pass
15× under all-core CPU stress (load avg ~4.6) reproduced the race 15/15 pass
sibling test (5-session aliasing) pass pass
canonical isolated runner 2/2 2/2

CPU-stress reproduction is the key check: the old version's failure mode (one thread finishing before the other starts) is now structurally impossible.

Infographic

de-flaked concurrent compression test

test_concurrent_compressions_same_session_serialize relied on a
time.sleep(0.25) inside the stubbed compressor to make the two threads
overlap inside the per-session lock window. Under CI CPU starvation that
sleep is insufficient: one thread can acquire -> compress -> rotate ->
RELEASE the lock before the other reaches try_acquire, so both acquire on
the shared session_id and both compress (the recurring 'Expected exactly
one agent to compress, got 2' failure on shard test (1)).

Replace the timing dependency with a threading.Barrier(2) wrapped around
the shared db's try_acquire_compression_lock: both threads rendezvous
immediately before the real (atomic) acquire, guaranteeing genuine
simultaneous contention regardless of scheduling. The real lock logic is
unchanged and still picks exactly one winner — this only fixes the test's
overlap guarantee. Restored after join so the post-join lock-leak
assertion hits the unwrapped method.

Verified: 20/20 plain + 15/15 under all-core CPU stress (load avg ~4.6),
where the old version flaked.
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: fix/compression-concurrency-test-flake vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 10394 on HEAD, 10392 on base (🆕 +2)

🆕 New issues (2):

Rule Count
invalid-assignment 2
First entries
tests/gateway/test_compression_concurrent_sessions.py:192: [invalid-assignment] invalid-assignment: Object of type `bound method SessionDB.try_acquire_compression_lock(session_id: str, holder: str, ttl_seconds: int | float = ...) -> bool` is not assignable to attribute `try_acquire_compression_lock` of type `def try_acquire_compression_lock(self, session_id: str, holder: str, ttl_seconds: int | float = ...) -> bool`
tests/gateway/test_compression_concurrent_sessions.py:171: [invalid-assignment] invalid-assignment: Object of type `def _barriered_acquire(...) -> Unknown` is not assignable to attribute `try_acquire_compression_lock` of type `def try_acquire_compression_lock(self, session_id: str, holder: str, ttl_seconds: int | float = ...) -> bool`

✅ Fixed issues: none

Unchanged: 5425 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/test Test coverage or test infrastructure comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have labels Jun 8, 2026
@teknium1
teknium1 merged commit 400e6e4 into main Jun 8, 2026
23 checks passed
@teknium1
teknium1 deleted the fix/compression-concurrency-test-flake branch June 8, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants