Skip to content

fix(tests): make FakeChatClient thread-safe to fix racy compaction test - #1579

Merged
Aaronontheweb merged 3 commits into
netclaw-dev:devfrom
Aaronontheweb:fix/racy-compaction-observer-list
Jul 5, 2026
Merged

fix(tests): make FakeChatClient thread-safe to fix racy compaction test#1579
Aaronontheweb merged 3 commits into
netclaw-dev:devfrom
Aaronontheweb:fix/racy-compaction-observer-list

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Problem

CompactionIntegrationTests.Successive_compactions_preserve_prior_summary_in_observer_system_prompt failed on ubuntu CI only (run 28690821775) — Windows and macOS passed, the classic racy-test signature:

System.InvalidOperationException : Collection was modified; enumeration operation may not execute.
  at System.Collections.Generic.List`1.Enumerator.MoveNext()
  at System.Linq...ToList()
  at CompactionIntegrationTests.cs:line 617

Root cause

FakeChatClient is a single shared instance whose GetResponseAsync is invoked concurrently from multiple actor dispatcher / ThreadPool threads — the main-model turn, the compaction summarizer sidecar, and a fire-and-forget memory-extraction sidecar that LlmSessionActor launches one line after it emits CompactionOutput (LlmSessionActor.cs:1273). It appended to three plain List<T> fields (ReceivedMessages / ReceivedOptions / ReceivedToolNames) with no synchronization, while the test thread enumerated ReceivedMessages via .Select().Where().ToList() at line 617. Only _callCount was Interlocked-guarded — a partial fix that left the lists exposed.

The test's ExpectMsg<CompactionOutput> barrier proves the summarizer call landed but does not quiesce the collection, so the enumeration raced a concurrent Add. Ubuntu-only reproduction is scheduling sensitivity — the bug is platform-independent.

Fix

Back the three recording collections with private lists and expose them as IReadOnlyList<T> getters that return a snapshot copy taken under a single _recordingGate lock; guard the appends (and the PlannedExceptions check) under the same lock. The lock is never held across an await.

Snapshot-on-read is load-bearing — locking the writes alone would not stop a caller's List<T> enumeration from throwing when an Add interleaves its MoveNext. The racing writer (memory-extraction, "memory extraction assistant" prompt) is filtered out of the assertion's .Where(Contains("session summarizer")) predicate, and the two summarizer calls the test inspects are ordered-before their CompactionOutput via the mailbox — so a thread-safe fake alone makes the test deterministic without adding an actor-side barrier.

Scope

Tightly scoped to the three recording collections + PlannedExceptions (the one field every call path touches). The deeper queue dequeues (PlannedToolCallDecisions / PlannedUsageOverrides / PlannedResponses) share the pattern but aren't reached concurrently in the current suite (sidecars branch earlier; one session's main-model calls are actor-serialized), so they were left alone to avoid widening the blast radius.

Follow-up worth filing: CompactionOutput isn't a "compaction fully settled" barrier because memory extraction fires fire-and-forget afterward with no completion signal. Fine for this test, but any future test asserting on the extraction call itself would need a real ack.

Verification (on Linux — the platform that reproduced it)

  • ✅ Build: 0 warnings / 0 errors (private lists proved no external mutator; all ~10 caller files compile against the IReadOnlyList shape)
  • ✅ Target test: 25/25
  • ✅ Full Netclaw.Actors.Tests: 2531 passed, 0 failed (was 2530 / 1-failed in CI)
  • dotnet slopwatch analyze: 0 issues

🤖 Analysis by akka-net-specialist + dotnet-concurrency-specialist via /analyze-racy-test.

CompactionIntegrationTests.Successive_compactions_preserve_prior_summary_in_observer_system_prompt
failed on ubuntu CI (run 28690821775) with:

  System.InvalidOperationException : Collection was modified;
  enumeration operation may not execute.
    at CompactionIntegrationTests.cs:617

FakeChatClient is a single shared instance whose GetResponseAsync is
invoked concurrently from multiple actor dispatcher / ThreadPool threads
(main-model turn + compaction summarizer sidecar + fire-and-forget
memory-extraction sidecar launched one line after CompactionOutput is
emitted). It appended to three plain List<T> fields — ReceivedMessages /
ReceivedOptions / ReceivedToolNames — with no synchronization, while the
test thread enumerated ReceivedMessages via .Select().Where().ToList().
Only _callCount was Interlocked-guarded — a partial fix that left the
lists exposed. The test's ExpectMsg<CompactionOutput> barrier does not
quiesce the collection, so the enumeration raced a concurrent Add.

Fix: back the three recording collections with private lists and expose
them as IReadOnlyList<T> getters that return a snapshot copy taken under
a single _recordingGate lock; guard the appends (and the PlannedExceptions
check) under the same lock. Snapshot-on-read is load-bearing — locking the
writes alone would not stop a caller's List<T> enumeration from throwing
when an Add interleaves its MoveNext. The lock is never held across an
await. The racing writer (memory-extraction, "memory extraction assistant"
prompt) is filtered out of the assertion, and the summarizer calls the
test inspects are ordered-before their CompactionOutput, so a thread-safe
fake alone makes the test deterministic without an added actor barrier.

Verified on Linux: build clean (0 warnings), target test 25/25,
full Netclaw.Actors.Tests suite 2531 passed / 0 failed, slopwatch clean.
@Aaronontheweb Aaronontheweb added tests All issues related to testing, quality assurance, and smoke testing. bug Something isn't working labels Jul 5, 2026
@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) July 5, 2026 14:54
@Aaronontheweb
Aaronontheweb merged commit 7efa247 into netclaw-dev:dev Jul 5, 2026
15 checks passed
Aaronontheweb added a commit that referenced this pull request Jul 5, 2026
… capability-provenance logging #1584, FakeChatClient thread-safety #1579)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working tests All issues related to testing, quality assurance, and smoke testing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant