Skip to content

Record the MQTT Broken pipe flake as accepted retry debt (GH-3763) - #3812

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3763/mqtt-broken-pipe-accepted-retry
Aug 4, 2026
Merged

Record the MQTT Broken pipe flake as accepted retry debt (GH-3763)#3812
jeremydmiller merged 1 commit into
mainfrom
gh-3763/mqtt-broken-pipe-accepted-retry

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Comments only; no behaviour change.

listen_with_topic_wildcards.broadcast and broadcast_to_topic_by_user_logic.route_by_derived_topics_2 each cost one retry in CIMQTT5, failing with the same error in the same run:

MQTTnet.Exceptions.MqttCommunicationException : Broken pipe
  ---- System.Net.Sockets.SocketException : Broken pipe

Neither is tagged Category=Flaky, deliberately. A tag would stop them running, and the trade this repo has been making all along is that a test which runs and occasionally retries is worth more than a test that runs nowhere. This is visible debt in the retry ledger rather than hidden debt in an exclusion list — the same trade already recorded on multi_tenancy_through_virtual_hosts.

What the note buys

The search that has already been done. Three candidates tested and eliminated — recorded so nobody repeats them:

candidate verdict
Teardown ordering — both classes stop the broker before the hosts still connected to it, and 7 of 12 broker-using classes share the shape Refuted. A faithful reproduction (sender + receiver, real tracked broadcast, broker-first teardown) throws nothing in 10 runs, either ordering
Port collision between worker processes via PortFinder's TOCTOU Refuted. MQTTnet throws SocketException: Address already in use, which fails the class at InitializeAsync rather than breaking a pipe mid-test
Local reproduction under concurrency No occurrence in six concurrent full-suite runs

Left standing is a keep-alive drop under CI load, which needs a call site. #3811 makes the ledger record one, so the next occurrence writes its stack into the test-ledger-CIMQTT5 artifact.

The note also says plainly what it costs — one retry, never a red main, two rounds of investigation already spent — so the next reader can weigh that before starting a third.

Also here

The note on multi_tenancy_through_virtual_hosts asked for a next step — "dump the tracked session on the FIRST attempt rather than infer from the assertion" — that has since been built (#3810) and now happens on its own. Amended with its first captured failure:

System.Exception : No messages of type MultiTenantResponse were received
Activity detected: | Service (Node Id) | Message Id | Message Type | ...

The request never produced its response, which is consistent with the fixed queue names already under suspicion there — without yet proving them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m

Comments only; no behaviour change.

listen_with_topic_wildcards.broadcast and
broadcast_to_topic_by_user_logic.route_by_derived_topics_2 each cost one retry
in CIMQTT5, failing with the same

    MQTTnet.Exceptions.MqttCommunicationException : Broken pipe
      ---- System.Net.Sockets.SocketException : Broken pipe

Neither is tagged Category=Flaky, deliberately. A tag would stop them running,
and the trade this repository has been making all along is that a test which
runs and occasionally retries is worth more than a test that runs nowhere. This
is visible debt in the retry ledger rather than hidden debt in an exclusion
list, which is the same trade already recorded on
multi_tenancy_through_virtual_hosts.

What the note buys is the search that has already been done. Three candidates
were tested and eliminated:

  - teardown ordering, which both classes get wrong in the same way and which
    7 of 12 broker-using classes share -- a faithful reproduction throws
    nothing in 10 runs, either ordering;
  - a port collision between worker processes via PortFinder's TOCTOU -- ruled
    out because MQTTnet throws "Address already in use", which fails the class
    at InitializeAsync rather than breaking a pipe mid-test;
  - local reproduction under concurrency -- six concurrent full-suite runs, no
    occurrence.

Left standing is a keep-alive drop under CI load, which needs a call site. PR
#3811 makes the ledger record one, so the next occurrence writes its stack into
the test-ledger-CIMQTT5 artifact.

Also amends the note on multi_tenancy_through_virtual_hosts, which asked for a
next step -- "dump the tracked session on the FIRST attempt" -- that has since
been built and now happens on its own. Its first captured failure shows the
request never produced its response, which is consistent with the fixed queue
names already under suspicion there without yet proving them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant