fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops - #6090
Open
mfethe1 wants to merge 1 commit into
Open
fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops#6090mfethe1 wants to merge 1 commit into
mfethe1 wants to merge 1 commit into
Conversation
…re drops
When a subscriber's outbound data channel is full, EVENT fan-out frames
are dropped silently: the socket stays up, the grace counter increments
server-side only, and the client's live view keeps a permanent hole
until the grace limit kills the connection (or a reconnect replay
accidentally heals it). Lane 3 of the sync-lag diagnosis proved this is
the first broken boundary behind "messages arrive late until restart".
Give the client a machine-readable gap signal instead:
- RelayMessage::sync_required() formats the extension frame
["BUZZ_SYNC_REQUIRED","backpressure"]. The constructor is deliberately
monomorphic so the relay cannot emit a frame the wire contract does
not define; protocol.rs module docs define the contract and delivery
rules (unknown relay->client heads are non-fatal per NIP-01 client
convention).
- The EVENT fan-out path — ConnectionManager::send_fanout_frame,
renamed from send_to_text_bytes to make its role explicit — routes
through try_send_fanout_frame:
* every dropped frame (full data channel, closed data channel, or
connection already gone) increments buzz_fanout_dropped_frames_total
with no sampling, so a silently lost EVENT always leaves a
telemetry trail;
* on a Full data channel with a live socket, the gap signal is queued
on the connection's priority control channel — never on the data
channel it signals about, never as a human-readable NOTICE — and
the existing consecutive-full grace counter decides cancellation
exactly as before;
* when the control channel is itself full or closed, in-band
signaling is impossible: cancel at once and let reconnect replay
recover the missed events;
* cancellation is idempotent under concurrent drops: a
compare-exchange on ConnEntry::backpressure_disconnect_counted
picks exactly one of the parallel droppers to increment
buzz_ws_backpressure_disconnects_total (a runtime flood harness
showed the unguarded is_cancelled->cancel sequence inflated the
disconnect counter 46x for one connection).
- Non-fan-out sends (CLOSED notices etc.) share the try_data_send
success path but keep the old drop semantics — no gap signal, no
dropped-frame counting.
Tests: 12 new tests plus a wire-shape pin (exact bytes) in the protocol
table test — full/closed/gone/ctrl-full branches, grace-counter
interaction, the signal never riding the data channel, disconnect
idempotence (including an 8-thread concurrent-drop case), and an
end-to-end case through the real send_fanout_frames path.
Gates on this Windows host: cargo fmt --check clean; cargo clippy
-p buzz-relay --tests --all-features -- -D warnings clean; cargo test
-p buzz-relay --lib = 885 passed / 43 ignored, failing only on 3
pre-existing host failures that reproduce on the base commit f956e6f
untouched by this diff (2 bash-HMAC tests resolve `bash` to WSL on
this host; the redis-gated mesh_demo test returns 504 with local
Redis up).
Co-authored-by: Michael Feth <michael@jira-flow.com>
Signed-off-by: Michael Feth <michael@jira-flow.com>
Author
|
Hi @themiguelamador — could you approve the pending CI workflow runs on this PR at head Context on why the CI run matters beyond the usual signal: both dev hosts available to us are Windows, where several workspace gates fail at base for platform reasons. We therefore verified with targeted crate gates at the exact head on two independent Windows hosts (full evidence table in the PR description): Per the repo's |
mfethe1
added a commit
to mfethe1/buzz
that referenced
this pull request
Sep 2, 2026
… product/main Signed-off-by: Michael Fethe <mfethe1@gmail.com>
loganj
added a commit
that referenced
this pull request
Sep 8, 2026
🤖 ## Summary When a Buzz agent falls behind on incoming messages, its connection can make the backlog worse while trying to recover. The connection buffers messages from the relay server until the agent is ready to process them; if that buffer overflows, recovery previously requested history for **every subscribed channel** and paused socket reads while sending those requests. That adds traffic to an already overloaded connection. This change requests history only for affected subscriptions, once the code consuming those messages has room, with at least five seconds between attempts. The recovery path now: - Combines repeated losses into one pending recovery per affected subscription, keeping the oldest dropped timestamp so replay starts early enough. - Waits until at least half the consumer queue is free and the relay's existing rate-limit delay has expired. The queue wakes recovery when space becomes available; recovery does not periodically sample capacity or hold queue space away from live messages. - Attempts one subscription at a time, choosing the least recently attempted so a busy channel cannot crowd out other channels or membership notifications. The five-second delay starts when an attempt finishes, including a failed write; failed writes leave recovery pending. Recovery is paced by available capacity, not by how often messages are lost. This is not a larger buffer or a cutoff that abandons recovery. Subscription identifiers, message filters, replay timestamp overlap and duplicate filtering are unchanged; no downstream agent changes are required. This targets a reproducible overload **amplifier**, not every cause of overload or every catch-up limitation. The initial live overload's cause has not been established. Recovery remains best effort: a successful request write is not proof of delivery, and existing history/retention limits, bounded duplicate tracking and replay limitations still apply. There is no exactly-once or complete catch-up guarantee. A stalled write can still pause socket reads for the existing ten-second timeout; the pacing bound does not cover initial subscriptions, reconnects or other retry paths. ### Related issue Closest related: #5014 (channel re-subscription); also #6661 (membership reconciliation) and #6090 (relay backpressure gap signaling). This addresses local overflow recovery scheduling, not those separate mechanisms. ### Testing Recorded offline comparisons against the previous behavior, with the final implementation at `8000636f3073167c5a5107bb179c7d91160f1729`: | Same fixture: 18 subscriptions, three overload rounds | Before | After | | --- | --- | --- | | Recovery history requests | 108 | 3 | | Ping-response delay | About 4.6 seconds | Below the measurement's 1 ms resolution | A separate bounded-history fixture delivered all 320 events plus subsequent live traffic in **both** versions. Regression coverage exercises the real socket-handling task, including intermittent consumer capacity, fairness, failed writes and cancellation of capacity waits before live delivery. These are synthetic results, not production throughput measurements or evidence of a deployed cure. The full local `RUST_TEST_THREADS=4 just ci` run passed on September 4, 2026. Earlier unsuccessful local runs remain part of the validation history. The [recorded validation evidence and separate desktop follow-up](#7325 (comment)) preserve the original desktop mock-history scroll failure, its passing rerun and the remaining investigation. That desktop path does not run the agent connection code; neither this repair nor the passing rerun fixes the observed scroll problem. --------- Signed-off-by: Logan Johnson <loganj@squareup.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Relay-side fix for the sync-lag diagnosis finding that the first broken boundary in relay->client sync is silent fan-out frame drops under backpressure. When the priority ctrl channel's fan-out buffer is full, the relay now emits a gap signal to the affected connection instead of dropping frames invisibly, letting clients know their view is stale.
Contract (frozen)
["BUZZ_SYNC_REQUIRED","backpressure"]on the priority ctrl channel.TrySendError::Fullfrom the fan-out path). A closed ctrl channel cancels immediately; no signal into a closed channel.sync_required(reason)is monomorphic overSyncRequiredReason::Backpressure— the relay cannot emit a reason the contract does not define.buzz_fanout_dropped_frames_totalcounts every drop unsampled.once_disconnect_counted) so concurrent disconnect paths count once per connection (8-thread regression test).protocol.rsmodule docs beside theRelayMessagedefinitions.Diff
3 files / 736 ins / 27 del, all under
crates/buzz-relay/(handlers/event.rs,protocol.rs,state.rs). Single commit on currentmain(f956e6fe), no rebase.Verification at exact head 0048e96
Two independent Windows (MSVC) hosts, separate gate stacks:
cargo test -p buzz-relay --lib --no-fail-fastcargo fmt --all -- --checkcargo clippy -p buzz-relay --tests --all-features -- -D warningsAll failures are pre-existing host-shaped issues reproducing on the
f956e6febase:api::git::policy::tests::bash_hmac_matches_rust_hmacandbash_hmac_single_ref— WSL bash resolution.api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo— redis-gated.telemetry::tests::trace_context_lookup_does_not_enable_callsites— passes in isolation (-- --exact); the test exists unchanged at base (telemetry.rs:475) and this diff touches zero telemetry code, so the flake is scheduler-shaped (global tracing callsite registration leaking between parallel tests), not diff-shaped. A Host A rerun reproduced the same intermittent flake.Per the repo's
ci.ymlpath-filter semantics, acrates/**diff triggersrust-lint/unit-tests/backend-integrationon ubuntu — maintainer-approved CI runs on this PR are the supported-environment complementary evidence.Attribution: commits carry
Co-authored-by+Signed-off-byfor the human operator; no AI attribution trailers.Evidence (contract 2026-08-17, tier T2 protocol/relay)
Locked-head exception applies: paired-gate receipts are locked at this exact head SHA, so this evidence lives in the body rather than a
docs/pr-evidence/6090/commit (a new commit would move the head and invalidate the receipts).Before — problem reproduced at base
f956e6fe(read-only diagnosis, 2026-08-16):try_send_ws_message(state.rs:592-620) onTrySendError::FullDROPS the frame with a server-side-onlywarn!;drop_countis never signaled to the client. Below the grace limit (default 15, config.rs:566-580) the socket stays "connected" with permanent holes in its event stream.relayReconnectReplay.ts, mobilerelay_session.dart:261-262).RESEARCH/SYNC_LAG_DIAGNOSIS_LANE3_HANDOFF_2026_08_16.md,.scratch/sync-lag-probe/probe_long.log,exp2.log(agent workspace).After — fix verified at head
0048e96a0(git rev-parse HEADin the same shell as the runs):BUZZ_SEND_BUFFER=1, scratch DBbuzz_gaptest; raw tokio-tungstenite + NIP-42 auth, victim stalled 8s while publisher floods kind-1): victim received 5 buffered EVENTs, then 8×["BUZZ_SYNC_REQUIRED","backpressure"]as the ctrl channel saturated at capacity 8 — the exact path that used to drop silently now emits the contracted client-visible gap signal. Capture taken during development at pre-fix head3cf682b50; the delta to this head is the metrics-inflation fix below + commit trailers (no mechanism change to the signal path).protocol.rs) and the end-to-end test throughsend_fanout_frames(handlers/event.rs) — both run green in the two-host gate table above at this exact head.buzz_fanout_dropped_frames_total); the harness also caught drops-after-cancel re-incrementingbuzz_ws_backpressure_disconnects_total(observed 46 for one connection) — fixed viais_cancelled()idempotence guards with regression testrepeated_drops_after_cancel_do_not_inflate_disconnect_counter, in this head.End-to-end: the first broken boundary in relay→client sync (silent fan-out drops under backpressure) is no longer silent: the relay emits a contracted gap signal on the priority ctrl channel, letting clients know their view is stale without tearing down the socket. Client-side replay on the signal is deliberately OUT of this PR (frozen client contract; follow-up lane).
Verified-by: paired Bingo + Winnie gate at
0048e96a0(two independent hosts, table above); gate-call PASS event77628f7d(Bingo/Winnie paired authority, Michael tiebreak, Ernie concurrence 2026-08-17T00:37Z). Contract: agent-workspaceGUIDES/PR_EVIDENCE_CONTRACT.md.