Repository navigation
fix state update timeout for remember-entities - #6479
Merged
Aaronontheweb merged 1 commit intoMar 2, 2023
Merged
Conversation
We were using the wrong state timeout value, which only gave us 2s to recover R-E data. We're now using the correct value.
Aaronontheweb
enabled auto-merge (squash)
March 2, 2023 03:19
Aaronontheweb
added a commit
that referenced
this pull request
Sep 10, 2026
…ng-state-timeout, as the config documents; the read setting had been used since #6479 Shard.SendToRememberStore armed the remember-entities WRITE timeout with TuningParameters.WaitingForStateTimeout (2s), while the exception it throws on timeout, reference.conf's own doc comment for updating-state-timeout ("Also used as timeout for writes of remember entities when that is enabled"), and DDataRememberEntitiesShardStore's write-majority retry sizing (3 retries at updating-state-timeout / 4) all agree the write should use UpdatingStateTimeout (5s). Commit dde36e9 (#6479) swapped both the read and write sites in one hunk but only described fixing the read path, leaving the write path armed with the wrong, shorter timeout and giving the store's retries zero time to run before the shard gave up and restarted, dropping any buffered messages for the entity. Restores UpdatingStateTimeout on the write arm; the read path (already using UpdatingStateTimeout, more leniently than Pekko) is untouched. Adds RememberEntitiesWriteTimeoutSpec, which fails on unpatched code (the shard restarts and the buffered message is dropped) and passes with the fix (the store's delayed-but-valid write ack arrives and no restart occurs).
Aaronontheweb
added a commit
that referenced
this pull request
Sep 10, 2026
…ng-state-timeout, as the config documents (#8574) * Cluster.Sharding: arm the remember-entities write timeout with updating-state-timeout, as the config documents; the read setting had been used since #6479 Shard.SendToRememberStore armed the remember-entities WRITE timeout with TuningParameters.WaitingForStateTimeout (2s), while the exception it throws on timeout, reference.conf's own doc comment for updating-state-timeout ("Also used as timeout for writes of remember entities when that is enabled"), and DDataRememberEntitiesShardStore's write-majority retry sizing (3 retries at updating-state-timeout / 4) all agree the write should use UpdatingStateTimeout (5s). Commit dde36e9 (#6479) swapped both the read and write sites in one hunk but only described fixing the read path, leaving the write path armed with the wrong, shorter timeout and giving the store's retries zero time to run before the shard gave up and restarted, dropping any buffered messages for the entity. Restores UpdatingStateTimeout on the write arm; the read path (already using UpdatingStateTimeout, more leniently than Pekko) is untouched. Adds RememberEntitiesWriteTimeoutSpec, which fails on unpatched code (the shard restarts and the buffered message is dropped) and passes with the fix (the store's delayed-but-valid write ack arrives and no restart occurs). * RememberEntitiesFailureSpec: prove the write timeout fix with the existing fake store instead of a new spec Deletes RememberEntitiesWriteTimeoutSpec.cs's standalone 180-line spec and its bespoke delayed-ack store, replacing it with one fact on RememberEntitiesFailureSpec that reuses the shared FakeStore/FakeShardStoreActor harness. The fact delays a remember-entities write's ack by 3s (strictly between the 2s waiting-for-state-timeout and 5s updating-state-timeout reference defaults) and asserts the buffered message is still delivered and the shard does not restart. FakeShardStoreActor's existing Delay failure mode replies to a delayed write with the wrong payload type, so the write never actually completes - every existing Delay fact only relies on that to force a timeout. Adds a new DelayedSuccess failure mode, additive and reusing the same Delayed/timer plumbing, that acknowledges the write correctly so a write can genuinely finish late but within its timeout - which the existing Delay mode could not exercise. Verified this fact discriminates: reverting Shard.cs's one-line fix by hand made it fail (shard restarts at ~2s per the log, buffered message lost); restoring the fix made it pass. All 26 facts in RememberEntitiesFailureSpec pass in three consecutive runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
We were using the wrong state timeout value, which only gave us 2s to recover R-E data. We're now using the correct value.
Addresses part of #6478 but there's still an issue with shard backoff + recreation that isn't handled correctly.
Checklist
For significant changes, please ensure that the following have been completed (delete if not relevant):