Add retry policy to NuCache content store lock acquisition#19429
Closed
AndyButland wants to merge 3 commits intov13/devfrom
Closed
Add retry policy to NuCache content store lock acquisition#19429AndyButland wants to merge 3 commits intov13/devfrom
AndyButland wants to merge 3 commits intov13/devfrom
Conversation
AndyButland
changed the title
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR addresses a regression in the NuCache content store locking mechanism by adding a retry strategy with incremental backoff.
- Introduces a call to WaitForWriteLock before attempting to acquire the lock
- Implements WaitForWriteLock which retries waiting for the lock with incremental delay
- Updates inline comments to document the new retry strategy
| _logger.LogDebug("Waiting for write lock, attempt {Attempt} of {Retries}. Sleeping for {DelayFor} milliseconds.", retryCount, retries, delayFor); | ||
| } | ||
|
|
||
| Thread.Sleep(delayFor); |
There was a problem hiding this comment.
Consider using an asynchronous wait mechanism instead of Thread.Sleep to avoid blocking threads in potentially async contexts.
Suggested change
| Thread.Sleep(delayFor); | |
| await Task.Delay(delayFor); |
Contributor
Author
There was a problem hiding this comment.
Don't believe we can do this here, or at least we wouldn't get any benefit, as this is called from synchronous code.
AndyButland
force-pushed
the
v13/bugfix/add-retry-to-nucache-content-store-lock-acquisition
branch
from
6f89b70 to
f0c83ae
Compare
… the check before the wait that threw the recursive lock exception. Added additional check to ensure we don't release a lock that's already released.
AndyButland
deleted the
v13/bugfix/add-retry-to-nucache-content-store-lock-acquisition
branch
Contributor
Author
|
Closed in favour of #19434 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prerequisites
Resolves: #19338
Description
The linked issue reports a regression after #17246, which was introduced to avoid locking exceptions in asynchronous code.
It seems that after this, near-simultaneous requests to save and publish content results in one or more of the requests failing due to NuCache being unable to acquire a lock (the
SemaphoneSlimthat's it's based on not being released from the previous operation). You would then find the content created and published, but the failed ones wouldn't be in the cache, which is manifested in the backoffice by the message of "published but not in the cache" shown where the URLs are displayed on the "Info" panel.To resolve I've added a retry strategy that will retry on an incrementally delayed basis for up to 2 second if the lock can't be acquired.
It's appreciated that this may not be the most elegant approach nor proposed with the deepest understanding of how the NuCache is working, but given this is a regression in a component that's now legacy, I think it'll be a reasonable fix for the issue.
Testing
The original issue can be replicated as shown in the linked issue from the browser console when logged into the backoffice.
You should find with this PR in place multiple requests to create, save and publish content will succeed.