Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only cache credentials on success #58594

Merged
merged 1 commit into from
Sep 9, 2021
Merged

Conversation

wfurt
Copy link
Member

@wfurt wfurt commented Sep 2, 2021

This is not what I expected. But I could reproduce the Safe handle has been closed error with few hundred runs.
With this change I did ~ 5000 runs without single failure. There may be more to it but this definitely seems to help.

I have some ideas why we do not see this more:

static void ShrinkCredentialCache()
{
//
// A simplest way of preventing infinite cache grows.
//
// Security relief (DoS):
// A number of active creds is never greater than a number of _outstanding_
// security sessions, i.e. SSL connections.
// So we will try to shrink cache to the number of active creds once in a while.
//
// We won't shrink cache in the case when NO new handles are coming to it.
//
if ((s_cachedCreds.Count % CheckExpiredModulo) == 0)

With this we would try to string the cache only on multiples of 32. With certificate, protocols and EncryptionPolicy that generally would be never on clients as there is typically no client certificate, few protocol variations and nobody sane should not use anything but Encrypt policy.

So it tales some effort to even hit the reduction code - perhaps by crafting many unique certificates.
For future investigations, lowering CheckExpiredModulo makes it much easier to test that code.

While we lock on cache changes, we don't seems to when we retrieving cached value.
And all the entries stored in the cache are disposed.
And we seems to allocate a lot while attempting to shrink:

cahced.Dispose();
cahced = SafeCredentialReference.CreateReference(creds);

So while the caching may be improved, I did not find anything particularly wrong.
My current speculation is that the observed issue is caused by polluting cache with bad values.
With the added extra check my local runs are now happy.

Fixes #30724
Fixes #46770

@wfurt wfurt requested a review from a team September 2, 2021 23:29
@wfurt wfurt self-assigned this Sep 2, 2021
@ghost
Copy link

ghost commented Sep 2, 2021

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

This is not what I expected. But I could reproduce the Safe handle has been closed error with few hundred runs.
With this change I did ~ 5000 runs without single failure. There may be more to it but this definitely seems to help.

I have some ideas why we do not see this more:

static void ShrinkCredentialCache()
{
//
// A simplest way of preventing infinite cache grows.
//
// Security relief (DoS):
// A number of active creds is never greater than a number of _outstanding_
// security sessions, i.e. SSL connections.
// So we will try to shrink cache to the number of active creds once in a while.
//
// We won't shrink cache in the case when NO new handles are coming to it.
//
if ((s_cachedCreds.Count % CheckExpiredModulo) == 0)

With this we would try to string the cache only on multiples of 32. With certificate, protocols and EncryptionPolicy that generally would be never on clients as there is typically no client certificate, few protocol variations and nobody sane should not use anything but Encrypt policy.

So it tales some effort to even hit the reduction code - perhaps by crafting many unique certificates.
For future investigations, lowering CheckExpiredModulo makes it much easier to test that code.

While we lock on cache changes, we don't seems to when we retrieving cached value.
And all the entries stored in the cache are disposed.
And we seems to allocate a lot while attempting to shrink:

cahced.Dispose();
cahced = SafeCredentialReference.CreateReference(creds);

So while the caching may be improved, I did not find anything particularly wrong.
My current speculation is that the observed issue is caused by polluting cache with bad values.
With the added extra check my local runs are now happy.

fixes #30724

Author: wfurt
Assignees: wfurt
Labels:

area-System.Net.Security

Milestone: -

@karelz
Copy link
Member

karelz commented Sep 3, 2021

I assume it fixes also #46770 -- top post edited.

@wfurt
Copy link
Member Author

wfurt commented Sep 3, 2021

I assume it fixes also #46770 -- top post edited.

Yes, I think it should. I was not assigned to me so I miss it.

@@ -818,7 +818,7 @@ private SecurityStatusPal GenerateToken(ReadOnlySpan<byte> inputBuffer, ref byte
// This call may bump up the credential reference count further.
// Note that thumbPrint is retrieved from a safe cert object that was possible cloned from the user passed cert.
//
if (!cachedCreds && _securityContext != null && !_securityContext.IsInvalid && _credentialsHandle != null && !_credentialsHandle.IsInvalid)
if (!cachedCreds && status.ErrorCode == SecurityStatusPalErrorCode.OK && _securityContext != null && !_securityContext.IsInvalid && _credentialsHandle != null && !_credentialsHandle.IsInvalid)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have tests that will end up with a non-OK ErrorCode?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a look. There seems to be some race condition so I see the "Close handle" only once in a while. It would be nice to have some predictability for error cases.

@wfurt wfurt merged commit b7ac52b into dotnet:main Sep 9, 2021
@wfurt wfurt deleted the cachedCreds_30724 branch September 9, 2021 15:28
@karelz karelz added this to the 7.0.0 milestone Sep 22, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
3 participants