Allow token refresh for multiple requests in a small window

[epixa]

The problem

The current behavior for refreshing a token is to immediately invalidate a refresh token when it is used the first time. In principle this is a sensible way to prevent the refresh token from being used maliciously, but in practice it can trivially break a client that is making many requests to Elasticsearch.

In Kibana, the consequence is that sessions using our token-based providers (saml and token) occasionally get destroyed as multiple requests race to refresh an expired access token. This problem will only get worse as we expand the usage of canvas expressions which can result in more requests in parallel.

Let's say a user allows their session to idle for a bit and their access token expires, then they click refresh on a dashboard. Requests A, B, and C are fired off to Elasticsearch in parallel, all three get rejected due to an expired access token, and all three attempt to refresh the session using the same refresh token. A succeeds and returns a new session cookie to the client. B and C fail since the refresh token has already been used. The client sees the failures of B and C and assumes the session must be dead, so it either errors or sends the user to the login form with a cleared session.

Only processing a single refresh token in Kibana for parallel requests isn't practical because there could be multiple Kibana instances behind a load balancer handling requests for the same session.

The proposal

I propose that we add an optional nonce property to the refresh_token grant type. If the first request to refresh a token contains a nonce, then subsequent refresh token requests for that same refresh token using the same nonce will return the same new access token.

A client like Kibana can generate a random nonce value each time a user does a page reload on Kibana, and it'll include the nonce in any refresh token requests that occurs for that user during this time. Unlike the access token and refresh token, the nonce is never stored in the session itself.

There can be a "refresh window" associated with the nonce feature as well, maybe 5 minutes or something like that, where afterwords the token cannot be refreshed even with a matching nonce.

cc @elastic/kibana-security

[elasticmachine]

Pinging @elastic/es-security

[jkakavas]

That's a tricky one :) My initial reaction ( and I shared this with @kobelb in the issue we came across this ) is that this needs to be fixed on the client side. All the solutions that come to mind though, either require a shared server-side state in Kibana or do not handle the "HA with many Kibana instances" scenario.

I think the suggested approach adds a lot of complexity and I'd like to propose a different solution, one where we do not invalidate refresh tokens on first use. The original decision to revoke refresh tokens on use, still makes sense and it is in the spirit of the specification:

The authorization server MAY revoke the old
refresh token after issuing a new refresh token to the client.

but I consider revisiting it as a sensible trade-off in order to resolve the issue at hand with low to no risk involved.

Suggested Solution

We stop revoking refresh tokens on first use. This will resolve the issue at hand as Requests A,B, and C are fired off to Elasticsearch in parallel, all three get rejected due to an expired access token, and all three attempt to refresh the session using the same refresh token. A succeeds and gets a new access token and returns a new session cookie to the client. B and C also succeed but get new access tokens and refresh tokens that are stored in the users session cookie and overwrite the existing one. The end result is that the latest of the concurrent requests to get a response will dictate the access token and refresh token to store in the user's session cookie.

This means that nothing needs to change on the Kibana's side. Any concerns regarding the possibility of A, B, C returning different access/refresh tokens , all of which will be valid and with the same expiration ( give or take a few ms ) ?

Standards and Best Practices compliance

There is nothing in the original standard, the OAuth 2.0 Threat Model and Security Considerations or the recent OAuth 2.0 Security Best Current Practice draft that mandates the revocation on use of the refresh tokens.

Threat Model

Threat : Retaining persistent access

Attack1 : A legitimate client could use the same refresh token multiple times.

Countermeasures : The legitimate client would get a new access token and a new refresh token each time but

• the access token would give them the same privileges that the original access token or the access token that they would get with the original refresh, would give them.
• the new refresh tokens will extend the duration of its access ( each new one will give the option to refresh access for 24h ) but this is in no way different than sequentially refreshing using a new refresh token each time.

As such, this threat degrades into normal functionality and should not be considered as a risk.

Attack2 : An attacker could use the same leaked refresh token multiple times.

Countermeasures : As above, this doesn't give any additional access to an attacker. The ability to explicitly invalidate tokens (either for specific tokens or for any user/realm combination) is another mitigating factor to this attack. Finally we do authenticate the client on the refresh_grant so leaked refresh tokens cannot be used by other clients ( outside Kibana )

Threat : Performance degradation / Denial of Service

Attack1: A malicious user can continuously refresh an existing refresh token. Each of these requests would give them a new refresh token and access token and they could then start also using this new refresh token to get additional ones, etc. Compared to the old approach, this would allow the malicious user to create documents in the security index in an exponential rate which depending on deployment specific factors could hurt performance of the Token Service.

Countermeasures: We already delete expired tokens so there is an upper limit to the amount of documents they can create, as we would periodically delete ones containing expired refresh tokens. Additionally, we can decide on a sensible maximum number of refreshes that would resolve the issue but further mitigate the mentioned threat.

[kobelb]

Do we need to take the logout/token-invalidation behavior into consideration here? If we could potentially have multiple access/refresh tokens that were created from the initial refresh token, it becomes quite complicated (perhaps impossible) to ensure that on logout "all" of the refresh/access tokens are invalidated. This is one of the main complaints which I hear from users about our existing basic auth provider, that on logout they can re-use the previous session cookies (which would contain the previous access/refresh tokens) to authenticate subsequently.

[jkakavas]

@kobelb wouldn't a call to invalidate all tokens of the user satisfy the logout requirement ?

DELETE /_xpack/security/oauth2/token
{
  "username" : "user"
}

[kobelb]

@jkakavas I forgot we could do that now, that's awesome! Would there be a way to limit this to only the tokens associated with the instance of Kibana so it wouldn't log them out of all other instances of Kibana, or their other usages of tokens.

[jkakavas]

Would there be a way to limit this to only the tokens associated with the instance of Kibana

No, that's a good catch. I didn't think about this scenario at all. I don't think that we have a good way of differentiating that. We could add the API parameter to also filter by oAuth2 client but there is also a high probability that these different kibana instances will use the same user to authenticate to Elasticsearch (that principal is what we keep track as client). The original proposal will also need a solution around this as B, C, D would create new refresh_tokens also and on logout we would revoke only the one that D created.

I'll go back to the drawing board and re-evaluate @epixa 's proposal also.

[albertzaharovits]

I agree with @jkakavas's assessment that we will not be introducing new attack vectors by not invalidating the refresh token on use. But I worry that we might increase the possibility of leaking refresh tokens if we generate new ones on each reuse. I would propose that we not generate a new refresh token for each use of another refresh token, and maintain the original one valid for subsequent uses.

[jkakavas]

I would propose that we not generate a new refresh token for each use of another refresh token, and maintain the original one valid for subsequent uses.

In terms of token leakage that would also broaden the attack surface in the time dimension as we would have fewer refresh tokens but with longer validity period. ( i.e. an attacker finds a refresh token logged somewhere from 4-5 days ago which might still be active )
This also suffers from the same issue with regards to access token invalidation on logout , as the previous proposal.

In general , the fact that the refresh token can only be used by the original client makes token leakage a reduced impact threat as an attacker would also need to compromise the client(usually kibana for now) credentials in order to exploit a leaked refresh token.

I'll go back to the drawing board and re-evaluate @epixa 's proposal also.

I'll get to it in the next couple of days.

[jkakavas]

We discussed this today in our team meeting. Jay came up with an idea to simplify the original proposal by not requiring a nonce and using a window of 1 min. We can probably bring this down to a few seconds as we're trying to solve for concurrent requests. The idea is that we would return the same access token for the duration of this window as the original proposal also suggests.
I have an idea of an implementation that would allow this and plan to give it a shot unless someone brings up a concern that we have missed. @epixa did you have any specific ideas why the nonce would be of use that is not covered by just a time window?

[epixa]

I don't have strong opinions about the nonce, but I do think it solves a different problem that is possible within the time window.

In a UI client, if an attacker gets access to the access token for another user's session, they almost certainly have access to the refresh token as well. Both of these things must be associated with a session in order to be useful.

We can't prevent an attacker from initiating the session reset themselves, but without the nonce feature, these two things are reality:

1. If the attacker is the one that first initiates the token refresh, the original user remains unaware that something is wrong as their session will also seamlessly refresh (to an already-exploited new token). If a nonce was in place, the original user's session would fail to refresh and we could potentially error with something like "this refresh token has already been used" which might tip them off to a problem.
2. If the original user is the one that first initiates the token refresh, the attacker can still safely refresh the token anyway (still unbeknownst to the user). If the nonce was in place, the attacker's request would be rejected.

To "quietly" exploit the nonce feature, the attacker needs to attack a different mechanism entirely in order to get a valid nonce. The act of getting the session info isn't enough.

With the time window alone, no additional exploit is necessary, the timing just needs to be right. This window could be made easier to exploit if the attacker can determine approximately when a token was due to expire. I notice the token service itself returns with expires_in, so if that info is available on an ongoing basis from ES, then it would be pretty easy for an attacker to exploit the window. If not available easily through ES, it's also not unlikely that a client would store the whole token service payload (including expires_in) along with a created_at timestamp with the session, so in that case the attacker could exploit the window using the info they had on hand.

I'm comfortable leaving it to this team to decide the extent to which this is acceptable in our auth threat model.

[albertzaharovits]

@epixa If I'm following correctly the nonce is a use-once token, that unlike the refresh token, which is also use-once atm, is generated by Kibana and not by ES. For that matter, the nonce is generated instead of stored.

I think the nonce is a raw method of binding the access-refresh token pair to a Kibana instance.
Instead, and this is not mentioned in the previous description, the refresh token is bound to the credentials of the kibana process that generated the access-refresh token pair in the first place. So, an attacker does not simply has to race for the refresh, or guess the generated nonce, it has to know the secret kibana password.

I guess, this is not how things work right now, but each authenticator process in a multiple kibana deployment should use different secret credentials. In this way, the nonce is not required to identify a Kibana instance (it is still useful, if credentials are shared, but then I don't think the refresh token is useful anymore).