Kibana throws internal server error when auth fails due to missing saml token

[tsullivan]

Kibana version: 6.3

Elasticsearch version: 6.3

A user authenticates to Kibana via an Active Directory service, and can use Kibana for the day. When they stop using Kibana for a long time (usually when returning the next day), the next time they open Kibana they see an error on the screen:

{ 
  statusCode: 500, 
  error: "Internal Server Error", 
  message: "An internal server error occurred" 
} 

Elasticsearch logs will show

[illegal_state_exception] token document is missing and must be present 

One can work around the problem by clearing browser cookies and refreshing the page.

Expected result:
If the token used for authentication is no longer valid, the user should be asked to log in again.

[tsullivan]

@kobelb in https://github.com/elastic/kibana/blob/master/x-pack/plugins/security/server/lib/authentication/providers/saml.js#L43, Kibana handles invalid refresh tokens with some hardcoded response body strings. Is this a case of ES not returning back the proper message body for an error?

[kobelb]

@elastic/es-security Kibana currently has the two error descriptions hard-coded to determine whether the SAML refresh token is invalid: token has already been refreshed and refresh token is expired.

We're seeing an additional error description token document is missing and must be present, should we be hard-coding this error message as well, or is there a better way that we should be determining whether the error thrown by Elasticsearch is indeed because the refresh token is expired?

[albertzaharovits]

We're seeing an additional error description token document is missing and must be present, should we be hard-coding this error message as well, or is there a better way that we should be determining whether the error thrown by Elasticsearch is indeed because the refresh token is expired?

No, this might be a bug. Did the user logged out (called ES /_xpack/security/saml/invalidate), but somehow still has the cookie?

We would be grateful for:

• a token (as submitted in the header) that manifests this behavior, ie it worked at some time but not anymore (you could turn up debug logging for org.elasticsearch.xpack.security.authc)

curl -u elastic http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'
{
 “transient”: {
   “logger”: {
     “org.elasticsearch.xpack.security.authc”: “DEBUG"
   }
 }
}
'

• the elasticsearch.log with the stacktrace (the token should be in the logs)
• ideally, the .security index contents

curl -u elastic -X GET "localhost:9200/.security/doc/_search"

Maybe someone else from @elastic/es-security has a better idea, that might not require work from the reporter.

[kobelb]

@dmlittle we've had trouble recreating this issue that you're seeing, would you mind elaborating on your SAML configuration? Which identity provider are you using? Do you use single logout?

[jkakavas]

Ι had missed this originally :/ The behavior described here can happen as we are removing tokens periodically. For this to manifest, all the above needs to happen:

• The user logs in via the saml authProvider in Kibana on timestamp X and leaves it be
• The user then attempts to open kibana again on Y , with Y > X + 24h
• The user's kibana authCookie has a lifetime of at least 24 hours
• The user never explicitly logs out
• Another user logs out or someone uses the Invalidate Token API on Z , with X + 24h < Z < Y ( we potentially trigger the removal of old tokens on any token invalidation while also honoring xpack.security.authc.token.delete.interval )

The event of a user logging out or someone using the invalidation API can trigger the deletion of the now expired access and refresh tokens that the user got on X . When kibana sends a request with this now deleted access token on Y , ES replies with the aforementioned exception as it can't find the associated token document any more.

Arguably, ES should respond with a 401 instead of a 500 here.

[kobelb]

@jkakavas that makes sense, we can update the SAML auth provider in Kibana to take this into consideration, thanks!

[jkakavas]

@kobelb I'm treating this as an ES bug, so hopefully this will be resolved for the next minor. I'll update this issue with the solution once it's merged

[kobelb]

@kobelb I'm treating this as an ES bug, so hopefully this will be resolved for the next minor. I'll update this issue with the solution once it's merged

Gotcha, this is in regard to the 500 vs 401 right? There's the potential that even with this change, we might have to augment our auth providers to handle this specific error situation differently.

[jkakavas]

If I understand the manifestation above correctly, Kibana gets the 500 ( which will be made into a 401 ) when the access token is used in a normal API request. I thought that Kibana expects 401 in this case and will then attempt to use the refresh token

It was originally thought that the 500 was a response to the refresh token API request, but I don't believe this is the case

[azasypkin]

I'll try to reproduce locally and see if there is anything we'll need to do in Kibana once we start getting 401 instead of 500 from Elasticsearch.