ambassador and oidc (identity server 4)

[mlushpenko]

Reference #16 (comment)

Here are my eas logs (basically empty):

kubectl logs eas-external-auth-server-7479497c4c-lfr2q            

> [email protected] start /home/eas/app
> node --nouse-idle-notification --expose-gc --max-old-space-size=8192 src/server.js

{"service":"external-auth-server","level":"debug","message":"cache opts: {\"store\":\"memory\",\"max\":0,\"ttl\":0}"}
{"service":"external-auth-server","level":"info","message":"revoked JTIs: []"}
{"service":"external-auth-server","level":"info","message":"starting server on port 8080"}

Could it be that token is wrong in a way and that's why ambassador can't talk to eas service?

here is my token config:

const jwt = require("jsonwebtoken");
const utils = require("../src/utils");

const config_token_sign_secret =
  process.env.EAS_CONFIG_TOKEN_SIGN_SECRET ||
  utils.exit_failure("missing EAS_CONFIG_TOKEN_SIGN_SECRET env variable");
const config_token_encrypt_secret =
  process.env.EAS_CONFIG_TOKEN_ENCRYPT_SECRET ||
  utils.exit_failure("missing EAS_CONFIG_TOKEN_ENCRYPT_SECRET env variable");

let config_token = {
  /**
   * future feature: allow blocking certain token IDs
   */
  //jti: <some known value>

  /**
   * using the same aud for multiple tokens allows sso for all services sharing the aud
   */
  //aud: "some application id", //should be unique to prevent cookie/session hijacking, defaults to a hash unique to the whole config
  eas: {
    plugins: [{
      type: "oidc",
      issuer: {
          /**
          * via discovery (takes preference)
          */
          discover_url: "https://dev.hal24k.nl/.well-known/openid-configuration",

          /**
          * via manual definition
          */
          //issuer: 'https://accounts.google.com',
          //authorization_endpoint: 'https://accounts.google.com/o/oauth2/v2/auth',
          //token_endpoint: 'https://www.googleapis.com/oauth2/v4/token',
          //userinfo_endpoint: 'https://www.googleapis.com/oauth2/v3/userinfo',
          //jwks_uri: 'https://www.googleapis.com/oauth2/v3/certs',
      },
      client: {
          /**
          * manually defined (preferred)
          */
          client_id: "k8s_ambassador",
          client_secret: "secretsecret"

          /**
          * via client registration
          */
          //registration_client_uri: "",
          //registration_access_token: "",
      },
      scopes: ["openid", "email", "profile"], // must include openid
      /**
      * static redirect URI
      * if your oauth provider does not support wildcards place the URL configured in the provider (that will return to this proper service) here
      */
      redirect_uri: "https://eas.hal24k.nl:8443/oauth/callback",
      features: {
          /**
          * how to expire the cookie
          * true = cookies expire will expire with tokens
          * false = cookies will be 'session' cookies
          * num seconds = expire after given number of seconds
          */
          cookie_expiry: false,

          /**
          * how frequently to refresh userinfo data
          * true = refresh with tokens (assuming they expire)
          * false = never refresh
          * num seconds = expire after given number of seconds
          */
          userinfo_expiry: true,

          /**
          * how long to keep a session (server side) around
          * true = expire with tokenSet (if applicable)
          * false = never expire
          * num seconds = expire after given number of seconds (enables sliding window)
          *
          * sessions become a floating window *if*
          * - tokens are being refreshed
          * or
          * - userinfo being refreshed
          * or
          * - session_expiry_refresh_window is a positive number
          */
          session_expiry: true,

          /**
          * window to update the session window based on activity if
          * nothing else has updated it (ie: refreshing tokens or userinfo)
          *
          * should be a positive number less than session_expiry
          *
          * For example, if session_expiry is set to 60 seconds and session_expiry_refresh_window value is set to 20
          * then activity in the last 20 seconds (40-60) of the window will 'slide' the window
          * out session_expiry time from whenever the activity occurred
          */
          session_expiry_refresh_window: 86400,

          /**
          * will re-use the same id (ie: same cookie) for a particular client if a session has expired
          */
          session_retain_id: true,

          /**
          * if the access token is expired and a refresh token is available, refresh
          */
          refresh_access_token: true,

          /**
          * fetch userinfo and include as X-Userinfo header to backing service
          */
          fetch_userinfo: true,

          /**
          * check token validity with provider during assertion process
          */
          introspect_access_token: false,

          /**
          * which token (if any) to send back to the proxy as the Authorization Bearer value
          * note the proxy must allow the token to be passed to the backend if desired
          *
          * possible values are id_token, access_token, or refresh_token
          */
          authorization_token: "id_token"
      },
      assertions: {
          /**
          * assert the token(s) has not expired
          */
          exp: true,

          /**
          * assert the 'not before' attribute of the token(s)
          */
          nbf: true,

          /**
          * assert the correct issuer of the token(s)
          */
          iss: true,

          /**
          * custom userinfo assertions
          */
          userinfo: [
              // {
              //     ...
              //     see ASSERTIONS.md for details
              // },
              // {
              //     ...
              // }
          ],

          /**
          * custom id_token assertions
          */
          id_token: [
              // {
              //     ...
              //     see ASSERTIONS.md for details
              // },
              // {
              //     ...
              // }
          ]
      },
      cookie: {
          //name: "_my_company_session",//default is _oeas_oauth_session
          //domain: "example.com", //defaults to request domain, could do sso with more generic domain
          //path: "/",
      },
      // see HEADERS.md for details
      headers: {},
    },], // list of plugin definitions, refer to PLUGINS.md for details
  }
};

config_token = jwt.sign(config_token, config_token_sign_secret);
const conifg_token_encrypted = utils.encrypt(
  config_token_encrypt_secret,
  config_token
);

//console.log("token: %s", config_token);
//console.log("");

console.log("encrypted token (for server-side usage): %s", conifg_token_encrypted);
console.log("");

console.log(
  "URL safe config_token: %s",
  encodeURIComponent(conifg_token_encrypted)
);
console.log("");

[travisghansen]

How many replicas are you running? For debugging it'll be much easier to launch 1 if not already doing so. If ambassador were making requests you would see some activity there..

[mlushpenko]

1 replica and no redis for now. 3 replicas of ambassador, I will decrease to 1 for debugging on Monday and try again using ClusterIP endpoint versus https and let you know.

[travisghansen]

Ok, I'll see if I can find my test setup I had and send it over too.

[mlushpenko]

Got some progress. As it often happens - my mistake, I had another auth service running from previous testing. But apart from that I changed https endpoint to service name and added namespace and port auth_service: eas-external-auth-server.external-auth-server:80 :

apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      auth_service: eas-external-auth-server.external-auth-server:80
      path_prefix: /ambassador/verify-params-url/%7B%22config_token%22%3A%22732h5VnooKJz0KSmmgMAup8OVSbwv2qVcQXQSAnVJ3Jyambq4uU%2BFNGZzUfFtem6qbkKPJNUoVwvhFBG0%2BkZrZ4WaC%2FuOg9X%2F6kDOoxNTXyuc0WqgBJDUwbA%3D%3D%22%2C%22fallback_plugin%22%3A0%7D
      allowed_request_headers:
        - authorization
      include_body:
        max_bytes: 4096
        allow_partial: true
      status_on_error:
        code: 503
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: eas_mapping
      host: eas.hal24k.nl:8443
      prefix: /
      bypass_auth: true
      service: eas-external-auth-server.external-auth-server:80
  creationTimestamp: null
  labels:
    app.kubernetes.io/instance: eas
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: external-auth-server
    helm.sh/chart: external-auth-server-0.1.0
  name: eas-external-auth-server
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/instance: eas
    app.kubernetes.io/name: external-auth-server
  sessionAffinity: None
  type: ClusterIP

And got some logs in eas, last line is:

{"service":"external-auth-server","level":"info","message":"end verify pipeline with status: 302"}

Will check identity server logs again, try a few more things and come back with more info.

[mlushpenko]

Another potentially interesting thing. During token generation, I put https://eas.hal24k.nl:8443/oauth/callback as redirect_uri param. Putting the same URI in identity server client configuration results in 404 error.

Looking at eas logs, I've found redirect_uri to be https://eas.hal24k.nl:8443/oauth/callback?__eas_oauth_handler__=authorization_callback and after updating client config to this URI, I got login page. Login worked . and I can see all user info in logs, but final results is still 403 error:

{"message":"tokenSet is premature","level":"verbose","service":"external-auth-server"}
{"service":"external-auth-server","level":"debug","message":"plugin response {\"statusCode\":403,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[],\"clearCookies\":[],\"headers\":{},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"oidc\",\"issuer\":{\"discover_url\":\"https://haldev.hal24k.nl/.well-known/openid-configuration\"},\"client\":{\"client_id\":\"k8s_ambassador\",\"client_secret\":\"secret\"},\"scopes\":[\"openid\",\"email\",\"profile\"],\"redirect_uri\":\"https://eas.hal24k.nl:8443/oauth/callback\",\"features\":{\"cookie_expiry\":false,\"userinfo_expiry\":true,\"session_expiry\":true,\"session_expiry_refresh_window\":86400,\"session_retain_id\":true,\"refresh_access_token\":true,\"fetch_userinfo\":true,\"introspect_access_token\":false,\"authorization_token\":\"id_token\"},\"assertions\":{\"exp\":true,\"nbf\":true,\"iss\":true,\"userinfo\":[],\"id_token\":[]},\"cookie\":{\"name\":\"_eas_oauth_session\",\"domain\":null,\"path\":\"/\"},\"headers\":{},\"pcb\":{}}}}"}
{"service":"external-auth-server","level":"info","message":"end verify pipeline with status: 403"}

[mlushpenko]

In Identity logs everything looks good:

{
  "Name": "Token Issued Success",
  "Category": "Token",
  "EventType": "Success",
  "Id": 2000,
  "ClientId": "k8s_ambassador",
  "ClientName": "k8s_ambassador_client",
  "Endpoint": "Token",
  "Scopes": "openid email profile",
  "GrantType": "authorization_code",
  "Tokens": [
    {
      "TokenType": "id_token",
      "TokenValue": "******"
    },
    {
      "TokenType": "access_token",
      "TokenValue": "****"
    }
  ],
  "TimeStamp": "2019-07-29T10:20:01Z",
  "ProcessId": 1,
}

So, I guess it's something in eas or ambassador, any ideas what to check?

[mlushpenko]

Maybe this line?

external-auth-server/src/plugin/oauth/index.js

Line 336 in 8d97265

const tokenSetValid = await plugin.token_set_asertions(tokenSet);

I didn't do any assertions, I mean I set introspect_access_token: false so was expecting assertions process will be skipped. I don't have good idea yet about assertions.

[travisghansen]

Ok this sounds like some great progress! Was going to mention the redirect uri needs to be something the client browser can access.

Given the message it seems you may have a time differential between the servers. For testing out the theory it appears you may be able to set nbf to false in the token and get past this particular issue. Try it out and let me know.

[mlushpenko]

Wow, you know your stuff :) it worked!

[mlushpenko]

@travisghansen thanks for your support! I am also happy to see I get all custom claims in logs like this (now I see time difference after you said it):

{
  "service": "external-auth-server",
  "level": "debug",
  "message": {
    "id_token": {
      "nbf": 1564393923,
      "exp": 1564394223,
      "iss": "https://haldev.hal24k.nl",
      "aud": "k8s_ambassador",
      "iat": 1564393923,
      "at_hash": "PFK5Uk-kDB",
      "sid": "1275ff339c87c5cc",
      "sub": "a942ec-8fbf-58a9",
      "auth_time": 1564393922,
      "idp": "local",
      "Tenant": "test123",
      "role": "Administrator",
      "preferred_username": "testuser",
      "name": "testuser",
      "email": "[email protected]",
      "amr": [
        "pwd"
      ]
    }
  }
}

Is it possible to protect different endpoints based on user/tenant/group info? I am thinking something in the lines of creating multiple tokens with different id_token or userinfo assertions and somehow specifying different tokens for different services, although so far I've seen only a single AuthServer definition, so not sure if multiple tokens are possible (but I read something like that in your docs)

Let me know if that sounds reasonable and if yes - please point me in the right direction 😊

Shall I open another issue? 😃

[travisghansen]

@mlushpenko great! I need to make some notes about the challenges you hit and make sure to get them documented. Can you share what you're using for the the oidc server by chance?

Within the service itself I don't currently support per-route logic (https://github.com/travisghansen/external-auth-server/blob/master/DEVELOPMENT.md#ideas). I've considered it but it's not super high on the priority list right now. Some proxies let you configure things a bit more fine-grained than ambassador (which from a configuration standpoint I found extremely limiting).

I've even had requests to effectively bind scopes to routes which is quite interesting but very specific. Having said all that, to build a custom plugin with specific support for this kind of thing wouldn't be terribly difficult either, especially if we added a couple 'hooks' in the plugins to check whether verification for the request is required or not.

Thanks for the patience and feedback to get it going with ambassador! Looking forward to more input :)

[travisghansen]

Maybe I misunderstood, by 'different endpoints' do you mean completely different services, or do you mean specific urls/routes within a single service?

[mlushpenko]

I meant completely different services running on kubernetes or rather different instances of the same application that belong to different tenants.

oidc is used to authenticate users to multiple web apps on our platform if that makes sense.

Rough idea is to separate users who run data science workloads. Kubeflow has good ideas how to support that but we don't know when it will be ready and so far they are planning to use dex. With with my understanding, looks like dex won't work for us due to limited group support.

So, researching if we can login to our identity provider, get group info and then use that info to either route to different instances of the same app or to integrate it with the application internal authorization system. I don't have clear idea/understanding yet, so kinda practicing to see what can work out.

[mlushpenko]

Looking at assertion examples and info from my comment #22 (comment), I was thinking that something like this will prohibit users from accessing the webapp if they are not in test123 tenant.

{
    query_engine: "jp",
    query: "$.id_token.Tenant",
    rule: {
        method: "eq",
        value: "test123",
    }
}

And also for this specific token we could add policy circuit breaker:

pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "test123.example.com",
                negate: true
              }
            }
          ],

And then, if we could add several tokens, I guess the whole authentication/authorization would work for multi-tenant environment or maybe I am just imagining things in my head 😆

(UPDATE after reading docs again): not several tokens, but multiple plugins if it's possible to configure several oidc plugins with a bit different configs

[mlushpenko]

I think I am quite close (haven't tested multiple oidc plugins yet). This is what I have, only relevant parts:

plugins: [
      {
        type: "oidc",
        pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "ambassador.hal24k.nl:8443",
              }
            }
          ],
          stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "ambassador.hal24k.nl:8443",
                negate: true
              }
            }
          ]
        },
        assertions: {
          id_token: [
            {
              query_engine: "jp",
              query: "$.name",
              rule: {
                  method: "eq",
                  value: "lushpenko",
              }
            }
          ]
        },
      },
      {
        type: "noop"
      },
]

This allows the following:

1. Going to ambassador.hal24k.nl:8443 without authentication via noop plugin
2. Going to demo.hal24k.nl:8443 which redirects me to login page and then I see the following in the logs:

{"service":"external-auth-server","level":"debug","message":"id_token {\"nbf\":1564481694,\"exp\":1564481994,\"iss\":\"https://dev.hal24k.nl\",\"aud\":\"k8s_ambassador\",\"iat\":1564481694,\"at_hash\":\"uX3VK6xAqN2Td6s1ndnH4A\",\"sid\":\"687b0c602a068e\",\"sub\":\"a969f-1c7b229e58a9\",\"auth_time\":1564481152,\"idp\":\"local\",\"Tenant\":\"test123\",\"role\":\"Administrator\",\"preferred_username\":\"lushpenko\",\"name\":\"lushpenko\",\"email\":\"[email protected]\",\"amr\":[\"pwd\"]}"}
{"service":"external-auth-server","level":"debug","message":"asserting: {\"query_engine\":\"jp\",\"query\":\"$.name\",\"rule\":{\"method\":\"eq\",\"value\":\"lushpenko\"}} against value: \"lushpenko\""}
{"service":"external-auth-server","level":"debug","message":"passed assertion: {\"query_engine\":\"jp\",\"query\":\"$.name\",\"rule\":{\"method\":\"eq\",\"value\":\"lushpenko\"}} against value: \"lushpenko\""}

just need to replace name for tenant parameter

[mlushpenko]

It worked! Multiple OIDC plugins with the same credentials worked, yoohoo!

pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "ambassador.hal24k.nl:8443",
                negate: true,
              }
            }
          ],
          stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "ambassador.hal24k.nl:8443",
              }
            }
          ]
        },
      assertions: {
          id_token: [
            {
              query_engine: "jp",
              query: "$.name",
              rule: {
                  method: "eq",
                  value: "lushpenko",
              }
            }
          ]
        }

versus

pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "demo.hal24k.nl:8443",
                negate: true,
              }
            }
          ],
          stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "eq",
                value: "demo.hal24k.nl:8443",
              }
            }
          ]
        },
      assertions: {
          id_token: [
            {
              query_engine: "jp",
              query: "$.name",
              rule: {
                  method: "eq",
                  value: "robert",
              }
            }
          ]
        }

Now, I can loging to ambassador.hal24k.nl and my colleague robert can login to demo.hal24k.nl

[travisghansen]

Oh wow! That's a very creative setup. I'm going to have to read it over a few times to better understand what you're doing. I really like the general idea though!

Did you mention what you're using for oidc provider?

[travisghansen]

Ok, yeah I'm following what you're doing now, very cool.

It sounds like the config_token data may get quite large which may result in a long auth URL... meaning you may start hit url length limits. That makes you a good candidate for 'server-side' tokens.

https://github.com/travisghansen/external-auth-server/blob/master/CONFIG_TOKENS.md#server-side-tokens

Also make sure to read the security implications with oidc sessions etc here if you haven't already: https://github.com/travisghansen/external-auth-server/blob/master/OAUTH_PLUGINS.md

[mlushpenko]

Indeed token is growing in size, I will check server-side if we agree this solution is suitable for our setup. Now that we have separation on tenant-level, each user will have their own instance of the app.

I understand it would be possible to keep updating pcb rules all the time manually, but some dynamic setup would be much better, like this:

pcbs:
  stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.host",
              rule: {
                method: "regex",
                value: "tenant123-${USERNAME_VARIABLE}.hal24k.nl:8443",
              }
            }
          ]

assertions: {
          id_token: [
            {
              query_engine: "jp",
              query: "$.tenant",
              rule: {
                  method: "eq",
                  value: "tenant123",
              }
            },
            {
              query_engine: "jp",
              query: "$.name",
              rule: {
                  method: "eq",
                  value: "${USERNAME_VARIABLE}",
              }
            }
          ]
        }

notice ${USERNAME_VARIABLE} in pcb and in assertion. Any plans to support variables? :) It's not even a variable, rather regex catch group that I want to pass to the assertion.

This is just an idea, looks like I have found my bottleneck. 95% sure authorization for multiple users will have to be handled by the app itself and in that case the whole external authentication service won't be needed.

[mlushpenko]

Or, I got another idea =D
We could use noop to embed username as header and then use ambassador header-based routing https://www.getambassador.io/reference/headers

I guess I should stop here and not spam you anymore, thanks for your help so far!

[travisghansen]

No problem! Keep me updated what works. noop won't get hit if one of the earlier plugins succeeds

[mlushpenko]

I see headers injection works:

{"service":"external-auth-server","level":"debug","message":"injecting header: X-Injected-User with value: lushpenko"}

Also configured ambassador:

allowed_request_headers:
      - "X-Injected-User"

and see that header actually gets propagated in ambassador:

'x-injected-user', 'lushpenko'
'x-envoy-expected-rq-timeout-ms', '10000'
'x-envoy-original-path', '/ambassador/v0/diag/grp-38f09d2dd04a6b2601c3e9b4e308de13209955c3'

but somehow ambassador ignores it in the end. Maybe need to open issue with them.

[travisghansen]

Very cool sounds like you're very close!

[travisghansen]

Crude ambassador documentation and support for url and header based envoy setups snapped in v0.5.0.