external-auth-server with keycloak and ambassador

[Romeren]

Hi,

So my setup is:
Ambassador v: 0.72.0
Keycloak v: 5.0.0
And external-auth-server as middle-ware providing openid-connect authentication to a webservice.

I followed the setup first for the Traefik example, substituting config where it was needed.
My configuration for the external-auth-server values.yaml looks as follows:

configTokenSignSecret: ZQTJMHgsRUYD4vaDJnmutYMU
configTokenEncryptSecret: MAuZtmwxfvcJCabSmhrvcAjv
issuerSignSecret: mEAYPr9bcZk8dFda8T6dBmzK
issuerEncryptSecret: QjjgXyVvVyVyUfBW6ZhFTG3w
cookieSignSecret: 5spCwm2jCtekwEb3G6Hcnav8
cookieEncryptSecret: YwX3gzTZmPNUL9v5RMhwsnZq
sessionEncryptSecret: jMV5WtUCfSme4xx9Qkmu2Jv2
logLevel: "info"
redis-ha:
  enabled: false

In addition i made a couple of changes to the service.yaml to accommodate ambassadors annotation-based configuration.... These looks like this:

  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      auth_service: {{ include "external-auth-server.fullname" . }}
      proto: http
      allowed_request_headers:
        - authorization
      include_body:
        max_bytes: 4096
        allow_partial: true
      ---
      apiVersion: ambassador/v1
      kind:  Mapping
      name:  eas_mapping
      prefix: /eas/
      bypass_auth: true
      service: {{ include "external-auth-server.fullname" . }}

I managed to create a CONFIG_TOKEN for the keycloak.
The configuration for the plugin added to generate-config-token.js that i used was:

      {
        type: "oidc",
        issuer: {
            /**
            * via discovery (takes preference)
            */
            discover_url: "http://keycloak.default.svc.cluster.local/auth/realms/master/.well-known/openid-configuration",
  
        },
        client: {
            /**
            * manually defined (preferred)
            */
            client_id: "lightningbadger",
            client_secret: "4dbd29da-5b81-417e-a230-abad914de57e"
    
            /**
            * via client registration
            */
            //registration_client_uri: "",
            //registration_access_token: "",
        },
        scopes: ["openid", "email", "profile"], // must include openid
        /**
        * static redirect URI
        * if your oauth provider does not support wildcards place the URL configured in the provider (that will return to this proper service) here
        */
        redirect_uri: "http://api.lightningbadger.io/eas/oauth/callback",
        features: {
            /**
            * how to expire the cookie
            * true = cookies expire will expire with tokens
            * false = cookies will be 'session' cookies
            * num seconds = expire after given number of seconds
            */
            cookie_expiry: true,
    
            /**
            * how frequently to refresh userinfo data
            * true = refresh with tokens (assuming they expire)
            * false = never refresh
            * num seconds = expire after given number of seconds
            */
            userinfo_expiry: true,
    
            /**
            * how long to keep a session (server side) around
            * true = expire with tokenSet (if applicable)
            * false = never expire
            * num seconds = expire after given number of seconds (enables sliding window)
            *
            * sessions become a floating window *if*
            * - tokens are being refreshed
            * or
            * - userinfo being refreshed
            * or
            * - session_expiry_refresh_window is a positive number
            */
            session_expiry: true,
    
            /**
            * window to update the session window based on activity if
            * nothing else has updated it (ie: refreshing tokens or userinfo)
            *
            * should be a positive number less than session_expiry
            *
            * For example, if session_expiry is set to 60 seconds and session_expiry_refresh_window value is set to 20
            * then activity in the last 20 seconds (40-60) of the window will 'slide' the window
            * out session_expiry time from whenever the activity occurred
            */
            session_expiry_refresh_window: 86400,
    
            /**
            * will re-use the same id (ie: same cookie) for a particular client if a session has expired
            */
            session_retain_id: true,
    
            /**
            * if the access token is expired and a refresh token is available, refresh
            */
            refresh_access_token: true,
    
            /**
            * fetch userinfo and include as X-Userinfo header to backing service
            */
            fetch_userinfo: true,
    
            /**
            * check token validity with provider during assertion process
            */
            introspect_access_token: false,
    
            /**
            * which token (if any) to send back to the proxy as the Authorization Bearer value
            * note the proxy must allow the token to be passed to the backend if desired
            *
            * possible values are id_token, access_token, or refresh_token
            */
            authorization_token: "access_token"
        },
        assertions: {
            /**
            * assert the token(s) has not expired
            */
            exp: true,
    
            /**
            * assert the 'not before' attribute of the token(s)
            */
            nbf: true,
    
            /**
            * assert the correct issuer of the token(s)
            */
            iss: true,
    
            /**
            * custom userinfo assertions
            */
            userinfo: [
            ],
    
            /**
            * custom id_token assertions
            */
            id_token: [
            ]
        },
        cookie: {
            //name: "_my_company_session",//default is _oeas_oauth_session
            //domain: "example.com", //defaults to request domain, could do sso with more generic domain
            //path: "/",
        },
        // see HEADERS.md for details
        headers: {},
    }

However, here comes the part on which i got stuck. The examples provided in this repo uses Traefik configuration and adds this to the

ingress.kubernetes.io/auth-url: https://eas.example.com/verify?fallback_plugin=0&config_token=PLACE_CONFIG_TOKEN_OUTPUT_HERE 

However, since we are using ambassador this is not an option.
So I expect I need to mount the configuration into a secret with the env adapter.
Which i tried without any luck.
So i tried adding the following to my values.yaml

configTokenStores:
 primary:
   adapter: env
   options:
     cache_ttl: 3600
     var: < I HAVE NO IDEA WHAT TO ADD HERE>
configTokens:
 1:  <MY CONFIG TOKEN NOT URL ENCODED >

Can you maybe help me out with this configuration :)
Then i can maybe help with writing some documentation :P

[travisghansen]

This is great! I've been hoping for some help with Ambassador. There's a lot to digest here so let me break it down as much as possible:

• First off, what is the current outcome? Do you get any logs etc showing up in the server pods?

• Without redis-ha enabled AND using oidc you're going to want to have only 1 replica. When you really get going with it you're going to want redis-ha enabled for sure.

• It appears you're mapping the eas service to http://api.lightningbadger.io/eas/. Can you confirm that the service is properly exposed at that endpoint by hitting http://api.lightningbadger.io/eas/metrics endpoint? You should see a bunch of prometheus metrics.

• This auth_service: {{ include "external-auth-server.fullname" . }} seems correct. Except it's missing the required route and get parameters. It needs to be something like: auth_service: {{ include "external-auth-server.fullname" . }}/verify?config_token=CONFIG_TOKEN

• The service needs to hand down more than the authorization headers. In fact for oidc that specific header is meaningless anyhow. Of critical importance is headers that let the service reconstruct the originally requested URL. For traefik I use the x-forwarded-* headers but I'm not sure what ambassador provides. The logic is here: https://github.com/travisghansen/external-auth-server/blob/master/src/utils.js#L77

• I could use some help knowing what headers ambassador can/does provide to know the originally requested URI. If you could point me to a doc or help me understand what gets passed down to the auth service that would be tremendously helpful.

• Ambassador must pass down the cookies (for oidc plugin) as well (maybe it does by default, not sure).

Let's (for the time being) remove the configToken/configTokenStores settings from values.yaml. Once we get you up and running then we can revisit that one.

Looking forward to getting it going for you and getting ambassador documented!

[Romeren]

• First off, what is the current outcome? Do you get any logs etc showing up in the server pods?

• Without redis-ha enabled AND using oidc you're going to want to have only 1 replica. When you really get going with it you're going to want redis-ha enabled for sure.
  Yes, i'm trying to take it one step at a time... Once i got it working with one replica, i'm going to enable it and get it working with redis as well

• It appears you're mapping the eas service to http://api.lightningbadger.io/eas/. Can you confirm that the service is properly exposed at that endpoint by hitting http://api.lightningbadger.io/eas/metrics endpoint? You should see a bunch of prometheus metrics.
  ** That is indeed correct. What i see is: **

 HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.14804600000000007 1561714370600

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.047775000000000005 1561714370600

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.1958210000000001 1561714370600

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1561714147

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 102178816 1561714370601

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1138524160 1561714370601

# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 151408640 1561714370601

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 22 1561714370600

# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 200934

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0.000375416 1561714370600

# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Socket"} 2 1561714370600
nodejs_active_handles{type="Server"} 1 1561714370600

# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 3 1561714370600

# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge
nodejs_active_requests{type="FSReqWrap"} 2

# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 2 1561714370600

# HELP nodejs_heap_size_total_bytes Process heap size from node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 68468736 1561714370600

# HELP nodejs_heap_size_used_bytes Process heap size used from node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 47404256 1561714370600

# HELP nodejs_external_memory_bytes Nodejs external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 2741328 1561714370600

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 524288 1561714370600
nodejs_heap_space_size_total_bytes{space="new"} 33554432 1561714370600
nodejs_heap_space_size_total_bytes{space="old"} 26447872 1561714370600
nodejs_heap_space_size_total_bytes{space="code"} 1572864 1561714370600
nodejs_heap_space_size_total_bytes{space="map"} 2109440 1561714370600
nodejs_heap_space_size_total_bytes{space="large_object"} 4259840 1561714370600

# HELP nodejs_heap_space_size_used_bytes Process heap space size used from node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 35200 1561714370600
nodejs_heap_space_size_used_bytes{space="new"} 15766776 1561714370600
nodejs_heap_space_size_used_bytes{space="old"} 25509648 1561714370600
nodejs_heap_space_size_used_bytes{space="code"} 1175776 1561714370600
nodejs_heap_space_size_used_bytes{space="map"} 2017136 1561714370600
nodejs_heap_space_size_used_bytes{space="large_object"} 2901800 1561714370600

# HELP nodejs_heap_space_size_available_bytes Process heap space size available from node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 480384 1561714370600
nodejs_heap_space_size_available_bytes{space="new"} 731912 1561714370600
nodejs_heap_space_size_available_bytes{space="old"} 1880 1561714370600
nodejs_heap_space_size_available_bytes{space="code"} 1312 1561714370600
nodejs_heap_space_size_available_bytes{space="map"} 240 1561714370600
nodejs_heap_space_size_available_bytes{space="large_object"} 8700239360 1561714370600

# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v10.16.0",major="10",minor="16",patch="0"} 1

# HELP http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.003",status_code="200",method="GET",path="/ping"} 41
http_request_duration_seconds_bucket{le="0.03",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_bucket{le="0.1",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_bucket{le="0.3",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_bucket{le="1.5",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_bucket{le="10",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_bucket{le="+Inf",status_code="200",method="GET",path="/ping"} 44
http_request_duration_seconds_sum{status_code="200",method="GET",path="/ping"} 0.048966237999999995
http_request_duration_seconds_count{status_code="200",method="GET",path="/ping"} 44

# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1

• This auth_service: {{ include "external-auth-server.fullname" . }} seems correct. Except it's missing the required route and get parameters. It needs to be something like: auth_service: {{ include "external-auth-server.fullname" . }}/verify?config_token=CONFIG_TOKEN
  That is also true, i am missing to put the /verify?config_token=CONFIG_TOKEN.... however since this is a kube service selector, this does not work.... I also tried the following without any luck:

auth_service: http://{{ include "external-auth-server.fullname" . }}.<namespace>.svc.cluster.local/verify?config_token=CONFIG_TOKEN

When i look at ambassador i see:

cluster_extauth_http___eas_external_auth_server_default_cluster_local_verify_config_token_yTgEQgAkLw_2B80AsrJyD7ouznnOkjBC4btgvh84Lh1LG3tHnvVi7xS86Vrb8c5GWBZwGIIEaur6LxqnNqXQXwCh_2BN_2BuKcMp0EG7sB8N4Zk1Z8J0hLFUjyW4SX_2BU3awa1vA8eSHw5K1XOsNvMJeaRL6eapc_2FmC7XvUyFcNhZQ_2FbXZw1wpQtt04_2FQd_2FcQX97AFOIJxRP5lQh1ilvm15qC9FkZ7Zwb_2FZ0bQFICdUf77XBXSjY8PEnWiJKuqXWHoL1Urcn48D8_2BC1qbcF_2BPR5ez1BURsJ7vSDQ2LfVXPhhQz_2ByjV5k1JHnMMHGkb5p9CURE1NryliELco6P_2F2P8H65akTji_2FEaqNnHb_2BFO73f6LecYFRvReDMJzKScr4VTbAppshKeeKiFGXw2vtNT058qXX5WUxy5YwSH6grjf7kwooITyDVnipoKJXgH7UUjFpycPNexjWofPC0_2FzM2Zf2au5akZL6wrYC4vMZ5zYtZAU3_2F7CHFpCU0ZpeHhs5LsKeuq1q6i6RA0yvcQbO4mF_2BF7UaUc0DWX3bOenJGITcioInrUJxdiodtMFn0c8CgLi3eqRcfxPxwo2led_2FEDaSKCWE5L_2FglJWH59Qj41DBrcLQm1E5DVGRF_2BdGIa_2FODO5Er8a7PI2zkK_2BzYgZWxnQCHOID6eT0fU4PCePZnnPWrC9_2FUgyPN29C2BD70bAAelL_2Bt8KCtY7JBupZed5sbe_2B2FCfBlnMIvp6b134arioCeFtMqy5vN_2B8dco7_2BgUqOeUsTwsJxpb3v4vH3GgPzwWqPqmLQsvPsyWTarog4FwFgwvrYiHl_2By128oBT8HPMMaZALQtKSbpzCA0UHFAwrYHch1KPIngf4C6kjp49xbvxe61HdY73AS4DiqCWKDLkWwYLhLk4RWSowWXHxK_2Bb19_2B2Z7zB23Hekmv7d5vVpWhZf5XIuhqQKNWQhwnccwporghZCSGHMKIm9cq4JEcmJARXorHfvN2dHgqwvi5O7_2F9f6Dnz9PlCC0VLtPTIBZQy_2FyrBsnUx_2B2scs0SRXda7rsxG_2FmdUs3d7nChf9mJnsLzZvh077nhBJEthGLQAIlMKxrxwQGNDMMJGH5i0ls1EtL36E_2FhrKMRYCK2t_2FG0Yb0wmMJHAjHLq6pAmIOUwQ_2BVcS5pPupZSfsImjKNqsZ5ynAtdc2YrSqcMDo4PAA3fivIFukoGtwF2EeHPuA4j8EsoYS_2BKmCz6jcT9jaZrjbOhYZtduORnMQp2zBN60FRyaJAyh2C9RqBvNijGWTL85vOYsM2z2z6J6vwdDKRkXta3fZcUfiTZYuSICBE8OPVHck0APZoKUg9Q4wXFed_2FfgmmrSf_2B9uR2RmG8E_2BIVxlNO755bw8SAJZF1SuGcLi7CgDZIKAaESZJa7MSEUszoa1B65j7TynsZraQ1gLfAZeKMyxVyOjk8qbA1Lw_3D_3D: service eas-external-auth-server.default.cluster.local/verify?config_token=<MY CONFIG TOKEN URL SAFE> has extra URL components; ignoring everything but the host and port

• The service needs to hand down more than the authorization headers. In fact for oidc that specific header is meaningless anyhow. Of critical importance is headers that let the service reconstruct the originally requested URL. For traefik I use the x-forwarded-* headers but I'm not sure what ambassador provides. The logic is here: https://github.com/travisghansen/external-auth-server/blob/master/src/utils.js#L77
  ** Again correct, i was testing a different AuthService before i decided to try out yours where i needed this header, so it is simply a legacy declaration that i will remove... Regarding the X-Forwarded- header i believe it is in the default allowed headers from ambassador ~For reference check: https://www.getambassador.io/reference/services/auth-service/ **

• I could use some help knowing what headers ambassador can/does provide to know the originally requested URI. If you could point me to a doc or help me understand what gets passed down to the auth service that would be tremendously helpful.
  ** This is available in the above link, but ill put it here as well :) ~Note that these are case sensitive **

allowed_request_headers:
    Authorization
    Cookie
    From
    Proxy-Authorization
    User-Agent
    X-Forwarded-For
    X-Forwarded-Host
    X-Forwarded-Proto
allowed_authorization_headers:
    Location
    Authorization
    Proxy-Authenticate
    Set-cookie
    WWW-Authenticate

• Ambassador must pass down the cookies (for oidc plugin) as well (maybe it does by default, not sure).
  ** I do think this is the case yes... Again i did a bit of testing with another bit more primitive AuthService and it seemed to be passing it along **

[Romeren]

If your service can pick up on it..... then i can add the config_token in a request header like this,

annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      auth_service: {{ include "external-auth-server.fullname" . }}
      proto: http
      add_request_headers:
             x-config_token:  <My TOKEN>

[Romeren]

Did a bit more reading up on the ambassador documentation... I found that you can add a path_prefix to the arguments, so i tried this:

      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      auth_service: {{ include "external-auth-server.fullname" . }}
      path_prefix: /verify?config_token=<My token>
      proto: http
      ---

And that seems to append the config to the routing correctly :)

[Romeren]

So..... Status update..... After updating the config with the path_prefix, i got a lot further in the process.
When i navigate to a webservice in my browser ambassador correctly, sends the request to the AuthService
The AuthService correctly, sees that the user is not logged in and redirects to keycloak.
I login with keycloak and get redirected back..... but here things goes worng....
The logs of the external-auth-service says:

{"message":"starting verify pipeline","level":"info","service":"external-auth-server"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: oidc"}
{"service":"external-auth-server","level":"info","message":"end verify pipeline with status: 302"}
{"level":"error","service":"external-auth-server","message":"error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length","stack":"Error: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length\n    at Object.decrypt (/home/eas/app/src/utils.js:45:11)\n    at server.WebServer.get (/home/eas/app/src/plugin/oauth/index.js:130:36)\n    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)\n    at next (/home/eas/app/node_modules/express/lib/router/route.js:137:13)\n    at Route.dispatch (/home/eas/app/node_modules/express/lib/router/route.js:112:3)\n    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)\n    at /home/eas/app/node_modules/express/lib/router/index.js:281:22\n    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)\n    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)\n    at middleware (/home/eas/app/node_modules/express-prom-bundle/src/index.js:164:5)\n    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/eas/app/node_modules/express/lib/router/index.js:317:13)\n    at /home/eas/app/node_modules/express/lib/router/index.js:284:7\n    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)\n    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)\n    at cookieParser (/home/eas/app/node_modules/cookie-parser/index.js:57:14)\n    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/eas/app/node_modules/express/lib/router/index.js:317:13)\n    at /home/eas/app/node_modules/express/lib/router/index.js:284:7\n    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)\n    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)\n    at urlencodedParser (/home/eas/app/node_modules/body-parser/lib/types/urlencoded.js:100:7)"}

[Romeren]

Or better formatted:

Error: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length
    at Object.decrypt (/home/eas/app/src/utils.js:45:11)
    at server.WebServer.get (/home/eas/app/src/plugin/oauth/index.js:130:36)
    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/eas/app/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/home/eas/app/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)
    at /home/eas/app/node_modules/express/lib/router/index.js:281:22
    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)
    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)
    at middleware (/home/eas/app/node_modules/express-prom-bundle/src/index.js:164:5)
    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/eas/app/node_modules/express/lib/router/index.js:317:13)
    at /home/eas/app/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)
    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)
    at cookieParser (/home/eas/app/node_modules/cookie-parser/index.js:57:14)
    at Layer.handle [as handle_request] (/home/eas/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/eas/app/node_modules/express/lib/router/index.js:317:13)
    at /home/eas/app/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/eas/app/node_modules/express/lib/router/index.js:335:12)
    at next (/home/eas/app/node_modules/express/lib/router/index.js:275:10)
    at urlencodedParser (/home/eas/app/node_modules/body-parser/lib/types/urlencoded.js:100:7)

[Romeren]

In case you want some debug info..... these are the configuration of that last call that fails:

{
    "url": "http://<External auth service host>/oauth/callback",
    "parameters": {
        "__eas_oauth_handler__": [
            "authorization_callback"
        ],
        "state": [
            "ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db4394797afe8204051fd08fa55fd101850a22d1b32fcf6798e4ab086f215ccbb2692595f08881eead3200c6bc9e184808f3725afcd16fd6c38c278d35abd32b112a8ab347c501be3bd15b487c01b45b472f7fed57faed99db4524f2746f6d8455d530eabac55a48277556a8c1fb18ba59e3887ae08d98d785d6ec734b5ec3f85472ac1"
        ],
        "session_state": [
            "da010069-bf7a-401f-b869-60a257f2c174"
        ],
        "code": [
            "07d99bfd-ecd7-482c-814b-489ed280bf02.da010069-bf7a-401f-b869-60a257f2c174.ddcb87d1-0937-49a2-af37-3253ca9520cf"
        ]
    },
    "cookies": {
        "AUTH_SESSION_ID": "da010069-bf7a-401f-b869-60a257f2c174.keycloak-0",
        "KC_RESTART": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5MDVlNzUzMC03Yjk3LTQwNjMtYjhlMi0xYjJhODBjMmM1YTEifQ.eyJjaWQiOiJsaWdodG5pbmdiYWRnZXIiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vYXBpLmxpZ2h0bmluZ2JhZGdlci5pby9lYXMvb2F1dGgvY2FsbGJhY2s_X19lYXNfb2F1dGhfaGFuZGxlcl9fPWF1dGhvcml6YXRpb25fY2FsbGJhY2siLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo1MDAwL2F1dGgvcmVhbG1zL21hc3RlciIsInJlc3BvbnNlX3R5cGUiOiJjb2RlIiwiY29kZV9jaGFsbGVuZ2VfbWV0aG9kIjoicGxhaW4iLCJyZWRpcmVjdF91cmkiOiJodHRwOi8vYXBpLmxpZ2h0bmluZ2JhZGdlci5pby9lYXMvb2F1dGgvY2FsbGJhY2s_X19lYXNfb2F1dGhfaGFuZGxlcl9fPWF1dGhvcml6YXRpb25fY2FsbGJhY2siLCJzdGF0ZSI6ImViYWE1OGMwNTAxMzQwOTM0M2U0MWJjZTRjNzAwN2IyMGEzMzE2Njk2ZDg2ZTA1YjZhNzliY2M1YjZjZWFmMTQwNzE1Njc1MzJmNDIxNzU5NWJiZjYyNjNiZjc2OWE0NWE5NGU2OGQ1OTFhNGEwZTZiNzA2OWE1NWIxMjVhN2I5MjdjNDhmZTVjMDgzNTUxMmQ2MTczN2E5ZDAzZTRmYWI2NDA5M2EzNWFkMmYwMDUzMWRlNTI5NjA3ZWM3YjYyMTJlNjI2NzFlNjhjZjljMTUwZmM4NjdmYmM0NmU2NzhiMjdiYTlhM2FiMzgwMThkYWJmNTcwZjZmZTFlZjdiNmQ3MzQ4NGMwYWMzMTE0MTY0MjdjMjA0ODY2YjBlNjk2MzFhZDM4MDdjNmZmODEwMjgwMjIxNTI3YThkYjQzOTQ3OTdhZmU4MjA0MDUxZmQwOGZhNTVmZDEwMTg1MGEyMmQxYjMyZmNmNjc5OGU0YWIwODZmMjE1Y2NiYjI2OTI1OTVmMDg4ODFlZWFkMzIwMGM2YmM5ZTE4NDgwOGYzNzI1YWZjZDE2ZmQ2YzM4YzI3OGQzNWFiZDMyYjExMmE4YWIzNDdjNTAxYmUzYmQxNWI0ODdjMDFiNDViNDcyZjdmZWQ1N2ZhZWQ5OWRiNDUyNGYyNzQ2ZjZkODQ1NWQ1MzBlYWJhYzU1YTQ4Mjc3NTU2YThjMWZiMThiYTU5ZTM4ODdhZTA4ZDk4ZDc4NWQ2ZWM3MzRiNWVjM2Y4NTQ3MmFjMSJ9fQ.PuMViFSetOaDeGTFODRNObRdXRb3pjb9INH3etOvCHc",
        "KEYCLOAK_IDENTITY": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5MDVlNzUzMC03Yjk3LTQwNjMtYjhlMi0xYjJhODBjMmM1YTEifQ.eyJqdGkiOiJiODk3YTUxOS0yMDBhLTRjMmUtYmVhMS1mMDU2NzMxZTNlMzEiLCJleHAiOjE1NjE3NTU2NDMsIm5iZiI6MCwiaWF0IjoxNTYxNzE5NjQzLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjUwMDAvYXV0aC9yZWFsbXMvbWFzdGVyIiwic3ViIjoiMzRlNjQ4OGQtNmMyNS00NDFhLTkwM2ItNDZlODdkNjRkNzhiIiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiZGEwMTAwNjktYmY3YS00MDFmLWI4NjktNjBhMjU3ZjJjMTc0Iiwic3RhdGVfY2hlY2tlciI6IkluWUtoVDVjTjBudkdhLXBLd2lNV2w0QXhmSlhwZWJvZWZ2LS1MUlFpMncifQ.eqF4K1g-PTE0UQIJT6VAmgVqk84urZK9YQ6p7-rfsJw",
        "KEYCLOAK_SESSION": "master/34e6488d-6c25-441a-903b-46e87d64d78b/da010069-bf7a-401f-b869-60a257f2c174"
    } 
}

[Romeren]

hmmm ..... that seemed to be a fuck-up on my side..... i accidentally had added some of the parameters twice in my request for the oauth callback.....

[Romeren]

Now the authentication looks like this:

{"message":"starting verify pipeline","level":"info","service":"external-auth-server"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: oidc"}
{"service":"external-auth-server","level":"info","message":"end verify pipeline with status: 302"}
{"service":"external-auth-server","level":"info","message":"redirecting browser to: \"http://undefinedundefined/?__eas_oauth_handler__=authorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947aae6eb5e340d7785d8c9cf44721953e078f99bfec01520658a214040f390cadaeeb242eb65181023ad42948d5b7afedc8d003a651e6499828b296b67e36952e4192f395dbc0635d44e47f5033de55aaf916a35512de89f9170a666813993dd121e7cf6619ef227510ce1bc3284d2525265efcbde7c47c5f7a47a4f1c235d8915&session_state=f894c457-bd69-4a1b-8ed1-f4d857e2afb1&code=e92f568b-5cad-463a-8cb0-21fdb6d67fe9.f894c457-bd69-4a1b-8ed1-f4d857e2afb1.ddcb87d1-0937-49a2-af37-3253ca9520cf\""}

I wonder what the undefinedundefined url is should have been

[travisghansen]

Wow great progress! This is all awesome info. That's likely related to the original uri detection. One thing that would be helpful now is to the the log level to silly and I dump the full request data to the auth service out.

That will help detect what headers are being passed and how they're formatted. Send that over and I'll read the doc you send previously and we'll get that last piece knocked out...I think you're very close, thanks for the willingness to help out!

[travisghansen]

I've read the doc and noticed that I'll also need to make the auth service support all the http verbs as ambassador doesn't always make a GET request like traefik does. That will be easy to fix mostly just an FYI.

So yeah, at this point just send me the output after enabling silly. You see an object dumped...it's the very first for a given request to the auth service that includes a headers and body property.

[Romeren]

This must be the debug message you were looking for:

{
   "service": "external-auth-server", "level": "verbose",
   "message": "parsed state redirect uri: 
    {\
         "scheme\": \"http\",\
         "host\":\ "undefinedundefined\",\
         "path\": \"\",\
         "reference\": \"absolute\"
     }"
}

[travisghansen]

Nope, it's before that even..you must have the log level set to silly. It's literally the first line of the endpoint handler.

https://github.com/travisghansen/external-auth-server/blob/master/src/server.js#L89

[Romeren]

A bit more log info:

{
    "service": "external-auth-server",
    "level": "debug",
    "message": "plugin response {\"statusCode\":302,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[[\"_eas_oauth_csrf\",\"5RkmDrKDRQA1ssG16eXMge/vjacF09+7NDI6mcdDeqPRNPe7A2+XGMVCxZUge4Ta\",{\"expires\":\"2019-06-29T00:55:20.118Z\",\"httpOnly\":true,\"signed\":true}]],\"clearCookies\":[],\"headers\":{\"Location\":\"http://keycloak-http.default.svc.cluster.local/auth/realms/master/protocol/openid-connect/auth?client_id=lightningbadger&scope=openid%20email%20profile&response_type=code&redirect_uri=http%3A%2F%2Fapi.lightningbadger.io%2Feas%2Foauth%2Fcallback%3F__eas_oauth_handler__%3Dauthorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947fcecae51791d42b94c0e258e8f929c46a3832b1b2a367a11c30aa2fcdc7bbb82cf5f5514f11077db17a54b4d8606148570ca60866846f37d58119767c2e07c1c4e1aa4cf2af27e8629a031678b0ff7b7741180ca67727c308855a22015b382a6e6a7727f8cf550e903557d288dd05fd068707d80be842a5e2f4b177f8b582ae4\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"oidc\",\"issuer\":{\"discover_url\":\"http://keycloak-http.default.svc.cluster.local/auth/realms/master/.well-known/openid-configuration\"},\"client\":{\"client_id\":\"lightningbadger\",\"client_secret\":\"4dbd29da-5b81-417e-a230-abad914de57e\"},\"scopes\":[\"openid\",\"email\",\"profile\"],\"redirect_uri\":\"http://api.lightningbadger.io/eas/oauth/callback\",\"features\":{\"cookie_expiry\":true,\"userinfo_expiry\":true,\"session_expiry\":true,\"session_expiry_refresh_window\":86400,\"session_retain_id\":true,\"refresh_access_token\":true,\"fetch_userinfo\":true,\"introspect_access_token\":false,\"authorization_token\":\"access_token\"},\"assertions\":{\"exp\":true,\"nbf\":true,\"iss\":true,\"userinfo\":[],\"id_token\":[]},\"cookie\":{\"name\":\"_eas_oauth_session\",\"domain\":null,\"path\":\"/\"},\"headers\":{},\"pcb\":{}}}}"
}
{
    "service": "external-auth-server",
    "level": "info",
    "message": "end verify pipeline with status: 302"
}
{
    "service": "external-auth-server",
    "level": "silly",
    "message": "{\"headers\":{\"host\":\"localhost:5000\",\"user-agent\":\"python-requests/2.22.0\",\"accept-encoding\":\"gzip, deflate\",\"accept\":\"*/*\",\"x-forwarded-for\":\"10.244.2.219\",\"x-forwarded-proto\":\"http\",\"x-envoy-internal\":\"true\",\"x-request-id\":\"8407eea6-4200-4e9d-ac18-7b3c90b0dfaa\",\"x-envoy-expected-rq-timeout-ms\":\"3000\",\"x-envoy-original-path\":\"/eas/oauth/callback?__eas_oauth_handler__=authorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947fcecae51791d42b94c0e258e8f929c46a3832b1b2a367a11c30aa2fcdc7bbb82cf5f5514f11077db17a54b4d8606148570ca60866846f37d58119767c2e07c1c4e1aa4cf2af27e8629a031678b0ff7b7741180ca67727c308855a22015b382a6e6a7727f8cf550e903557d288dd05fd068707d80be842a5e2f4b177f8b582ae4&session_state=2011b2a2-04ce-4a79-bf9a-c0cced24031c&code=d970c333-c559-42f5-8d82-16b44a11aeef.2011b2a2-04ce-4a79-bf9a-c0cced24031c.ddcb87d1-0937-49a2-af37-3253ca9520cf\",\"content-length\":\"0\"},\"body\":{}}"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "parsed state redirect uri: {\"scheme\":\"http\",\"host\":\"undefinedundefined\",\"path\":\"\",\"reference\":\"absolute\"}"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "parsed request uri: {\"path\":\"/oauth/callback\",\"query\":\"__eas_oauth_handler__=authorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947fcecae51791d42b94c0e258e8f929c46a3832b1b2a367a11c30aa2fcdc7bbb82cf5f5514f11077db17a54b4d8606148570ca60866846f37d58119767c2e07c1c4e1aa4cf2af27e8629a031678b0ff7b7741180ca67727c308855a22015b382a6e6a7727f8cf550e903557d288dd05fd068707d80be842a5e2f4b177f8b582ae4&session_state=2011b2a2-04ce-4a79-bf9a-c0cced24031c&code=d970c333-c559-42f5-8d82-16b44a11aeef.2011b2a2-04ce-4a79-bf9a-c0cced24031c.ddcb87d1-0937-49a2-af37-3253ca9520cf\",\"reference\":\"relative\"}"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "parsed redirect uri: {\"scheme\":\"http\",\"host\":\"undefinedundefined\",\"path\":\"\",\"query\":\"__eas_oauth_handler__=authorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947fcecae51791d42b94c0e258e8f929c46a3832b1b2a367a11c30aa2fcdc7bbb82cf5f5514f11077db17a54b4d8606148570ca60866846f37d58119767c2e07c1c4e1aa4cf2af27e8629a031678b0ff7b7741180ca67727c308855a22015b382a6e6a7727f8cf550e903557d288dd05fd068707d80be842a5e2f4b177f8b582ae4&session_state=2011b2a2-04ce-4a79-bf9a-c0cced24031c&code=d970c333-c559-42f5-8d82-16b44a11aeef.2011b2a2-04ce-4a79-bf9a-c0cced24031c.ddcb87d1-0937-49a2-af37-3253ca9520cf\",\"reference\":\"absolute\"}"
}
{
    "service": "external-auth-server",
    "level": "info",
    "message": "redirecting browser to: \"http://undefinedundefined/?__eas_oauth_handler__=authorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db43947fcecae51791d42b94c0e258e8f929c46a3832b1b2a367a11c30aa2fcdc7bbb82cf5f5514f11077db17a54b4d8606148570ca60866846f37d58119767c2e07c1c4e1aa4cf2af27e8629a031678b0ff7b7741180ca67727c308855a22015b382a6e6a7727f8cf550e903557d288dd05fd068707d80be842a5e2f4b177f8b582ae4&session_state=2011b2a2-04ce-4a79-bf9a-c0cced24031c&code=d970c333-c559-42f5-8d82-16b44a11aeef.2011b2a2-04ce-4a79-bf9a-c0cced24031c.ddcb87d1-0937-49a2-af37-3253ca9520cf\""
}

[Romeren]

If you are wondering about the user-agent...... im using python to intercept all the calls and redirect so that i can see the headers, content, parameters and cookie on every step along the way

[travisghansen]

Ok, that 3rd entry is what I'm after. Can you send that on a fresh request to your end user service? When you send it also send the actual url you requested like: http://localhost:5000/some/path?foo=bar

[travisghansen]

Adding that in as an additional proxy may be harmful to uri detection but we'll see...

[Romeren]

I think i have found the issue......
In the first step, when i try to connect to my test web-page localhost:5000/test
I get redirected to external-auth-server, and this is the output in the logs:

{
    "service": "external-auth-server",
    "level": "silly",
    "message": "{\"headers\":{\"host\":\"localhost:5000\",\"content-length\":\"0\",\"x-forwarded-for\":\"10.244.2.219,10.244.2.219\",\"x-forwarded-proto\":\"http\",\"user-agent\":\"python-requests/2.22.0\",\"x-envoy-internal\":\"true\",\"x-envoy-expected-rq-timeout-ms\":\"5000\"},\"body\":{}}"
}
{
    "message": "starting verify pipeline",
    "level": "info",
    "service": "external-auth-server"
}
{
    "service": "external-auth-server",
    "level": "debug",
    "message": "config token: {\"eas\":{\"plugins\":[{\"type\":\"oidc\",\"issuer\":{\"discover_url\":\"http://keycloak-http.default.svc.cluster.local/auth/realms/master/.well-known/openid-configuration\"},\"client\":{\"client_id\":\"lightningbadger\",\"client_secret\":\"4dbd29da-5b81-417e-a230-abad914de57e\"},\"scopes\":[\"openid\",\"email\",\"profile\"],\"redirect_uri\":\"http://api.lightningbadger.io/eas/oauth/callback\",\"features\":{\"cookie_expiry\":true,\"userinfo_expiry\":true,\"session_expiry\":true,\"session_expiry_refresh_window\":86400,\"session_retain_id\":true,\"refresh_access_token\":true,\"fetch_userinfo\":true,\"introspect_access_token\":false,\"authorization_token\":\"access_token\"},\"assertions\":{\"exp\":true,\"nbf\":true,\"iss\":true,\"userinfo\":[],\"id_token\":[]},\"cookie\":{},\"headers\":{}}]},\"iat\":1561717749,\"audMD5\":\"3d3a4f625f913e2b3bc69ed508a6d1a6\"}"
}
{
    "service": "external-auth-server",
    "level": "info",
    "message": "starting verify for plugin: oidc"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "parent request info: {\"uri\":\"http://undefinedundefined\",\"parsedUri\":{\"scheme\":\"http\",\"host\":\"undefinedundefined\",\"path\":\"\",\"reference\":\"absolute\"},\"parsedQuery\":{}}"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "audMD5: 3d3a4f625f913e2b3bc69ed508a6d1a6"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "cooking name: _eas_oauth_session"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "redirect_uri: http://api.lightningbadger.io/eas/oauth/callback?__eas_oauth_handler__=authorization_callback"
}
{
    "service": "external-auth-server",
    "level": "verbose",
    "message": "callback redirect_uri: http://keycloak-http.default.svc.cluster.local/auth/realms/master/protocol/openid-connect/auth?client_id=lightningbadger&scope=openid%20email%20profile&response_type=code&redirect_uri=http%3A%2F%2Fapi.lightningbadger.io%2Feas%2Foauth%2Fcallback%3F__eas_oauth_handler__%3Dauthorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db439474f8625b965e9b072c43b66ffbe49c9b636112506a0e8c94ddda94cb31ac1d78ba69a1886c18711a9c2ff6152abea0263575103459a10c01033beab832965e891120af46c3468e44ec48a55d1d9e65ddcacd8d9ceb8b93f870e1baf0a5c98c5628a88b9adf5ffb394fb7504bbecfbc9d2cfb830adec4311f31fc468ff5e227cc2"
}
{
    "service": "external-auth-server",
    "level": "debug",
    "message": "plugin response {\"statusCode\":302,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[[\"_eas_oauth_csrf\",\"xHdNpafpTQ/4Z2YqM6lX0kJIGupexvu/x+CuvQoUIySraMEbRZxiAMMP3k8f7/VJ\",{\"expires\":\"2019-06-29T01:28:29.022Z\",\"httpOnly\":true,\"signed\":true}]],\"clearCookies\":[],\"headers\":{\"Location\":\"http://keycloak-http.default.svc.cluster.local/auth/realms/master/protocol/openid-connect/auth?client_id=lightningbadger&scope=openid%20email%20profile&response_type=code&redirect_uri=http%3A%2F%2Fapi.lightningbadger.io%2Feas%2Foauth%2Fcallback%3F__eas_oauth_handler__%3Dauthorization_callback&state=ebaa58c05013409343e41bce4c7007b20a3316696d86e05b6a79bcc5b6ceaf14071567532f4217595bbf6263bf769a45a94e68d591a4a0e6b7069a55b125a7b927c48fe5c0835512d61737a9d03e4fab64093a35ad2f00531de529607ec7b6212e62671e68cf9c150fc867fbc46e678b27ba9a3ab38018dabf570f6fe1ef7b6d73484c0ac311416427c204866b0e69631ad3807c6ff810280221527a8db439474f8625b965e9b072c43b66ffbe49c9b636112506a0e8c94ddda94cb31ac1d78ba69a1886c18711a9c2ff6152abea0263575103459a10c01033beab832965e891120af46c3468e44ec48a55d1d9e65ddcacd8d9ceb8b93f870e1baf0a5c98c5628a88b9adf5ffb394fb7504bbecfbc9d2cfb830adec4311f31fc468ff5e227cc2\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"oidc\",\"issuer\":{\"discover_url\":\"http://keycloak-http.default.svc.cluster.local/auth/realms/master/.well-known/openid-configuration\"},\"client\":{\"client_id\":\"lightningbadger\",\"client_secret\":\"4dbd29da-5b81-417e-a230-abad914de57e\"},\"scopes\":[\"openid\",\"email\",\"profile\"],\"redirect_uri\":\"http://api.lightningbadger.io/eas/oauth/callback\",\"features\":{\"cookie_expiry\":true,\"userinfo_expiry\":true,\"session_expiry\":true,\"session_expiry_refresh_window\":86400,\"session_retain_id\":true,\"refresh_access_token\":true,\"fetch_userinfo\":true,\"introspect_access_token\":false,\"authorization_token\":\"access_token\"},\"assertions\":{\"exp\":true,\"nbf\":true,\"iss\":true,\"userinfo\":[],\"id_token\":[]},\"cookie\":{\"name\":\"_eas_oauth_session\",\"domain\":null,\"path\":\"/\"},\"headers\":{},\"pcb\":{}}}}"
}
{
    "service": "external-auth-server",
    "level": "info",
    "message": "end verify pipeline with status: 302"
}

As you can see, in info block 5
** "parent request info: {"uri":"http://undefinedundefined\" **
i have lost the information about the original site that i was trying to access

[Romeren]

In other words.... the missing link is in src/util. js line 80
originalRequestURI += req.headers["x-forwarded-host"]

the header x-forwarded-host is missing or empty

[travisghansen]

Yes, exactly. That's what extra logic I need to code to work better with ambassador (should be very simple once I have a better idea of the passed headers).

Notice the first block, is there any way you can configure ambassador to send down the path info there? ie: /test along with any applicable query params? I'm a little perplexed as the doc says it should be sending down the Location header and I don't see it...

In the end, I just need enough info in the headers to completely reconstruct the full originally requested URI.

[travisghansen]

Actually location is for the response (duh) sorry. Anyhow, if you know of a way to configure ambassador to send down the full request details in some shape or form we can get you going quickly..

[travisghansen]

It probably would be helpful to remove the python proxy at this point and observe output without that additional layer too..

[travisghansen]

I've read the ambassador doc more thoroughly and it looks like the approach is even more different than I originally thought. You mentioned earlier you can inject the token as a header, have you tried that with another service and does it work (header injection)? I think I know how ambassador can be tooled to work nicely but we'll need that for sure from what I can tell..

[Romeren]

Yep, header injection is possible and quite simple, as long as the value is static.... i dont know if we can do dynamic headers if that is what you are thinking :)

[Romeren]

I seem to be missing something somewhere..... i end up in a endless loop, with request to page => redirect to authservice => redirect to keycloak => redirect to authservice => redirect back to the initial web-page which redirects to authservice (authentication fails ) => redirects to keycloak
hahaha

[travisghansen]

Ouch.

Can you send me all the eas output for 1 cycle of that process?

Were you able to get any more details on what headers can be (or are) being passed down by bypassing the python proxy? I noticed in one of the first examples you sent it had this header x-envoy-original-path for example but did not see it in the later examples.

I'm going to do some preliminary work on better supporting ambassador and get into my next branch/images. I'm confident we can get it working with fairly limited effort.

Regarding the header, yes static. Ambassador takes a very different approach to this than traefik does...contrary to my original perception. I uses a sort of overlay idea whereas traefik just requests an (static) external arbitrary URL.

What I intend to do is the following:

1. Add a wildcard path to the eas service, something lke /ambassador/verify
2. Ensure that path/route can handle all HTTP verbs
3. Add support for parsing the token from a header instead of a query param

At that point the only missing piece will be proper URI reconstruction (only really required for oidc/oauth2 plugins) which is obviously trivial once we get ambassador configured/setup to send headers that support that.

[Romeren]

I just sat up traefik to see if i can get it working with that.... ill circle back to this one, once i have verified that all of the other components are working exactly as they should..... ill give a update once im further in the process :) ..... but thank you so much for your help so far

[travisghansen]

OK, I've already coded everything to support ambassador too minus the host stuff.

I don't even need the path portion as that can be derrived based on the ambassador spec and how it works. But I still need the original scheme/host/port somehow...

[travisghansen]

Worst case scenario if ambassador can't send that info down as a header(s) then I can introduce a simple configuration param that's specific to ambassador to be passed in as a static value via header injection...pretty simple stuff.

I am headed out shortly for the weekend so likely won't be very responsive until Monday (in North America) comes around.

[Romeren]

Hey... no worries.... im going off for weekend in -2 hours as well ;)

[Romeren]

Right now im just testing and yes i have two plugins in one config.... however, before i move this to production, im going to split it into one config for the front end and one for api's....
Nice, where can i find the code for the jwks for the endpoint...?

[travisghansen]

You can use a single config and use the pcb functionality to target a specific plugin based on the nature of the request (ie: contains Authorization: Bearer xxxxx header).

I'll just clean up the jwks code and commit it to next branch in the next day or so.

I've never used tokens from keycloak like that (only the authorization code flow so far), if you've got a link you used send it over so I can see the general idea.

If you're interested in pcb let me know.

Going back to the very beginning when you're ready to revisit server-side tokens let me know as well.

[Romeren]

Awesome!... ill use the pcb functionality and keep a close eye on next brance 👍 ...
Regarding the pcb, this is what you are referring to, right

pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.authorization",
              rule: {
                method: "regex",
                value: "/^bearer/i",
                negate: true
              }
            }
          ],
          stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.authorization",
              rule: {
                method: "regex",
                value: "/^bearer/i",
              }
            }
          ]
        }

Im currently working on the documentation of the setup, all parameters and steps to set it up, can definitely share it all with you once im done

[travisghansen]

Yeah, add that block to the jwt plugin...you should then see 40X response if you send in a bearer token that's not valid instead of a redirect from the oidc plugin.

Probably best to put the jwt plugin before the oidc plugin as well..

[travisghansen]

Also note that if the authorization header with bearer is not present the plugin will be skipped and go straight to oidc in your case..

[Romeren]

Noted, ill try that out.....
Last thing, for the redis-ha config,
i just need to set:

redis-ha:
  enabled: true
  auth: true
  redisPassword: <whatever password>

and it should be all good...? ~It seems to work

[travisghansen]

See the current example in the readme, you also need to set the storeOpts.* params for that app to actually use redis.

[Romeren]

Super 👍

[travisghansen]

I recommend reading this to understand some of the considerations with oidc: https://github.com/travisghansen/external-auth-server/blob/master/OAUTH_PLUGINS.md

[Romeren]

Setup

We are about to take you onto a journey... a long lasting adventure ~with ups' downs' and and a little bit of personal growth.

In this tutorial we'll be setting up a star-constallation of three interconnected micro-services that will enable SSO.

• Setup a database for keycloak
• Setup keycloak as a SSO authentication provider
• Setup external-auth-server (eas) as a authentication service middle-ware between traefik and keycloak
• Configure eas with plugins for openid-connect ~OIDC (web auth) and JWT (json-web-tokens for apis)
• Setup of traefik as an ingress provider
• Test that it is all working

Setup a database for keycloak

Repo OIDC/Oauth2 Provider:

https://github.com/helm/charts/tree/master/stable/keycloak

Before we get going on setting up anything, we need a place to percist data for keycloak, so that multiple instances can run.

If you have a db, you can use that one, thats fine.... otherwise go ahead and set one up now.

Create:

    db: keycloakdb,
    user: keycloakusr (and a password for that one)

Open keycloak_values.yaml and edit the following fields to match your setup:

    deployPostgres: false
    dbVendor: postgres (dbType)
    dbHost: <database service name.namespace.svc.cluster.local> (host)
    dbPort: 5432
    dbName: keycloakdb
    dbUser: keycloakusr
    dbPassword: putAnEasyToBreakPasswordHere_like_password1_or_something

Set also set a default username and password for the admin user of keycloak:

keycloak:
  username: keycloak
  password: PleaseNotATooHardPasswordHere!

Next set host and tls for dashboard:

keycloak:
  ingress:
    enabled: true
    hosts: [auth.<Your domain name>] (endpoints)
    annotations:
      kubernetes.io/ingress.class: traefik ( what ingress to use)
    tls: 
      - hosts: 
        - auth.<Your domain name> (endpoints for https)

Note this many not get an appropriate https cert unless cert-manager and an ngnix ingress controller are in place.

Also, you probaly wanna put a theme onto the login page of the keycloak.
You can add these by putting it into the emptyDir called "theme" which is declared in the keycloak_values.yaml.
But how do i get it in there you might ask.... Good question, since emptyDirs only exists in the lifespan of the container, you'll have to put them there after the pod is created.
But isnt there a way to do this programmatically, i mean come-on, i was expecting to have my theme in a git repo....

Well i got some stuff for you then. Use:

  extraInitContainers: |
    - name: theme-provider
      image: ansilh/debug-tools
      imagePullPolicy: IfNotPresent
      args:
        - git
        - clone
        - <Theme from git repo> (git repo)
        - theme
      volumeMounts:
        - name: theme
          mountPath: /theme

Now you are all ready to go...

Setup keycloak as an SSO authentication provider

To install keycloak simply run:

helm upgrade --install -f keycloak_values.yaml keycloak --tls codecentric/keycloak

You can port forward to the service using:

kubectl port-forward svc/keycloak-http 5000:80

Now login go to admin console and login with keycloak user.

This configuration by default happens in the master realm:

• Create a new client and give it a name
• Set Service Accounts Enabled
• And add Valid Redirect URIs (i.e. the sites that you want to protect)
• Go to the Credentials tab and get the client_secret
• Add whatever users you need as well

Keycloak done!
Eventually you may want to find a client for interacting with Keycloak for user administration
- https://python-keycloak-client.readthedocs.io/en/latest/

Setup external-auth-server eas:

• Get keycloak key for signing JWT-tokens by running this python script (we'll need this in a later step):

from jwkest.jwk import load_jwks_from_url
keys = load_jwks_from_url("<keycloak-url>/auth/realms/master/protocol/openid-connect/certs")
keys[0].get_key().export_key() # will give you a pem encoded key

• Clone the following repo:

https://github.com/travisghansen/external-auth-server

• Create a file called eas_values.yaml
• Setup eas_values.yaml by generating random strings for these fields:

configTokenSignSecret: <random1>
configTokenEncryptSecret: <random2>
issuerSignSecret: <random3>
issuerEncryptSecret: <random4>
cookieSignSecret: <random5>
cookieEncryptSecret: <random6>
sessionEncryptSecret: <random7>

• Configure eas_values.yaml ingress:

ingress:
  enable: true
  annotations:
    kubernetes.io/ingress.class: traefik
  hosts:
    - eas.<public-host-dns>
  paths:
    - '/'
  tls:
  - hosts: 
    - eas.<public-host-dns>

• Configure redis-ha services for enabling multiple replicas of eas:

redis-ha:
  enabled: true
  auth: true
  redisPassword: <You gotta put a password here>

storeOpts:
  store: ioredis
  password: <You gotta put a password here>
  name: mymaster
  sentinels:
    - host: eas-redis-ha-announce-0
      port: 26379
    - host: eas-redis-ha-announce-1
      port: 26379
    - host: eas-redis-ha-announce-2
      port: 26379
  keyPrefix: "eas:"

• Install the service:

helm upgrade --install -f eas_values.yaml --tls eas ./chart/

• Install the repo with npm install
• Make a copy of bin/generate-config-token-stores.js and call it oidc-generate-config.js
• Fill in the plugins with:

plugins: [
      {
        type: "jwt",
        realm: "<Realm as configured in keycloak>",
        header_name: "Authorization",
        scheme: "Bearer",
        config: {
          secret: `-----BEGIN PUBLIC KEY-----
<Add the output of step 1 : Get keycloak key for signing JWT-tokens by running (we'll need this in a later step) >
-----END PUBLIC KEY-----`
        },
        assertions: {
          exp: true,
          nbf: true,
          iss: true,
          userinfo: [],
          id_token: []
        },
        pcb: {
          skip: [
            {
              query_engine: "jp",
              query: "$.req.headers.authorization",
              rule: {
                method: "regex",
                value: "/^bearer/i",
                negate: true
              }
            }
          ],
          stop: [
            {
              query_engine: "jp",
              query: "$.req.headers.authorization",
              rule: {
                method: "regex",
                value: "/^bearer/i",
              }
            }
          ]
        }
      },{
        type: "oidc",
        issuer: {
            discover_url: "https://<keycloak-url>/auth/realms/<realm as confugured in keycloak>/.well-known/openid-configuration",
        },
        client: {
            client_id: "name of the client, see configuration of keycloak",
            client_secret: "client_secret, see configuration of keycloak"
        },
        scopes: ["openid", "email", "profile"], // must include openid
        features: {
            cookie_expiry: true,
            userinfo_expiry: true,
            session_expiry: true,
            session_expiry_refresh_window: 86400,
            session_retain_id: true,
            refresh_access_token: true,
            fetch_userinfo: true,
            introspect_access_token: false,
            authorization_token: "access_token"
        },
        assertions: {
            exp: true,
            nbf: true,
            iss: true,
            userinfo: [],
            id_token: []
        },
        cookie: {
        },
        headers: {},
      }
    ] // list of plugin definitions, refer to PLUGINS.md for details

• Generate config_token later referenced as <CONFIG_TOKEN URL SAFE> by running:

EAS_CONFIG_TOKEN_SIGN_SECRET="<random1 from values>" \
EAS_CONFIG_TOKEN_ENCRYPT_SECRET="<random2 from values>" \
node bin/oidc-generate-config.js

Setup Traefik

Traefik is available from:

https://github.com/helm/charts/tree/master/stable/traefik

Before you start deploying this, there are a few things you will need

Traefik has a dashboard for overview of endpoint to secure it, add the following to the values:

dashboard:
  enabled: true
  domain: traefik.<your-domain>
  serviceType: NodePort
  service: {}
  ingress: 
    annotations:
      kubernetes.io/ingress.class: traefik
      ingress.kubernetes.io/auth-response-headers: X-Userinfo, X-Id-Token, X-Access-Token, Authorization
      ingress.kubernetes.io/auth-type: forward
      ingress.kubernetes.io/auth-url: https://eas.<your domain>/verify?fallback_plugin=0&config_token=<CONFIG_TOKEN URL SAFE>
    tls:
      - hosts:
        - traefik.<your-domain>

In addition i have added configuration to support SSL to traefik_values.yaml:

ssl:
  enabled: true
  enforced: true
  permanentRedirect: true
  generateTLS: true

acme:
  enabled: true
  email: [email protected]
  challengeType: tls-alpn-01

Once ready run:

helm upgrade --install --tls --version 1.61.0 -f traefik_values.yaml traefik stable/traefik

NOTE THE --version 1.61.0 (i could not get it working with current version) theres a ticket open and people are working on it. See this

https://github.com/containous/traefik/issues/4850

Configure ingress

Use the following to track the state of the traefik deployment:

kubectl get svc --watch

Once the traefik services' ip no longer says PENDING and has an ip, it is time to set-up the dns.

Go to your dns provider. I believe we're using transip.nl ;)
Here map a domain-name to the ip of the newly created ingress.

Test that it is working

You can try and navigate to the dashboard page for Traefik or you can be fancy and test the JWT plugin by running this py script:

import requests
import sys

args = sys.argv
args = {args[x]: args[x+1] for x in range(1, len(args), 2)}

user = args['-u']
password = args['-p']

keycloak_url = args['-keycloak']
protected_url = args['-protected-site']
ci = args['-client_id']
cs = args['-client_secret']

payload = dict(
    response_type='code',
    scope=["openid",'email', 'profile'],
    username=args['-u'],
    password=args['-p'],
    grant_type=['password'],
    code='',
    redirect_uri=protected_url,
    client_id=ci,
    client_secret=cs
)

res = requests.post(keycloak_url + 'auth/realms/master/protocol/openid-connect/token', data=payload)
res = res.json()
token = res.get('access_token')
print('LOGIN and get token:', token)

echo = requests.get(protected_url, headers={
    'Authorization': f"Bearer {token}"
}, allow_redirects=False)

print(echo)

[travisghansen]

That's a lot to digest! I'll share a few comments on the eas aspect:

• bin/generate-config-token-stores.js should probably be bin/generate-config-token.js
• the jwt realm parameter is actually what gets sent back in failed auth requests as part of the WWW-Authenticate header (it only indirectly relates to keycloak realms). This is the way docker registry works for example. The idea is if the request responds with an error it's effectively telling the client "go authenticate here". To that end, you'd likely want to set it to keycloak_url + 'auth/realms/master/protocol/openid-connect/token. A client hypothetically could parse that value when appropriate and then post username/password to get a new token etc.

Thanks for sharing your setup!

[Romeren]

• bin/generate-config-token-stores.js should probably be bin/generate-config-token.js
  Yes you are correct, that is a mistake
• the jwt realm parameter is actually what gets sent back in failed auth requests...
  Noted, makes sense... Also since the api's mainly is for machine to machine, the authentication should also follow the Client Credentials Grant Flow, with only client_id and client_secret.

In addition, this is just an example, i do not think securing infrastructure (as i did) with oidc makes sense, unless roles and or different clients get set up in keycloak to differentiate what different users have access to.

• Thanks for sharing your setup!
  Any time..! and thank you for the help and the development of this awesome piece of software.

[Romeren]

Hey Travis,
You mentioned earlier that you had a working example with ambassador...? -Can you elaborate on how you got passed the issue of the missing forwarded headers.
I'm currently running into some issues when securing a gRPC endpoint with traefik (unrelated to eas), and i might need to switch back to ambassador since they support that part.

[travisghansen]

There is a powerful assertion engine that can be used to run assertions against the various tokens for both oidc and jwt. With keycloak you could send group membership down for example and have eas deny access based on groups or any other arbitrary data point (claims in jwt parlance).

I haven't documented the ambassador stuff but it would be good to get feedback regardless. Just setup your auth service as you did previously but use the file below to generate your prefix: https://github.com/travisghansen/external-auth-server/blob/master/bin/generate-ambassador-prefix.js

Let me know how it goes. Given ambassador's limitations you can only have 1 configuration/token per auth service/'cluster' which may be quite inhibiting..

[Romeren]

Kweel.... i'll try that out as well... in the meantime i got secure tls connection between gRCP-client and traefik and regular http/2 connection between traefik and gRPC micro-service with OIDC authentication going so thats done :)

[travisghansen]

Very good!

[Romeren]

Do you want doc on that as well...?

[travisghansen]

Sure.

[Romeren]

To get the ingress going i added:

kind: Ingress
metadata:
annotations:
    ingress.kubernetes.io/protocol: h2c

Which made the traefik frontend and backend communicate with HTTP/2 or h2c.
Client authenicates with keycloak to get a JWT like previously.
However, grpc does not support Authorization header, since it essentially is transmitting vie http/2 as well, and therefore upper case letters are not allowed. Instead i use authorization: . And pass it to the gRPC stub like this:

res = stub.Echo(echo_pb2.envelope(message="Test 1 2 3 Test"), metadata=[('authorization': 'Bearer whatever')])

In addition just so other people can find it (because i could not), to establish a secure connection from gRPC with a server and validating the servers certificate i used grpc.ssl_channel_credentials():

channel = grpc.secure_channel(f'{domain}:443', grpc.ssl_channel_credentials())

However it is also necessary to add ssl enforced on the Traefik configuration

[mlushpenko]

hi @travisghansen, I tried reading and following this huge thread to get ambassador working with Identity Server 4 (oidc). I am getting 403 with the following error and nothing in aes logs even with silly log level, so I guess ambassador can't reach it. Here is log from ambassador:

ACCESS [2019-07-26T13:24:23.322Z] "GET / HTTP/1.1" 403 UAEX 0 0 5000 - "192.168.0.4" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" "4e6fcc04-50eb-4156-9bd0-8161e8a9e38d" "demo.hal24k.nl:8443" "-"

Here is my config for eas (I used generate_ambassador_prefix.js for prefix):

getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind:  AuthService
      name:  authentication
      auth_service: https://eas.hal24k.nl:8443
      path_prefix: /ambassador/verify-params-url/%7B%22config_token%22%3A__REDACTED__%3D%22%2C%22fallback_plugin%22%3A0%7D
      allowed_request_headers:
        - authorization
      include_body:
        max_bytes: 4096
        allow_partial: true
      failure_mode_allow: false
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: eas_mapping
      host: eas.hal24k.nl:8443
      prefix: /
      bypass_auth: true
      service: eas-external-auth-server.external-auth-server:80

eas.hal24k.nl:8443 is accessible from private network, I checked metrics (see below).

I've found your project somewhere in DEX issues if I am not mistaken and decided to give it a try hoping it support group claims and maybe custom claims with oidc (DEX doesn't dexidp/dex#1065).

We may decide to use Istio ingress, so can help you with testing that as well, but spent some time with Ambassador so would like to get it working (hoping you are interested in that as well to have more material for docs/use cases )

# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 1.2999229999999993 1564147639483

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.36606100000000036 1564147639483

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.6659840000000008 1564147639483

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1564145744

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 64819200 1564147639484

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1201950720 1564147639484

# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 117469184 1564147639484

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 23 1564147639483

# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 809837

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0.000508906 1564147639483

# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Socket"} 3 1564147639483
nodejs_active_handles{type="Server"} 1 1564147639483

# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 4 1564147639483

# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge
nodejs_active_requests{type="FSReqWrap"} 2

# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 2 1564147639483

# HELP nodejs_heap_size_total_bytes Process heap size from node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 31756288 1564147639483

# HELP nodejs_heap_size_used_bytes Process heap size used from node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 26202176 1564147639483

# HELP nodejs_external_memory_bytes Nodejs external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 505687 1564147639483

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 524288 1564147639483
nodejs_heap_space_size_total_bytes{space="new"} 1048576 1564147639483
nodejs_heap_space_size_total_bytes{space="old"} 22777856 1564147639483
nodejs_heap_space_size_total_bytes{space="code"} 1572864 1564147639483
nodejs_heap_space_size_total_bytes{space="map"} 2109440 1564147639483
nodejs_heap_space_size_total_bytes{space="large_object"} 3723264 1564147639483

# HELP nodejs_heap_space_size_used_bytes Process heap space size used from node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 35200 1564147639483
nodejs_heap_space_size_used_bytes{space="new"} 1027936 1564147639483
nodejs_heap_space_size_used_bytes{space="old"} 20617920 1564147639483
nodejs_heap_space_size_used_bytes{space="code"} 964448 1564147639483
nodejs_heap_space_size_used_bytes{space="map"} 1180872 1564147639483
nodejs_heap_space_size_used_bytes{space="large_object"} 2377472 1564147639483

# HELP nodejs_heap_space_size_available_bytes Process heap space size available from node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 480384 1564147639483
nodejs_heap_space_size_available_bytes{space="new"} 3232 1564147639483
nodejs_heap_space_size_available_bytes{space="old"} 1666440 1564147639483
nodejs_heap_space_size_available_bytes{space="code"} 388192 1564147639483
nodejs_heap_space_size_available_bytes{space="map"} 884520 1564147639483
nodejs_heap_space_size_available_bytes{space="large_object"} 8736951808 1564147639483

# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v10.16.0",major="10",minor="16",patch="0"} 1

# HELP http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.003",status_code="200",method="GET",path="/ping"} 377
http_request_duration_seconds_bucket{le="0.03",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_bucket{le="0.1",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_bucket{le="0.3",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_bucket{le="1.5",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_bucket{le="10",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_bucket{le="+Inf",status_code="200",method="GET",path="/ping"} 378
http_request_duration_seconds_sum{status_code="200",method="GET",path="/ping"} 0.10201838199999992
http_request_duration_seconds_count{status_code="200",method="GET",path="/ping"} 378

# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1

[travisghansen]

@mlushpenko generally looks like you're on the right track, do you have any logs from the auth service containers you can send?

I think istio does the same overlay behavior as ambassador (meaning, I don't think ambassador is doing anything special to the envoy config) so hypothetically running in instio would be the same general idea (I may rename the route if that thinking is correct but I don't have much experience with istio/envoy).

It may be good to open another issue too, as you said this is quite long. I simply left it open until I commit support for jwks in the jwt plugin :)

[travisghansen]

emissary-ingress/emissary#1764

[travisghansen]

I've implemented jwks, probably snapping a new release this week.

[travisghansen]

v0.5.0 snapped and latest updated etc with jwks support and crude ambassador doc.

[vishalgoel1988]

HI, It might be off topic, but I need to check one thing.
If I put eas (with oidc/keycloak) ahead of traefik ingress, my webapp client would be redirected to oidc login. Any REST calls further from webapp would carry cookie and go through fine. This is all good.
I want to check that if my client gets the jwt access token directly from auth server and pass it as Authorization header in a REST call to traefik (and further to eas server), would eas server allow that? I am asking as eas server just works on sessions? or does it verify the input Auth header token also and if token is good, pass through it otherwise throw 401?

[travisghansen]

@vishalgoel1988 yes eas can handle that scenario (and even more complex if you wanted to add basic auth backed by ldap as a 3rd option which is what I do).

This comment has a pretty good overview of how it can be achieved: #16 (comment)

Basically you want to simply add additional plugins to your config token and then optionally (and recommended) target those plugins with the pcb functionality.

I’d recommend opening a new issue if you need help with your specific configuration and we’ll get you all up and going :)

[travisghansen]

@vishalgoel1988 also note, if your primary app is a SPA then there are a couple of special options you should set especially if you own/develop the app and can add some logic there to better interact with eas.

[vishalgoel1988]

Thanks @travisghansen , I will give it a try