Allow usage of go-gatekeeper as auth-service for traefik forward-auth (or nginx auth_request or similar)

[ChristianCiach]

Let me preface my issue by saying that I absolutely love this project! This is exactly what we need and I am especially excited for the teased OPA integration.

Previously we were using oauth2-proxy, but this leaves a lot to be desired and the configuration is awkward at best. With gogatekeeper we are only missing a single little feature, which leads me to this request.

Summary

Instead of using gogatekeeper as a reverse-proxy, we would like to use it as an auth-service together with Traefik's forward-auth. This is actually possible with oauth2-proxy, but the configuration is really awkward. The relevant part of our pod-spec for the oauth2-proxy looks like this:

      containers:
        - name: apis-oauth-proxy
          image: 'quay.io/oauth2-proxy/oauth2-proxy' 
          args:  # See https://medium.com/in-the-weeds/service-to-service-authentication-on-kubernetes-94dcb8216cdc
            - '--client-id=dashboard'
            - '--client-secret=UNUSED'
            - '--cookie-secret=A7fS115w-K0gm9-7ChbkVA=='  # 100% unused!
            - '--http-address=0.0.0.0:4180'
            - '--oidc-issuer-url={{ keycloak_realm_url }}'
            - '--provider=oidc'
            - '--reverse-proxy'
            - '--skip-jwt-bearer-tokens'  # Means that requests that already contain a JWT are not redirected to the login page.
            - '--upstream=static://202'
            - '--email-domain=*'  # We don't care for "email" claim
            - '--insecure-oidc-allow-unverified-email'  # We don't care for "email" claim
            - '--oidc-email-claim=sub'  # We don't care for "email" claim

This is the corresponding Traefik-Middleware configuration:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: apis-oauth
spec:
  forwardAuth:
    # Important: Add namespace to URL, because it gets called by traefik which runs in kube-system.
    address: http://apis-oauth-proxy.default:4180/oauth2/auth

Why?

The idea is to (additionally) secure our Rest-APIs at the ingress controller (Traefik) before a request hits the actual backend service. Traefik takes the requests and forwards it to the configured "forward-auth" service.

If the service answers with a 2XX code, access is granted, and the original request is performed. Otherwise, the response from the authentication server is returned.

How

As you can see from the example above, oauth2-proxy allows the configuration of a special upstream-url with the schema static://, for example: --upstream=static://202. This makes oauth2-proxy return a simple 202 response with an empty body if a token is successfully validated. This makes it very easy to use it with traefik's forward-auth. Of course, there is no need exactly replicate this in gogatekeeper. I would be totally fine with something like..

listen: :3000
upstream-url: http://127.0.0.1:3000/static/202

Additional Information

After having written all of this... Does gogatekeeper even allow simple JWT-validation without initiating the auth-code-flow if the token is missing or invalid? I've not tried this yet and the documention is not perfectly clear to me in this regard.

[ChristianCiach]

The static:// upstream schema was added to oauth2-proxy to support ext_authz of Envoy: oauth2-proxy/oauth2-proxy#265

This means that adding something like this to go-gatekeeper will probably allow it to be used for many similar use-cases (if go-gatekeeper is able to return a simple 401 response for failed token validations instead of redirecting to a login page, which I am still unsure about).

[p53]

@ChristianCiach you can switch off redirecting to a login page by using option:
--no-redirects: true
It is right in first example in documentation, but agree that maybe it would be better to explicitly document.

[ChristianCiach]

@p53 Thanks. I think the docs are probably explicit enough. I just didn't try out this stuff before creating this issue.

The comment "used with forward proxies" probably also threw me off. But we should probably change this line if Gatekeeper will be made to work with forward-auth to mention this use-case also.

[p53]

@ChristianCiach yes i can update it, it is useful also when you would like protect just APIs and use client credentials grant (that could be probably also mentioned more explicitly in docs), but for your use case when token will be valid, request will be forwarded to upstream, that means in current situation either request will hit your backend twice, one for gatekeeper and one for traefik or you will have to have some dummy backend which always returns 200 and that would be configured as upstream in gogatekeeper. For forward-auth request it would be probably best to have some /oauth/validate endpoint or similar that would do same thing as validation with no-redirect except it would not forward, need to check/think about this.

[ChristianCiach]

For forward-auth request it would be probably best to have some /oauth/validate endpoint or similar that would do same thing as validation with no-redirect except it would not forward, need to check/think about this

Yes, I agree, that would be even better than a dummy upstream-url like oauth2-proxy is using. Thanks for actually considering this!

[ChristianCiach]

or you will have to have some dummy backend which always returns 200 and that would be configured as upstream in gogatekeeper

I actually was considering that, but I would rather avoid that to keep things as simple as possible. I also have to consider the footprint, because resources are scarce and we may need to deploy many instances of go-gatekeeper.

[p53]

@ChristianCiach i implemented features which enable you to configure gogatekeeper as forward-auth https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md#forward-auth, done it this way because it is more general and allows more use-cases, you can try 1.7.0-rc1 release, there is image in quay.io

[ChristianCiach]

Awesome, thank you very much! I will test this tomorrow or later this week at the latest.

[ChristianCiach]

@p53 I see that you also included a new "header matching" mechanism together with the NoProxy option. You clearly envisioned some specific use-case for using these two features together, but I am not sure I understand it.

What is the header matching good for? What problems are you trying to solve with it? I have the feeling I am missing something important.

[p53]

@ChristianCiach as i checked traefik and nginx forward-auth configuration, with traefik you must have fixed forward-auth url:
forwardAuth: # Important: Add namespace to URL, because it gets called by traefik which runs in kube-system. address: http://apis-oauth-proxy.default:4180/oauth2/auth
That means when you forward request from your proxy to gogatekeeper you lose /path and also other information (like original IP), which is quite important. Traefik solves this by forwarding headers as stated in documentation and similar possibility is there for nginx-ingress. So now with traefik/nginx+gatekeeper you can match those headers and additionally (gogatekeeper can insert response headers there is option response-headers) traefik/nginx can insert responded headers to forwarded request.

[p53]

@ChristianCiach how it worked to you?

[ChristianCiach]

@p53 I am very embarrassed that testing this stuff takes so much longer than you took to implement it 😅

I am currently busy with unforseen stuff at work, but I promise I get back to you very soon!

That means when you forward request from your proxy to gogatekeeper you lose /path and also other information (like original IP), which is quite important.

Yes, this makes perfect sense! Thank you for thinking ahead.

[ChristianCiach]

Most likely I will thoroughly test this with multiple traefik IngressRoutes tomorrow! You may want to hold back the final release of 1.7.0 until then. Sorry for leaving you hanging!

(Actually, when I created this feature request, I thought "well, this will probably take a few months...". You took me by surprise by implementing it that quickly. So I didn't reserve time for testing it 😄)

[p53]

@ChristianCiach no hurry, just take your time...., i will leave this issue opened anyway until you confirm that it works for you

[ChristianCiach]

I am testing 1.7.0-rc2, but I cannot get it to work.

This is my current Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dashboard-apis-oauth
  namespace: censored
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dashboard-apis-oauth
  template:
    metadata:
      labels:
        app.kubernetes.io/name: dashboard-apis-oauth
    spec:
      containers:
      - args:
        - --client-id=dashboard
        - --no-redirects=true
        - --no-proxy=true
        - --listen=0.0.0.0:4180
        - --discovery-url=https://auth-censored.experimental.local/auth/realms/censored
        env:
        - name: SSL_CERT_FILE
          value: /cacerts/tls.ca
        image: quay.io/gogatekeeper/gatekeeper:1.7.0-rc2
        livenessProbe:
          httpGet:
            path: /oauth/health
            port: 4180
          initialDelaySeconds: 5
          periodSeconds: 60
        name: oauth-proxy
        ports:
        - containerPort: 4180
          name: oauth
        volumeMounts:
        - mountPath: /cacerts
          name: cacerts
      volumes:
      - name: cacerts
        secret:
          items:
          - key: tls.ca
            path: tls.ca
          secretName: dashboard-apis-cacerts-25k7g68k7t

This is the traefik middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  labels:
    app.kubernetes.io/name: dashboard-apis-oauth
    app.kubernetes.io/part-of: dashboard
  name: dashboard-apis-oauth
  namespace: censored
spec:
  forwardAuth:
    address: http://dashboard-apis-oauth.censored:4180/oauth/authorize

Observations, in no specific order:

• It took me a while to figure out that I need to enable no-proxy. I find the documentation a bit lacking in general, but that's not a big issue.
• At first the gatekeeper refused to connect to our keycloak, because our SSL certificate is self-signed. I tried the tls-ca-certificate option, but this did nothing. It took a while to figure out that Go allows to specify custom certificates using the SSL_CERT_FILE env variable. Oauth2-proxy allows to set the ca-certificate file (or bundle) as an option. Again, this is not a big issue, but adding this to the documentation would be nice. (I can help out with extending the documentation if you like.)
• The gatekeeper logs enabled reverse proxy mode, upstream url {"url": ""} and no redirection url has been set, will use host headers on startup. This is slightly confusing when using no-proxy and no-redirects, especially since the second message is a warning.
• I am not sure which URL to use in my middleware. /oauth/authorize always redirects to our keycloak with a 303 - See Other header (even though no-redirects=true), while /oauth/callback always responds with 400 - Bad Request.

Do you have an idea? Can I provide anything to debug this?

[ChristianCiach]

Interestingly, Traefik passes the 303 - See Other redirect back to the originating caller. When this happens, Gatekeeper logs nothing.

[p53]

@ChristianCiach

• hmm there is explicitly mentioned no-proxy in forward-auth documentation, i can give there it is MUST
• you can use --skip-openid-provider-tls-verify, (usually when you cannot find something in user-docs you can try settings.md or --help - it is mentiond on front page at the beginning, there is so much options that really is hard to document every possible thing)
• you should not use /oauth/authorize (thats used for authorization code flow which is completly different use case as forward-auth), just give no path there "http://dashboard-apis-oauth.censored:4180"
• and use --enable-default-deny or --enable-default-deny-strict - that will protect all your paths if you don't permit them explicitly with resources

[p53]

@ChristianCiach i think i will have to give some example in docs on protecting REST API's with gatekeeper because it seems it is not very known thing

[ChristianCiach]

hmm there is explicitly mentioned no-proxy in forward-auth documentation, i can give there it is MUST

Yes, it is in the documentation, but maybe there should be a complete example for the forward-auth use case. I still think it's a bit confusing, but maybe that's just me :)

you can use --skip-openid-provider-tls-verify, (usually when you cannot find something in user-docs you can try settings.md or --help - it is mentiond on front page at the beginning, there is so much options that really is hard to document every possible thing)

I've seen this, but I would rather avoid this option, since this hostname is especially important to verify. Thanks for the pointer, though!

and use --enable-default-deny or --enable-default-deny-strict - that will protect all your paths if you don't permit them explicitly with resources

Thanks for this! --enable-default-deny seems to be the default and that is what I actually want in this case.

you should not use /oauth/authorize (thats used for oauth flow which is completly different use case as forward-auth), just give no path there "http://dashboard-apis-oauth.censored:4180/"

Thanks, this works! I would probably never have guessed that 😄 Maybe my brain is still too attached to how oauth2-proxy works.

I can confirm that this works with a valid token and properly returns 401 when the token is missing, expired or corrupt.
Very nice!

In case of a corrupt token (just removed a character from the JWT signature part), gatekeeper logs this:

error    no session found in request, redirecting for authorization    {"error": "illegal base64 data at input byte 340"}

When the token is missing, it logs this:

error    no session found in request, redirecting for authorization    {"error": "authentication session not found"}

In both cases it states a redirect, which isn't actually happening. This is not a big issue, just a cosmetic thing.

I will also try the header-matching thingy, but probably not today. I see that you have unit tests for this, so I trust that this will work.

[ChristianCiach]

and use --enable-default-deny or --enable-default-deny-strict - that will protect all your paths if you don't permit them explicitly with resources

Will this even have any effect when using forward-auth? As I understand it, Traefik will only ever request the / path of the gatekeeper using GET. It will never attach the requested path when forwarding the request to the auth-service, but will send it as a header instead. Traefik will add the requested path as an additional header X-Forwarded-Uri, but Gatekeeper won't check this when using URL-resource-matchers, I guess...? So URL-resource-matchers will probably not make any sense in this use case.

[p53]

--skip-openid-provider-tls-verify yup this is meant especially for quick testing, when you don't want to bother with certs.
if you give --enable-default-deny=true it will protect all paths /*, when you will give =false, it will return 200 even for /, unless you explicitly set conditions in resources, yes matchers on path other than / don't make sense in this case

[ChristianCiach]

Thanks for everything so far! It seems to work just fine, ignoring the small cosmetic issues in the logs. Have a nice evening!

[ChristianCiach]

Now that 1.7.0 is out and this seems to work fine, I will close this issue. Thanks again!

[bcm0]

ChristianCiach: Do you use header matching feature or do you use --resources? I can't get around error 401 with keycloak.

I use the exact settings from forward-auth documentation example but I don't understand the last two lines.

- --enable-default-deny=true # this option will ensure protection of all paths /*, according our traefik config, traefik will send it to /
- --match-headers=headers=x-some-header:somevalue,x-other-header:othervalue

https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md#forward-auth

How to set allowed resources with keycloak? Are there defaults for x-some-header:somevalue?

[p53]

@bcm0 replied to you also on discord

• checked, yes docu is bad will need to update, resources should be used