Kubeflow OIDC Okta Integration EKS cluster

[tppalani]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

I'm team i have installed latest version 1.9 in my EKS cluster and also i have my okta meta details client ID and secret followed by callback url and so on. all the pods are up and running inside Kubeflow components.

Below the are files i have modified

Config map oauth2-proxy

apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
  labels:
    app: oauth2-proxy
data:
  oauth2_proxy.cfg: >-
    provider = "oidc"

    scope = "profile email groups openid"

    oidc_issuer_url =
    "https://my-internal-app.com/oauth2/abcdef123456789"
    upstreams = "https://eks-sbx-aws.com"
    email_domains = [ "*" ]
    insecure_oidc_skip_issuer_verification = true
    client_id = "hjiklhsmshuwowoaalala"
    client_secret =
    "kijkahoolmnbjkoiplosuerkrlsjslslslsosjkslsskslslsls"
    pass_access_token = true
    cookie_secret = "R-F4Rh_9mLZVFLpbOe9saGggEunKZXUrRRWIXKlDT9c="
    skip_provider_button = true
    skip_auth_regex=["/dex/.*"]
    set_authorization_header = true
    set_xauthrequest = true
    cookie_name = "oauth2_proxy_kubeflow"
    cookie_expire = "24h"
    cookie_refresh = 0
    code_challenge_method = "S256"
    redirect_url =
    "https://eks-sbx-aws.com/oauth2/callback"
    relative_redirect_url = true

Post updating above configmap i have restarted the oauth2-proxy deployment and verified the oauth2-proxy pod logs
i can see auth success but followed by with message "No valid authentication in request. Initiating login"

[2024/09/13 16:40:34] 1.2.3.4:8080 GET - "/" HTTP/1.1 "" 302 462 0.001
[2024/09/13 16:40:34] [oauthproxy.go:1017] No valid authentication in request. Initiating login.
[2024/09/13 16:40:34] 1.2.3.4:8080 GET - "/" HTTP/1.1 "" 302 462 0.001
[2024/09/13 16:40:41] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[2024/09/13 16:40:41] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[2024/09/13 16:40:49] [oauthproxy.go:1017] No valid authentication in request. Initiating login.
[2024/09/13 16:40:49] 1.2.3.4:8080 GET - "/" HTTP/1.1 "" 302 462 0.001
[2024/09/13 16:40:51] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[2024/09/13 16:40:51] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[email protected] [2024/09/13 16:40:57] [AuthSuccess] Authenticated via OAuth2: Session{email:[email protected] user:abdefghiklmon PreferredUsername:[email protected] token:true id_token:true created:2024-09-13 16:40:57.315224546 +0000 UTC m=+4055.816491254 expires:2024-09-13 17:40:56.815439276 +0000 UTC m=+7655.316705964 groups:[KF-ADMIN]}
[2024/09/13 16:40:56] https://eks-sbx-aws.com GET - "/oauth2/callback?code=Q9pC8_ertuijslkalopqrs_tg9Q8sCrnwahAObTylQgjg&state=x9IF5isbdefghiklaopquytrewqahjdeJ-kmlmnhoaqsE%3A%2F" HTTP/1.1 "" 302 24 1.137
[2024/09/13 16:41:01] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[2024/09/13 16:41:01] 2.3.4.5:4180 GET - "/ping" HTTP/1.1 "kube-probe/1.27+" 200 2 0.000
[2024/09/13 16:41:04] [oauthproxy.go:1017] No valid authentication in request. Initiating login.
[2024/09/13 16:41:04] 1.2.3.4:8080 GET - "/" HTTP/1.1 "" 302 462 0.001
[2024/09/13 16:41:04] [oauthproxy.go:1017] No valid authentication in request. Initiating login.
[2024/09/13 16:41:04] 1.2.3.4:8080 GET - "/" HTTP/1.1 "" 302 462 0.001

Here auth pods logs

ime="2024-09-13T07:02:46Z" level=info msg="config using log level: debug"
time="2024-09-13T07:02:46Z" level=info msg="config issuer: https://eks-sbx-aws.com/dex"
time="2024-09-13T07:02:46Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2024-09-13T07:02:46Z" level=info msg="creating custom Kubernetes resources"
time="2024-09-13T07:02:46Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..."
time="2024-09-13T07:02:46Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2024-09-13T07:02:46Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..."
time="2024-09-13T07:02:46Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2024-09-13T07:02:46Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..."
time="2024-09-13T07:02:46Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2024-09-13T07:02:46Z" level=info msg="config storage: kubernetes"
time="2024-09-13T07:02:46Z" level=info msg="config static client: oauth2-proxy"
time="2024-09-13T07:02:46Z" level=info msg="config connector: local passwords enabled"
time="2024-09-13T07:02:46Z" level=info msg="config skipping approval screen"
time="2024-09-13T07:02:46Z" level=info msg="config refresh tokens rotation enabled: true"
time="2024-09-13T07:02:46Z" level=info msg="listening (http) on 0.0.0.0:5556"
time="2024-09-13T09:53:17Z" level=info msg="keys expired, rotating"
time="2024-09-13T09:53:17Z" level=info msg="keys rotated, next rotation: 2024-09-13 15:53:17.182046073 +0000 UTC"
time="2024-09-13T15:53:17Z" level=info msg="keys expired, rotating"
time="2024-09-13T15:53:17Z" level=info msg="keys rotated, next rotation: 2024-09-13 21:53:17.709302111 +0000 UTC"

Here is the dex config map value.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: dex
  namespace: auth
data:
  config.yaml: |
    issuer: https://eks-sbx-aws.com/dex
    storage:
      type: kubernetes
      config:
        inCluster: true
    web:
      http: 0.0.0.0:5556
    logger:
      level: "debug"
      format: text
    oauth2:
      skipApprovalScreen: true
    enablePasswordDB: true
    staticPasswords:
    - email: [email protected]
      hash: $2y$10$CwnuwG65Jibf2NxFtxiskOnOcW2HIRBWJfx7fE2D1To0ItZKoqBl2
      username: admin
      userID: "15841185641784"
    staticClients:
    - idEnv: oauth2-proxy
      redirectURIs: 
       - 'https://eks-sbx-aws.com/oauth2/callback'
      name: 'oauth2-proxy'
      secret: proxy

When accessing my dns name from browser https://eks-sbx-aws.com kubeflow dashboard waiting for username and pasword after passing the login details i'm getting page error like acces denied to access the webpage.

Here is my k8s ingress.yaml file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/security-groups: security-group
      alb.ingress.kubernetes.io/certificate-arn: arn-details 
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-2016-08
      alb.ingress.kubernetes.io/ssl-redirect: "443"
  name: istio-ingress
  namespace: istio-system
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - backend:
          service:
            name: istio-ingressgateway
            port:
              number:  80
        path: /*
        pathType: ImplementationSpecific

Here is istio-ingressgateway

"ELB-HealthChecker/2.0" "0a774d4c-127e-4743-993a-f34a6e34cc29" "1.2.3.4:8080" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 1.2.3.4:8080 12.13.14.15:62948 - -
[2024-09-13T16:40:49.751Z] "GET / HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 462 3 - "10.1.13.7" "ELB-HealthChecker/2.0" "1f73a262-7cb4-4323-aaf9-f0629e3f8cda" "1.2.3.4:8080" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 1.2.3.4:8080 10.188.213.187:7614 - -
[2024-09-13T16:40:49.875Z] "GET / HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 462 1 - "10.1.13.9" "ELB-HealthChecker/2.0" "e3d2483c-4550-4784-b399-6099db3c1020" "1.2.3.4:8080" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 1.2.3.4:8080 10.188.212.29:14712 - -
[2024-09-13T16:40:56.178Z] "GET /oauth2/callback?code=jookksloiuytreqahskxx&state=lmnhojuahskskkskksaaa-khHxF8LE%3A%2F HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 24 1139 - "03.11.11.11,12.13.14.15" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "32e3a6fe-a6eb-4a63-aae8-b7e091e89b8b" "https://eks-sbx-aws.com/oauth2/callback" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 1.2.3.4:8080 12.13.14.15:48990 - -
[2024-09-13T16:40:57.622Z] "GET / HTTP/1.1" 403 UAEX ext_authz_error - "-" 0 0 86 - "03.11.11.11,12.13.14.15" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "e17642ac-ba7c-41f8-ab87-5fb0667987d7" "https://eks-sbx-aws.com/oauth2/callback" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 1.2.3.4:8080 12.13.14.15:48990 - -
[2024-09-13T16:41:04.586Z] "GET / HTTP/1.1" 302 UAEX ext_authz_denied - "-" 0 462 4 - "12.13.14.15" "ELB-HealthChecker/2.0" "3ddba49f-4ed3-4fe8-b2c1-0d61639663ff" "1.2.3.4:8080" "-" outbound|80||centraldashboard.kub

Steps to reproduce the issue

yaml and logs has been added

Put here any screenshots or videos (optional)

yaml and logs has been added

[tppalani]

Hi @juliusvonkohout will you able to help me on this or please give me some right direction that would be really helpful for entire team

[juliusvonkohout]

Hello, there is commercial consulting and distributions available. Feel free to reach out if you are interested.

Otherwise I can redirect to #2884 as a community solution and the official Dex and oauth2-proxy documentation.