Invalid client_id ("kubeflow-oidc-authservice").

[majorinche]

kubeflow version: manifests-1.3.0
try to visit kubeflow from istio-ingress, it reports this error

Bad Request

Invalid client_id ("kubeflow-oidc-authservice").

i check the dex log
kubectl logs -f dex-59cd9bd699-8spx6 -n auth
find error:
http://istio-ingressgateway.cluster.local/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTYzMTg0Njk4OXxFd3dBRUdNeVYwUTRSakp4VG1aSVN6VmhPRFE9fPtE69iYBqaT71Aq42jAxTmfhhVN58DyaQ8qeg3a9Sjc

i already reinstall cluster several times. error still exist.

[rc-coderepo]

@majorinche Try re-installing https://github.com/kubeflow/manifests/tree/v1.3.1 with a clean kubernetes cluster.

Is auth-service-0 in istio-sytem namespace running?

[majorinche]

hi, i already re-install three times, but error exist

auth-service-0 is running as showing below.
[root@e0302 ~]# kubectl get pod -n auth
NAME READY STATUS RESTARTS AGE
dex-59cd9bd699-8spx6 1/1 Running 0 22h
[root@e0302 ~]# kubectl get pod -n istio-system
NAME READY STATUS RESTARTS AGE
authservice-0 1/1 Running 0 22h
cluster-local-gateway-8558bdd679-97k4k 1/1 Running 0 22h
istio-ingressgateway-78f7794d66-w6pdz 1/1 Running 0 22h
istiod-574485bfdc-287pp 1/1 Running 0 22h

how to debug this error except reinstalling?

thank you

[rc-coderepo]

Hi,
Did you spin a new kubernetes cluster for every new install?

if you re-installed kubeflow, did you delete the kubeflow namespace and other releated namespaces or uninstalled by component by component by using kubectl delete -f -

Any changes made to kubeflow/manifests before install? I had Invalid client_id when I made changes to dex or odic modules in the manifest but don't exactly remember.

Which version of kubernetes are you using?
Is the kubernetes cluster on-prem or cloud?

Cert-manager
kustomize build common/cert-manager/cert-manager-kube-system-resources/base | kubectl apply -f -
kustomize build common/cert-manager/cert-manager-crds/base | kubectl apply -f -
kustomize build common/cert-manager/cert-manager/overlays/self-signed | kubectl apply -f -

Istio
kustomize build common/istio-1-9-0/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-9-0/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-9-0/istio-install/base | kubectl apply -f -

Dex
kustomize build common/dex/overlays/istio | kubectl apply -f -

odic-authservice
kustomize build common/oidc-authservice/base | kubectl apply -f -

if all the above steps are completed try?

uinnstall odic-authservice
kustomize build common/oidc-authservice/base | kubectl delete -f -

Reinstall oidc-authservice
kustomize build common/oidc-authservice/base | kubectl Apply-f -

[majorinche]

on-prem(bare-metal based)
kubernetes 1.17.9

i just install a new K8S cluster. now i have two k8s cluster to verify kubeflow

only change docker image address (as gcr.io can not be access here)

and change persistentvolumeclaim, as my cluster only as nfs storage
like
kubeflow-mysql-pv-claim,katib-mysql,minio
istio-system-authservice

already do steps as you mentioned. but not work!

[rc-coderepo]

what are the OIDC_CLIENT_ID and OIDC_CLIENT_SECRET values you see when you run the below command?

kubectl -n auth get secret -o yaml dex-oidc-client

I remember now the error is cause by the oidc-image. Can you please check which image your pod is using from the output
kubectl -n istio-system describe pod authservice-0

it should be using the image with following tag: gcr.io/arrikto/kubeflow/oidc-authservice:28c59ef

[majorinche]

[root@e0501 ~]# kubectl -n auth get secret -o yaml dex-oidc-client
apiVersion: v1
data:
OIDC_CLIENT_ID: a3ViZWZsb3ctb2lkYy1hdXRoc2VydmljZQ==
OIDC_CLIENT_SECRET: cFVCbkJPWTgwU25YZ2ppYlRZTTlaV056WTJ4cmVOR1Fvaw==

[root@e0501 ~]# kubectl -n istio-system describe pod authservice-0 | grep image
Normal Pulling 2m39s kubelet, e0601 Pulling image "chepkoyallan/oidc-authservice:28c59ef"

for the link you mentioned "gcr.io/arrikto/kubeflow/oidc-authservice:28c59ef", we cannot connect gcr.io, so i just search oidc-authservice in hub.docker.com, but not sure these are the same image.

[rc-coderepo]

Can you check the same for dex-xxx pod and dex cm. Try deleting dex pod if not deleted recently

[majorinche]

dex pod has been deleted again, but not working, and configmap shows below
[root@e0501 ~]# kubectl get configmap dex -n auth -o yaml
apiVersion: v1
data:
config.yaml: |
issuer: http://dex.auth.svc.cluster.local:5556/dex
storage:
type: kubernetes
config:
inCluster: true
web:
http: 0.0.0.0:5556
logger:
level: "debug"
format: text
oauth2:
skipApprovalScreen: true
enablePasswordDB: true
staticPasswords:
- email: [email protected]
hash: $2y$12$4K/VkmDd1q1Orb3xAt82zu8gk7Ad6ReFR4LCP9UeYE90NLiN9Df72
# https://github.com/dexidp/dex/pull/1601/commits
# FIXME: Use hashFromEnv instead
username: user
userID: "15841185641784"
staticClients:
# dexidp/dex#1664
- idEnv: OIDC_CLIENT_ID
redirectURIs: ["/login/oidc"]
name: 'Dex Login Application'
secretEnv: OIDC_CLIENT_SECRET
kind: ConfigMap
metadata:

[rc-coderepo]

Everything looks good here as well.

Which port are you trying to access istio-ingress are you using node port?
Are you able to see the initial login screen?

Also can post the version of dex docker image? it should be dex:v2.24.0

[majorinche]

yes, i using node port

i cannot see the login screen in browser

i check inside host by using curl command ,can see the login

[root@e0501 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost6 localhost6.localdomain6 localhost6.localdomain
10.20.0.51 e0501.cluster.local e0501
10.20.0.52 e0502.cluster.local e0502
10.20.0.53 e0503.cluster.local e0503
10.20.0.54 e0504.cluster.local e0504
10.20.0.55 e0505.cluster.local e0505
10.20.0.56 e0506.cluster.local e0506
10.20.0.61 e0601.cluster.local e0601
10.20.0.42 e0402.cluster.local e0402
10.20.0.45 e0405.cluster.local e0405
10.20.0.46 e0406.cluster.local e0406

10.20.0.51 istio-ingressgateway.cluster.local
[root@e0501 ~]# curl istio-ingressgateway.cluster.local
a href="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTYzMjcxMzcyNXxFd3dBRUdsWGRuRmFWR0V5VGprMVVuaFNWRm89fImcXwVWqs6AdfYeMooimBN4Rr3S-2IsnUg0q5medTiE">Found

[majorinche]

1,ingress-nginx expose a nodeport
[root@e0501 ~]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx NodePort 10.233.29.4 80:31255/TCP 8d

2,and another nginx redirect 80 to 31255
[root@e0305 ~]# cat /etc/nginx/nginx.conf| grep -A 2 -B 6 31255
stream{
server {
listen 80;
proxy_pass sonic_k8s;
}
upstream sonic_k8s {
server 10.20.0.51:31255;
}
}

3,then, in my laptop, using http://istio-ingressgateway.cluster.local to visit

a little complex, as my network environment is limited.

[rc-coderepo]

All the outputs you posted from curl and other looks exactly the same as my working cluster. Except I use istio-ingressgateway nodeport and node ip

Are you trying to access on ingress-ngnix or istio-ingressgateway?

As you are running on a single node which is your laptop can you try on the nodeport for 80 istio-ingressgateway?

[majorinche]

so if you use node ip + node port, it will report the same error as mine?

[rc-coderepo]

No mine doesn't have any errors right now. I just saying I am using nodeip+nodeport to access the UI.

Did you try nodeip+nodeport?

is central-dashboard pod running in kubeflow namespace just to make sure?

[majorinche]

central-dashboard is running
[root@e0501 ~]# kubectl get pod -n kubeflow | grep central
centraldashboard-797484bd5b-ss7tw 1/1 Running 0 8d
[root@e0501 ~]# kubectl logs -f centraldashboard-797484bd5b-ss7tw -n kubeflow

[email protected] start /app
npm run serve

[email protected] serve /app
node dist/server.js

Initializing Kubernetes configuration
Unable to fetch Application information: 404 page not found

"other" is not a supported platform for Metrics
Using Profiles service at http://profiles-kfam.kubeflow:8081/kfam
Server listening on port http://localhost:8082 (in production mode)

i try visit by nodeip+nodeport, but still same error

can i skip this auth or dex? is it really necessary for kubeflow?

[rc-coderepo]

There is a way to use a standalone kubeflow without dex but haven't tried it.

https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/

Did you install the below to component in the current instllation?

Profiles + KFAM
Install the Profile Controller and the Kubeflow Access-Management (KFAM) official Kubeflow components:

kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -

User Namespace
Finally, create a new namespace for the the default user (named kubeflow-user-example-com).

kustomize build common/user-namespace/base | kubectl apply -f -

[majorinche]

Profiles+KFAM and User Namespace already done.

kubectl get pod -n chejinguo -w
NAME READY STATUS RESTARTS AGE
ml-pipeline-ui-artifact-7d6b76c7f9-fdhgv 2/2 Running 0 2m32s
ml-pipeline-visualizationserver-77d54cf95b-bv2vk 2/2 Running 0 2m31s

[rc-coderepo]

you have to look for profile-controller pod for KFAM+profile

[majorinche]

if i visit nodeip+nodeport directly, it will redirect for me and has invalid id error

but after using following link, i can see a login page
http://172.18.165.51:32479/dex/auth/local?req=hlxdseon6244zaoj6jmn33svn

remind me to input email address and password

Log in to Your Account
Email Address
Password

and where can i add a new account manually?

[rc-coderepo]

account is added by default
user - [email protected]
password - 12341234

is 172.18.168.51 your node id?

[majorinche]

yes, node ip is 172.18.165.51

[majorinche]

invalid client id error url
http://172.18.165.51:32479/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTYzMjczNzg4NHxFd3dBRURsbVVXSjZUMDVLUVVGM1pFTjRiVTA9fBM-oZdZ4I0vYmA6_JA-Chl-fOSXpSJB9djzr3RTSmx_

after login with user([email protected]) and password(12341234), new error appear.
time="2021-09-27T10:19:25Z" level=error msg="Failed to exchange authorization code with token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"invalid_client","error_description":"Invalid client credentials."}" ip=10.233.91.204 request="/login/oidc?code=knpcvannnx2j2txmyj4pzjkny&state=MTYzMjczNzk1MnxFd3dBRUdSbU1tVlRTbmxaZEhGM1kwWnBWVWs9fGM26RB6pgh-7-YYKHCSnq45GjtDmscWVGnEfDQa8m0e"

[rc-coderepo]

did you run this it will create that user
User Namespace
Finally, create a new namespace for the the default user (named kubeflow-user-example-com).

kustomize build common/user-namespace/base | kubectl apply -f -

[majorinche]

yes, already done

[rc-coderepo]

Ok delete dex-xxx, auth-service, istio-ingressgateway, central-dashboard and profile-controller pods and try again to login

Can you clearly mention the setup of your kubeflow

Like how did you install kubernetes?
Is it running on a virtualbox on your laptop?
any related details?

did you run kustomize command for each component to install individually or did your run kustomize to install all kubeflow components at once?

[majorinche]

1, all pods you mentioned has been delete again

2, install kubernetes in 6 bare-mental servers. kubernetes version is 1.17.9, and its clean, only to test kubeflow. no other application.
[root@e0501 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
e0402 Ready 9d v1.17.9
e0405 Ready 9d v1.17.9
e0406 Ready 9d v1.17.9
e0501 Ready master 9d v1.17.9
e0502 Ready master 9d v1.17.9
e0503 Ready master 9d v1.17.9

3, i follow up with https://github.com/kubeflow/manifests/tree/v1.3.0, and Install individual components

[majorinche]

hi,
i just replace dex image from default: dexidp/dex:v2.22.0 to the latest version in https://hub.docker.com/r/dexidp/dex

now the error is gone, i can open the kubeflow.

[rc-coderepo]

like i said in the initial comments of the discussion this is just an image version issue. 🙂

[majorinche]

yes, i review the comment, you said "Also can post the version of dex docker image? it should be dex:v2.24.0"

but i did not notice. my fault. sorry.

thanks you very much!!!

[rc-coderepo]

You're Welcome. I think we can close this

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to inactivity.