[frontend] KFP version is unknown and the cluster information is missing in the KFP UI left sidenav

[devina-ekawati]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  Google Kubernetes Engine (GKE), standalone KFP

• KFP version:
  2.4.0

Steps to reproduce

Open the KFP dashboard then check the KFP version and cluster information in the left sidenav (refer to the screenshots below)

Expected result

The KFP UI left sidenav show 2.4.0 as the version and show the cluster information correctly.

Materials and Reference

To get the value of the KFP version, the frontend calls /apis/v1beta1/healthz. When I tried to call this API, it returns:

{
    "buildDate": "",
    "frontendCommitHash": "",
    "frontendTagName": "",
    "apiServerReady": true,
    "apiServerCommitHash": "unknown",
    "apiServerTagName": "unknown",
    "apiServerMultiUser": false,
    "multi_user": false
}

After further reading, it seems that those values are populated from COMMIT_SHA and TAG_NAME, which are read from environment variable (ref). These environment variables are set inside the Dockerfile of the kfp-api-server.

For the cluster metadata, I got GKE metadata endpoints are disabled. error message when inspecting the UI using browser developer console. It worked as expected in the previous version (2.3.0). During the version upgrade, I also didn't make any changes on the cluster.

---

Impacted by this bug? Give it a 👍.

[izuku-sds]

/assign

[izuku-sds]

Hello @devina-ekawati ,

Thankyou for this hint!, literally saved a lot of time for me as a beginner.

After further reading, it seems that those values are populated from COMMIT_SHA and TAG_NAME, which are read from environment variable (ref). These environment variables are set inside the Dockerfile of the kfp-api-server.

Reason for Version: Unknown :
As you mentioned, the environment variables are set inside Dockerfile of the kfp-api-server. These variables are set using corresponding ARG values. We need to pass values of these ARG while building Docker image.
But the image pushed to registry were created without passing those ARG due to which these env variables took unknown as value.
Images which correspond to Version 2.3.0 were build properly, so no issue there.

Solution:
A new image could be build with proper ARG values and updating image path in corresponding yaml file should fix the issue.

You could refer: R1 , R2
Thankyou

[izuku-sds]

Hello @juliusvonkohout,
Could you confirm if I’m mistaken or it's correct?

[devina-ekawati]

Hi @izuku-sds! Yup correct, my initial hunch is that these variables are not passed during the build process. What I observed is that previously the image are build and pushed to gcr.io using Cloudbuild (these values are passed correctly as mentioned in the reference you mentioned) . But starting from 2.4.0, the build processes are moved to Github Action (ref). When I looked into the YAML file, it seems that the ARG values are not passed.

[juliusvonkohout]

I think even more relevant for most users is kubeflow/dashboard#25 if you want on this (right now still in Kubeflow/kubeflow) This change here is only for people running KFP without the full Kubeflow.