[bug] Execution not caching

[Itega]

What steps did you take:

• Deploy Kubeflow Pipelines on a new GKE cluster
• Run the Kubeflow caching sample

What happened:

The sample failed with error :
RuntimeError: The cached execution was not re-used or pipeline run took to long to complete.

What did you expect to happen:

The sample should have run successfully.

Environment:

Kubernetes : 1.18.18-gke.1100
KFP version : 1.6.0
KFP SDK version: 1.6.2

[Bobgy]

@Ark-kun is this sample still working properly?
It fails in our test infra too:
https://testgrid.k8s.io/googleoss-kubeflow-pipelines#kubeflow-pipeline-postsubmit-integration-test

[Itega]

I can give you more information about my case as I have now found a way to install KFP 1.6 with caching on GCP.
My problem was that I was using a GKE private cluster, I tried with a public cluster and it's working fine.
Did some search and Kubeflow does not seems to be compatible with private cluster on GKE : GoogleCloudPlatform/kubeflow-distribution#267 Maybe you have more informations on this ?

Can confirm the sample is working well on my public cluster so I really think there is a problem with caching on some Kubernetes cluster configurations.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.