chore(sdk): Loosening kubernetes dependency constraint

[egeucak]

Description of your changes:
Loosen Kubernetes dependency version requirement. Addresses #11078

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

Resolves #11078

[google-oss-prow]

Hi @egeucak. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chensun]

@egeucak Can you sign the DCO please?

/ok-to-test

[egeucak]

sorry i always forget to signoff, signed it now. also the errors in test seems to be caused by absl, is it caused by my changes?

/retest

[hbelmiro]

/rerun-all

[hbelmiro]

@egeucak can you please add the following to #11079 (comment) so the issue is automatically closed when this PR gets merged?

Resolves https://github.com/kubeflow/pipelines/issues/11078

[egeucak]

Some of the tests are failing. Is there anything I can do to fix those?

[egeucak]

Can this PR please get some attention and love?

[hbelmiro]

Some of the tests are failing. Is there anything I can do to fix those?

@egeucak the version upgrades caused incompatibilities. You need to go through the failing tests and figure out why.
Sorry if that sounds too obvious to you. Feel free to let us know if you need more specific help.

[egeucak]

I need some help reproducing the errors for "kfp-kubernetes library tests" and "test-run-all-gcpc-modules" locally. i don't know how to get more specific than that

[hbelmiro]

I need some help reproducing the errors

@egeucak Now you're being specific 🙂

For kfp-kubernetes library tests, you can run the steps in https://github.com/kubeflow/pipelines/blob/master/.github/workflows/kfp-kubernetes-library-test.yaml.

For test-run-all-gcpc-modules, you can see https://github.com/GoogleCloudPlatform/oss-test-infra/blob/3738b70f78b971cc281b1791cf8e73cfbe08ae55/prow/prowjobs/kubeflow/pipelines/kubeflow-pipelines-presubmits.yaml#L332C11-L332C36.
The last time I ran this one locally, I did the following: (not sure it still works)

source_root=$(pwd)
pip install --upgrade pip
pip install $source_root/sdk/python
pushd api
make clean python
popd
python3 -m pip install api/v2alpha1/python
pip install components/google-cloud
pip install $(grep 'pytest==' sdk/python/requirements-dev.txt)
pytest test/gcpc-tests/run_all_gcpc_modules.py

[hbelmiro]

/rerun-all

[egeucak]

/rerun-all

[egeucak]

so i figured out the issue wrt kfp-kubernetes library tests. it seems like SDK depended on setuptools which had been brought by the previous version of kubernetes. please let me know if installing it at test/presubmit-test-kfp-kubernetes-library.sh is acceptable.

"test-run-all-gcpc-modules" is giving me timeout, it didn't even start but locally it seemed to work

and I would like to get some guidance on replicating the error with KFP SDK execution tests / sdk-execution-tests.

thanks

[hbelmiro]

@egeucak

please let me know if installing it at test/presubmit-test-kfp-kubernetes-library.sh is acceptable.

My quick answer would be: Yes, it's acceptable. But I'd like to know @diegolovison's opinion on this.
We're trying to make the tests easier to run on other environments than GitHub Actions. Maybe separating installs in a different script would be better.

"test-run-all-gcpc-modules" is giving me timeout, it didn't even start but locally it seemed to work

I'll leave this one for @chensun or @zijianjoy to answer.

I would like to get some guidance on replicating the error with KFP SDK execution tests / sdk-execution-tests.

At which point are you blocked?
Did you try to run ./test/presubmit-test-kfp-kubernetes-library.sh? What happened?

[diegolovison]

Maybe separating installs in a different script would be better.

Yes. Maybe in other PR?

[egeucak]

At which point are you blocked?
Did you try to run ./test/presubmit-test-kfp-kubernetes-library.sh? What happened?

The error is coming from .github/workflows/sdk-execution.yml which is then invoking ./test/sdk-execution-tests/sdk_execution_tests.py, and it is requiring me to deploy KFP. I was wondering if there is an easy and straightforward way of running those tests other than having a local deployment.

[egeucak]

I installed kubeflow locally by following the guide at https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/ and executed pytest ./test/sdk-execution-tests/sdk_execution_tests.py --asyncio-task-timeout $TIMEOUT_SECONDS. The tests look flaky as I get inconclusive results. I can run all tests locally without any issues but it is inconsistent when the resource limits are low.

/rerun-all

[egeucak]

seems like the bot doesn't listen to my commands. how can I force it to rerun checks?

[egeucak]

Can I please get a re-run? Tagging @chensun @zijianjoy and @hbelmiro for visibility

[hbelmiro]

At which point are you blocked?
Did you try to run ./test/presubmit-test-kfp-kubernetes-library.sh? What happened?

The error is coming from .github/workflows/sdk-execution.yml which is then invoking ./test/sdk-execution-tests/sdk_execution_tests.py, and it is requiring me to deploy KFP. I was wondering if there is an easy and straightforward way of running those tests other than having a local deployment.

There isn't. It tests executions. So, it needs KFP somewhere to run those executions.

The tests look flaky as I get inconclusive results. I can run all tests locally without any issues but it is inconsistent when the resource limits are low.

Can you please open an issue with details?

/rerun-all

[hbelmiro]

/ok-to-test
/rerun-all

[egeucak]

KFP SDK execution tests / sdk-execution-tests succeeds and test-run-all-gcpc-modules is still pending. Do you have any idea why that might be?

The reason for failure in test-run-all-gcpc-modules was caused by a missing dependency and the issue was resolved after #11087.

Link to the previous failing run: https://oss.gprow.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/11079/test-run-all-gcpc-modules/1821086011313098752

[egeucak]

Prow is still timing out. Any ideas or anything I can do to fix it?

https://oss.gprow.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/11079/test-run-all-gcpc-modules/1826224907701194752

Init container clonerefs not ready: (state: waiting, reason: "PodInitializing", message: "") Init container initupload not ready: (state: waiting, reason: "PodInitializing", message: "") Init container place-entrypoint not ready: (state: waiting, reason: "PodInitializing", message: "")

[hbelmiro]

/retest

[hbelmiro]

/lgtm

cc @chensun

[egeucak]

Can we get reviews and approvals on this today?

[egeucak]

please let's not forget this PR and let's give this the love and attention it deserves.

cc @chensun @zijianjoy @gregsheremeta

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [chensun]
• test/OWNERS [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hbelmiro]

/retest

[hbelmiro]

/retest

[hbelmiro]

/retest

[egeucak]

I hate prow urrghh

[hbelmiro]

/retest

[hbelmiro]

/retest

[hbelmiro]

@chensun @zijianjoy there's something wrong with Prow. test-run-all-gcpc-modules always fails with "Pod pending timeout".