Reorganize Kubeflow tests

[kromanow94]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

The tests for the Kubeflow Components are currently placed either in the GitHub Actions Workflow Templates or under the tests/gh-actions path, or under contrib in an unorganized manner. It's confusing and lacks organization. There should be well-organized place or alignment for the tests. Those should preferably run with a testing framework like pytest. If we were to improve this situation, we can start implementing a more robust and flexible test framework that can be simply run from GH Actions or in the user development environment.

To point a few tests that could be reorganized:

• tests/gh-actions/runasnonroot.sh
• tests/gh-actions/run_and_wait_kubeflow_pipeline.py
• contrib/bentoml/test.sh
  • .github/workflows/bentoml_test.yaml
• .github/workflows/katib_test.yaml
• .github/workflows/kserve_m2m_test.yaml
• contrib/metacontroller/test/catset-controller.yaml
  • .github/workflows/metacontroller_test.yaml
• .github/workflows/model_registry_test.yaml
• .github/workflows/notebook_controller_m2m_test.yaml
• .github/workflows/pipeline_run_from_notebook.yaml
• .github/workflows/pipeline_test.yaml
• contrib/ray/test.sh
  • .github/workflows/ray_test.yaml
• contrib/seldon
  • .github/workflows/seldon_test.yaml
• .github/workflows/train_operator_test.yaml

Additionally, some of the components lacks a proper e2e tests. Here is the list of GH Actions that performs sanity tests by just deploying the manifests on the cluster but are not providing any more insights than that:

• .github/workflows/admission_webhook_test.yaml
• .github/workflows/centraldashboard_test.yaml
• .github/workflows/dex_test.yaml
• .github/workflows/jupyter_web_application_test.yaml
• .github/workflows/kserve_cni_test.yaml
• .github/workflows/kserve_test.yaml
• .github/workflows/manifests_example_test.yaml
• .github/workflows/notebook_controller_test.yaml
• .github/workflows/profiles_test.yaml
• .github/workflows/tensorboard_controller_test.yaml
• .github/workflows/tensorboards_web_application_test.yaml
• .github/workflows/volumes_web_application_test.yaml

Additionally, we could start using some nice utilities like pre-commit to automatically run some actions like linters, formatters, check if commit is signed and potentially more. Then, such a pre-commit configuration could be used both from the GH Actions and from the Developer Environment. These are the GH Actions that could benefit from this pattern:

• .github/workflows/bash_formatter.yaml
• .github/workflows/python_formatter.yaml
• .github/workflows/yaml_formatter.yaml

This was recently touched upon here: 2795#discussion_r1674228094.

Steps to reproduce the issue

N/A

[juliusvonkohout]

Let's discuss this in the next kubeflow platform meeting :-)

[juliusvonkohout]

@hansinikarunarathne is also working on this and has a draft. Can we maybe merge both issues?

[kromanow94]

@juliusvonkohout sounds like a good plan. I can share a few thoughts on where and how I think this should head. We can talk at the next KF Platform/Manifests WG Meeting.

[juliusvonkohout]

I think https://docs.google.com/document/d/1FSbGDPko3rZ77oU5F4xxDwGhzgfYbvq5kX-ZciUzaT4/edit?usp=sharing is the document from @hansinikarunarathne

[juliusvonkohout]

Reopen after merging #2805