Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(frontend): retrieve archived logs from correct location #11010

Merged
merged 8 commits into from
Aug 15, 2024

Conversation

droctothorpe
Copy link
Contributor

@droctothorpe droctothorpe commented Jul 16, 2024

Description of your changes:
This PR solves #10036 by making it possible to inject the AWF keyFormat, which determines where archived workflow container logs are saved, into the UI via configs.ts / env vars so that the UI retrieves them from the correct location.

Currently, for v2, the UI is looking in the wrong object store path (which is hard-coded) and coming up empty. We could update the hard-coded path, but loading it from configs.ts makes it more future proof and flexible in case end users want to modify the keyFormat for any reason.

This is an interim solution that's still a huge improvement over the current state of affairs (logs are inaccessible in v2 once the AWF CR has been GCed), but it has two important drawbacks that we want to delineate:

  1. It doesn't show logs for the dag driver or the task driver(s). This can be helpful if there are issues in the driver rather than the component code. The only way to debug this currently is to interact directly with the pods via the K8s API. This PR doesn't introduce this limitation. It just doesn't solve for it.
  2. If the AWF keyFormat is updated in both the workflow-controller-configmap and the UI (via configs.ts / env vars), the logs of new runs will be accessible, but the logs of old runs will no longer be accessible. In defense of this PR as an interim solution, the logs of NO runs are currently accessible.

In spite of these drawbacks, we still think it may be worthwhile to merge this PR as an interim fix since keyFormat is rarely modified and this will solve 99% of use cases for a really significant regression (with, at present, 21 👍 emojis).

Looking forward to feedback. Thanks!

PS. We can add support for additional AWF template tags, as long as the actual parameters are available to the handler. We can also list supported tags via comments in the configmap.

PPS. I am not a frontend engineer.

Checklist:

@thesuperzapper
Copy link
Member

@droctothorpe if we are going to do this interim fix can we please change the implementation slightly:

  1. Use the go template package (which is what Argo uses) rather than native string replacement, so we:
    • allow for key formats that use non perfect spaces {{ workflow.name}}, which is still valid
    • fail to start KFP if an unsupported key format template is passed (e.g. using an unsupported variable/function)
  2. Also support {{ workflow.namespace }} because this is a very common element of keyFormat to allow for namespace isolation by IAM scoped to object store prefixes

@droctothorpe
Copy link
Contributor Author

droctothorpe commented Jul 16, 2024

  1. Use the go template package (which is what Argo uses) rather than native string replacement, so we:

    • allow for key formats that use non perfect spaces {{ workflow.name}}, which is still valid

Apologies if I'm missing something obvious, but we can't use the Go templating package in typescript. Is there a well-maintained typescript package for handling go templates that you're aware of? Is it worth adding a dep to avoid a half dozen string replacements?

  1. Also support {{ workflow.namespace }} because this is a very common element of keyFormat to allow for namespace isolation by IAM scoped to object store prefixes

I added a TODO for this (since namespaces are not supported in the existing implementation), but should be able to incorporate it into this PR.

@droctothorpe
Copy link
Contributor Author

droctothorpe commented Jul 16, 2024

We added namespace tag handling and validated that it works as expected, which is a good step towards improved user isolation for log artifacts.

We also added template tag validation. The node server throws an error which bubbles up to the UI and provides clear remediation instructions.

@thesuperzapper
Copy link
Member

@droctothorpe can you try and make it support arbitrary spaces within the {{ xxx }}, probably using a regex like {{\s*some_token\s*}}

It's very common to format go templates like{{ xxxxx }} and not like {{xxxx}}, and it would be helpful if users can use the exact same value as their Argo Workflows keyFormat config.

For example, these should all be equivalent:

  • {{aaaa}}
  • {{ aaaa}}
  • {{aaaa }}
  • {{ aaaa }}
  • {{ aaaa }}

@thesuperzapper
Copy link
Member

@droctothorpe also, just to be clear, does this affect KFP V1 pipelines?

If so, we should support separate keyFormat configs for KFP V1 vs V2, because many deployments store their V1 and V2 pipeline logs separately.

@droctothorpe
Copy link
Contributor Author

@thesuperzapper addressed your penultimate comment with .replace(/\s+/g, '') since the keyFormat should not contain any whitespace anyway.

To your second point, my understanding is that keyFormat is defined at the AWF level rather than the KFP level and is set globally for both v1 and v2. This PR just instructs the UI where to look for the logs. The UI has no say over where the logs are actually being saved.

Apologies for the delayed response. Had to pivot to some dask-kubernetes / xgboost issues for two weeks.

@thesuperzapper
Copy link
Member

@droctothorpe why not just use the approach I described in #11010 (comment)?

While I agree removing spaces might seem like a solution, if keyFormat ever adds functions (beyond the default ones with every go template) removing spaces will break the inputs to those functions.

Also, while it would be a little crazy, users might want to include spaces in their S3 object store names.

@droctothorpe
Copy link
Contributor Author

@droctothorpe why not just use the approach I described in #11010 (comment)?

While I agree removing spaces might seem like a solution, if keyFormat ever adds functions (beyond the default ones with every go template) removing spaces will break the inputs to those functions.

Also, while it would be a little crazy, users might want to include spaces in their S3 object store names.

This was my rationale:

Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).

https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

Bucket names can only contain lowercase letters, numeric characters, dashes (-), underscores (_), and dots (.). Spaces are not allowed.

https://cloud.google.com/storage/docs/buckets#:~:text=Your%20bucket%20names%20must%20meet,Spaces%20are%20not%20allowed.

https://github.com/minio/minio-go/blob/cc7769f319debdee18b5b304e2b35166b0165fdb/pkg/s3utils/utils.go#L346

Copy link
Collaborator

@zijianjoy zijianjoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Run npm run format to pass presubmit test.

frontend/server/configs.ts Show resolved Hide resolved
frontend/server/workflow-helper.ts Outdated Show resolved Hide resolved
@droctothorpe
Copy link
Contributor Author

I'm going to do some manual validation against v1 to make sure that this didn't introduce any breaking changes.

@droctothorpe
Copy link
Contributor Author

droctothorpe commented Aug 14, 2024

Manual validation evidence

KFP v2

The run code:

from kfp import dsl
from kfp.compiler.compiler import Compiler
from kfp import Client


@dsl.component()
def foo():
    from random import random

    print(random())


@dsl.pipeline()
def pipeline():
    task1 = foo()
    task1.set_caching_options(False)


# Compiler().compile(pipeline_func=pipeline, package_path="two.yaml")
client = Client()
client.create_run_from_pipeline_func(pipeline)

Logs in the UI:
image

Deleting the workflow:

> k delete workflows --all && k get workflows
workflow.argoproj.io "pipeline-fzhgs" deleted
No resources found in kubeflow namespace.

Logs still available in the UI:
image

KFP v1

The run code:

from kfp import Client
from kfp import dsl

# from kfp.compiler.compiler import Compiler
from kfp.components import create_component_from_func


def foo():
    from random import random

    print(random())


foo_comp = create_component_from_func(foo)


@dsl.pipeline()
def pipeline():
    task1 = foo_comp()
    task1.set_caching_options(False)


# Compiler().compile(pipeline_func=pipeline, package_path="two.yaml")
client = Client()
client.create_run_from_pipeline_func(pipeline, arguments={})

Logs in the UI:
image

Deleting the workflow:

> k delete workflows --all && k get workflows
workflow.argoproj.io "pipeline-vkq4t" deleted
No resources found in kubeflow namespace.

Logs are NOT available in the UI from the logs tab. They must be accessed via output artifacts. This is a known bug that we could probably fix pretty easily, though I recommend doing so in a discrete PR since this PR doesn't introduce this bug, it just doesn't fix it.

A new regression

I decided to run another manual validation and found an unrelated regression.

In the master branch, if you create a run then delete the pod but preserve the workflow manifest, it still fails to retrieve logs!

image

Unless I'm somehow misconfiguring something, I think it's parsing the AWF manifest incorrectly. All of the workflow parsing logic here seems to be deeply misaligned with the actual workflow structure.

I started working on a fix but it was quite complicated and seemed like scope creep so I figured it might make more sense to tackle it in a standalone PR. That being said, the fact that retrieving artifact keys from the workflow doesn't work arguably increases the urgency of this PR since we always need to fallback to the final handler now when the pod is GCed.

@droctothorpe
Copy link
Contributor Author

Screw it. @quinnovator 🧙 and I decided to just fix the workflow parsing as part of this PR. Confirmed it works. May have to update tests. Additional commits coming soon.

droctothorpe and others added 6 commits August 15, 2024 12:43
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>
@droctothorpe
Copy link
Contributor Author

@zijianjoy, @quinnovator and I decided to just fix getPodLogsStreamFromWorkflow in this PR. It broke as a result of changes to the AWF manifest structure that were probably introduced a while ago. We updated the interfaces and brought the logic up to speed.

To summarize, the frontend tries to get logs from the following three locations in order:

  1. Directly from the K8s API.
  2. Extracting the artifact path from the AWF manifest (if the pods are GCed but the AWF manifest is not).
  3. Inferring the artifact path directly (if the AWF manifest is GCed).

Both 2 and 3 were broken! This PR fixes both. 🎉

Signed-off-by: droctothorpe <[email protected]>
@zijianjoy
Copy link
Collaborator

/lgtm
/approve

Thank you @droctothorpe for the contribution and validation!

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zijianjoy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 2e6e634 into kubeflow:master Aug 15, 2024
9 checks passed
@droctothorpe droctothorpe deleted the logs branch August 15, 2024 20:49
KevinGrantLee pushed a commit that referenced this pull request Sep 17, 2024
* fix(frontend): retrieve archived logs from correct location

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Add namespace tag handling and validation

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Remove whitespace from keyFormat

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Update frontend unit tests

Signed-off-by: droctothorpe <[email protected]>

* Remove superfluous log statements

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Add link to keyFormat in manifests

Signed-off-by: droctothorpe <[email protected]>

* Fix workflow parsing for log artifact

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Fix unit test

Signed-off-by: droctothorpe <[email protected]>

---------

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Co-authored-by: quinnovator <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>
R3hankhan123 pushed a commit to R3hankhan123/pipelines that referenced this pull request Sep 20, 2024
…#11010)

* fix(frontend): retrieve archived logs from correct location

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Add namespace tag handling and validation

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Remove whitespace from keyFormat

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Update frontend unit tests

Signed-off-by: droctothorpe <[email protected]>

* Remove superfluous log statements

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Add link to keyFormat in manifests

Signed-off-by: droctothorpe <[email protected]>

* Fix workflow parsing for log artifact

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Fix unit test

Signed-off-by: droctothorpe <[email protected]>

---------

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Co-authored-by: quinnovator <[email protected]>
google-oss-prow bot pushed a commit that referenced this pull request Sep 20, 2024
…etters (#11097)

* temp title: change title

Signed-off-by: KevinGrantLee <[email protected]>

* add release notes

Signed-off-by: KevinGrantLee <[email protected]>

* formatting

Signed-off-by: KevinGrantLee <[email protected]>

* feat(backend): move comp logic to workflow params (#10979)

* feat(backend): move comp logic to workflow params

Signed-off-by: zazulam <[email protected]>
Co-authored-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: MonicaZhang1 <[email protected]>
Co-authored-by: kylekaminky <[email protected]>
Co-authored-by: CarterFendley <[email protected]>
Signed-off-by: zazulam <[email protected]>

* address pr comments

Signed-off-by: zazulam <[email protected]>

* Use function name instead of base name and address edge cases

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: zazulam <[email protected]>

* Improve logic and update tests

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: zazulam <[email protected]>

* POC hashing command and args

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: zazulam <[email protected]>

* Add comments to clarify the logic

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: zazulam <[email protected]>

* Hash entire PipelineContainerSpec

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: zazulam <[email protected]>

---------

Signed-off-by: zazulam <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: MonicaZhang1 <[email protected]>
Co-authored-by: kylekaminky <[email protected]>
Co-authored-by: CarterFendley <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* feat(component): internal

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 660985413
Signed-off-by: KevinGrantLee <[email protected]>

* feat(components): internal

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 661332120
Signed-off-by: KevinGrantLee <[email protected]>

* fix(components): Fix to model batch explanation component for Structured Data pipelines

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 661475667
Signed-off-by: KevinGrantLee <[email protected]>

* feat(components): Support dynamic values for boot_disk_type, boot_disk_size in preview.custom_job.utils.create_custom_training_job_from_component

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 662242688
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Upgrade Argo to v3.4.17 (#10978)

Signed-off-by: Giulio Frasca <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* test: Moved kubeflow-pipelines-manifests to GitHub Actions (#11066)

Signed-off-by: vmudadla <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix: re-enable exit hanler test. (#11100)

Signed-off-by: Liav Weiss (EXT-Nokia) <[email protected]>
Co-authored-by: Liav Weiss (EXT-Nokia) <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(frontend): retrieve archived logs from correct location (#11010)

* fix(frontend): retrieve archived logs from correct location

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Add namespace tag handling and validation

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Remove whitespace from keyFormat

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>

* Update frontend unit tests

Signed-off-by: droctothorpe <[email protected]>

* Remove superfluous log statements

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Add link to keyFormat in manifests

Signed-off-by: droctothorpe <[email protected]>

* Fix workflow parsing for log artifact

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: quinnovator <[email protected]>

* Fix unit test

Signed-off-by: droctothorpe <[email protected]>

---------

Signed-off-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: owmasch <[email protected]>
Co-authored-by: quinnovator <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* feat(component): internal

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 663774557
Signed-off-by: KevinGrantLee <[email protected]>

* feat(component): internal

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 663872006
Signed-off-by: KevinGrantLee <[email protected]>

* chore(components): GCPC 2.16.1 Release

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 663883139
Signed-off-by: KevinGrantLee <[email protected]>

* test: Fail fast when image build fails on tests #11102 (#11115)

* Fail fast when image build fails on tests #11102

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>

* Fail fast when image build fails on tests #11102

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>

---------

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Co-authored-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(components): Use instance.target_field_name format for text-bison models only, use target_field_name for gemini models

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 665638487
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Renamed GitHub workflows from *.yaml to *.yml for consistency (#11126)

Signed-off-by: hbelmiro <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* Fix view edit cluster roles (#11067)

* Fixing incorrect typing in loop_parallism example

Signed-off-by: Oswaldo Gomez <[email protected]>

* Fixing samples/core/loop_parameter example

Signed-off-by: Oswaldo Gomez <[email protected]>

* Fixing aggregate-to-kubeflow-pipelines-edit

Signed-off-by: Oswaldo Gomez <[email protected]>

* keeping MRs separate.

Signed-off-by: Oswaldo Gomez <[email protected]>

* Adding blank line

Signed-off-by: Oswaldo Gomez <[email protected]>

---------

Signed-off-by: Oswaldo Gomez <[email protected]>
Co-authored-by: Oswaldo Gomez <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(components): Pass moddel name to eval_runner to process batch prediction's output as per the output schema of model used

Signed-off-by: Googler <[email protected]>
PiperOrigin-RevId: 665977093
Signed-off-by: KevinGrantLee <[email protected]>

* feat(components): release LLM Model Evaluation image version v0.7

Signed-off-by: Jason Dai <[email protected]>
PiperOrigin-RevId: 666102687
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Adding @DharmitD to SDK reviewers (#11131)

Signed-off-by: ddalvi <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* test: Kubeflow Pipelines V2 integration Tests (#11125)

Signed-off-by: Diego Lovison <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Add make targets for building driver and launcher images (#11103)

Signed-off-by: Giulio Frasca <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* feat(Backend + SDK): Update kfp backend and kubernetes sdk to support EmptyDir (#10913)

Update kfp backend and kubernetes sdk to support mounting EmptyDir
volumes to task pods.

Inspired by #10427

Fixes: #10656

Signed-off-by: Greg Sheremeta <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* docs:fixing broken links in readme (#11108)

Signed-off-by: Fiona Waters <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(deps): bump micromatch from 4.0.5 to 4.0.8 in /test/frontend-integration-test (#11132)

Bumps [micromatch](https://github.com/micromatch/micromatch) from 4.0.5 to 4.0.8.
- [Release notes](https://github.com/micromatch/micromatch/releases)
- [Changelog](https://github.com/micromatch/micromatch/blob/4.0.8/CHANGELOG.md)
- [Commits](micromatch/micromatch@4.0.5...4.0.8)

---
updated-dependencies:
- dependency-name: micromatch
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: KevinGrantLee <[email protected]>

* Fix: Basic sample tests - sequential is flaky (#11138)

Signed-off-by: Diego Lovison <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Wrapped "Failed GetContextByTypeAndName" error for better troubleshooting (#11098)

Signed-off-by: hbelmiro <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(components): Update AutoSxS and RLHF image tags

Signed-off-by: Michael Hu <[email protected]>
PiperOrigin-RevId: 668536503
Signed-off-by: KevinGrantLee <[email protected]>

* test: Improvements to wait_for_pods function (#11162)

Signed-off-by: hbelmiro <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(frontend): fixes filter pipeline text box shows error when typing anything in it. Fixes #10241 (#11096)

* Filter pipeline text box shows error when typing anything in it #10241

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>

* Filter pipeline text box shows error when typing anything in it #10241

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>

---------

Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Co-authored-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* correct artifact preview behavior in UI (#11059)

This change allows KFP UI to fallback to UI host namespace when no
namespaces are provided when referencing the artifact object store
provider secret, in default kubeflow deployments this namespace is
simply "kubeflow", however the user can customize this behavior by
providing the environment variable "SERVER_NAMESPACE" to the KFP UI
deployment.

Further more, this change addresses a bug that caused URL
parse to fail when parsing endpoints without a protocol, this will
support such endpoint types as <ip>:<port> for object store endpoints,
as is the case in the default kfp deployment manifests.

Signed-off-by: Humair Khan <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Added DCO link to PR template (#11176)

Signed-off-by: Helber Belmiro <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(backend): Update driver and launcher licenses (#11177)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(backend): update driver and launcher default images (#11178)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Add instructions for releasing driver and launcher images (#11179)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* test: Fixed `kfp-runtime-tests` to run on master branch (#11158)

Signed-off-by: hbelmiro <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* (fix): reduce executor logs (#11169)

* remove driver logs from executor

These logs congest the executor runtime logs making it difficult for the
user to differentiate between logs. The driver logs are unnecessary here
and can be removed to reduce this clutter.

Signed-off-by: Humair Khan <[email protected]>

* remove duplicate emissary call in executor

As per the initial inline dev comment, argo podspecpatch did not add the
emissary call, and had to be manualy added. This was fixed a couple of
argo versions back. However, as a result executor pod makes double calls
to the executor, which as a consequence also results in superflous logs.

This change removes the additional call to emissary to resolve this.

Signed-off-by: Humair Khan <[email protected]>

---------

Signed-off-by: Humair Khan <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: add PaulinaPacyna and ouadakarim as reviewers (#11180)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* test: Move run-all-gcpc-modules to GitHub Actions  (#11157)

* add gcpc modules tests to gha

Signed-off-by: Amanpreet Singh Bedi <[email protected]>

* remove run-all-gcpc-modules test driver script

Signed-off-by: Amanpreet Singh Bedi <[email protected]>

* fix path under gcpc modules tests github action

Signed-off-by: Amanpreet Singh Bedi <[email protected]>

* upgrade ubuntu base image

Signed-off-by: Amanpreet Singh Bedi <[email protected]>

* upgrade python version to 3.9

Signed-off-by: Amanpreet Singh Bedi <[email protected]>

---------

Signed-off-by: Amanpreet Singh Bedi <[email protected]>
Signed-off-by: Amanpreet Singh Bedi <[email protected]>
Co-authored-by: Amanpreet Singh Bedi <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(sdk): Kfp support for pip trusted host (#11151)

Signed-off-by: Diego Lovison <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(sdk): Loosening kubernetes dependency constraint (#11079)

* Loosening kubernetes dependency constraint

Signed-off-by: egeucak <[email protected]>

* added setuptools in test script

Signed-off-by: egeucak <[email protected]>

---------

Signed-off-by: egeucak <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Remove unwanted Frontend test files (#10973)

Signed-off-by: ddalvi <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* fix(ui): fixes empty string value in pipeline parameters (#11175)

Signed-off-by: Jan Staněk <[email protected]>
Co-authored-by: Jan Staněk <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(backend): update driver and launcher default images (#11182)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(release): bumped version to 2.3.0

Signed-off-by: KevinGrantLee <[email protected]>

* chore: Update RELEASE.md to remove obsolete instructions (#11183)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: Release kfp-pipeline-spec 0.4.0 (#11189)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: release kfp-kubernetes 1.3.0 (#11190)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore: update kfp-kubernetes release scripts and instructions (#11191)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* feat(sdk)!: Pin kfp-pipeline-spec==0.4.0, kfp-server-api>=2.1.0,<2.4.0 (#11192)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* chore(sdk): release KFP SDK 2.9.0 (#11193)

Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: KevinGrantLee <[email protected]>

* Delete test pipelines as they are duplicate with
pipeline_with_resource_spec

Signed-off-by: KevinGrantLee <[email protected]>

---------

Signed-off-by: KevinGrantLee <[email protected]>
Signed-off-by: zazulam <[email protected]>
Signed-off-by: droctothorpe <[email protected]>
Signed-off-by: Googler <[email protected]>
Signed-off-by: Giulio Frasca <[email protected]>
Signed-off-by: vmudadla <[email protected]>
Signed-off-by: Liav Weiss (EXT-Nokia) <[email protected]>
Signed-off-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Signed-off-by: hbelmiro <[email protected]>
Signed-off-by: Oswaldo Gomez <[email protected]>
Signed-off-by: Jason Dai <[email protected]>
Signed-off-by: ddalvi <[email protected]>
Signed-off-by: Diego Lovison <[email protected]>
Signed-off-by: Greg Sheremeta <[email protected]>
Signed-off-by: Fiona Waters <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Michael Hu <[email protected]>
Signed-off-by: Humair Khan <[email protected]>
Signed-off-by: Helber Belmiro <[email protected]>
Signed-off-by: Chen Sun <[email protected]>
Signed-off-by: Amanpreet Singh Bedi <[email protected]>
Signed-off-by: Amanpreet Singh Bedi <[email protected]>
Signed-off-by: egeucak <[email protected]>
Signed-off-by: Jan Staněk <[email protected]>
Co-authored-by: Michael <[email protected]>
Co-authored-by: droctothorpe <[email protected]>
Co-authored-by: andreafehrman <[email protected]>
Co-authored-by: MonicaZhang1 <[email protected]>
Co-authored-by: kylekaminky <[email protected]>
Co-authored-by: CarterFendley <[email protected]>
Co-authored-by: Googler <[email protected]>
Co-authored-by: Giulio Frasca <[email protected]>
Co-authored-by: Vani Haripriya Mudadla <[email protected]>
Co-authored-by: Liav Weiss <[email protected]>
Co-authored-by: Liav Weiss (EXT-Nokia) <[email protected]>
Co-authored-by: owmasch <[email protected]>
Co-authored-by: quinnovator <[email protected]>
Co-authored-by: ElayAharoni <[email protected]>
Co-authored-by: Elay Aharoni (EXT-Nokia) <[email protected]>
Co-authored-by: Helber Belmiro <[email protected]>
Co-authored-by: Oswaldo Gomez <[email protected]>
Co-authored-by: Oswaldo Gomez <[email protected]>
Co-authored-by: Jason Dai <[email protected]>
Co-authored-by: Dharmit Dalvi <[email protected]>
Co-authored-by: Diego Lovison <[email protected]>
Co-authored-by: Greg Sheremeta <[email protected]>
Co-authored-by: Fiona Waters <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Hu <[email protected]>
Co-authored-by: Humair Khan <[email protected]>
Co-authored-by: Chen Sun <[email protected]>
Co-authored-by: aman23bedi <[email protected]>
Co-authored-by: Amanpreet Singh Bedi <[email protected]>
Co-authored-by: ege uçak <[email protected]>
Co-authored-by: Jan Staněk <[email protected]>
Co-authored-by: Jan Staněk <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants