[backend] Query Strings in the Pipeline Root are ignored

[TobiasGoerke]

To my understanding, using custom artifact stores in pipelines v2 requires configuring the pipelineRoot.
According to the blob documentation, an endpoint needs to be configured by using a queryString that gets appended to the pipelineRoot.

However, the kfp driver ignores query strings and assembles wrong bucket names.

This issue effectively blocks using custom S3 artifact stores for pipelines v2.

Environment

KFP 2.0.5.

Steps to reproduce

Set a custom Pipeline Root, e.g. by setting the kfp-launcher's ConfigMap key defaultPipelineRoot to s3://my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true.

Expected result

We'll see errors like

failed to execute component: Failed to open bucket "my-bucket": open bucket s3//my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true%2Fmy-task-name%2Fmy-artifact-name: invalid value for query parameter "s3ForcePathStyle": strconv.ParseBool: parsing true/my-task-name/my-artifact-name: invalid syntax

Also, after the driver has been executed, the wrongly concatenated pipelineRoot is visible in the workflow yaml.

---

Impacted by this bug? Give it a 👍.

[TobiasGoerke]

Using custom s3 endpoints now works for me (given #10319):

• Set the defaultPipelineRoot in the kfp-launcher ConfigMap to e.g. s3://my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true
• Set AWS_REGION env
• Mount mlpipeline-minio-artifact (or another secret that contains your s3 credentials) to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

[RonaldFletcher]

Using custom s3 endpoints now works for me (given #10319):

• Set the defaultPipelineRoot in the kfp-launcher ConfigMap to e.g. s3://my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true
• Set AWS_REGION env
• Mount mlpipeline-minio-artifact (or another secret that contains your s3 credentials) to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

I used minio, I configured it this way, but it didn't work @TobiasGoerke

[TobiasGoerke]

Using custom s3 endpoints now works for me (given #10319):

• Set the defaultPipelineRoot in the kfp-launcher ConfigMap to e.g. s3://my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true
• Set AWS_REGION env
• Mount mlpipeline-minio-artifact (or another secret that contains your s3 credentials) to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

I used minio, I configured it this way, but it didn't work @TobiasGoerke

Hard to say what's going wrong given your screenshot.
I'd assume your container is misconfigured and is missing permissions to create directories under `/tmp/outputs/', see last two lines

[RonaldFletcher]

Using custom s3 endpoints now works for me (given #10319):

• Set the defaultPipelineRoot in the kfp-launcher ConfigMap to e.g. s3://my-bucket/my-dir?endpoint=my.endpoint&s3ForcePathStyle=true
• Set AWS_REGION env
• Mount mlpipeline-minio-artifact (or another secret that contains your s3 credentials) to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

I used minio, I configured it this way, but it didn't work @TobiasGoerke

Hard to say what's going wrong given your screenshot. I'd assume your container is misconfigured and is missing permissions to create directories under `/tmp/outputs/', see last two lines

This is the example I use

I don't know why there is such an error，and How do AWS_SECRET_ACCESS_KEY inject these environment variables into the environment variables of that service? @TobiasGoerke

[TobiasGoerke]

You'll either need to specify these vars manually or use tools like OPA / Kyverno.
However, it does not look like the message is related to auth.

[RonaldFletcher]

You'll either need to specify these vars manually or use tools like OPA / Kyverno. However, it does not look like the message is related to auth.

defaultPipelineRoot: s3://kubeflow-pipelines/v2?endpoint=minio.examplecom:443&s3ForcePathStyle=true
and ml-pipeline-apiserver deploymeny add env

Is this all the configuration? Are the changes I made correct? @TobiasGoerke

[TobiasGoerke]

Seems correct. Although you will also need to mount the env variables to you pipeline pods, like this (better you OPA / Kyverno though):

task.set_env_variable("AWS_REGION", "eu-central-1")
task.set_env_variable("AWS_ACCESS_KEY_ID", "minio")
task.set_env_variable("AWS_SECRET_ACCESS_KEY", "minio123")