Artifacts in Azure Blob Storage can't be downloaded from UI

[svenakela]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

We've been fighting this issue a few days and start to think this might be a bug in argo-server and how the authentication towards Azure works.

We have two separate Argo WF instances and they have their own Azure Blob storage configured with a managed identity authentication as the artifact repo. Workflows can save logs and output/input artifacts with no problem.

We also have SSO with Entra configured in both clusters in case that matters and if I go to any finalized step and try to download the artifacts I get a black screen, a 500 internal error and the log below is written to the argo-server log.

All suggestions we've found is tested with the same error.
Adding the azure.workload.identity/use: "true" label, and the client ID as an annotation for the argo-server pod via the deployment. We've tried to set both AZURE_TENANT_ID and AZURE_CLIENT_ID as env vars hardcoded in the deployment for the argo-server pod. We've also tried to label and annotate all imaginable service accounts with no effect.

We still get the error below.
Workflows are working seamlessly and both logs and artifacts are stored in the storage account, so we are pretty darn sure that the workload configuration is working. But whatever we try, the UI refuse to allow a download.

For reference, the screen shots represents the links where the UI will dump the error.

I've backported to v3.6.7 and back with no difference. There are a few similar discussions back when v3.5.5 was a thing but that version is too old for me to try.

Version(s)

v3.6.7, v3.7.1

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-artifact-template
  namespace: argo
spec:
  volumes:
    - name: tmp
      emptyDir: {}
  templates:
    - name: main
      inputs:
        parameters:
          - name: message
            value: "{{workflow.parameters.message}}"
      container:
        image: busybox
        command: [sh, -c]
        args: ["echo {{inputs.parameters.message}} | tee /tmp/hello_world.txt"]
        volumeMounts:
          - name: tmp
            mountPath: /tmp
      outputs:
        artifacts:
        - name: inputfile
          path: /tmp/hello_world.txt
  entrypoint: main

Logs from the workflow controller

workflow-controller logs nothing, but the argo-server do:


level=error msg="Artifact Server returned internal error" error="error listing blobs xyz/opq/inputfile.tgz in Azure Blob Storage container: DefaultAzureCredential: failed to acquire a token.
Attempted credentials:
  EnvironmentCredential: missing environment variable AZURE_TENANT_ID
    WorkloadIdentityCredential: no client ID specified. Check pod configuration or set ClientID in the options
      ManagedIdentityCredential: failed to authenticate a system assigned identity. The endpoint responded with {\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"}
        AzureCLICredential: fork/exec /bin/sh: no such file or directory\n\tAzureDeveloperCLICredential: fork/exec /bin/sh: no such file or directory"

Logs from in your workflow's wait container

n/a