cannot submit WorkflowTemplate with {{workflow.labels.xxxx}} values from v3.3.0

[yu-croco]

Checklist

• Double-checked my configuration.
• Tested using the latest version.
• Used the Emissary executor.

Summary

What happened/what you expected to happen?
When I submit WorkflowTemplate with {{workflow.labels.workflow-name}} , I get status 500, templates.argosay: failed to resolve {{workflow.labels.workflow-name}} though the attribute is set on .metadata.labels below Diagnostics .

It does not happen on v3.2.11, but it happens from v3.3.0 to v3.3.5 .
I couldn't find this kind of breaking change on Upgrading to v3.3, so I am wondering if this is a bug. 🤔

What version are you running?

v3.3.5

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

metadata:
  name: sparkly-rhino
  namespace: foo
  labels:
    example: 'true'
    workflow-name: hello-world
spec:
  templates:
    - name: argosay
      container:
        name: main
        image: 'argoproj/argosay:v2'
        command:
          - /argosay
        args:
          - echo
          - '{{workflow.labels.workflow-name}}'
        resources: {}
  entrypoint: argosay
  arguments: {}
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion
  workflowMetadata:
    labels:
      example: 'true'
      workflow-name: hello-world

# Logs from workflow server when I submitted WorkflowTemplate:
time="2022-05-25T11:37:29.175Z" level=error msg="finished unary call with code Unknown" error="templates.argosay: failed to resolve {{workflow.labels.workflow-name}}" grpc.code=Unknown grpc.method=SubmitWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2022-05-25T11:37:29Z" grpc.time_ms=29.021 span.kind=server system=grpc

# response on Google Chrome developer tool when I submitted WorkflowTemplate:
{
  code: 2,
  message: "templates.argosay: failed to resolve {{workflow.labels.workflow-name}}"
}

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[juliev0]

@sarabala1979 Can you assign this to me?

[juliev0]

I just tried this using the latest master and couldn't reproduce it. To make sure I did the same thing: I went to "Workflow Templates" on the UI, clicked "Create New Workflow Template", entered the yaml that you showed (except changing the namespace to mine), and clicked "Create". I didn't see an error in the UI or see "finished unary call with code Unknown" in the Workflow Controller log. I suppose I can try with the versions you specified.

[yu-croco]

@juliev0
Thank you for investigating!

and clicked "Create". I didn't see an error in the UI or ....

In my report, it happened when I clicked Submit button. 🙋
*I can create Workflow Template, but I cannot submit.

[juliev0]

Thanks for clarifying! I'll try that then

[juliev0]

So, I did reproduce your error and am looking into it. But just curious: if you try running with 3.2.11 does it work end to end? I see the call to "submit" works fine but then the node itself shows this error:

Error (exit code 1): WorkflowTaskResult.argoproj.io "" is invalid: [metadata.name: Required value: name or generateName is required, metadata.ownerReferences.uid: Invalid value: "": uid must not be empty]

Do you see that as well? Perhaps the spec changed?

[yu-croco]

It worked without errors in my case, as below. 🙋
This time, I installed Argo Workflow v3.2.11 by referring Quick Start on my local environment and checked.
*Precondition is different from original report (this was checked on production related environment) but I could submit Workflow Template successfully in v3.2.11 and failed in v3.3.x

full yaml is below in this time 🙋

metadata:
  name: sparkly-rhino
  namespace: argo
  uid: 898c9721-bff3-4980-8a6f-f3803e9f76ba
  resourceVersion: '1452735'
  generation: 1
  creationTimestamp: '2022-05-28T00:03:41Z'
  labels:
    example: 'true'
    workflow-name: hello-world
    workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server
  managedFields:
    - manager: argo
      operation: Update
      apiVersion: argoproj.io/v1alpha1
      time: '2022-05-28T00:03:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:example': {}
            'f:workflow-name': {}
            'f:workflows.argoproj.io/creator': {}
        'f:spec': {}
spec:
  templates:
    - name: argosay
      inputs: {}
      outputs: {}
      metadata: {}
      container:
        name: main
        image: 'argoproj/argosay:v2'
        command:
          - /argosay
        args:
          - echo
          - '{{workflow.labels.workflow-name}}'
        resources: {}
  entrypoint: argosay
  arguments: {}
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion
  workflowMetadata:
    creationTimestamp: null
    labels:
      example: 'true'
      workflow-name: hello-world

[juliev0]

Thanks so much for the quick response and for posting the full yaml. I realized I was accidentally still running the latest version of the underlying Docker image while running 3.2.11 for the argo server and Workflow Controller. Once I fixed that, I can also get it to work successfully for 3.2.11.

[juliev0]

@alexec Removing the code from workflow/common/convert.go which transfers the Labels from the WorkflowTemplate to the Workflow as part of this change appears to cause this:

This causes them to not get added to ctx.globalParams here, which causes them to not get picked up here.

Any thoughts on why the code which transfers Labels from the WorkflowTemplate to the Workflow would have been removed?

[sandeepk8s]

Hi @juliev0 @yu-croco I see the label name has '-' in it. I faced similar issue when wf submitted using workfloweventbinding thru payload. Resolved by removing '-'

Did you try using something like below ? If not could you please try

workflow_name: hello-world

we are on v3.3.5

[yu-croco]

@sandeepitachi
Thank you for your comment! I tried and failed to submit... 😇

Full yaml is here.

metadata:
  name: sparkly-rhino
  namespace: argo
  uid: 898c9721-bff3-4980-8a6f-f3803e9f76ba
  resourceVersion: '1542289'
  generation: 2
  creationTimestamp: '2022-05-28T00:03:41Z'
  labels:
    example: 'true'
    workflow_name: hello-world
    workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server
  managedFields:
    - manager: argo
      operation: Update
      apiVersion: argoproj.io/v1alpha1
      time: '2022-05-31T00:42:16Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:example': {}
            'f:workflow_name': {}
            'f:workflows.argoproj.io/creator': {}
        'f:spec': {}
spec:
  templates:
    - name: argosay
      inputs: {}
      outputs: {}
      metadata: {}
      container:
        name: main
        image: 'argoproj/argosay:v2'
        command:
          - /argosay
        args:
          - echo
          - '{{workflow.labels.workflow_name}}'
        resources: {}
  entrypoint: argosay
  arguments: {}
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion
  workflowMetadata:
    labels:
      example: 'true'
      workflow_name: hello-world

According to #8837 (comment) , additional attributes are not accepted... 👀

This causes them to not get added to ctx.globalParams here, which causes them to not get picked up here.

[yu-croco]

Oh.... 😇

Could you please check your wf events or pod logs and see what executor image version you are running 😄 ??

executor image version means the version of argoproj/argoexec ? 👀
If so, I tried 2 patterns and both of them are same result.

1.

I used quick start and the version is latest .

$ k logs -f -n argo workflow-controller-b99cbc8bf-v582b
time="2022-05-31T02:07:52Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2022-05-31T02:07:52Z" level=info msg="cron config" cronSyncPeriod=10s
time="2022-05-31T02:07:52Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
time="2022-05-31T02:07:52.622Z" level=info msg="not enabling pprof debug endpoints"
time="2022-05-31T02:07:52.629Z" level=info msg="Get configmaps 200"
time="2022-05-31T02:07:52.632Z" level=info msg="Configuration:\nartifactRepository:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: my-minio-cred\n    bucket: my-bucket\n    endpoint: minio:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: my-minio-cred\nexecutor:\n  name: \"\"\n  resources:\n    requests:\n      cpu: 10m\n      memory: 64Mi\nimages:\n  docker/whalesay:latest:\n    cmd:\n    - cowsay\ninitialDelay: 0s\nlinks:\n- name: Workflow Link\n  scope: workflow\n  url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Pod Link\n  scope: pod\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Pod Logs Link\n  scope: pod-logs\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Event Source Logs Link\n  scope: event-source-logs\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Sensor Logs Link\n  scope: sensor-logs\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\nmetricsConfig:\n  enabled: true\n  path: /metrics\n  port: 9090\nnamespaceParallelism: 10\nnodeEvents: {}\npersistence:\n  archive: true\n  archiveTTL: 168h0m0s\n  connectionPool:\n    maxIdleConns: 100\n  nodeStatusOffLoad: true\n  postgresql:\n    database: postgres\n    host: postgres\n    passwordSecret:\n      key: password\n      name: argo-postgres-config\n    port: 5432\n    tableName: argo_workflows\n    userNameSecret:\n      key: username\n      name: argo-postgres-config\npodSpecLogStrategy: {}\nretentionPolicy:\n  completed: 10\n  errored: 3\n  failed: 3\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
time="2022-05-31T02:07:52.632Z" level=info msg="Persistence configuration enabled"
time="2022-05-31T02:07:52.632Z" level=info msg="Get secrets 200"
time="2022-05-31T02:07:52.633Z" level=info msg="Get secrets 200"
time="2022-05-31T02:07:52.636Z" level=info msg="Persistence Session created successfully"
time="2022-05-31T02:07:52.636Z" level=info msg="Migrating database schema" clusterName=default dbType=postgres
time="2022-05-31T02:07:52.665Z" level=info msg="Node status offloading is enabled"
time="2022-05-31T02:07:52.665Z" level=info msg="Workflow archiving is enabled"
time="2022-05-31T02:07:52.665Z" level=info executorImage="quay.io/argoproj/argoexec:latest" executorImagePullPolicy= managedNamespace=argo
I0531 02:07:52.666062       1 leaderelection.go:248] attempting to acquire leader lease argo/workflow-controller...
...

2.

I used helm chart and version is v3.3.5 , same as argo workflow controller.

[sandeepk8s]

@yu-croco my bad! I misunderstood the issue was using (global variables) wf labels as input parameters!

Yah you are right - the workflow labels variables are unavailable

I tried and getting the same error 😔says failed to resolve!

[yu-croco]

@sandeepitachi
Thank you for ckecking! It's ok, it's kinda confusing one 😇
I will wait for the reply against this #8837 (comment)

[juliev0]

FYI, I got some more information on this one so now have a path to resolving.