Template level lifecycle hooks unable to access workflow.status

[Mastergalen]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

In Argo v3.4.8 the template level lifecycle hooks fail to trigger when using workflow.status

The reason I am using a template level lifecycle hook instead of just an onExit handler, is that I want to able to destroy a Resource created by my workflow (such as a PVC) even if the workflow fails in the middle of execution.

See example use case

metadata:
    name: dynamic-pvc-6794c
  spec:
    templates:
      - name: whalesay
        inputs:
          parameters:
            - name: i
            - name: pvc-name
        outputs: {}
        metadata: {}
        container:
          name: ''
          image: docker/whalesay:latest
          command:
            - sh
            - '-c'
          args:
            - >-
              echo generating message in volume {{inputs.parameters.pvc-name}};
              cowsay hello world | tee
              /mnt/vol/hello_world-{{inputs.parameters.i}}.txt
          resources: {}
          volumeMounts:
            - name: workdir
              mountPath: /mnt/vol
        volumes:
          - name: workdir
            persistentVolumeClaim:
              claimName: '{{inputs.parameters.pvc-name}}'
      - name: print-message
        inputs:
          parameters:
            - name: i
            - name: pvc-name
        outputs: {}
        metadata: {}
        container:
          name: ''
          image: alpine:latest
          command:
            - sh
            - '-c'
          args:
            - >-
              echo getting message from volume; find /mnt/vol; cat
              /mnt/vol/hello_world-{{inputs.parameters.i}}.txt
          resources: {}
          volumeMounts:
            - name: workdir
              mountPath: /mnt/vol
        volumes:
          - name: workdir
            persistentVolumeClaim:
              claimName: '{{inputs.parameters.pvc-name}}'
      - name: generate-pvc
        inputs:
          parameters:
            - name: pvc-size
        outputs:
          parameters:
            - name: pvc-name
              valueFrom:
                jsonPath: '{.metadata.name}'
        metadata: {}
        resource:
          action: create
          manifest: |-
            apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              generateName: hera-scratch-pvc-
            spec:
              accessModes: ['ReadWriteOnce']
              resources:
                requests:
                  storage: '{{inputs.parameters.pvc-size}}'
              storageClassName: standard-rwo
                        
          setOwnerReference: true
      - name: delete-pvc
        inputs:
          parameters:
            - name: pvc-name
        outputs: {}
        metadata: {}
        resource:
          action: delete
          flags:
            - pvc
            - '{{inputs.parameters.pvc-name}}'
            - '--grace-period=0'
            - '--force'
      - name: worker
        inputs:
          parameters:
            - name: i
        outputs: {}
        metadata: {}
        steps:
          - - name: generate-volume
              template: generate-pvc
              arguments:
                parameters:
                  - name: pvc-size
                    value: 1Gi
              # This hook fails to trigger
              hooks:
                exit:
                  template: delete-pvc
                  arguments: {}
                  expression: >-
                    (workflow.status == "Error") || (workflow.status == "Failed")
                    ||(workflow.status == "Succeeded")
          - - name: whalesay
              template: whalesay
              arguments:
                parameters:
                  - name: i
                    value: '{{inputs.parameters.i}}'
                  - name: pvc-name
                    value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
          - - name: print
              template: print-message
              arguments:
                parameters:
                  - name: i
                    value: '{{inputs.parameters.i}}'
                  - name: pvc-name
                    value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
          - - name: delete-pvc
              template: delete-pvc
              arguments:
                parameters:
                  - name: pvc-name
                    value: '{{steps.generate-volume.outputs.parameters.pvc-name}}'
      - name: entrypoint
        inputs: {}
        outputs: {}
        metadata: {}
        steps:
          - - name: parallel-worker-0
              template: worker
              arguments:
                parameters:
                  - name: i
                    value: '0'
            - name: parallel-worker-1
              template: worker
              arguments:
                parameters:
                  - name: i
                    value: '1'
            - name: parallel-worker-2
              template: worker
              arguments:
                parameters:
                  - name: i
                    value: '2'
    entrypoint: entrypoint
    arguments: {}
      strategy: OnPodCompletion

Version

v3.3.8

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
      generateName: lifecycle-hook-tmpl-level-
    spec:
      entrypoint: main
      templates:
        - name: main
          steps:
            - - name: step-1
                hooks:
                  running: # Name of hook does not matter
                    # Expr will not support `-` on variable name. Variable should wrap with `[]`
                    expression: steps["step-1"].status == "Running"
                    template: echo
                  success:
                    expression: steps["step-1"].status == "Succeeded"
                    template: echo
                 
                  # None of the below hooks get triggered
                  workflow-running:
                    expression: workflow.status == "Running"
                    template: echo
                  workflow-done:
                    expression: workflow.status == "Succeeded"
                    template: echo
                  workflow-failed:
                    expression: workflow.status == "Failed"
                    template: echo
                  workflow-error:
                    expression: workflow.status == "Error"
                    template: echo
                template: echo

        - name: echo
          container:
            image: alpine:3.6
            command: [sh, -c]
            args: ["echo \"it was heads\""]

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2023-06-26T11:52:09.896Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="Updated phase  -> Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="Steps node lifecycle-hook-tmpl-level-zm7zf initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.900Z" level=info msg="StepGroup node lifecycle-hook-tmpl-level-zm7zf-4058935595 initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.901Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-3586332675 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.923Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1 (lifecycle-hook-tmpl-level-zm7zf-echo-3586332675)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.924Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:09.937Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858033 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.924Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-3586332675 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.925Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-3586332675 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.926Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.926Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-156141424 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.948Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success (lifecycle-hook-tmpl-level-zm7zf-echo-156141424)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.950Z" level=info msg="Running hooks" hookName=workflow-running lifeCycleHook=workflow-running namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.951Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-4170442462 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running (lifecycle-hook-tmpl-level-zm7zf-echo-4170442462)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.964Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:19.976Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858139 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:24.977Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-3586332675/deletePod
time="2023-06-26T11:52:29.948Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="Task-result reconciliation" namespace=default numObjs=2 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-156141424 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-4170442462 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-4170442462 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.949Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-156141424 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Running hooks" hookName=workflow-running lifeCycleHook=workflow-running namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.workflow-running" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[0].step-1.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Step group node lifecycle-hook-tmpl-level-zm7zf-4058935595 successful" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-4058935595 phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-4058935595 finished: 2023-06-26 11:52:29.950516792 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="StepGroup node lifecycle-hook-tmpl-level-zm7zf-904993486 initialized Running" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.950Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-512981488 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.962Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[1].step2 (lifecycle-hook-tmpl-level-zm7zf-echo-512981488)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-904993486 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.963Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:29.980Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858252 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:34.981Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-156141424/deletePod
time="2023-06-26T11:52:34.981Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-4170442462/deletePod
time="2023-06-26T11:52:39.964Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-512981488 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.965Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-512981488 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.966Z" level=info msg="Pod node lifecycle-hook-tmpl-level-zm7zf-1227685471 initialized Pending" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="Created pod: lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success (lifecycle-hook-tmpl-level-zm7zf-echo-1227685471)" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="Workflow step group node lifecycle-hook-tmpl-level-zm7zf-904993486 not yet completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.984Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:39.999Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=181858357 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:45.000Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-512981488/deletePod
time="2023-06-26T11:52:49.985Z" level=info msg="Processing workflow" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.986Z" level=info msg="Task-result reconciliation" namespace=default numObjs=1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.986Z" level=info msg="task-result changed" namespace=default nodeID=lifecycle-hook-tmpl-level-zm7zf-1227685471 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=lifecycle-hook-tmpl-level-zm7zf-1227685471 old.message= old.phase=Pending old.progress=0/1 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="SG Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-3586332675 are [lifecycle-hook-tmpl-level-zm7zf-3586332675]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="Running hooks" hookName=success lifeCycleHook=success namespace=default node="lifecycle-hook-tmpl-level-zm7zf[1].step2.hooks.success" workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="Step group node lifecycle-hook-tmpl-level-zm7zf-904993486 successful" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-904993486 phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.987Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf-904993486 finished: 2023-06-26 11:52:49.987975644 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Outbound nodes of lifecycle-hook-tmpl-level-zm7zf-512981488 is [lifecycle-hook-tmpl-level-zm7zf-512981488]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Outbound nodes of lifecycle-hook-tmpl-level-zm7zf is [lifecycle-hook-tmpl-level-zm7zf-512981488]" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="node lifecycle-hook-tmpl-level-zm7zf finished: 2023-06-26 11:52:49.988032529 +0000 UTC" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Checking daemoned children of lifecycle-hook-tmpl-level-zm7zf" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg=reconcileAgentPod namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Marking workflow completed" namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.988Z" level=info msg="Checking daemoned children of " namespace=default workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:49.993Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-1340600742-agent/deletePod
time="2023-06-26T11:52:50.001Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=181858458 workflow=lifecycle-hook-tmpl-level-zm7zf
time="2023-06-26T11:52:50.001Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h59m59s due to TTL"
time="2023-06-26T11:52:55.026Z" level=info msg="cleaning up pod" action=deletePod key=default/lifecycle-hook-tmpl-level-zm7zf-echo-1227685471/deletePod
time="2023-06-26T12:24:17.471Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h28m32s due to TTL"
time="2023-06-26T12:44:17.472Z" level=info msg="Queueing Succeeded workflow default/lifecycle-hook-tmpl-level-zm7zf for delete in 167h8m32s due to TTL"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

None