Workflows fail when cluster-scoped resources have successCondition

[Kyle-Wong]

Checklist

• Double-checked my configuration.
• Tested using the latest version.
• Used the Emissary executor.

Summary

What happened/what you expected to happen?

I expected this workflow to succeed. It applies a namespace and has a successCondition. The resource is applied successfully, but the workflow is marked as failed because of The resource has been deleted while its status was still being checked. Will not be retried: the server could not find the requested resource. This is because the selflink for the namespace resource is malformed: SelfLink: api/v1/namespaces//namespaces/kubeflow. This selflink is used during the evaluation of successConditions. This issue still exists in latest because inferSelfObjectLink doesn't account for cluster-scoped resources. According to kubernetes docs, the correct selflink should be /api/v1/namespaces/kubeflow

What version are you running?
Argo workflows v2.3.11, but the issue still exists in latest version

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        metadata:
          labels:
            workflows.argoproj.io/controller-instanceid: workflow-controller
          name: workflow
        spec:
          entrypoint: entry
          serviceAccountName: default
          templates:
          - name: entry
            steps:
            - - name: prereq-resources
                template: prereq-resources-install
          - name: prereq-resources-install
            resource:
              action: apply
              successCondition: status.phase == Active
              manifest: |
                apiVersion: v1
                kind: Namespace
                metadata:
                  name: kubeflow

# Logs from workflow pod
time="2022-08-05T23:33:33.997Z" level=info msg="Resource: /namespace./kubeflow. SelfLink: api/v1/namespaces//namespaces/kubeflow"
time="2022-08-05T23:33:33.998Z" level=info msg="Waiting for conditions: status.phase==Active"
time="2022-08-05T23:33:34.003Z" level=info msg="Get namespaces 404"
time="2022-08-05T23:33:34.003Z" level=warning msg="Non-transient error: The resource has been deleted while its status was still being checked. Will not be retried: the server could not find the requested resource"

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow} 

time="2022-08-08T17:18:20.676Z" level=info msg="Processing workflow" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="Pod failed: Error (exit code 1): The resource has been deleted while its status was still being checked. Will not be retried: the server could not find the requested resource" displayName=prereq-resources namespace=addon-manager-system pod=kubeflow-prereqs-b9539c3c-wf-2764095841 templateName=prereq-resources-install workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="Updating node kubeflow-prereqs-b9539c3c-wf-2764095841 exit code 1" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="Updating node kubeflow-prereqs-b9539c3c-wf-2764095841 status Pending -> Failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="Updating node kubeflow-prereqs-b9539c3c-wf-2764095841 message: Error (exit code 1): The resource has been deleted while its status was still being checked. Will not be retried: the server could not find the requested resource" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="Step group node kubeflow-prereqs-b9539c3c-wf-3136752568 deemed failed: child 'kubeflow-prereqs-b9539c3c-wf-2764095841' failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-3136752568 phase Running -> Failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-3136752568 message: child 'kubeflow-prereqs-b9539c3c-wf-2764095841' failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.677Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-3136752568 finished: 2022-08-08 17:18:20.67792632 +0000 UTC" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="step group kubeflow-prereqs-b9539c3c-wf-3136752568 was unsuccessful: child 'kubeflow-prereqs-b9539c3c-wf-2764095841' failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="Outbound nodes of kubeflow-prereqs-b9539c3c-wf-2764095841 is [kubeflow-prereqs-b9539c3c-wf-2764095841]" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="Outbound nodes of kubeflow-prereqs-b9539c3c-wf-683026374 is [kubeflow-prereqs-b9539c3c-wf-2764095841]" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-683026374 phase Running -> Failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-683026374 message: child 'kubeflow-prereqs-b9539c3c-wf-2764095841' failed" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf
time="2022-08-08T17:18:20.678Z" level=info msg="node kubeflow-prereqs-b9539c3c-wf-683026374 finished: 2022-08-08 17:18:20.678149711 +0000 UTC" namespace=addon-manager-system workflow=kubeflow-prereqs-b9539c3c-wf

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/psp: eks.privileged
      vpc.amazonaws.com/pod-ips: ""
      workflows.argoproj.io/node-id: kubeflow-prereqs-b9539c3c-wf-2764095841
      workflows.argoproj.io/node-name: kubeflow-prereqs-b9539c3c-wf(0)[0].prereq-resources
    creationTimestamp: "2022-08-08T17:18:10Z"
    labels:
      workflows.argoproj.io/completed: "true"
      workflows.argoproj.io/controller-instanceid: addon-manager-workflow-controller
      workflows.argoproj.io/workflow: kubeflow-prereqs-b9539c3c-wf
    name: kubeflow-prereqs-b9539c3c-wf-2764095841
    namespace: addon-manager-system
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: kubeflow-prereqs-b9539c3c-wf
      uid: 912fc861-58ea-41fa-bf07-f8ab3c877d6a
    resourceVersion: "5067407"
    uid: 85a4b071-a9b2-410e-ad80-024c6c80c9ff
  spec:
    activeDeadlineSeconds: 599
    containers:
    - command:
      - argoexec
      - resource
      - apply
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: pns
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: kubeflow-prereqs-b9539c3c-wf
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_TEMPLATE
        value: '{"name":"prereq-resources-install","inputs":{},"outputs":{},"metadata":{},"resource":{"action":"apply","manifest":"apiVersion:
          v1\nkind: Namespace\nmetadata:\n    annotations:\n        iks.intuit.com/allowHostIPC:
          \"false\"\n        iks.intuit.com/allowHostNetwork: \"false\"\n        iks.intuit.com/allowHostPID:
          \"false\"\n        iks.intuit.com/allowHostPort: \"false\"\n        iks.intuit.com/allowPrivilegeEscalation:
          \"true\"\n        iks.intuit.com/allowPrivileged: \"true\"\n        iks.intuit.com/allowed-igs:
          nodes\n        iks.intuit.com/allowedHostPaths: \"\"\n        iks.intuit.com/managed:
          \"false\"\n        iks.intuit.com/service-asset-alias: Intuit.data.mlplatform.mlpinfrastructure\n        iks.intuit.com/service-asset-id:
          \"8001788230453915925\"\n    labels:\n        app.kubernetes.io/managed-by:
          addonmgr.keikoproj.io\n        app.kubernetes.io/name: kubeflow\n        app.kubernetes.io/part-of:
          kubeflow\n        app.kubernetes.io/version: v1.0.0\n        istio-injection:
          enabled\n    name: kubeflow\n","successCondition":"status.phase == Active"}}'
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "2022-08-08T17:28:10Z"
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imagePullPolicy: IfNotPresent
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-vpdp2
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: ip-10-205-126-187.us-east-2.compute.internal
    nodeSelector:
      node.kubernetes.io/instancegroup: system
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: addon-manager-workflow-installer-sa
    serviceAccountName: addon-manager-workflow-installer-sa
    shareProcessNamespace: true
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    - key: ig/system
    volumes:
    - name: kube-api-access-vpdp2
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:10Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:12Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:12Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:10Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://b7c4179226c99ea28d9cce22d2ae660068253e2eb86504d9d62d91263cfb458b
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imageID: docker.intuit.com/quay-rmt/argoproj/argoexec@sha256:96390f7ea826f7a918d697f9ff2bdc79c74a343712275f8e3c90c18f474f6b92
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://b7c4179226c99ea28d9cce22d2ae660068253e2eb86504d9d62d91263cfb458b
          exitCode: 1
          finishedAt: "2022-08-08T17:18:12Z"
          message: 'The resource has been deleted while its status was still being
            checked. Will not be retried: the server could not find the requested
            resource'
          reason: Error
          startedAt: "2022-08-08T17:18:11Z"
    hostIP: 10.205.126.187
    phase: Failed
    podIP: 10.205.124.58
    podIPs:
    - ip: 10.205.124.58
    qosClass: BestEffort
    startTime: "2022-08-08T17:18:10Z"
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/psp: eks.privileged
      vpc.amazonaws.com/pod-ips: ""
      workflows.argoproj.io/node-id: kubeflow-prereqs-b9539c3c-wf-3201628028
      workflows.argoproj.io/node-name: kubeflow-prereqs-b9539c3c-wf(1)[0].prereq-resources
    creationTimestamp: "2022-08-08T17:18:20Z"
    labels:
      workflows.argoproj.io/completed: "true"
      workflows.argoproj.io/controller-instanceid: addon-manager-workflow-controller
      workflows.argoproj.io/workflow: kubeflow-prereqs-b9539c3c-wf
    name: kubeflow-prereqs-b9539c3c-wf-3201628028
    namespace: addon-manager-system
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: kubeflow-prereqs-b9539c3c-wf
      uid: 912fc861-58ea-41fa-bf07-f8ab3c877d6a
    resourceVersion: "5067517"
    uid: b4e0c0f5-3c7e-4370-87d6-c0522736e150
  spec:
    activeDeadlineSeconds: 589
    containers:
    - command:
      - argoexec
      - resource
      - apply
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: pns
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: kubeflow-prereqs-b9539c3c-wf
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_TEMPLATE
        value: '{"name":"prereq-resources-install","inputs":{},"outputs":{},"metadata":{},"resource":{"action":"apply","manifest":"apiVersion:
          v1\nkind: Namespace\nmetadata:\n    annotations:\n        iks.intuit.com/allowHostIPC:
          \"false\"\n        iks.intuit.com/allowHostNetwork: \"false\"\n        iks.intuit.com/allowHostPID:
          \"false\"\n        iks.intuit.com/allowHostPort: \"false\"\n        iks.intuit.com/allowPrivilegeEscalation:
          \"true\"\n        iks.intuit.com/allowPrivileged: \"true\"\n        iks.intuit.com/allowed-igs:
          nodes\n        iks.intuit.com/allowedHostPaths: \"\"\n        iks.intuit.com/managed:
          \"false\"\n        iks.intuit.com/service-asset-alias: Intuit.data.mlplatform.mlpinfrastructure\n        iks.intuit.com/service-asset-id:
          \"8001788230453915925\"\n    labels:\n        app.kubernetes.io/managed-by:
          addonmgr.keikoproj.io\n        app.kubernetes.io/name: kubeflow\n        app.kubernetes.io/part-of:
          kubeflow\n        app.kubernetes.io/version: v1.0.0\n        istio-injection:
          enabled\n    name: kubeflow\n","successCondition":"status.phase == Active"}}'
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "2022-08-08T17:28:10Z"
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imagePullPolicy: IfNotPresent
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-8nhww
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: ip-10-205-126-187.us-east-2.compute.internal
    nodeSelector:
      node.kubernetes.io/instancegroup: system
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: addon-manager-workflow-installer-sa
    serviceAccountName: addon-manager-workflow-installer-sa
    shareProcessNamespace: true
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    - key: ig/system
    volumes:
    - name: kube-api-access-8nhww
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:20Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:23Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:23Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:20Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://dc8d8c0dbf63fe016e881e76a6b493c881a08d4775754e2c8af06a95e4abd29a
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imageID: docker.intuit.com/quay-rmt/argoproj/argoexec@sha256:96390f7ea826f7a918d697f9ff2bdc79c74a343712275f8e3c90c18f474f6b92
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://dc8d8c0dbf63fe016e881e76a6b493c881a08d4775754e2c8af06a95e4abd29a
          exitCode: 1
          finishedAt: "2022-08-08T17:18:23Z"
          message: 'The resource has been deleted while its status was still being
            checked. Will not be retried: the server could not find the requested
            resource'
          reason: Error
          startedAt: "2022-08-08T17:18:21Z"
    hostIP: 10.205.126.187
    phase: Failed
    podIP: 10.205.125.39
    podIPs:
    - ip: 10.205.125.39
    qosClass: BestEffort
    startTime: "2022-08-08T17:18:20Z"
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/psp: eks.privileged
      vpc.amazonaws.com/pod-ips: ""
      workflows.argoproj.io/node-id: kubeflow-prereqs-b9539c3c-wf-4228090283
      workflows.argoproj.io/node-name: kubeflow-prereqs-b9539c3c-wf(2)[0].prereq-resources
    creationTimestamp: "2022-08-08T17:18:30Z"
    labels:
      workflows.argoproj.io/completed: "true"
      workflows.argoproj.io/controller-instanceid: addon-manager-workflow-controller
      workflows.argoproj.io/workflow: kubeflow-prereqs-b9539c3c-wf
    name: kubeflow-prereqs-b9539c3c-wf-4228090283
    namespace: addon-manager-system
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: kubeflow-prereqs-b9539c3c-wf
      uid: 912fc861-58ea-41fa-bf07-f8ab3c877d6a
    resourceVersion: "5067630"
    uid: fc8058a9-b096-4783-a9e6-503a876d03c5
  spec:
    activeDeadlineSeconds: 579
    containers:
    - command:
      - argoexec
      - resource
      - apply
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
        value: pns
      - name: GODEBUG
        value: x509ignoreCN=0
      - name: ARGO_WORKFLOW_NAME
        value: kubeflow-prereqs-b9539c3c-wf
      - name: ARGO_CONTAINER_NAME
        value: main
      - name: ARGO_TEMPLATE
        value: '{"name":"prereq-resources-install","inputs":{},"outputs":{},"metadata":{},"resource":{"action":"apply","manifest":"apiVersion:
          v1\nkind: Namespace\nmetadata:\n    annotations:\n        iks.intuit.com/allowHostIPC:
          \"false\"\n        iks.intuit.com/allowHostNetwork: \"false\"\n        iks.intuit.com/allowHostPID:
          \"false\"\n        iks.intuit.com/allowHostPort: \"false\"\n        iks.intuit.com/allowPrivilegeEscalation:
          \"true\"\n        iks.intuit.com/allowPrivileged: \"true\"\n        iks.intuit.com/allowed-igs:
          nodes\n        iks.intuit.com/allowedHostPaths: \"\"\n        iks.intuit.com/managed:
          \"false\"\n        iks.intuit.com/service-asset-alias: Intuit.data.mlplatform.mlpinfrastructure\n        iks.intuit.com/service-asset-id:
          \"8001788230453915925\"\n    labels:\n        app.kubernetes.io/managed-by:
          addonmgr.keikoproj.io\n        app.kubernetes.io/name: kubeflow\n        app.kubernetes.io/part-of:
          kubeflow\n        app.kubernetes.io/version: v1.0.0\n        istio-injection:
          enabled\n    name: kubeflow\n","successCondition":"status.phase == Active"}}'
      - name: ARGO_INCLUDE_SCRIPT_OUTPUT
        value: "false"
      - name: ARGO_DEADLINE
        value: "2022-08-08T17:28:10Z"
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imagePullPolicy: IfNotPresent
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-bzwkx
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: ip-10-205-126-187.us-east-2.compute.internal
    nodeSelector:
      node.kubernetes.io/instancegroup: system
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: addon-manager-workflow-installer-sa
    serviceAccountName: addon-manager-workflow-installer-sa
    shareProcessNamespace: true
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    - key: ig/system
    volumes:
    - name: kube-api-access-bzwkx
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:30Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:32Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:32Z"
      message: 'containers with unready status: [main]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2022-08-08T17:18:30Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://194ebe8cbe4d5098751e15df67f9d009181db6519853b8c8997752d1f6030e84
      image: docker.intuit.com/quay-rmt/argoproj/argoexec:v3.2.11
      imageID: docker.intuit.com/quay-rmt/argoproj/argoexec@sha256:96390f7ea826f7a918d697f9ff2bdc79c74a343712275f8e3c90c18f474f6b92
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: containerd://194ebe8cbe4d5098751e15df67f9d009181db6519853b8c8997752d1f6030e84
          exitCode: 1
          finishedAt: "2022-08-08T17:18:32Z"
          message: 'The resource has been deleted while its status was still being
            checked. Will not be retried: the server could not find the requested
            resource'
          reason: Error
          startedAt: "2022-08-08T17:18:31Z"
    hostIP: 10.205.126.187
    phase: Failed
    podIP: 10.205.124.15
    podIPs:
    - ip: 10.205.124.15
    qosClass: BestEffort
    startTime: "2022-08-08T17:18:30Z"
kind: List
metadata:
  resourceVersion: ""

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[terrytangyuan]

Good catch. Would you like to submit a PR to fix this?

[Kyle-Wong]

Good catch. Would you like to submit a PR to fix this?

Sure, I'll do that

[sarabala1979]

@Kyle-Wong can you try to reproduce on 3.3 release?

[Kyle-Wong]

I was able to reproduce the issue on 3.3 release.

[terrytangyuan]

Yeah it should still be a problem. Please submit a PR and we'll review.