[test] All log is 404 in argo

[gaocegege]

http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-tf-operator-presubmit-tfjob-e2e-492-b317aef-225-99c4?tab=workflow&nodeId=kubeflow-tf-operator-presubmit-tfjob-e2e-492-b317aef-225-99c4-3605379172&sidePanel=logs%3Akubeflow-tf-operator-presubmit-tfjob-e2e-492-b317aef-225-99c4-3605379172%3Amain

The log of element in argo DAG is 404 although the element is succeeded

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"kubeflow-tf-operator-presubmit-tfjob-e2e-492-b317aef-225-99c4-3605379172\" not found","r
eason":"NotFound","details":{"name":"kubeflow-tf-operator-presubmit-tfjob-e2e-492-b317aef-225-99c4-3605379172","kind":"pods"},"code":404}

[gaocegege]

/assign @jlewi

[jlewi]

Duplicate of argoproj/argo-workflows#777

The work around is to access the logs directly via stackdriver as described here.

@gaocegege @ScorpioCPH You should both have access to stackdriver.

[gaocegege]

Thanks for your comment!