CopyFile and GetFileContents of kubelet executor don't work

[dtaniwaki]

Is this a BUG REPORT or FEATURE REQUEST?:

BUG REPORT

What happened:

I'm implementing K8sAPI executor and almost finished the implementation based on the kubelet executor. However, I found CopyFile and GetFileContents of the kubelet executor don't work because the container to execute commands is terminated.

Here's the wait container log when I tried the artifact-passing example.

wait-container.log

time="2018-09-14T03:02:10Z" level=info msg="Creating a kubelet executor"
time="2018-09-14T03:02:10Z" level=info msg="Non configured envvar ARGO_KUBELET_PORT, defaulting the kubelet port to 10250"
time="2018-09-14T03:02:10Z" level=warning msg="Using a kubelet client with insecure options"
time="2018-09-14T03:02:10Z" level=info msg="Executor (version: v2.2.0+7080bfc.dirty, build_date: 2018-09-14T02:21:09Z) initialized with template:\narchiveLocation:\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: my-minio-cred\n    bucket: my-bucket\n    endpoint: argo-artifacts-minio.default:9000\n    insecure: true\n    key: artifact-passing-qp8kj/artifact-passing-qp8kj-36053786\n    secretKeySecret:\n      key: secretkey\n      name: my-minio-cred\ncontainer:\n  args:\n  - cowsay hello world | tee /tmp/hello_world.txt\n  command:\n  - sh\n  - -c\n  image: docker/whalesay:latest\n  name: \"\"\n  resources: {}\ninputs: {}\nmetadata: {}\nname: whalesay\noutputs:\n  artifacts:\n  - name: hello-art\n    path: /tmp/hello_world.txt\n"
time="2018-09-14T03:02:10Z" level=info msg="Waiting on main container"
time="2018-09-14T03:02:12Z" level=info msg="main container started with container ID: 52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4"
time="2018-09-14T03:02:12Z" level=info msg="Starting annotations monitor"
time="2018-09-14T03:02:12Z" level=info msg="Starting to wait completion of containerID 52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4 ..."
time="2018-09-14T03:02:12Z" level=info msg="Starting deadline monitor"
time="2018-09-14T03:02:13Z" level=info msg="ContainerID \"52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4\" is terminated: &ContainerStatus{Name:main,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2018-09-14 03:02:10 +0000 UTC,FinishedAt:2018-09-14 03:02:10 +0000 UTC,ContainerID:docker://52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:docker/whalesay:latest,ImageID:docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b,ContainerID:docker://52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4,}"
time="2018-09-14T03:02:13Z" level=info msg="Main container completed"
time="2018-09-14T03:02:13Z" level=info msg="No sidecars"
time="2018-09-14T03:02:13Z" level=info msg="Saving output artifacts"
time="2018-09-14T03:02:13Z" level=info msg="Saving artifact: hello-art"
time="2018-09-14T03:02:13Z" level=info msg="Archiving 52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4:/tmp/hello_world.txt to /argo/outputs/artifacts/hello-art.tgz"
time="2018-09-14T03:02:13Z" level=info msg="Annotations monitor stopped"
time="2018-09-14T03:02:13Z" level=info msg="Alloc=4035 TotalAlloc=11865 Sys=9830 NumGC=4 Goroutines=10"
time="2018-09-14T03:02:13Z" level=fatal msg="container docker://52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4 is terminated: &ContainerStateTerminated{ExitCode:0,Signal:0,Reason:Completed,Message:,StartedAt:2018-09-14 03:02:10 +0000 UTC,FinishedAt:2018-09-14 03:02:10 +0000 UTC,ContainerID:docker://52ef7dd9893d8a2ce85816a0339032a4eb64999f22d913aea5f1985528ca31c4,}"

I confirmed the docker executor works because docker cp can be executed for terminated containers.

What you expected to happen:

CopyFile and GetFileContents of the kubelet executor should work. Am I missing something to use the kubelet executor?

If it doesn't work and Argo doesn't support it, it should fail with an unsupported error message instead of the raw output from kubelet API.

How to reproduce it (as minimally and precisely as possible):

Use the kubelet executor for artifact-passing example.

Anything else we need to know?:

Environment:

• Argo version:

$ argo version
argo: v2.2.0
  BuildDate: 2018-08-30T08:51:40Z
  GitCommit: af636ddd8455660f307d835814d3112b90815dfd
  GitTreeState: clean
  GitTag: v2.2.0
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: 2018-04-26T04:07:23Z
  compiler: gc
  gitCommit: 9e4010d067ee4d799330980e62830b06a07296bd
  gitTreeState: clean
  gitVersion: v0.0.0-master+$Format:%h$
  goVersion: go1.10.1
  major: ""
  minor: ""
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-07-17T18:43:26Z
  compiler: gc
  gitCommit: b1b29978270dc22fecc592ac55d903350454310a
  gitTreeState: clean
  gitVersion: v1.11.1
  goVersion: go1.10.3
  major: "1"
  minor: "11"
  platform: linux/amd64

[dtaniwaki]

@jessesuen I remember we discussed how this feature should be implemented instead of #1022 in offline and you had the idea of utilizing volumeMounts of the main container to automatically mount the volumes in the corresponded wait container. If you're too busy to implement it, I will do it as I need it in my projects ASAP.

[jessesuen]

@dtaniwaki i have the work staged in a branch right now:
https://github.com/jessesuen/argo/commits/pns

The logic to mirror the volume mounts to main container is here:
jessesuen@e00f40a#diff-799231f95f7b5d8d9fe078b2cc4ebc1dR563

What is not implemented yet is to update the wait logic for K8s/kubelet executor and make use of the mirrored volume mounts.

[dtaniwaki]

Cool! I'm looking forward to the feature.

[dtaniwaki]

I made a comment inline in the diff. Please check it.
jessesuen@e00f40a#r31891710

[jessesuen]

This was fixed as part of #1214.

We now have limited support for copying out artifacts from kubelet/k8s executors, as long as the artifact is in a volume (e.g. an emptyDir). All volumes are now mirrored to the wait sidecar, which allows this to happen.