"kubectl cp" on large file causes crash: eviction manager: attempting to reclaim imagefs

[michalzubkowicz]

BUG REPORT:

Please provide the following details:

Environment:

minikube version: v0.30.0

OS:
PRETTY_NAME="Deepin 15"
NAME="Deepin"
VERSION_ID="15.7"
VERSION="15.7"
ID=deepin
HOME_URL="https://www.deepin.org/"
BUG_REPORT_URL="http://feedback.deepin.org/feedback/"

VM driver:
"DriverName": "virtualbox",

ISO version
"Boot2DockerURL": "file:///home/q/.minikube/cache/iso/minikube-v0.30.0.iso",

What happened:
I've tried to copy large file (20GB) into pod, then minikube crashes and I cannot do any action by kubectl, only delete and create of minikube helpes.
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port? I think it's something to PVC, because even importing file into database directly does the same.

What you expected to happen:
File should be copied

How to reproduce it (as minimally and precisely as possible):
Create for example cluster following these instructions:
https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/#deploy-mysql
Change size of volume claim

Try to copy file into cluster:
kubectl cp ./BIGFILE.sql mysql-0:/var/lib/mysql/restore.sql

Output of minikube logs (if applicable):
See attached file
logs.txt.gz

[tstromberg]

This sounds similar to kubernetes-retired/kube-aws#1300

Nov 21 08:18:06 minikube kubelet[2762]: W1121 08:18:06.490471    2762 eviction_manager.go:343] eviction manager: attempting to reclaim imagefs
Nov 21 08:18:06 minikube kubelet[2762]: I1121 08:18:06.490530    2762 container_gc.go:85] attempting to delete unused containers
Nov 21 08:18:08 minikube kubelet[2762]: I1121 08:18:08.847093    2762 image_gc_manager.go:317] attempting to delete unused images
Nov 21 08:18:09 minikube kubelet[2762]: I1121 08:18:09.182453    2762 eviction_manager.go:357] eviction manager: "
Nov 21 08:18:09 minikube kubelet[2762]: I1121 08:18:09.235634    2762 eviction_manager.go:375] eviction manager: pods ranked for eviction: kube-addon-manager-minikube_kube-system(8b52f08746ac78d32737b5f7fdffec52), kube-apiserver-minikube_kube-system(220ed20ff22681b3d8cce28ac46419da), mysql-0_default(fbfa11e3-ed63-11e8-aabb-080027284fad), kube-dns-86f4d74b45-rzxfh_kube-system(912640cf-ed63-11e8-aabb-080027284fad), etcd-minikube_kube-system(1e5d7926d38fe76a281b0f09980a6d55), kube-controller-manager-minikube_kube-system(6f992fb03cfe41a50b544d374d1ec649), mysql-684b8d4cfc-k5t7w_default(fb069f43-ed63-11e8-aabb-080027284fad), kube-proxy-6pm42_kube-system(90f6fe92-ed63-11e8-aabb-080027284fad), kube-scheduler-minikube_kube-system(31cf0ccbee286239d451edb6fb511513), coredns-c4cffd6dc-nc626_kube-system(929676d7-ed63-11e8-aabb-080027284fad), kubernetes-dashboard-6f4cfc5d87-5n75j_kube-system(9205e221-ed63-11e8-aabb-080027284fad), storage-provisioner_kube-system(92aebe29-ed63-11e8-aabb-080027284fad)
Nov 21 08:18:15 minikube kubelet[2762]: E1121 08:18:15.526533    2762 kuberuntime_container.go:65] Can't make a ref to pod "kube-addon-manager

I wonder if the VM is just running low on disk space? What is your minikube start command? if it's disk size related, you can try going with a much larger --disk-size: the default is 20G.

[michalzubkowicz]

@tstromberg
Yes. It's probably caused by low disk space I was copying 25GB file inside

[tstromberg]

Not much we can do here other than make it easier to debug. Opened #3574 to help with the latter.