Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wip: fix mounting /var/lib/contaired to tmpfs in VM ISO #7834

Closed
wants to merge 2 commits into from
Closed

wip: fix mounting /var/lib/contaired to tmpfs in VM ISO #7834

wants to merge 2 commits into from

Conversation

medyagh
Copy link
Member

@medyagh medyagh commented Apr 21, 2020 •
edited
Loading

while debugging preload on containerd, it said there is no diskpace left.
@tstromberg found out that /var/lib/contained is living on tmpfs

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           5.2G  496M  4.7G  10% /
$

Before this PR:

$  grep /var /proc/mounts | egrep -v "^(overlay|shm)"
/dev/vda1 /var/lib/boot2docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/containers ext4 rw,relatime 0 0
/dev/vda1 /var/log ext4 rw,relatime 0 0
/dev/vda1 /var/lib/kubelet ext4 rw,relatime 0 0
/dev/vda1 /var/lib/cni ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minikube ext4 rw,relatime 0 0
/dev/vda1 /var/lib/toolbox ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minishift ext4 rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/6a9c4fa6-7027-48db-8f80-1ecb3c6e568b/volumes/kubernetes.io~secret/storage-provisioner-token-th5bp tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/6a9c4fa6-7027-48db-8f80-1ecb3c6e568b/volumes/kubernetes.io~secret/storage-provisioner-token-th5bp tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/615c7df5-e19c-440c-8688-000febf34ade/volumes/kubernetes.io~secret/kube-proxy-token-gscfv tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/615c7df5-e19c-440c-8688-000febf34ade/volumes/kubernetes.io~secret/kube-proxy-token-gscfv tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/1bc7a49f-4420-4e8e-a9d1-f2d100558136/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/1bc7a49f-4420-4e8e-a9d1-f2d100558136/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/b96c5a5a-5e2c-4373-a481-2a4a571a936e/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/b96c5a5a-5e2c-4373-a481-2a4a571a936e/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 21, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 21, 2020
@tstromberg
Copy link
Contributor

tstromberg commented Apr 21, 2020 •
edited
Loading

Can you comment with the output of:

$ grep /var /proc/mounts | egrep -v "^(overlay|shm)"

after making this change?

@tstromberg
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Apr 21, 2020
@minikube-pr-bot
Copy link

kvm2 Driver
docker Driver

@minikube-pr-bot
Copy link

kvm2 Driver
docker Driver

@medyagh
Copy link
Member Author

medyagh commented Apr 22, 2020

first attemp failed error for logs:

/usr/bin/make  subdir=localedata -C localedata ..=../ subdir_lib
make[5]: Entering directory '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/localedata'
.././scripts/mkinstalldirs /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
.././scripts/mkinstalldirs /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
.././scripts/mkinstalldirs /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
mkdir -p -- /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
mkdir -p -- /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
mkdir -p -- /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata
rm -f /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.o; > /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.o
rm -f /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.os; > /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.os
rm -f /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.oS; > /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.oS
/bin/sh: 1: cannot create /mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.os: File exists
make[5]: *** [/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.os] Error 2
../Makerules:924: recipe for target '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build/localedata/stamp.os' failed
make[5]: Leaving directory '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/localedata'
Makefile:258: recipe for target 'localedata/subdir_lib' failed
make[4]: *** [localedata/subdir_lib] Error 2
make[4]: Leaving directory '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f'
Makefile:9: recipe for target 'all' failed
make[3]: *** [all] Error 2
make[3]: Leaving directory '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/build'
package/pkg-generic.mk:252: recipe for target '/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/.stamp_built' failed
make[2]: *** [/mnt/out/buildroot/output/build/glibc-glibc-2.28-133-g6578d89c170cc7b524b9bccafffd5b4207bf646f/.stamp_built] Error 2
make[2]: Leaving directory '/mnt/out/buildroot'
Makefile:202: recipe for target 'minikube_iso' failed
make[1]: *** [minikube_iso] Error 2
make[1]: Leaving directory '/mnt'
Makefile:226: recipe for target 'out/minikube.iso' failed
make: *** [out/minikube.iso] Error 2
make: *** [out/minikube.iso] Error 2

@medyagh medyagh changed the title fix mounting /var/lib/contaired to tmpfs in VM ISO wip: fix mounting /var/lib/contaired to tmpfs in VM ISO Apr 22, 2020
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 22, 2020
@afbjorklund
Copy link
Collaborator

afbjorklund commented Apr 22, 2020 •
edited
Loading

This commit 270756e was much easier to review than this one...

This one does two unrelated things (/var and /var/run)
And seems to now have turned into a duplicate of #7843 ?

Can't we just do a simple fix for containerd, like the title says ?

     mkdir -p /mnt/$PARTNAME/var/lib/docker
     mkdir -p /var/lib/docker
     mount --bind /mnt/$PARTNAME/var/lib/docker /var/lib/docker
 
     mkdir -p /mnt/$PARTNAME/var/lib/containers
     mkdir -p /var/lib/containers
     mount --bind /mnt/$PARTNAME/var/lib/containers /var/lib/containers
 
+    mkdir -p /mnt/$PARTNAME/var/lib/containerd
+    mkdir -p /var/lib/containerd
+    mount --bind /mnt/$PARTNAME/var/lib/containerd /var/lib/containerd
+

@medyagh medyagh closed this Apr 22, 2020
@medyagh medyagh deleted the var_lib_mount branch May 2, 2020 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tstromberg tstromberg tstromberg requested changes

@afbjorklund afbjorklund Awaiting requested review from afbjorklund

@josedonizetti josedonizetti Awaiting requested review from josedonizetti

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@medyagh @k8s-ci-robot @tstromberg @minikube-pr-bot @afbjorklund