Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'podman system df' is slow #7406

Closed
Nuc1eoN opened this issue Aug 22, 2020 · 8 comments · Fixed by #7604
Closed

'podman system df' is slow #7406

Nuc1eoN opened this issue Aug 22, 2020 · 8 comments · Fixed by #7604
Assignees
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@Nuc1eoN
Copy link

Nuc1eoN commented Aug 22, 2020

podman system df in rootless mode hangs indefinitely for me.

Can anybody reproduce?

@rhatdan
Copy link
Member

rhatdan commented Aug 24, 2020

It works for me.
Does podman ps -a and podman images take a long time?

@Nuc1eoN
Copy link
Author

Nuc1eoN commented Aug 26, 2020

podman images is instant for me.

Also a little update. I let podman system df run for a while and it turns out it does actually finish but the process takes ~3 minutes.

@paravz
Copy link

paravz commented Aug 27, 2020

journalctl -l --all --no-pager -f while sudo podman system df is running shows that all (most?) image layers is being mounted during this process, takes 5 minutes on my system, output is funky to say the least (containers were created using podman 1.5 or 1.6, now running F32's podman 2.0.4):

$ time sudo podman system df -v 2>&1 | tee -a podman-2.0.4-system-df.log

Images space usage:

REPOSITORY                         TAG       IMAGE ID      CREATED    SIZE     SHARED SIZE  UNIQUE SIZE  CONTAINERS
<none>                             <none>    289289d1a15b  14 months  326.2MB  0B           326.2MB      1
localhost/test                     d0a6310   bf62eb48712c  14 months  3.771GB  0B           3.771GB      4
<none>                             <none>    e06a00c9c1d7  13 months  326.2MB  326.2MB      -3333B       1
<none>                             <none>    b49d797fed63  13 months  326.2MB  326.2MB      -1.381e+04B  1
<none>                             <none>    72979d8140b0  13 months  326.2MB  326.2MB      -2.712e+04B  1
<none>                             <none>    91c62fb55d3a  13 months  332.6MB  338.9MB      -6.368e+06B  1
<none>                             <none>    d1ac9ff6e033  13 months  1.357GB  2.381GB      -1.024e+09B  1
<none>                             <none>    0c1e23b2d31c  13 months  1.357GB  1.357GB      -3.212e+05B  1
<none>                             <none>    7433b67d788e  13 months  1.364GB  1.371GB      -6.777e+06B  1
<none>                             <none>    fba2cc484d33  13 months  1.399GB  1.435GB      -3.539e+07B  1
<none>                             <none>    81f74ebe319e  13 months  1.399GB  1.399GB      -1.124e+04B  1
<none>                             <none>    e4c8edf468a6  13 months  1.399GB  1.399GB      -4.451e+04B  1
<none>                             <none>    6d03e3f2fc06  13 months  1.399GB  1.399GB      -4.451e+04B  1
<none>                             <none>    e5c52a1880a5  13 months  326.2MB  326.2MB      -3333B       0
<none>                             <none>    2b8114581ee4  13 months  326.2MB  326.2MB      -1.381e+04B  0
<none>                             <none>    05f71c6eeb8e  13 months  326.2MB  326.2MB      -2.712e+04B  0
<none>                             <none>    d78678de3e20  13 months  332.6MB  338.9MB      -6.368e+06B  0
<none>                             <none>    490baf730067  13 months  1.357GB  2.381GB      -1.024e+09B  0
<none>                             <none>    3fa5d6820708  13 months  1.357GB  1.357GB      -3.212e+05B  0
<none>                             <none>    f0e5ee5846e0  13 months  1.364GB  1.371GB      -6.777e+06B  0
<none>                             <none>    ee798016f66b  13 months  1.399GB  1.434GB      -3.539e+07B  0
<none>                             <none>    ae3ca2acb94e  13 months  1.399GB  1.399GB      -1.124e+04B  0
<none>                             <none>    4a05cef439a9  13 months  1.399GB  1.399GB      -4.451e+04B  0
<none>                             <none>    7f93e174adbe  13 months  1.399GB  1.399GB      -4.451e+04B  0
<none>                             <none>    f30c316ff16b  13 months  3.664GB  5.928GB      -2.264e+09B  0
<none>                             <none>    604f1dbf9b24  13 months  3.664GB  3.664GB      -1.747e+04B  0
<trimmed>

Note pid 1 and systemd --user's pid in log (trimmed):

systemd[1]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
kernel: overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
systemd[1]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-cdfd8542e233f64902ae0e217d3c024ed07ea879ff8a42dd17dfd5760688fdbe-merged.mount: Succeeded.
systemd[1463]: var-lib-containers-storage-overlay-1a3669dbba07ed4bb912fffbafcf777bfe85af54616cf1ea382e06349db1ba92-merged.mount: Succeeded.
systemd[1]: var-lib-containers-storage-overlay-1a3669dbba07ed4bb912fffbafcf777bfe85af54616cf1ea382e06349db1ba92-merged.mount: Succeeded.
systemd[1]: var-lib-containers-storage-overlay-98acca75f0b46e527d6ffaaf547e3f9b8743c70ff84daa7f0001128821158741-merged.mount: Succeeded.

Edit: also related to #7405

@rhatdan
Copy link
Member

rhatdan commented Sep 10, 2020

@vrothberg PTAL

@rhatdan
Copy link
Member

rhatdan commented Sep 10, 2020

Seems similar to the issues you took care of with images.

@vrothberg
Copy link
Member

I'll take a look, sure. A friendly reminder to use our issue template as it's invaluable to have data about the host and its environment.

@vrothberg vrothberg added the bug label Sep 11, 2020
@vrothberg vrothberg changed the title 'podman system df' hangs in rootless mode 'podman system df' is slow Sep 11, 2020
@vrothberg
Copy link
Member

Eeek ... looking deeper, the image sizes reported by df --verbose are wrong. I'll tackle that as well.

vrothberg added a commit to vrothberg/libpod that referenced this issue Sep 11, 2020
Fix the image-size calculations of system-df, where the shared size is
the actual shared size with other images (including children) and the
(total) size is the sum of the shared and unique size [1].

To calculate parent/child relations, make use of the recently added
layer tree which allows for quick (and cached!) calculations.

Break calculating image disk usages into the image runtime to a) access
the layer tree, and b) make the code easier to maintain and extend.

[1] https://docs.docker.com/engine/reference/commandline/system_df/

Fixes: containers#7406
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg
Copy link
Member

#7604 to the rescue

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants