Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

tar command fails inside docker #1095

Open
olalonde opened this issue Jan 26, 2016 · 16 comments
Open

tar command fails inside docker #1095

olalonde opened this issue Jan 26, 2016 · 16 comments

Comments

@olalonde
Copy link

First reported here: moby/moby#19647

Command to reproduce:

docker run -it --rm heroku/cedar:14 /bin/bash -c "curl -sS https://install.meteor.com | /bin/sh"

On Docker v1.8.1, aufs storage driver, the command completes successfully:

Downloading Meteor distribution
######################################################################## 100.0%

Meteor 1.2.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com

On Docker v1.8.3, overfayfs storage driver, the command fails:

Downloading Meteor distribution
######################################################################## 100.0%
tar: .meteor/packages/coffeescript/.1.0.11.148kw9n++os+web.browser+web.cordova/plugin.compileCoffeescript.os/npm/babel-compiler/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler: Directory renamed before its status could be extracted
tar: Exiting with failure status due to previous errors
Installation failed.

More logs:

core@deis-03 ~ $ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64
core@deis-03 ~ $ docker info
Containers: 11
Images: 69
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-coreos-r1
Operating System: CoreOS 835.9.0
CPUs: 1
Total Memory: 1.959 GiB
Name: deis-03
ID: SSCW:KJB5:3MI4:WCSV:RXKY:WG2W:HB7U:TRYL:36L4:ZDTC:5KUL:JIMD

Related issues: deis/deis#4867 meteorhacks/meteord#64 meteor/meteor#5762

@mischief
Copy link

i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.

@AkihiroSuda
Copy link

@mischief I confirmed it still fails in CoreOS 1000.0.0

$ docker info
docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.6-coreos
Operating System: CoreOS 1000.0.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.958 GiB

@mitar
Copy link

mitar commented May 31, 2016

Interesting. I am getting this error sometimes for Docker hub automatic builds for my Meteor apps in exactly the same step, when install Meteor inside, which uses tar to extract files. But what is interesting is that sometimes it happens and sometimes it does not. Does Docker hub build images on different underlying configurations?

@kenXengineering
Copy link

I am seeing the same issue on docker hub with our meteor build image.
https://hub.docker.com/r/onmodulus/build-meteor/builds/bxnluvw33haxrgvhlylfmpv/

@vcaputo
Copy link

vcaputo commented Jul 19, 2016

Judging from the relevant code in tar, the ancestor directory must have undergone a copy_up in the middle of the extraction, resulting in its st_dev/st_ino changing which tar is using as a heuristic for rename detection:

 839       if (check_for_renamed_directories)
 840         {
 841           if (fstatat (chdir_fd, data->file_name, &st, data->atflag) != 0)
 842             {
 843               stat_error (data->file_name);
 844               skip_this_one = 1;
 845             }
 846           else
 847             {
 848               current_mode = st.st_mode;
 849               current_mode_mask = ALL_MODE_BITS;
 850               if (! (st.st_dev == data->dev && st.st_ino == data->ino))
 851                 {
 852                   ERROR ((0, 0,
 853                           _("%s: Directory renamed before its status could be extracted"),
 854                           quotearg_colon (data->file_name)));
 855                   skip_this_one = 1;
 856                 }
 857             }
 858         }

That's unfortunate, and I'm not yet clear on if any recent overlayfs developments have improved on this situation. I also don't see an obvious flag to turn off this heuristic in tar.

An immediate workaround would be to use btrfs instead of ext4+overlayfs, which CoreOS still supports. You can automate the switch to btrfs on first boot using Ignition as well.

Addendum:
Upon further study of the overlayfs implementation, it doesn't seem like the st_dev/st_ino values should be changing spuriously, and I've only managed to reproduce this once out of ~50 attempts in a simplified tar extraction of the meteor release tar on a plain overlayfs mount. I'll keep digging, but it's very likely to be overlayfs-related considering the nature of the failure.

Addendum II:
These inode numbers are being generated on-the-fly by overlayfs:
https://github.com/coreos/linux/blob/v4.6-coreos/fs/inode.c#L848
https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/inode.c#L408
https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/super.c#L559

So the persistence of the inode number is entirely bound to the cache, which explains why this can be frustrating to reproduce.

Here's a simple test one can use to observe the problem tar is complaining about:

overlay # mkdir u l w mnt 
overlay # mount -t overlay overlay -o upperdir=u,lowerdir=l,workdir=w mnt
overlay # mkdir mnt/foo
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743    Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743    Links: 2
overlay # echo 2 > /proc/sys/vm/drop_caches 
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163    Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163    Links: 2

I see no immediate solution to this without significant changes to overlayfs.

@vcaputo vcaputo self-assigned this Jul 20, 2016
@vcaputo
Copy link

vcaputo commented Jul 22, 2016

Definitely another overlayfs issue, for the upstream discussion see http://thread.gmane.org/gmane.linux.file-systems.union/879/focus=879

@tzapu
Copy link

tzapu commented Aug 26, 2016

we re experiencing this as well here: https://hub.docker.com/r/tzapu/development-meteor/builds/biszzqx9oawewfe8hnrzm4c/

some builds make it through, some fail, even if it s the same thing...

@gsc-dev
Copy link

gsc-dev commented Jan 4, 2017

Same happens when using docker cloud to build...
All 4 builds have failed when extracting the tar.

Docker Repo: https://cloud.docker.com/app/monbelle/repository/docker/monbelle/meteor/builds
Github: https://github.com/gsc-dev/docker-meteor

@idmontie
Copy link

I'm trying to install meteor in my gitlab ci/cd so that it can bundle and deploy the project. I'm getting the same error when the tar is being extracted. For me it happens every single time.

@Koleok
Copy link

Koleok commented Mar 3, 2017

@idmontie Same here for me, fails every time, except in my case only when i add the --delete flag which i do in my on_stop for my review apps setup.

@xmjiao
Copy link

xmjiao commented Apr 7, 2017

I was getting the same errors on Mac and on Docker Hub.

A workaround was to use bsdtar instead of tar in the guest OS. I was using Ubuntu 14.04 in the Docker image. You may need to install bsdtar using the command apt-get install -y --no-install-recommends bsdtar and then use bsdtar in place of tar.

rbuckland pushed a commit to rbuckland/diskimage-builder that referenced this issue May 2, 2017
rbuckland pushed a commit to rbuckland/diskimage-builder that referenced this issue May 2, 2017
rbuckland pushed a commit to rbuckland/diskimage-builder that referenced this issue May 2, 2017
shapiromatron added a commit to shapiromatron/tblBuilder that referenced this issue Jul 28, 2017
@euank
Copy link
Contributor

euank commented Jul 28, 2017

I think kernel 4.13's OVERLAY_FS_INDEX=y config should fix this. See https://lwn.net/Articles/725276/

@povesteam
Copy link

You can install and replace tar with bsdtar in the Dockerfile, right before installing meteor:

RUN apt-get install -y bsdtar && ln -sf $(which bsdtar) $(which tar)
RUN curl "https://install.meteor.com/?release=1.5.2.2" | sh

@Jeyanthinath
Copy link

@povesteam your fix works fine, but making other apt-get install to fail :(

@povesteam
Copy link

povesteam commented Dec 10, 2017

@Jeyanthinath How about putting the original tar back after meteor installation?

# meteor installer doesn't work with the default tar binary
RUN apt-get install -y bsdtar \
    && cp $(which tar) $(which tar)~ \
    && ln -sf $(which bsdtar) $(which tar)

# install Meteor forcing its progress
RUN curl "https://install.meteor.com/?release=1.6.0.1" \
    | sed 's/VERBOSITY="--silent"/VERBOSITY="--progress-bar"/' \
    | sh

# put back the original tar
RUN mv $(which tar)~ $(which tar)

Thanks @xmjiao for the idea!

lukemarsden added a commit to dotmesh-io/zfs-builder that referenced this issue Jan 20, 2018
mhutter added a commit to abacusresearch/s2i-phpnode that referenced this issue Mar 12, 2018
According to infos in [1] `tar` does not like the way overlay2 plays
tricks with inodes. As a workaround it is proposed to use `bsdtar` which
handles open files and dirs differently.

[1]: coreos/bugs#1095
DanielRuthardt added a commit to baristi/moka that referenced this issue Aug 2, 2018
Additionally replaced tar with bsdtar, to workaround the following issue: coreos/bugs#1095.
fishilico added a commit to fishilico/shared that referenced this issue Sep 4, 2018
tar is complaining when extracting Arch Linux iso install:

    tar: root.x86_64/etc/xdg/systemd: Directory renamed before its status could be extracted
    tar: root.x86_64/etc/systemd/system/getty.target.wants: Directory renamed before its status could be extracted
    tar: root.x86_64/etc/systemd/user/sockets.target.wants: Directory renamed before its status could be extracted
    tar: root.x86_64/etc/systemd/user: Directory renamed before its status could be extracted
    tar: root.x86_64/etc/ssl/certs: Directory renamed before its status could be extracted
    tar: root.x86_64/var: Directory renamed before its status could be extracted
    tar: Exiting with failure status due to previous errors

This is due to the way tar keeps track of directories and the way
overlayfs presents inodes, according to coreos/bugs#1095
Use bsdtar instead of tar in order to work around this issue.

While at it, download the signing key in the repository in order to no
longer rely on PGP keyservers, which are quite unstable.
Draggha pushed a commit to Draggha/deis-workflow-cli-docker that referenced this issue Oct 1, 2018
Draggha added a commit to Draggha/deis-workflow-cli-docker that referenced this issue Oct 1, 2018
felixbuenemann added a commit to felixbuenemann/slugbuilder that referenced this issue Oct 11, 2018
This should fix problems with gnu tar failing to create directories due
to changing inodes on overlayfs.

See coreos/bugs#1095 for details.
lerwys added a commit to lnls-sirius/debian-rootfs that referenced this issue Nov 1, 2018
This is a known issue with overlayfs discussed here:
coreos/bugs#1095
@kgbook
Copy link

kgbook commented Nov 1, 2019

apt-get clean before tar, it works for me .

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests