tar command fails inside docker

[olalonde]

First reported here: moby/moby#19647

Command to reproduce:

docker run -it --rm heroku/cedar:14 /bin/bash -c "curl -sS https://install.meteor.com | /bin/sh"

On Docker v1.8.1, aufs storage driver, the command completes successfully:

Downloading Meteor distribution
######################################################################## 100.0%

Meteor 1.2.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com

On Docker v1.8.3, overfayfs storage driver, the command fails:

Downloading Meteor distribution
######################################################################## 100.0%
tar: .meteor/packages/coffeescript/.1.0.11.148kw9n++os+web.browser+web.cordova/plugin.compileCoffeescript.os/npm/babel-compiler/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler: Directory renamed before its status could be extracted
tar: Exiting with failure status due to previous errors
Installation failed.

More logs:

core@deis-03 ~ $ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64
core@deis-03 ~ $ docker info
Containers: 11
Images: 69
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-coreos-r1
Operating System: CoreOS 835.9.0
CPUs: 1
Total Memory: 1.959 GiB
Name: deis-03
ID: SSCW:KJB5:3MI4:WCSV:RXKY:WG2W:HB7U:TRYL:36L4:ZDTC:5KUL:JIMD

Related issues: deis/deis#4867 meteorhacks/meteord#64 meteor/meteor#5762

[mischief]

i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.

[AkihiroSuda]

@mischief I confirmed it still fails in CoreOS 1000.0.0

$ docker info
docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.6-coreos
Operating System: CoreOS 1000.0.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.958 GiB

[mitar]

Interesting. I am getting this error sometimes for Docker hub automatic builds for my Meteor apps in exactly the same step, when install Meteor inside, which uses tar to extract files. But what is interesting is that sometimes it happens and sometimes it does not. Does Docker hub build images on different underlying configurations?

[kenXengineering]

I am seeing the same issue on docker hub with our meteor build image.
https://hub.docker.com/r/onmodulus/build-meteor/builds/bxnluvw33haxrgvhlylfmpv/

[vcaputo]

Judging from the relevant code in tar, the ancestor directory must have undergone a copy_up in the middle of the extraction, resulting in its st_dev/st_ino changing which tar is using as a heuristic for rename detection:

 839       if (check_for_renamed_directories)
 840         {
 841           if (fstatat (chdir_fd, data->file_name, &st, data->atflag) != 0)
 842             {
 843               stat_error (data->file_name);
 844               skip_this_one = 1;
 845             }
 846           else
 847             {
 848               current_mode = st.st_mode;
 849               current_mode_mask = ALL_MODE_BITS;
 850               if (! (st.st_dev == data->dev && st.st_ino == data->ino))
 851                 {
 852                   ERROR ((0, 0,
 853                           _("%s: Directory renamed before its status could be extracted"),
 854                           quotearg_colon (data->file_name)));
 855                   skip_this_one = 1;
 856                 }
 857             }
 858         }

That's unfortunate, and I'm not yet clear on if any recent overlayfs developments have improved on this situation. I also don't see an obvious flag to turn off this heuristic in tar.

An immediate workaround would be to use btrfs instead of ext4+overlayfs, which CoreOS still supports. You can automate the switch to btrfs on first boot using Ignition as well.

Addendum:
Upon further study of the overlayfs implementation, it doesn't seem like the st_dev/st_ino values should be changing spuriously, and I've only managed to reproduce this once out of ~50 attempts in a simplified tar extraction of the meteor release tar on a plain overlayfs mount. I'll keep digging, but it's very likely to be overlayfs-related considering the nature of the failure.

Addendum II:
These inode numbers are being generated on-the-fly by overlayfs:
https://github.com/coreos/linux/blob/v4.6-coreos/fs/inode.c#L848
https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/inode.c#L408
https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/super.c#L559

So the persistence of the inode number is entirely bound to the cache, which explains why this can be frustrating to reproduce.

Here's a simple test one can use to observe the problem tar is complaining about:

overlay # mkdir u l w mnt 
overlay # mount -t overlay overlay -o upperdir=u,lowerdir=l,workdir=w mnt
overlay # mkdir mnt/foo
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743    Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743    Links: 2
overlay # echo 2 > /proc/sys/vm/drop_caches 
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163    Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163    Links: 2

I see no immediate solution to this without significant changes to overlayfs.

[vcaputo]

Definitely another overlayfs issue, for the upstream discussion see http://thread.gmane.org/gmane.linux.file-systems.union/879/focus=879

[tzapu]

we re experiencing this as well here: https://hub.docker.com/r/tzapu/development-meteor/builds/biszzqx9oawewfe8hnrzm4c/

some builds make it through, some fail, even if it s the same thing...

[gsc-dev]

Same happens when using docker cloud to build...
All 4 builds have failed when extracting the tar.

Docker Repo: https://cloud.docker.com/app/monbelle/repository/docker/monbelle/meteor/builds
Github: https://github.com/gsc-dev/docker-meteor

[idmontie]

I'm trying to install meteor in my gitlab ci/cd so that it can bundle and deploy the project. I'm getting the same error when the tar is being extracted. For me it happens every single time.

[Koleok]

@idmontie Same here for me, fails every time, except in my case only when i add the --delete flag which i do in my on_stop for my review apps setup.

[xmjiao]

I was getting the same errors on Mac and on Docker Hub.

A workaround was to use bsdtar instead of tar in the guest OS. I was using Ubuntu 14.04 in the Docker image. You may need to install bsdtar using the command apt-get install -y --no-install-recommends bsdtar and then use bsdtar in place of tar.

[euank]

I think kernel 4.13's OVERLAY_FS_INDEX=y config should fix this. See https://lwn.net/Articles/725276/

[povesteam]

You can install and replace tar with bsdtar in the Dockerfile, right before installing meteor:

RUN apt-get install -y bsdtar && ln -sf $(which bsdtar) $(which tar)
RUN curl "https://install.meteor.com/?release=1.5.2.2" | sh

[Jeyanthinath]

@povesteam your fix works fine, but making other apt-get install to fail :(

[povesteam]

@Jeyanthinath How about putting the original tar back after meteor installation?

# meteor installer doesn't work with the default tar binary
RUN apt-get install -y bsdtar \
    && cp $(which tar) $(which tar)~ \
    && ln -sf $(which bsdtar) $(which tar)

# install Meteor forcing its progress
RUN curl "https://install.meteor.com/?release=1.6.0.1" \
    | sed 's/VERBOSITY="--silent"/VERBOSITY="--progress-bar"/' \
    | sh

# put back the original tar
RUN mv $(which tar)~ $(which tar)

Thanks @xmjiao for the idea!

[kgbook]

apt-get clean before tar, it works for me .