Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unsupported entry type: hardlink #101

Closed
maleadt opened this issue Apr 15, 2021 · 4 comments · Fixed by #102
Closed

Unsupported entry type: hardlink #101

maleadt opened this issue Apr 15, 2021 · 4 comments · Fixed by #102

Comments

@maleadt
Copy link

maleadt commented Apr 15, 2021 

julia> using ArtifactUtils

julia> add_artifact!(
                  "Artifacts.toml",
                  "ubuntu",
                  "https://partner-images.canonical.com/core/focal/20210414/ubuntu-focal-core-cloudimg-amd64-root.tar.gz",
                  force=true,
              )
ERROR: unsupported entry type
Tar.Header("usr/bin/bzcat", :hardlink, 0o755, 0, "usr/bin/bunzip2")

I was trying to use a rootfs tarball as an artifact, but it seems to contain a hardlink which results in Tar.jl unable to process it.
@StefanKarpinski
Copy link
Member

Yup, Tar doesn't support hard links. We possibly could, but it's a bit of a significant feature.

@maleadt
Copy link
Author

maleadt commented Apr 16, 2021

I decided to work around it: https://github.com/JuliaCI/PkgEval.jl/blob/0cd947e632f5986cab203b1e596de1e5d804117a/rootfs/debian.sh#L27-L36. I thought it might have been an oversight since there's some traces of it in the code base, e.g.:

Tar.jl/src/header.jl

Lines 93 to 94 in ddca329

hdr.type ∉ (:hardlink, :symlink) && !isempty(hdr.link) &&
err("non-link with link path")

@StefanKarpinski
Copy link
Member

I'm digging into how hard links work in tarballs, which is not entirely obvious. If it's possible to support extracting them without too much trouble, I could add the feature, but I have to figure out how they work first.

@StefanKarpinski
Copy link
Member

So it turns out that this isn't that hard to implement but it's very tricky to make sure that extracting the tarball does't do anything dangerous with some combination of symlinks and hard links.

@StefanKarpinski StefanKarpinski mentioned this issue Apr 17, 2021
Merged
StefanKarpinski added a commit that referenced this issue Apr 17, 2021
@StefanKarpinski
support for hard links: extract, tree_hash, rewrite
691ece7 
This adds support for hard links, including:

- extracting them by copying the linked file (no hard link created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hard links whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hard link that appears before the file that is linked. If the
target of a hard link is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hard links where the link involves symlinks, even if the link refers to
a path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 18, 2021
@StefanKarpinski
support for hard links: extract, tree_hash, rewrite
d7c1233 
This adds support for hard links, including:

- extracting them by copying the linked file (no hard link created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hard links whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hard link that appears before the file that is linked. If the
target of a hard link is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hard links where the link involves symlinks, even if the link refers to
a path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 18, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
a9140a8 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 21, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
4337b47 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
@StefanKarpinski StefanKarpinski mentioned this issue Apr 21, 2021
Merged
StefanKarpinski added a commit that referenced this issue Apr 22, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
b0a0645 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 23, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
4b70bba 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 23, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
f9bbb4e 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 23, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
411e8cf 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
StefanKarpinski added a commit that referenced this issue Apr 23, 2021
@StefanKarpinski
support for hardlinks: extract, tree_hash, rewrite
c4b28fb 
This adds support for hardlinks, including:

- extracting them by copying the linked file (no hardlink created)
- tree hashing them as they are extracted
- rewriting by duplicating the linked file

This only supports hardlinks whose target is a plain file that has
already been seen in the tarball that is being processed. You cannot
have a hardlink that appears before the file that is linked. If the
target of a hardlink is overwritten later, the link copies the current
version of the file at the time of extraction. Tree hashing and rewrite
are both consistent with this behavior. It is not supported to extract
hardlinks where the link involves symlinks, even if the link refers to a
path that would be a file — the target must be a plain file.

Close #101.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support for extracting hard links JuliaIO/Tar.jl
2 participants
@StefanKarpinski @maleadt