Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support eStargz TOC conversion and run for RAFS v6 #486

Merged
merged 12 commits into from
Jun 22, 2022

Conversation

imeoer
Copy link
Collaborator

@imeoer imeoer commented Jun 12, 2022

No description provided.

@hsiangkao
Copy link
Contributor

hsiangkao commented Jun 12, 2022

Moved as a common branch for better cooperation.
related topic: opencontainers/image-spec#877 and #463

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch 8 times, most recently from a2411ee to 2281c2d Compare June 18, 2022 09:02
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

storage/src/meta/mod.rs Outdated Show resolved Hide resolved
storage/src/meta/mod.rs Outdated Show resolved Hide resolved
storage/src/meta/mod.rs Outdated Show resolved Hide resolved
@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch 4 times, most recently from 253bd20 to 76a5a8e Compare June 21, 2022 03:04
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , Test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , The CI test is completed, please check result:

Test CaseTest Result
merge-target-branch✅SUCCESS
build-docker-image✅SUCCESS
compile-nydus❌FAIL

Sorry, your test job failed. Please get the details in the link.

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from 76a5a8e to d8f0f53 Compare June 21, 2022 03:20
@yqleng1987
Copy link
Contributor

@imeoer , Your pull request has been updated. A new test job has been submitted. Please wait in patience.

@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from a1e1b65 to cf0f18e Compare June 21, 2022 11:44
@yqleng1987
Copy link
Contributor

@imeoer , your pull request has been updated. A new test job will be submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , your test job has passed, and no need to test again.

@imeoer
Copy link
Collaborator Author

imeoer commented Jun 22, 2022

The smoke test passed but docker cache export hang, I think we can move on first. cc @jiangliu

imeoer added 12 commits June 22, 2022 16:18
Add stargz TOC build and run in smoke test for rafs v6.

Signed-off-by: Yan Song <[email protected]>
Set rafs v6's blob meta header for stargz chunks.

Signed-off-by: Yan Song <[email protected]>
1. Remove the chunk size limitation RAFS_DEFAULT_CHUNK_SIZE.
2. Fill stargz chunk's uncompressed_offset to 4k aligned.

Signed-off-by: Yan Song <[email protected]>
Add `--blob-meta` option to allow dump a separate blob meta file
to local disk, so that we can convert stargz TOC to bootstrap and
generate blob meta file for rafs v6 runtime.

Signed-off-by: Yan Song <[email protected]>
Allow loading separate blob meta file from local cache directory
for stargz image, this commit just remove some useless checks.

Signed-off-by: Yan Song <[email protected]>
For uncompressed blob meta, keeping 4k alignment to make compatible
with nydusd runtime, allowing runtime to use the blob meta in blob
cache directory directly.

Signed-off-by: Yan Song <[email protected]>
Merge should inherit the chunk size stored in parent bootstrap,
not the default value.

Signed-off-by: Yan Song <[email protected]>
Convert all TOC to multiple bootstraps for each stargz layers,
and then merge all bootstrap into final bootstrap. This method
has better performance than loading parent bootstrap layer by
layer.

Signed-off-by: Yan Song <[email protected]>
The stargz chunk size is 4MB by default, and we need store
its compressed/uncompressed size into BlobChunkInfoOndisk.

So wen need use reserved 4bits as the high bits of compressed_size
and uncompressed_size.

This change is still compatible with old format, update ut
case to ensure that.

Signed-off-by: Yan Song <[email protected]>
The estargz image format has changed a lot, for a real estargz image,
the TOC file in each layer does not contain root inode, so we need
prepend one before dump bootstrap.

Signed-off-by: Yan Song <[email protected]>
For stargz chunk, it does not provide compressed size, so the
value of `compressed_blob_size` is always 0, we need to check
decompressed_blob_size here.

Signed-off-by: Yan Song <[email protected]>
For rafs v6 in fscache daemon, we must make compatible with
stargz chunks.

Signed-off-by: Yan Song <[email protected]>
@imeoer imeoer force-pushed the support-estargz-in-rafs-v6 branch from cf0f18e to 8b746e9 Compare June 22, 2022 08:21
@yqleng1987
Copy link
Contributor

@imeoer , your pull request has been updated. A new test job will be submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , the test job has been submitted. Please wait in patience.

@yqleng1987
Copy link
Contributor

@imeoer , The CI test is completed, please check result:

Test CaseTest Result
merge-target-branch✅SUCCESS
build-docker-image✅SUCCESS
compile-nydus✅SUCCESS
compile-ctr-remote✅SUCCESS
compile-nydus-snapshotter✅SUCCESS
start-nydus-snapshotter-config-containerd✅SUCCESS
run-container-with-nydus-image✅SUCCESS

Congratulations, your test job passed!

@hsiangkao hsiangkao self-requested a review June 22, 2022 14:48
@jiangliu jiangliu merged commit c839875 into master Jun 22, 2022
@jiangliu jiangliu deleted the support-estargz-in-rafs-v6 branch September 30, 2022 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants