Add support for containers/storage #63

nalind · 2016-08-31T14:27:56Z

This patch adds containers/storage as a backend type called 'oci-storage', using a storage.Mall initialized with the library's default settings, or using an externally-initialized storage.Mall.

An image's blobs are stored either as oci-storage layers (if they look like filesystem layers) or (alongside the manifest and signatures) as named big data items associated with the image.

Record-keeping (so that we can remember which blobs were layers and which weren't) are encoded as a JSON object which is stored as the oci-storage image's metadata.

The type of compression which we detected around a layer blob when it was imported is encoded as a field in a JSON object which is stored as the oci-storage layer object's metadata, and is reapplied if we need to export the layer later.

If we find ourselves importing an image with the same ID as an image that we already have, we return an error. If we find ourselves importing an image which contains layers which we already have, we flag
an error if its contents differ from the one we already have, otherwise we accept it.

If we find ourselves importing an image that wants to be tagged with a name that is already in use, the name is assigned to the new image and the old image remains otherwise unmodified.

mtrmac · 2016-08-31T17:25:26Z

storage/storage_image.go

+	if (n > 0) && archive.IsArchive(header[:n]) {
+		// It's a filesystem layer.  If it's not the first one in the
+		// image, we assume that the most recently added layer is its
+		// parent.


This feels a bit like the ImageDestination interface should be better typed / more specific about the content being transferred; knowing about configs / layers / layer orders

Right, the order matters for filesystem blobs when this library is being used to store them, but I don't think the rest of the transport types care, since I'm pretty sure they don't need to look at the contents. Still, it's probably worth clarifying that we depend on the ordering being right in types.go.

Sadly, looking at the current genericImage.BlobDigests, the order is not currently preserved on input. That can change of course, and I think docker-daemon: will need this to change as well.

For these kinds of API changes, it would be very convenient to have both the definition of the interfaces, the transports implementing the interfaces, and the core of skopeo copy, the primary user, in a single repo — then we could change the ImageSource/ImageDestination interfaces without breaking anything else and without needing to keep a skopeo branch+PR in sync.

#56 + containers/skopeo#175 does that, so if this PR ended up changing the types, perhaps it would be practical to base it on that?

(Warning: #56 is being rebased as other PRs are merged; but I don't expect it to substantially change.)

@runcom Note that #56 currently depends on #52 , primarily for the types.SystemContext API changes; I could split the API changes from #52 so that #52 is only the lookaside implementation, which would allow merging the API changes + copy.go before reviewing the lookaside—if that helped you in any way.

@runcom Note that #56 currently depends on #52 , primarily for the types.SystemContext API changes; I could split the API changes from #52 so that #52 is only the lookaside implementation, which would allow merging the API changes + copy.go before reviewing the lookaside—if that helped you in any way.

that would be awesome and I can go ahead and review #56 and maybe getting it before this (#63) so Nalin's work can benefit from it.

wdyt?

@runcom OK, #64 contains the API changes , and #56 re-merged to not include the lookaside. skopeo PRs update will follow.

@nalind the API churn in #64 will affect this I'm afraid, but it should be easy to adapt.

mtrmac · 2016-08-31T17:41:49Z

Thanks!

For now just a few comments from a quick skim, I haven’t read all of the code.

Structurally it is a bit surprising to have both ImageSource and ImageDestination as the same object, though it doesn’t hurt anything at all.

runcom · 2016-08-31T17:48:33Z

@nalind tests failures seems related (missing or wrong vendoring of containers/storage?)

mtrmac · 2016-08-31T17:56:00Z

@nalind tests failures seems related (missing or wrong vendoring of containers/storage?)

Yeah, containers/image does not do any vendoring, but make test-skopeo users (only) the libraries already vendored in skopeo. containers/skopeo#179 could help with that

Calls to ParseReference() eventually end up calling GetStore(), and GetStore() needs more information than a reference can tell it, so a few test cases that appear to be valid at first glance still fail to parse. Add some more detail on why that happens: in order to match a valid store, that store needs to have been previously opened by the calling application, but since the store is in a temporary directory created just for the test, and we aren't opening that location with GetStore() first, it's expected to fail. Signed-off-by: Nalin Dahyabhai <[email protected]>

Convert the helper functions which copy.Image() used, often passing in several arguments, into methods of a helper object, which can remember most of them. Signed-off-by: Nalin Dahyabhai <[email protected]>

Fix stylistic nits from PR review of containers#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

When HasBlob() returns false, always return a non-nil error. Signed-off-by: Nalin Dahyabhai <[email protected]>

Return more context in error values instead of merely logging them and returning a more generic error. Signed-off-by: Nalin Dahyabhai <[email protected]>

Don't use the name of a builtin ("close") as a variable name. Signed-off-by: Nalin Dahyabhai <[email protected]>

Vendor a copy of containers/storage, the proposed version of containers/image that knows how to use it, and new dependencies that they pull in. Signed-off-by: Nalin Dahyabhai <[email protected]>

runcom · 2016-12-15T10:29:27Z

@nalind make lint is failing https://travis-ci.org/containers/image/builds/184081374#L349

I believe we're pretty good to go after you fix that

runcom · 2016-12-15T14:54:15Z

@nalind just have the last question: this is adding a transport just for containers/storage but would it be possible to still use a specific transport and just stream/pass blobs to contianers/storage? I mean, can contianers/storage and image be used fully decoupled?

like:

streamOrPathToBlob = src.GetBlob
storage.AddLyaer(streamOrPathToBlob)

Signed-off-by: Nalin Dahyabhai <[email protected]>

nalind · 2016-12-15T16:46:04Z

Re-pushed without the Makefile changes.

mtrmac · 2017-01-03T18:24:37Z

image/memory.go

+	if err != nil {
+		return -1, err
+	}
+	return int64(len(s)), nil


This is only the length of the manifest, which does not really make sense. Considering that memoryImage is only returned by types.Image.UpdatedImage, it should say something semantically consistent with sourcedImage.Size.

Doh. Opened #199 to just return -1.

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

nalind force-pushed the add-storage-transport branch 5 times, most recently from 2ae8dd9 to 3cbb921 Compare August 31, 2016 15:32

mtrmac reviewed Aug 31, 2016
View reviewed changes

mtrmac mentioned this pull request Aug 31, 2016

Vendor containers/storage containers/skopeo#179

Merged

nalind force-pushed the add-storage-transport branch 15 times, most recently from 1295c5b to c111f3e Compare September 8, 2016 14:17

nalind added 6 commits December 14, 2016 17:37

Make copy.Image() helpers into an object

0051fdf

Convert the helper functions which copy.Image() used, often passing in several arguments, into methods of a helper object, which can remember most of them. Signed-off-by: Nalin Dahyabhai <[email protected]>

Fix stylistic nits from PR review of image#63

5de46ff

Fix stylistic nits from PR review of containers#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

Return ErrBlobNotFound errors with HasBlob() false

453de84

When HasBlob() returns false, always return a non-nil error. Signed-off-by: Nalin Dahyabhai <[email protected]>

storage: return more context in errors

271db42

Return more context in error values instead of merely logging them and returning a more generic error. Signed-off-by: Nalin Dahyabhai <[email protected]>

Avoid using builtin "close" as a variable name

8e0f748

Don't use the name of a builtin ("close") as a variable name. Signed-off-by: Nalin Dahyabhai <[email protected]>

nalind force-pushed the add-storage-transport branch from 706129b to 0616be9 Compare December 14, 2016 22:37

Fix an unreachable-code validation error

fa91766

Signed-off-by: Nalin Dahyabhai <[email protected]>

nalind force-pushed the add-storage-transport branch from 9f0ba20 to fa91766 Compare December 15, 2016 16:45

runcom merged commit d14bd91 into containers:master Dec 15, 2016

mtrmac reviewed Jan 3, 2017

View reviewed changes

mtrmac mentioned this pull request Jan 3, 2017

Restore in-memory check for duplicate layers #196

Closed

nalind mentioned this pull request Jan 6, 2017

Don't fake a bad answer from memoryImage.Size() #199

Merged

This was referenced Jan 7, 2017

Avoid duplication in dockerImageDestionation.{Has,Put}Blob #202

Merged

RFC WIP: Define the three return states of HaveBlob as true/false/error #204

Merged

mtrmac mentioned this pull request May 9, 2017

storage images IDs leak and performance cri-o/cri-o#500

Closed

Jamesjaj2dc6j added a commit to Jamesjaj2dc6j/lucascalsilvaa that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

e743b92

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

mrhyperbit23z0d added a commit to mrhyperbit23z0d/thomasdarimont that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

f2120bc

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

easyriderk0c9g added a commit to easyriderk0c9g/rnin9 that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

5d02a5d

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

oguzturker8ijm8l added a commit to oguzturker8ijm8l/diwir that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

adc948d

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

wesnowm added a commit to wesnowm/MatfOOP- that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

b4a8863

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

gaveen3 added a commit to gaveen3/KimJeongHoon3x that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

d540535

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

gaveen3 added a commit to gaveen3/KimJeongHoon3x that referenced this pull request Jun 6, 2022

Fix stylistic nits from PR review of image#63

3f0d77c

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

Manuel-Suarez-Abascal80n9y added a commit to Manuel-Suarez-Abascal80n9y/knimeu that referenced this pull request Oct 7, 2022

Fix stylistic nits from PR review of image#63

1135d9e

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

straiforos8bsh5n added a commit to straiforos8bsh5n/tokingsq that referenced this pull request Oct 7, 2022

Fix stylistic nits from PR review of image#63

7790868

Fix stylistic nits from PR review of containers/image#63 Signed-off-by: Nalin Dahyabhai <[email protected]>

Add support for containers/storage #63

Add support for containers/storage #63

Uh oh!

Conversation

nalind commented Aug 31, 2016

Uh oh!

mtrmac Aug 31, 2016

Choose a reason for hiding this comment

Uh oh!

nalind Aug 31, 2016

Choose a reason for hiding this comment

Uh oh!

mtrmac Aug 31, 2016

Choose a reason for hiding this comment

Uh oh!

runcom Aug 31, 2016

Choose a reason for hiding this comment

Uh oh!

mtrmac Aug 31, 2016

Choose a reason for hiding this comment

Uh oh!

mtrmac commented Aug 31, 2016

Uh oh!

runcom commented Aug 31, 2016

Uh oh!

mtrmac commented Aug 31, 2016

Uh oh!

runcom commented Dec 15, 2016

Uh oh!

runcom commented Dec 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nalind commented Dec 15, 2016

Uh oh!

mtrmac Jan 3, 2017

Choose a reason for hiding this comment

Uh oh!

nalind Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

runcom commented Dec 15, 2016 •

edited

Loading