conformance: initial commit for references test #375

rchincha · 2023-01-19T18:10:46Z

Based off of 01_pull_test.go

Fixes #370

Signed-off-by: Ramkumar Chinchani [email protected]

rchincha · 2023-01-24T01:25:08Z

Testing against zot ...

      you have skipped this test; if this is an error, check your environment variable settings:
        OCI_TEST_CONTENT_DISCOVERY=0
        OCI_TEST_CONTENT_MANAGEMENT=1
        OCI_REFERRERS=1
        OCI_TEST_PULL=0
        OCI_TEST_PUSH=0


      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:408
------------------------------
•••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 23 of 78 Specs in 0.085 seconds
SUCCESS! -- 23 Passed | 0 Failed | 0 Pending | 55 Skipped

jdolitsky · 2023-04-19T12:58:00Z

can you rebase?

rchincha · 2023-05-26T20:21:29Z

@jdolitsky @sudo-bmitch @sajayantony
Some eyes on this pls.

FYI, top of zot/main passes.

sudo-bmitch

@rchincha thanks for getting this started! I've got a bunch of feedback, but we're on the right path with this.

conformance/05_referrers.go

sudo-bmitch · 2023-05-28T13:26:18Z

conformance/05_referrers.go

+				// also check resp header "OCI-Filters-Applied: artifactType" denoting that an artifactType filter was applied
+				Expect(resp.Header().Get("OCI-Filters-Applied")).ToNot(BeEmpty())
+				Expect(resp.Header().Get("OCI-Filters-Applied")).To(Equal("application/vnd.oci.conformance"))


Filter support is a SHOULD, so at most we can warn. I'd prefer we push a third referrer with a different type, and then conditionally handle the response based on if the registry returns the OCI-Filters-Applied header. If it's missing, all 3 referrers should be returned. Otherwise if the header is defined with the requested value, we should verify the two requested referrers were returned and not the 3rd.

WARN: you have failed this test, but ignoring Expected <string>: application/vnd.nhl.peanut.butter.bagel to equal <string>:

^ warn support added

Modified implementation.

conformance/05_referrers.go

sudo-bmitch · 2023-05-28T13:34:49Z

conformance/setup.go

+	scratchDescriptor                 = Descriptor{
+		MediaType: "application/vnd.oci.scratch.v1+json",
+		Size:      2,
+		Digest:    godigest.FromString("sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"),
+	}


Pending the result of opencontainers/image-spec#1068

Used same empty JSON descriptor

conformance/05_referrers.go

sudo-bmitch · 2023-06-02T15:30:23Z

conformance/05_referrers.go

+				Expect(len(index.Manifests)).To(Equal(2))
+
+				// also check resp header "OCI-Filters-Applied: artifactType" denoting that an artifactType filter was applied
+				Expect(resp.Header().Get("OCI-Filters-Applied")).To(EqualOrWarn(testRefArtifactTypeA))


Since filter support is a "SHOULD", the length and header check here needs to be conditional. If the header is set, then we expect 2 entries and the header must match the expected value. Else, we'd expect 4 manifests.

Maybe we don't need the "warn" after all.

sudo-bmitch

Couple minor updates. Does Zot still pass after the latest changes?

sudo-bmitch · 2023-06-04T14:22:19Z

conformance/image.go

+// Artifact describes an artifact manifest.
+// This structure provides `application/vnd.oci.artifact.manifest.v1+json` mediatype when marshalled to JSON.
+type Artifact struct {
+	// MediaType is the media type of the object this schema refers to.
+	MediaType string `json:"mediaType"`
+
+	// ArtifactType is the IANA media type of the artifact this schema refers to.
+	ArtifactType string `json:"artifactType,omitempty"`
+
+	// Blobs is a collection of blobs referenced by this manifest.
+	Blobs []Descriptor `json:"blobs,omitempty"`
+
+	// Subject (reference) is an optional link from the artifact to another manifest forming an association between the artifact and the other manifest.
+	Subject *Descriptor `json:"subject,omitempty"`
+
+	// Annotations contains arbitrary metadata for the artifact manifest.
+	Annotations map[string]string `json:"annotations,omitempty"`
+}


conformance/05_referrers.go

rchincha · 2023-06-05T02:46:21Z

Couple minor updates. Does Zot still pass after the latest changes?

Yes, top of zot/main does.

      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:491
------------------------------
••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.152 seconds
SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped

rchincha · 2023-06-09T19:36:17Z

bin/zot-linux-amd64 serve examples/config-conformance.json
^ to test

Made the warning code/output a little "nicer"

sudo-bmitch · 2023-06-15T17:15:33Z

conformance/setup.go

+func Warn(message string) {
+	fmt.Fprintf(colorable.NewColorableStderr(), "\n")
+	// print file:line
+	_, file, line, _ := runtime.Caller(1)
+	warncolor := color.New(color.FgMagenta).FprintfFunc()
+	warncolor(colorable.NewColorableStderr(), formatter.Fi(2, "\nWARNING: "+message))
+	fmt.Fprintf(colorable.NewColorableStderr(), "\n")
+	// print message
+	msgcolor := color.New(color.FgWhite).FprintfFunc()
+	msgcolor(colorable.NewColorableStderr(), formatter.Fi(2, "\n"))
+	msgcolor(colorable.NewColorableStderr(), formatter.Fi(2, "%s:%d\n", file, line))
+}


@jdolitsky is there any issue generating warnings in the conformance test this way?

Rebased and updated after recent merges.

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api Based off of 01_pull_test.go Fixes opencontainers#370 https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage Signed-off-by: Ramkumar Chinchani <[email protected]>

rchincha · 2023-06-21T23:24:28Z

As a data point, top of zot/main passes

••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.255 seconds
SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped

rchincha · 2023-06-21T23:25:54Z

Instrumented zot warns like so,


      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:490
------------------------------
••••••••••••••••••••••••••••
    WARNING: filtering by artifact-type is not implemented                                                                                                                                                                                                                                                                                              /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/05_referrers.go:240

rchincha · 2023-06-21T23:34:15Z

opencontainers/image-spec#1077
^ tests for this in a separate PR after it lands

outdated

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#enabling-the-referrers-api Based off of 01_pull_test.go Fixes opencontainers#370 https://github.com/opencontainers/image-spec/blob/main/manifest.md#guidelines-for-artifact-usage Signed-off-by: Ramkumar Chinchani <[email protected]>

rchincha force-pushed the references branch 2 times, most recently from e28419c to eb5ccad Compare January 24, 2023 00:59

jdolitsky previously approved these changes Jan 24, 2023

View reviewed changes

sudo-bmitch added this to the v1.1.0 milestone Mar 14, 2023

rchincha dismissed jdolitsky’s stale review via d526d56 May 26, 2023 19:59

rchincha force-pushed the references branch from eb5ccad to d526d56 Compare May 26, 2023 19:59

rchincha marked this pull request as ready for review May 26, 2023 20:00

rchincha requested review from dmcgowan, jzelinskie, jonjohnsonjr, mikebrow, stevvooe, sudo-bmitch and vbatts as code owners May 26, 2023 20:00

rchincha force-pushed the references branch 2 times, most recently from 180974d to 18ab2cb Compare May 26, 2023 20:17

rchincha requested a review from jdolitsky May 26, 2023 20:20

rchincha force-pushed the references branch from 18ab2cb to 151f848 Compare May 26, 2023 20:24

sudo-bmitch requested changes May 28, 2023

View reviewed changes

sudo-bmitch reviewed May 29, 2023

View reviewed changes

conformance/05_referrers.go Outdated Show resolved Hide resolved

rchincha force-pushed the references branch 3 times, most recently from 4683bf1 to 2e3cfc1 Compare June 2, 2023 05:13

rchincha requested a review from sudo-bmitch June 2, 2023 05:15

sudo-bmitch reviewed Jun 2, 2023

View reviewed changes

rchincha force-pushed the references branch 2 times, most recently from 9769a96 to 627be76 Compare June 2, 2023 23:44

rchincha force-pushed the references branch from 627be76 to 9b233d5 Compare June 2, 2023 23:48

rchincha requested a review from sudo-bmitch June 2, 2023 23:56

rchincha force-pushed the references branch from 9b233d5 to da259b1 Compare June 3, 2023 17:04

sudo-bmitch previously requested changes Jun 4, 2023

View reviewed changes

rchincha force-pushed the references branch from da259b1 to af46363 Compare June 5, 2023 02:45

rchincha requested a review from sudo-bmitch June 5, 2023 06:00

rchincha force-pushed the references branch from af46363 to d705dff Compare June 9, 2023 19:35

sudo-bmitch reviewed Jun 15, 2023

View reviewed changes

rchincha force-pushed the references branch from d705dff to e586e62 Compare June 16, 2023 20:14

rchincha mentioned this pull request Jun 17, 2023

conformance: skip check (by default) for zero layers in image manifest #421

Merged

jdolitsky mentioned this pull request Jun 21, 2023

Reference types conformance tests #370

Closed

jdolitsky added the conformance label Jun 21, 2023

rchincha force-pushed the references branch from e586e62 to 68fb0ce Compare June 21, 2023 23:17

rchincha force-pushed the references branch from 68fb0ce to 3af85ef Compare June 21, 2023 23:21

rchincha requested a review from sudo-bmitch June 21, 2023 23:31

jonjohnsonjr approved these changes Jun 22, 2023

View reviewed changes

jdolitsky approved these changes Jun 22, 2023

View reviewed changes

jdolitsky merged commit 99aba31 into opencontainers:main Jun 22, 2023

jdolitsky mentioned this pull request Jun 27, 2023

tag 1.1.0-rc3 #433

Closed

8 tasks

jdolitsky mentioned this pull request Jul 6, 2023

tag 1.1.0-rc3 (part 2) #440

Merged

8 tasks

andaaron mentioned this pull request Jul 11, 2023

conformance: make references tests part of content discovery workflow #430

Merged

sudo-bmitch mentioned this pull request Feb 1, 2024

Release v1.1.0 #507

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conformance: initial commit for references test #375

conformance: initial commit for references test #375

rchincha commented Jan 19, 2023

rchincha commented Jan 24, 2023 •

edited

Loading

jdolitsky commented Apr 19, 2023

rchincha commented May 26, 2023

sudo-bmitch left a comment

sudo-bmitch May 28, 2023

rchincha Jun 1, 2023

rchincha Jun 2, 2023

sudo-bmitch May 28, 2023

rchincha Jun 2, 2023

sudo-bmitch Jun 2, 2023

rchincha Jun 2, 2023

rchincha Jun 2, 2023

sudo-bmitch left a comment

sudo-bmitch Jun 4, 2023

rchincha Jun 5, 2023

rchincha commented Jun 5, 2023 •

edited

Loading

rchincha commented Jun 9, 2023

sudo-bmitch Jun 15, 2023

rchincha Jun 21, 2023 •

edited

Loading

rchincha commented Jun 21, 2023

rchincha commented Jun 21, 2023 •

edited

Loading

rchincha commented Jun 21, 2023

conformance: initial commit for references test #375

conformance: initial commit for references test #375

Conversation

rchincha commented Jan 19, 2023

rchincha commented Jan 24, 2023 • edited Loading

jdolitsky commented Apr 19, 2023

rchincha commented May 26, 2023

sudo-bmitch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sudo-bmitch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rchincha commented Jun 5, 2023 • edited Loading

rchincha commented Jun 9, 2023

Choose a reason for hiding this comment

rchincha Jun 21, 2023 • edited Loading

Choose a reason for hiding this comment

rchincha commented Jun 21, 2023

rchincha commented Jun 21, 2023 • edited Loading

rchincha commented Jun 21, 2023

rchincha commented Jan 24, 2023 •

edited

Loading

rchincha commented Jun 5, 2023 •

edited

Loading

rchincha Jun 21, 2023 •

edited

Loading

rchincha commented Jun 21, 2023 •

edited

Loading