OCI-based devifle registry support

[GeekArthur]

Signed-off-by: jingfu wang [email protected]

What type of PR is this?
/kind feature

What does this PR do / why we need it:
This PR implements the OCI-based devfile registry support, more specifically:

1. Consume devfile registry library or REST API (list devfile components + create devfile component)
2. Change the default devfile registry URL to OCI-based public devfile registry

Which issue(s) this PR fixes:

Related issues:

1. odo Migration to OCI based public community devfile registry devfile/api#367
2. Use official devfile registry #4504

PR acceptance criteria:

• Unit test

• Integration test

• Documentation

• I have read the test guidelines

How to test changes / Special notes to the reviewer:

1. Run odo registry add OCI https://registry.devfile.io to add OCI-based public devfile registry if it's not in the registry list
2. Run odo catalog list components then verify the devfiles from OCI-based public devfile registry can be listed
3. Run odo create nodejs --registry OCI then verify the nodejs devfile can be downloaded from OCI-based public devfile registry

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[kadel]

We need to find a way how to change DefaultDevfileRegistry for the current users.
Existing odo users will already have the existing DefaultDevfileRegistry record in preference.yaml, odo needs to update it to the new registry, otherwise, all current users will get stuck with the default GitHub based registry.

[johnmcollier]

We need to find a way how to change DefaultDevfileRegistry for the current users.
Existing odo users will already have the existing DefaultDevfileRegistry record in preference.yaml, odo needs to update it to the new registry, otherwise, all current users will get stuck with the default GitHub based registry.

Yup, agreed. Some useful discussion in #3298 about this as well that might provide some insight on how we can do this

[johnmcollier]

@GeekArthur The odo catalog describe tests are failing due to this error:

 [odo] Starter Projects:
[odo] I0330 15:04:12.105604   59639 validate.go:50] Successfully validated devfile sections
[odo] ---
[odo] name: nodejs-starter
[odo] attributes: {}
[odo] description: ""
[odo] subdir: ""
[odo] projectsource:
[odo]   sourcetype: ""
[odo]   git:
[odo]     gitlikeprojectsource:
[odo]       commonprojectsource: {}
[odo]       checkoutfrom: null
[odo]       remotes:
[odo]         origin: https://github.com/odo-devfiles/nodejs-ex.git
[odo]   github: null
[odo]   zip: null
[odo]   custom: null
[odo] 
[odo] * Registry: DefaultDevfileRegistry
[odo]  ✗  Get https://registry.devfile.iodevfile-catalog/nodejs:latest: dial tcp: lookup registry.devfile.iodevfile-catalog on 172.30.0.10:53: no such host 

You need to update the GetDevfile function to properly handle cases where the devfile is in an OCI registry (specifically the call to devfile.ParseFromURLAndValidate):

https://github.com/openshift/odo/blob/12b03e5924b1db8ea1634ecf22e9aab215bc7ab5/pkg/odo/cli/catalog/describe/component.go#L197-L209

[GeekArthur]

We need to find a way how to change DefaultDevfileRegistry for the current users.
Existing odo users will already have the existing DefaultDevfileRegistry record in preference.yaml, odo needs to update it to the new registry, otherwise, all current users will get stuck with the default GitHub based registry.

Yup, agreed. Some useful discussion in #3298 about this as well that might provide some insight on how we can do this

Yeah agree, I also wanna link Maysun's issue here, thanks John.

[GeekArthur]

@GeekArthur The odo catalog describe tests are failing due to this error:

 [odo] Starter Projects:
[odo] I0330 15:04:12.105604   59639 validate.go:50] Successfully validated devfile sections
[odo] ---
[odo] name: nodejs-starter
[odo] attributes: {}
[odo] description: ""
[odo] subdir: ""
[odo] projectsource:
[odo]   sourcetype: ""
[odo]   git:
[odo]     gitlikeprojectsource:
[odo]       commonprojectsource: {}
[odo]       checkoutfrom: null
[odo]       remotes:
[odo]         origin: https://github.com/odo-devfiles/nodejs-ex.git
[odo]   github: null
[odo]   zip: null
[odo]   custom: null
[odo] 
[odo] * Registry: DefaultDevfileRegistry
[odo]  ✗  Get https://registry.devfile.iodevfile-catalog/nodejs:latest: dial tcp: lookup registry.devfile.iodevfile-catalog on 172.30.0.10:53: no such host 

You need to update the GetDevfile function to properly handle cases where the devfile is in an OCI registry (specifically the call to devfile.ParseFromURLAndValidate):

https://github.com/openshift/odo/blob/12b03e5924b1db8ea1634ecf22e9aab215bc7ab5/pkg/odo/cli/catalog/describe/component.go#L197-L209

Yep, will fix that with other test automation failures.

[elsony]

@GeekArthur , @kadel and I have discussed the default registry migration. We'll just take a look at the existing preference, if there is a default entry that points to the old repo, we'll just migrate the registry URL to the new URL silently in the preference file without prompting the user.

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[GeekArthur]

/retest

[girishramnani]

Is this a genuine failure?

[GeekArthur]

Is this a genuine failure?

@girishramnani No, it's test infra issue, I posted it on odo slack channel: https://kubernetes.slack.com/archives/C01D6L2NUAG/p1618243057007700

[GeekArthur]

@johnmcollier @kadel All comments addressed, I already discussed with Tomas, due to the test infra issues, the PR can be merged once it satisfies the following requirements:

1. Has approve and lgtm labels
2. Pass unit and ci/prow/v4.7-integration-e2e tests

Please review again, thanks.

[johnmcollier]

No longer a reviewer on this repo, but approving from the devfile side of things

/lgtm

Edit: Didn't expect that to actually add the label 😅 , thought you had to be in the OWNERS file?

[kadel]

Edit: Didn't expect that to actually add the label 😅 , thought you had to be in the OWNERS file?

wow. Interesting. You still have to write permission on this repo, that might be why. But this is new to me :-)

[kadel]

/approve

I have one small issue with the current migration approach. But that can be addressed separately.
Opened #4626 to track it

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GeekArthur, kadel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kadel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kadel]

/lgtm

[openshift-ci]

@GeekArthur: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/psi-openshift-integration-e2e	b15145e	link	/test psi-openshift-integration-e2e
ci/prow/psi-minishift-integration-e2e	b15145e	link	/test psi-minishift-integration-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[GeekArthur]

/retest