[odo] Failure to fetch extra stack files along with Devfile from the staging registry

[valaparthvi]

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area api

/area ci
/area api
/area library
/area registry
/area devworkspace
/area integration-tests
/area test-automation
/area releng
/area landing-page

What versions of software are you using?

Go project

Operating System and version:

Go Pkg Version:

Node.js project

Operating System and version:

Node.js version:

Yarn version:

Project.json:

Web browser

Operating System and version:

Browser name and version:

Bug Summary

Describe the bug:
odo uses the Staging Registry to fetch Devfiles for it's integration test on CI.

https://github.com/redhat-developer/odo/blob/08c3904ff249f1eb0a937152e01bd38ee9e7ce42/tests/helper/helper_generic.go#L376

Things were working as expected until March 29, when we were able to fetch the extra stack files along with the Devfile (e.g. go:latest, java-springboot:latest, python:latest), but we can't seem to fetch it now, and this is blocking us (We are in the process of using an alternate solution, but the issue still remains if one wants to use the staging registry).

To Reproduce:

1. odo preference add registry StagingDevfileRegistry https://registry.stage.devfile.io
2. odo init --devfile go --devfile-version 2.0.0 --name my-go-app --devfile-registry StagingDevfileRegistry --starter go-starter

Expected behavior

It should be possible to fetch the extra stack files.

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable

➜  odo preference view
[...]
Devfile registries:
 NAME                    URL                                SECURE
 StagingDevfileRegistry  https://registry.stage.devfile.io/  No
 DefaultDevfileRegistry  https://registry.devfile.io/        No

➜  odo init --devfile go --devfile-version 2.0.0 --name my-go-app --devfile-registry StagingDevfileRegistry
  __
 /  \__     Initializing a new component
 \__/  \
 /  \__/    odo version: v3.8.0
 \__/

 ✓  Downloading devfile "go:2.0.0" from registry "StagingDevfileRegistry" [3s]
 ✗  unable to parse devfile: failed to convert kubernetes uri to inlined for component 'deploy': failed to read kubernetes resources definition from path '/tmp/nodejs-debug/kubernetes/deploy.yaml': open /tmp/nodejs-debug/kubernetes/deploy.yaml: no such file or directory
the command failed, the devfile has been removed from current directory

Additional context

Any workaround?

Suggestion on how to fix the bug

Target Date: April 4, 2023

[feloy]

After some debugging on odo, I can see that the difference between the Staging and the Production registries can be seen at (https://github.com/devfile/registry-support/blob/main/registry-library/library/library.go#L275):

	_, err = oras.Copy(ctx, resolver, ref, fileStore, ref, oras.WithAllowedMediaTypes(allowedMediaTypes))

After this call, for a stack containing extra files:

• with the Staging registry, only the devfile.yaml file is downloaded
• with the Production registry, the devfile.yaml and archive.tar files are downloaded.

[kim-tsao]

@feloy @valaparthvi, I confirmed this is a bug with the way I set up the manifest for pushing artifacts to the OCI server. I have a potential fix with my PR but I'm still adding test cases to ensure there's proper coverage. Hopefully, I can get this PR in review state tomorrow.

[kim-tsao]

The changes are now in staging. I re-ran the odo command and verified the artifacts are downloaded:

./odo init --devfile go --devfile-version 2.0.0 --name my-go-app --devfile-registry StagingRegistry
  __
 /  \__     Initializing a new component
 \__/  \    
 /  \__/    odo version: v3.8.0
 \__/

 ✓  Downloading devfile "go:2.0.0" from registry "StagingRegistry" [1s]

Your new component 'my-go-app' is ready in the current directory.
To start editing your component, use 'odo dev' and open this folder in your favorite IDE.
Changes will be directly reflected on the cluster.
To deploy your component to a cluster use "odo deploy".

[valaparthvi]

Does this also work for other devfiles(java-springboot:latest, python:latest) with extra stack files?

[kim-tsao]

Does this also work for other devfiles(java-springboot:latest, python:latest) with extra stack files?

yes, I just confirmed it works for both. By extra stack files, I assume you mean all the contents in this folder for example: https://github.com/devfile/registry/tree/main/stacks/java-springboot/2.0.0. The dockerfile, deploy.yaml and devfile.yaml are all downloaded and available in their respective folders

[valaparthvi]

I've confirmed that this works for me as well, thank you for fixing this @kim-tsao :)