Devfile Library should have the ability to get resources from a private repo

[maysunfaisal]

/kind user-story

Which area this user story is related to?

/area library

User Story

We would like if the devfile/library had the ability to get resources from a private repository. This would be required for reading devfile.yaml or an outerloop kubernetes deploy.yaml defined in the devfile for example.

The library should have the ability to read the private resources for most of the leading repositories like github, gitlab, bitbucket, etc.

In AppStudio, this is done using SPI but it would require installing their CRs and Controllers. But we could also look at the OpenShift Developer Console(ODC) as to how they do it for their codebase but their implementation would be in TS.

Acceptance Criteria

• Should be able to get resources from a private repository
• Should be able to support the leading repositories like github, gitlab, bitbuckket

Target Date: 2023-04-28

[mike-hoang]

context

Currently, when a devfile is parsed from a uri

schemaVersion: 2.2.0
metadata:
  name: my-project-dev
parent:
  uri: https://raw.githubusercontent.com/devfile/registry/main/stacks/nodejs/devfile.yaml

that is in a repo from a supported git provider, the resources from the repo are downloaded.

problem

If we want to download from private repos, we need to authenticate with a token. How should this git token be passed into the library?

1. Using a Kubernetes secret

type ParserArgs struct {
	...
	// K8sClient is the Kubernetes client instance used for interacting with a cluster
	K8sClient client.Client
	...
}

2. Passing in the token itself; could be from an env variables, etc

3. Other?

[mike-hoang]

01/23 - Devfile community call

• potential duplication of work; align with SPI
• consider explicitly mapping repo url with credentials instead of inferring from url

[mike-hoang]

removing from hold; will continue with implementation with the potential of swapping out with SPI in the future

[mike-hoang]

token flow: will scope to single scenario where only the private parent resources are downloaded for now. will need to consider private child and private parent scenario in the future

experimenting with mocking a happy path test for cloning a repo

edit 03/08
two cases when the git token is used:

1. main devfile comes from a private repo (requires using a token in util.DownloadInMemory())
2. downloading parent devfile resources from a private repo (requires using a token in the parser)

consumers will pass the token through the ParserArgs struct:

type ParserArgs struct {
        ...
+       // Token is a GitHub, GitLab, or Bitbucket personal access token used with a private git repo uri
+	Token string
        ...
}

the single token passed from client means that:

• main devfile is public and token works only for private parent repo
• main devfile and parent devfile are in the same private repo

future: will need to consider the case where the main and parent devfiles are in different private repos

edit 03/21
wasn't able to mock the happy path for downloading a repo or file; SetToken validates token with a network call and is wrapped by the file and repo downloading

found edge case where DownloadInMemory only works if Github url is in the raw format, will need to address

edit 03/28
added mocks for testing and GitRawFileAPI to get the raw file endpoints for the different providers

edit 03/29
addressing review comments; fixed private main devfile support

[mike-hoang]

ongoing review progress; updating token location and url parsing

[mike-hoang]

moving back to in progress; unit tests are completed, but functional testing needs to be added / at the minimum explored properly before the PR can be merged. if functional tests can't be added, manual testing needs to be documented as a final option

[mike-hoang]

Found a way to add functional tests and completed the mocking setup; moved parent resource downloading into a separate method that can then be mocked. Currently creating the test cases with the mocks

[mike-hoang]

04/14 - added functional test cases, which required:

• adding the git interface to the devfile context struct
• storing the git interface in the resolverTools for parent cases (context is lost after parseFromKubeCRD)
• token duplication in the context and the git interface