Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasource): add debian datasource #30071

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

oxdev03
Copy link

@oxdev03 oxdev03 commented Jul 7, 2024

Changes

  • Built upon @Ka0o0 implementation of the Debian datasource.
  • Includes minor improvements.
  • Addresses coverage issues identified in the previous review of the Debian datasource.
  • Fixes earlier review findings (except the caching issue).

Context

Resolves: #7041, #24906

Concept

Based on the discussion and the reused implementation (see the related issue for more details):

  1. Evaluation of Provided or Default Registry URL

    • Example: The URL https://ftp.debian.org/debian?release=bullseye&components=main,contrib&binaryArch=amd64 is evaluated to:
      • .../debian/dists/bullseye/main/binary-amd64
      • .../debian/dists/bullseye/contrib/binary-amd64
    • Supports multiple component URLs.
  2. Check for Existing Compressed Package

    • If the compressed Packages.gz file has already been downloaded and extracted:
      • Send a HEAD request to check if the upstream file has been updated.
    • If not already downloaded:
      • Download the compressed file.
      • Extract the contents.
      • Delete the compressed file.
    • Currently, only gzip compression is supported.
  3. Iterate through Package Index

    • Gather relevant information for the requested package.
    • Stop iteration, if found
  4. Return relevant release information

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

included documentation for new datasource

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

Real Test

Open Points

  • Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

  • Is the current approach generic enough? Are there any points that were not considered?

  • If significant rework is required, please let me know in advance.

@CLAassistant
Copy link

CLAassistant commented Jul 7, 2024

CLA assistant check
All committers have signed the CLA.

@viceice viceice self-requested a review July 7, 2024 17:15
Copy link
Collaborator

@secustor secustor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, and the well documented functionality!

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Would this work for every APT repository?

lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/modules/datasource/deb/index.ts Outdated Show resolved Hide resolved
lib/util/fs/index.ts Outdated Show resolved Hide resolved
* @returns An array of component URLs.
* @throws Will throw an error if required parameters are missing from the URL.
*/
constructComponentUrls(registryUrl: string): string[] {
Copy link
Collaborator

@secustor secustor Jul 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use functions and not methods where possible. Also move functions into separate files.

});
};

const getReleaseParam = (url: URL): string => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No local functions, please.

@secustor
Copy link
Collaborator

secustor commented Jul 7, 2024

Does the current caching implementation for the datasource need to be reworked? Can the Renovate team provide a concept/details for this? (e.g., invalidation of the compressed file based on its timestamp)

We should start simple and then start to optimize.
The Package.gz will be cached non transparently by the http layer, so the file will be downloaded only when necessary.
I suggest to simply cache the extracted file on a time base ( extract it to a different function and use the annotation ).

@oxdev03
Copy link
Author

oxdev03 commented Jul 7, 2024

This is my first pass as there is a lot going on here and we have to be sure that the caching is setup efficiently.

Thx for the initial review, I will fix the findings soon.

Would this work for every APT repository?

should work with every apt repository, which follows the https://wiki.debian.org/DebianRepository standard.

});
});

const getComponentUrl = (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use proper function declarations

Comment on lines 67 to 70
<!-- prettier-ignore -->
!!! tip
We recommend you try `loose` versioning for distribution packages first.
This is because the version number usually doesn't match Renovate's default `semver-coerced` specification.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No don't set any versioning, the datasource should set deb versioning as default, which should work in those cases.

@@ -33,7 +33,7 @@ First you would set a custom manager in your `renovate.json` file for `Dockerfil

Then you would put comments in your Dockerfile, to tell Renovate where to find the updates:

```docker
```dockerfile
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants