Tool to help creating distributions of packages by jsoriano · Pull Request #1335 · elastic/package-registry

jsoriano · 2025-07-05T16:17:25Z

The tool makes queries to package registry to discover and download packages.

Users provide a configuration file that includes the set of queries to perform, each query receives parameters that map with query parameters. A matrix can be defined with parameters that should be included in all queries.
Actions can be also defined for each package.

Some small examples:

Download all available packages for 9.1.0, with format versions between 2.3 and 3.4.

actions:
  - download:
      destination: ./build/distribution
queries:
  - kibana.version: 9.1.0
    spec.min: 2.3
    spec.max: 3.4

Download all available packages for 9.1.0, with format versions between 2.3 and 3.4, including prereleases

actions:
  - download:
      destination: ./build/distribution
matrix:
  - {}
  - prerelease: true
queries:
  - kibana.version: 9.1.0
    spec.min: 2.3
    spec.max: 3.4

A couple of more complex examples can be found in the PR:

lite.yml downloads the latest version of the packages included in the lite distribution, for each one of the given versions of kibana.
sample.yml downloads the latest version of all packages, including their prereleases.

prodsecmachine · 2025-07-05T16:18:31Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

mrodm

Just take a quick look to this approach, and it looks nice!
This is a tool quite flexible allowing to set different queries to get the desired packages for different scenarios.

For instance it looks easy to create a distribution with just the packages available for Kibana 9.1.0.

The only consideration that I see is that it requires to know how the /search API works along with its query parameters. It will be needed to review the documentation to ensure that there is nothing missing.

cmd/distribution/examples/sample.yaml

cmd/distribution/examples/lite.yaml

cmd/distribution/main.go

jsoriano · 2026-02-09T17:18:04Z

Branch updated, and opening for review.

We were missing the scripts and docs to make this tool useful, but we have found that it is already being used by some users. @fred-maussion is using it, and prepared a PR based on this one with the scripts he is using. It can be found in #1554

So I think we can review and merge this one, and keep iterating on it on follow ups.

teresaromero · 2026-02-10T08:17:05Z

cmd/distribution/main.go

+	baseURL, err := url.Parse(basePath)
+	if err != nil {
+		// This should not happen because JoinPath already parses the url.
+		panic("invalid url")


can we use fmt.Fatalf and attach the error?

teresaromero · 2026-02-10T08:27:03Z

cmd/distribution/main.go

+	os.Exit(status)
+}
+
+type config struct {


can we move this config code into its own internal or to a file under main pkg (like download file) ?

teresaromero · 2026-02-10T08:28:07Z

cmd/distribution/tasks.go

+	"sync"
+)
+
+type taskPool struct {


can we use the internal/workers pkg instead of duplicating the code?

teresaromero · 2026-02-10T08:31:53Z

cmd/distribution/examples/all.yaml

can we organize the yml files under an examples folder or testdata?

teresaromero · 2026-02-10T08:32:33Z

cmd/distribution/Makefile

why do we need a copy of Makefile from root here?

To have the same release targets, though I could remove it as this is not used by now...

cmd/distribution/download.go

…sert in the project

jsoriano · 2026-02-10T12:47:37Z

workers/taskpool.go

 	close(p.pool)
 	p.wg.Wait()
-	close(p.errC)
 	return errors.Join(p.errors...)


There was a race condition here, there could be error that haven't been appended yet at this point. Fixed by directly appending the error instead of using a channel.

workers/taskpool_test.go

magefile.go

cmd/distribution/config.go

elasticmachine · 2026-02-10T15:41:15Z

💚 Build Succeeded

Buildkite Build
Commit: 787984d

History

💚 Build #1680 succeeded e628e53
💚 Build #1678 succeeded 002720e
💚 Build #1674 succeeded a4bc751
💔 Build #1673 failed dcc6d5e
💚 Build #1665 succeeded ff94b0d
💚 Build #1614 succeeded 973d306

cc @jsoriano

mrodm

👍

nimarezainia · 2026-02-17T00:40:32Z

@jsoriano @mrodm was this tool and options documented?

jsoriano · 2026-02-17T12:30:55Z

@jsoriano @mrodm was this tool and options documented?

This PR adds some reference documentation in https://github.com/elastic/package-registry/blob/main/cmd/distribution/README.md

There is also an open follow up PR by @fred-maussion that helps making the tool easier to use, including more docs: #1554

nimarezainia · 2026-02-18T00:59:32Z

@theletterf I think we should be documenting this tool in our public docs as well. I tried to look around but couldn't find any references. We do have a section on using elastic-package itself. We also have section on EPR and folks building their own containers. Not sure where the best place would be.

theletterf · 2026-02-18T09:32:44Z

Sounds good. @jsoriano could you open an issue for this in docs-content?

jsoriano added 5 commits May 16, 2025 18:30

Add scripts to prepare distributions

a7f3b21

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

6e0af0f

Run actions in parallel

e70a21c

Select lite packages per version

10118ea

Add 8.19 and 9.1

b31b25e

jsoriano self-assigned this Jul 5, 2025

jsoriano added 2 commits July 7, 2025 09:31

linting

0b47e15

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

d93fdf9

mrodm reviewed Aug 14, 2025

View reviewed changes

cmd/distribution/examples/sample.yaml Show resolved Hide resolved

cmd/distribution/examples/lite.yaml Show resolved Hide resolved

cmd/distribution/examples/lite.yaml Show resolved Hide resolved

cmd/distribution/examples/lite.yaml Show resolved Hide resolved

jsoriano mentioned this pull request Aug 26, 2025

feat: Add package registry to eck elastic/cloud-on-k8s#8800

Merged

jsoriano mentioned this pull request Oct 6, 2025

[Fleet] Replace package-registry "lite" image in integration tests elastic/kibana#237688

Open

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

10b90f5

mrodm reviewed Nov 14, 2025

View reviewed changes

cmd/distribution/main.go Outdated Show resolved Hide resolved

jsoriano added 7 commits November 14, 2025 13:32

Support pinned package versions

9cc14d4

Reuse existing packages

a967daa

Add Dockerfile

8aa3e5b

Add sample for pinned versions

1148a75

Use ProtonMail fork of opengpg

3ac5e33

Use github.com/Masterminds/semver/v3

e6ca112

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

973d306

jsoriano force-pushed the cmd-distribution-scripts branch from 01055ed to 973d306 Compare January 20, 2026 11:06

jsoriano added 3 commits January 23, 2026 10:12

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

ce4573e

Remove unused dockerfile

468598f

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

ff94b0d

jsoriano marked this pull request as ready for review February 9, 2026 17:18

jsoriano requested a review from a team as a code owner February 9, 2026 17:18

jsoriano requested a review from fred-maussion February 9, 2026 17:18

teresaromero reviewed Feb 10, 2026

View reviewed changes

Apply comments from review

5ecac86

mrodm reviewed Feb 10, 2026

View reviewed changes

cmd/distribution/download.go Show resolved Hide resolved

jsoriano added 6 commits February 10, 2026 10:33

Add tests, fix race condition

dcc6d5e

Fix tests

a4bc751

Move cmd/distribution to its own module

35463c9

Merge remote-tracking branch 'origin/main' into cmd-distribution-scripts

135ac07

Update AGENTS.md with recommendations about the use of require and as…

edcd868

…sert in the project

Update magefile

e9ee6c9

jsoriano commented Feb 10, 2026

View reviewed changes

jsoriano requested review from mrodm and teresaromero February 10, 2026 12:48

jsoriano added 3 commits February 10, 2026 13:51

Remove too specific readme section

002720e

Address some small issues

27ae7af

Fix test names

e628e53

mrodm reviewed Feb 10, 2026

View reviewed changes

workers/taskpool_test.go Outdated Show resolved Hide resolved

workers/taskpool_test.go Outdated Show resolved Hide resolved

workers/taskpool_test.go Outdated Show resolved Hide resolved

magefile.go Show resolved Hide resolved

cmd/distribution/config.go Outdated Show resolved Hide resolved

Apply suggestions from review

787984d

jsoriano requested a review from mrodm February 10, 2026 15:16

mrodm approved these changes Feb 10, 2026

View reviewed changes

jsoriano merged commit 500b635 into elastic:main Feb 11, 2026
5 checks passed

jsoriano deleted the cmd-distribution-scripts branch February 11, 2026 19:04

jsoriano mentioned this pull request Feb 12, 2026

Ensure code stays compatible with the Go version in go.mod #1568

Open

jsoriano mentioned this pull request Feb 18, 2026

[REQUEST]: Prepare public documentation for the distribution helper tool for Package Registry elastic/ingest-docs#1890

Open

Conversation

jsoriano commented Jul 5, 2025

Uh oh!

prodsecmachine commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano commented Feb 9, 2026

Uh oh!

teresaromero Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

teresaromero Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

teresaromero Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

teresaromero Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

teresaromero Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

jsoriano Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsoriano Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Feb 10, 2026

💚 Build Succeeded

History

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nimarezainia commented Feb 17, 2026

Uh oh!

jsoriano commented Feb 17, 2026

Uh oh!

nimarezainia commented Feb 18, 2026

Uh oh!

theletterf commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

prodsecmachine commented Jul 5, 2025 •

edited

Loading