bundle templates of required input packages by teresaromero · Pull Request #3329 · elastic/elastic-package

teresaromero · 2026-03-03T08:47:45Z

Bundle input package templates for composable integrations

Composable packages stack — part 1 of 4.
Merge order: this PR → teresaromero#1 → teresaromero#2 → teresaromero#3.
Each subsequent PR is stacked on this one and will be rebased onto main once this merges.

Summary

Implements build-time template bundling for integration packages that declare dependencies on input packages via requires.input in their manifest. At build time, elastic-package downloads the required input packages from EPR and copies their agent templates into the integration's build output so Fleet can render them correctly. Both template_path and template_paths fields are handled — input package templates are prepended to the integration's own templates, so the integration takes precedence during rendering.

Bundling runs in two paths:

Policy template inputs (bundlePolicyTemplatesInputPackageTemplates): triggered when policy_templates[].inputs[].package references an input package. Templates from agent/input/ are copied to the integration's agent/input/ directory (prefixed with the package name), and the manifest is rewritten to use template_paths.
Data stream streams (bundleDataStreamTemplates): triggered when data_stream/<name>/manifest.yml → streams[].package references an input package. Templates from the input package's agent/input/ are copied to the data stream's agent/stream/ directory and template_paths is updated. Bundling is driven solely by this per-stream package reference — Fleet then selects the appropriate templates at render time.

What changed

internal/requiredinputs (new package)

RequiredInputsResolver: downloads required input packages from EPR into a temp directory; supports directory and zip formats.
bundlePolicyTemplatesInputPackageTemplates: copies and wires input package agent/input/ templates for policy template inputs.
bundleDataStreamTemplates: copies and wires input package agent/input/ templates for data stream stream entries.
YAML in-place update utilities that preserve comments and formatting.

internal/packages — extended manifest types

PolicyTemplate: adds DataStreams, TemplatePaths.
PackageManifest.Input: adds Package, TemplatePath, TemplatePaths.
DataStreamManifest.Stream: adds Package, TemplatePath, TemplatePaths.
New Requires / PackageDependency types; PackageManifest.Requires wired up.

internal/registry — DownloadPackage fetches and extracts packages from EPR; TLS support via profile CA cert or env-provided cert.

Build pipeline — BundleInputPackageTemplates called during build after package assembly; RequiredInputsResolver threaded through BuildOptions and installer.

Test fixtures — test/manual_packages/required_inputs/test_input_pkg (minimal input package) and test/manual_packages/required_inputs/with_input_package_requires (integration consuming it).

How to test

Build the input package so it is available in the local build output:

cd test/manual_packages/required_inputs/test_input_pkg
elastic-package build

Start the stack from anywhere inside the elastic-package repository so the built package is served by the stack's built-in registry:
```
elastic-package stack up --version 9.4.0-SNAPSHOT
```
Build the local tool binary from the repo root, then run it against the integration package:
```
go build -o elastic-package .
cd test/manual_packages/required_inputs/with_input_package_requires
../../elastic-package build
```
Inspect the build/ directory — the input package's agent templates should appear in agent/input/ and data_stream/test_logs/agent/stream/, and the manifests should reference them via template_paths.

Dependencies

Support input references by package: Support input references by package (input type no longer required) package-registry#1605
package-spec v3.5.8: Release 3.6 package-spec#1060

Not included

Local source override for input package resolution (bypass EPR during development) — removed pending a cleaner design, to be addressed in a follow-up.

teresaromero · 2026-03-03T08:49:18Z

blocked for merge until 3.6 elastic/package-spec#1060

Replace the existing package-spec dependency with a specific pre-release version that includes support for the requires.input manifest section. NOTE: Remove the replace directive before merging and use the latest released version when available. Made-with: Cursor

Add Requires, PackageDependency, and RequiresOverride types. Extend Stream with PackageRef and TemplatePaths, PolicyTemplate with TemplatePaths, and PackageManifest with a Requires field. Add FindPackageInRepo to search the repository for a package by name, scanning up to 4 directory levels to accommodate monorepo layouts.

Add Client.DownloadPackage which fetches a package by name and version from the Elastic Package Registry, extracts the zip archive into a temporary directory, and returns the path to the extracted package root. Add bundleInputPackageTemplates, which runs after the package content is copied to the build directory. For each integration that declares requires.input, it resolves every required input package (from the registry or a local source override), copies agent input templates into data_stream/<ds>/agent/stream/ with a "<pkgname>-" prefix, and rewrites the data stream manifest to use template_paths. Add test fixtures (test_input_pkg, with_input_package_requires) and unit tests for the YAML rewriting helpers and the full bundling path.

… pipeline Thread RequiresOverrides and RegistryURL through FleetPackage, installer.Options, and BuildOptions so test runners can supply source or version overrides for required input packages. Support requires overrides at both top-level and per-type sections in the global test config (_dev/test/config.yml). Per-type entries override global entries by package name. Inject the registry client from app config values to enable local or custom package registries during the build. Add an integration test that exercises the full template bundling path using a local fixture package, bypassing the package registry.

Add a new HOWTO guide explaining how to configure elastic-package to use a local or custom package registry when developing composable integrations that declare required input packages. Update the build command description, README, and dependency management docs to document the relationship between the build process and the package registry for composable packages.

…ed before merge

…undle-templates-required-input-packages

cmd/install.go

…undle-templates-required-input-packages

jsoriano

Nice, thanks!

docs/howto/dependency_management.md

jsoriano · 2026-03-10T18:44:16Z

docs/howto/local_package_registry.md

+This guide explains how to point elastic-package at a local or custom registry, which is
+useful when the required input packages are still under development and not yet published to
+the production registry.


elastic-package stack up with the compose provider (the default), starts a local package registry, that serves the packages under build/packages. This can be used to serve local packages. You only need to remember building the ones you need.

This mechanism would even be improved after your change in elastic/package-registry#1482, what would allow reloading the packages without needing to restart the registry.

In the documentation here we are adding another method to serve local packages. We should probably converge both methods in a single one.

docs/howto/local_package_registry.md

internal/builder/input_packages.go

jsoriano · 2026-03-10T19:19:47Z

internal/builder/packages.go

 		return "", fmt.Errorf("resolving transform manifests failed: %w", err)
 	}

+	err = bundleInputPackageTemplates(options.PackageRoot, buildPackageRoot, options.RegistryClient, options.RequiresOverrides)


For a future refactor. We have several methods that receive options.PackageRoot to read the manifest. We could maybe make these methods members of an object that shares the manifest and maybe other parts of a package.

this was only required to extract the _dev/test configs... as i've removed this part i will see how to include this param when i implement the testing part

jsoriano · 2026-03-10T19:34:43Z

internal/builder/input_packages.go

+	dsRoot := filepath.Dir(dsManifestPath)
+	agentStreamDir := filepath.Join(dsRoot, "agent", "stream")
+
+	// Parse the YAML document preserving formatting for targeted modifications.


Preserving the format is nice, though we don't really need to keep the format here, as we are building the package, that is not intended for later human-driven modifications. I would be concerned of complicating things unnecessarily.

does this mean we can modify using the structs and save a new copy from them? i thought we needed to preserve how the manifest was and working directly with yml

We don't need the built package to be human-friendly, so we can do what is simpler to maintain.

i've gave a try to not use the yaml node aproach, but the models (structs) from elastic-package do not cover all the manifest data... so when marshalling and unmarshalling, data is lost. I was thinking that perhps we should export the models from the spec as a library, and a single point of truth for go structs reflecting the manifest from yaml.

however, i kept the yaml implementation so we just modify the necessary fields, without data loss

internal/builder/input_packages.go

jsoriano · 2026-03-10T19:46:28Z

internal/packages/installer/factory.go

+	RegistryClient    *registry.Client                     // Registry client for downloading input packages.
+	RequiresOverrides map[string]packages.RequiresOverride // Pre-merged requires overrides (test builds only).


Nit. Maybe for a future change. Instead of receiving the registry and the overrides, this could receive a higher level abstraction that gets dependencies. This abstraction would handle the different package sources, caches, overrides and so on.

i've removed for now this, i am still wondering how to do the overrides. for now i am injecting an interface with the epr client, i am not sure if there should be an implementation for tests and other for build/install

i am still understanding how this should work, could you provide an example of what should be expected when the override is defined?

internal/packages/packages.go

internal/registry/download.go

…ncies resolution. Enhance local package registry instructions with built-in stack registry usage and standalone registry setup.

…undle-templates-required-input-packages

…readability and performance

…undle-templates-required-input-packages

…ociation bundleDataStreamTemplates now reads the integration's root manifest and builds a dsName→packages lookup from policy templates that explicitly declare data_streams[]. Stream entries are only bundled when their package is present in this lookup, preventing implicit 1:1 assumptions between data streams and policy templates. Adds TestBundleDataStreamTemplates_SkipWhenNoDataStreamsAssociation, moves shared test helpers to testhelpers_test.go, and updates the with_input_package_requires fixture with an explicit data_streams field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Bundle templates based solely on the data stream manifest's streams[].package reference, without requiring an explicit data_streams association in the root policy template. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

teresaromero · 2026-03-19T10:39:35Z

When installing the package or running policy tests for a composable package fleet returns error

Error: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Invalid manifest for data stream test_logs: stream is missing one or more fields of: input, title"}

Implementing tests and running installation of the composable will require #3380

…esolver Add NoopRequiredInputsResolver to avoid nil interface panics in integration tests that don't have an EPR client. Replace filepath.Join/Dir with path.Join/Dir in policytemplates.go and streams.go so fs.FS and os.Root receive forward-slash paths on all platforms including Windows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…tests

Relocate test_input_pkg and with_input_package_requires out of test/packages so CI build/install zip loops no longer pick them up. Add test/manual_packages/README.md with manual testing steps. Made-with: Cursor

…registry" This reverts commit 86557d8.

…undle-templates-required-input-packages

docs/howto/local_package_registry.md

internal/requiredinputs/requiredinputs.go

mrodm · 2026-03-24T15:52:52Z

internal/requiredinputs/policytemplates.go

+	// for each policy template, with an input package reference:
+	// collect the templates from the input package and copy them to the agent/input directory of the build package
+	// then update the policy template manifest to include the copied templates as template_paths


Have this process being tested with linked files ?

Checking the spec, template paths could also be linked files. For instance:
https://github.com/elastic/package-spec/blob/73ad6a82946c4880c327b09bb28327bb0d44c0f8/spec/integration/data_stream/agent/spec.yml#L30

For packages downloaded from EPR those files should be already resolved, but maybe while testing using local folders (e.g. in system tests), could there be any issue if package contains linked files?

i have not tested with linked files yet. the feature for testing and replacing epr for a local source is still WIP

mrodm · 2026-03-24T17:11:09Z

internal/registry/download.go

+	zipPath := filepath.Join(destDir, fmt.Sprintf("%s-%s.zip", name, version))
+	if err := os.WriteFile(zipPath, zipBytes, 0644); err != nil {
+		return "", fmt.Errorf("writing package zip %s-%s: %w", name, version, err)
+	}


During this process, once the package has been downloaded, should it be validated that package with the corresponding signature (if exists) here too ? signature_path from the package response (metadata in this method). cc @jsoriano

internal/requiredinputs/policytemplates.go

Use os.OpenRoot and root.FS() instead of os.DirFS so reads stay scoped to the package subtree like the build root path. Callers must close the root via the returned close function; update openPackageFS godoc accordingly. Made-with: Cursor

Use a "failed to <action>" pattern across policy template bundling, data stream bundling, and YAML formatting. Distinguish read versus parse failures for package manifests. Update tests that assert on error substrings. Made-with: Cursor

Use docker.elastic.co/package-registry/package-registry:v1.37.0 to match PackageRegistryBaseImage. Document distribution:lite as a smaller alternative and point readers to internal/stack/versions.go for the canonical tag. Made-with: Cursor

docs/howto/local_package_registry.md

mrodm · 2026-03-26T11:17:57Z

internal/packages/installer/factory.go

+type requiredInputsResolver interface {
+	BundleInputPackageTemplates(buildPackageRoot string) error
+}


I see this is duplicated in different packages.
If this is moved for instance under the new requiredinputs package, does it results in circular dependencies?

i used this as a pattern to avoid exporting interfaces and make implementations independent from each other. i've moved the interface to the package as it does not create circular dependencies.

Replace per-package private requiredInputsResolver definitions with a single exported Resolver interface in internal/requiredinputs. Consumers import requiredinputs for the contract; *RequiredInputsResolver still implements it. Made-with: Cursor

…undle-templates-required-input-packages

elasticmachine · 2026-03-26T15:24:33Z

💚 Build Succeeded

Buildkite Build
Commit: 95c83df

History

💚 Build #7579 succeeded f0eebb2
💚 Build #7563 succeeded 4c381ed
💚 Build #7536 succeeded 32a2ff5
💔 Build #7532 failed 0f962e7
💔 Build #7509 failed 86557d8
💔 Build #7507 failed f4d026b

teresaromero marked this pull request as ready for review March 3, 2026 08:49

teresaromero requested a review from a team as a code owner March 3, 2026 08:49

teresaromero marked this pull request as draft March 3, 2026 09:36

teresaromero added 5 commits March 3, 2026 14:50

teresaromero changed the title ~~builder: bundle templates of required input packages~~ bundle templates of required input packages Mar 4, 2026

teresaromero force-pushed the 3278-bundle-templates-required-input-packages branch from 9b55229 to b43bf5b Compare March 4, 2026 10:30

teresaromero added 2 commits March 4, 2026 11:39

chore: update with latests package-spec. this commit should be revert…

5da820b

…ed before merge

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

09fdab8

…undle-templates-required-input-packages

teresaromero marked this pull request as ready for review March 4, 2026 10:42

mrodm reviewed Mar 9, 2026

View reviewed changes

cmd/install.go Outdated Show resolved Hide resolved

teresaromero added 3 commits March 10, 2026 10:54

use profile epr url on install command. update readme

7468026

Revert "update readme"

50c3b2b

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

a690ef0

…undle-templates-required-input-packages

mrodm requested a review from jsoriano March 10, 2026 17:10

jsoriano reviewed Mar 10, 2026

View reviewed changes

teresaromero added 2 commits March 11, 2026 09:38

Update dependency management documentation to clarify package depende…

d76ffad

…ncies resolution. Enhance local package registry instructions with built-in stack registry usage and standalone registry setup.

Add TLS support to registry client

ad98db8

teresaromero marked this pull request as draft March 11, 2026 11:32

teresaromero added 7 commits March 12, 2026 12:10

Add bundle policy templates input package handling and tests

5d69e4b

Add bundle data stream templates handling and tests

1d8e4dd

Add YAML utility functions for mapping and formatting

11fe9df

Add function to read and unpack data stream manifest bytes

22047e8

refactor bundle templates for builder

2cd96e7

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

9f6ade8

…undle-templates-required-input-packages

remove overrides implementation

e490bb3

teresaromero added 3 commits March 13, 2026 11:53

Remove FindPackageInRepo function from packages.go

0ba7f63

rename PackageRef

dbe671b

Refactor YAML utility functions to use slices.IndexFunc for improved …

8337f12

…readability and performance

jsoriano mentioned this pull request Mar 13, 2026

[Fleet] Resolve and merge templates listed in template_paths elastic/kibana#257730

Merged

10 tasks

teresaromero and others added 5 commits March 18, 2026 13:52

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

9c3301f

…undle-templates-required-input-packages

chore: update package-spec dependency to v3.5.8 in go.mod and go.sum

3dc281d

docs: update howto guide

42ad433

teresaromero marked this pull request as ready for review March 19, 2026 09:53

teresaromero and others added 6 commits March 19, 2026 12:18

test: ensure proper resource cleanup in policy templates and streams …

f4d026b

…tests

tests: update test script to include test package into stack registry

86557d8

Move required_inputs fixtures to test/manual_packages

0f962e7

Relocate test_input_pkg and with_input_package_requires out of test/packages so CI build/install zip loops no longer pick them up. Add test/manual_packages/README.md with manual testing steps. Made-with: Cursor

Revert "tests: update test script to include test package into stack …

32c11ae

…registry" This reverts commit 86557d8.

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

32a2ff5

…undle-templates-required-input-packages

mrodm reviewed Mar 24, 2026

View reviewed changes

teresaromero added 3 commits March 25, 2026 09:20

mrodm reviewed Mar 26, 2026

View reviewed changes

teresaromero added 3 commits March 26, 2026 14:27

Align docs with package-registry and elastic-package

f0eebb2

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

95c83df

…undle-templates-required-input-packages

		RegistryClient *registry.Client // Registry client for downloading input packages.
		RequiresOverrides map[string]packages.RequiresOverride // Pre-merged requires overrides (test builds only).

Conversation

teresaromero commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle input package templates for composable integrations

Summary

What changed

How to test

Dependencies

Not included

Uh oh!

teresaromero commented Mar 3, 2026

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

teresaromero commented Mar 19, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 26, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Mar 3, 2026 •

edited

Loading

mrodm Mar 24, 2026 •

edited

Loading