Support input references by package (input type no longer required) by teresaromero · Pull Request #1605 · elastic/package-registry

teresaromero · 2026-03-04T09:22:56Z

What

Added Package field to Input and Stream structs, allowing inputs/streams to reference another package instead of specifying a type/input directly
Made Input.Type and Stream.Input optional (removed validate:"required" tags, added omitempty)
Added validation in Package.validatePackageReference() and DataStream.validStreamsInput() to enforce that each input/stream has either a type/input or a package reference, but not both and not neither
Added test packages (package_reference, stream_validation, stream_validation_invalid, etc.) covering valid and invalid scenarios
Moved test files to a dedicated folder to avoid marshalling errors on good packages

Why

Package spec: As of elastic/package-spec#1071 (merged), the spec defines package dependencies and input references. For format_version 3.6.0 and onwards, either package or type is required for inputs in policy templates and data streams; type is no longer required when package is used.
elastic-package: This work aligns the registry with the composable packages model and unblocks elastic/elastic-package#3278 (Bundle templates of required input packages), where integration packages depend on input packages and their templates are bundled at build time. The registry must correctly load and serve packages that reference input packages via package instead of type.

Checklist

Input struct supports optional type and package
Test package and golden files updated
No breaking change: packages that still use type continue to work

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Made-with: Cursor

mrodm

As here it does not run the validation from package-spec, would it be helpful to add the validation to ensure that Type or Package is set in the manifest (policy template)? and the Input or Package is set in Datastreams ?

package-registry/packages/package.go

Line 695 in 05720fb

func (p *Package) Validate() error {

package-registry/packages/datastream.go

Line 243 in 05720fb

func (d *DataStream) Validate() error {

mrodm · 2026-03-04T10:52:18Z

packages/datastream.go

According to the changes in package-spec PR elastic/package-spec#1071, it looks like that under Stream it should also be added package

https://github.com/elastic/package-spec/blob/5f23052266aab46c2b17423f07ac528c4026fded/spec/integration/data_stream/manifest.spec.yml#L576-L577

And it this struct, it should be one of the Input or Package mandatory

https://github.com/elastic/package-spec/blob/5f23052266aab46c2b17423f07ac528c4026fded/spec/integration/data_stream/manifest.spec.yml#L615-L617

I missed the datastream one! thanks. I've added the field to Streams and remove the validation tag at the Input. I was going through and did not see why input was missing the yaml tag at the struct, so i added too.

- Introduced `validStreamsInput` method in `DataStream` to validate stream inputs. - Added `validatePackageReference` method in `Package` to enforce XOR rule for policy template inputs. - Implemented unit tests for various package validation scenarios, including valid and invalid configurations. - Created test data packages to validate stream input rules and ensure proper error handling for invalid cases.

teresaromero · 2026-03-04T13:31:55Z

As here it does not run the validation from package-spec, would it be helpful to add the validation to ensure that Type or Package is set in the manifest (policy template)? and the Input or Package is set in Datastreams ?

@mrodm i thought about it on the initial implementation, but i finally thought that validation is done when building the package, thanks to the package-spec validation. Also an argument was that this is special from a certain version of the spec... so not all the packages share this.

However, i've added after your comment as i think there is no harm in double check this and make sure as we remove a requirement, the rule is well stablished. I've added tests with mock packages to validate the aproach, i wonder if package-spec should be a dependency here 🤔 as we are kind of duplicating the validations

…od packages

packages/datastream.go

packages/package.go

Co-authored-by: Mario Rodriguez Molins <marrodmo@gmail.com>

mrodm

LGTM
Just add some final comments to update changelog and test packages.

mrodm · 2026-03-09T12:00:25Z

CHANGELOG.md


 ### Added

+* Support input references by package: input `type` is no longer required when `package` is set (format_version 3.6.0+, aligns with package-spec and composable packages). [#1605](https://github.com/elastic/package-registry/pull/1605)


Nit.
This should add a reference to the streams (input and package fields).

mrodm · 2026-03-09T12:00:59Z

testdata/input_validation/stream_validation_invalid/0.1.0/manifest.yml

@@ -0,0 +1,11 @@
+format_version: 3.0.0


Nit. Would these packages require spec 3.6.0?

Suggested change

format_version: 3.0.0

format_version: 3.6.0

mrodm · 2026-03-09T12:01:26Z

testdata/input_validation/policy_template_type_input/0.1.0/manifest.yml

@@ -0,0 +1,20 @@
+format_version: 3.0.0


Nit. spec 3.6.0 ?

Suggested change

format_version: 3.0.0

format_version: 3.6.0

mrodm · 2026-03-09T12:01:51Z

testdata/input_validation/policy_template_both_inputs/0.1.0/manifest.yml

@@ -0,0 +1,21 @@
+format_version: 3.0.0


As mentioned in the other test packages.

Suggested change

format_version: 3.0.0

format_version: 3.6.0

mrodm · 2026-03-09T12:03:59Z

packages/package_test.go

As these test packages are used as part of the packages module, I'd move those to https://github.com/elastic/package-registry/tree/main/packages/testdata

i've moved the success test cases to the package folder. i still want to test validation introduced with wrong packages

…mplates

…y package

…put validation - Updated package paths in `package_test.go` to use relative paths. - Introduced new test data packages for policy templates and stream validation, including: - `policy_template_both_inputs` with both type and package set (invalid case). - `policy_template_type_input` with type only (valid case). - `stream_validation_invalid` with an invalid stream definition. - Added corresponding README files for documentation.

elasticmachine · 2026-03-10T09:54:41Z

💚 Build Succeeded

Buildkite Build
Commit: 915f379

History

💚 Build #1784 succeeded b15eab5
💚 Build #1782 succeeded ea7715a
💔 Build #1779 failed 2a4be82
💚 Build #1773 succeeded f755679
💔 Build #1772 failed 75ee197
💚 Build #1771 succeeded 4a53042

Support input references by package (input type no longer required)

69675f1

Made-with: Cursor

teresaromero requested a review from a team as a code owner March 4, 2026 09:22

Add changelog entry for input references by package (elastic#1605)

4a53042

Made-with: Cursor

teresaromero mentioned this pull request Mar 4, 2026

bundle templates of required input packages elastic/elastic-package#3329

Open

mrodm reviewed Mar 4, 2026

View reviewed changes

teresaromero added 2 commits March 4, 2026 14:26

Remove validation tag to Stream.Input. Add Package field into Stream.

eebf4af

Move test files to dedicated folder to avoid marshalling errors of go…

f755679

…od packages

mrodm reviewed Mar 4, 2026

View reviewed changes

packages/datastream.go Outdated Show resolved Hide resolved

packages/package.go Outdated Show resolved Hide resolved

teresaromero and others added 2 commits March 9, 2026 08:38

Apply suggestions from code review

2a4be82

Co-authored-by: Mario Rodriguez Molins <marrodmo@gmail.com>

Add case where both are set

ea7715a

mrodm reviewed Mar 9, 2026

View reviewed changes

teresaromero added 3 commits March 10, 2026 09:08

Update manifest files to format version 3.6.0 for input validation te…

2978852

…mplates

Update changelog to clarify input and stream reference requirements b…

67713ee

…y package

Move success test to testdata/package and update snapshots

b15eab5

teresaromero mentioned this pull request Mar 10, 2026

Add requires field to package manifest output #1612

Merged

mrodm approved these changes Mar 10, 2026

View reviewed changes

teresaromero merged commit 47fe54d into elastic:main Mar 10, 2026
5 checks passed


		### Added

		* Support input references by package: input `type` is no longer required when `package` is set (format_version 3.6.0+, aligns with package-spec and composable packages). [#1605](https://github.com/elastic/package-registry/pull/1605)

Conversation

teresaromero commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Checklist

Related

Uh oh!

prodsecmachine commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Mar 4, 2026 •

edited

Loading

prodsecmachine commented Mar 4, 2026 •

edited

Loading