Composable packages: input name qualifiers and stream input refs#3480
Composable packages: input name qualifiers and stream input refs#3480teresaromero merged 37 commits intoelastic:mainfrom
name qualifiers and stream input refs#3480Conversation
…e-type inputs When a composable policy template ends up with multiple required input packages that share the same type (e.g. two otelcol inputs), set a unique name on each input derived from the required package name so Fleet can disambiguate them (package-spec SVR00010). Data stream manifests then reference inputs by name instead of type, matching the Fleet/Kibana expectation from package-spec 3.6.0+. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fiers Updated the logic in `BuildIntegrationPackagePolicy` to utilize a unique name qualifier for inputs that share the same type, ensuring proper disambiguation in data stream manifests. Introduced the `inputRef` function to determine the correct identifier for inputs based on their name or type. Enhanced the `FindInput` method in `PolicyTemplate` to accommodate this change, allowing for more flexible input resolution. Added tests to validate the new behavior with dual input packages.
…release version - Added a temporary replace directive in go.mod for github.com/elastic/package-spec/v3 to use a pre-release version until the official v3.6.1 is released. - Updated go.sum to reflect the new dependency version and its associated module information.
… v3.6.1 Removed temporary replace directive for github.com/elastic/package-spec/v3 and updated go.sum to reflect the stable version 3.6.1. This change ensures the project uses the official release instead of a pre-release version.
… to ^9.4.0 Created new stack version files for composable packages, all set to version 9.5.0-SNAPSHOT. Updated the Kibana version requirement in the manifest files of existing packages to ^9.4.0 for compatibility.
name qualifiers and stream input refs (package-spec #1135)name qualifiers and stream input refs
…iable setup Updated the test-composable-packages.sh script to include environment variable exports for PACKAGE_TEST_TYPE and PACKAGE_UNDER_TEST. This allows for better configuration of stack version and provider settings during testing, improving the flexibility of the testing process.
…bootstrap inputs Updated the test-composable-packages.sh script to build and install multiple bootstrap input packages in a defined order. This change enhances the testing process by ensuring that all necessary input packages are handled correctly before and after the stack is set up, improving the overall reliability of the composable package testing workflow.
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579 |
| for i := range pt.Inputs { | ||
| input := &pt.Inputs[i] | ||
| if (input.Name != "" && input.Name == identifier) || input.Type == identifier { | ||
| return input | ||
| } | ||
| } |
There was a problem hiding this comment.
Q: Should this loop duplicated to first look for input.Name and if it does not find any , then look for input.Type ?
Is it ok to search at the same time for both ?
There was a problem hiding this comment.
as defiened on spec https://github.com/elastic/package-spec/blob/61cee23beb811b6c93ddb71caf822914ee30afba/code/go/internal/validator/semantic/validate_integration_input_qualifier.go#L68 the propoerty name is required when inputs share the same type. so i think is safe to make this comparison, so when name is found, its due to the fact that type is the same. when name is not found (empty) then we can trust type
| if [[ -z "${PACKAGE_UNDER_TEST:-}" ]]; then | ||
| export PACKAGE_UNDER_TEST="${COMPOSABLE_STACK_PIN_PACKAGE:-${COMPOSABLE_INPUT_PKG}}" | ||
| fi |
There was a problem hiding this comment.
As this test script is intended to test all packages at the same Buildkite step, I was wondering that it could be set directly in the elastic-package stack up command the required verion.
Helpers for doing this task, just ignore the targets that do not set PACKAGE_UNDER_TEST
stack_version_args() {
if [[ -z "${PACKAGE_UNDER_TEST:-""}" ]]; then
# Don't force stack version if we are testing multiple packages.
return
fi
...
It would be needed to add a TODO note to remove that version once the default one in elastic-package is the right one (or create an issue for that ?)
There was a problem hiding this comment.
i've added the version flag to the command directly on the script. does it make more sense?
There was a problem hiding this comment.
Yes, let's keep it as that for now.
There was a problem hiding this comment.
Given how the Buildkite step works for this new test step, just one file should be needed because there is just one Elastic stack started.
Given that, I was wondering if it could be just pinned in the shellscript adding a TODO to remove it (or maybe an issue). WDYT?
| Enabled: enabled, | ||
| Streams: streams, | ||
| inputType: streamInput, | ||
| inputType: input.Type, // always the real agent type, not the name qualifier |
There was a problem hiding this comment.
Q: So then here, would be "metrics", "logs", "traces" or "otelcol"?
There was a problem hiding this comment.
these are the agent input types, not the data stream types, added a comment for clarification
… definitions for access logs and updating policy files to include data stream structure. Introduced expected output files for both access and stubstatus data streams to improve testing coverage.
… package testing Modified the Makefile to set PACKAGE_TEST_TYPE for the test-build-install-packages-composable target. Updated the test-composable-packages.sh script to clarify the usage of PACKAGE_TEST_TYPE and ensure proper environment variable handling during testing.
Eliminated the FindInputByType method from the PolicyTemplate struct to streamline the codebase. This change simplifies the input retrieval process, as the method was not utilized in the current implementation.
…le-packages.sh Deleted obsolete stack version files for composable packages, which were all set to 9.5.0-SNAPSHOT. Updated the test-composable-packages.sh script to streamline the handling of PACKAGE_UNDER_TEST, ensuring better clarity and functionality in the testing process.
…rationPackagePolicy Updated comments for the inputType field in PackagePolicyInput and the BuildIntegrationPackagePolicy function to provide clearer explanations regarding the Fleet agent input type, distinguishing it from name qualifiers and data stream types.
… configurations Introduced a new stack version file set to 9.5.0-SNAPSHOT. Updated the configuration files for access and stubstatus data streams to include a data_stream structure, enhancing the organization and clarity of variable definitions.
…e input refs Source manifests for composable integrations carry unresolved `package:` references in stream/input definitions; RequiredInputsResolver only materialises these during the build step. Using source manifests in CreatePackagePolicy produced empty inputRef keys (e.g. "nginx-") that Fleet rejects with "input not found". Refactor CreatePackagePolicy to accept policy template and data stream names instead of pre-loaded source manifests, and always read from the built package tree via builder.BuildPackagesDirectory. Both input-type and integration packages now go through the built bundle, removing the source/built inconsistency and ensuring the same resolved input keys Fleet sees. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nifest reading This change adds the ReadBuiltPackageManifest function to encapsulate the logic for locating and reading the built package manifest. It simplifies the code in CreatePackagePolicy and related functions by reducing redundancy and improving clarity. The new function handles errors and returns both the built root path and the parsed manifest, ensuring consistent access to package data across the codebase.
…bstatus This commit introduces new sample event JSON files for the nginx composable access and stubstatus data streams. The access sample event includes log attributes and a message format, while the stubstatus sample event captures metrics related to NGINX connections. Additionally, updates to expected policy files and configuration files reflect changes in dataset naming conventions and minimum hit counts for assertions.
…y templates This update enhances the input resolution process in the addPackagePolicy function by streamlining error handling and ensuring that the correct input type is assigned from the built package manifest. The changes include clearer error messages and checks for input availability, improving overall robustness and maintainability.
| # Prefer the elastic-package binary built in this repo (see Makefile `build`) so | ||
| # composable install uses the same code as the checkout, not an older global install. | ||
| if [[ -x "${REPO_ROOT}/elastic-package" ]]; then | ||
| PATH="${REPO_ROOT}:${PATH}" | ||
| fi |
There was a problem hiding this comment.
This should not be needed. elastic-package is installed here
And it should be available already in the PATH.
Here it is the installation in the Buildkite build: https://buildkite.com/elastic/elastic-package/builds/7852#019dd956-4795-49ef-b207-d1d2b25e1155/L436
| stack_args=$(set +x; stack_version_args) | ||
| # TODO: Remove --version below when composable tests pass on elastic-package's default stack | ||
| # (Fleet/Kibana no longer require this snapshot). | ||
| stack_args="--version 9.5.0-SNAPSHOT" |
There was a problem hiding this comment.
This should be tried with 9.4.0-SNAPSHOT before merging if possible.
I haven't checked if the required changes are available in the latest Snapshot of 9.4.0
| if builtRoot, builtPkg, berr := builder.ReadBuiltPackageManifest(manifestRoot); berr == nil { | ||
| if builtPT, berr := packages.SelectPolicyTemplateByName(builtPkg.PolicyTemplates, *polName); berr == nil { | ||
| inputName := config.Input | ||
| if inputName == "" { | ||
| if builtDS, berr := packages.ReadDataStreamManifestFromPackageRoot(builtRoot, dsMan.Name); berr == nil && len(builtDS.Streams) > 0 { | ||
| inputName = builtDS.Streams[0].Input | ||
| } | ||
| } |
There was a problem hiding this comment.
Errors from these functions are not validated. Should they be raised if any?
| for bootstrap_name in "${COMPOSABLE_BOOTSTRAP_PKGS[@]}"; do | ||
| echo "--- Building bootstrap input package: ${COMPOSABLE_PACKAGES_PATH}/${bootstrap_name}" | ||
| # Bootstrap packages have no (or satisfied) requires.input; no registry yet. | ||
| # After build, artifacts land in build/packages/ for the stack local registry. | ||
| elastic-package build -C "${COMPOSABLE_PACKAGES_PATH}/${bootstrap_name}" -v | ||
| done |
There was a problem hiding this comment.
Not required for this PR, but just thought that there could be added support for some kind of hooks for the packages tested under parallel or false positives could take advantage in case they need to run specific commands.
There could be some hooks:
- before and after building the package
- before and after installing the package
- before and after testing the package
AFAIK there is no need for that now, so just commenting here this idea.
There was a problem hiding this comment.
i am not sure i understand what you mean with hook here. you mean that before building a packagae we could have defined commands running? can you extend with an example?
There was a problem hiding this comment.
In the main script (scripts/test-check-packages.sh), it could be added some code like:
for d in test/packages/${PACKAGE_TEST_TYPE}/${PACKAGE_UNDER_TEST}/; do
echo "--- Checking package ${d}"
if [ -f "$d/pre-check.sh" ]; then
bash "$d/pre-check.sh"
fi
elastic-package check -C "$d" -v
if [ -f "$d/post-check.sh" ]; then
bash "$d/post-check.sh"
fi
done echo "--- Run tests for package ${d}"
if [ -f "$d/pre-test.sh" ]; then
bash "$d/pre-test.sh"
fi
# Run all tests
# defer-cleanup is set to a short period to verify that the option is available
elastic-package test -C "$d" -v \
--report-format xUnit \
--report-output file \
--defer-cleanup 1s \
--test-coverage \
--coverage-format=generic
if [ -f "$d/post-test.sh" ]; then
bash "$d/post-test.sh"
fiNot sure if this could be helpful in the future, to try to allow customize even further the scenario for each package.
As I mentioned, just an idea that it came to my mind, no need to do it now.
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579 |
…-SNAPSHOT and remove obsolete nginx_composable.stack_version file
## Summary Found while working on composable packages elastic/elastic-package#3480 When an integration is bundled with inputs (composable packages) API reference should use the effective name instead of the input type. Effective name is introduced when two inputs share the same type, so they can be idetified. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### How to test Tested using the nginx_composable text fixture under [elastic-package ](elastic/elastic-package#3480) ### Screenshots Before <img width="1368" height="610" alt="Screenshot 2026-04-30 at 09 35 31" src="https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2" /> After <img width="1134" height="738" alt="Screenshot 2026-05-04 at 14 45 42" src="https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34" />
jsoriano
left a comment
There was a problem hiding this comment.
Tested and seems to work great, thanks!
I have a couple of concerns on the code:
- There is some complexity on deciding when the source or the built manifests need to be used. In most testers I think we should always use built packages, and thus we can assume that we have the build manifests available.
- [Nit] For the qualifier name/type we use "effective input name" in Kibana, and "input ref" here. If possible it would be great to use the same terms to avoid confusion.
| if r.testFolder.DataStream == "" { | ||
| return false | ||
| } | ||
| builtRoot, builtPkg, err := builder.ReadBuiltPackageManifest(r.packageRoot) |
There was a problem hiding this comment.
This method already receives a manifest. Do callers need to be able to work both with source and built manifests?
If this is always working with built packages callers could use always built manifests so we don't need to read it again here.
There was a problem hiding this comment.
this is not trivial change, for static tests we need both package source and built.
There was a problem hiding this comment.
It would be good to check why static tests need both, we can leave this for a future refactor.
|
|
||
| // Always resolve against the built tree so that RequiredInputsResolver has already | ||
| // materialized package: references into concrete input types. | ||
| builtRoot, builtPkg, err := builder.ReadBuiltPackageManifest(packageRoot) |
There was a problem hiding this comment.
We are reading and using the built package manifest in several places. I would prefer to read it only once and then pass it.
My main concern is not about reusing the read manifest, but about having to decide every time what manifest should be used.
As this is a system tester, we can probably use always the built manifest, as a built package should be available.
There was a problem hiding this comment.
i think this change is related to a test structure refactor. we needed to introduced the built bundle to work with composables, can we aford this debt and address the refactor as a followup? i can open an issue for it
There was a problem hiding this comment.
Definitely, we can do this in a follow up.
| ts.Check(decoratedWith("finding policy template", err)) | ||
|
|
||
| policy, dsType, dsDataset, err := system.CreatePackagePolicy(installed.testingPolicy, pkgMan, templ, dsMan, config.Input, config.Vars, config.DataStream.Vars, installed.testingPolicy.Namespace, manifestRoot) | ||
| // For composable integration packages the source manifest has unresolved "package:" references, |
There was a problem hiding this comment.
We should probably test always with built packages to avoid this complexity.
There was a problem hiding this comment.
Ok, let's do it in a follow up.
mrodm
left a comment
There was a problem hiding this comment.
LGTM!
I've also tested locally and the nginx_composable built package looks good too 💪
| builtPkg, err := packages.ReadPackageManifestFromPackageRoot(builtRoot) | ||
| if err != nil { | ||
| return kibana.PackagePolicy{}, "", "", fmt.Errorf("reading built package manifest at %s: %w", builtRoot, err) | ||
| } |
There was a problem hiding this comment.
When the package is of type "integration" this call would produced the same builtPkg value as in https://github.com/elastic/elastic-package/pull/3480/changes#diff-0bdd696fb9abd5e4aa12a3bab634cbac31ec1a1349b3a61e8a784a6bdeb6e7d8R2272-R2275
Would it be valid to set as parameter builtPkg instead of builtRoot? Would it be equivalent ?
| } | ||
|
|
||
| upsertKey(streamNode, "input", strVal(info.identifier)) | ||
| upsertKey(streamNode, "input", strVal(streamInputRefs[stream.Package])) |
There was a problem hiding this comment.
Should it be ensured that stream.Package is present in streamInputRefs to return some error before doing this call ? As it has been done for infoByPkg at line 195.
… (#267703) # Backport This will backport the following commits from `main` to `9.4`: - [Fix composable inputs on API reference integration docs (#267472)](#267472) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Tere","email":"teresa.romero@elastic.co"},"sourceCommit":{"committedDate":"2026-05-05T08:44:26Z","message":"Fix composable inputs on API reference integration docs (#267472)\n\n## Summary\n\nFound while working on composable packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen an integration is bundled with inputs (composable packages) API\nreference should use the effective name instead of the input type.\nEffective name is introduced when two inputs share the same type, so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n### How to test\n\nTested using the nginx_composable text fixture under [elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n### Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\" alt=\"Screenshot 2026-04-30 at 09 35 31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45 42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:version","v9.4.0","v9.5.0"],"title":"Fix composable inputs on API reference integration docs ","number":267472,"url":"https://github.com/elastic/kibana/pull/267472","mergeCommit":{"message":"Fix composable inputs on API reference integration docs (#267472)\n\n## Summary\n\nFound while working on composable packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen an integration is bundled with inputs (composable packages) API\nreference should use the effective name instead of the input type.\nEffective name is introduced when two inputs share the same type, so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n### How to test\n\nTested using the nginx_composable text fixture under [elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n### Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\" alt=\"Screenshot 2026-04-30 at 09 35 31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45 42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab"}},"sourceBranch":"main","suggestedTargetBranches":["9.4"],"targetPullRequestStates":[{"branch":"9.4","label":"v9.4.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/267472","number":267472,"mergeCommit":{"message":"Fix composable inputs on API reference integration docs (#267472)\n\n## Summary\n\nFound while working on composable packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen an integration is bundled with inputs (composable packages) API\nreference should use the effective name instead of the input type.\nEffective name is introduced when two inputs share the same type, so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n\n### How to test\n\nTested using the nginx_composable text fixture under [elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n### Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\" alt=\"Screenshot 2026-04-30 at 09 35 31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45 42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab"}}]}] BACKPORT--> Co-authored-by: Tere <teresa.romero@elastic.co>
Updated the input resolution logic in BuildIntegrationPackagePolicy and FindInput functions to utilize effective names instead of references. This change enhances clarity and consistency in identifying inputs across data stream manifests. Additionally, adjusted related tests to reflect the new naming convention.
Updated the inputPkgInfo struct to eliminate the pkgName field, simplifying the data structure used for input package metadata. Adjusted related test cases to align with this change, ensuring consistency in input handling across the codebase.
Refactored the inputPkgInfo structure and related functions to replace the identifier field with effectiveName, enhancing clarity in input package metadata. Adjusted test cases to ensure consistency with the new naming convention across the codebase.
Updated the inputPkgInfo structure to inputPolicyTemplateInfo, replacing effectiveName with input for clarity. Adjusted related functions and tests to ensure consistency in input resolution across policy templates, enhancing the overall structure and readability of the codebase.
Updated the comments in the buildStreamInputRefs function to provide a clearer explanation of how input references are constructed for required input packages. The new documentation outlines the handling of duplicate input types within policy templates and the significance of stable qualifiers, improving understanding of the input resolution process.
…putTypesToDataStreamManifests function Updated the applyInputTypesToDataStreamManifests function to include a check for the existence of stream input references. If a reference is not found for a given package, an informative error message is returned, enhancing the robustness of the input handling process.
…lt to improve data stream handling Updated the CreatePackagePolicy function to read data stream manifests and all data streams directly from the built package root. Refactored the buildIntegrationPackagePolicyFromBuilt function to accept the built package manifest, data stream manifest, policy template, and all data streams as parameters, enhancing clarity and error handling in policy creation.
jsoriano
left a comment
There was a problem hiding this comment.
👍 ready to go, and we can address the requested refactors in follow ups.
|
Opening followup issue to address test refactor #3530 ; |
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579 |
💛 Build succeeded, but was flaky
Failed CI Steps
History
|
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579 |
Why
Composable packages with
requires.inputcan have multiple required inputs of the sametype. package-spec#1135 (3.6.0+) requires distinctpolicy_templates[].inputs[].nameandstreams[].inputthat reference that name—not a single derived id for both policy and streams. elastic-package previously did not match that pattern, which caused ambiguous Fleet policies and errors (e.g. missing stream templates).This branch also builds policies from the built package manifest (so composable
package:refs resolve for Fleet), adds CI/fixtures (nginx composable, dual same-type inputs, bootstrap input pkgs), and includesdata_stream.typefor input-package system/policy tests (#3509) plus docs for Kibana/Fleet version behavior.What changed
internal/requiredinputs/internal/kibana/internal/packages/internal/resources/fleetpolicy: Stable inputnamewhen types collide;typestays the agent type; streams useinputRef(name or type);PackagePolicyInput.inputTypeis the Fleet input type;FindInputmatches name or type. Policy built from built tree (ReadBuiltPackageManifest, etc.).test/packages/composable/andtest/packages/parallel/nginx_composable/.How to verify
go build ./.../go test ./...(see AGENTS.md for narrower scopes). Composable: Makefile +scripts/test-composable-packages.sh. Example:go run . test -C ./test/packages/parallel/nginx_composableon a stack with the Fleet fixes you need.go test ./internal/kibana/... ./internal/testrunner/...for golden/policy coverage.Related
Closes Composable
requires.input: set inputnameand streaminputper package-spec qualified inputs (#1135) #3465#3465 / package-spec#1135
Fleet: kibana#264366; nginx/system discussion: kibana#266101
#3509; Fleet vars / legacy API: kibana#214216, kibana#266321
Depends on
Stack: Snapshots (or pins) that include the Fleet fixes your integration tests assume—align CI stack version with what you document.
data_stream.type: Usepolicy_api_format: legacywhere the simplified API still rejects the var (kibana#266321); see howto docs.