Skip to content

Composable packages: input name qualifiers and stream input refs#3480

Merged
teresaromero merged 37 commits intoelastic:mainfrom
teresaromero:composable/input-name
May 7, 2026
Merged

Composable packages: input name qualifiers and stream input refs#3480
teresaromero merged 37 commits intoelastic:mainfrom
teresaromero:composable/input-name

Conversation

@teresaromero
Copy link
Copy Markdown
Contributor

@teresaromero teresaromero commented Apr 20, 2026

Why

Composable packages with requires.input can have multiple required inputs of the same type. package-spec#1135 (3.6.0+) requires distinct policy_templates[].inputs[].name and streams[].input that reference that name—not a single derived id for both policy and streams. elastic-package previously did not match that pattern, which caused ambiguous Fleet policies and errors (e.g. missing stream templates).

This branch also builds policies from the built package manifest (so composable package: refs resolve for Fleet), adds CI/fixtures (nginx composable, dual same-type inputs, bootstrap input pkgs), and includes data_stream.type for input-package system/policy tests (#3509) plus docs for Kibana/Fleet version behavior.

What changed

  • internal/requiredinputs / internal/kibana / internal/packages / internal/resources/fleetpolicy: Stable input name when types collide; type stays the agent type; streams use inputRef (name or type); PackagePolicyInput.inputType is the Fleet input type; FindInput matches name or type. Policy built from built tree (ReadBuiltPackageManifest, etc.).
  • Test runners & scripts: Policy/system/script updates for composable flows, overrides, and data stream naming; composable Makefile/script and packages under test/packages/composable/ and test/packages/parallel/nginx_composable/.

How to verify

go build ./... / go test ./... (see AGENTS.md for narrower scopes). Composable: Makefile + scripts/test-composable-packages.sh. Example: go run . test -C ./test/packages/parallel/nginx_composable on a stack with the Fleet fixes you need. go test ./internal/kibana/... ./internal/testrunner/... for golden/policy coverage.

Related

Depends on

Stack: Snapshots (or pins) that include the Fleet fixes your integration tests assume—align CI stack version with what you document. data_stream.type: Use policy_api_format: legacy where the simplified API still rejects the var (kibana#266321); see howto docs.

teresaromero and others added 7 commits April 20, 2026 14:31
…e-type inputs

When a composable policy template ends up with multiple required input packages
that share the same type (e.g. two otelcol inputs), set a unique name on each
input derived from the required package name so Fleet can disambiguate them
(package-spec SVR00010). Data stream manifests then reference inputs by name
instead of type, matching the Fleet/Kibana expectation from package-spec 3.6.0+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fiers

Updated the logic in `BuildIntegrationPackagePolicy` to utilize a unique name qualifier for inputs that share the same type, ensuring proper disambiguation in data stream manifests. Introduced the `inputRef` function to determine the correct identifier for inputs based on their name or type. Enhanced the `FindInput` method in `PolicyTemplate` to accommodate this change, allowing for more flexible input resolution. Added tests to validate the new behavior with dual input packages.
…release version

- Added a temporary replace directive in go.mod for github.com/elastic/package-spec/v3 to use a pre-release version until the official v3.6.1 is released.
- Updated go.sum to reflect the new dependency version and its associated module information.
… v3.6.1

Removed temporary replace directive for github.com/elastic/package-spec/v3 and updated go.sum to reflect the stable version 3.6.1. This change ensures the project uses the official release instead of a pre-release version.
… to ^9.4.0

Created new stack version files for composable packages, all set to version 9.5.0-SNAPSHOT. Updated the Kibana version requirement in the manifest files of existing packages to ^9.4.0 for compatibility.
@teresaromero teresaromero changed the title [Draft] Composable packages: input name qualifiers and stream input refs (package-spec #1135) Composable packages: input name qualifiers and stream input refs Apr 22, 2026
…iable setup

Updated the test-composable-packages.sh script to include environment variable exports for PACKAGE_TEST_TYPE and PACKAGE_UNDER_TEST. This allows for better configuration of stack version and provider settings during testing, improving the flexibility of the testing process.
…bootstrap inputs

Updated the test-composable-packages.sh script to build and install multiple bootstrap input packages in a defined order. This change enhances the testing process by ensuring that all necessary input packages are handled correctly before and after the stack is set up, improving the overall reliability of the composable package testing workflow.
@teresaromero teresaromero marked this pull request as ready for review April 22, 2026 09:33
@teresaromero
Copy link
Copy Markdown
Contributor Author

test integrations

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579

Comment on lines +860 to +865
for i := range pt.Inputs {
input := &pt.Inputs[i]
if (input.Name != "" && input.Name == identifier) || input.Type == identifier {
return input
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Should this loop duplicated to first look for input.Name and if it does not find any , then look for input.Type ?

Is it ok to search at the same time for both ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as defiened on spec https://github.com/elastic/package-spec/blob/61cee23beb811b6c93ddb71caf822914ee30afba/code/go/internal/validator/semantic/validate_integration_input_qualifier.go#L68 the propoerty name is required when inputs share the same type. so i think is safe to make this comparison, so when name is found, its due to the fact that type is the same. when name is not found (empty) then we can trust type

Comment thread scripts/test-composable-packages.sh Outdated
Comment thread scripts/test-composable-packages.sh Outdated
Comment on lines +90 to +92
if [[ -z "${PACKAGE_UNDER_TEST:-}" ]]; then
export PACKAGE_UNDER_TEST="${COMPOSABLE_STACK_PIN_PACKAGE:-${COMPOSABLE_INPUT_PKG}}"
fi
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this test script is intended to test all packages at the same Buildkite step, I was wondering that it could be set directly in the elastic-package stack up command the required verion.

Helpers for doing this task, just ignore the targets that do not set PACKAGE_UNDER_TEST

stack_version_args() {
  if [[ -z "${PACKAGE_UNDER_TEST:-""}" ]]; then
    # Don't force stack version if we are testing multiple packages.
    return
  fi
  ...

It would be needed to add a TODO note to remove that version once the default one in elastic-package is the right one (or create an issue for that ?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've added the version flag to the command directly on the script. does it make more sense?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, let's keep it as that for now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given how the Buildkite step works for this new test step, just one file should be needed because there is just one Elastic stack started.

Given that, I was wondering if it could be just pinned in the shellscript adding a TODO to remove it (or maybe an issue). WDYT?

Comment thread internal/packages/packages.go Outdated
Comment thread internal/kibana/packagepolicy.go Outdated
Enabled: enabled,
Streams: streams,
inputType: streamInput,
inputType: input.Type, // always the real agent type, not the name qualifier
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: So then here, would be "metrics", "logs", "traces" or "otelcol"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are the agent input types, not the data stream types, added a comment for clarification

Comment thread test/packages/composable/01_ci_input_pkg/manifest.yml
jsoriano and others added 14 commits April 22, 2026 13:10
… definitions for access logs and updating policy files to include data stream structure. Introduced expected output files for both access and stubstatus data streams to improve testing coverage.
… package testing

Modified the Makefile to set PACKAGE_TEST_TYPE for the test-build-install-packages-composable target. Updated the test-composable-packages.sh script to clarify the usage of PACKAGE_TEST_TYPE and ensure proper environment variable handling during testing.
Eliminated the FindInputByType method from the PolicyTemplate struct to streamline the codebase. This change simplifies the input retrieval process, as the method was not utilized in the current implementation.
…le-packages.sh

Deleted obsolete stack version files for composable packages, which were all set to 9.5.0-SNAPSHOT. Updated the test-composable-packages.sh script to streamline the handling of PACKAGE_UNDER_TEST, ensuring better clarity and functionality in the testing process.
…rationPackagePolicy

Updated comments for the inputType field in PackagePolicyInput and the BuildIntegrationPackagePolicy function to provide clearer explanations regarding the Fleet agent input type, distinguishing it from name qualifiers and data stream types.
… configurations

Introduced a new stack version file set to 9.5.0-SNAPSHOT. Updated the configuration files for access and stubstatus data streams to include a data_stream structure, enhancing the organization and clarity of variable definitions.
…e input refs

Source manifests for composable integrations carry unresolved `package:` references
in stream/input definitions; RequiredInputsResolver only materialises these during
the build step. Using source manifests in CreatePackagePolicy produced empty inputRef
keys (e.g. "nginx-") that Fleet rejects with "input not found".

Refactor CreatePackagePolicy to accept policy template and data stream names instead
of pre-loaded source manifests, and always read from the built package tree via
builder.BuildPackagesDirectory. Both input-type and integration packages now go
through the built bundle, removing the source/built inconsistency and ensuring the
same resolved input keys Fleet sees.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nifest reading

This change adds the ReadBuiltPackageManifest function to encapsulate the logic for locating and reading the built package manifest. It simplifies the code in CreatePackagePolicy and related functions by reducing redundancy and improving clarity. The new function handles errors and returns both the built root path and the parsed manifest, ensuring consistent access to package data across the codebase.
…bstatus

This commit introduces new sample event JSON files for the nginx composable access and stubstatus data streams. The access sample event includes log attributes and a message format, while the stubstatus sample event captures metrics related to NGINX connections. Additionally, updates to expected policy files and configuration files reflect changes in dataset naming conventions and minimum hit counts for assertions.
…y templates

This update enhances the input resolution process in the addPackagePolicy function by streamlining error handling and ensuring that the correct input type is assigned from the built package manifest. The changes include clearer error messages and checks for input availability, improving overall robustness and maintainability.
Comment thread scripts/test-composable-packages.sh Outdated
Comment on lines +19 to +23
# Prefer the elastic-package binary built in this repo (see Makefile `build`) so
# composable install uses the same code as the checkout, not an older global install.
if [[ -x "${REPO_ROOT}/elastic-package" ]]; then
PATH="${REPO_ROOT}:${PATH}"
fi
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be needed. elastic-package is installed here

And it should be available already in the PATH.

Here it is the installation in the Buildkite build: https://buildkite.com/elastic/elastic-package/builds/7852#019dd956-4795-49ef-b207-d1d2b25e1155/L436

Comment thread scripts/test-composable-packages.sh Outdated
stack_args=$(set +x; stack_version_args)
# TODO: Remove --version below when composable tests pass on elastic-package's default stack
# (Fleet/Kibana no longer require this snapshot).
stack_args="--version 9.5.0-SNAPSHOT"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be tried with 9.4.0-SNAPSHOT before merging if possible.
I haven't checked if the required changes are available in the latest Snapshot of 9.4.0

Comment thread test/packages/parallel/nginx_composable.stack_version
Comment on lines +128 to +135
if builtRoot, builtPkg, berr := builder.ReadBuiltPackageManifest(manifestRoot); berr == nil {
if builtPT, berr := packages.SelectPolicyTemplateByName(builtPkg.PolicyTemplates, *polName); berr == nil {
inputName := config.Input
if inputName == "" {
if builtDS, berr := packages.ReadDataStreamManifestFromPackageRoot(builtRoot, dsMan.Name); berr == nil && len(builtDS.Streams) > 0 {
inputName = builtDS.Streams[0].Input
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errors from these functions are not validated. Should they be raised if any?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

Comment on lines +75 to +80
for bootstrap_name in "${COMPOSABLE_BOOTSTRAP_PKGS[@]}"; do
echo "--- Building bootstrap input package: ${COMPOSABLE_PACKAGES_PATH}/${bootstrap_name}"
# Bootstrap packages have no (or satisfied) requires.input; no registry yet.
# After build, artifacts land in build/packages/ for the stack local registry.
elastic-package build -C "${COMPOSABLE_PACKAGES_PATH}/${bootstrap_name}" -v
done
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not required for this PR, but just thought that there could be added support for some kind of hooks for the packages tested under parallel or false positives could take advantage in case they need to run specific commands.

There could be some hooks:

  • before and after building the package
  • before and after installing the package
  • before and after testing the package

AFAIK there is no need for that now, so just commenting here this idea.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am not sure i understand what you mean with hook here. you mean that before building a packagae we could have defined commands running? can you extend with an example?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the main script (scripts/test-check-packages.sh), it could be added some code like:

for d in test/packages/${PACKAGE_TEST_TYPE}/${PACKAGE_UNDER_TEST}/; do
  echo "--- Checking package ${d}"
  if [ -f "$d/pre-check.sh" ]; then
    bash "$d/pre-check.sh"
  fi
  elastic-package check -C "$d" -v
  if [ -f "$d/post-check.sh" ]; then
    bash "$d/post-check.sh"
  fi
done
  echo "--- Run tests for package ${d}"
  if [ -f "$d/pre-test.sh" ]; then
    bash "$d/pre-test.sh"
  fi
  # Run all tests
  # defer-cleanup is set to a short period to verify that the option is available
  elastic-package test -C "$d" -v \
    --report-format xUnit \
    --report-output file \
    --defer-cleanup 1s \
    --test-coverage \
    --coverage-format=generic
  if [ -f "$d/post-test.sh" ]; then
    bash "$d/post-test.sh"
  fi

Not sure if this could be helpful in the future, to try to allow customize even further the scenario for each package.

As I mentioned, just an idea that it came to my mind, no need to do it now.

@teresaromero
Copy link
Copy Markdown
Contributor Author

test integrations

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579

teresaromero added a commit to elastic/kibana that referenced this pull request May 5, 2026
## Summary

Found while working on composable packages
elastic/elastic-package#3480

When an integration is bundled with inputs (composable packages) API
reference should use the effective name instead of the input type.
Effective name is introduced when two inputs share the same type, so
they can be idetified.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### How to test

Tested using the nginx_composable text fixture under [elastic-package
](elastic/elastic-package#3480)


### Screenshots

Before 
<img width="1368" height="610" alt="Screenshot 2026-04-30 at 09 35 31"
src="https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2"
/>

After
<img width="1134" height="738" alt="Screenshot 2026-05-04 at 14 45 42"
src="https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34"
/>
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and seems to work great, thanks!

I have a couple of concerns on the code:

  • There is some complexity on deciding when the source or the built manifests need to be used. In most testers I think we should always use built packages, and thus we can assume that we have the build manifests available.
  • [Nit] For the qualifier name/type we use "effective input name" in Kibana, and "input ref" here. If possible it would be great to use the same terms to avoid confusion.

Comment thread internal/kibana/packagepolicy.go Outdated
Comment thread internal/kibana/packagepolicy.go Outdated
Comment thread internal/packages/packages.go Outdated
Comment thread internal/requiredinputs/streamdefs.go Outdated
if r.testFolder.DataStream == "" {
return false
}
builtRoot, builtPkg, err := builder.ReadBuiltPackageManifest(r.packageRoot)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method already receives a manifest. Do callers need to be able to work both with source and built manifests?
If this is always working with built packages callers could use always built manifests so we don't need to read it again here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not trivial change, for static tests we need both package source and built.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to check why static tests need both, we can leave this for a future refactor.


// Always resolve against the built tree so that RequiredInputsResolver has already
// materialized package: references into concrete input types.
builtRoot, builtPkg, err := builder.ReadBuiltPackageManifest(packageRoot)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are reading and using the built package manifest in several places. I would prefer to read it only once and then pass it.
My main concern is not about reusing the read manifest, but about having to decide every time what manifest should be used.

As this is a system tester, we can probably use always the built manifest, as a built package should be available.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this change is related to a test structure refactor. we needed to introduced the built bundle to work with composables, can we aford this debt and address the refactor as a followup? i can open an issue for it

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, we can do this in a follow up.

Comment thread internal/testrunner/runners/system/tester_test.go
Comment thread internal/testrunner/runners/system/tester_test.go Outdated
ts.Check(decoratedWith("finding policy template", err))

policy, dsType, dsDataset, err := system.CreatePackagePolicy(installed.testingPolicy, pkgMan, templ, dsMan, config.Input, config.Vars, config.DataStream.Vars, installed.testingPolicy.Namespace, manifestRoot)
// For composable integration packages the source manifest has unresolved "package:" references,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably test always with built packages to avoid this complexity.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as here #3480 (comment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's do it in a follow up.

Copy link
Copy Markdown
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
I've also tested locally and the nginx_composable built package looks good too 💪

Comment thread internal/requiredinputs/streamdefs.go Outdated
builtPkg, err := packages.ReadPackageManifestFromPackageRoot(builtRoot)
if err != nil {
return kibana.PackagePolicy{}, "", "", fmt.Errorf("reading built package manifest at %s: %w", builtRoot, err)
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the package is of type "integration" this call would produced the same builtPkg value as in https://github.com/elastic/elastic-package/pull/3480/changes#diff-0bdd696fb9abd5e4aa12a3bab634cbac31ec1a1349b3a61e8a784a6bdeb6e7d8R2272-R2275

Would it be valid to set as parameter builtPkg instead of builtRoot? Would it be equivalent ?

Comment thread internal/requiredinputs/streamdefs.go Outdated
}

upsertKey(streamNode, "input", strVal(info.identifier))
upsertKey(streamNode, "input", strVal(streamInputRefs[stream.Package]))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be ensured that stream.Package is present in streamInputRefs to return some error before doing this call ? As it has been done for infoByPkg at line 195.

Comment thread internal/requiredinputs/streamdefs_test.go
kibanamachine added a commit to elastic/kibana that referenced this pull request May 5, 2026
… (#267703)

# Backport

This will backport the following commits from `main` to `9.4`:
- [Fix composable inputs on API reference integration docs
(#267472)](#267472)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT
[{"author":{"name":"Tere","email":"teresa.romero@elastic.co"},"sourceCommit":{"committedDate":"2026-05-05T08:44:26Z","message":"Fix
composable inputs on API reference integration docs (#267472)\n\n##
Summary\n\nFound while working on composable
packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen
an integration is bundled with inputs (composable packages)
API\nreference should use the effective name instead of the input
type.\nEffective name is introduced when two inputs share the same type,
so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies
following conditions. \n\nReviewers should verify this PR satisfies this
list as well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*` labels.\n\n### How to test\n\nTested using
the nginx_composable text fixture under
[elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n###
Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\"
alt=\"Screenshot 2026-04-30 at 09 35
31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img
width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45
42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:version","v9.4.0","v9.5.0"],"title":"Fix
composable inputs on API reference integration docs
","number":267472,"url":"https://github.com/elastic/kibana/pull/267472","mergeCommit":{"message":"Fix
composable inputs on API reference integration docs (#267472)\n\n##
Summary\n\nFound while working on composable
packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen
an integration is bundled with inputs (composable packages)
API\nreference should use the effective name instead of the input
type.\nEffective name is introduced when two inputs share the same type,
so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies
following conditions. \n\nReviewers should verify this PR satisfies this
list as well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*` labels.\n\n### How to test\n\nTested using
the nginx_composable text fixture under
[elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n###
Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\"
alt=\"Screenshot 2026-04-30 at 09 35
31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img
width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45
42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab"}},"sourceBranch":"main","suggestedTargetBranches":["9.4"],"targetPullRequestStates":[{"branch":"9.4","label":"v9.4.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/267472","number":267472,"mergeCommit":{"message":"Fix
composable inputs on API reference integration docs (#267472)\n\n##
Summary\n\nFound while working on composable
packages\nhttps://github.com/elastic/elastic-package/pull/3480\n\nWhen
an integration is bundled with inputs (composable packages)
API\nreference should use the effective name instead of the input
type.\nEffective name is introduced when two inputs share the same type,
so\nthey can be idetified.\n\n### Checklist\n\nCheck the PR satisfies
following conditions. \n\nReviewers should verify this PR satisfies this
list as well.\n\n- [ ] Any text added follows [EUI's
writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\nsentence case text and includes
[i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n-
[
]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas
added for features that require explanation or tutorials\n- [ ] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [ ] If a plugin
configuration key changed, check if it needs to be\nallowlisted in the
cloud and added to the
[docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n-
[ ] This was checked for breaking HTTP API changes, and any
breaking\nchanges have been approved by the breaking-change committee.
The\n`release_note:breaking` label should be applied in these
situations.\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*` labels.\n\n### How to test\n\nTested using
the nginx_composable text fixture under
[elastic-package\n](https://github.com/elastic/elastic-package/pull/3480)\n\n\n###
Screenshots\n\nBefore \n<img width=\"1368\" height=\"610\"
alt=\"Screenshot 2026-04-30 at 09 35
31\"\nsrc=\"https://github.com/user-attachments/assets/e0ea3cba-ee2b-4ede-9b8d-30f413d336a2\"\n/>\n\nAfter\n<img
width=\"1134\" height=\"738\" alt=\"Screenshot 2026-05-04 at 14 45
42\"\nsrc=\"https://github.com/user-attachments/assets/7aecde20-4b0e-4384-8a61-bb1ea0c40d34\"\n/>","sha":"8c42329f33a15146e7919302a1cf6daa72a0d4ab"}}]}]
BACKPORT-->

Co-authored-by: Tere <teresa.romero@elastic.co>
Updated the input resolution logic in BuildIntegrationPackagePolicy and FindInput functions to utilize effective names instead of references. This change enhances clarity and consistency in identifying inputs across data stream manifests. Additionally, adjusted related tests to reflect the new naming convention.
Updated the inputPkgInfo struct to eliminate the pkgName field, simplifying the data structure used for input package metadata. Adjusted related test cases to align with this change, ensuring consistency in input handling across the codebase.
Refactored the inputPkgInfo structure and related functions to replace the identifier field with effectiveName, enhancing clarity in input package metadata. Adjusted test cases to ensure consistency with the new naming convention across the codebase.
Updated the inputPkgInfo structure to inputPolicyTemplateInfo, replacing effectiveName with input for clarity. Adjusted related functions and tests to ensure consistency in input resolution across policy templates, enhancing the overall structure and readability of the codebase.
Updated the comments in the buildStreamInputRefs function to provide a clearer explanation of how input references are constructed for required input packages. The new documentation outlines the handling of duplicate input types within policy templates and the significance of stable qualifiers, improving understanding of the input resolution process.
…putTypesToDataStreamManifests function

Updated the applyInputTypesToDataStreamManifests function to include a check for the existence of stream input references. If a reference is not found for a given package, an informative error message is returned, enhancing the robustness of the input handling process.
…lt to improve data stream handling

Updated the CreatePackagePolicy function to read data stream manifests and all data streams directly from the built package root. Refactored the buildIntegrationPackagePolicyFromBuilt function to accept the built package manifest, data stream manifest, policy template, and all data streams as parameters, enhancing clarity and error handling in policy creation.
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 ready to go, and we can address the requested refactors in follow ups.

@teresaromero
Copy link
Copy Markdown
Contributor Author

Opening followup issue to address test refactor #3530 ;

@teresaromero
Copy link
Copy Markdown
Contributor Author

test integrations

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579

@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented May 7, 2026

@teresaromero
Copy link
Copy Markdown
Contributor Author

test integrations

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Created or updated PR in integrations repository to test this version. Check elastic/integrations#18579

Copy link
Copy Markdown
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@teresaromero teresaromero merged commit 2ebd2bc into elastic:main May 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Composable requires.input: set input name and stream input per package-spec qualified inputs (#1135)

4 participants