Add new field migrate_from to handle inputs migration by criamico · Pull Request #1021 · elastic/package-spec

criamico · 2025-11-18T10:45:10Z

What does this PR do?

Add a new migrate_from field that can be used to auto-migrate equivalent inputs or streams on Fleet side. An example is migrating httpjson to cel. The field can be added at two levels:

Example 1 - policy template

- policy_templates:
	- name: sample
	  title: Sample logs  
	  description: Collect sample data  
	  inputs:  
		- type: cel
		  migrate_from: logfile

Example 2 - streams

- streams:
	- input: filestream
	  title: "Collect logs via filestream"  
	  description: "Collect logs using the filestream input"  
	  template_path: filestream.yml.hbs  
	  migrate_from: logfile  
	  vars:  
		- name: paths
		  type: text  
		  title: "Paths"  
		  multi: true  
		  required: true  
		  show_user: true  
		  default:  
			- /var/log/*.log

Please note: for some integrations this migration path will require a change at elastic agent level, so package authors should consider carefully if it's safe to use it. See comment

Why is it important?

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

Relates [Change Proposal] Support migration to equivalent inputs #956
Kibana change: [Fleet] Migrate inputs when migrate_from field is found kibana#242934

teresaromero · 2026-02-25T11:14:16Z

spec/changelog.yml

    - description: Allow otelcol input type and dynamic_signal_types field in integration packages.
      type: enhancement
      link: https://github.com/elastic/package-spec/pull/1091
+    - description: Add definition for migrate_from field.


nit:

Suggested change

- description: Add definition for migrate_from field.

- description: Add support for migrating input types at integrations policy templates and data streams.

teresaromero · 2026-02-25T11:16:11Z

spec/integration/manifest.spec.yml

                    - ["default"]
                    - ["agentless"]
                    - ["default", "agentless"]
+                migrate_from:


can we add this to a definition at the integrations spec and use the ref in both places?

teresaromero · 2026-02-25T11:17:20Z

code/go/pkg/validator/validator_test.go

+		"good_migrate_from":                   {},
+		"bad_migrate_from": {
+			"manifest.yml",
+			[]string{`field policy_templates.0.inputs.0: Additional property migrate_from is not allowed`},


other than being present as a property, should the text be validated with some pattern?

That's a good question. I didn't add any because it could match very different cases. For example, it could be an input type identifiers likelogfile, filestream, httpjson, cel, winlog, udp, tcp, aws/metrics, but I didn't find a definition of these anywhere in this repo. Do you have suggestions on how to match them? I thought of a regex but I don't know it it's the right choice.

I found (via AI) the field is validated in runtime by elastic-agent, as values come from different sources. So if the value is not valid an error will rise at elastic-agent. I got cursor to go through all the input types on the current packages and come with a common regex:

Proposed Regex

^[a-z0-9][a-z0-9_\-]*(/[a-z0-9][a-z0-9_\-]*)*$

Explanation:
[a-z0-9] — must start with alphanumeric (lowercase)
[a-z0-9_-]* — segment continues with alphanumeric, underscore, or hyphen
(/[a-z0-9][a-z0-9_-]) — optional additional segments after / (e.g., provider/type)

Matches: filestream, azure-eventhub, cloudbeat/cis_k8s, log_file, prometheus/metrics
Rejects: empty string, Azure-EventHub (uppercase), //double, leading/trailing slashes

After learning this, i think is better to leave it open and delegate this validation (format/value) to the elastic-agent where there is the "source of truth", as introducing the regex could cause some false negatives... 👍🏻 wdyt?

yes, I also got this regex but thought that is not the best way. It's basically a generic string validation, so it wouldn't add much value.

teresaromero · 2026-02-25T11:17:57Z

spec/changelog.yml

    - description: Allow otelcol input type and dynamic_signal_types field in integration packages.
      type: enhancement
      link: https://github.com/elastic/package-spec/pull/1091
+    - description: Add definition for migrate_from field.


if this depends on fleet/kibana PR, can we add it as a comment

coderabbitai · 2026-03-05T09:21:24Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 104a0d44-5356-4b2d-bb45-99d9dd695a4a

📥 Commits

Reviewing files that changed from the base of the PR and between b094881 and 100de75.

📒 Files selected for processing (1)

test/packages/bad_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml

📝 Walkthrough

Walkthrough

Adds a temporary migrate_from property to integration schemas and manifests, with patches that remove the property in version 3.6.0. Updates spec/integration/manifest.spec.yml and spec/integration/data_stream/manifest.spec.yml to reference a migrate_from definition and to add/remove it in the versioned patches. Introduces two test packages (good_migrate_from, bad_migrate_from) and extends validator tests with "good_migrate_from" and "bad_migrate_from" cases. Adds a changelog entry documenting the migrate_from wiring. No public API signatures were changed.

Possibly related PRs

Adjust property names for package dependencies to original proposal #1102: Modifies validator test cases and error expectations related to schema validation, aligning with the migrate_from test additions in this change.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@test/packages/bad_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml`:
- Around line 4-7: The ingest pipeline step with tag "set_dataset" erroneously
sets field "data_stream.dataset" to "good_migrate_from.logs"; update that value
to "bad_migrate_from.logs" so this package routes documents to its own
dataset/index (locate the "set" block with tag: set_dataset and change the value
property accordingly).

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f68e4bbc-b2a4-4f0e-af3c-044fa27e21ea

📥 Commits

Reviewing files that changed from the base of the PR and between 5f23052 and b094881.

⛔ Files ignored due to path filters (4)

test/packages/bad_migrate_from/img/sample-logo.svg is excluded by !**/*.svg
test/packages/bad_migrate_from/img/sample-screenshot.png is excluded by !**/*.png
test/packages/good_migrate_from/img/sample-logo.svg is excluded by !**/*.svg
test/packages/good_migrate_from/img/sample-screenshot.png is excluded by !**/*.png

📒 Files selected for processing (24)

code/go/pkg/validator/validator_test.go
spec/changelog.yml
spec/integration/data_stream/manifest.spec.yml
spec/integration/manifest.spec.yml
test/packages/bad_migrate_from/LICENSE.txt
test/packages/bad_migrate_from/_dev/build/docs/README.md
test/packages/bad_migrate_from/changelog.yml
test/packages/bad_migrate_from/data_stream/logs/agent/stream/filestream.yml.hbs
test/packages/bad_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
test/packages/bad_migrate_from/data_stream/logs/fields/base-fields.yml
test/packages/bad_migrate_from/data_stream/logs/manifest.yml
test/packages/bad_migrate_from/docs/README.md
test/packages/bad_migrate_from/manifest.yml
test/packages/bad_migrate_from/sample_event.json
test/packages/good_migrate_from/LICENSE.txt
test/packages/good_migrate_from/_dev/build/docs/README.md
test/packages/good_migrate_from/changelog.yml
test/packages/good_migrate_from/data_stream/logs/agent/stream/filestream.yml.hbs
test/packages/good_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
test/packages/good_migrate_from/data_stream/logs/fields/base-fields.yml
test/packages/good_migrate_from/data_stream/logs/manifest.yml
test/packages/good_migrate_from/docs/README.md
test/packages/good_migrate_from/manifest.yml
test/packages/good_migrate_from/sample_event.json

test/packages/bad_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml

elasticmachine · 2026-03-05T09:43:51Z

💚 Build Succeeded

Buildkite Build
Commit: 100de75

History

💚 Build #1891 succeeded 8b854f7
💚 Build #1889 succeeded 20cff11
💚 Build #1877 succeeded d8cad66
💚 Build #1604 succeeded cc64614
💚 Build #1603 succeeded e3faca8

cc @criamico

jsoriano

Thanks!

jsoriano · 2026-03-05T11:15:22Z

spec/integration/manifest.spec.yml

+      - op: remove
+        path: "/properties/policy_templates/items/properties/inputs/items/properties/migrate_from"
+      - op: remove
+        path: "/definitions/migrate_from"


Nit. This should go on top. Though this is relevant when removing elements from lists, that we are not doing on this version. So in this case it doesn't matter, and can stay here.

Related to elastic/package-spec#956 Package-spec PR: elastic/package-spec#1021 ## Summary Migrate inputs and vars when `migrate_from` field is found. **Please note:** for some integrations this migration path will require a change at elastic agent level. See elastic/package-spec#956 (comment) - Enable the new input and carries over its values from the old one - Inputs and variables that were enabled are now carried to the new version; however the state is maintained: if a variable is disabled, it stays disabled - Modified the secret handling so the existing secret references get carried over to the new inputs - Verified that this logic only run on package policy upgrade, when there is migrate_from present and Fleet detects a difference in input type - Add visual information about the migration Example with a test integration: https://github.com/user-attachments/assets/aaa56904-05d7-47b2-955b-5ae7a2c26c97 ### Testing This was tested with a test version of sentinelOne (not finalized yet): https://github.com/elastic/integrations/compare/main...kcreddy:sentinel_one-migrate-inputs?expand=1 ### Screenshots <img width="1641" height="1540" alt="Screenshot 2026-03-19 at 17 07 17" src="https://github.com/user-attachments/assets/819e46cd-823a-47df-8ef2-4fbce8900b53" /> <img width="1628" height="1524" alt="Screenshot 2026-03-19 at 17 07 43" src="https://github.com/user-attachments/assets/c5f97183-bcfd-496a-9882-e170323246e3" /> ### Checklist Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

criamico added 2 commits November 18, 2025 11:43

Add new field migrate_from under policy_templates

e3faca8

Fix position of new variable and add changelog

cc64614

This was referenced Nov 20, 2025

[Fleet] Migrate inputs when migrate_from field is found elastic/kibana#242934

Merged

[Change Proposal] Support migration to equivalent inputs #956

Closed

criamico added 2 commits February 23, 2026 16:05

Merge main and fix conflicts

1014dd6

Add field to datastreams and add test packages

d8cad66

criamico changed the title ~~Add new field migrate_from under policy_templates~~ Add new field migrate_from to handle inputs migration Feb 23, 2026

criamico self-assigned this Feb 24, 2026

Improve descriptions

20cff11

criamico marked this pull request as ready for review February 25, 2026 11:09

criamico requested a review from a team as a code owner February 25, 2026 11:09

criamico requested a review from jsoriano February 25, 2026 11:11

teresaromero reviewed Feb 25, 2026

View reviewed changes

Address code review

8b854f7

teresaromero previously approved these changes Feb 27, 2026

View reviewed changes

Merge main and fix conflicts

b094881

criamico dismissed teresaromero’s stale review via b094881 March 5, 2026 09:15

coderabbitai bot reviewed Mar 5, 2026

View reviewed changes

test/packages/bad_migrate_from/data_stream/logs/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Code review comment

100de75

criamico requested a review from teresaromero March 5, 2026 10:04

jsoriano approved these changes Mar 5, 2026

View reviewed changes

jsoriano merged commit 9b2e7ee into elastic:main Mar 5, 2026
4 checks passed

	- description: Add definition for migrate_from field.
	- description: Add support for migrating input types at integrations policy templates and data streams.

Conversation

criamico commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Example 1 - policy template

Example 2 - streams

Why is it important?

Checklist

Related issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Possibly related PRs

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Mar 5, 2026

💚 Build Succeeded

History

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

criamico commented Nov 18, 2025 •

edited

Loading

coderabbitai bot commented Mar 5, 2026 •

edited

Loading