Skip to content

[streams][content pack] archive format and portable dashboards#217288

Merged
klacabane merged 75 commits intoelastic:mainfrom
klacabane:content-pack-archive-format
Apr 22, 2025
Merged

[streams][content pack] archive format and portable dashboards#217288
klacabane merged 75 commits intoelastic:mainfrom
klacabane:content-pack-archive-format

Conversation

@klacabane
Copy link
Contributor

@klacabane klacabane commented Apr 7, 2025 •
edited by kibanamachine
Loading

Summary

Allows one to export and import content packs in archive format. The format follows the integration content package's format so it becomes possible to import existing integration packages.

Content packs only support dashboard assets at the moment.
A pattern replacement logic has been implemented for dashboards and referenced data views:

  • at export time, any pattern matching the source stream will be replaced with a placeholder. Other patterns will remain as-is unless user explicitly ask to replace them
  • at import time, the placeholders are replaced with the target stream pattern

For example, if a dashboard is first exported from stream logs.nodejs and reads data from patterns logs.nodejs and logs.nodejs.prod, the patterns will be updated to logs.ruby and logs.ruby.prod when imported into logs.ruby stream.

The relevant UI components are hidden behind a feature flag, set the following in kibana.dev.yml to enable them:
feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled: true

export-import-replace-esql-panels.mov

klacabane and others added 26 commits April 7, 2025 11:22
@klacabane
content pack archive
28e0686
@klacabane
export func
9b76d5c
@klacabane
install fleet managed index
3595b4e
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/notice'
e5b900b
@kibanamachine
[CI] Auto-commit changed files from 'ts-node .buildkite/pipeline-reso…
59eb390 
…urce-definitions/scripts/fix-location-collection.ts'
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/capture_oas_snapsho…
d1ed14d 
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --update'
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
2597303 
…-fix'
@klacabane
content-packs-schema package
8dbab64
@klacabane
UI components and read archive in browser
a911ab3
@klacabane
Merge branch 'main' into content-pack-archive-format
d1693ae
@klacabane
fix merge
f34b1b8
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/build_plugin_list_d…
0ae3935 
…ocs'
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/capture_oas_snapsho…
cdb2356 
…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --update'
@klacabane
pattern replacement
3e517c4
@klacabane
fix replacements logic
a8097af
@klacabane
remove dependencies
571a09c
@klacabane
Merge branch 'main' into content-pack-archive-format
ddc68ba
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/notice'
ced569c
@kibanamachine
[CI] Auto-commit changed files from 'ts-node .buildkite/pipeline-reso…
519bbf3 
…urce-definitions/scripts/fix-location-collection.ts'
@kibanamachine
[CI] Auto-commit changed files from 'make api-docs'
b04dae6
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
3a2386d 
…-fix'
@klacabane
preview endpoint instead of browser parsing
2e08897
@klacabane
remove jszip
4ea52ee
@klacabane
bootstrap
329a890
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'
45a8aae
@tommyers-elastic
add feature flag checks for content pack UI elements
e007f25
@tommyers-elastic tommyers-elastic force-pushed the content-pack-archive-format branch from 0d5ca1a to e007f25 Compare April 8, 2025 11:43
@tommyers-elastic tommyers-elastic force-pushed the content-pack-archive-format branch from 9b27e08 to 58d0285 Compare April 8, 2025 11:48
@kc13greiner kc13greiner self-requested a review April 16, 2025 13:01
@kc13greiner
Copy link
Contributor

kc13greiner commented Apr 16, 2025

Heya @klacabane ! I am trying to test this, but I cant seem to get the Streams nav option to appear, I probably have my local configured wrong.

Is feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled: true all I need to add to my dev yml?

@klacabane
Copy link
Contributor Author

klacabane commented Apr 17, 2025

@kc13greiner streams app is disabled by default, you'll need to enable it with POST kbn:/api/streams/_enable. The feature flag in the summary is only to enable the feature implemented in the change. Feel free to reach out on slack

kc13greiner
kc13greiner approved these changes Apr 17, 2025
View reviewed changes
Copy link
Contributor

@kc13greiner kc13greiner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the changes and info ➕

@tommyers-elastic tommyers-elastic self-requested a review April 17, 2025 13:23
gsoldevila
gsoldevila approved these changes Apr 17, 2025
View reviewed changes
Copy link
Contributor

@gsoldevila gsoldevila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, new file with test data added to the KBN archiver fixtures

@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 18, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #16 / ObservableActionsPopoverButton clicking the button opens the popover

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
cloudSecurityPosture 688 689 +1
dashboard 651 652 +1
data 536 537 +1
dataVisualizer 736 737 +1
discover 1206 1207 +1
esql 191 192 +1
esqlDataGrid 420 421 +1
eventAnnotationListing 592 593 +1
infra 1413 1414 +1
lens 1365 1366 +1
lists 412 413 +1
maps 1257 1258 +1
ml 2382 2383 +1
observability 1289 1290 +1
observabilityAIAssistantApp 437 438 +1
securitySolution 7264 7265 +1
slo 1117 1118 +1
stackAlerts 162 163 +1
streamsApp 437 473 +36
unifiedHistogram 197 198 +1
unifiedSearch 350 351 +1
total +56

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/content-packs-schema - 25 +25
@kbn/esql-utils 92 95 +3
@kbn/streams-schema 379 376 -3
total +25

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
streamsApp 397.0KB 409.6KB +12.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
streamsApp 9.7KB 9.9KB +156.0B
Unknown metric groups

API count

id before after diff
@kbn/content-packs-schema - 25 +25
@kbn/esql-utils 105 108 +3
@kbn/streams-schema 394 391 -3
total +25

History

@kdelemme kdelemme added v8.19.0 and removed v8.9.0 labels Apr 21, 2025
@klacabane klacabane merged commit 6a56939 into elastic:main Apr 22, 2025
12 checks passed
@kibanamachine
Copy link
Contributor

kibanamachine commented Apr 22, 2025

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/14590258341

@kibanamachine
Copy link
Contributor

kibanamachine commented Apr 22, 2025

💔 All backports failed

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 217288

Questions ?

Please refer to the Backport tool documentation

@klacabane klacabane mentioned this pull request Apr 22, 2025
Merged
@klacabane
Copy link
Contributor Author

klacabane commented Apr 22, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

klacabane added a commit to klacabane/kibana that referenced this pull request Apr 22, 2025
@klacabane
[streams][content pack] archive format and portable dashboards (elast…
c90d094 
…ic#217288)

## Summary

Allows one to export and import content packs in archive format. The
format follows the integration content package's format so it becomes
possible to import existing integration packages.

Content packs only support dashboard assets at the moment.
A pattern replacement logic has been implemented for dashboards and
referenced data views:
- at export time, any pattern matching the source stream will be
replaced with a placeholder. Other patterns will remain as-is unless
user explicitly ask to replace them
- at import time, the placeholders are replaced with the target stream
pattern

For example, if a dashboard is first exported from stream `logs.nodejs`
and reads data from patterns `logs.nodejs` and `logs.nodejs.prod`, the
patterns will be updated to `logs.ruby` and `logs.ruby.prod` when
imported into `logs.ruby` stream.

The relevant UI components are hidden behind a feature flag, set the
following in `kibana.dev.yml` to enable them:
`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled: true`

https://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
(cherry picked from commit 6a56939)

# Conflicts:
#	.github/CODEOWNERS
#	oas_docs/bundle.json
#	oas_docs/output/kibana.yaml
This was referenced Apr 22, 2025
Merged
Merged
klacabane added a commit that referenced this pull request Apr 23, 2025
@klacabane @kibanamachine
[8.19] [streams][content pack] archive format and portable dashboards (…
ff193c6 
…#217288) (#218870)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[streams][content pack] archive format and portable dashboards
(#217288)](#217288)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Kevin
Lacabane","email":"kevin.lacabane@elastic.co"},"sourceCommit":{"committedDate":"2025-04-22T08:27:52Z","message":"[streams][content
pack] archive format and portable dashboards (#217288)\n\n##
Summary\n\nAllows one to export and import content packs in archive
format. The\nformat follows the integration content package's format so
it becomes\npossible to import existing integration packages.\n\nContent
packs only support dashboard assets at the moment.\nA pattern
replacement logic has been implemented for dashboards and\nreferenced
data views:\n- at export time, any pattern matching the source stream
will be\nreplaced with a placeholder. Other patterns will remain as-is
unless\nuser explicitly ask to replace them\n- at import time, the
placeholders are replaced with the target stream\npattern\n\nFor
example, if a dashboard is first exported from stream `logs.nodejs`\nand
reads data from patterns `logs.nodejs` and `logs.nodejs.prod`,
the\npatterns will be updated to `logs.ruby` and `logs.ruby.prod`
when\nimported into `logs.ruby` stream.\n\nThe relevant UI components
are hidden behind a feature flag, set the\nfollowing in `kibana.dev.yml`
to enable
them:\n`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled:
true`\n\n\n\nhttps://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
tommyers-elastic
<106530686+tommyers-elastic@users.noreply.github.com>","sha":"6a569398acd21ceefaa5cb62fb77f1df67cd63a3","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:obs-entities","backport:version","Feature:Streams","v9.1.0","v8.19.0"],"title":"[streams][content
pack] archive format and portable
dashboards","number":217288,"url":"https://github.com/elastic/kibana/pull/217288","mergeCommit":{"message":"[streams][content
pack] archive format and portable dashboards (#217288)\n\n##
Summary\n\nAllows one to export and import content packs in archive
format. The\nformat follows the integration content package's format so
it becomes\npossible to import existing integration packages.\n\nContent
packs only support dashboard assets at the moment.\nA pattern
replacement logic has been implemented for dashboards and\nreferenced
data views:\n- at export time, any pattern matching the source stream
will be\nreplaced with a placeholder. Other patterns will remain as-is
unless\nuser explicitly ask to replace them\n- at import time, the
placeholders are replaced with the target stream\npattern\n\nFor
example, if a dashboard is first exported from stream `logs.nodejs`\nand
reads data from patterns `logs.nodejs` and `logs.nodejs.prod`,
the\npatterns will be updated to `logs.ruby` and `logs.ruby.prod`
when\nimported into `logs.ruby` stream.\n\nThe relevant UI components
are hidden behind a feature flag, set the\nfollowing in `kibana.dev.yml`
to enable
them:\n`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled:
true`\n\n\n\nhttps://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
tommyers-elastic
<106530686+tommyers-elastic@users.noreply.github.com>","sha":"6a569398acd21ceefaa5cb62fb77f1df67cd63a3"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/217288","number":217288,"mergeCommit":{"message":"[streams][content
pack] archive format and portable dashboards (#217288)\n\n##
Summary\n\nAllows one to export and import content packs in archive
format. The\nformat follows the integration content package's format so
it becomes\npossible to import existing integration packages.\n\nContent
packs only support dashboard assets at the moment.\nA pattern
replacement logic has been implemented for dashboards and\nreferenced
data views:\n- at export time, any pattern matching the source stream
will be\nreplaced with a placeholder. Other patterns will remain as-is
unless\nuser explicitly ask to replace them\n- at import time, the
placeholders are replaced with the target stream\npattern\n\nFor
example, if a dashboard is first exported from stream `logs.nodejs`\nand
reads data from patterns `logs.nodejs` and `logs.nodejs.prod`,
the\npatterns will be updated to `logs.ruby` and `logs.ruby.prod`
when\nimported into `logs.ruby` stream.\n\nThe relevant UI components
are hidden behind a feature flag, set the\nfollowing in `kibana.dev.yml`
to enable
them:\n`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled:
true`\n\n\n\nhttps://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
tommyers-elastic
<106530686+tommyers-elastic@users.noreply.github.com>","sha":"6a569398acd21ceefaa5cb62fb77f1df67cd63a3"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
@klacabane @kibanamachine
@tommyers-elastic @akowalska622
[streams][content pack] archive format and portable dashboards (elast…
b50d95e 
…ic#217288)

## Summary

Allows one to export and import content packs in archive format. The
format follows the integration content package's format so it becomes
possible to import existing integration packages.

Content packs only support dashboard assets at the moment.
A pattern replacement logic has been implemented for dashboards and
referenced data views:
- at export time, any pattern matching the source stream will be
replaced with a placeholder. Other patterns will remain as-is unless
user explicitly ask to replace them
- at import time, the placeholders are replaced with the target stream
pattern

For example, if a dashboard is first exported from stream `logs.nodejs`
and reads data from patterns `logs.nodejs` and `logs.nodejs.prod`, the
patterns will be updated to `logs.ruby` and `logs.ruby.prod` when
imported into `logs.ruby` stream.

The relevant UI components are hidden behind a feature flag, set the
following in `kibana.dev.yml` to enable them:
`feature_flags.overrides.featureFlagsStreams.contentPackUIEnabled: true`



https://github.com/user-attachments/assets/9fb07daf-9fb9-4c62-9f5b-387e1833eaf0

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: tommyers-elastic <106530686+tommyers-elastic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flash1293 flash1293 flash1293 left review comments

@stratoula stratoula stratoula left review comments

@vadimkibana vadimkibana vadimkibana left review comments

Copilot code review Copilot Copilot left review comments

@drewdaemon drewdaemon drewdaemon approved these changes

@pheyos pheyos pheyos approved these changes

@miltonhultgren miltonhultgren miltonhultgren approved these changes

@kc13greiner kc13greiner kc13greiner approved these changes

@gsoldevila gsoldevila gsoldevila approved these changes

@tommyers-elastic tommyers-elastic tommyers-elastic approved these changes

Assignees

No one assigned

Labels

backport:version Backport to applied version labels Feature:Streams This is the label for the Streams Project release_note:skip Skip the PR/issue when compiling release notes Team:obs-entities DEPRECATED - Observability Entities Team v8.19.0 v9.1.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[streams][content pack] find and replace dashboard index patterns [streams][content pack] ui to import and export a content pack