[streams][content pack] track installed content packs

[klacabane]

Summary

When importing a content pack into a stream, we now store an object to audit this import.

In addition to the content pack metadata that can now be surfaced, we also store metadata of the saved objects imported to remember the source object it was copied from. This allows updates of existing objects instead of creating a duplicate.
The stored metadata is restricted to the stream/content pack name on purpose as it's not clear yet how versioning will be handled for content pack and dashboards.

[elasticmachine]

Pinging @elastic/obs-entities (Team:obs-entities)

[flash1293]

I tested with this and it generally works fine for dashboards without references.

However for referenced saved objects I'm not sure whether the current behavior makes sense. I have a dashboard that has the following references (twice the same data view and once a library visualization):

   {
      "name": "6d1cb3a3-08c9-4427-a741-ecaa2b0d9149:panel_6d1cb3a3-08c9-4427-a741-ecaa2b0d9149",
      "type": "lens",
      "id": "5ca48846-63ea-446a-8f3f-17de9262d482"
    },
    {
      "type": "index-pattern",
      "id": "6fd421f2-547c-4d9a-b946-62e078631f91",
      "name": "8b18d450-266e-44f1-a520-037d0769e47f:indexpattern-datasource-layer-6d821e87-a35a-4404-968a-cca57c45c7c0"
    },
    {
      "type": "index-pattern",
      "id": "6fd421f2-547c-4d9a-b946-62e078631f91",
      "name": "6d1cb3a3-08c9-4427-a741-ecaa2b0d9149:indexpattern-datasource-layer-a18791e3-bd58-4483-bdd5-ffd15b625311"
    }

The library visualization is just ignored as part of the content pack, which works as intended AFAIK (if the library vis is there, it works fine, if not, the panel won't render correctly).

For the referenced data view however the following is generated in the content pack:

"references": [
                {
                  "source_id": "5ca48846-63ea-446a-8f3f-17de9262d482",
                  "target_id": "0ec88155-c330-4e61-b351-99e3c61cd1e8"
                },
                {
                  "source_id": "6fd421f2-547c-4d9a-b946-62e078631f91",
                  "target_id": "e143357c-fecb-452f-b891-0a47b5ec038c"
                },
                {
                  "source_id": "6fd421f2-547c-4d9a-b946-62e078631f91",
                  "target_id": "c8402171-24cc-474c-a8fd-7203987a3b04"
                }
              ]

• The non-resolved vis library panel is listed here (does that make sense)?
• The same data view is listed twice - only one saved object is actually created (the first one)
  • Should it even create it or reference the existing data view that is already present?
  • Should it deduplicate so only one reference is tracked?

I think it would make sense to double-check the behavior of references and add a functional test for how we want it to behave. The case of having a single data view referenced multiple times in a dashboard is a very common one.

I also have some general questions, maybe this is planned to be done in follow-ups:

• The generated dashboards are not tagged/named that they come from a content pack - when installing the same one to multiple streams, this is quickly getting messy. Are we planning to do this?
• Updating a content pack can't remove assets from a stream - is this something that would make sense? Integrations have this ability, it seems to make sense here as well
• Should we somehow track whether the content pack assets are modified? That would be good to know when updating a content pack, otherwise local changes can be overwritten

[klacabane]

Thanks for the review @flash1293

The library visualization is just ignored as part of the content pack, which works as intended AFAIK (if the library vis is there, it works fine, if not, the panel won't render correctly).

Yes only dashboard and index-pattern are currently supported but all the code necessary to support lens exists, it just needs to be enabled. I'll include that change

The non-resolved vis library panel is listed here (does that make sense)?

Good catch, we should not create links for references that are not included in the content pack (fixed in 3974efe)

Should it even create it or reference the existing data view that is already present?

The import logic will by default duplicate any object referenced by the dashboard that is also included in the content pack as long as it is a supported type. This is because we assume there is a pattern replacement being done on the object so it has to be duplicated. We can improve the logic and duplicate only if there is a replacement on the object, but this raises questions:

• what if the source object does not exist in the cluster ? this can be fixed by running a validation step and only create if it is not persisted
• what if the source object exists but has a different version than the one included in the content pack - we could duplicate only if the versions are different, or always reference the source object. Keeping a reference to the source object allows free updates to the parent objects, but I don't know if this is sensible considering content packs could be forked indefinitely. Creating a copy on every import eventually generates unnecessary duplication, but allows safe and predictable updates to individual objects. imo references complicate things and Allow exporting self contained saved objects #218034 would be nice to have
• do we apply this logic to dashboard or references only ?

wdyt ?

Should it deduplicate so only one reference is tracked?

Absolutely, good catch (fixed in 218b23d)

The generated dashboards are not tagged/named that they come from a content pack - when installing the same one to multiple streams, this is quickly getting messy. Are we planning to do this?

Yes tagging/adding metadata for content pack objects is planned, there's no issue tracking this yet but I'll create one

Updating a content pack can't remove assets from a stream - is this something that would make sense? Integrations have this ability, it seems to make sense here as well

Agreed, we should leave a clean state corresponding to the latest version of the content pack imported. I'll create an issue for that

Should we somehow track whether the content pack assets are modified? That would be good to know when updating a content pack, otherwise local changes can be overwritten

Definitely. I've left out the version out of this initial change, but the content pack links should carry the version of the object that was installed so we can make smarter decisions when updating

[flash1293]

Thanks for your answers, @klacabane

In general this all makes sense to me, I think there are a couple downstream tasks we will need to do, but that's fine, they will become apparent once we built out these features.

Especially the question with duplicating vs. reusing saved objects is important I think - maybe we can leverage the content pack tracking information also in the dashboarding listing UI and collapse the same dashboards linked with different streams? Just an idea, I think that can come later.

I'm approving, but I would love to have another review from @tommyers-elastic on this

[klacabane]

Added lens support in f4293ff

Created #219556 to track updates of content pack

Created #219550 to tag content pack assets

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 340dc82

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
streamsApp	462	466	+4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/content-packs-schema	25	45	+20

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	416.4KB	421.0KB	+4.6KB

Unknown metric groups

API count

id	before	after	diff
@kbn/content-packs-schema	25	45	+20

History

• 💛 Build #297577 was flaky 40879ec
• 💛 Build #296679 was flaky d67a123
• 💔 Build #296469 failed 218b23d
• 💚 Build #295158 succeeded 29a6c4e
• 💔 Build #295061 failed 2ae4c8d
• 💔 Build #295016 failed 58abbae

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/14901975111

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation