[Security Solution] update endpoint list api to support united index

[joeypoon]

Summary

metadata list api will now first attempt to query the united metadata index to save on extra fleet-agent queries.
if the index is not found, it will fallback to the existing query.
this PR is backwards compatible and will work even without endpoint package updates.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[joeypoon]

adding an optional path to the fields since we copy .fleet-agents mapping but are additionally nesting it

[joeypoon]

use the endpoint package version to determine which index the kql bar should use for auto completion

[joeypoon]

existing logic was just moved into this method with no change in logic.

[joeypoon]

this is the new query to the new united index

[joeypoon]

do we need to support HostDetails... like the legacy query here? if this is being written by endpoint I'm guessing it's possible they can run an older version which still uses HostDetails?

[elasticmachine]

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

[joeypoon]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

Fleet changes 🚀 (these strings interpolation seems a little fragile to me and we should probably move to another pattern like a query builder or other one day to have something more robust here, at least we have some functional tests that cover these queries)

[kevinlog]

@joeypoon - awesome job on this so far.

I check it out, tried it, and found a few issues. Let me know if you have any questions.

Here are the cases I tested.

Original/backwards compatible mode

• Enroll a single real Agent with Endpoint, Endpoint shows in Endpoint list ✅
• Unenroll Agent, Endpoint is removed from Endpoint list ✅
• Enroll Agent with Endpoint, Endpoint appears in list, remove Endpoint integration ONLY, Endpoint is removed from list ✅
• Generate 100 docs with the generator, all docs appear in list ✅

New transforms

• Enroll a single real Agent with Endpoint, Endpoint shows in Endpoint list ✅
• Unenroll Agent, Endpoint is removed from Endpoint list ❌
  • The Endpoint record remained in the list while the Agent was removed from the Agents list
• Enroll Agent with Endpoint, Endpoint appears in list, remove Endpoint integration ONLY, Endpoint is removed from list ✅
  • I tried this scenario a few times. I added the Endpoint integration to the Agent policy and deleted it a handful of times. It worked as expected each time
• Generate 100 docs with the generator, all docs appear in list ❌ ❓
  • I saw some odd behavior on this one. I could see the docs coming into the destination indices, but the Endpoint list didn't render correctly. I think the generator may need to be updated? There are some screenshots below of the scenario.

You can see pagination and "Showing" number, but it doesn't render.

I can see the docs in the indices, so may it has to do with the shape of them?

[joeypoon]

thanks for checking @kevinlog

• Unenroll Agent, Endpoint is removed from Endpoint list ❌

latest push should fix that one and also fixes the issue with generated docs kind of showing up in the list (they don't show up at all now, this is expected since generated docs don't currently merge in the endpoint metadata into the doc).

[joeypoon]

endpoint generator also works with united index with latest commit 👌

[pzl]

🐑 ?

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: c8e6451
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	2224	2225	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	4.2MB	4.2MB	+231.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	132.4KB	132.3KB	-15.0B
securitySolution	108.4KB	108.4KB	+80.0B
total			+65.0B

History

• 💔 Build #155506 failed 6730ea8a7180508c7d6217c3561ced669bf6bc67
• 💔 Build #155501 failed 7fea80600d02ac554a00165335146297a3cc70c5
• 💚 Build #155084 succeeded 822942ddc8658a83c182f897f4cc63a5d72df496
• 💔 Build #154997 failed 1ee59e2e1b5d3e72c99d4dd48a31179992780b20
• 💔 Build #154790 failed 0f95c98a8b68f5298564735f01b11942416b306b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.