[Privmon] Synchronize Privileged User index based on configured Entity Sources

[CAWilson94]

Summary

This PR introduces support for populating the Privileged Users index from custom data sources (Elasticsearch indices) by querying Privileged User Monitoring Saved Objects from a Kibana task.

Related Issue
Figma Breakdown Working from the "Synchronize Privileged User index based on configured Entity Sources" section, to the right.

✅ Implemented

• Registered the temporary type and mappings
• Created a Descriptor Client that saves it via soClient.create()
• Called Descriptor Client create method from PrivMon initialisation.

• Testing above default Saved Object creation within PrivMon Initialisation
• Update the entity_analytics:monitoring:privileges:engine task to:
  • Read all Saved objects of index type
  • query user.name values from given indexPattern
  • Apply any filters or matchers defined in the Saved Object
  • Insert matched user.names into .entity_analytics.monitoring.users- with source type index.
  • Cleanup logic:
  • Target pattern does not exist, log warning and continue task
  • Failures will be caught and do not interrupt task loop
  • Remove previously stored user.names with source type index, that are no longer present.
  • Component Testing

🚧 Wish List

• Pull out sync functions from privilege monitoring data client into their own sync service class / similar
• • Currently in the data client, they should not be on this layer.
• Update GET and DELETE methods with dynamic id's for monitoring data client as per this ticket

Follow-up Issues:

• https://github.com/elastic/security-team/issues/12887
• https://github.com/elastic/security-team/issues/12851 (May be changed based on 🅿️ discussion on dynamic ID's)

How to Test

• Pull branch into local machine
• Security experimental flag enable: 'privilegeMonitoringEnabled'
• Start up ElasticSearch and Kibana

1. Optional - create the default index, this should just skip if you don't make it.

POST entity_analytics.privileged_monitoring/_doc
{
  "user": {
    "name": "default name"
  }
}

2. Create test index/ indicies

POST tatooine-/_bulk
{ "index": {} }
{ "user": { "name": "Luke Skywalker" } }
{ "index": {} }
{ "user": { "name": "Leia Organa" } }
{ "index": {} }
{ "user": { "name": "Han Solo" } }
{ "index": {} }
{ "user": { "name": "Chewbacca" } }
{ "index": {} }
{ "user": { "name": "Obi-Wan Kenobi" } }
{ "index": {} }
{ "user": { "name": "Yoda" } }
{ "index": {} }
{ "user": { "name": "R2-D2" } }
{ "index": {} }
{ "user": { "name": "C-3PO" } }
{ "index": {} }
{ "user": { "name": "Darth Vader" } }

3. Register Monitoring Entity Source Saved Objects

POST kbn:/api/entity_analytics/monitoring/entity_source
{
  "type": "index",
  "name": "StarWars",
  "managed": true,
  "indexPattern": "tatooine-",
  "enabled": true,
 "matchers": [
    {
      "fields": ["user.role"],
      "values": ["admin"]
    }
  ],
  "filter": {}
}

- OPTIONAL: You can check what is in the monitoring entity_source SO:

GET kbn:/api/entity_analytics/monitoring/entity_source/list

4. Initialise monitoring engine:

POST kbn:/api/entity_analytics/monitoring/engine/init {}

5. Verify Users in Monitoring Index

• Check the list of synced user, should include:
• • The created users
• • The default user (if you created it)

GET kbn:/api/entity_analytics/monitoring/users/list

e.g. output:

[
 {
   "id": "FkMJoZcB7muj1aiwb_eQ",
   "user": {
     "name": "C-3PO",
     "is_privileged": true
   },
   "labels": {
     "sources": [
       "index"
     ],
     "source_indices": [
       "tatooine-"
     ],
   }
 },
 {
   "id": "F0MJoZcB7muj1aiwb_eQ",
   "user": {
     "name": "Chewbacca", 
      "is_privileged": true
   },
   "labels": {
     "sources": [
       "index"
     ],
     "source_indices": [
       "tatooine-"
     ],     
   }
 },
// ... more here 

Testing: Removing Stale Users

The engine should soft delete users from the internal index if they no longer appear in the synced sources - e.g. label as monitoring.privileged_users: "not_monitored"
Example:

• Delete users in index:

POST tatooine-/_delete_by_query
{
 "query": {
   "terms": {
     "user.name.keyword": ["Chewbacca", "Han Solo"]
   }
 }
}

• re-run engine init

POST kbn:/api/entity_analytics/monitoring/engine/init
{}

-Fetch the updated user list:

GET kbn:/api/entity_analytics/monitoring/users/list

You should now only see both Chewbacca and Han Solo are no longer privileged:

// ..
{
   "id": "GUMJoZcB7muj1aiwb_eQ",
   "user": {
     "name": "Han Solo"
     "is_privileged": false 
   },
   "labels": {
     "sources": [],
     "source_indices": [],
   }
 }
]

[machadoum]

getAllUsernamesFromIndex does much more than its name suggests. A ' get `function should only retrieve data. This one adds data to an index.

[machadoum]

I would suggest splitting it into 2 functions

[CAWilson94]

Added a comment onto this for now - will split into two after merge

[tiansivive]

🚢
Thanks for all the work! This was quite the epic

[tiansivive]

???

Have you gone back and checked? looks like a really puzzling issue

[CAWilson94]

ty for nudge, lost it somewhere. BUT classically I don't know why it didn't work and I don't know why it does work now - probably some typo somewhere down the line. All good now 🎉

[CAWilson94]

Actually, previous index makes more sense. Struggling to find reference for this in the spec, will bring up with @hop-dev when back, its easy to change after which is ok.

[machadoum]

The code looks ok, we can do the improvement in a follow-up PR due to the time constraintsThe code seems acceptable; improvements can be made in a follow-up PR due to time constraints.

I haven't tested it locally.

Thank you for handling this complex feature! 👏 👏 👏

[CAWilson94]

The code looks ok, we can do the improvement in a follow-up PR due to the time constraintsThe code seems acceptable; improvements can be made in a follow-up PR due to time constraints.

I haven't tested it locally.

Thank you for handling this complex feature! 👏 👏 👏

Added a ticket here: https://github.com/elastic/security-team/issues/12887 thanks @machadoum and @tiansivive for having a look and giving feedback 🚀

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e254da2

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	625	627	+2

Total ESLint disabled count

id	before	after	diff
securitySolution	723	725	+2

History

• 💔 Build #311985 failed d11a704
• 💔 Build #311935 failed e548366
• 💔 Build #311908 failed b352566
• 💔 Build #311504 failed ff5f7be
• 💔 Build #311466 failed 128b716
• 💔 Build #311361 failed 838f94b

cc @CAWilson94