Misleading SIEM Warning When Attaching Custom Roles With Valid But Complex RegEx

[wathian]

Describe the bug:
Observed the following SIEM warning when login as a user with custom roles (Refer to video link below)

It seems like the warning is shown because the role's index pattern is unable to view the indices, where in reality, it did permit the indices which made us believed it's a misleading visual bug. (Refer to video link below)

If possible, we would like the team to confirm whether said warnings are impeding the functionality of the rules or that it's actually a bug. Our suspicion is the latter where we think it is only doing a naive check and it does not actually impedes the functionality since it has sufficient privilege as shown in the video.

Kibana/Elasticsearch Stack version:
Elastic Cloud SaaS v8.12.2

Steps to reproduce:

1. Create the user and attach the roles: siem warnings filebeat params.txt
2. Create an SIEM rule with said user that targets filebeat-* indices
3. Mock the data in order to trigger the SIEM alerts

Current behavior:
Misleading warnings are shown despite the rule "working"

Expected behavior:
SIEM Rule should raise alerts properly
Warnings are not shown when the RBAC is sufficient
No functionality of the rules are affected.

Screenshots (if relevant):
Video Link: https://youtu.be/8A8Du31NwEs

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[yctercero]

Hi @wathian ! Thanks for the post. Could you attach the video on the ticket? I unfortunately can't access the link.

[wathian]

Hi @yctercero, I've set the video to unlisted visibility, hopefully you can see it now.

[louisong]

@yctercero See if you are able to access the video here?
https://www.youtube.com/watch?v=8A8Du31NwEs

[dhurley14]

Hi @wathian

I noticed in your video you have the role stackopsdevsyste_Admin and rcr_stackopsdevsyste_elasticnonprodmo assigned to the user you are testing with. However, the role rcr_stackopsdevsyste_elasticnonprodmo definition does not have read access to the filebeat index. I pulled down the text file you attached and the role rcr_stackopsdevsyste_elasticnonprodmo only has access to the following indices [".monitoring-*", "metricbeat-*", "*elastic-cloud-logs*"]

Can you try updating this role or removing it from the user you test with and then re-enable the rule? The warning should disappear after re-enabling it.

cc: @yctercero @louisong

[tianlang8158]

Hi @dhurley14 ,

I am following up on the issue.

User has two roles assigned, rcr_stackopsdevsyste_elasticnonprodmo is only for cross-cluster read permission.
The permission for local index pattern filebeat-* is granted in stackopsdevsyste_Admin role using regex:

/@&~((filebeat-|partial-)?(\.ds-)?(\.monitoring-.*|metricbeat-.*|elastic-cloud-.*|filebeat-ess-.*))/

I have verified by login to the user with the two roles, I am able to read index with pattern filebeat-*, but when I created SIEM Rule include this pattern, the error This rule may not have the required read privileges to the following index patterns: ["filebeat-*"] is thrown.

Are you saying both roles need to have explicit access to filebeat-* pattern?

Regards
Chen

[dhurley14]

Hi @tianlang8158 and @wathian ,

My apologies I just saw the customer support thread related to this issue. I have one question - is the objective to block access to the filebeat index or to allow access to it? I'm guessing block access but want to double check.

From what I can tell, this looks like there might be a discrepancy with how elasticsearch compares user privileges for data streams. I used auditbeat instead of filebeat for testing but I believe the issue is still the same.

The code that renders the warning you reference in the video checks the _has_privileges api using the index patterns defined on the rule as the values for the read permission check:

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/utils.ts

Lines 178 to 198 in 2664ccf

	export const checkPrivilegesFromEsClient = async (
	esClient: ElasticsearchClient,
	indices: string[]
	): Promise<Privilege> =>
	withSecuritySpan(
	'checkPrivilegesFromEsClient',
	async () =>
	(await esClient.transport.request({
	path: '/_security/user/_has_privileges',
	method: 'POST',
	body: {
	index: [
	{
	names: indices ?? [],
	allow_restricted_indices: true,
	privileges: ['read'],
	},
	],
	},
	})) as Privilege
	);

Replicating this in the Kibana Dev Tools I get the following:

# this works, and returns values
GET auditbeat-*/_search

The above request is what the security rule executes. And just like you saw in your deployment there are alerts generated / the search yields results. This is because when I check privileges for the auditbeat datastream I have read access.

# This says I have read privileges
POST /_security/user/_has_privileges
{
  "index": [
    {
      "names": ".ds-auditbeat-8.13.2-2024.10.21",
      "allow_restricted_indices": true,
      "privileges": [
        "read"
      ]
    }
  ]
}

# RESPONSE:
{
  "username": "testuser",
  "has_all_requested": true,
  "cluster": {},
  "index": {
    ".ds-auditbeat-8.13.2-2024.10.21": {
      "read": true
    }
  },
  "application": {}
}

Now when I check for read privileges on the auditbeat-* alias I have none, which reflects the warning on the rule details page

POST /_security/user/_has_privileges
{
  "index": [
    {
      "names": "auditbeat-*",
      "allow_restricted_indices": true,
      "privileges": [
        "read"
      ]
    }
  ]
}


# RESPONSE:
{
  "username": "testuser",
  "has_all_requested": false,
  "cluster": {},
  "index": {
    "auditbeat-*": {
      "read": false
    }
  },
  "application": {}
}

This definitely is confusing. To remedy this would require a change in how elasticsearch checks data stream privileges. And again, I just want to confirm that I'm understanding correctly, you would like to block read access to filebeat with this regex, right?

[dhurley14]

After playing around with the provided regex I believe I have figured out the issue.

The warning on the rule's details page regarding filebeat-* is accurate because in the original regex:

/@&~((filebeat-|partial-)?(\.ds-)?(\.monitoring-.*|metricbeat-.*|elastic-cloud-.*|filebeat-ess-.*))/

The section /@&~((filebeat-| will not match filebeat-*, which is the case when we query _has_privileges with filebeat-* and read we get false (see my comment above.)

The reason we are seeing the data still come back is because the data stream part of the regular expression does not cover .ds-filebeat-*

Give this, I came up with the following which I believe is the desired outcome

/@&~((\.ds-.*)?(filebeat-.*|partial-||\.monitoring-.*|metricbeat-.*|elastic-cloud-.*|filebeat-ess-.*))/

Note the additional .* after the \.ds- section and after the filebeat- section.

This will prevent the rule from querying any filebeat data stream, which will match with what the error message said. I hope this covers your use case!

[tianlang8158]

Hi @dhurley14 ,

The original pattern /@&~((filebeat-|partial-)?(\.ds-)?(\.monitoring-.*|metricbeat-.*|elastic-cloud-.*|filebeat-ess-.*))/ aims to:

MATCH ANY BUT MONITORING DATA which is to allow all indices while block below indice pattern:

filebeat-ess-*
filebeat-.ds-.monitoring-*
filebeat-.ds-metricbeat-*
filebeat-.ds-elastic-cloud-*
filebeat-.ds-filebeat-ess-*
filebeat-.monitoring-*
filebeat-metricbeat-*
filebeat-elastic-cloud-*
filebeat-filebeat-ess-*
partial-.ds-.monitoring-*
partial-.ds-metricbeat-*
partial-.ds-elastic-cloud-*
partial-.ds-filebeat-ess-*
partial-.monitoring-*
partial-metricbeat-*
partial-elastic-cloud-*
partial-filebeat-ess-*
.ds-.monitoring-*
.ds-metricbeat-*
.ds-elastic-cloud-*
.ds-filebeat-ess-*
.monitoring-*
metricbeat-*
elastic-cloud-*
filebeat-ess-*

With your new pattern /@&~((\.ds-.*)?(filebeat-.*|partial-||\.monitoring-.*|metricbeat-.*|elastic-cloud-.*|filebeat-ess-.*))/, the warning message disappeared for new SIEM rule with query pattern filebeat-*. However, at same time, it also denied access to other normal filebeat indices such as filebeat-endpoint-test.

For your advice please.

[electra06]

Hi @louisong @dhurley14 , following up on this, we have related tickets pending for this issue, please see the last comment from my colleague @tianlang8158 . Would appreciate it if you can provide an update. Thank you.

[dhurley14]

Hello @electra06 👋

Apologies for the delay. Thank you for clarifying which exact indices you wish to exclude. That is helpful. Knowing that you are NOT trying to exclude all filebeat-* indices is helpful as that was not clear to me earlier.

As I mentioned in my earlier comment the yellow warning banner on the rule details page is correct due to the reasons I outlined in the above comment:

The section /@&~((filebeat-| will not match filebeat-*, which is the case when the security solution rule runs a query on the _has_privileges api with filebeat-* and read we get false

Given this above information I believe that warning on the rule details page is not a bug and is not preventing the rule from running correctly, which I believe was confirmed by the rule generating correct alerts.

If you know that the regex is performing as expected then I think we can close out this ticket. I believe you can double check this though by testing out the original regular expression using the scripts I posted in the above comment within Kibana Dev Tools.

[tianlang8158]

Hello @dhurley14,

Yes the yellow warning banner is not preventing the rule from running correctly, however, it is misleading to our users which we received a lot of feedback on this, and its not just filebeat-* indices, same goes to other index pattern that is not explicitly defined such as logs-aws.cloudtrail-*.

Below screenshot is from tenant which all AWS custom rules hit the same warning.

Is there a better way to achieve the same without this warning?

Regards
Chen