Consider a redesign of the mutelist tags feature

[dlouzan]

New feature motivation

This is a bit of a follow-up / feedback of the discussions in a previous bug report #4782 around mutelists and tags behaviour.

After having implemented mutelists for 3 different AWS accounts in the last weeks, I have realized that when working with tag combinations, the syntax gets really messy and leads to easy errors.

The main problem stems from the behaviour of tags: they are ANDed, which is fine for some use cases, but for others, the only option to OR them is to create complex regex rules inside single yaml array elements.

As an example, this is the syntax I ended up using for the following use case: Filter out any failures on resources tagged with Name a.domain.com OR b.domain.com OR c.domain.com:

        "some_prowler_check":
          Regions:
            - "*"
          Resources:
            - "*"
          Tags:
            - "Name=(\
                a\\.domain\\.com|\
                b\\.domain\\.com|\
                c\\.domain\\.com\
              )"

The syntax above is combined with yaml flow scalar style ending each line with \ to make it a bit more readable in multiple lines, otherwise we have to write the regex in a single line, which gets unwieldly really fast. The need to make ORs in the regex via | also makes it quite error-prone.

If we have to combine tags it gets even worse, e.g. Mute for any host that has as Name any of the values as the example above, OR Type someType1 OR someType2:

        "some_prowler_check":
          Regions:
            - "*"
          Resources:
            - "*"
          Tags:
            - "Name=(\
                a\\.domain\\.com|\
                b\\.domain\\.com|\
                c\\.domain\\.com\
              )|\
              Type=(\
                someType1|\
                someType2\
              )"

All of the complications of the syntax are compounded by having to escape regex-interpreted characters inside yaml with double quotes, making the code quite unreadable. Plus regexes are not anchored, so we run also the danger of matching parts of the strings unintendedly.

Solution Proposed

I think a new syntax for being able to combine AND/OR on mutelist tag rules should be reviewed. Probably with a default behaviour that keeps current behaviour so current deployments are not broken, but that allows to overwrite the behaviour as required.

Inspiration could e.g. be taken from filebeat's processors syntax for and/or in yaml: https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html. There's no need I think for something as complicated as their support for different filters, but the and/or/not functionality would be very welcome.

Describe alternatives you've considered

Of course, if you have a better approach for implementing such a use case, I'll be happy to learn something 👍

Additional context

Thank you very much for the great project!

/cc @pedrooot

[MrCloudSec]

Hi @dlouzan, what would your ideal syntaxes for the above examples?

In Prowler, you can also use the following syntax for the examples:

1. Filter out any failures on resources tagged with Name a.domain.com OR b.domain.com OR c.domain.com:

"some_prowler_check":
  Regions:
    - "*"
  Resources:
    - "*"
  Tags:
    - "Name=a.domain.com|Name=a.domain.com|Name=a.domain.com"

2. Mute for any host that has as Name any of the values as the example above, OR Type someType1 OR someType2:

"some_prowler_check":
  Regions:
    - "*"
  Resources:
    - "*"
  Tags:
    - "Name=a.domain.com|Name=a.domain.com|Name=a.domain.com|Type=someType1|Type=someType1"

Is that approach better for your use case?

[dlouzan]

@sergargar I imagine something inspired by filebeat's syntax would work, they allow combinations of and/or/not.

# Mute when `Name` is any of the 3 values below

"some_prowler_check":
  Regions:
    - "*"
  Resources:
    - "*"
  Tags:
    or:
      - "Name=a.domain.com"
      - "Name=b.domain.com"
      - "Name=c.domain.com"

Most probably, this could be supported in general for the different tags available, keeping compatibility with the current syntax. In filebeat they can be combined, but I think for starters a single level would be enough:

# Only mute when resource has `Name=a.domain.com`, or when it has both `Name=b.domain.com` and `Type=sometype`

"some_prowler_check":
  Regions:
    - "*"
  Resources:
    - "*"
  Tags:
    or:
      - "Name=a.domain.com"
      - and:
          - "Name=b.domain.com"
          - "Type=sometype"

If you'd like to get fancier, we might support different matching modes with equals & regexp, e.g.:

"some_prowler_check":
  Regions:
    - "*"
  Resources:
    - "*"
  Tags:
    or:
      - equals:
          "Name": "a.domain.com"
      - regexp:
          "Name": "some.*\\.domain\\.com"

Another issue I mentioned earlier is the way the regex matching is done, based on a pipe character-separated stringified list from unroll_dict, which makes regex anchors depend on where in the list a certain tag is when converted to a string to be matched (see discussion here).

As an example of what I just mentioned, see a rule I just implemented recently with 4.4.0: limit iam admin rights to a subset of users:

"iam_user_administrator_access_policy":
  Regions:
    - "*"
  Resources:
    - "^user\\.one@siemens\\.com$"
    - "^user\\.two@siemens\\.com$"

In the example above, if the anchors are not used, the mute rule would also mute entries for a user [email protected]; yes, the match is a regex so it's kind of expected that anchors would be needed, but a substring match here I think is a bit tricky for users and error-prone.

Sorry for the wall-of-text ™️