[RAM] Add shareable rule tag filter

[JiaweiWu]

Summary

Resolves: #130158

Adds a shareable rule tags filter, this is a new component that does not exist as a part of the rule list currently. Therefore, it is not being used, except as a shareable component (hidden behind a feature flag).

This change also required new aggregations on the tags field since we need a list of all tags that the user can filter on. So the alerting aggregation function was modified. API and E2E tests were written to cover this.

[mgiota]

@JiaweiWu LGTM so far!

For the tags filter to work, I expect one change in this function call https://github.com/elastic/kibana/blob/main/x-pack/plugins/triggers_actions_ui/public/application/lib/rule_api/rules.ts#L40 where we need to pass the new tagsFilter along with the rest filters that are currently being passed. This means that this file https://github.com/elastic/kibana/blob/main/x-pack/plugins/triggers_actions_ui/public/application/lib/rule_api/map_filters_to_kql.ts needs to be changed as well.

[JiaweiWu]

@elasticmachine merge upstream

[JiaweiWu]

@mgiota I added changes to the loadRules and API changes, feel free to give it a test

[JiaweiWu]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mgiota]

@JiaweiWu I gave it a try and yep it works!

I will have a look again a bit later and approve the PR.

[mgiota]

@JiaweiWu Since I had to do the loadRuleAggregations logic on our side same way you did above in order to setTags, I was wondering if it would make sense to make the tags prop optional and have this logic implemented within the RuleTagFilter, if you know what I mean. The code I had to add is similar to what you have on your side, that's why I am wondering if we could avoid this repetition.

I don't mind merging current PR and have a follow up PR when there's more time. I just wanted to hear your opinion, regarding moving the aggregations on the tags field within the RuleTagFilter, so that consumer doesn't have to write this logic. What do you think?

[JiaweiWu]

We had a chat and I brought up the issue that the rules_list aggregations require filter parameters from other filters as well (actionTypes, ruleStatuses, etc.) which makes it outside of the responsibility of the tag filter component.

I'm currently working on refactoring and making the rules list shareable, which will involves moving the fetching logic into hooks. So I think that will indirectly solve this problem.

[mgiota]

I approve this PR! Later on we could refactor so that the server requests the tags and the consumer doesn't have to pass any tags prop.

[mgiota]

It would be great if we could get this PR merged soon, so that I can working on #127013

[gmmorris]

Are we confident the buckets are always there?
ES has a tendency to drop keys in aggs when there are no hits (such as a cluster with no rules in it or when a filter is applied).

Worth treading with caution and checking for undefined keys.

[JiaweiWu]

I tested it with rules that have no tags, but I'll add a fall back just in case

[gmmorris]

Always build these things defensively, otherwise the UI breaks and we have no idea unless someone files an SDH.

Thanks 🙏

[chrisronline]

Looking great so far, had a couple of thoughts/questions in my first pass

[chrisronline]

I don't think you need to worry about this in this PR but this code should be improved. It looks like we're doing multiple JSON.stringify calls every render - we should redo this a bit to be more performant. WDYT?

[mgiota]

@JiaweiWu In case it is useful, here's what I used https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability/public/hooks/use_fetch_rules.ts#L36 when I moved this code under observability in a use_fetch_rules hook. I got rid of JSON.stringify and used useCallback.

[JiaweiWu]

Yep, this is getting improved in my rules_list refactor

[JiaweiWu]

@elasticmachine merge upstream

[JiaweiWu]

I want to give more context to this function here. So there's an awkward behaviour when I added the tag filters to the aggregation, because we use 1 aggregation function to get a list of tags and all of the rule execution status values, if we started filtering down the aggregation with tag filters, the list of tags would also get filtered, and thus it would replace the list of available tags with a smaller subset of tags (which is incorrect).

To fix this, i considered only calling setState on the tags list if the tagsFilter is empty. But this is also incorrect because we miss the edge case where the user adds new tags or a new rule while the tag filter is applied.

From there, I decided to add this function, which calls the aggregation with empty params, as a part of the data fetch. I think the downside here is we're doing the aggregation twice now, which has performance implications.

But aside from doing something client side like merging in new tags in react, we might eventually want to have a separate aggregation just to get a list of tags. Let me know if we want to do that in this PR or I could do it in another PR.

[chrisronline]

Is it possible to understand this more? How can I simulate this? Or can I see this happening on an older version of this PR?

[JiaweiWu]

I didn't have the tag filters passed into the aggregation for the previous commit, you can simulate it by checking out the previous commit and passing the tag filters:

rules_list.tsx:
async function loadRuleAggs() {
  try {
    const rulesAggs = await loadRuleAggregations({
      http,
      searchText,
      typesFilter,
      actionTypesFilter,
      ruleExecutionStatusesFilter,
      ruleStatusesFilter,
      tagsFilter,
  });
  ...
}

aggregate.ts:
export async function loadRuleAggregations({
  const filters = mapFiltersToKql({
    typesFilter,
    actionTypesFilter,
    ruleExecutionStatusesFilter,
    ruleStatusesFilter,
    tagsFilter,
  });
  ...
}

Switching the experiment on and filter by tags using the dropdown, you'll notice the entire list of available tags gets filtered as well

[chrisronline]

Hmm

What about something like:

 async function loadRuleAggs() {
    try {
      const rulesAggs = await loadRuleAggregations({
        http,
        searchText,
        typesFilter,
        actionTypesFilter,
        ruleStatusesFilter,
        tagsFilter,
      });
      if (rulesAggs?.ruleExecutionStatus) {
        setRulesStatusesTotal(rulesAggs.ruleExecutionStatus);
      }
      if (rulesAggs?.ruleTags && rulesAggs?.ruleTags.length > tags.length) {
        setTags(rulesAggs.ruleTags);
      }
    } 

I guess this doesn't work if we delete a rule and a tag is only associated with that rule though

I think we should separate these two things out into separate "things" - api that is called that is assumed to return filtered data and a separate api call that is expected to return total data. We might need to add something to the lib/rule_api layer too but it's too confusing to keep this logic as-is - definitely recommend some delineation here to make it clear in the future.

[JiaweiWu]

yea thought so, ok I'll add a new endpoint to expose tags

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: c35c626
• Interpreting CI Failures
• Storybooks Preview
• Webpack Bundle Analyzer

Failed CI Steps

• Post-Build

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
triggersActionsUi	373	376	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
alerting	316	317	+1
triggersActionsUi	316	320	+4
total			+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
triggersActionsUi	728.2KB	732.4KB	+4.1KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
triggersActionsUi	27	28	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff	errors
triggersActionsUi	100.5KB	101.1KB	+611.0B	❌ 169.0B over limit

Unknown metric groups

API count

id	before	after	diff
alerting	325	326	+1
triggersActionsUi	330	334	+4
total			+5

async chunk count

id	before	after	diff
triggersActionsUi	80	81	+1

ESLint disabled line counts

id	before	after	diff
triggersActionsUi	171	172	+1

Total ESLint disabled count

id	before	after	diff
triggersActionsUi	177	178	+1

History

• 💚 Build #42881 succeeded 7e2d6df
• 💚 Build #42269 succeeded 74fd8ab
• 💚 Build #41023 succeeded f0b2615
• 💔 Build #41018 failed df698fc
• 💛 Build #40944 was flaky 3ca21d7
• 💔 Build #40689 failed 93c1358

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[chrisronline]

LGTM! Great work here

[JiaweiWu]

@elasticmachine merge upstream

[JiaweiWu]

Hey there! It seems like I have to slightly bump the bundle size for triggers_actions_ui again. For context, we have determined ways to lower our bundle size (#130728) + our current efforts to make make our components shareable to other plugins (which is what this PR is doing). Once that happens, our bundle size should go down quite a bit, however, right now we're in this awkward point where there's a lot of PRs happening to try and make this happen from various teams, so we're going to work on reducing the bundle size near the end of this release once the dust settles a bit.

[JiaweiWu]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 010bec3
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
triggersActionsUi	373	376	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
alerting	316	317	+1
triggersActionsUi	316	320	+4
total			+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
triggersActionsUi	725.3KB	729.4KB	+4.1KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
triggersActionsUi	27	28	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
triggersActionsUi	100.2KB	100.8KB	+611.0B

Unknown metric groups

API count

id	before	after	diff
alerting	325	326	+1
triggersActionsUi	330	334	+4
total			+5

async chunk count

id	before	after	diff
triggersActionsUi	80	81	+1

ESLint disabled line counts

id	before	after	diff
triggersActionsUi	171	172	+1

Total ESLint disabled count

id	before	after	diff
triggersActionsUi	177	178	+1

History

• 💛 Build #43123 was flaky 869545a
• 💔 Build #43104 failed 5428990
• 💔 Build #42927 failed c35c626
• 💚 Build #42881 succeeded 7e2d6df
• 💚 Build #42269 succeeded 74fd8ab
• 💚 Build #41023 succeeded f0b2615

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream