[Rule-based Auto-tagging] restructure the in-memory trie to store values as a set #19344

ruai0511 · 2025-09-19T21:02:43Z

In this PR, we restructured the in-memory rule tries so that values are stored as a set instead of a single string.
The reason is that as we add more attributes into the system, a single attribute key may correspond to multiple valid labels.

For example:

Rule 1: index_pattern: [a], label: label1
Rule 2: index_pattern: [a], username: [b], label: label2

These two rules are not duplicates—both should be stored. In the trie for index_pattern, the key a would map to:

a → { label1, label2 }

By switching from a string to a set, we can correctly represent multiple labels for the same attribute key without overwriting or losing data.

More documentation on the feature is here: https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/rule-based-autotagging/autotagging/

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
[] API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

jainankitk

Thanks @ruai0511 for breaking into smaller code changes. Mostly LGTM! Couple of minor comments to understand the code changes better

...utotagging-commons/common/src/main/java/org/opensearch/rule/storage/AttributeValueStore.java

...ing-commons/common/src/main/java/org/opensearch/rule/storage/DefaultAttributeValueStore.java

github-actions · 2025-09-19T21:47:55Z

❌ Gradle check result for 5550d32: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-19T23:14:53Z

❌ Gradle check result for 9fa3af8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk · 2025-09-22T22:16:19Z

One of the failure looks related to this PR:

[Test Result](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/) (3 failures / -1)

    [org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/junit/org.opensearch.http/SearchRestCancellationIT/testAutomaticCancellationMultiSearchDuringQueryPhase/)
    [org.opensearch.gateway.remote.RemoteClusterStateServiceTests.testReadClusterStateInParallel_ExceptionDuringRead](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/junit/org.opensearch.gateway.remote/RemoteClusterStateServiceTests/testReadClusterStateInParallel_ExceptionDuringRead/)

@ruai0511 - Can you check this once?

github-actions · 2025-09-22T22:47:26Z

❌ Gradle check result for 9fa3af8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ruai0511 · 2025-09-22T23:32:05Z

One of the failure looks related to this PR:

[Test Result](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/) (3 failures / -1)

    [org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/junit/org.opensearch.http/SearchRestCancellationIT/testAutomaticCancellationMultiSearchDuringQueryPhase/)
    [org.opensearch.gateway.remote.RemoteClusterStateServiceTests.testReadClusterStateInParallel_ExceptionDuringRead](https://build.ci.opensearch.org/job/gradle-check/63939/testReport/junit/org.opensearch.gateway.remote/RemoteClusterStateServiceTests/testReadClusterStateInParallel_ExceptionDuringRead/)

@ruai0511 - Can you check this once?

Sorry which test is related?

jainankitk · 2025-09-23T00:35:36Z

Sorry which test is related?

#19354

ruai0511 · 2025-09-23T17:54:44Z

Sorry which test is related?

#19354

Oh yeah I already asked @Lindsay-00 yesterday to take a look since she's in charge of IT.

...utotagging-commons/common/src/main/java/org/opensearch/rule/storage/AttributeValueStore.java

...ing-commons/common/src/main/java/org/opensearch/rule/storage/DefaultAttributeValueStore.java

github-actions · 2025-09-23T23:12:13Z

❌ Gradle check result for 403d34b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk · 2025-09-23T23:13:44Z

Still seeing some flaky tests related to WLM:

[Test Result](https://build.ci.opensearch.org/job/gradle-check/64193/testReport/) (3 failures / -55)

    [org.opensearch.plugin.wlm.WlmAutoTaggingIT.testMultipleRulesDoNotInterfereWithEachOther {p0={"search.concurrent_segment_search.enabled":"false"}}](https://build.ci.opensearch.org/job/gradle-check/64193/testReport/junit/org.opensearch.plugin.wlm/WlmAutoTaggingIT/testMultipleRulesDoNotInterfereWithEachOther__p0___search_concurrent_segment_search_enabled___false___/)
    [org.opensearch.plugin.wlm.WlmAutoTaggingIT.testDeleteRuleForNonexistentId {p0={"search.concurrent_segment_search.enabled":"true"}}](https://build.ci.opensearch.org/job/gradle-check/64193/testReport/junit/org.opensearch.plugin.wlm/WlmAutoTaggingIT/testDeleteRuleForNonexistentId__p0___search_concurrent_segment_search_enabled___true___/)
    [org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation](https://build.ci.opensearch.org/job/gradle-check/64193/testReport/junit/org.opensearch.action.admin.cluster.node.tasks/ResourceAwareTasksTests/testTaskResourceTrackingDuringTaskCancellation/)

CHANGELOG.md

github-actions · 2025-09-24T01:16:27Z

✅ Gradle check result for 6433f37: SUCCESS

codecov · 2025-09-24T01:16:48Z

Codecov Report

❌ Patch coverage is 79.16667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.90%. Comparing base (a132bb5) to head (a36aaa4).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...earch/rule/storage/DefaultAttributeValueStore.java	76.47%	2 Missing and 2 partials ⚠️
...opensearch/rule/InMemoryRuleProcessingService.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19344      +/-   ##
============================================
+ Coverage     72.81%   72.90%   +0.09%     
- Complexity    69838    69871      +33     
============================================
  Files          5674     5675       +1     
  Lines        320850   320886      +36     
  Branches      46383    46390       +7     
============================================
+ Hits         233624   233945     +321     
+ Misses        68361    67965     -396     
- Partials      18865    18976     +111

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Ruirui Zhang <[email protected]>

github-actions · 2025-09-24T06:15:09Z

❕ Gradle check result for a36aaa4: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…pensearch-project#19344) Signed-off-by: Ruirui Zhang <[email protected]>

ruai0511 requested a review from a team as a code owner September 19, 2025 21:02

ruai0511 force-pushed the trie-restructure branch from d020ffb to 5550d32 Compare September 19, 2025 21:04

ruai0511 added the v3.3.0 label Sep 19, 2025

jainankitk reviewed Sep 19, 2025

View reviewed changes

...utotagging-commons/common/src/main/java/org/opensearch/rule/storage/AttributeValueStore.java Outdated Show resolved Hide resolved

...ing-commons/common/src/main/java/org/opensearch/rule/storage/DefaultAttributeValueStore.java Outdated Show resolved Hide resolved

ruai0511 force-pushed the trie-restructure branch from 5550d32 to 00f931c Compare September 19, 2025 22:20

opensearch-ci-bot mentioned this pull request Sep 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

ruai0511 force-pushed the trie-restructure branch 4 times, most recently from fc729f0 to 58d35c6 Compare September 23, 2025 22:05

jainankitk reviewed Sep 23, 2025

View reviewed changes

ruai0511 force-pushed the trie-restructure branch from 58d35c6 to 403d34b Compare September 23, 2025 22:37

jainankitk reviewed Sep 23, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

ruai0511 force-pushed the trie-restructure branch from 403d34b to 6433f37 Compare September 23, 2025 23:52

opensearch-ci-bot mentioned this pull request Sep 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ResourceAwareTasksTests #14293

Open

restructure trie to store values as a set

a36aaa4

Signed-off-by: Ruirui Zhang <[email protected]>

ruai0511 force-pushed the trie-restructure branch from 6433f37 to a36aaa4 Compare September 24, 2025 04:49

jainankitk approved these changes Sep 24, 2025

View reviewed changes

jainankitk merged commit 8f59bf2 into opensearch-project:main Sep 24, 2025
33 checks passed

This was referenced Sep 22, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AwarenessAllocationIT #17930

Open

[AUTOCUT] Gradle Check Flaky Test Report for CreateIndexIT #14312

Open

vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025

[Rule-based Auto-tagging] Restructure trie to store values as a set (o…

4a36e63

…pensearch-project#19344) Signed-off-by: Ruirui Zhang <[email protected]>

BrewTestBot mentioned this pull request Oct 15, 2025

opensearch 3.3.0 Homebrew/homebrew-core#249627

Merged

[Rule-based Auto-tagging] restructure the in-memory trie to store values as a set #19344

[Rule-based Auto-tagging] restructure the in-memory trie to store values as a set #19344

Uh oh!

Conversation

ruai0511 commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

jainankitk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

jainankitk commented Sep 22, 2025

Uh oh!

github-actions bot commented Sep 22, 2025

Uh oh!

ruai0511 commented Sep 22, 2025

Uh oh!

jainankitk commented Sep 23, 2025

Uh oh!

ruai0511 commented Sep 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 23, 2025

Uh oh!

jainankitk commented Sep 23, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

codecov bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ruai0511 commented Sep 19, 2025 •

edited

Loading

codecov bot commented Sep 24, 2025 •

edited

Loading