Skip to content

Commit e985fc2

Browse files
kaushalmahi12kaushalmahi12
authored
[Backport 2.x] [Rule based auto tagging] Add in-memory rule processing service (#17781)
* [Rule based auto tagging] Add in-memory rule processing service (#17365) * [rule based autotagging] add attribute value store Signed-off-by: Kaushal Kumar <[email protected]> * add in-memory rule processing service Signed-off-by: Kaushal Kumar <[email protected]> * add missing javadoc Signed-off-by: Kaushal Kumar <[email protected]> * merge the in-memory store changes: commit d02e544 Author: Kaushal Kumar <[email protected]> Date: Mon Feb 17 13:05:20 2025 -0800 add licenses directory Signed-off-by: Kaushal Kumar <[email protected]> commit 3f98f9d Author: Kaushal Kumar <[email protected]> Date: Mon Feb 17 11:52:56 2025 -0800 improve binary search bisecting expression Signed-off-by: Kaushal Kumar <[email protected]> commit 630a3ee Author: Kaushal Kumar <[email protected]> Date: Mon Feb 17 11:14:39 2025 -0800 improve javadoc for attribute value store Signed-off-by: Kaushal Kumar <[email protected]> commit acdb27c Author: Kaushal Kumar <[email protected]> Date: Fri Feb 14 10:09:58 2025 -0800 add missing javadoc Signed-off-by: Kaushal Kumar <[email protected]> commit 24c4ea6 Author: Kaushal Kumar <[email protected]> Date: Fri Feb 14 09:28:46 2025 -0800 run spotless apply Signed-off-by: Kaushal Kumar <[email protected]> commit 75b6e68 Author: Kaushal Kumar <[email protected]> Date: Fri Feb 14 09:24:32 2025 -0800 make the store interface generic Signed-off-by: Kaushal Kumar <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]> * fix generics error Signed-off-by: Kaushal Kumar <[email protected]> * add CHANGELOG entry Signed-off-by: Kaushal Kumar <[email protected]> * remove stubs Signed-off-by: Kaushal Kumar <[email protected]> * move generic logic to lib Signed-off-by: Kaushal Kumar <[email protected]> * fix javadoc error Signed-off-by: Kaushal Kumar <[email protected]> * fix javadoc error Signed-off-by: Kaushal Kumar <[email protected]> * delete licenses from the wlm plugin Signed-off-by: Kaushal Kumar <[email protected]> * expose feature level attribute value store init method Signed-off-by: Kaushal Kumar <[email protected]> * add extra space to remove unwanted entry from the changelog diff Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * use constructors over static methods Signed-off-by: Kaushal Kumar <[email protected]> * make member var final in InMemoryRuleProcessingService Signed-off-by: Kaushal Kumar <[email protected]> * make concurrency checks more granular Signed-off-by: Kaushal Kumar <[email protected]> * add concurrent test Signed-off-by: Kaushal Kumar <[email protected]> * remove forbidden api usage Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]> * add changelog entry Signed-off-by: Kaushal Kumar <[email protected]> * remove generic type usage with instanceof operator Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>
1 parent 05bb96b commit e985fc2

File tree

21 files changed

+661
-80
lines changed

21 files changed

+661
-80
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1010
- Add execution_hint to cardinality aggregator request (#[17419](https://github.com/opensearch-project/OpenSearch/pull/17419))
1111
- Fix Bug - Handle unsigned long in sorting order assertion of LongHashSet ([#17207](https://github.com/opensearch-project/OpenSearch/pull/17207))
1212
- [Rule Based Auto-tagging] Add in-memory attribute value store ([#17342](https://github.com/opensearch-project/OpenSearch/pull/17342))
13+
- [Rule Based Auto-tagging] Add in-memory rule processing service ([#17365](https://github.com/opensearch-project/OpenSearch/pull/17365))
1314
- Change priority for scheduling reroute during timeout([#16445](https://github.com/opensearch-project/OpenSearch/pull/16445))
1415

1516
### Dependencies

‎libs/autotagging-commons/build.gradle‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
dependencies {
10+
api 'org.apache.commons:commons-collections4:4.4'
11+
api project(":server")
12+
13+
testImplementation(project(":test:framework")) {
14+
exclude group: 'org.opensearch', module: 'opensearch-core'
15+
}
16+
}
17+
18+
tasks.named("dependencyLicenses").configure {
19+
mapping from: /commons-collections.*/, to: 'commons-collections'
20+
}

‎plugins/workload-management/licenses/commons-collections-LICENSE.txt‎ renamed to ‎libs/autotagging-commons/licenses/commons-collections-LICENSE.txt‎

File renamed without changes.

‎plugins/workload-management/licenses/commons-collections-NOTICE.txt‎ renamed to ‎libs/autotagging-commons/licenses/commons-collections-NOTICE.txt‎

File renamed without changes.

‎plugins/workload-management/licenses/commons-collections4-4.4.jar.sha1‎ renamed to ‎libs/autotagging-commons/licenses/commons-collections4-4.4.jar.sha1‎

File renamed without changes.

‎libs/autotagging-commons/src/main/java/org/opensearch/rule/InMemoryRuleProcessingService.java‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.rule;
10+
11+
import org.opensearch.autotagging.Attribute;
12+
import org.opensearch.autotagging.FeatureType;
13+
import org.opensearch.autotagging.Rule;
14+
import org.opensearch.rule.attribute_extractor.AttributeExtractor;
15+
import org.opensearch.rule.storage.AttributeValueStore;
16+
import org.opensearch.rule.storage.AttributeValueStoreFactory;
17+
18+
import java.util.List;
19+
import java.util.Map;
20+
import java.util.Optional;
21+
import java.util.Set;
22+
import java.util.function.BiConsumer;
23+
import java.util.function.Supplier;
24+
25+
/**
26+
* This class is responsible for managing in-memory view of Rules and Find matching Rule for the request
27+
* Each auto-tagging feature should use a separate instance of this class as this avoid potential concurrency overhead
28+
* in case of dynamic updates and attribute sharing scenarios
29+
*/
30+
public class InMemoryRuleProcessingService {
31+
32+
private final AttributeValueStoreFactory attributeValueStoreFactory;
33+
34+
/**
35+
* Constrcutor
36+
* @param featureType
37+
* @param attributeValueStoreSupplier
38+
*/
39+
public InMemoryRuleProcessingService(
40+
FeatureType featureType,
41+
Supplier<AttributeValueStore<String, String>> attributeValueStoreSupplier
42+
) {
43+
attributeValueStoreFactory = new AttributeValueStoreFactory(featureType, attributeValueStoreSupplier);
44+
}
45+
46+
/**
47+
* Adds the rule to in-memory view
48+
* @param rule to be added
49+
*/
50+
public void add(final Rule rule) {
51+
perform(rule, this::addOperation);
52+
}
53+
54+
/**
55+
* Removes the rule from in-memory view
56+
* @param rule to be removed
57+
*/
58+
public void remove(final Rule rule) {
59+
perform(rule, this::removeOperation);
60+
}
61+
62+
private void perform(Rule rule, BiConsumer<Map.Entry<Attribute, Set<String>>, Rule> ruleOperation) {
63+
for (Map.Entry<Attribute, Set<String>> attributeEntry : rule.getAttributeMap().entrySet()) {
64+
ruleOperation.accept(attributeEntry, rule);
65+
}
66+
}
67+
68+
private void removeOperation(Map.Entry<Attribute, Set<String>> attributeEntry, Rule rule) {
69+
AttributeValueStore<String, String> valueStore = attributeValueStoreFactory.getAttributeValueStore(attributeEntry.getKey());
70+
for (String value : attributeEntry.getValue()) {
71+
valueStore.remove(value);
72+
}
73+
}
74+
75+
private void addOperation(Map.Entry<Attribute, Set<String>> attributeEntry, Rule rule) {
76+
AttributeValueStore<String, String> valueStore = attributeValueStoreFactory.getAttributeValueStore(attributeEntry.getKey());
77+
for (String value : attributeEntry.getValue()) {
78+
valueStore.put(value, rule.getFeatureValue());
79+
}
80+
}
81+
82+
/**
83+
* Evaluates the label for the current request. It finds the matches for each attribute value and then it is an
84+
* intersection of all the matches
85+
* @param attributeExtractors list of extractors which are used to get the attribute values to find the
86+
* matching rule
87+
* @return a label if there is unique label otherwise empty
88+
*/
89+
public Optional<String> evaluateLabel(List<AttributeExtractor<String>> attributeExtractors) {
90+
assert attributeValueStoreFactory != null;
91+
Optional<String> result = Optional.empty();
92+
for (AttributeExtractor<String> attributeExtractor : attributeExtractors) {
93+
AttributeValueStore<String, String> valueStore = attributeValueStoreFactory.getAttributeValueStore(
94+
attributeExtractor.getAttribute()
95+
);
96+
for (String value : attributeExtractor.extract()) {
97+
Optional<String> possibleMatch = valueStore.get(value);
98+
99+
if (possibleMatch.isEmpty()) {
100+
return Optional.empty();
101+
}
102+
103+
if (result.isEmpty()) {
104+
result = possibleMatch;
105+
} else {
106+
boolean isThePossibleMatchEqualResult = possibleMatch.get().equals(result.get());
107+
if (!isThePossibleMatchEqualResult) {
108+
return Optional.empty();
109+
}
110+
}
111+
}
112+
}
113+
return result;
114+
}
115+
}

‎libs/autotagging-commons/src/main/java/org/opensearch/rule/attribute_extractor/AttributeExtractor.java‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.rule.attribute_extractor;
10+
11+
import org.opensearch.autotagging.Attribute;
12+
13+
/**
14+
* This interface defines the contract for extracting the attributes for Rule based auto-tagging feature
15+
* @param <V>
16+
*/
17+
public interface AttributeExtractor<V> {
18+
/**
19+
* This method returns the Attribute which it is responsible for extracting
20+
* @return attribute
21+
*/
22+
Attribute getAttribute();
23+
24+
/**
25+
* This method returns the attribute values in context of the current request
26+
* @return attribute value
27+
*/
28+
Iterable<V> extract();
29+
}

‎libs/autotagging-commons/src/main/java/org/opensearch/rule/attribute_extractor/package-info.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
/**
10+
* This package contains feature attribute extractor interface and its implementations
11+
*/
12+
package org.opensearch.rule.attribute_extractor;

‎libs/autotagging-commons/src/main/java/org/opensearch/rule/package-info.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
/**
10+
* Rule based auto-tagging generic constructs
11+
*/
12+
package org.opensearch.rule;

‎plugins/workload-management/src/main/java/org/opensearch/plugin/wlm/rule/storage/AttributeValueStore.java‎ renamed to ‎libs/autotagging-commons/src/main/java/org/opensearch/rule/storage/AttributeValueStore.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
* compatible open source license.
77
*/
88

9-
package org.opensearch.plugin.wlm.rule.storage;
9+
package org.opensearch.rule.storage;
1010

1111
import java.util.Optional;
1212

0 commit comments

Comments
 (0)