Skip to content

Add search relevance stats API #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
martin-gaievski merged 4 commits into from
Jun 10, 2025
Merged

Add search relevance stats API #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5670353
Add stats API
q-andy Jun 4, 2025
a06bb75
Rename all stats references to search relevance
q-andy Jun 7, 2025
f3bb744
Add experiment stats
q-andy Jun 10, 2025
79e44b0
Add include category parameters
q-andy Jun 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Changing URL for plugin APIs to /_plugin/_search_relevance [backend] ([#62](https://github.com/opensearch-project/search-relevance/pull/62)
- Added lazy index creation for all APIs ([#65](https://github.com/opensearch-project/search-relevance/pull/65)
- Realistic test data set based on ESCI (products, queries, judgements) ([#70](https://github.com/opensearch-project/search-relevance/pull/70)
- [Stats] Add stats API ([#63](https://github.com/opensearch-project/search-relevance/pull/63)))

### Removed

Expand Down
20 changes: 20 additions & 0 deletions src/main/java/org/opensearch/searchrelevance/dao/ExperimentDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
import static org.opensearch.searchrelevance.indices.SearchRelevanceIndices.EXPERIMENT;

import java.io.IOException;
import java.util.Map;
import java.util.Optional;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
Expand All @@ -25,10 +27,21 @@
import org.opensearch.searchrelevance.exception.SearchRelevanceException;
import org.opensearch.searchrelevance.indices.SearchRelevanceIndicesManager;
import org.opensearch.searchrelevance.model.Experiment;
import org.opensearch.searchrelevance.model.ExperimentType;
import org.opensearch.searchrelevance.stats.events.EventStatName;
import org.opensearch.searchrelevance.stats.events.EventStatsManager;

public class ExperimentDao {
private static final Logger LOGGER = LogManager.getLogger(ExperimentDao.class);
private final SearchRelevanceIndicesManager searchRelevanceIndicesManager;
private static Map<ExperimentType, Runnable> experimentTypeIncrementers = Map.of(
ExperimentType.PAIRWISE_COMPARISON,
() -> EventStatsManager.increment(EventStatName.EXPERIMENT_PAIRWISE_COMPARISON_EXECUTIONS),
ExperimentType.POINTWISE_EVALUATION,
() -> EventStatsManager.increment(EventStatName.EXPERIMENT_POINTWISE_EVALUATION_EXECUTIONS),
ExperimentType.HYBRID_OPTIMIZER,
() -> EventStatsManager.increment(EventStatName.EXPERIMENT_HYBRID_OPTIMIZER_EXECUTIONS)
);

public ExperimentDao(SearchRelevanceIndicesManager searchRelevanceIndicesManager) {
this.searchRelevanceIndicesManager = searchRelevanceIndicesManager;
Expand All @@ -52,6 +65,8 @@ public void putExperiment(final Experiment experiment, final ActionListener list
listener.onFailure(new SearchRelevanceException("Experiment cannot be null", RestStatus.BAD_REQUEST));
return;
}
// Increment stats
recordStats(experiment);
try {
searchRelevanceIndicesManager.putDoc(
experiment.id(),
Expand Down Expand Up @@ -121,4 +136,9 @@ public SearchResponse listExperiment(SearchSourceBuilder sourceBuilder, ActionLi

return searchRelevanceIndicesManager.listDocsBySearchRequest(sourceBuilder, EXPERIMENT, listener);
}

private void recordStats(Experiment experiment) {
EventStatsManager.increment(EventStatName.EXPERIMENT_EXECUTIONS);
Optional.ofNullable(experimentTypeIncrementers.get(experiment.type())).ifPresent(Runnable::run);
Copy link
Copy Markdown
Member

@martin-gaievski martin-gaievski Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this same pattern for other plugins, this total number of executions is effectively the sum of all metrics for individual types? This saves us efforts for aggregating them outside OS but increases numbers of KPIs we're collecting.

Copy link
Copy Markdown
Contributor Author

@q-andy q-andy Jun 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's on a case by case basis but here it's the same pattern, the idea for this API design is to reduce external aggregations when possible. For hybrid query stats in neural search for example, we track the total number of normalization processor executions in addition to the normalization/combination technique breakdowns, see here

In this case the primary way users will be interacting with SRW is by running experiments. So I think have a coarse grained metric like this is worthwhile, even if we can aggregate the info in other ways

}
}
3 changes: 3 additions & 0 deletions src/main/java/org/opensearch/searchrelevance/judgments/ImportJudgmentsProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
import org.opensearch.core.rest.RestStatus;
import org.opensearch.searchrelevance.exception.SearchRelevanceException;
import org.opensearch.searchrelevance.model.JudgmentType;
import org.opensearch.searchrelevance.stats.events.EventStatName;
import org.opensearch.searchrelevance.stats.events.EventStatsManager;
import org.opensearch.transport.client.Client;

public class ImportJudgmentsProcessor implements BaseJudgmentsProcessor {
Expand All @@ -37,6 +39,7 @@ public JudgmentType getJudgmentType() {

@Override
public void generateJudgmentRating(Map<String, Object> metadata, ActionListener<List<Map<String, Object>>> listener) {
EventStatsManager.increment(EventStatName.IMPORT_JUDGMENT_RATING_GENERATIONS);

List<Map<String, Object>> sourceJudgementRatings = (List<Map<String, Object>>) metadata.get("judgmentRatings");
metadata.remove("judgmentRatings");
Expand Down
3 changes: 3 additions & 0 deletions src/main/java/org/opensearch/searchrelevance/judgments/LlmJudgmentsProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@
import org.opensearch.searchrelevance.ml.MLAccessor;
import org.opensearch.searchrelevance.model.JudgmentCache;
import org.opensearch.searchrelevance.model.JudgmentType;
import org.opensearch.searchrelevance.stats.events.EventStatName;
import org.opensearch.searchrelevance.stats.events.EventStatsManager;
import org.opensearch.searchrelevance.utils.TimeUtils;
import org.opensearch.transport.client.Client;

Expand Down Expand Up @@ -87,6 +89,7 @@ public JudgmentType getJudgmentType() {

@Override
public void generateJudgmentRating(Map<String, Object> metadata, ActionListener<List<Map<String, Object>>> listener) {
EventStatsManager.increment(EventStatName.LLM_JUDGMENT_RATING_GENERATIONS);
String querySetId = (String) metadata.get("querySetId");
List<String> searchConfigurationList = (List<String>) metadata.get("searchConfigurationList");
int size = (int) metadata.get("size");
Expand Down
3 changes: 3 additions & 0 deletions src/main/java/org/opensearch/searchrelevance/judgments/UbiJudgmentsProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
import org.opensearch.searchrelevance.judgments.clickmodel.coec.CoecClickModel;
import org.opensearch.searchrelevance.judgments.clickmodel.coec.CoecClickModelParameters;
import org.opensearch.searchrelevance.model.JudgmentType;
import org.opensearch.searchrelevance.stats.events.EventStatName;
import org.opensearch.searchrelevance.stats.events.EventStatsManager;
import org.opensearch.transport.client.Client;

public class UbiJudgmentsProcessor implements BaseJudgmentsProcessor {
Expand All @@ -38,6 +40,7 @@ public JudgmentType getJudgmentType() {

@Override
public void generateJudgmentRating(Map<String, Object> metadata, ActionListener<List<Map<String, Object>>> listener) {
EventStatsManager.increment(EventStatName.UBI_JUDGMENT_RATING_GENERATIONS);
String clickModel = (String) metadata.get("clickModel");
int maxRank = (int) metadata.get("maxRank");

Expand Down
24 changes: 20 additions & 4 deletions src/main/java/org/opensearch/searchrelevance/plugin/SearchRelevancePlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import static org.opensearch.searchrelevance.common.PluginConstants.EXPERIMENT_INDEX;
import static org.opensearch.searchrelevance.common.PluginConstants.JUDGMENT_CACHE_INDEX;
import static org.opensearch.searchrelevance.settings.SearchRelevanceSettings.SEARCH_RELEVANCE_STATS_ENABLED;
import static org.opensearch.searchrelevance.settings.SearchRelevanceSettings.SEARCH_RELEVANCE_WORKBENCH_ENABLED;

import java.util.Collection;
Expand Down Expand Up @@ -63,7 +64,10 @@
import org.opensearch.searchrelevance.rest.RestPutJudgmentAction;
import org.opensearch.searchrelevance.rest.RestPutQuerySetAction;
import org.opensearch.searchrelevance.rest.RestPutSearchConfigurationAction;
import org.opensearch.searchrelevance.rest.RestSearchRelevanceStatsAction;
import org.opensearch.searchrelevance.settings.SearchRelevanceSettingsAccessor;
import org.opensearch.searchrelevance.stats.events.EventStatsManager;
import org.opensearch.searchrelevance.stats.info.InfoStatsManager;
import org.opensearch.searchrelevance.transport.experiment.DeleteExperimentAction;
import org.opensearch.searchrelevance.transport.experiment.DeleteExperimentTransportAction;
import org.opensearch.searchrelevance.transport.experiment.GetExperimentAction;
Expand All @@ -90,6 +94,9 @@
import org.opensearch.searchrelevance.transport.searchConfiguration.GetSearchConfigurationTransportAction;
import org.opensearch.searchrelevance.transport.searchConfiguration.PutSearchConfigurationAction;
import org.opensearch.searchrelevance.transport.searchConfiguration.PutSearchConfigurationTransportAction;
import org.opensearch.searchrelevance.transport.stats.SearchRelevanceStatsAction;
import org.opensearch.searchrelevance.transport.stats.SearchRelevanceStatsTransportAction;
import org.opensearch.searchrelevance.utils.ClusterUtil;
import org.opensearch.threadpool.ThreadPool;
import org.opensearch.transport.client.Client;
import org.opensearch.watcher.ResourceWatcherService;
Expand All @@ -112,6 +119,8 @@ public class SearchRelevancePlugin extends Plugin implements ActionPlugin, Syste
private MLAccessor mlAccessor;
private MetricsHelper metricsHelper;
private SearchRelevanceSettingsAccessor settingsAccessor;
private ClusterUtil clusterUtil;
private InfoStatsManager infoStatsManager;

@Override
public Collection<SystemIndexDescriptor> getSystemIndexDescriptors(Settings settings) {
Expand Down Expand Up @@ -149,6 +158,10 @@ public Collection<Object> createComponents(
this.mlAccessor = new MLAccessor(mlClient);
this.metricsHelper = new MetricsHelper(clusterService, client, judgmentDao, evaluationResultDao, experimentVariantDao);
this.settingsAccessor = new SearchRelevanceSettingsAccessor(clusterService, environment.settings());
this.clusterUtil = new ClusterUtil(clusterService);
this.infoStatsManager = new InfoStatsManager(settingsAccessor);
EventStatsManager.instance().initialize(settingsAccessor);

return List.of(
searchRelevanceIndicesManager,
querySetDao,
Expand All @@ -159,7 +172,8 @@ public Collection<Object> createComponents(
evaluationResultDao,
judgmentCacheDao,
mlAccessor,
metricsHelper
metricsHelper,
infoStatsManager
);
}

Expand All @@ -186,7 +200,8 @@ public List<RestHandler> getRestHandlers(
new RestGetSearchConfigurationAction(settingsAccessor),
new RestPutExperimentAction(settingsAccessor),
new RestGetExperimentAction(settingsAccessor),
new RestDeleteExperimentAction(settingsAccessor)
new RestDeleteExperimentAction(settingsAccessor),
new RestSearchRelevanceStatsAction(settingsAccessor, clusterUtil)
);
}

Expand All @@ -205,12 +220,13 @@ public List<RestHandler> getRestHandlers(
new ActionHandler<>(GetSearchConfigurationAction.INSTANCE, GetSearchConfigurationTransportAction.class),
new ActionHandler<>(PutExperimentAction.INSTANCE, PutExperimentTransportAction.class),
new ActionHandler<>(DeleteExperimentAction.INSTANCE, DeleteExperimentTransportAction.class),
new ActionHandler<>(GetExperimentAction.INSTANCE, GetExperimentTransportAction.class)
new ActionHandler<>(GetExperimentAction.INSTANCE, GetExperimentTransportAction.class),
new ActionHandler<>(SearchRelevanceStatsAction.INSTANCE, SearchRelevanceStatsTransportAction.class)
);
}

@Override
public List<Setting<?>> getSettings() {
return List.of(SEARCH_RELEVANCE_WORKBENCH_ENABLED);
return List.of(SEARCH_RELEVANCE_WORKBENCH_ENABLED, SEARCH_RELEVANCE_STATS_ENABLED);
}
}
Loading