From 0025bddb132da8775642aaa52b7bbcd011c4fe12 Mon Sep 17 00:00:00 2001 From: enomott Date: Sun, 11 Jan 2026 10:52:31 +0900 Subject: [PATCH] docs: add Index Management Enhancements report for v3.3.0 --- .../index-management/index-management.md | 4 + .../index-management-enhancements.md | 93 +++++++++++++++++++ docs/releases/v3.3.0/index.md | 4 + 3 files changed, 101 insertions(+) create mode 100644 docs/releases/v3.3.0/features/index-management/index-management-enhancements.md diff --git a/docs/features/index-management/index-management.md b/docs/features/index-management/index-management.md index f8f692961..a777dd40b 100644 --- a/docs/features/index-management/index-management.md +++ b/docs/features/index-management/index-management.md @@ -232,6 +232,9 @@ PUT _plugins/_rollup/jobs/sample_rollup | Version | PR | Description | |---------|-----|-------------| +| v3.3.0 | [#1460](https://github.com/opensearch-project/index-management/pull/1460) | Using Scripted Avg Class in AvgAggregationBuilder for rollup aggregation fix | +| v3.3.0 | [#1491](https://github.com/opensearch-project/index-management/pull/1491) | Fix the build for upstream OpenSearch changes | +| v3.3.0 | [#1473](https://github.com/opensearch-project/index-management/pull/1473) | Dependabot: bump 1password/load-secrets-action from 2 to 3 | | v3.2.0 | [#1440](https://github.com/opensearch-project/index-management/pull/1440) | Support for no_alias and min_state_age in ISM Transitions | | v3.2.0 | [#1444](https://github.com/opensearch-project/index-management/pull/1444) | Add history index pattern to System Index descriptors | | v3.2.0 | [#1442](https://github.com/opensearch-project/index-management/pull/1442) | Fix Integration test and lint errors | @@ -265,6 +268,7 @@ PUT _plugins/_rollup/jobs/sample_rollup ## Change History +- **v3.3.0** (2026-01-11): Fixed rollup aggregation reduction bug when searching rollup and raw indices together by using ScriptedAvg class, build fixes for upstream OpenSearch changes, dependency updates - **v3.2.0** (2026-01-10): Added `no_alias` and `min_state_age` transition conditions for ISM, registered ISM history index as System Index descriptor, fixed integration tests and lint errors - **v3.1.0** (2026-01-10): Fixed false positive notifications in Snapshot Management by suppressing user notifications for internal VersionConflictEngineException errors - **v3.0.0** (2025-05-06): Added ISM unfollow action for CCR, rollup target index settings, CVE fixes, Java Agent migration diff --git a/docs/releases/v3.3.0/features/index-management/index-management-enhancements.md b/docs/releases/v3.3.0/features/index-management/index-management-enhancements.md new file mode 100644 index 000000000..012200df6 --- /dev/null +++ b/docs/releases/v3.3.0/features/index-management/index-management-enhancements.md @@ -0,0 +1,93 @@ +# Index Management Enhancements + +## Summary + +This release includes bug fixes and maintenance updates for the Index Management plugin, including a fix for rollup aggregation reduction when searching across rollup and raw indices together, build fixes for upstream OpenSearch changes, and a dependency update for the 1password/load-secrets-action GitHub Action. + +## Details + +### What's New in v3.3.0 + +#### Rollup Aggregation Fix (ScriptedAvg Class) + +The main enhancement addresses a bug in the rollup feature when searching rollup and raw indices together (introduced in v2.18.0). The fix uses the `ScriptedAvg` class in `AvgAggregationBuilder` instead of raw double arrays, enabling proper handling during aggregation reduction. + +**Problem**: When searching across both rollup and non-rollup indices, a `ClassCastException` occurred during the reduce phase of `InternalValueCount` and `InternalAvg` aggregations because both used `InternalScriptedMetric` internally. + +**Solution**: Updated the scripted metric aggregation scripts in `RollupUtils.kt` to use `ScriptedAvg` class for proper type handling: + +```kotlin +// Before (v3.2.0 and earlier) +combineScript: "def d = new double[2]; d[0] = state.sums; d[1] = state.counts; return d" +reduceScript: "... for (a in states) { sum += a[0]; count += a[1]; } ..." + +// After (v3.3.0) +combineScript: "def d = new org.opensearch.search.aggregations.metrics.ScriptedAvg(state.sums, state.counts); return d" +reduceScript: "... for (a in states) { sum += a.getSum(); count += a.getCount(); } ..." +``` + +This change depends on OpenSearch core PR [#18411](https://github.com/opensearch-project/OpenSearch/pull/18411) which added support for `InternalScriptedMetric` in `InternalValueCount` and `InternalAvg` reduction. + +#### Build Fixes + +Updated test infrastructure to accommodate upstream OpenSearch changes: +- Changed `LockService` instantiation in unit tests from direct constructor to mock +- Updated `randomUser()` helper to generate proper custom attribute names format (`key=value`) +- Added `error_prone_annotations` dependency resolution +- Removed deprecated `log4j-core` test dependency + +#### Dependency Update + +Bumped `1password/load-secrets-action` from v2 to v3 in GitHub Actions workflows. This is a breaking change in the action that sets `export-env` to `false` by default. + +### Technical Changes + +#### Modified Files + +| File | Change | +|------|--------| +| `RollupUtils.kt` | Updated scripted metric scripts to use `ScriptedAvg` class | +| `RollupInterceptorIT.kt` | Added `value_count` aggregation to test coverage | +| `spi/build.gradle` | Added `error_prone_annotations` dependency | +| `TestHelpers.kt` | Updated `randomUser()` for custom attribute format | +| Multiple `*StepTests.kt` | Changed `LockService` to mock | + +### Usage Example + +Searching across rollup and raw indices with avg aggregation now works correctly: + +```json +GET sample-data-*,sample-rollup/_search +{ + "size": 0, + "aggs": { + "avg_value": { "avg": { "field": "value" } }, + "sum_value": { "sum": { "field": "value" } }, + "count_value": { "value_count": { "field": "value" } } + } +} +``` + +## Limitations + +- The rollup aggregation fix requires OpenSearch core v3.3.0+ with the `ScriptedAvg` class support +- Mixed rollup/raw index search still requires `plugins.rollup.search.search_source_indices: true` + +## Related PRs + +| PR | Description | +|----|-------------| +| [#1460](https://github.com/opensearch-project/index-management/pull/1460) | Using Scripted Avg Class in AvgAggregationBuilder in place of double | +| [#1491](https://github.com/opensearch-project/index-management/pull/1491) | Fix the build | +| [#1473](https://github.com/opensearch-project/index-management/pull/1473) | Dependabot: bump 1password/load-secrets-action from 2 to 3 | + +## References + +- [OpenSearch PR #18411](https://github.com/opensearch-project/OpenSearch/pull/18411): Core changes for InternalScriptedMetric support in aggregation reduction +- [OpenSearch PR #18288](https://github.com/opensearch-project/OpenSearch/pull/18288): Related aggregation changes +- [Index Rollups Documentation](https://docs.opensearch.org/3.3/im-plugin/index-rollups/index/) +- [Index Rollups API](https://docs.opensearch.org/3.3/im-plugin/index-rollups/rollup-api/) + +## Related Feature Report + +- [Full feature documentation](../../../../features/index-management/index-management.md) diff --git a/docs/releases/v3.3.0/index.md b/docs/releases/v3.3.0/index.md index 4cbc1b5e4..7f9990ec2 100644 --- a/docs/releases/v3.3.0/index.md +++ b/docs/releases/v3.3.0/index.md @@ -125,6 +125,10 @@ - [Geospatial Deprecation Fixes](features/geospatial/geospatial-deprecation-fixes.md) +### Index Management + +- [Index Management Enhancements](features/index-management/index-management-enhancements.md) + ### Query Insights - [Query Insights Bug Fixes](features/query-insights/query-insights-bug-fixes.md)