Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/features/index-management/index-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,9 @@ PUT _plugins/_rollup/jobs/sample_rollup

| Version | PR | Description |
|---------|-----|-------------|
| v3.3.0 | [#1460](https://github.com/opensearch-project/index-management/pull/1460) | Using Scripted Avg Class in AvgAggregationBuilder for rollup aggregation fix |
| v3.3.0 | [#1491](https://github.com/opensearch-project/index-management/pull/1491) | Fix the build for upstream OpenSearch changes |
| v3.3.0 | [#1473](https://github.com/opensearch-project/index-management/pull/1473) | Dependabot: bump 1password/load-secrets-action from 2 to 3 |
| v3.2.0 | [#1440](https://github.com/opensearch-project/index-management/pull/1440) | Support for no_alias and min_state_age in ISM Transitions |
| v3.2.0 | [#1444](https://github.com/opensearch-project/index-management/pull/1444) | Add history index pattern to System Index descriptors |
| v3.2.0 | [#1442](https://github.com/opensearch-project/index-management/pull/1442) | Fix Integration test and lint errors |
Expand Down Expand Up @@ -265,6 +268,7 @@ PUT _plugins/_rollup/jobs/sample_rollup

## Change History

- **v3.3.0** (2026-01-11): Fixed rollup aggregation reduction bug when searching rollup and raw indices together by using ScriptedAvg class, build fixes for upstream OpenSearch changes, dependency updates
- **v3.2.0** (2026-01-10): Added `no_alias` and `min_state_age` transition conditions for ISM, registered ISM history index as System Index descriptor, fixed integration tests and lint errors
- **v3.1.0** (2026-01-10): Fixed false positive notifications in Snapshot Management by suppressing user notifications for internal VersionConflictEngineException errors
- **v3.0.0** (2025-05-06): Added ISM unfollow action for CCR, rollup target index settings, CVE fixes, Java Agent migration
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Index Management Enhancements

## Summary

This release includes bug fixes and maintenance updates for the Index Management plugin, including a fix for rollup aggregation reduction when searching across rollup and raw indices together, build fixes for upstream OpenSearch changes, and a dependency update for the 1password/load-secrets-action GitHub Action.

## Details

### What's New in v3.3.0

#### Rollup Aggregation Fix (ScriptedAvg Class)

The main enhancement addresses a bug in the rollup feature when searching rollup and raw indices together (introduced in v2.18.0). The fix uses the `ScriptedAvg` class in `AvgAggregationBuilder` instead of raw double arrays, enabling proper handling during aggregation reduction.

**Problem**: When searching across both rollup and non-rollup indices, a `ClassCastException` occurred during the reduce phase of `InternalValueCount` and `InternalAvg` aggregations because both used `InternalScriptedMetric` internally.

**Solution**: Updated the scripted metric aggregation scripts in `RollupUtils.kt` to use `ScriptedAvg` class for proper type handling:

```kotlin
// Before (v3.2.0 and earlier)
combineScript: "def d = new double[2]; d[0] = state.sums; d[1] = state.counts; return d"
reduceScript: "... for (a in states) { sum += a[0]; count += a[1]; } ..."

// After (v3.3.0)
combineScript: "def d = new org.opensearch.search.aggregations.metrics.ScriptedAvg(state.sums, state.counts); return d"
reduceScript: "... for (a in states) { sum += a.getSum(); count += a.getCount(); } ..."
```

This change depends on OpenSearch core PR [#18411](https://github.com/opensearch-project/OpenSearch/pull/18411) which added support for `InternalScriptedMetric` in `InternalValueCount` and `InternalAvg` reduction.

#### Build Fixes

Updated test infrastructure to accommodate upstream OpenSearch changes:
- Changed `LockService` instantiation in unit tests from direct constructor to mock
- Updated `randomUser()` helper to generate proper custom attribute names format (`key=value`)
- Added `error_prone_annotations` dependency resolution
- Removed deprecated `log4j-core` test dependency

#### Dependency Update

Bumped `1password/load-secrets-action` from v2 to v3 in GitHub Actions workflows. This is a breaking change in the action that sets `export-env` to `false` by default.

### Technical Changes

#### Modified Files

| File | Change |
|------|--------|
| `RollupUtils.kt` | Updated scripted metric scripts to use `ScriptedAvg` class |
| `RollupInterceptorIT.kt` | Added `value_count` aggregation to test coverage |
| `spi/build.gradle` | Added `error_prone_annotations` dependency |
| `TestHelpers.kt` | Updated `randomUser()` for custom attribute format |
| Multiple `*StepTests.kt` | Changed `LockService` to mock |

### Usage Example

Searching across rollup and raw indices with avg aggregation now works correctly:

```json
GET sample-data-*,sample-rollup/_search
{
"size": 0,
"aggs": {
"avg_value": { "avg": { "field": "value" } },
"sum_value": { "sum": { "field": "value" } },
"count_value": { "value_count": { "field": "value" } }
}
}
```

## Limitations

- The rollup aggregation fix requires OpenSearch core v3.3.0+ with the `ScriptedAvg` class support
- Mixed rollup/raw index search still requires `plugins.rollup.search.search_source_indices: true`

## Related PRs

| PR | Description |
|----|-------------|
| [#1460](https://github.com/opensearch-project/index-management/pull/1460) | Using Scripted Avg Class in AvgAggregationBuilder in place of double |
| [#1491](https://github.com/opensearch-project/index-management/pull/1491) | Fix the build |
| [#1473](https://github.com/opensearch-project/index-management/pull/1473) | Dependabot: bump 1password/load-secrets-action from 2 to 3 |

## References

- [OpenSearch PR #18411](https://github.com/opensearch-project/OpenSearch/pull/18411): Core changes for InternalScriptedMetric support in aggregation reduction
- [OpenSearch PR #18288](https://github.com/opensearch-project/OpenSearch/pull/18288): Related aggregation changes
- [Index Rollups Documentation](https://docs.opensearch.org/3.3/im-plugin/index-rollups/index/)
- [Index Rollups API](https://docs.opensearch.org/3.3/im-plugin/index-rollups/rollup-api/)

## Related Feature Report

- [Full feature documentation](../../../../features/index-management/index-management.md)
4 changes: 4 additions & 0 deletions docs/releases/v3.3.0/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@

- [Geospatial Deprecation Fixes](features/geospatial/geospatial-deprecation-fixes.md)

### Index Management

- [Index Management Enhancements](features/index-management/index-management-enhancements.md)

### Query Insights

- [Query Insights Bug Fixes](features/query-insights/query-insights-bug-fixes.md)
Expand Down