diff --git a/release-notes/opensearch.release-notes-3.6.0.md b/release-notes/opensearch.release-notes-3.6.0.md index d4d17c3f3b5d5..8aca4794245e8 100644 --- a/release-notes/opensearch.release-notes-3.6.0.md +++ b/release-notes/opensearch.release-notes-3.6.0.md @@ -7,78 +7,80 @@ Compatible with OpenSearch and OpenSearch Dashboards version 3.6.0 * Add bitmap64 query support ([#20606](https://github.com/opensearch-project/OpenSearch/pull/20606)) * Add warmup phase for pull-based ingestion to prevent serving stale data before catching up with the streaming source ([#20526](https://github.com/opensearch-project/OpenSearch/pull/20526)) * Implement field_mapping ingestion message mapper for pull-based ingestion ([#20729](https://github.com/opensearch-project/OpenSearch/pull/20729)) +* Add mapper_settings support and field_mapping mapper type for pull-based ingestion ([#20722](https://github.com/opensearch-project/OpenSearch/pull/20722)) * Remove experimental tag for pull-based ingestion, marking it as public API ([#20704](https://github.com/opensearch-project/OpenSearch/pull/20704)) ### Enhancements * Add index warmer support for replica shards using segment replication ([#20650](https://github.com/opensearch-project/OpenSearch/pull/20650)) * Add indices to search request slow log for easier request identification ([#20588](https://github.com/opensearch-project/OpenSearch/pull/20588)) -* Add mapper_settings support and field_mapping mapper type for pull-based ingestion ([#20722](https://github.com/opensearch-project/OpenSearch/pull/20722)) -* Add new `Sensitive` setting property for tiering dynamic settings authorization ([#20901](https://github.com/opensearch-project/OpenSearch/pull/20901)) * Add node-level JVM and CPU runtime metrics following OpenTelemetry semantic conventions ([#20844](https://github.com/opensearch-project/OpenSearch/pull/20844)) +* Add new `Sensitive` setting property for tiering dynamic settings authorization ([#20901](https://github.com/opensearch-project/OpenSearch/pull/20901)) * Add adaptive shard selection for bulk writes on append-only indices ([#20065](https://github.com/opensearch-project/OpenSearch/pull/20065)) * Add support for expected remote cluster name validation in CCS sniff mode ([#20532](https://github.com/opensearch-project/OpenSearch/pull/20532)) -* Add Scroll API support for workload management rule-based autotagging ([#20151](https://github.com/opensearch-project/OpenSearch/pull/20151)) -* Add custom search settings support for workload management groups, starting with timeout ([#20536](https://github.com/opensearch-project/OpenSearch/pull/20536)) -* Expose wrapped scorer in `ProfileScorer` via `getWrappedScorer` method for plugin access ([#20549](https://github.com/opensearch-project/OpenSearch/pull/20549)) -* Fix `ProfileScorer.getChildren()` to expose wrapped scorer in the scorer tree hierarchy ([#20607](https://github.com/opensearch-project/OpenSearch/pull/20607)) -* Fallback to Netty HTTP client when AWS CRT client is unavailable on the target platform ([#20698](https://github.com/opensearch-project/OpenSearch/pull/20698)) +* Add `search_settings` support to WLM workload groups with initial `timeout` setting ([#20536](https://github.com/opensearch-project/OpenSearch/pull/20536)) +* Add scroll API support for workload management rule-based autotagging ([#20151](https://github.com/opensearch-project/OpenSearch/pull/20151)) +* Add stream request flag to SearchRequestContext for plugin consumption ([#20530](https://github.com/opensearch-project/OpenSearch/pull/20530)) * Support Docker distribution builds for ppc64le, arm64, and s390x architectures ([#20678](https://github.com/opensearch-project/OpenSearch/pull/20678)) -* Add intra-segment concurrency support for single-value metric aggregations ([#20503](https://github.com/opensearch-project/OpenSearch/pull/20503)) -* Add `ref_path` support for package-based Hunspell dictionary loading ([#20840](https://github.com/opensearch-project/OpenSearch/pull/20840)) -* Add streaming request flag to `SearchRequestContext` for plugin consumption ([#20530](https://github.com/opensearch-project/OpenSearch/pull/20530)) -* Make telemetry `Tags` immutable with sorted parallel arrays and precomputed hash for improved performance ([#20788](https://github.com/opensearch-project/OpenSearch/pull/20788)) -* Add indexer interface to decouple `IndexShard` from underlying engine implementations ([#20675](https://github.com/opensearch-project/OpenSearch/pull/20675)) +* Fallback to Netty HTTP client when AWS CRT client is unavailable on the target platform ([#20698](https://github.com/opensearch-project/OpenSearch/pull/20698)) +* Add intra-segment support for single-value metric aggregations (sum, min, max, avg, stats, cardinality, value_count) ([#20503](https://github.com/opensearch-project/OpenSearch/pull/20503)) +* Expose wrapped scorer in ProfileScorer for plugin access to custom scorer methods ([#20549](https://github.com/opensearch-project/OpenSearch/pull/20549)) +* Fix ProfileScorer.getChildren() to expose wrapped scorer in the scorer tree hierarchy ([#20607](https://github.com/opensearch-project/OpenSearch/pull/20607)) +* Remove X-Request-Id format restrictions and make maximum length configurable ([#21048](https://github.com/opensearch-project/OpenSearch/pull/21048)) +* Make telemetry Tags immutable with allocation-efficient factories and content-based equality ([#20788](https://github.com/opensearch-project/OpenSearch/pull/20788)) +* Add `ref_path` support for package-based Hunspell dictionary loading with multi-tenant isolation ([#20840](https://github.com/opensearch-project/OpenSearch/pull/20840)) * Prevent criteria field updates for context-aware indices to simplify version management ([#20250](https://github.com/opensearch-project/OpenSearch/pull/20250)) +* Add indexer interface to decouple IndexShard from Engine for pluggable engine architectures ([#20675](https://github.com/opensearch-project/OpenSearch/pull/20675)) +* Use `ReadAdviseByContext` for MMapDirectory instead of Lucene default read advise ([#21062](https://github.com/opensearch-project/OpenSearch/pull/21062)) ### Bug Fixes -* Add range validations in query builder and field mapper ([#20518](https://github.com/opensearch-project/OpenSearch/pull/20518)) -* Expose `STRICT_EXPAND_OPEN_HIDDEN_FORBID_CLOSED` index option via static method ([#20980](https://github.com/opensearch-project/OpenSearch/pull/20980)) -* Fix `opensearch.cgroups.hierarchy.override` causing `SecurityException` due to missing file permissions ([#20565](https://github.com/opensearch-project/OpenSearch/pull/20565)) -* Relax `updatedAt` validation for workload group creation to handle minor clock skew across nodes ([#20486](https://github.com/opensearch-project/OpenSearch/pull/20486)) -* Fix `copy_to` functionality for `geo_point` fields with object or array values ([#20542](https://github.com/opensearch-project/OpenSearch/pull/20542)) +* Fix `copy_to` functionality for geo_point fields with object and array values ([#20542](https://github.com/opensearch-project/OpenSearch/pull/20542)) * Fix `field_caps` returning empty results for `disable_objects` mappings and field name corruption ([#20814](https://github.com/opensearch-project/OpenSearch/pull/20814)) -* Fix `listBlobsByPrefixInSortedOrder` in `EncryptedBlobContainer` to respect limit and avoid loading all files into memory ([#20514](https://github.com/opensearch-project/OpenSearch/pull/20514)) * Fix terms lookup subquery to use cluster `max_clause_count` setting instead of hardcoded fallback ([#20823](https://github.com/opensearch-project/OpenSearch/pull/20823)) -* Fix terms aggregation performance regression on high-cardinality fields by introducing a max cardinality setting ([#20623](https://github.com/opensearch-project/OpenSearch/pull/20623)) -* Fix index template pattern collision detection producing false positives for multi-wildcard patterns ([#20702](https://github.com/opensearch-project/OpenSearch/pull/20702)) +* Fix terms aggregation performance regression on high-cardinality fields by adding a max cardinality setting ([#20623](https://github.com/opensearch-project/OpenSearch/pull/20623)) +* Fix terms aggregation performance regression using segment-to-global ordinals mapping ([#20683](https://github.com/opensearch-project/OpenSearch/pull/20683)) +* Fix `array_index_out_of_bounds_exception` with wildcard and aggregations under concurrent access ([#20842](https://github.com/opensearch-project/OpenSearch/pull/20842)) +* Add range validations in query builder and field mapper ([#20518](https://github.com/opensearch-project/OpenSearch/pull/20518)) +* Fix synonym_graph filter failure with word_delimiter_graph by handling analyzer dependency ordering ([#19248](https://github.com/opensearch-project/OpenSearch/pull/19248)) +* Fix index template pattern collision false positives for multi-wildcard patterns ([#20702](https://github.com/opensearch-project/OpenSearch/pull/20702)) +* Fix `SecurityException` when using `opensearch.cgroups.hierarchy.override` setting ([#20565](https://github.com/opensearch-project/OpenSearch/pull/20565)) +* Fix SLF4J component error caused by slf4j-api version mismatch with log4j binding ([#20587](https://github.com/opensearch-project/OpenSearch/pull/20587)) * Fix JSON escaping in task details log metadata ([#20802](https://github.com/opensearch-project/OpenSearch/pull/20802)) -* Fix SLF4J component error by updating to `log4j-slf4j2-impl` for SLF4J 2.x compatibility ([#20587](https://github.com/opensearch-project/OpenSearch/pull/20587)) -* Fix `array_index_out_of_bounds_exception` with wildcard fields and aggregations under concurrent access ([#20842](https://github.com/opensearch-project/OpenSearch/pull/20842)) -* Delegate `getMin`/`getMax` methods for `ExitableTerms` to avoid slow path in field sort ([#20775](https://github.com/opensearch-project/OpenSearch/pull/20775)) -* Fix `CriteriaBasedCodec` to work with delegate codec ([#20442](https://github.com/opensearch-project/OpenSearch/pull/20442)) -* Fix synonym_graph filter failure with word_delimiter_graph when using custom synonym analyzers ([#19248](https://github.com/opensearch-project/OpenSearch/pull/19248)) -* Improve exception messaging when encountering legacy Elasticsearch version IDs during upgrade ([#20512](https://github.com/opensearch-project/OpenSearch/pull/20512)) +* Fix `listBlobsByPrefixInSortedOrder` in EncryptedBlobContainer to respect limit and prevent JVM exhaustion ([#20514](https://github.com/opensearch-project/OpenSearch/pull/20514)) +* Fix batched deletion of stale cluster metadata manifests to prevent remote storage pile-up ([#20566](https://github.com/opensearch-project/OpenSearch/pull/20566)) * Fix segment replication infinite retry caused by stale metadata checkpoint ([#20551](https://github.com/opensearch-project/OpenSearch/pull/20551)) -* Show heap percent threshold instead of moving-average value in search cancellation messages ([#20779](https://github.com/opensearch-project/OpenSearch/pull/20779)) -* Defer stored field reader initialization in `SourceLookup` to avoid unnecessary `madvise` calls ([#20827](https://github.com/opensearch-project/OpenSearch/pull/20827)) -* Fix terms aggregation performance regression by leveraging segment-to-global ordinal mapping ([#20683](https://github.com/opensearch-project/OpenSearch/pull/20683)) -* Fix batched deletion of stale cluster metadata manifests in remote cluster state cleanup ([#20566](https://github.com/opensearch-project/OpenSearch/pull/20566)) -* Fix stale segments cleanup to also trigger based on map size threshold ([#20976](https://github.com/opensearch-project/OpenSearch/pull/20976)) +* Fix `ExitableTerms` to delegate `getMin`/`getMax` methods avoiding slow path in field sort ([#20775](https://github.com/opensearch-project/OpenSearch/pull/20775)) +* Lazily initialize stored field reader in SourceLookup to avoid unnecessary `madvise` calls ([#20827](https://github.com/opensearch-project/OpenSearch/pull/20827)) +* Fix CriteriaBasedCodec to work with delegate codec ([#20442](https://github.com/opensearch-project/OpenSearch/pull/20442)) +* Show heap percent threshold instead of moving average in search cancellation messages ([#20779](https://github.com/opensearch-project/OpenSearch/pull/20779)) +* Relax `updatedAt` validation for WLM workload group creation to handle minor clock skew ([#20486](https://github.com/opensearch-project/OpenSearch/pull/20486)) +* Improve exception messaging when encountering legacy Elasticsearch version IDs during upgrade ([#20512](https://github.com/opensearch-project/OpenSearch/pull/20512)) * Fix Windows service startup with OpenJDK by updating procrun to 1.5.1 ([#20615](https://github.com/opensearch-project/OpenSearch/pull/20615)) -* Move `Randomness` from server to `lib/common` to reduce FIPS demo installer size ([#20570](https://github.com/opensearch-project/OpenSearch/pull/20570)) +* Expose `STRICT_EXPAND_OPEN_HIDDEN_FORBID_CLOSED` index option via static method ([#20980](https://github.com/opensearch-project/OpenSearch/pull/20980)) +* Fix stale segments cleanup to also trigger based on segments map size threshold ([#20976](https://github.com/opensearch-project/OpenSearch/pull/20976)) +* Move `Randomness` from server to lib/common to reduce FIPS demo installer size ([#20570](https://github.com/opensearch-project/OpenSearch/pull/20570)) * Add `accessUnixDomainSocket` permission for transport-grpc on Windows ([#20649](https://github.com/opensearch-project/OpenSearch/pull/20649)) -* Harden HTTP/3 support detection by verifying Quic native library availability for the target platform ([#20680](https://github.com/opensearch-project/OpenSearch/pull/20680)) +* Harden HTTP/3 support detection by checking Quic native library availability for the target platform ([#20680](https://github.com/opensearch-project/OpenSearch/pull/20680)) * Use `OPENSEARCH_FIPS_MODE` environment variable to enable FIPS enforced mode instead of checking for bcFIPS jars ([#20625](https://github.com/opensearch-project/OpenSearch/pull/20625)) -* Update MMapDirectory to use `ReadAdviseByContext` rather than default read advise ([#21062](https://github.com/opensearch-project/OpenSearch/pull/21062)) -* Ensure transient `ThreadContext` headers with propagators survive context restore ([#20854](https://github.com/opensearch-project/OpenSearch/pull/20854)) +* Fix Arrow Flight transport TLS certificate hot-reload ([#20700](https://github.com/opensearch-project/OpenSearch/pull/20700)) +* Fix stream transport TLS certificate hot-reload by using live SSLContext from SecureTransportSettingsProvider ([#20734](https://github.com/opensearch-project/OpenSearch/pull/20734)) +* Ensure transient ThreadContext headers with propagators survive context restore ([#20854](https://github.com/opensearch-project/OpenSearch/pull/20854)) ### Maintenance * Bump Apache Lucene from 10.3.2 to 10.4.0 ([#20735](https://github.com/opensearch-project/OpenSearch/pull/20735)) -* Bump OpenTelemetry to 1.60.1 and OpenTelemetry Semconv to 1.40.0 ([#20737](https://github.com/opensearch-project/OpenSearch/pull/20737), [#20797](https://github.com/opensearch-project/OpenSearch/pull/20797)) -* Bump Project Reactor to 3.8.4 and Reactor Netty to 1.3.4 ([#20589](https://github.com/opensearch-project/OpenSearch/pull/20589), [#20834](https://github.com/opensearch-project/OpenSearch/pull/20834)) -* Bump Netty to 4.2.12.Final ([#20586](https://github.com/opensearch-project/OpenSearch/pull/20586), [#20997](https://github.com/opensearch-project/OpenSearch/pull/20997), [#20998](https://github.com/opensearch-project/OpenSearch/pull/20998)) +* Update Netty to 4.2.12.Final ([#20586](https://github.com/opensearch-project/OpenSearch/pull/20586)), ([#20997](https://github.com/opensearch-project/OpenSearch/pull/20997)), ([#20998](https://github.com/opensearch-project/OpenSearch/pull/20998)) * Update Jackson to 2.21.2 ([#20989](https://github.com/opensearch-project/OpenSearch/pull/20989)) -* Bump `ch.qos.logback:logback-classic` to 1.5.32 and `logback-core` to 1.5.27 ([#20525](https://github.com/opensearch-project/OpenSearch/pull/20525), [#20761](https://github.com/opensearch-project/OpenSearch/pull/20761)) +* Bump OpenTelemetry to 1.60.1 and OpenTelemetry Semconv to 1.40.0 ([#20737](https://github.com/opensearch-project/OpenSearch/pull/20737)), ([#20797](https://github.com/opensearch-project/OpenSearch/pull/20797)) +* Bump Project Reactor to 3.8.4 and Reactor Netty to 1.3.4 ([#20589](https://github.com/opensearch-project/OpenSearch/pull/20589)), ([#20834](https://github.com/opensearch-project/OpenSearch/pull/20834)) +* Bump `ch.qos.logback:logback-classic` to 1.5.32 and `logback-core` to 1.5.27 ([#20525](https://github.com/opensearch-project/OpenSearch/pull/20525)), ([#20761](https://github.com/opensearch-project/OpenSearch/pull/20761)) * Bump `com.nimbusds:nimbus-jose-jwt` from 10.7 to 10.8 ([#20715](https://github.com/opensearch-project/OpenSearch/pull/20715)) * Bump `com.sun.xml.bind:jaxb-impl` from 4.0.6 to 4.0.7 ([#20886](https://github.com/opensearch-project/OpenSearch/pull/20886)) * Bump `com.netflix.nebula.ospackage-base` from 12.2.0 to 12.3.0 ([#20799](https://github.com/opensearch-project/OpenSearch/pull/20799)) * Bump `org.apache.commons:commons-text` from 1.14.0 to 1.15.0 ([#20576](https://github.com/opensearch-project/OpenSearch/pull/20576)) * Bump `org.jline:jline` from 3.30.6 to 4.0.0 ([#20800](https://github.com/opensearch-project/OpenSearch/pull/20800)) * Bump `org.jruby.jcodings:jcodings` from 1.0.63 to 1.0.64 ([#20713](https://github.com/opensearch-project/OpenSearch/pull/20713)) -* Bump `org.jruby.joni:joni` from 2.2.3 to 2.2.7 ([#20714](https://github.com/opensearch-project/OpenSearch/pull/20714), [#20759](https://github.com/opensearch-project/OpenSearch/pull/20759)) +* Bump `org.jruby.joni:joni` from 2.2.3 to 2.2.7 ([#20714](https://github.com/opensearch-project/OpenSearch/pull/20714)), ([#20759](https://github.com/opensearch-project/OpenSearch/pull/20759)) * Bump `org.tukaani:xz` from 1.11 to 1.12 ([#20760](https://github.com/opensearch-project/OpenSearch/pull/20760)) -* Bump `shadow-gradle-plugin` from 8.3.9 to 9.3.1 ([#20569](https://github.com/opensearch-project/OpenSearch/pull/20569)) -* Bump `com.netflix.nebula:gradle-info-plugin` to 16.2.1 ([#20825](https://github.com/opensearch-project/OpenSearch/pull/20825)) +* Bump shadow-gradle-plugin from 8.3.9 to 9.3.1 ([#20569](https://github.com/opensearch-project/OpenSearch/pull/20569))