diff --git a/release-notes/opensearch.release-notes-3.7.0.md b/release-notes/opensearch.release-notes-3.7.0.md new file mode 100644 index 0000000000000..2234a4b81a5af --- /dev/null +++ b/release-notes/opensearch.release-notes-3.7.0.md @@ -0,0 +1,160 @@ +## Version 3.7.0 Release Notes + +Compatible with OpenSearch and OpenSearch Dashboards version 3.7.0 + +### Features + +* Add dynamic properties support for pattern-based field definitions without cluster state mapping updates ([#20816](https://github.com/opensearch-project/OpenSearch/pull/20816)) +* Add pluggable data format engine with DataFormatAwareEngine for multi-format indexing ([#21181](https://github.com/opensearch-project/OpenSearch/pull/21181)) +* Add Lucene engine implementation for pluggable data formats ([#21299](https://github.com/opensearch-project/OpenSearch/pull/21299)) +* Add merge support for Parquet data format plugin via streaming k-way merge sort ([#21079](https://github.com/opensearch-project/OpenSearch/pull/21079)) +* Add directory and IndexInput layers for WritableWarm tiered storage ([#21178](https://github.com/opensearch-project/OpenSearch/pull/21178)) +* Add server-side implementation for tiering status APIs (GetTieringStatus and ListTieringStatus) ([#21220](https://github.com/opensearch-project/OpenSearch/pull/21220)) +* Add server-side implementation for HotToWarm, WarmToHot, and CancelTiering APIs ([#21295](https://github.com/opensearch-project/OpenSearch/pull/21295)) +* Add prefetch settings and stored fields prefetch for WritableWarm tiered storage ([#21285](https://github.com/opensearch-project/OpenSearch/pull/21285)) +* Add slow logs, per-query metrics, and migration metrics for WritableWarm tiered storage ([#21332](https://github.com/opensearch-project/OpenSearch/pull/21332)) +* Add module wiring and integration tests for WritableWarm tiered storage ([#21427](https://github.com/opensearch-project/OpenSearch/pull/21427)) +* Add tiered object storage crate for warm node file routing ([#21204](https://github.com/opensearch-project/OpenSearch/pull/21204)) +* Add event-driven scheduler and stage execution for analytics engine ([#21242](https://github.com/opensearch-project/OpenSearch/pull/21242)) +* Add coordinator-side DataFusion reduce with streaming Arrow batches ([#21356](https://github.com/opensearch-project/OpenSearch/pull/21356)) +* Add distributed aggregation with partial/final mode for analytics engine ([#21457](https://github.com/opensearch-project/OpenSearch/pull/21457)) +* Add distributed join planning and execution for analytics engine ([#21639](https://github.com/opensearch-project/OpenSearch/pull/21639)) +* Add PPL `append` command support with multi-child stage runtime for Union ([#21474](https://github.com/opensearch-project/OpenSearch/pull/21474)) +* Add PPL `dedup` command support via ROW_NUMBER window function ([#21622](https://github.com/opensearch-project/OpenSearch/pull/21622)) +* Add PPL `eventstats` and `streamstats` window function support ([#21734](https://github.com/opensearch-project/OpenSearch/pull/21734)) +* Add PPL `top` and `rare` command support via window functions ([#21593](https://github.com/opensearch-project/OpenSearch/pull/21593)) +* Add PPL `parse` command with regex mode via Rust UDFs ([#21573](https://github.com/opensearch-project/OpenSearch/pull/21573)) +* Add PPL `rex` command with sed and extract modes ([#21550](https://github.com/opensearch-project/OpenSearch/pull/21550)) +* Add PPL `spath` command with auto-extract mode via json_extract_all UDF ([#21664](https://github.com/opensearch-project/OpenSearch/pull/21664)) +* Add 7 PPL JSON scalar functions to analytics engine route ([#21513](https://github.com/opensearch-project/OpenSearch/pull/21513)) +* Add 23 PPL datetime scalar functions to analytics engine route ([#21556](https://github.com/opensearch-project/OpenSearch/pull/21556)) +* Add 14 additional PPL datetime functions (Wave A) including strftime, date_format, maketime ([#21582](https://github.com/opensearch-project/OpenSearch/pull/21582)) +* Add 30+ PPL math scalar functions to analytics engine ([#21520](https://github.com/opensearch-project/OpenSearch/pull/21520)) +* Add PPL string scalar functions to analytics engine (18 functions) ([#21543](https://github.com/opensearch-project/OpenSearch/pull/21543)) +* Add PPL conditional functions (coalesce, isempty, isblank, case, if, ifnull) to analytics engine ([#21643](https://github.com/opensearch-project/OpenSearch/pull/21643)) +* Add PPL conversion scalar functions (num, auto, memk, rmcomma, dur2sec, ctime, mktime) to analytics engine ([#21628](https://github.com/opensearch-project/OpenSearch/pull/21628)) +* Add PPL cryptographic functions (md5, sha1, sha2, crc32) to analytics engine ([#21611](https://github.com/opensearch-project/OpenSearch/pull/21611)) +* Add PPL array constructor and 8 multivalue functions to analytics engine ([#21554](https://github.com/opensearch-project/OpenSearch/pull/21554)) +* Add PPL bucketing scalars (span_bucket, width_bucket, minspan_bucket, range_bucket) ([#21621](https://github.com/opensearch-project/OpenSearch/pull/21621)) +* Add PPL TAKE, FIRST, LAST, LIST, VALUES aggregate functions ([#21731](https://github.com/opensearch-project/OpenSearch/pull/21731)) +* Add Lucene filter delegation from DataFusion for full-text search predicates ([#21555](https://github.com/opensearch-project/OpenSearch/pull/21555)) +* Add performance delegation to Lucene for selective filter predicates ([#21701](https://github.com/opensearch-project/OpenSearch/pull/21701)) +* Add native Arrow transport path with zero-copy transfer for stream transport ([#21253](https://github.com/opensearch-project/OpenSearch/pull/21253)) +* Stream Arrow batches on data-node fragment execution path ([#21418](https://github.com/opensearch-project/OpenSearch/pull/21418)) +* Add support for `extra_fields` outside `_source` indexing for improved vector ingestion throughput ([#20635](https://github.com/opensearch-project/OpenSearch/pull/20635)) +* Add gRPC support for Min, Max, and Terms aggregations ([#21205](https://github.com/opensearch-project/OpenSearch/pull/21205)) +* Add partition strategy setting for flexible shard-to-partition mapping in pull-based ingestion ([#21165](https://github.com/opensearch-project/OpenSearch/pull/21165)) +* Add SplitToFieldsProcessor for distributing split values to target fields ([#21216](https://github.com/opensearch-project/OpenSearch/pull/21216)) +* Add native memory based admission control for transport request throttling ([#21191](https://github.com/opensearch-project/OpenSearch/pull/21191)) +* Add native memory search backpressure for off-heap query cancellation ([#21647](https://github.com/opensearch-project/OpenSearch/pull/21647)) +* Add unified native allocator framework for Arrow allocations with elastic rebalancing ([#21703](https://github.com/opensearch-project/OpenSearch/pull/21703)) +* Add on-demand jemalloc heap profiling support via JMX CLI tool ([#21599](https://github.com/opensearch-project/OpenSearch/pull/21599)) +* Add `search.max_buckets` to workload group settings for per-tenant bucket limits ([#21721](https://github.com/opensearch-project/OpenSearch/pull/21721)) +* Add additional search settings and `override_request_values` to workload management groups ([#21523](https://github.com/opensearch-project/OpenSearch/pull/21523)) +* Add hunspell dictionary hot-reload support via `_refresh_search_analyzers` API ([#21559](https://github.com/opensearch-project/OpenSearch/pull/21559)) + +### Enhancements + +* Add adaptive query budget for DataFusion engine with bounded memory and improved throughput ([#21695](https://github.com/opensearch-project/OpenSearch/pull/21695)) +* Add DynamicLimitPool for runtime memory pool limit changes in DataFusion ([#21286](https://github.com/opensearch-project/OpenSearch/pull/21286)) +* Add configurable coordinator buffer limit for per-query Arrow allocator ([#21726](https://github.com/opensearch-project/OpenSearch/pull/21726)) +* Add CPU task cancellation for DataFusion queries ([#21560](https://github.com/opensearch-project/OpenSearch/pull/21560)) +* Add IO task cancellation support for DataFusion queries ([#21531](https://github.com/opensearch-project/OpenSearch/pull/21531)) +* Add DataFusion logical and physical plan logging at DEBUG level ([#21646](https://github.com/opensearch-project/OpenSearch/pull/21646)) +* Add dynamic settings for indexed query execution path ([#21522](https://github.com/opensearch-project/OpenSearch/pull/21522)) +* Add dedicated `analytics_scheduler` thread pool to prevent coordinator deadlock ([#21771](https://github.com/opensearch-project/OpenSearch/pull/21771)) +* Add dedicated `analytics_reduce` thread pool for coordinator reduce drains ([#21800](https://github.com/opensearch-project/OpenSearch/pull/21800)) +* Add native memory stats and task cancellation stats to node stats API ([#21637](https://github.com/opensearch-project/OpenSearch/pull/21637)) +* Add `current_application_duration_ms` to cluster state download stats in node stats API ([#20922](https://github.com/opensearch-project/OpenSearch/pull/20922)) +* Add segments and segment stats support for DataFormatAwareEngine ([#21696](https://github.com/opensearch-project/OpenSearch/pull/21696)) +* Add DataFormat-aware NRT replication engine and remote-store wiring ([#21311](https://github.com/opensearch-project/OpenSearch/pull/21311)) +* Add DataFormat-aware shallow snapshot v2 support ([#21742](https://github.com/opensearch-project/OpenSearch/pull/21742)) +* Add DataFormat-aware read-only engine for warm primaries with tiering service improvements ([#21720](https://github.com/opensearch-project/OpenSearch/pull/21720)) +* Add dynamic mapping support for pluggable data formats ([#21444](https://github.com/opensearch-project/OpenSearch/pull/21444)) +* Add delete execution engine abstraction for DataFormatAwareEngine ([#21313](https://github.com/opensearch-project/OpenSearch/pull/21313)) +* Add cluster-scope defaults for pluggable dataformat settings ([#21435](https://github.com/opensearch-project/OpenSearch/pull/21435)) +* Add indexing support for metadata fields in pluggable data formats ([#21585](https://github.com/opensearch-project/OpenSearch/pull/21585)) +* Add Lucene merge support for pluggable data format composite engine ([#21422](https://github.com/opensearch-project/OpenSearch/pull/21422)) +* Add composite merge handler and merge policy for data-format-aware engine ([#21128](https://github.com/opensearch-project/OpenSearch/pull/21128)) +* Add sort-on-refresh for composite engine with cross-format row-ID consistency ([#21468](https://github.com/opensearch-project/OpenSearch/pull/21468)) +* Add warm+format directory wiring with per-format tiered directory routing ([#21361](https://github.com/opensearch-project/OpenSearch/pull/21361)) +* Add block cache SPI and Foyer plugin for warm nodes ([#21530](https://github.com/opensearch-project/OpenSearch/pull/21530)) +* Add REST API paths for block cache prune and detailed file cache stats ([#21705](https://github.com/opensearch-project/OpenSearch/pull/21705)) +* Add cancellation checkpoints in field data loading and aggregation paths ([#21318](https://github.com/opensearch-project/OpenSearch/pull/21318)) +* Add `queryTimeout` to IndexSearcher for KNN vector search timeout enforcement ([#21316](https://github.com/opensearch-project/OpenSearch/pull/21316)) +* Add index-level authorization to analytics engine via ActionFilter dispatch ([#21789](https://github.com/opensearch-project/OpenSearch/pull/21789)) +* Add `/_analytics/ppl/_explain` endpoint with stage profiling ([#21660](https://github.com/opensearch-project/OpenSearch/pull/21660)) +* Add relevance function support (match_phrase, multi_match, query_string, etc.) to analytics engine ([#21562](https://github.com/opensearch-project/OpenSearch/pull/21562)) +* Add relevance functions optional parameter support and new functions (wildcard_query, query, match_all) ([#21661](https://github.com/opensearch-project/OpenSearch/pull/21661)) +* Add filter pushdown rules and Calcite rule metrics for profiling ([#21684](https://github.com/opensearch-project/OpenSearch/pull/21684)) +* Add per-column encoding and compression configuration for Parquet data format ([#21665](https://github.com/opensearch-project/OpenSearch/pull/21665)) +* Avoid repeated encoding and compression for sort column writes in Parquet ([#21464](https://github.com/opensearch-project/OpenSearch/pull/21464)) +* Add pipeline execution metrics to PollingIngestStats for pull-based ingestion ([#21024](https://github.com/opensearch-project/OpenSearch/pull/21024)) +* Add batching for persistent task cluster service to reduce cluster manager load ([#21245](https://github.com/opensearch-project/OpenSearch/pull/21245)) +* Refactor BitsetFilterCache to node-level cache with configurable size limit ([#21179](https://github.com/opensearch-project/OpenSearch/pull/21179)) +* Skip zone awareness when `auto_expand_replicas` is set to all ([#21217](https://github.com/opensearch-project/OpenSearch/pull/21217)) +* Relax field-level meta validation constraints to allow any number of entries with string values ([#20578](https://github.com/opensearch-project/OpenSearch/pull/20578)) +* Deprecate boolean constructor of FetchSourceContext in favor of static constants ([#21235](https://github.com/opensearch-project/OpenSearch/pull/21235)) +* Add validation and deprecation warnings for ambiguous `_source` filtering ([#21203](https://github.com/opensearch-project/OpenSearch/pull/21203)) +* Speed up Painless Script Engine initialization by ~10% ([#21463](https://github.com/opensearch-project/OpenSearch/pull/21463)) +* Fix accumulation of file sizes when multiple files share the same extension in segment stats ([#21000](https://github.com/opensearch-project/OpenSearch/pull/21000)) +* Improve native memory admission control precision with auto-derived budget and JVM non-heap subtraction ([#21749](https://github.com/opensearch-project/OpenSearch/pull/21749)) +* Tighten DataFusion memory guard with RSS-based hard guard to prevent OOM under concurrent load ([#21814](https://github.com/opensearch-project/OpenSearch/pull/21814)) +* Support `indices_boost_2` array format for gRPC search ([#21300](https://github.com/opensearch-project/OpenSearch/pull/21300)) +* Add configurable Kafka metadata timeout for pull-based ingestion ([#21425](https://github.com/opensearch-project/OpenSearch/pull/21425)) +* Expose tokio-metrics as DataFusion plugin stats ([#21303](https://github.com/opensearch-project/OpenSearch/pull/21303)) +* Add Lucene FFM callbacks to task resource tracking ([#21610](https://github.com/opensearch-project/OpenSearch/pull/21610)) + +### Bug Fixes + +* Fix YAML parser corrupting string values that resemble booleans after Jackson 3.x migration ([#21294](https://github.com/opensearch-project/OpenSearch/pull/21294)) +* Fix `map_unmapped_fields_as_text` lost after dynamic mapping update in PercolatorFieldMapper ([#21301](https://github.com/opensearch-project/OpenSearch/pull/21301)) +* Fix O(n²) removeAll in remote translog metadata cleanup causing CPU spikes ([#21350](https://github.com/opensearch-project/OpenSearch/pull/21350)) +* Fix `Rounding.isUTC()` to recognize UTC timezone aliases for date histogram optimization ([#21221](https://github.com/opensearch-project/OpenSearch/pull/21221)) +* Fix NPE in QueryPhaseResultConsumer when all shards fail ([#21158](https://github.com/opensearch-project/OpenSearch/pull/21158)) +* Fix bulk request hang when index is deleted during primary phase ([#21305](https://github.com/opensearch-project/OpenSearch/pull/21305)) +* Fix deadlock between engineMutex and writeLock during index close and engine reset ([#21404](https://github.com/opensearch-project/OpenSearch/pull/21404)) +* Fix FlightOutboundHandler clearing caller's ThreadContext ([#21167](https://github.com/opensearch-project/OpenSearch/pull/21167)) +* Fix `IndicesRequestCacheCleanupIT` flakiness by removing too-short assertBusy timeouts ([#21494](https://github.com/opensearch-project/OpenSearch/pull/21494)) +* Fix negative fielddata stats by guarding against stale removals after shard reallocation ([#21667](https://github.com/opensearch-project/OpenSearch/pull/21667)) +* Fix `half_float` ingest writing wrong fp16 bit pattern in Parquet ([#21783](https://github.com/opensearch-project/OpenSearch/pull/21783)) +* Fix StringView buffer bloat in DataFusion stream_next FFI export causing 435x data amplification ([#21753](https://github.com/opensearch-project/OpenSearch/pull/21753)) +* Fix Utf8View/Utf8 schema mismatch panic in indexed parquet path ([#21826](https://github.com/opensearch-project/OpenSearch/pull/21826)) +* Fix memory leak in `transport-reactor-netty4` plugin with persistent connections ([#21788](https://github.com/opensearch-project/OpenSearch/pull/21788)) +* Fix `ExitablePostingsEnum` to extend `FilterPostingsEnum` for proper delegation ([#21558](https://github.com/opensearch-project/OpenSearch/pull/21558)) +* Fix local recovery from flush for DataFormatAwareEngine ([#21553](https://github.com/opensearch-project/OpenSearch/pull/21553)) +* Fix safe-commit info and replication checksum for DFA shards ([#21787](https://github.com/opensearch-project/OpenSearch/pull/21787)) +* Fix DFA recovery failures: file-handle leak and reset-path crash ([#21759](https://github.com/opensearch-project/OpenSearch/pull/21759)) +* Handle null scripted metric combine results ([#21534](https://github.com/opensearch-project/OpenSearch/pull/21534)) +* Demote "No resource usage stats available for node" log from WARN to DEBUG ([#21638](https://github.com/opensearch-project/OpenSearch/pull/21638)) +* Fix pull-based ingestion document mapper usage to reflect mapping updates ([#21183](https://github.com/opensearch-project/OpenSearch/pull/21183)) +* Fix pull-based ingestion consumer factory to be stateless and prevent race conditions ([#21652](https://github.com/opensearch-project/OpenSearch/pull/21652)) +* Fix pull-based ingestion multi-threaded writer batchStartPointer computation ([#21697](https://github.com/opensearch-project/OpenSearch/pull/21697)) +* Fix Netty4Http3ServerTransport to use configured HeaderVerifier and Decompressor instances ([#21281](https://github.com/opensearch-project/OpenSearch/pull/21281)) +* Convert varchar to str in analytics engine Project operations to fix DataFusion type errors ([#21794](https://github.com/opensearch-project/OpenSearch/pull/21794)) +* Fix `microsecond()` function and add timestamp lower-bound validation in analytics engine ([#21793](https://github.com/opensearch-project/OpenSearch/pull/21793)) +* Enforce write blocks for DFA hot-to-warm tiering to survive DiskThresholdMonitor removal ([#21828](https://github.com/opensearch-project/OpenSearch/pull/21828)) + +### Maintenance + +* Bump Netty to 4.2.14.Final ([#21772](https://github.com/opensearch-project/OpenSearch/pull/21772)) +* Update Jackson to 2.21.3 / 3.1.3 ([#21493](https://github.com/opensearch-project/OpenSearch/pull/21493)) +* Update ASM to 9.10 ([#21764](https://github.com/opensearch-project/OpenSearch/pull/21764)) +* Update OpenTelemetry to 1.62.0 and SemConv to 1.41.0 ([#21595](https://github.com/opensearch-project/OpenSearch/pull/21595)) +* Update Project Reactor to 3.8.5 and Reactor Netty to 1.3.5 ([#21226](https://github.com/opensearch-project/OpenSearch/pull/21226)) +* Update bundled JDK to JDK 25.0.3 ([#21353](https://github.com/opensearch-project/OpenSearch/pull/21353)) +* Update log4j2 to 2.25.4 ([#21416](https://github.com/opensearch-project/OpenSearch/pull/21416)) +* Update `httpclient5` to 5.6.1 ([#21441](https://github.com/opensearch-project/OpenSearch/pull/21441)) +* Bump commons-configuration2 from 2.14.0 to 2.15.0 ([#21806](https://github.com/opensearch-project/OpenSearch/pull/21806)) +* Bump org.apache.commons:commons-configuration2 from 2.13.0 to 2.14.0 ([#21213](https://github.com/opensearch-project/OpenSearch/pull/21213)) +* Bump com.google.protobuf from 0.9.6 to 0.10.0 ([#21291](https://github.com/opensearch-project/OpenSearch/pull/21291)) +* Bump org.apache.hadoop:hadoop-minicluster from 3.4.2 to 3.5.0 ([#21138](https://github.com/opensearch-project/OpenSearch/pull/21138)) +* Bump org.codehaus.woodstox:stax2-api from 4.2.2 to 4.3.0 ([#21137](https://github.com/opensearch-project/OpenSearch/pull/21137)) +* Bump org.jline:jline from 4.0.0 to 4.0.14 ([#21471](https://github.com/opensearch-project/OpenSearch/pull/21471)) +* Bump org.jsoup:jsoup from 1.22.1 to 1.22.2 ([#21290](https://github.com/opensearch-project/OpenSearch/pull/21290)) +* Bump com.nimbusds:nimbus-jose-jwt from 10.8 to 10.9 ([#21214](https://github.com/opensearch-project/OpenSearch/pull/21214)) +* Remove Unsafe class injection from Java agent ([#21542](https://github.com/opensearch-project/OpenSearch/pull/21542)) +* Replace mimalloc with jemalloc as global allocator for native sandbox plugins ([#21497](https://github.com/opensearch-project/OpenSearch/pull/21497)) +* Upgrade DataFusion to v53 and Arrow to v58 ([#21590](https://github.com/opensearch-project/OpenSearch/pull/21590)) +* Pin GitHub Actions to commit SHAs for supply chain security ([#21808](https://github.com/opensearch-project/OpenSearch/pull/21808)) +* Update FIPS bootstrap check to use OpenSearch env var instead of BouncyCastle system property ([#21415](https://github.com/opensearch-project/OpenSearch/pull/21415))