diff --git a/release-notes/opensearch-sql.release-notes-3.3.0.0.md b/release-notes/opensearch-sql.release-notes-3.3.0.0.md index ee0eb83b902..7a11f50b899 100644 --- a/release-notes/opensearch-sql.release-notes-3.3.0.0.md +++ b/release-notes/opensearch-sql.release-notes-3.3.0.0.md @@ -3,84 +3,122 @@ Compatible with OpenSearch and OpenSearch Dashboards version 3.3.0 ### Features +* Add EARLIEST/LATEST aggregate functions for PPL ([#4100](https://github.com/opensearch-project/sql/pull/4100)) +* Add earliest/latest aggregate function for eventstats PPL command ([#4212](https://github.com/opensearch-project/sql/pull/4212)) +* Add max/min eval functions ([#4333](https://github.com/opensearch-project/sql/pull/4333)) +* [Feature] Core Implementation of `rex` Command In PPL ([#4109](https://github.com/opensearch-project/sql/pull/4109)) +* [Feature] Implementation of `regex` Command In PPL ([#4083](https://github.com/opensearch-project/sql/pull/4083)) +* [Feature] Implementation of mode `sed` and `offset_field` in rex PPL command ([#4241](https://github.com/opensearch-project/sql/pull/4241)) +* Support ```bin``` command with Calcite ([#3878](https://github.com/opensearch-project/sql/pull/3878)) +* Support timechart command with Calcite ([#3993](https://github.com/opensearch-project/sql/pull/3993)) +* Implement `Append` command with Calcite ([#4123](https://github.com/opensearch-project/sql/pull/4123)) +* Starter implementation for `spath` command ([#4120](https://github.com/opensearch-project/sql/pull/4120)) +* [Experimental] Support direct query data sources ([#4375](https://github.com/opensearch-project/sql/pull/4375)) + +### Enhancements +* Add `values` stats function with UDAF ([#4276](https://github.com/opensearch-project/sql/pull/4276)) +* Add asc keyword to sort command ([#4113](https://github.com/opensearch-project/sql/pull/4113)) +* Add count alias to count() ([#4157](https://github.com/opensearch-project/sql/pull/4157)) +* Add non-numeric field support for max/min functions ([#4281](https://github.com/opensearch-project/sql/pull/4281)) +* Add regex_match function for PPL with Calcite engine support ([#4092](https://github.com/opensearch-project/sql/pull/4092)) +* Add shortcut for count() ([#4142](https://github.com/opensearch-project/sql/pull/4142)) +* Add support for `list()` multi-value stats function ([#4161](https://github.com/opensearch-project/sql/pull/4161)) +* Add support for `median()` ([#4234](https://github.com/opensearch-project/sql/pull/4234)) +* Add wildcard support for rename command ([#4019](https://github.com/opensearch-project/sql/pull/4019)) +* Allow type checkering on nested & struct fields for isnull, isnotnull, and ispresent ([#4044](https://github.com/opensearch-project/sql/pull/4044)) * Change the default search sort tiebreaker to `_shard_doc` for PIT search ([#4378](https://github.com/opensearch-project/sql/pull/4378)) -* Support direct query data sources ([#4375](https://github.com/opensearch-project/sql/pull/4375)) +* Date/Time based Span aggregation should always not present null bucket ([#4327](https://github.com/opensearch-project/sql/pull/4327)) +* Dynamic source selector in PPL Grammar. ([#4116](https://github.com/opensearch-project/sql/pull/4116)) * Enable Calcite by default and implicit fallback the unsupported commands ([#4372](https://github.com/opensearch-project/sql/pull/4372)) +* Enable pushdown optimization for filtered aggregation ([#4213](https://github.com/opensearch-project/sql/pull/4213)) * Enhance the cost computing mechanism and push down context ([#4353](https://github.com/opensearch-project/sql/pull/4353)) -* Add error handling for known limitation of sql `JOIN` ([#4344](https://github.com/opensearch-project/sql/pull/4344)) +* [Feature][Enhancement] Enhance patterns command with additional sample_logs output field ([#4155](https://github.com/opensearch-project/sql/pull/4155)) +* Make fields optional parameter in multi field relevance function. ([#4018](https://github.com/opensearch-project/sql/pull/4018)) * Optimize count aggregation performance by utilizing native doc_count in v3 ([#4337](https://github.com/opensearch-project/sql/pull/4337)) -* Date/Time based Span aggregation should always not present null bucket ([#4327](https://github.com/opensearch-project/sql/pull/4327)) -* Add non-numeric field support for max/min functions ([#4281](https://github.com/opensearch-project/sql/pull/4281)) -* Push down project operator with non-identity projections into scan ([#4279](https://github.com/opensearch-project/sql/pull/4279)) -* Add `values` stats function with UDAF ([#4276](https://github.com/opensearch-project/sql/pull/4276)) -* Support ISO8601-formatted string in PPL ([#4246](https://github.com/opensearch-project/sql/pull/4246)) +* PPL Coalesce Function Enhancement for Calcite - Advanced Type Handling and Field Support ([#4041](https://github.com/opensearch-project/sql/pull/4041)) +* PPL percentile function shortcut `perc()` and `p()` support ([#4085](https://github.com/opensearch-project/sql/pull/4085)) * Push down limit operator into aggregation bucket size ([#4228](https://github.com/opensearch-project/sql/pull/4228)) -* Support time modifiers in search command ([#4224](https://github.com/opensearch-project/sql/pull/4224)) -* Support first/last aggregate functions for PPL ([#4223](https://github.com/opensearch-project/sql/pull/4223)) -* `mvjoin` support in PPL Caclite ([#4217](https://github.com/opensearch-project/sql/pull/4217)) -* Enable pushdown optimization for filtered aggregation ([#4213](https://github.com/opensearch-project/sql/pull/4213)) +* Push down project operator with non-identity projections into scan ([#4279](https://github.com/opensearch-project/sql/pull/4279)) +* Push down stats with bins on time field into auto_date_histogram ([#4329](https://github.com/opensearch-project/sql/pull/4329)) * Pushdown earliest/latest aggregate functions ([#4166](https://github.com/opensearch-project/sql/pull/4166)) -* Add support for `list()` multi-value stats function ([#4161](https://github.com/opensearch-project/sql/pull/4161)) -* [Enhancement] Enhance patterns command with additional sample_logs output field ([#4155](https://github.com/opensearch-project/sql/pull/4155)) +* ResourceMonitor only checks the memory health by calculating the memory usage after GC of old gen in v3 ([#3983](https://github.com/opensearch-project/sql/pull/3983)) +* SUM aggregation enhancement on operations with literal ([#3971](https://github.com/opensearch-project/sql/pull/3971)) * Search command revamp. ([#4152](https://github.com/opensearch-project/sql/pull/4152)) -* Add shortcut for count() ([#4142](https://github.com/opensearch-project/sql/pull/4142)) -* Starter implementation for `spath` command ([#4120](https://github.com/opensearch-project/sql/pull/4120)) -* strftime function implementation ([#4106](https://github.com/opensearch-project/sql/pull/4106)) -* Add regex_match function for PPL with Calcite engine support ([#4092](https://github.com/opensearch-project/sql/pull/4092)) +* Speed up aggregation pushdown for single group-by expression ([#3550](https://github.com/opensearch-project/sql/pull/3550)) +* Support ISO8601-formatted string in PPL ([#4246](https://github.com/opensearch-project/sql/pull/4246)) * Support distinct_count/dc in eventstats ([#4084](https://github.com/opensearch-project/sql/pull/4084)) -* Add wildcard support for rename command ([#4019](https://github.com/opensearch-project/sql/pull/4019)) -* Support timechart command with Calcite ([#3993](https://github.com/opensearch-project/sql/pull/3993)) -* SUM aggregation enhancement on operations with literal ([#3971](https://github.com/opensearch-project/sql/pull/3971)) +* Support first/last aggregate functions for PPL ([#4223](https://github.com/opensearch-project/sql/pull/4223)) +* Support for == and like operator in where and eval clauses. ([#4101](https://github.com/opensearch-project/sql/pull/4101)) * Support join field list and join options ([#3803](https://github.com/opensearch-project/sql/pull/3803)) -* Speed up aggregation pushdown for single group-by expression ([#3550](https://github.com/opensearch-project/sql/pull/3550)) -* Add max/min eval functions ([#4333](https://github.com/opensearch-project/sql/pull/4333)) -* Implementation of mode `sed` and `offset_field` in rex PPL command ([#4241](https://github.com/opensearch-project/sql/pull/4241)) -* Add earliest/latest aggregate function for eventstats PPL command ([#4212](https://github.com/opensearch-project/sql/pull/4212)) -* Core Implementation of `rex` Command In PPL ([#4109](https://github.com/opensearch-project/sql/pull/4109)) -* Implementation of `regex` Command In PPL ([#4083](https://github.com/opensearch-project/sql/pull/4083)) +* Support pushdown dedup with Calcite ([#3972](https://github.com/opensearch-project/sql/pull/3972)) +* Support pushdown sort by simple expressions ([#4071](https://github.com/opensearch-project/sql/pull/4071)) +* Support serializing & deserializing UDTs when pushing down scripts ([#4245](https://github.com/opensearch-project/sql/pull/4245)) +* Support time modifiers in search command ([#4224](https://github.com/opensearch-project/sql/pull/4224)) +* Support `count(eval)` expression with stats command ([#4103](https://github.com/opensearch-project/sql/pull/4103)) +* [Enhancement] Add error handling for known limitation of sql `JOIN` ([#4344](https://github.com/opensearch-project/sql/pull/4344)) +* [PPL] Patterns command add UUID regex into log template parsing ([#3989](https://github.com/opensearch-project/sql/pull/3989)) +* `fields` Command Enhancement - Advanced Field Selection Features (Calcite) ([#3970](https://github.com/opensearch-project/sql/pull/3970)) +* `mvjoin` support in PPL Caclite ([#4217](https://github.com/opensearch-project/sql/pull/4217)) +* adding capability in SQLQueryUtils to identify if SQL query is for creating a table or not. ([#4029](https://github.com/opensearch-project/sql/pull/4029)) +* strftime function implementation ([#4106](https://github.com/opensearch-project/sql/pull/4106)) ### Bug Fixes -* Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER #4416 ([#4418](https://github.com/opensearch-project/sql/pull/4418)) -* Mod function should return decimal instead of float when handle the operands are decimal literal ([#4407](https://github.com/opensearch-project/sql/pull/4407)) -* Fix numbered token bug and make it optional output in patterns command ([#4402](https://github.com/opensearch-project/sql/pull/4402)) -* Scale of decimal literal should always be positive in Calcite ([#4401](https://github.com/opensearch-project/sql/pull/4401)) +* Filter expression with `OR isnull(x)` should be pushed down ([#4055](https://github.com/opensearch-project/sql/pull/4055)) +* Fix CI failure because of plan having changed ([#4077](https://github.com/opensearch-project/sql/pull/4077)) +* Fix `ClassCastException` for value-storing aggregates on nested PPL fields ([#4360](https://github.com/opensearch-project/sql/pull/4360)) * Fix bug of missed analyzed node when pushdown filter for Search call ([#4388](https://github.com/opensearch-project/sql/pull/4388)) +* Fix geopoint issue in complex data types ([#4325](https://github.com/opensearch-project/sql/pull/4325)) +* Fix numbered token bug and make it optional output in patterns command ([#4402](https://github.com/opensearch-project/sql/pull/4402)) * Fix parse related functions return behavior in case of NULL input ([#4381](https://github.com/opensearch-project/sql/pull/4381)) -* Prevent limit pushdown before action building instead of in action executing ([#4377](https://github.com/opensearch-project/sql/pull/4377)) +* Fix value parsing bug. ([#4095](https://github.com/opensearch-project/sql/pull/4095)) +* FollowUp: Disable GCedMemoryUsage if no concurrent GC MXBean ([#4107](https://github.com/opensearch-project/sql/pull/4107)) +* Mod function should return decimal instead of float when handle the operands are decimal literal ([#4407](https://github.com/opensearch-project/sql/pull/4407)) * No index found with given index pattern should throw IndexNotFoundException ([#4369](https://github.com/opensearch-project/sql/pull/4369)) -* Fix `ClassCastException` for value-storing aggregates on nested PPL fields ([#4360](https://github.com/opensearch-project/sql/pull/4360)) -* change Anonymizer to mask PPL ([#4352](https://github.com/opensearch-project/sql/pull/4352)) -* Fix alphanumeric search which starts with number ([#4334](https://github.com/opensearch-project/sql/pull/4334)) -* Push down stats with bins on time field into auto_date_histogram ([#4329](https://github.com/opensearch-project/sql/pull/4329)) -* Fix geopoint issue in complex data types ([#4325](https://github.com/opensearch-project/sql/pull/4325)) -* Support serializing & deserializing UDTs when pushing down scripts ([#4245](https://github.com/opensearch-project/sql/pull/4245)) +* Prevent limit pushdown before action building instead of in action executing ([#4377](https://github.com/opensearch-project/sql/pull/4377)) +* Scale of decimal literal should always be positive in Calcite ([#4401](https://github.com/opensearch-project/sql/pull/4401)) * Bugfix: SQL type mapping for legacy JDBC output ([#3613](https://github.com/opensearch-project/sql/pull/3613)) +* [SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER #4416 ([#4418](https://github.com/opensearch-project/sql/pull/4418)) +* Fix alphanumeric search which starts with number ([#4334](https://github.com/opensearch-project/sql/pull/4334)) +* change Anonymizer to mask PPL ([#4352](https://github.com/opensearch-project/sql/pull/4352)) ### Infrastructure -* Spotless precommit: apply instead of check ([#4320](https://github.com/opensearch-project/sql/pull/4320)) -* Add spotless precommit hook + license check ([#4306](https://github.com/opensearch-project/sql/pull/4306)) -* Fix doctest branch ([#4292](https://github.com/opensearch-project/sql/pull/4292)) -* Doctest: Use 1.0 branch of CLI instead of main ([#4219](https://github.com/opensearch-project/sql/pull/4219)) +* Add issue template for doc issues ([#4194](https://github.com/opensearch-project/sql/pull/4194)) * Add merge_group trigger to test workflows ([#4216](https://github.com/opensearch-project/sql/pull/4216)) +* Add spotless precommit hook + license check ([#4306](https://github.com/opensearch-project/sql/pull/4306)) +* Avoid unnecessary security plugin download in integ-test ([#4368](https://github.com/opensearch-project/sql/pull/4368)) +* Don't recreate indices on every doctest ([#4222](https://github.com/opensearch-project/sql/pull/4222)) +* Migrate from sonatype snapshot to ci.opensearch.org snapshots ([#4141](https://github.com/opensearch-project/sql/pull/4141)) +* Print links to test logs after integTest ([#4273](https://github.com/opensearch-project/sql/pull/4273)) +* Reorder the maven repositories to resolve the service unavailable issue ([#4153](https://github.com/opensearch-project/sql/pull/4153)) +* Run unit test suites in parallel ([#4190](https://github.com/opensearch-project/sql/pull/4190)) * Split up our test actions into unit, integ, and doctest. ([#4193](https://github.com/opensearch-project/sql/pull/4193)) +* Spotless precommit: apply instead of check ([#4320](https://github.com/opensearch-project/sql/pull/4320)) +* Adding new maintainer @yuancu ([#4045](https://github.com/opensearch-project/sql/pull/4045)) +* `fields` and `table` command big5 queries ([#4126](https://github.com/opensearch-project/sql/pull/4126)) +* disable commit-history uploading to maven ([#4088](https://github.com/opensearch-project/sql/pull/4088)) +* Doctest: Use 1.0 branch of CLI instead of main ([#4219](https://github.com/opensearch-project/sql/pull/4219)) ### Documentation +* Add documents on how to develop a UDF / UDAF ([#4094](https://github.com/opensearch-project/sql/pull/4094)) +* Add new command checklist doc ([#4073](https://github.com/opensearch-project/sql/pull/4073)) +* Add splunk to ppl cheat sheet ([#3726](https://github.com/opensearch-project/sql/pull/3726)) +* Doc enhancement for eventstats and bin command ([#4117](https://github.com/opensearch-project/sql/pull/4117)) * Update bin.rst and add `bin` to doctest ([#4384](https://github.com/opensearch-project/sql/pull/4384)) +* Update developer guide ([#4128](https://github.com/opensearch-project/sql/pull/4128)) +* Update fields.rst ([#4134](https://github.com/opensearch-project/sql/pull/4134)) * Update timechart in SPL/PPL cheat sheet ([#4382](https://github.com/opensearch-project/sql/pull/4382)) -* Enable doctest with Calcite ([#4379](https://github.com/opensearch-project/sql/pull/4379)) -* Correct the comparision table for rex doc ([#4321](https://github.com/opensearch-project/sql/pull/4321)) * Updating coalesce documentation ([#4305](https://github.com/opensearch-project/sql/pull/4305)) * Updating documentation for `fields` and `table` commands ([#4177](https://github.com/opensearch-project/sql/pull/4177)) -* Add documents on how to develop a UDF / UDAF ([#4094](https://github.com/opensearch-project/sql/pull/4094)) -* Add splunk to ppl cheat sheet ([#3726](https://github.com/opensearch-project/sql/pull/3726)) +* [Doc] Correct the comparision table for rex doc ([#4321](https://github.com/opensearch-project/sql/pull/4321)) +* [Doc] Enable doctest with Calcite ([#4379](https://github.com/opensearch-project/sql/pull/4379)) ### Maintenance -* Avoid unnecessary security plugin download in integ-test ([#4368](https://github.com/opensearch-project/sql/pull/4368)) -* Fix timezone dependent test failures ([#4367](https://github.com/opensearch-project/sql/pull/4367)) -* Update grammar files and developer guide ([#4301](https://github.com/opensearch-project/sql/pull/4301)) -* Introduce YAML formatter for better testing/debugging ([#4274](https://github.com/opensearch-project/sql/pull/4274)) -* Print links to test logs after integTest ([#4273](https://github.com/opensearch-project/sql/pull/4273)) -* Fix the IT issue caused by merging conflict ([#4270](https://github.com/opensearch-project/sql/pull/4270)) -* Fix gitignore to ignore symbolic link ([#4263](https://github.com/opensearch-project/sql/pull/4263)) -* Add gitignore for Cline ([#4258](https://github.com/opensearch-project/sql/pull/4258)) * Add Ryan as a maintainer ([#4257](https://github.com/opensearch-project/sql/pull/4257)) +* Add gitignore for Cline ([#4258](https://github.com/opensearch-project/sql/pull/4258)) +* Add missing license headers from #4120 ([#4160](https://github.com/opensearch-project/sql/pull/4160)) +* Adding big5 `coalesce` queries ([#4127](https://github.com/opensearch-project/sql/pull/4127)) +* Extract getOnlyForCalciteException ([#4150](https://github.com/opensearch-project/sql/pull/4150)) +* Fix AnalyzerTest ([#4215](https://github.com/opensearch-project/sql/pull/4215)) +* [Hotpatch][Infra] Fix CI artifact sources ([#4131](https://github.com/opensearch-project/sql/pull/4131)) +* Fix doctest branch ([#4292](https://github.com/opensearch-project/sql/pull/4292)) +* Fix doctest requirements install ([#4175](https://github.com/opensearch-project/sql/pull/4175)) \ No newline at end of file