Add release notes for 0.274

[highker]

Missing Release Notes

Amit Dutta

• Add escape character in like predicate for SHOW CATALOGS #17767 Add escape character in like predicate for SHOW CATALOGS (Merged by: Timothy Meehan)

Eduard Tudenhoefner

• Bump Iceberg to 0.13.2 #17866 Bump Iceberg to 0.13.2 (Merged by: Rebecca Schlussel)

Pranjal Shankhdhar

• Add external statistics provider #17845 Add external statistics provider (Merged by: Rebecca Schlussel)

Todd Gao

• Add hudi connector #17149 Add hudi connector (Merged by: ARUNACHALAM THIRUPATHI)

maswin

• Optimize filter condition with switch case #17065 Optimize filter condition with switch case (Merged by: Rebecca Schlussel)

vaishnavibatni

• Revert "Adding Presto Version Class and version checks for session properties" #17898 Revert "Adding Presto Version Class and version checks for session properties" (Merged by: James Sun)
• Adding Presto Version Class and version checks for session properties #17760 Adding Presto Version Class and version checks for session properties (Merged by: Timothy Meehan)

Extracted Release Notes

• Initial phase of support for constraint optimization to the PrestoDB optimizer in response to issue #16413 #16416 (Author: David Simmen): Initial phase of support for constraint optimization to the PrestoDB optimizer in response to issue Constraint Support and Optimizations  #16413
  • Add support for constraint optimizations in Hive MetaStore. This feature can be turned on by setting the session property "exploit-constraints" and config property "optimizer.exploit-constraints" to true.
• Add hudi integeration tests and bump version #17463 (Author: Todd Gao): Add hudi integeration tests and bump version
  • Upgrade Hudi support to 0.11.0.
• Add rate limiting functionality for coordinator #17628 (Author: ericyuliu): Add rate limiting functionality for coordinator
  • Add rate limiting functionality for each query on coordinator http endpoints.
• Handle missing StorageDescriptor in Hive Glue tables #17655 (Author: v-jizhang): Handle missing StorageDescriptor in Hive Glue tables
  • Handle missing StorageDescriptor in Hive Glue tables.
• Add Connector Serde Infrastructure #17684 (Author: Ajay George): Add Connector Serde Infrastructure
  • Add connector type serde support.
  • Add connector type serde support in SPI.
• Add support for Nessie Catalog in Iceberg Connector #17719 (Author: Eduard Tudenhoefner): Add support for Nessie Catalog in Iceberg Connector
  • Add support for NessieCatalog.
• Fix infinite loop during planning caused by PickTableLayout #17720 (Author: v-jizhang): Fix infinite loop during planning caused by PickTableLayout
  • Fix infinite loop during planning caused by PickTableLayout.
• Add unit to RuntimeMetric #17737 (Author: Zac): Add unit to RuntimeMetric
  • Add the support for setting units of runtime metrics.
• Add support for SHOW SESSION LIKE #17752 (Author: Amit Dutta): Add support for SHOW SESSION LIKE
  • Add support for LIKE predicate to SHOW SESSION.
• Fix Pinot timestamp format error #17756 (Author: Xiaoman Dong): Fix Pinot timestamp format error
  • Fix query error when applying aggregation function over timestamp columns (:issue:17755).
• Make open writers limit configurable in iceberg connector #17771 (Author: v-jizhang): Make open writers limit configurable in iceberg connector
  • Add iceberg.max-partitions-per-writer configuration property to allow configuring the limit on partitions per writer.
• Handle Pinot Project Expressions - SWITCH #17800 (Author: Andy Li): Handle Pinot Project Expressions - SWITCH
  • Add support for CASE statement pushdown to Pinot.
  • Add support for logical and arithmetic functions for CASE statements.
• Add clickhouse document in connector list #17813 (Author: Reetika Agrawal): Add clickhouse document in connector list
  • Add clickhouse connector document in connector list.
• Introducing LocalMode to DistributedClusterStats  #17817 (Author: Harleen Kaur): Introducing LocalMode to DistributedClusterStats
  • Add support for local mode on /v1/cluster/ to get information on locally running/queued queries. Can be enabled with the includeLocalInfoOnly property.
• [Pinot Connector] Enable pinot grpc by default #17819 (Author: Xiang Fu): [Pinot Connector] Enable pinot grpc by default
  • Prefer querying Pinot server grpc endpoint by default (through config pinot.use-streaming-for-segment-queries).
• Add laplace_cdf and inverse_laplace_cdf functions #17838 (Author: Jonathan Hehir): Add laplace_cdf and inverse_laplace_cdf functions
  • Add :func:laplace_cdf and :func:inverse_laplace_cdf functions.
• Consider table size when choosing join distribution type #17849 (Author: Rongrong Zhong): Consider table size when choosing join distribution type
  • When CBO fails, consider source table size when choosing join order and join distribution type. This might improve query plan for queries with complex projections on small tables. This optimization can be disabled by setting session property size_based_join_distribution_type to false.
• Use REQUIRED repetition level for MAP keys in parquet writer #17879 (Author: Zhenxiao Luo): Use REQUIRED repetition level for MAP keys in parquet writer
  • Use REQUIRED repetition level for MAP keys in parquet writer.
• Add parquet ecryption functionality into presto #17881 (Author: Xinli shang): Add parquet ecryption functionality into presto
  • Add decryption functionality to Presto. When a Parquet file is encrypted following Parquet Modular Encryption, this change enables Presto to be able to decrypt.
  • No flag is introduced. Presto-Hive was changed by adding the loading DecryptionPropertiesFactory(implemented in parquet-mr) and using it to get the file decryptor and pass it to presto-parquet.
• Implement segmented aggregation execution #17886 (Author: Zac): Implement segmented aggregation execution
  • Add ability to flush the aggregated data when at lease one segment from the input has been exhausted. It can help reduce the memory footprint and improves the performance of aggregation when the data is already ordered by a subset of the group-by keys. This can be enabled with the segmented_aggregation_enabled session property or the optimizer.segmented-aggregation-enabled configuration property.
  • Add support for segmented aggregation to reduce the memory footprint and improve query performance when the order-by keys are a subset of the group-by keys. This can be enabled with the order_based_execution_enabled session property or the hive.order-based-execution-enabled configuration property.
• Add memory limit check for HashBuilderOperator's spilling #17905 (Author: Jaromir Vanek): Add memory limit check for HashBuilderOperator's spilling
  • Add memory limit check for HashBuilderOperator's spilling and fail fast if exceeding to avoid unnecessary processing.
• Fix the UI displaying name and systemMemoryContext allocationTag of OptimizedPartitionedOutputOperator #17910 (Author: junyi1313): Fix the UI displaying name and systemMemoryContext allocationTag of OptimizedPartitionedOutputOperator
  • Fix the UI displaying name and systemMemoryContext allocationTag of OptimizedPartitionedOutputOperator.
• Support renaming table in MongoDB #17914 (Author: v-jizhang): Support renaming table in MongoDB
  • Support renaming tables. Users can use ALTER TABLE RENAME TO to rename a table.
• Query failing with exceed local memory does not free memory correctly #17984 (Author: Swapnil Tailor): Query failing with exceed local memory does not free memory correctly
  • Fix broadcast join memory leak caused by exceeding local memory query failure.

All Commits

• 04e4900 Add memory limit check for HashBuilderOperator's spilling (Jaromir Vanek)
• d42d354 Query failing with exceed local memory doesn't free memory correctly (Swapnil Tailor)
• 9cf90ad Upgrade TestNG Version to 7.5 (Arunachalam Thirupathi)
• 8a2ea29 Disable Flaky Kudu integration test (Arunachalam Thirupathi)
• e0f4249 Disable Failing Cassandra Tests (Arunachalam Thirupathi)
• 7e45adc Fix OracleServer Tests (Arunachalam Thirupathi)
• e4fc261 Fix Iceberg Tests (Arunachalam Thirupathi)
• 15058ec Ignore Exit code exceptions on Presto stop (Arunachalam Thirupathi)
• b89766c add hudi connector doc (zhangyanbing)
• 3d3bcd2 Change readPositions to long to avoid overflow (Dongsheng Wang)
• 19397a1 Document the Iceberg split weight property (v-jizhang)
• 31979f7 Attach weights to IcebergSplits (v-jizhang)
• 03d9533 Skip unused bin packing in IcebergSplitSource (v-jizhang)
• 629f4da Fix tests in preparation of TestNG upgrade (Arunachalam Thirupathi)
• 4739320 Move MaterializedViewQueryOptimizer logic to inner class (Patrick Sullivan)
• 88e7df6 Support queryUpdate event in EventListener (Arjun Gupta)
• 55e24cf Add query level aggregated shuffle bytes and rows in QueryStats (xiaoxmeng)
• de6ef8b Minor improvements for DwrfMetadataReader (Sergii Druzkin)
• bff66c4 Add a singleton for NoOpOrcWriterStats (Sergii Druzkin)
• 238f42b Fix the UI displaying name and systemMemoryContext allocationTag of OptimizedPartitionedOutputOperator (junyi1313)
• cd71cf0 Add a flat map writer key limit (Sergii Druzkin)
• d1b20f1 Add read/write support for map stats (Sergii Druzkin)
• f4d977c Fix MergeJoinNode stage detail in UI (Amit Adhikari)
• 1290033 Close down OrcReaderOptions c-tor for good (Sergii Druzkin)
• 5c03da3 Add codeowners for Hudi connector (Tim Meehan)
• 3ee8cb3 Implement segmented aggregation execution (Zac)
• c61c2e4 Support renaming table in MongoDB (v-jizhang)
• e53d846 Add metadata classes for map statistics (Sergii Druzkin)
• 66079f8 Add RemoteMetadataManager implementation (Eric Kwok)
• c13f510 Enable pinot grpc by default (Xiang Fu)
• 1ecf0c4 Add mapStatisticsEnabled flag to OrcWriterOptions (Sergii Druzkin)
• 11f2085 Fix external statistics initialization (Pranjal Shankhdhar)
• 439f31f Creating and testing local info in coordinator stats (Harleen Kaur)
• 3f5595e Improve benchmark for flat map writer (Sergii Druzkin)
• d796781 Fix pinot connector generated query by adding single quote for time value in timeBoundary predicate (Xiang Fu)
• 8fe1a05 Make OrcOutputBuffer lazy (Sergii Druzkin)
• 386e460 Previous partition info for overwrite (abhiseksaikia)
• d5ba813 Fail the merge join query if grouped execution can't be enabled (Ke Wang)
• 48266ed Optimize ByteOutputStream used for IN_MAP and PRESENT streams (Sergii Druzkin)
• 2f88aa7 Optimize ByteOutputStream used for IN_MAP and PRESENT streams (Sergii Druzkin)
• 6b62e5e Optimize merging of stats in DoubleStatisticsBuilder (Sergii Druzkin)
• 941905a Bundle hudi connector to presto-server (Todd Gao)
• 0d6cb1e Add hudi related sample configurations (Todd Gao)
• 766cafb Fix metastore bindings in HudiModule (Todd Gao)
• 836b905 Add CatalogServer thrift service (Eric Kwok)
• 05a2771 Annotate metadata related objects as Thrift objects (Eric Kwok)
• e6f0a70 Allow transaction manager to register a transaction (Eric Kwok)
• 49311b0 Revert "Adding Version Checks for Session Properties" (vaishnavibatni)
• bf9cff7 Remove source location from CanonicalPlanGenerator (Pranjal Shankhdhar)
• da4f2b5 Consider table size when choosing join distribution type (Rongrong Zhong)
• 039c107 Bump Iceberg to 0.13.2 (Eduard Tudenhoefner)
• a647858 Optimize merging of IntegerStatistics (Sergii Druzkin)
• 79421bf Add grouped execution session property check for merge join optimization (Ke Wang)
• e429cb8 Handle Pinot Project Expressions (Andy Li)
• 842b469 Use REQUIRED repetition level for MAP keys in parquet writer (Zhenxiao Luo)
• 20795f7 Add TestParquetSchemaConverter (Zhenxiao Luo)
• 07b7f0d Parameterize repetition for types in ParquetSchemaConverter (Zhenxiao Luo)
• 71dc62a Add parquet ecryption functionality into presto. (Xinli shang)
• 84a7350 Only support inner join for merge join optimization (Ke Wang)
• b72dbf1 Upgrade Airlift to 0.205 (prithvip)
• 36c5fe1 Revert "Only request client certificate when using certificate authentication" (prithvip)
• 2b61ec8 Update presto_cpp 20220613 (Michael Shang)
• 5c3ac4c Implement constraints based optimization rules (David Simmen)
• 44482f6 Compute and propagate logical properties (David Simmen)
• 9c6b35c Read Hive table primary key and unique constraints (David Simmen)
• d512ae1 Check overflow in ORC Timestamp decoder (vigneshwarselvaraj)
• 6ecfd81 Add hudi connector (Todd Gao)
• e1f8b4b Use history based statistics provider (Pranjal Shankhdhar)
• 0eab55a Add API for history based plan statistics using external service (Pranjal Shankhdhar)
• db461e6 Refactor HashAggregationOperator (Zac)
• 51cb821 Add materialized view rewrite optimization for COUNT (Patrick Sullivan)
• f8ae713 Fix stats validation for flat map files (Sergii Druzkin)
• 90dd818 Integrate new stream layout API(s) with the OrcWriter (Harsha Rastogi)
• 11f4b09 Increase readability of DefaultTraversalVisitor (Patrick Sullivan)
• f25ce87 Introduce AlternativeMinMax aggregations for Velox with REAL intermediate type. (Sergey Pershin)
• 633f469 Implement stream reordering in the stream layout (Harsha Rastogi)
• b7d3d4c Adding Version Checks for Session Properties (vaishnavibatni)
• ef7d408 Allow unit to be null in RuntimeMetric (Zac)
• ac78220 Add laplace_cdf and inverse_laplace_cdf functions (Jonathan Hehir)
• f579385 Deltalake document minor update (Reetika Agrawal)
• c97f527 Add rate limiting functionality for coordinator (ericyuliu)
• d422c76 Fix flaky TestQueryTaskLimit (James Sun)
• 525e69e Add read hints to ExtendedFileSystem open API (Arunachalam Thirupathi)
• 65eb78e Enable multiway joins in MV definition (Patrick Sullivan)
• 49cb9c8 Upgrade Airlift to 0.204 (Ajay George)
• 190d0bd Revert "Add WrapperJobConf, CopyOnWriteConfiguration to reduce config copies" (Ajay George)
• 4d1add0 Revert "Reduce number of hdfsConfiguration copies" (Ajay George)
• 0f6f9a4 Add Thrift Support for ServerInfoResource (Xiaoxuan Meng)
• b34cb6a Add plan matching to materialized view join test (Patrick Sullivan)
• dc9d089 Fix a bug with empty maps in MapFlatSelectiveStreamReader (Sergii Druzkin)
• 2bc8e67 Add a StreamOrdering config (Harsha Rastogi)
• 71663cf Add clickhouse document in connector list (Reetika Agrawal)
• da39f03 Switch comments in example to match the correct block (v-jizhang)
• 51f5cc1 Fix flat map writer's getBufferedBytes (Sergii Druzkin)
• 3131bea Optimize creation of empty column stats in flat map writer (Sergii Druzkin)
• 0ad522a Add implementation of DWRF flat map column writer. (Sergii Druzkin)
• aa7dd4b Full flat map writer (Sergii Druzkin)
• 8b6a71d Allow to create a copy of ColumnWriterOptions with disabled dictionaries (Sergii Druzkin)
• 60f4583 Refactor iceberg hadoop and hive catalog tests (Chunxu Tang)
• 15d97bf Add microsecond timestamp support in orc reader / writer (vigneshwarselvaraj)
• ed9e11b Fix Iceberg statistics for empty table (v-jizhang)
• 2bbf869 Add empty java class to fix the test coverages staging deploy (Guy Moore)
• ae074a1 Fix mongodb smoke test with varchar keyword (James Sun)
• a2cfd24 Simplify the format of Hive CommitOutput (Lin Liu)
• 554e823 Remove enableTimestampMicroPrecision option (vigneshwarselvaraj)
• 12dfba2 Upgrade hudi to version 0.11.0 and fix breakings (Todd Gao)
• 3aec561 Add integration tests for querying Hudi tables (Todd Gao)
• 66c7001 Add support for Nessie Catalog in Iceberg Connector (Eduard Tudenhoefner)
• d03c809 Let some stats builders to add a value from block (Sergii Druzkin)
• e9eeff5 Make StripeEncryptionGroup work with flat map column encodings (Sergii Druzkin)
• d5ecd88 Allow special character when creating subfield from column (Pranjal Shankhdhar)
• a9037be Migrate existing fucntions to adapt runtime metric unit (Eric Kwok)
• e2c1132 Allow # in SubfieldTokenizer which is a valid column name (Pranjal Shankhdhar)
• 09c15eb Add hive connector type serde support (Ajay George)
• 5d7035d Add test for connector type serde (Ajay George)
• ac56e53 Add connector type serde support (Ajay George)
• e68c9bd Add SPI connector type serde support (Ajay George)
• 43192d3 Add Any type (Ajay George)
• 1a4deb8 Add unit to Presto coordinator runtime metrics (Eric Kwok)
• 2547605 Add Iceberg configuration property to docs (v-jizhang)
• 2c32a8c Make open writers limit configurable in iceberg connector (v-jizhang)
• 8ee3bea Log epoch for SerializedCommitOutput (Lin Liu)
• d27f61d Return ConnectorCommitHandle from HiveConnector (Lin Liu)
• 90539ac Overwrite partition using previous partition info (abhiseksaikia)
• 705889d Add support for setting unit of RuntimeMetric (Zac)
• d61ca17 Add Nullable annotation to OperatorInfoUnion (Ajay George)
• 9a73930 Fix reading Delta Table metadata from Glue (v-jizhang)
• dd39f46 Always ignore StorageDescriptors for Iceberg and Delta tables (v-jizhang)
• 760537a Handle missing StorageDescriptor in Hive Glue tables (v-jizhang)
• 08c38f9 Fix Pinot timestamp format parsing error (Xiaoman Dong)
• 41460f9 Let flat map writer to add IN_MAP checkpoints to sub nodes (Sergii Druzkin)
• 71631e4 Fix coordinator queryID tooltip and query string truncation (Eric Kwok)
• 81a347b Add escape character in like predicate for SHOW CATALOGS (Amit Dutta)
• 47b1fe3 Fix phased execution for merge join (Ke Wang)
• 497e3d5 Fix grouped execution property for merge join (Ke Wang)
• be7eec5 Add stream property check for merge join plan generation (Ke Wang)
• 95cc08c Add support for SHOW SESSION LIKE. (Amit Dutta)
• 67c8e6d Optimize filter condition with CASE predicate (maswin)
• b17ec68 Allow RowExpressionRewriteRuleSet to be enabled or disabled (maswin)
• 5596bd9 Enable use of the method readTable from child classes (Harsha Rastogi)
• bd48fc3 Fix infinite loop during planning caused by PickTableLayout (v-jizhang)