[DO NOT MERGE] Add release notes for 0.230

[leiqingc]

Missing Release Notes

Ami Tavory

• Refactor Differential Entropy for Mutual Information Classification #13574 Refactor Differential Entropy for Mutual Information Classification (Merged by: Rebecca Schlussel)

Gaurav Mittal

• Add support for JNI based decompression for zstd files #13704 Add support for JNI based decompression for zstd files (Merged by: James Sun)
• Wrap OrcReader parameters into OrcReaderOptions #13697 Wrap OrcReader parameters into OrcReaderOptions (Merged by: James Sun)

Leiqing Cai

• f189ca7 Make TestUpgradeMetadata single threaded

Masha Basmanova

• c4d0c7f Fix pushdown for filters on multiple subfields of a struct
• 9d60d38 Create NestedField using lowercase field name
• 2b6e974 Make subfield name matching case insensitive

Sahar Massachi

• merge  mayunlei/presto#2 merge

Saksham Sachdev

• Transaction Sharing across Connectors #13660 Transaction Sharing across Connectors (Merged by: James Sun)

Extracted Release Notes

• imported combinations function from prestosql, complete with tests an… #13283 (Author: Laila A. Wahedi): imported combinations function from prestosql, complete with tests an…
  • Added combinations function, a function that returns n combinations of values in an array, up to n=5.
• Reduce the identifier length in generated code for nested columns #13405 (Author: Venki Korukanti): Reduce the identifier length in generated code for nested columns
  • Fix compilation errors for expressions over types containing an extremely large number of nested types.
• Support DROP FUNCTION #13412 (Author: Leiqing Cai): Support DROP FUNCTION
  • Add support for DROP FUNCTION.
• Modularize Hive Metastore #13601 (Author: Islam Ismailov): Modularize Hive Metastore
  • Metastore interface is separated into a separate module to reduce monolithicness.
• Improve BatchStreamReader - Part 1 #13602 (Author: Ying Su): Improve BatchStreamReader - Part 1
  • Improve BatchStreamReader performance.
• Add geospatial expand_envelope function. #13616 (Author: James A. Gill): Add geospatial expand_envelope function.
  • Add expand_envelope function to return a geometry's envelope expanded by a distance.
• Introduce caching file system #13644 (Author: James Sun): Introduce caching file system
  • Introduced new cache module to allow using local disk for to cache files from remote file systems.
  • Allow Raptor to read data from HDFS while caching the files on local disks.
• Fix GeometryToBingTiles for certain geometries #13645 (Author: James A. Gill): Fix GeometryToBingTiles for certain geometries
  • Fix geometry_to_bing_tile for geometries at -180 longitude or 85.05112878 latitude.
• Output verifier events for queries being skipped by pre-filters #13647 (Author: Leiqing Cai): Output verifier events for queries being skipped by pre-filters
  • Add skipped verification results to the output for queries being filtered.
• Optimize lambda body when possible #13650 (Author: Yi He): Optimize lambda body when possible
  • Fix regression on lambda evaluation (Issue#13648).
• Make commitPartition an async operation #13651 (Author: Shixuan Fan): Make commitPartition an async operation
  • Change ConnectorMetadata#commitPartition into async operation, and rename it to ConnectorMetadata#commitPartitionAsync.
• fix organizationDiscoveryIntervalMillis bug #13652 (Author: Ke Wang): fix organizationDiscoveryIntervalMillis bug
  • Fix organizer not running scheduled job due to misreading storage.organization-discovery-interval config.
• Fix PRESTO_EXTRA_CREDENTIAL parsing #13653 (Author: Yi He): Fix PRESTO_EXTRA_CREDENTIAL parsing
  • Improved PRESTO_EXTRA_CREDENTIAL header parsing to allow value contain multiple '=' and urlEncode characters.
• Add all_match, any_match and none_match functions for arrays #13734 (Author: Wenlei Xie): Add all_match, any_match and none_match functions for arrays
  • Add all_match(), any_match(), and none_match() functions.
• Enable creating table with table_supports_delta_delete property for raptor #13741 (Author: Ke Wang): Enable creating table with table_supports_delta_delete property for raptor
  • Add table_supports_delta_delete property in Raptor to allow deletion happening in background. DELETE queries in Raptor can now delete data logically but relying on compactors to delete physical data.
• Add Alluxio client jar to hive-hadoop2 #13743 (Author: Bin Fan): Add Alluxio client jar to hive-hadoop2
  • Add an Alluxio client jar to plugin/hive-hadoop2/ (as a runtime dependency) to avoid copying Alluxio client jar to all Presto servers manually to connect to Alluxio.
• Fix SQL function compilation failure #13748 (Author: Rongrong Zhong): Fix SQL function compilation failure
  • Fix SQL function compilation failure when the function parameter is not referenced in function body.
• Respect null-call clause in SQL function execution #13764 (Author: Rongrong Zhong): Respect null-call clause in SQL function execution
  • Fix SQL function execution not respecting null-call clause.
• Cast constants in SQL function to proper types #13784 (Author: Rongrong Zhong): Cast constants in SQL function to proper types
  • Fix SQL function byte code compilation error for functions implemented with constants.
• List non built-in functions in SHOW FUNCTIONS #13787 (Author: Leiqing Cai): List non built-in functions in SHOW FUNCTIONS
  • Add support to list non-builtin functions in SHOW FUNCTIONS. The feature can be turned on by the configuration property list-non-built-in-functions.

All Commits

• c4d0c7f Fix pushdown for filters on multiple subfields of a struct (Masha Basmanova)
• f189ca7 Make TestUpgradeMetadata single threaded (Leiqing Cai)
• 9d60d38 Create NestedField using lowercase field name (Masha Basmanova)
• 2b6e974 Make subfield name matching case insensitive (Masha Basmanova)
• c5c3c67 Use LazyBlock for columns with no filters (Masha Basmanova)
• 23e4773 Add Alluxio client jar to hive-hadoop2 (Bin Fan)
• 2b7baf2 List non built-in functions in SHOW FUNCTIONS (Leiqing Cai)
• 7236e03 Cast constants in SQL function to proper types (Rongrong Zhong)
• b35550c Remove some usages of QualifiedFunctionName#of(String) (Leiqing Cai)
• cb1b514 Throw SYNTAX_ERROR when creating function with invalid name (Leiqing Cai)
• 943b4d0 Support DROP FUNCTION execution (Leiqing Cai)
• 71a23da Support dropFunction in FunctionNamespaceManager (Leiqing Cai)
• 8138d2f Add syntax support for DROP FUNCTION (Leiqing Cai)
• eb47fcf Remove print statement in TestFunctionNamespaceManager (Leiqing Cai)
• 7c26b6e Invert if condition for better readability (Ajay George)
• 1fbfb8e Add support for column names with slashes to SubfieldTokenizer (Masha Basmanova)
• 0f1fe83 Fix backward compatible issue of Raptor delta delete (Ke Wang)
• d254d5f Fix materialized exchange planning with filter pushdown (Masha Basmanova)
• df0fd4d Fix slice size overflow in DictionaryBlock.getIds() (Ying Su)
• 80613f9 Add support for column names with hyphen to SubfieldTokenizer (Masha Basmanova)
• 3242715 Respect null-call clause in SQL function execution (Rongrong Zhong)
• 9aa8f80 Optimize lambda body when possible (Yi He)
• 83b7d17 Propagate correct error code in RowExpressionInterpreter (Yi He)
• bf4f7ec introduce warnings for nonReserved words moving to reserved (Sujay Jain)
• 5772869 Materialize stripe stream slice when caching (Shixuan Fan)
• 98f54e7 Fix stripe metadata cache weigher to use retained size (Shixuan Fan)
• a472c23 Fix filter pushdown for always-true range filter on a boolean column (Bhavani Hari)
• bc2e8b2 chunk merge files during cache write to avoid reading excessively large data into memory and other optimizations (Sujay Jain)
• dcb3e1b Construct QualifiedFunctionName with CatalogSchemaName and function name (Vic Zhang)
• 5985800 Add array combinations function (Laila A. Wahedi)
• 36ec9d2 Enable creating table with table_supports_delta_delete property (Ke Wang)
• d355b8b Add support for filter functions on columns with coercions (Bhavani Hari)
• 6dc5468 Fix SQL function compilation failure (Rongrong Zhong)
• dedea3b Support supplying benchmark queries through MySQL (Nikhil Collooru)
• 456487e Minor refactor of ApplyNode constructor (Vic Zhang)
• 19339ee Throw PrestoException for unsupported correlated subquery (Vic Zhang)
• bd08887 Use distinct table names in TestHiveRecoverableGroupedExecution (Shixuan Fan)
• e18bb83 Expose H2QueryRunner Handle for Modular Testing (Saksham Sachdev)
• 0a0a443 Add support for Companion Catalogs in Transactions (Saksham Sachdev)
• 57f9632 Revert "Optimize PlanFragment serialization" (Nikhil Collooru)
• cafa9e2 Minor fixes to documents (Wenlei Xie)
• 4eb8549 Rename session variable to be more specific (Bhavani Hari)
• 0426340 Fix size type when calculating retained size. (Ying Su)
• 48e58fd Make restProxyServiceForQuery nullable for the Pinot connector (Devesh Agrawal)
• fca4353 Add all_match, any_match and none_match functions for arrays (Wenlei Xie)
• 2bee126 Extract and rename functional interfaces in ArrayFilterFunction (Wenlei Xie)
• df7a0e5 Remove unused FilterVoidLambda interface in ArrayFilterFunction (Wenlei Xie)
• 7a5cc80 Fix minor typo in CLA instructions (Brian Warner)
• bba36ec Update CLA information to reflect EasyCLA (Brian Warner)
• ef3456a Introduce caching file system (James Sun)
• 229b60b Make commitPartition an async operation (Shixuan Fan)
• 3d2f2a1 Add JNI decompressor for zstd files (Gaurav Mittal)
• 01c66c7 Fix selective struct reader to make a copy of input positions (Bhavani Hari)
• 8e5051c Wrap OrcReader parameters into OrcReaderOptions (Gaurav Mittal)
• 70308fa Fix Values are too large error message (Masha Basmanova)
• cbf1ee7 Allow range filters on newly added struct fields (Masha Basmanova)
• 998db97 Add test for range filter on a newly added column (Masha Basmanova)
• d294ebc Allow range filters on subfields of structs with coercion (Masha Basmanova)
• e668b1a Reset allNulls flag in SliceDirectxxx#read (Masha Basmanova)
• db8067e Reset allNulls flag in SliceDictxxx#read (Masha Basmanova)
• c0fdc90 Skip map entries with null keys (Masha Basmanova)
• d9e9f39 Run filters without inputs first in OrcSelectiveStreamReader (Sahar Massachi)
• c0b0edd Evalute predicate on partition columns to prune partitions (Masha Basmanova)
• dd16d8a Add a method to interpret an expression to ExpressionOptimizer (Masha Basmanova)
• ca4e94c Add functionMetadataManager to HiveMetadataFactory (Masha Basmanova)
• 39a16b3 Fix filter pushdown for DELETE queries (Masha Basmanova)
• 6cced01 Support reading RC, TEXTFILE fromats with pushdown enabled (Bhavani Hari)
• d3ee8ec Replace Joda-Time libraries with java.time (Ajay George)
• 315868d Add final to histogram class members (Ami Tavory)
• 9f2359a Fixe minor issues in reservoir sample tests (Ami Tavory)
• 4b11292 Move logic for handling different strategy implementations to DifferentialEntropyStateStrategy (Ami Tavory)
• 62299ab Add cloneEmpty method to differential entropy strategy classes (Ami Tavory)
• 3530831 Add population weight/count methods to reservoir samples (Ami Tavory)
• 8e15a3a Expand entropy utils class and rename methods (Ami Tavory)
• 9b0a1a0 Update names of variables in code, and function letters in docs. (Ami Tavory)
• dacfc45 Remove unused types from CostCalculator (James Sun)
• 6cc6e5b Fix the build broken by 9633991 (Ying Su)
• 906006b Add memory tracking from StreamReader local buffers (Ying Su)
• 7be0c9f Improve BenchmarkBatchStreamReaders (Ying Su)
• f92fa25 Improve ORC LongReader (Ying Su)
• 61408c2 Improve ORC timestamp reader (Ying Su)
• 0814f33 Improve ORC list and map readers (Ying Su)
• bf96a44 Improve ORC slice direct reader (Ying Su)
• 9633991 Improve ORC byte reader (Ying Su)
• 95a57bd Improve ORC boolean reader (Ying Su)
• d45be17 Remove type from OrcRecordReader.readBlock (Ying Su)
• 96bdef0 Add support for bucket adaptation (Masha Basmanova)
• 97bd945 Fix getTableStatistics with subfield pruning (Masha Basmanova)
• fb2079e Clarify variable names in getTableStatistics (Masha Basmanova)
• 9805be6 Refactor FilterStatsCalculatorService (Masha Basmanova)
• c851382 Fix subfield pruning for legacy unnest (Masha Basmanova)
• 53a84b0 Add test to verify the size of all-null blocks returned (Bhavani Hari)
• bd306cb Fix the size of all-null blocks returned from Decimal reader (Bhavani Hari)
• 7ae9f43 Fix the size of all-null blocks returned from MapFlat reader (Bhavani Hari)
• 5b294b0 Fix the size of all-null blocks returned from Timestamp reader (Bhavani Hari)
• b835ac1 Fix the size of all-null blocks returned from List reader (Bhavani Hari)
• 1b5d86d Fix the size of all-null blocks returned from Slice reader (Bhavani Hari)
• 874255d Fix the size of all-null blocks returned from Long reader (Bhavani Hari)
• 6d4250a Fix the size of all-null blocks returned from Double reader (Bhavani Hari)
• 68ca8a6 Fix the size of all-null blocks returned from Float reader (Bhavani Hari)
• 300ba88 Fix the size of all-null blocks returned from Byte reader (Bhavani Hari)
• 1b6d7f9 Fix the size of all-null blocks returned from Boolean reader (Bhavani Hari)
• 91af80f Enable testGroupByKeyPredicatePushdown with filter pushdown (Masha Basmanova)
• 4e71bf3 Optimize byte reader when reading contiguous rows with no nulls and no filter (Ying Su)
• 4e66289 Test filter-only columns with coercions (Masha Basmanova)
• 604b707 Allow range filters on columns with coercion (Masha Basmanova)
• f5e14e9 Static import TupleDomain.withColumnDomains (Masha Basmanova)
• 7e19193 Add coercing tuple domain filters (Masha Basmanova)
• 5b874b4 Introduce HiveCoercer interface (Masha Basmanova)
• 9c21aaa Move out the entire Hive metastore (James Sun)
• b7dcdb9 Add requireNonNull checks to ConnectorManager (Sahar Massachi)
• 8f1b1ae Incorporate pushed down filter into stats used by CBO (Sahar Massachi)
• 272a0e9 Rename toSymbolStatistics to toVariableStatsEstimate (Sahar Massachi)
• 154c446 Extract toSymbolStatistics from TableScanStatsRule (Sahar Massachi)
• 04019d4 Add layout handle parameter to ConnectorMetadata.getTableStatistics (Masha Basmanova)
• abfef50 Change DomainTranslator.toPredicate to allow any RowExpression (Sahar Massachi)
• 282a7f1 Add Subfield to RowExpression converter (Sahar Massachi)
• 35970d3 Fix filter pushdown for non-partitioned tables (Masha Basmanova)
• ed16368 Static import Domain.singleValue (Masha Basmanova)
• 669f9b5 Move ExtendedHiveMetastore with its deps to new presto-hive-metastore module (Islam Ismailov)
• 1a18eb9 Do not prune subfields when filter uses entire column (Masha Basmanova)
• d2a3157 Add basic support for schema evolution to OrcSelectiveRecordReader (Masha Basmanova)
• 685475f Extract coercers from HivePageSource (Masha Basmanova)
• 7c61a7e Add TestHiveClientInMemoryMetastoreWithFilterPushdown (Masha Basmanova)
• f180de3 Refactor getTableLayouts logic in AbstractTestHiveClient (Masha Basmanova)
• 564d288 Change testBucketSortedTables to use ORC format (Masha Basmanova)
• 630a592 Add TestHiveUtils.getDefaultHiveSelectivePageSourceFactories (Masha Basmanova)
• fadcc90 Rename HiveTestUtils.getDefaultHiveDataStreamFactories (Masha Basmanova)
• 40f9db0 Fix selective readers to make a copy of input positions (Bhavani Hari)
• b8acb68 Reset values after getBlock in LongSelectiveStreamReader (Bhavani Hari)
• 662a00a Decode urlencoded property for PRESTO_EXTRA_CREDENTIAL parsing (Yi He)
• 323bd92 Add Hive tests with filter pushdown and subfield pruning (Bhavani Hari)
• dcc5bb2 Output verifier events for queries being skipped by pre-filters (Leiqing Cai)
• cad9d12 Revert "Add Subfield to RowExpression converter" (Sahar Massachi)
• eac554c Add Subfield to RowExpression converter (Sahar Massachi)
• 0637dcc Revert "Add non-linear transformation for structural hash" (Wenlei Xie)
• d9c1e20 fix organizationDiscoveryIntervalMillis bug (Ke Wang)
• f601571 Fix double and float NaN comparisons (Bhavani Hari)
• 22adf22 Move TestHiveDistributedQueriesWithOptimizedRepartitioning to separate Travis Job (Bhavani Hari)
• e4769ad Add Hive smoke tests with filter pushdown and subfield pruning (Bhavani Hari)
• 89a41a9 Allow creating flat-map reader for non-supported key types (Bhavani Hari)
• 1a65486 Allow reading non-ORC tables when pushdown filter is enabled (Bhavani Hari)
• ffed56f Fix GeometryToBingTiles for certain geometries (James A. Gill)
• 7018ccd Reduce the identifier length in generated code for nested columns (Venki Korukanti)
• 69c6573 Add expand_envelope function. (James A. Gill)
• 593e790 Allow empty Envelope serialization (James A. Gill)
• 60aa23d Verify that outputPositions increase monotonically (Bhavani Hari)
• 2def2e1 Add JSON annotations to SqlFunctionId (Leiqing Cai)

[linux-foundation-easycla]

One or more committers are not authorized under a signed CLA as indicated below. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

• ❌ Leiqing Cai (4d5e368)