Add release notes for 0.234

[caithagoras]

Missing Release Notes

Cem Cayiroglu

• A new end point to get resource group stats #14177 A new end point to get resource group stats (Merged by: Andrii Rosa)

Jonas Bauer

• 54dd54c Minor exception message fix for ArrayBlockBuilder

Ke

• 0834046 Update connectors.rst
• 8f1c093 Update connectors.rst
• 446e7f5 Update connectors.rst
• 82f8a4b Update connectors.rst

Rohit Jain

• Add ZSTD support for writing ORC and DWRF tables #14113 Add ZSTD support for writing ORC and DWRF tables (Merged by: Maria Basmanova)

Sreeni Viswanadha

• Fix bucket pruning when value is null and the predicate is 'IS NULL'. #14276 Fix bucket pruning when value is null and the predicate is 'IS NULL'. (Merged by: Rongrong Zhong)
• Add @JsonProperty tag to ignoreNulls flag so it works in distributed … #14221 Add @JsonProperty tag to ignoreNulls flag so it works in distributed … (Merged by: Rongrong Zhong)

Extracted Release Notes

• Add KHyperLogLog type and UDF #14119 (Author: Matías Correa): Add KHyperLogLog type and UDF
  • Add KHyperLogLog Type.
  • Add MAKE_KHYPERLOGLOG(K, V) -> KHYPERLOGLOG aggregate function.
  • Add KHyperLogLog related scalar functions.
• Add cast bingtile to/from bigint #14125 (Author: James A. Gill): Add cast bingtile to/from bigint
  • Add cast(tile AS bigint) and cast(bigint_value AS bingtile) to encode/decode Bing tiles to/from bigints. This is a more efficient storage format that also reduces bucket skew in some cases.
• Enable spatial join for spherical geography ST_Distance #14137 (Author: James A. Gill): Enable spatial join for spherical geography ST_Distance
  • Enable spatial joins for ST_Distance(p1, p2) < r for spherical geography points p1 and p2.
• Support determinism analysis for simple queries with top-level ORDER BY LIMIT clause #14181 (Author: Leiqing Cai): Support determinism analysis for simple queries with top-level ORDER BY LIMIT clause
  • Support determinism analysis for simple queries with top-level ORDER BY LIMIT clause. (:pr:14181).
• Improve scale writer creation based on producer buffer #14193 (Author: Wenlei Xie): Improve scale writer creation based on producer buffer
  • Improve the scale writer heuristics by considering overall producer buffer utilization. This can be enabled by using the session property optimized_scale_writer_producer_buffer and the configuration property optimized-scale-writer-producer-buffer.
• Keep queries in the queue when task count is high #14194 (Author: Vic Zhang): Keep queries in the queue when task count is high
  • Add support for stopping new query execution when total number of tasks exceeds a set limit to help with reliability. The limit can be set by the cluster property by experimental.max-total-running-task-count-to-halt-scheduling.
• Limit pushdown for Druid connector #14200 (Author: Zhenxiao Luo): Limit pushdown for Druid connector
  • Add LIMIT evaluation pushdown to Druid connector.
• Allow forcing streaming exchange for Mark Distinct #14216 (Author: Ariel Weisberg): Allow forcing streaming exchange for Mark Distinct
  • Allow forcing streaming exchange for Mark Distinct using 'query.use-streaming-exchange-for-mark-distinct'.
• Aggregation Pushdown for Druid connector #14224 (Author: Zhenxiao Luo): Aggregation Pushdown for Druid connector
  • Aggregation Pushdown for Druid connector.
• Optimize socket pooling #14225 (Author: Andrii Rosa): Optimize socket pooling
  • Optimize connection pooling to avoid running out of sockets in certain cases.
• Auto resolve time limit exceeded on checksum queries #14230 (Author: Leiqing Cai): Auto resolve time limit exceeded on checksum queries
  • Add support to auto-resolve control check query failures due to EXCEEDED_TIME_LIMIT.
• Add AccessControlContext to store client information for security purposes #14248 (Author: Nikhil Collooru): Add AccessControlContext to store client information for security purposes
  • All the methods in SystemAccessControl now take additional parameter AccessControlContext context.
• Set use-legacy-scheduler true by default #14264 (Author: Rebecca Schlussel): Set use-legacy-scheduler true by default
  • Change the default for configuration property use-legacy-scheduler to true in order to mitigate a regression in the new scheduler.
• Support verifying SELECT queries with DATE or UNKNOWN columns #14266 (Author: Leiqing Cai): Support verifying SELECT queries with DATE or UNKNOWN columns
  • Add support for verifying SELECT queries that produce DATE or UNKNOWN (null) columns.
• Handle coalesce partition handle when bucket count does not match but compatible #14267 (Author: Rongrong Zhong): Handle coalesce partition handle when bucket count does not match but compatible
  • Fix query failures with incompatible partition handle when session property partial_merge_pushdown_strategy is set to PUSH_THROUGH_LOW_MEMORY_OPERATORS and optimize_full_outer_join_with_coalesce is set to true and query has mismatched partition and uses FULL OUTER JOIN with COALESCE.
• Support verifying structured types containing DATE and UNKNOWN #14273 (Author: Leiqing Cai): Support verifying structured types containing DATE and UNKNOWN
  • Add support for verifying SELECT queries that produce structured types containing DATE or UNKNOWN (null).

All Commits

• 0f2b203 Reuse buffers in BlockEncodingBuffer (Ying Su)
• 5b8b26e Add byte[] support to ArrayAllocator (Ying Su)
• 3c2e9ee Return arrays in reverse order of borrowing (Ying Su)
• ec684fb Clean up queries in DispatchManager (Tim Meehan)
• efd979a Add AccessControlContext to store client information for security purposes (Nikhil Collooru)
• 61e3c88 Move SerializedPage to SPI (Vic Zhang)
• 4e5dd5f Remove guava dependency from SerializedPage (Vic Zhang)
• 96409c3 Move PageCodecMarker to SPI (Vic Zhang)
• 151f57b Move BlockSerdeUtil to SPI (Vic Zhang)
• acb8af4 An end point to get resource group level stats (Cem Cayiroglu)
• e9b4d1f Fix getRetainedSizeInBytes for DecodedBlockNode (Ying Su)
• 410dd09 Move RemoteSplit URI construction from coordinator to worker (Shixuan Fan)
• 59ddb0b Remove unused ConnectorIdentity#{equals, hashCode} (Wenlei Xie)
• 7af7880 Add partitioned spherical spatial joins (James A. Gill)
• 0638aa0 Extract spherical geography functions to SphericalGeoFunctions (James A. Gill)
• 9bebca5 Enable spatial join for spherical geography ST_Distance (James A. Gill)
• 8bc8c94 Extract some spherical geometry functions to helper class (James A. Gill)
• 6028f4d Create ThriftBufferResult and ThriftSerializedPage (Vic Zhang)
• 5f21344 Add PrestoSparkAuthenticatorProvider (Wenlei Xie)
• 9c32d6b Introduce TokenAuthenticator (Wenlei Xie)
• 68e4367 Handle coalesce partition handle with mismatched bucket count (Rongrong Zhong)
• dd38222 Change default value for colocated_join to true (Rongrong Zhong)
• 6613a6a Fix bucket pruning when value is null and the predicate is 'IS NULL'. (Sreeni Viswanadha)
• e90fd6b Reuse positions, mappedPositions and offsets in BlockEncodingBuffer (Ying Su)
• acac2e4 Add isNested flag to BlockEncodingBuffer (Ying Su)
• ed23d49 Reuse offsetsCopy in BlockEncodingBuffer (Ying Su)
• df3b10a Add ArrayAllocator to BlockEncodingBuffer (Ying Su)
• d3f9cfc Reuse serializedRowSizes buffer between different PartitionBuffers (Ying Su)
• 1bbd131 Support verifying structured types containing DATE and UNKNOWN (Leiqing Cai)
• ed7a6ef Use Type instead of TypeSignature in QueryRewriter (Leiqing Cai)
• 613e19e Remove usage of String.format in HTTP client hot path (James Petty)
• 5078fd6 Support verifying SELECT queries with UNKNOWN columns (Leiqing Cai)
• 65a29b3 Support verifying SELECT queries with DATE columns (Leiqing Cai)
• ae3cafe Improve JdbcPrestoAction (Leiqing Cai)
• 7dc0453 Optimize parquet gzip decompression (James Petty)
• 075b70e Set use-legacy-scheduler true by default (Rebecca Schlussel)
• a3df4d1 Avoid Exceptions when closing queryRunner (Leiqing Cai)
• 0bfc8cc Add toString() to SimpleArrayAllocator (Ying Su)
• ac18342 Add benchmarks for Parquet reader (Venki Korukanti)
• 32011d8 Make PageReader class public (Venki Korukanti)
• a99684e Removed unused method (Venki Korukanti)
• 528bd5c Move ColumnIOConverter to presto-parquet package (Venki Korukanti)
• 5bad5a3 Add FileParquetDataSource for testing purpose (Venki Korukanti)
• fd5a7dd Removed unused ParquetDataSource#size (Venki Korukanti)
• 11ae1f2 Revert "Update connectors.rst" (Ke Wang)
• 0491783 Fix subscript expression optimization during subfield pushdown (Bhavani Hari)
• caff58e Disable Presto-on-Spark query stats collection (Wenlei Xie)
• ad1be42 Aggregation Pushdown for Druid connector (Zhenxiao Luo)
• c61194f Do not cancel running HTTP requests (Andrii Rosa)
• aa77278 Update Airlift to 0.191 (Andrii Rosa)
• 7c7fa10 Remove overrides for idle connection timeout (Andrii Rosa)
• 7dda502 Refactor PrestoSparkLauncher (Andrii Rosa)
• 000fa29 Reduce positions array size in PartitionBuffer (Ying Su)
• dd86faa Fix memory tracking for PartitionedOutputOperator (Ying Su)
• 2b43154 Add support for testing DictionaryBlock with non-zero offset (Ying Su)
• 660f0a7 Fix BlockFlattener when DictionaryBlock's idsOffset is nonzero (Ying Su)
• f88d652 Refactor TestBlockFlattenner (Ying Su)
• 5d1797c Add ZSTD support for writing ORC and DWRF tables (Rohit Jain)
• b081418 Fix ArrayBlockEncodingBuffer#getRetainedSizeInBytes() (Ying Su)
• 39c43b6 Add @JsonProperty tag to ignoreNulls flag so it works in distributed execution as well. (Sreeni Viswanadha)
• f35c660 Keep queries in the queue when task count is high (Vic Zhang)
• f6fd9ef Refactor TestQueryTaskLimit (Vic Zhang)
• f13cee1 Add configuration property to control query queuing (Vic Zhang)
• 66e94ef Rename configuration properties for task count limit (Vic Zhang)
• f9bf064 Allow forcing streaming exchange for Mark Distinct (Ariel Weisberg)
• 4d5bcff Use static imports in TestAddExchangePlans (Ariel Weisberg)
• 7bfc203 Add purger to ExecutingStatementResource (Tim Meehan)
• 3af06bd Simplify token management in protocol Query (Tim Meehan)
• 916cc6c Fix result caching in protocol Query (Tim Meehan)
• 4a992da Catch errors from LocalDispatchQuery querySubmitter (Tim Meehan)
• 58c0b7e Change local dispatch to finish immediately after query submission (Tim Meehan)
• 87b8729 Cleanup dispatcher executor management (Tim Meehan)
• c372011 Fix handling of failures during query creation (Tim Meehan)
• 42b40c7 Simplify query manager stats tracking (Tim Meehan)
• 9c17fc1 Rename SqlQueryManagerStats to QueryManagerStats (Tim Meehan)
• b93f31e Simplify DispatchInfo construction (Tim Meehan)
• 39d639f Remove Optional from QueryStateMachine resourceGroup (Tim Meehan)
• 7999f84 Improve query event stats for immediately failed queries (Tim Meehan)
• e23009a Add LocalCoordinatorLocation (Tim Meehan)
• 2240313 Add peak tasks to BasicQueryStats (Tim Meehan)
• a92f161 Split out queued phase from QueryManager (Tim Meehan)
• f72d511 Add DISPATCHING query states (Tim Meehan)
• 8b7c734 Remove system startup minimum worker requirement (Tim Meehan)
• 153761b Add query id to NoSuchElementException (Tim Meehan)
• 8d5283d Optimize initial batch size for scan (Masha Basmanova)
• a902a34 Harden blockEncodingBuffers in OptimizedPartitionedOutputOperator (Ying Su)
• 0834046 Update connectors.rst (Ke)
• 8f1c093 Update connectors.rst (Ke)
• 446e7f5 Update connectors.rst (Ke)
• 82f8a4b Update connectors.rst (Ke)
• 3977cbf Improve entropy for bing tile bigint encoding (James A. Gill)
• e167e46 Add BingTile cast to/from bigint (James A. Gill)
• 59f137a Auto-resolve EXCEEDED_TIME_LIMIT on control checksum queries (Leiqing Cai)
• a021f7f Fix reading a map column having negative integers as keys (Bhavani Hari)
• 0b7ab4b Rename PinotConnectorPlanOptimizer to PinotPlanOptimizer (James Sun)
• b03e989 Rename DruidConnectorPlanOptimizer to DruidPlanOptimizer (James Sun)
• e57af67 Fix typo in NodeSelectionStats (Shixuan Fan)
• f4b7869 Create KHyperLogLog type (Matías Correa)
• 411b5c6 Add nodeSelectionStats to monitor affinity scheduler (Ke Wang)
• fd0ec6b Add test methods to BenchmarkPartitionedOutputOperator (Ying Su)
• 65606a1 Fix BenchmarkPartitionedOutputOperator (Ying Su)
• f5d93fb Minor fix to BenchmarkPartitionedOutputOperator (Ying Su)
• 2ab2543 Improve scale writer creation based on producer buffer (Wenlei Xie)
• a4608e1 Fix candiateNodes selection for Soft Affinity (Ke Wang)
• 54dd54c Minor exception message fix for ArrayBlockBuilder (Jonas Bauer)
• 0e144a6 Add allocation info to Task/Stage/QueryStats (Masha Basmanova)
• 76e311d Add allocation info to PipelineStats (Masha Basmanova)
• ec3dd65 Add allocation into to DriverStats (Masha Basmanova)
• e10de40 Refactor TestQueryStats (Masha Basmanova)
• 419d155 Add allocation info to OperatorStats (Masha Basmanova)
• 5fe659f Track allocations per operator (Masha Basmanova)
• a9d1d33 Propagate flags to track allocation to OperationTimer (Masha Basmanova)
• a64e5d8 Add config properties to enable tracking memory allocations (Masha Basmanova)
• 569be23 Split test-hive-materialized on Travis (Leiqing Cai)
• 757fd25 Remove version tags for presto projects from sub-module pom files (Leiqing Cai)
• b1c5015 Move Hive filter pushdown logic out of PickTableLayout (Saksham Sachdev)
• 9fa0cae Separate some modules into their own travis jobs to reduce test times (Sujay Jain)
• 5dadf5d Throw OrcCorruptionException for column type mismatch (Bhavani Hari)
• 05407a7 Remove unused HiveFileContext in OrcReader (James Sun)
• f8ee9fd Add LIMIT evaluation pushdown to Druid connector (Zhenxiao Luo)
• d461465 Allow column pruning in SHOW STATS (Masha Basmanova)
• d8d3939 Materialize LazyBlock in scan (Ying Su)
• b36f463 Break presto-tests Travis jobs to reduce test times (Bhavani Hari)
• 6d662bd Fix finishStatisticsCollection to have correct ConnectorSession (Mayank Garg)
• b429ddb Wait for final task info on abort (Andrii Rosa)
• 1f77466 Always use JSON for PlanFragment serialization (Tim Meehan)
• 7aed008 Support determinism analysis for simple ORDER BY LIMIT queries (Leiqing Cai)
• 1e070b2 Improve Presto results conversion (Leiqing Cai)

[caithagoras]

@cemcayiroglu Can you please add the missing release notes here from the comment?

[caithagoras]

@rschlussel @aweisberg Can you help taking a look? Thanks!

[rschlussel]

Just one question, the rest looks good. Thanks!

[aweisberg]

Left one suggestion to make clear that forcing streaming exchange for mark distinct is used with materialized exchange.