Add release notes for 0.231

[caithagoras]

Missing Release Notes

Ariel Weisberg

• Cherry-pick documentation for cost, statistics, CBO #13155 Cherry-pick documentation for cost, statistics, CBO (Merged by: Rebecca Schlussel)

Ke Wang

• Raptor background jobs #13849 Raptor background jobs (Merged by: Jiexi Lin)

Rohit Jain

• Add max buffer size to Raptor session properties #13905 Add max buffer size to Raptor session properties (Merged by: James Sun)

Saksham Sachdev

• Revert "Consuming max buffer size from the session properties " #13815 Revert "Consuming max buffer size from the session properties " (Merged by: Shixuan Fan)

ptkool

• Fix IS DISTINCT FROM for decimals with precision > 18 #13668 Fix IS DISTINCT FROM for decimals with precision > 18 (Merged by: Maria Basmanova)
• Add IGNORE NULLS clause to various Window functions #10568 Add IGNORE NULLS clause to various Window functions (Merged by: Rongrong Zhong)

Extracted Release Notes

• Merge partition preference when adding exchange nodes #11262 (Author: Jiexi Lin): Merge partition preference when adding exchange nodes
  • Add new session property "aggregation_partitioning_merging_strategy" to control merging partitioning preference when adding repartition remote exchange around aggregation node. Default option is LEGACY and can be overwritten to TOP_DOWN or BOTTOM_UP.
• Use JTS instead of ESRI for geometries, Parts 1 and 2 #13604 (Author: James A. Gill): Use JTS instead of ESRI for geometries, Parts 1 and 2
  • Use JTS instead of Esri for many geometrical operations. + Polygon WKTs must have closed loops. Previously Esri would close the loops for you. + Certain other invalid geometries will fail to be created from WKTs, such as LINESTRING(0 0, 0 0, 0 0). + Returned WKTs may have a different point order. + Fixes incorrect calculation of extreme points in certain cases.
• Implement Parallel Partition Pruning for Glue Hive Metastore #13729 (Author: Anoop Johnson): Implement Parallel Partition Pruning for Glue Hive Metastore
  • Add support for parallel partition fetching for the Glue metastore.
• Raptor read and write with delta delete functionality #13756 (Author: Ke Wang): Raptor read and write with delta delete functionality
  • Add the ability to read and write with delta deletes.
• Verifier: Improve determinism analysis and checksum query recording #13758 (Author: Leiqing Cai): Verifier: Improve determinism analysis and checksum query recording
  • Fix an issue where checksum query text and ID are not recorded if the checksum query fails.
  • Add new columns control_session_properties and test_session_properties to verifier_queries, and remove column session_properties_json. The value of the removed column can be copied to the two new columns for the schema change.
  • Add details of determinism analysis runs to the output.
  • Add configuration property max-determinism-analysis-runs to control maximum number of determinism analysis runs in case of column mismatch.
  • Add configuration property run-teardown-for-determinism-analysis to allow disabling teardown for determinism analysis runs.
• Make SQL function work for input parameters with lambda #13796 (Author: Rongrong Zhong): Make SQL function work for input parameters with lambda
  • Fix SQL function compilation error when input parameters contain lambda.
• Sort OOM killer log output #13820 (Author: Ariel Weisberg): Sort OOM killer log output
  • OOM killer log output sorted to put memory heavy nodes and queries first.
• Use JTS for ST_Buffer #13824 (Author: James A. Gill): Use JTS for ST_Buffer
  • Use more efficient implementation for ST_Buffer. This produces fewer buffer points on rounded corners, which will produce very similar but different results. JTS also better handles buffering with small (<1e-9) distances.
• Fix function namespace manager instantiation failure #13842 (Author: Leiqing Cai): Fix function namespace manager instantiation failure
  • Fix an issue where server fails to start when two function namespace managers of the same type are specified.
• Skip verification when checksum query fails to compile #13853 (Author: Leiqing Cai): Skip verification when checksum query fails to compile
  • Improve Verifier to skip verification when checksum query fails to compile.
• Fix schema mismatch w/Parquet INT64 & Timestamp #13857 (Author: Ariel Weisberg): Fix schema mismatch w/Parquet INT64 & Timestamp
  • Fix schema mismatch w/Parquet INT64 & Timestamp.
• Get PrestoS3FileSystem to work with the AWS Default Credentials Provider #13858 (Author: Anoop Johnson): Get PrestoS3FileSystem to work with the AWS Default Credentials Provider
  • Get PrestoS3FileSystem to work with the AWS Default Credentials Provider.
• Add approx_percentile forms with accuracy #13871 (Author: James A. Gill): Add approx_percentile forms with accuracy
  • Add forms of approx_percentile accepting an accuracy parameter.
• Fix raw input stats for ScanFilterAndProject #13873 (Author: James Petty): Fix raw input stats for ScanFilterAndProject
  • Fix ScanFilterAndProjectOperator raw input bytes accounting for LazyBlocks.
• Optimize array_join by supporting PROVIDED_BLOCKBUILDER convention #13874 (Author: Wenlei Xie): Optimize array_join by supporting PROVIDED_BLOCKBUILDER convention
  • Optimizer performance for array_join.
• Add query integrity check in AccessControlManager #13878 (Author: Yi He): Add query integrity check in AccessControlManager
  • Add API to check if the query is unexpectedly modified using the credentials passed in the identity.
• Keep sum stat for string when min/max stat is too long in ORC writer #13879 (Author: Islam Ismailov): Keep sum stat for string when min/max stat is too long in ORC writer
  • Output (sum) string stat even if min/max values are too long. This is needed for the read-path to be able to better estimate the size of row.
• Add support for get partition names by filter #13898 (Author: Ajay George): Add support for get partition names by filter
  • Add support for get partition names by filter.
• Fix invalid plan for repeated lambdas in order by #13901 (Author: Rebecca Schlussel): Fix invalid plan for repeated lambdas in order by
  • Fix failures caused by invalid plans for queries with repeated lambda expressions in the order by clause.
• Add caching file system to hive connector #13904 (Author: Rohit Jain): Add caching file system to hive connector
  • Allow reading data from HDFS while caching the fetched data on local disks. Turn on the feature by specifying the cache directory config cache.base-directory.
• Add sum of cardinality for array column verifier results #13916 (Author: Yi Shen): Add sum of cardinality for array column verifier results
  • Add checks for the cardinalities sum when validating an array column.
• Improve error handling for geometry deserialization edge cases #13922 (Author: James A. Gill): Improve error handling for geometry deserialization edge cases
  • Improve error handling for geometry deserialization edge cases.

All Commits

• 9008f5a Extract Verifier intergation test into abstract test (Leiqing Cai)
• 9a9ed94 Check retryable errors are all recoginzed (Leiqing Cai)
• ca6a2d3 Allow more flexible construction of PrestoExceptionClassifier (Leiqing Cai)
• a5bb640 Add support for get partition names by filter (Ajay George)
• d22cd7d Remove totalPartitions from TaskUpdateRequest (Andrii Rosa)
• 2b7926f Add approx_percentile forms with accuracy (James A. Gill)
• 262b74f Refactor approx_percentile functions (James A. Gill)
• fbb9e98 Capture geometry deserialization failures (James A. Gill)
• 910374b Use helper method jtsGeometryFromWkt (James A. Gill)
• 82c9ea0 Publish tarball for Verifier (Leiqing Cai)
• 628afed Fix corrupted verifier executable jar (Leiqing Cai)
• fd59547 Fix invalid plan for repeated lambdas in order by (Rebecca Schlussel)
• f948092 Move OptimizeMixedDistinctAggregations below Logical Optimization (Saksham Sachdev)
• 3f8d9da Inject cache manager for Raptor HdfsModule with provider (James Sun)
• 2762513 Add caching file system to hive connector (Rohit Jain)
• 5e57428 Add high level CBO documentation (Ariel Weisberg)
• 7370838 Add documentation for cost and statistics (Lukasz Osipiuk)
• b219152 Increase visibility of the PrestoSystemRequirements (Andrii Rosa)
• 48fe432 Introduce RemoteSourceFactory abstraction (Andrii Rosa)
• 8a11650 Expose connector property for TPCH partitioning (Andrii Rosa)
• 0f14563 Make require Hadoop native configurable (Andrii Rosa)
• 072f645 Refactor TableScanOperator stats collection (James Petty)
• 0191168 Fix raw input stats for ScanFilterAndProject (James Petty)
• 176e8f8 Add checks for the cardinalities sum when validating an array column (Yi Shen)
• ad99885 Repurpose OrderableArrayColumnValidator to support all array types (Yi Shen)
• 659c3ee Optimize ByteSelectiveStreamReader for no filter with nulls (Ying Su)
• bdea351 Allow hive outstanding splits size limit larger than 2GB (James Petty)
• 58247b0 Skip logging of GCStatusMonitor state if there is no activity (Ajay George)
• 3a4f2e3 Optimize boolean reader when reading contiguous rows with no nulls and no filter (Eugene Kalenkovich)
• 0185288 Clean up arrays in ORC dictionary after close (James Sun)
• b2a5e70 Make BenchmarkSelectiveStreamReaders load LazyBlock (Eugene Kalenkovich)
• 9016a06 This patch updates the description for redis.key-prefix-schema-table to (Joseph Gmitter)
• 1a3d5ff Fix IS DISTINCT FROM for long decimals (ptkool)
• 49ea39b Add session property to disable adaptive filter reordering (Masha Basmanova)
• eb207f8 Cache results of TupleDomainFilterUtils::toFilter (Masha Basmanova)
• 7360bce Add max buffer size to Raptor session properties (Rohit Jain)
• 84d9f0a Eliminate duplicate table name to avoid conflict in TestRaptorIntegrationSmokeTest (Ke Wang)
• 241a3f5 Keep sum stat for string when min/max stat is too long in ORC writer (Islam Ismailov)
• 25e1d79 Add query integrity check in AccessControlManager (Yi He)
• c8fc21e Integrate PrestoS3FileSystem with AWS Default Credentials Provider (Anoop Johnson)
• f4f8882 Cache optimized RowExpression predicate (Bhavani Hari)
• 5177bea Fix requiredField logic in StructSelectiveStreamReader (Bhavani Hari)
• 5f781d3 Add delta in reassign process (Ke Wang)
• a60985c Add priority to Compaction process (Ke Wang)
• 618b0af Enable delta compaction and organization (Ke Wang)
• 145a652 Handle failed queries and execute remaining (Nikhil Collooru)
• 522c912 Create benchmark runner entry point (Nikhil Collooru)
• 0945361 Add benchmark runner logic (Nikhil Collooru)
• d58d687 Export results using event clients (Nikhil Collooru)
• 96d1a04 Support execution of concurrent phases (Nikhil Collooru)
• eacf134 Provide retry utilities for benchmark runner (Nikhil Collooru)
• 2c4b93b Support querying Presto in benchmark runner (Nikhil Collooru)
• fdb7611 Optimize array_join by supporting PROVIDED_BLOCKBUILDER convention (Wenlei Xie)
• ce68813 Move ExtractSpatialJoins to after LogicalOptimization (Saksham Sachdev)
• 82e90bf Control merging partitioning in aggregation node via session property (Jiexi Lin)
• 43e8c02 Force repartition when aggregation node has mixed grouping sets (Jiexi Lin)
• 8abba9e Use JTS for ST_Buffer (James A. Gill)
• 584a92f Add Benchmarks for ST_Buffer (James A. Gill)
• 7db5fbf Add Geometry construction methods to GeometryBenchmarkUtils (James A. Gill)
• 7a38da8 Fix double and float comparisons for NaNs (Bhavani Hari)
• 7841028 Refactor OrcPageSource (Ke Wang)
• c533c6f Export statistics of delta delete (Ke Wang)
• e2af436 Commit protocol for rewrite delete, delta delete (Ke Wang)
• 2a97a86 Add delta delete functionality on worker (Ke Wang)
• 3749dad Enhance RaptorSplit and enable delta read (Ke Wang)
• e01dda2 Increase timeout in TestBackupManager#testCorruption (Leiqing Cai)
• 58b3767 Remove unused variable from BigintValuesUsingHashTable (Masha Basmanova)
• c7d8f55 Use bitmask for IN filter when range of values is small (Masha Basmanova)
• 49a9cd9 Rename BigintValues to BigintValuesUsingHashTable (Masha Basmanova)
• bdd91af Optimize and refactor array_distinct, array_filter and array_sort (Sreeni Viswanadha)
• a123e5c Enable subfield pruning to pass through arbitrary() function (Gautam Parai)
• 62c4524 Modify table definition for new testcases (Gautam Parai)
• cb4ffb3 Disable TestDistributedSpilledQueries#testLimitWithJoin (Leiqing Cai)
• 5545874 Skip verification when checksum query fail to compile (Leiqing Cai)
• bb17af3 Fix schema mismatch w/Parquet INT64 & Timestamp (Ariel Weisberg)
• 53bb79f Fix SQL formatting for CREATE FUNCTION (Leiqing Cai)
• ab80d8d Fix function namespace manager instantiation failure (Leiqing Cai)
• c270e80 Sort OOM killer log output (Ariel Weisberg)
• 762b183 Fix selective stream readers (Masha Basmanova)
• 469637d Set boolean value to 0 if the position is null (Andrii Rosa)
• 8c2dac4 Check aggregation mask values for null (Andrii Rosa)
• 378e063 Log queries and tasks memory reservation on full GC (Kelvin Fann)
• 77eb6e0 Create zero row files without overwrite flag (Andrii Rosa)
• efb0c6b Rename PrestoResourceClient to NodeResourceClient (Nikhil Collooru)
• 692eb2d Fix deserailization error in verifier (Nikhil Collooru)
• 95eb8a4 Remove Unused Argument in ParquetPageSource (James Petty)
• 85fa3e7 Remove Redundant Parquet Column Index Lookups (James Petty)
• b6aa6e9 Populate basic query info for skipped events in Verifier (Leiqing Cai)
• 95a6f6d Optimize zero row files creation for bucketed tables (Andrii Rosa)
• 0ade474 Optimize loop over filter functions in applyFilterFunctions (Masha Basmanova)
• e91595d Reorder filters at runtime (Masha Basmanova)
• 789364e Track filter function performance (Masha Basmanova)
• 48e6220 Add toString method to TIMESTAMP reader (Masha Basmanova)
• ac93bd6 Evaluate filter function as soon as all inputs have been read (Masha Basmanova)
• 5589d3d Read columns of primitive types first (Masha Basmanova)
• 3938c79 Use a constant for empty page (Masha Basmanova)
• b526246 Fix BooleanInputStream#getSetBits (Masha Basmanova)
• 98b8c8d Minor refactor to TableFinishOperator (Wenlei Xie)
• 46ae108 Rename GeometrySerde to EsriGeometrySerde (James A. Gill)
• ab32977 Convert GeoFunctions to JTS: Part 2 (James A. Gill)
• a5eec6c Convert WKT functions to JTS (James A. Gill)
• f66cb96 Upgrade JTS to 1.16.1 (James Gill)
• 951eb8d Reduce Memory used by Finished AsyncQueue Instances (James Petty)
• 0f7061f Enable bucket pruning for IN predicates (Gautam Parai)
• 40d35ee Display bucket pruning info in hive table layout (Gautam Parai)
• e94f2c1 Revert "Consuming max buffer size from the session properties instead of configs in the Raptor page sink provider." (Saksham)
• 1d2be6c Consuming max buffer size from the session properties instead of configs in the Raptor page sink provider. (Rohit Jain)
• b978025 Implement Parallel Partition Pruning for Glue Hive Metastore (Anoop Johnson)
• 6436aa6 Fail creation of table with unsupported bucket count (Vic Zhang)
• f582c6f Add a method to compare SqlInvokedFunction (Leiqing Cai)
• 22f70d4 Fix pushdown for filters on multiple subfields of a struct (Masha Basmanova)
• 1d32324 Make SQL function work for input parameters with lambda (Rongrong Zhong)
• 91f3ce7 Make TestUpgradeMetadata single threaded (Leiqing Cai)
• d838c9b Add an option to disable teardown for determinism analysis runs (Leiqing Cai)
• 2d5ded9 Fix variable name in VerifierConfig (Leiqing Cai)
• 5de945e Make determinism analysis run count configurable (Leiqing Cai)
• bba5aa2 Export determinism analysis details (Leiqing Cai)
• 7cedd37 Export checksum query and query id even if the query fails (Leiqing Cai)
• 2d3a241 Make tableName required in QueryBundle (Leiqing Cai)
• ed547be Move LimitQueryDeterminismAnalyzer.Analysis to top level (Leiqing Cai)
• e91519a Use different columns for control and test session properties (Leiqing Cai)
• f074d63 Move verifier_queries table creation SQL to VerifierDao (Leiqing Cai)
• bc565eb Fix flaky test TestRetryDriver (Leiqing Cai)
• 70ab7ac Check column exist before alter table for raptor delta delete (Ke Wang)
• 36e4e86 Update hive.rst with Alluxio docs (Haoyuan Li)
• 7efeecc Add support for null treatment clause to various window functions (ptkool)
• 350f1cd Create NestedField using lowercase field name (Masha Basmanova)
• e9c57c8 Make subfield name matching case insensitive (Masha Basmanova)

[aweisberg]

No release note for #13155

[rschlussel]

Please make sure to follow the order of sections and changes within sections that's specified here: https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines

Also, make sure each line ends with a period for consistency.

[rongrong]

In general check the quoting of variables, properties, keywords, etc.

[rongrong]

LGTM in general! Though I'd trust @rschlussel more than myself. 😛

[rschlussel]

Basically looks good and I have to head out. @rongrong No need to wait for me to review again before merging.

[rschlussel]

I still don't think this is very clear -- what is a partition merging strategy? How would a user decide when to set this property or what the appropriate setting should be?

Or if it's not really user facing, maybe it doesn't need a release note.

[jessesleeping]

It's a little bit tricky (at least for me) to summarize the change of behavior in a short note. If it's doable, can we keep this note with a link to the PR where I have detailed description on the change of behavior?

Another option is to exclude this note as I can imagine only very sophisticated user will need to leverage it. By default it does not change the current behavior.

[bhhari]

ok added a link to the PR

[rschlussel]

we can probably leave out the note then, but adding a link to a PR or issue is also an option

you can link to an issue or pr using the following syntax as appropriate.

For issue #1234
:issue:`1234`

For pr #1234
:pr:`1234`

[bhhari]

yes added a link to the PR

[rschlussel]

Thanks @bhhari i'll merge this once tests pass