Add release notes for 0.240

[caithagoras]

Missing Release Notes

Beinan Wang

• Upgrade presto-hadoop-apache2 for GCS token refreshing #14890 Upgrade presto-hadoop-apache2 for GCS token refreshing (Merged by: Zhenxiao Luo)

George Wang

• Add support for limit pushdown through union #14895 Add support for limit pushdown through union (Merged by: Rongrong Zhong)

Jinyang Li

• Handle create page source for Druid segment on s3 #14922 Handle create page source for Druid segment on s3 (Merged by: Zhenxiao Luo)

Saksham Sachdev

• Implement Window Function Spilling #14909 Implement Window Function Spilling (Merged by: James Sun)

Weidong Duan

• Support more than 1 druid source #14980 Support more than 1 druid source (Merged by: Zhenxiao Luo)
• Fix druid connector bugs related to pushed down queries #14967 Fix druid connector bugs related to pushed down queries (Merged by: Zhenxiao Luo)

agrawalreetika

• Add support for ip data type in ES Connector #14996 Add support for ip data type in ES Connector (Merged by: Zhenxiao Luo)

Extracted Release Notes

• Use local private credentials (json key file) to refresh GCS access token #14585 (Author: Beinan Wang): Use local private credentials (json key file) to refresh GCS access token
  • Use local private credentials (json key file) to refresh GCS access token presto-cli --extra-credential hive.gcs.credentials.path="${PRIVATE_KEY_JSON_PATH}".
• Add a limit on total number of bytes read from storage in table scan #14739 (Author: George Wang): Add a limit on total number of bytes read from storage in table scan
  • Add query.max-scan-physical-bytes configuration and query_max_scan_physical_bytes session properties to limit total number of bytes read from storage during table scan. The default limit is 1PB.
• Add support for Hudi MOR queries #14795 (Author: Brandon Scheller): Add support for Hudi MOR queries
  • Allows presto-hive to use custom parquet input formats.
  • Add support for Hudi realtime input format for hudi realtime queries.
• Support customizable ways of launching Presto queries in Verifier #14823 (Author: Leiqing Cai): Support customizable ways of launching Presto queries in Verifier
  • Fix an issue where Verifier fails to start when failure resolver is disabled.
  • Add support to implement customized way of launching Presto queries.
  • Add support to run helper queries on a separate cluster other than the control cluster.
• Pushdown dereference #14829 (Author: Zhenxiao Luo): Pushdown dereference
  • Push down dereference expression.
• Make skip.header.line.count=1 files splittable #14866 (Author: James Petty): Make skip.header.line.count=1 files splittable
  • Adds support for splitting hive files when skip.header.line.count=1.
• Avoid checking isSplittable for files smaller than the split max size #14877 (Author: James Petty): Avoid checking isSplittable for files smaller than the split max size
  • Improves split generation by avoiding an unncessary splittable check when files are smaller than the initial split max size, regardless of their input format.
• Add bing_tile_children and bing_tile_parent functions #14896 (Author: James Gill): Add bing_tile_children and bing_tile_parent functions
  • Introduce bing_tile_children and bing_tile_parent functions to get parents and children of a Bing tile.
• Add geometry_to_dissolved_bing_tiles and improve geometry_to_bing_tiles #14899 (Author: James Gill): Add geometry_to_dissolved_bing_tiles and improve geometry_to_bing_tiles
  • Add geometry_to_dissolved_bing_tiles function, which dissolves complete sets of child tiles to their parent.
  • Improve geometry_to_bing_tiles. It is 50x faster on complex polygons, the limit on polygon complexity is removed, and some correctness bugs have been fixed.
• Kafka insert #14901 (Author: Zhenxiao Luo): Kafka insert
  • Support insert in Kafka connector.
• Fix Parquet long statistics handling when min/max not set #14903 (Author: Zhenxiao Luo): Fix Parquet long statistics handling when min/max not set
  • Fix parquet statistics when min/max is not set.
• Handle CAST in canonicalized LambdaDefinitionExpression #14912 (Author: Rongrong Zhong): Handle CAST in canonicalized LambdaDefinitionExpression
  • Fix LambdaDefinitionExpression canonicalization did not handle CAST.
• Inline SQL functions at plan time #14931 (Author: prithvip): Inline SQL functions at plan time
  • Add support for inlining SQL functions at query planning time. This feature is enabled by default, and can be disabled with the inline_sql_functions session property.
• Populate ClientInfo in Verifier JDBC queries #14941 (Author: Leiqing Cai): Populate ClientInfo in Verifier JDBC queries
  • Add support to populate client info for the queries issued by Verifier.
  • Add configuration property test_name, to be passed in to the client info blob.
• Connect to Cassandra cluster using SSL #14950 (Author: SandishKumarHN): Connect to Cassandra cluster using SSL
  • Add TLS security support.
• Support skipping control in Verifier #14958 (Author: Leiqing Cai): Support skipping control in Verifier
  • Add support to skip running control queries and comparing results. This can be enabled by configuration property skip-control.
• Use the same RowExpressionCompiler when compiling expressions #14959 (Author: Rongrong Zhong): Use the same RowExpressionCompiler when compiling expressions
  • Fix compiler error in certain situations where sql functions with same lambda are used multiple times.
• Added conversion functions between Geometry and GeoJSON format.  #14968 (Author: Rong Rong): Added conversion functions between Geometry and GeoJSON format.
  • Added :func:geometry_from_geojson and :func:geometry_as_geojson to convert geometries from and to GeoJSON format.
• Resubmit verification if test query fails with HIVE_PARTITION_OFFLINE #14970 (Author: Leiqing Cai): Resubmit verification if test query fails with HIVE_PARTITION_OFFLINE
  • Add support to resubmit verification if test query fails with HIVE_PARTITION_OFFLINE.
• Thrift udf api #14985 (Author: Rongrong Zhong): Thrift udf api
  • Rename presto-thrift-connector-api to presto-thrift-api and have separate packages for datatypes, valuesets and connector.
• Allow procedures to have optional arguments with default values #14988 (Author: Vivek): Allow procedures to have optional arguments with default values
  • Allow procedures to have optional arguments with default values.
• Add IF EXISTS and IF NOT EXISTS checks to ALTER TABLE #15003 (Author: Vivek): Add IF EXISTS and IF NOT EXISTS checks to ALTER TABLE
  • Add IF EXISTS and IF NOT EXISTS syntax to ALTER TABLE.

All Commits

• 95e5ee4 Update java version parser for java 10, 11 or later (Beinan Wang)
• 7a0c140 Support more than 1 druid source, allow union and union all operation (Weidong Duan)
• a9d5c4c Inline SQL functions at plan time (prithvip)
• dfee819 Add casts to expressions in SQL functions if coercible (prithvip)
• 6f1db90 Fix improperly scoped argument binding in lambda expressions (prithvip)
• 4c869d6 Fix OrcSelectivePageSource incorrect memory reporting (Sujay Jain)
• 06b6f96 Deprecating NullableValue class (Saumitra Shahapure)
• 32dfebd Add IF EXISTS and IF NOT EXISTS checks to ALTER TABLE (Vivek)
• ec5b590 Implement dynamic filtering collection for semi join (Ke Wang)
• 849b6a4 Add semi join dynamic filter placeholder (Ke Wang)
• 603b77c Move Presto Spark execution exceptions to presto-spark-classloader-interface (Andrii Rosa)
• c838111 Change SQL function version to String (Rongrong Zhong)
• 569d71e Add language to FunctionMetadata (Rongrong Zhong)
• 22c79d8 Fix equals, hashCode and toString for JoinFilterCacheKey (Shixuan Fan)
• a70b415 Fix hashCode for CacheKey in ExpressionCompiler (Shixuan Fan)
• 1b66b42 Allow procedures to have optional arguments with default values (Vivek)
• eed2581 Add support for ip data type in ES Connector (agrawalreetika)
• 53467ec Support legacy date/timestamp to varchar coercion (Shixuan Fan)
• d9c1e84 Add functions geometry_from_geojson and geometry_as_geojson (Rong Rong)
• 4c143d9 Upgrade Kudu to latest release 1.12.0 (SandishKumarHN)
• 922afac Remove unnecessary annotation in ThriftUdfService (Rongrong Zhong)
• 4479e85 Ensure exception thrown by scheduled tasks are logged (Andrii Rosa)
• 269c3ac Prefer builtin Page methods that avoid Block[] copies (James Petty)
• 91eb4b8 Add copyPositions and extractChannels utility methods to Page (James Petty)
• 0f8a20e Fix memorySize calculation for quantiles (Bhavani Hari)
• b990542 Add test for timestamp literal pushdown (beinan)
• be949c0 Remove the unnecessary logic on time literal pushdown (beinan)
• 67c9fad Timestamp literal push down for Druid connector (beinan)
• 7074294 Handle the error message from Druid broker (beinan)
• 42a4729 Add druid connector to devel configs (beinan)
• b15ee86 Export slim query info to file upon PoS query completion (Wenlei Xie)
• 521df87 Support FilterNode on top of AggregationNode (Weidong Duan)
• d560b6e Fix unhandled HTTP response error for druid client (Weidong Duan)
• 53c89ab Run distributed Hive join tests with and without dynamic filtering (Ke Wang)
• 8792a92 Push dynamic filter to HivePageSourceProvider (Ke Wang)
• 740e18c Fix generation of dynamic filter (Ke Wang)
• eb4e582 Fix error message in SessionPropertyDefaults (Leiqing Cai)
• 1d52c57 prestodb connect to cassandra cluster with tls security (SandishKumarHN)
• dd95430 Introduce Thrift UDF service APIs (Rongrong Zhong)
• b8a4ab3 Refactor presto-thrift-connector-api (Rongrong Zhong)
• e1b4066 Resubmit verification if test query fails with HIVE_PARTITION_OFFLINE (Leiqing Cai)
• e12f5f4 Allow Raptor to run custom check before table creation (Shixuan Fan)
• a2183fa Convert memory to user pool on HashAggregation spill finish (Saksham Sachdev)
• cfb2e7a Add hive custom split support for Hudi (Brandon Scheller)
• c406344 Support skipping control in Verifier (Leiqing Cai)
• 35ce159 Collect shuffle statistics in Presto on Spark (Andrii Rosa)
• ea81b9f Propagate fragment id to PrestoSparkShufflePageInput (Andrii Rosa)
• 1a909d5 Refactor PrestoSparkMutableRowPageInput (Andrii Rosa)
• 46499f5 Update QueryDetails UI to use new TaskStats response structure (Ajay George)
• 88e0bd0 Flatten TaskStats and PipelineStats (Ajay George)
• c960ab4 Use the same RowExpressionCompiler when compiling expressions (Rongrong Zhong)
• e9f49df Add support for limit pushdown through union (George Wang)
• e0718bb Add a constructor to PrestoQueryException (Leiqing Cai)
• d45ad47 Make PrestoQueryException.queryActionStats non-optional (Leiqing Cai)
• 4959b79 Allow extra stats to be emitted even when query stats are missing (Leiqing Cai)
• 1c406b0 Allow TableScanNode#getCurrentConstraint returns null on worker (Wenlei Xie)
• 2c247c5 Add logging on Presto-on-Spark worker (Wenlei Xie)
• fecb4a9 Use totalSize statistics for simple join plan (Peizhen Guo)
• fb60458 Add unit tests for LocalDynamicFilter and LocalDynamicFilterCollector (Ke Wang)
• c9db6e2 Short circuit page source when dynamic filter is none and Clean up LocalDynamicFiltersCollector (Ke Wang)
• 4332408 Implement local dynamic filtering for broadcast inner-joins (Ke Wang)
• 882fb23 Allow creating unlimited TypedSet (Ke Wang)
• 56503cf Add spilled bytes to QueryStatistics (Saksham Sachdev)
• 5823741 Extract dynamic filters in Hive ORC reader (Ke Wang)
• 66c22ab Extract dynamic filters in LocalExecutionPlanner (Ke Wang)
• dab5a60 Remove unsupported dynamic filters (Ke Wang)
• 05553f8 Add query plan test for DynamicFilter (Ke Wang)
• 4d4f29f Generate DynamicFilter in optimization and cost calculation process (Ke Wang)
• c2ad784 Introduce dynamicFilters into JoinNode and PlanPrinter (Ke Wang)
• 1f5da2e Introduce DynamicFilter placeholder and its builtin function (Ke Wang)
• af8d095 Add session properties for dynamic filtering (Ke Wang)
• 28b6062 Close spiller in Operator#close() for window spilling (Saksham Sachdev)
• 9925cd3 Extract window queries tests to separate class (Saksham Sachdev)
• 0eeaf84 Convert revocable memory to user memory on fully buffered group (Saksham Sachdev)
• 19097b0 Use OrderingCompiler in WindowOperator spilling (Saksham Sachdev)
• 085e1e6 Implement spill to disk For WindowOperator (Saksham Sachdev)
• d8b4fa0 Introduce revocable aggregated memory context in OperatorContext (Saksham Sachdev)
• c4fc2d9 Separate input PageSource work processor (Saksham Sachdev)
• 5351fa4 Port window operator to work processors (Saksham Sachdev)
• 74cc2dd Update airbase to 99 (Mayank Garg)
• 8e272aa Add a user specified test name that is passed into client info (Leiqing Cai)
• 0cade1d Populate ClientInfo in Verifier JDBC queries (Leiqing Cai)
• 9a3b041 Fix nondeterminism in TestQueryResource (Tim Meehan)
• 5ebb3e2 Shade classes in presto-jdbc uber jar (Mayank Garg)
• 24ac335 Add blank line for format (Beinan Wang)
• e50f5db Add Google OAuth scopes to extra credentials (Beinan Wang)
• 756f9f0 Fix transitive dependencies conflict of google oauth libraries (Beinan Wang)
• d49e768 Make presto jdbc driver support Google OAuth private key handling (Beinan Wang)
• d0dfd13 Make QueryRunner support GCS private key handling (Beinan Wang)
• 29c5f3e Implement GCS OAuth token handler (Beinan Wang)
• 48be2a9 Complete removal of joda-to-java-time-bridge. (Beinan Wang)
• fb43861 Add the timezone Asia/Qostanay into zone-index (Beinan Wang)
• 0e3cd14 Fix incompatible types error on jdk 9 and 11 (Beinan Wang)
• fc69952 Add dependency of javax.annotation into presto-catch's pom to fix build failure on java 11 (Beinan Wang)
• bd0c3e2 Move dereference pushdown below TranslateExpressions (Zhenxiao Luo)
• 6a2e94e Add session property to enable dereference pushdown (Zhenxiao Luo)
• d45962c Push down dereference expression (Zhenxiao Luo)
• a4f14e4 Handle create page source for Druid segment on s3 (Jinyang Li)
• 4a37013 Simplify integration with Alluxio local cache (Bin Fan)
• fb8bb9f Add a limit on total number of bytes read from storage in table scan (George Wang)
• 404581f Disable TestDistributedSpilledQueries#testJoinPredicatePushdown (Saksham Sachdev)
• 186ea6d Fix flaky testTransactionMetadataCleanup (Vic Zhang)
• a3feff4 Fix typo in "longitudeToTileY" function name (James Gill)
• d43913f Make skip.header.line.count=1 files splittable (James Petty)
• 52a7529 Introduce additional methods in SqlExceptionClassifier interface (Leiqing Cai)
• 1704705 Move Query#toStatementStats to a utility class (Leiqing Cai)
• 9f97c9c Fix binding failure when failure resolver is disabled (Leiqing Cai)
• b027e1e Move the logic of mangling session properties into a utility method (Leiqing Cai)
• 6a2ed8b Use a single metadata-timeout and checksum-timeout (Leiqing Cai)
• 6be7b05 Allow additional data to be emitted in QueryStatsEvent (Leiqing Cai)
• 8bd1a52 Support customizable ways of launching Presto queries in Verifier (Leiqing Cai)
• 7a4dc5a Make the entire JDBC QueryStats available in verification results (Leiqing Cai)
• e85e4e1 Make control.http-port optional (Leiqing Cai)
• 426b48b Style improvments in DeterminismAnalyzer (Leiqing Cai)
• 8473d79 Refactor geometry_to_bing_tiles to use new algorithm (James Gill)
• 5e5bb96 Reformatting some geometry_to_bing_tiles tests (James Gill)
• 22c1167 Add test cases for geometry_to_bing_tiles (James Gill)
• b96c2a2 Add geometry_to_dissolved_bing_tiles (James Gill)
• 52e6eb9 Extract accelerateGeometry into GeometryUtils (James A. Gill)
• c53732e [Test Only] Remove allFragments from PlanPrinter#formatFragment (Wenlei Xie)
• 4110ec6 Avoid checking isSplittable for files smaller than initial split size (James Petty)
• e4aa4e3 Minor refactor to LocalQueryRunner (Wenlei Xie)
• 53a26bb Add BingTile.findChildren() and findParent() (James Gill)
• 040a48a Rename RowExpressionPredicatePushDown to PredicatePushDown (James Sun)
• 0cfbeff Remove unused functions in ExpressionUtils (James Sun)
• 990a119 Remove SubExpressionExtractor (James Sun)
• 04beda3 Rename RowExpressionEqualityInference to EqualityInference (James Sun)
• ae6d0ff Remove EqualityInference (James Sun)
• 502bc06 Rename RowExpressionPredicateExtractor to EffectivePredicateExtractor (James Sun)
• 40abdce Remove EffectivePredicateExtractor (James Sun)
• 9ae92b5 Move TranslateExpressions above all PredicatePushDowns (James Sun)
• 6528824 Add "Test plan" field to PR template (Mayank Garg)
• d34897d Put time limit on TestDistributedSpillQueries#testJoinPredicatePushdown (Mayank Garg)
• 0438c0e Upgrade hadoop-apache for refreshing GCS OAuth token automatically (Beinan Wang)
• 55a685d Fix Kafka product tests avro schema (Zhenxiao Luo)
• b52bab6 Add Kafka raw encoder (Zhenxiao Luo)
• ad1bca3 Improve Kafka round trip test (Zhenxiao Luo)
• 27865e4 Fix resource warning in KafkaPageSink#appendPage (Zhenxiao Luo)
• 3c4d47b Add row to Kafka round trip test (Zhenxiao Luo)
• 4563769 Add Kafka Avro encoder (Zhenxiao Luo)
• 6ff5921 Make Kafka RowEncoder Closeable (Zhenxiao Luo)
• 5a00a63 Move column handle validation for Kafka encoders into constructor (Zhenxiao Luo)
• 9c74b62 Remove Kafka connector suppress warnings labels (Zhenxiao Luo)
• a1635df Add Kafka CSV encoder (Zhenxiao Luo)
• 8eee859 Implements Inserts for Kafka connector (Zhenxiao Luo)
• 611d210 Remove unused functions in ExpressionUtils (James Sun)
• 56bfdeb Move TranslateExpressions above all PickTableLayouts (James Sun)
• ddd2766 Move TranslateExpressions above PushAggregationThroughOuterJoin (James Sun)
• 5f30def Move TranslateExpressions below PushAggregationThroughOuterJoin (James Sun)
• fe32cbd Move TranslateExpressions above IndexJoinOptimizer (James Sun)
• 3f1023d Move TranslateExpressions above SimplifyCountOverConstant (James Sun)
• a07948a Move TranslateExpressions above LimitPushDown (James Sun)
• d46226c Move TranslateExpressions above WindowFilterPushDown (Yi He)
• 19634dc Move TranslateExpressions above GatherAndMergeWindows (James Sun)
• b39ecad Move TranslateExpressions below GatherAndMergeWindows (James Sun)
• fa288b8 Handle CAST in canonicalized LambdaDefinitionExpression (Rongrong Zhong)
• 3eb8442 Fix Parquet long statistics handling when min/max not set (Zhenxiao Luo)

[ajaygeorge]

Can you please comment with the missing release notes for the PRs that you authored/merged - @zhenxiao , @rongrong , @highker

[highker]

General Changes
* Add window function spilling capacity to local disk.

[zhenxiao]

Druid connector
* Support union all operation with more than 1 druid source
* Support Filter on top of Aggregation
* Fix unhandled HTTP response error for druid client

Elasticserarch connector
* Add support for ip data type in ES Connector

[ajaygeorge]

capturing the release notes for #14895 from the PR itself.

General Changes
* Add support for limit pushdown through union

[caithagoras]

Please add a highlight section: https://www.internalfb.com/intern/wiki/DataInfra/Presto/Releasing/Release_Preparation/#polish-release-notes

You can see 239 release notes as a reference.

[ajaygeorge]

Please add a highlight section:

@caithagoras I have added a placeholder for now. If a committer can mention the highlights I can add that in.

Also, created a PR to update the template to add in a highlights section. prestodb/presto-release-tools#12

[rschlussel]

I think this is missing some words between token and presto-cli. Is that how you use it?

[ajaygeorge]

I think this should be in two separate lines. See original PR for this #14585 .
Let me add token. Usage : presto-cli --extra-credential hive.gcs.credentials.path="${PRIVATE_KEY_JSON_PATH}".

[rschlussel]

not a user friendly description

[ajaygeorge]

@rongrong this is from #14895 . Can you help me rephrase this in a user friendly way.

[rongrong]

I don't know how to rephrase it. It basically pushes limit through union. I think it's pretty easy to understand and if we consider description of plan optimization not user friendly / not user facing we can remove this from release note.

[rschlussel]

this isn't a user friendly description

[ajaygeorge]

@zhenxiao this is from #14829 . Can you help me rephrase this in a user friendly way.

[zhenxiao]

Provide experimental.pushdown-dereference-enabled to enable pushdown of dereference expressions for querying nested data

[ajaygeorge]

I'm going to reword it as
Add support for pushdown of dereference expressions for querying nested data. This can be enabled with the pushdown_dereference_enabled session property or the experimental.pushdown-dereference-enabled configuration property.

[ajaygeorge]

@rschlussel / @tdcmeehan / @caithagoras Most of the comments are addressed . Can you please take another pass.

[tdcmeehan]

LGTM pending @caithagoras comments