Add release notes for 0.229

[caithagoras]

Missing Release Notes

Ajay George

• [pull] master from prestodb:master jzmq/presto#41 [pull] master from prestodb:master
• [pull] master from prestodb:master jzmq/presto#18 [pull] master from prestodb:master

James A. Gill

• c006c80 Fix GeometryToBingTiles for certain geometries

Ke Wang

• b4752cc fix organizationDiscoveryIntervalMillis bug

Leiqing Cai

• 7e12d42 Add JSON annotations to SqlFunctionId

Rongrong Zhong

• Function implementation #13384 Function implementation (Merged by: Rongrong Zhong)
• Revert HashBuilderOperator changes #13615 Revert HashBuilderOperator changes (Merged by: Rongrong Zhong)

Saksham Sachdev

• Implement Basic Example For Jdbc Row Expression Translation #13526 Implement Basic Example For Jdbc Row Expression Translation (Merged by: James Sun)

Yi He

• Row expression predicate pushdown #13112 Row expression predicate pushdown (Merged by: James Sun)
• Add RowExpressionPredicateExtractor #13560 Add RowExpressionPredicateExtractor (Merged by: James Sun)

oerling@fb.com

• Fix position count of prefilled Blocks #13631 Fix position count of prefilled Blocks (Merged by: Maria Basmanova)
• Fix getBlockView for all-null SliceDictionarySelectiveReader #13630 Fix getBlockView for all-null SliceDictionarySelectiveReader (Merged by: Maria Basmanova)
• Use fast varint read path in LongInputStreamV1 #13607 Use fast varint read path in LongInputStreamV1 (Merged by: James Sun)
• Intersect pushdown predicates #13505 Intersect pushdown predicates (Merged by: Rebecca Schlussel)

Extracted Release Notes

• Add peak memory distribution among tasks of each stage #13379 (Author: Vic Zhang): Add peak memory distribution among tasks of each stage
  • Add peak total memory distribution among tasks of each stage.
• Support max_tasks_per_stage for scan #13477 (Author: Cem Cayiroglu): Support max_tasks_per_stage for scan
  • Respect stage.max-tasks-per-stage to limit number of tasks for scan.
• Adding presto-pinot with pushdown working #13504 (Author: Devesh Agrawal): Adding presto-pinot with pushdown working
  • Added a new Pinot connector.
  • Split ConnectorPlanOptimizerProvider to have two phases for connectors to participate query optimization: LOGICAL and PHYSICAL. The two phases correspond to pre- and post-shuffle optimization respectively.
• Passing X_Forwarded_For in header from proxy server #13527 (Author: Swapnil Tailor): Passing X_Forwarded_For in header from proxy server
  • Start forwarding X_Forwarded_For in header from Proxy.
• Support using remote HDFS as storage in Raptor #13535 (Author: Jiexi Lin): Support using remote HDFS as storage in Raptor
  • Change storage.data-directory from path to URI. For existing deployment on local flash, a scheme header "file://" should be added to the original config value.
  • Change error code name RAPTOR_LOCAL_FILE_SYSTEM_ERROR to RAPTOR_FILE_SYSTEM_ERROR.
• Move SetOperationNode to SPI #13546 (Author: James Sun): Move SetOperationNode to SPI
  • Move SetOperationNode to SPI. Connectors can now alter query plan to have union, intersect, and except.
• Support CREATE FUNCTION execution #13561 (Author: Leiqing Cai): Support CREATE FUNCTION execution
  • Add support for CREATE FUNCTION.
• Fail query with large taskUpdate size #13584 (Author: Nikhil Collooru): Fail query with large taskUpdate size
  • Add configuration property experimental.internal-communication.max-task-update-size to limit the size of the taskUpdate.
• Add common InternalCommunicationConfig to turn on HTTPS #13592 (Author: Yi He): Add common InternalCommunicationConfig to turn on HTTPS
  • Added configs to use common HTTPS settings for internal communications.
• Parquet Dictionary Predicate Pushdown Fixes #13594 (Author: James Petty): Parquet Dictionary Predicate Pushdown Fixes
  • Fix parquet predicate pushdown on dictionaries to consider more than just the first predicate column.
  • Improve parquet predicate pushdown on dictionaries to avoid reading additional data after successfully eliminating a block.
• Add non-linear transformation for structural hash #13610 (Author: Wenlei Xie): Add non-linear transformation for structural hash
  • Improve hash value computation for structrual types to avoid collision.
• Add geospatial function line_interpolate_point #13612 (Author: James Gill): Add geospatial function line_interpolate_point
  • Add geospatial function line_interpolate_point.

All Commits

• b4752cc fix organizationDiscoveryIntervalMillis bug (Ke Wang)
• c006c80 Fix GeometryToBingTiles for certain geometries (James A. Gill)
• 7e12d42 Add JSON annotations to SqlFunctionId (Leiqing Cai)
• fbdbeeb Add geospatial function line_interpolate_point (James Gill)
• f424e1b Fix slice reader reading empty strings column (Bhavani Hari)
• 5d8105c Fix slice reader check for presentStream (Bhavani Hari)
• b82b56b Remove TypeProvider from FilterStatsCalculator.filterStats (Sahar Massachi)
• 34c846e Ensure TableFinishOperator input is ordered (Andrii Rosa)
• 62bf698 Remove synchronized method in FixedSourcePartitionedScheduler (Shixuan Fan)
• c61d6bb Make updateTaskStatus not synchronized within SqlStageExecution (Andrii Rosa)
• be1db83 Fix recoverable grouped execution for task_writer_count=1 (Andrii Rosa)
• dd00be9 Improve TestHiveRecoverableGroupedExecution (Andrii Rosa)
• 91fb5c1 Fix NPE in SimpleNodeSelector (Andrii Rosa)
• 9fd0400 Add non-linear transformation for structural hash (Wenlei Xie)
• 2413924 Fix position count of prefilled Blocks (oerling@fb.com)
• 1373a9f Fix getBlockView for all-null SliceDictionarySelectiveReader (oerling@fb.com)
• 808043f Add cache for stripe metadata (Shixuan Fan)
• 177a680 Move cache related class into separate package (Shixuan Fan)
• e1f0dd6 Remove unused method in OrcTestingUtil (Shixuan Fan)
• f24a452 Fix an issue when fetching function implementation in FunctionManager (Leiqing Cai)
• dabeccc Support CREATE FUNCTION execution (Leiqing Cai)
• 866f665 Add support for refreshing functions cache in namespace manager (Leiqing Cai)
• 109ad5e Rename FunctionNamespaceManager#rollback to #abort (Leiqing Cai)
• 3b7f143 Add ParametersAreNonnullByDefault annotation to cache loading methods (Leiqing Cai)
• 02b8c0e Move SqlInvokedRegularFunctionTestUtils to presto-sql-functions (Leiqing Cai)
• e81b812 Move classes related to SQL-invoked functions into spi (Leiqing Cai)
• a13b4c5 Rename SqlInvokedRegularFunction#comment to description (Leiqing Cai)
• dd7f571 Require function namespace to be exactly 2 parts (Leiqing Cai)
• 4051fd7 Move spi.relation.FullyQualifiedName to spi.QualifiedFunctionName (Leiqing Cai)
• ab9d0ec Remove unused FeatureConfig from PlanOptimizers (James Sun)
• 694bd5d Support SQL function execution (Rongrong Zhong)
• d47c75d Add ScalarFunctionImplementation abstraction (Rongrong Zhong)
• 7294fa5 Add funcion implementation type and argument names to FunctionMetadata (Rongrong Zhong)
• 04ead21 Rename ScalarFunctionImplementation to BuiltInScalarFunctionImplementation (Rongrong Zhong)
• b064542 Add SqlFunctionProperties to bytecode compiler cache keys (Rongrong Zhong)
• a5701ca Remove Session from SqlToRowExpressionTranslator (Rongrong Zhong)
• a80dea9 Factor out Expression related session configs into SqlFunctionProperties (Rongrong Zhong)
• 297e2ba Fix error message in FunctionManager (Rongrong Zhong)
• c1a0c65 Check call convention once in RowExpressionInterpreter.visitCall (Rongrong Zhong)
• 096c1de Check call convention once in ExpressionInterpreter.visitFunctionCall (Rongrong Zhong)
• 929e8a9 Remove TypeProvider from LocalExecutionPlanContext (Rongrong Zhong)
• b54d248 Improve HashGenerator.getPartition performance (Ying Su)
• 898037b Fix hive columnIndex with default partition (Bhavani Hari)
• 052e0eb Use URI in raptor.properties (Jiexi Lin)
• 2be543e Optimize PlanFragment serialization (Nikhil Collooru)
• af3b9dd Revert " Include predicted size of built hash in HashBuilderOperator" (Rongrong Zhong)
• 2404248 Revert "Do not overcommit memory in HashBuilderOperator" (Rongrong Zhong)
• b0fed6e Make Polygon WKTs valid in test cases (James A. Gill)
• d28e5de Improve Geometry Serialization Tests (James Gill)
• de1dda7 Improve error message when transaction is gone (Andrii Rosa)
• a140fba Refactor Hive transaction management not to use metastore getter (Andrii Rosa)
• e6b7687 Do not update query failureCause when query is FINISHING (Andrii Rosa)
• 2b5711c Do not fail finishing query unless the transaction failed (Andrii Rosa)
• aa47ca3 Check if transaction still exist on transitioning to failed query (Andrii Rosa)
• d75bd6b Use fast varint read path in LongInputStreamV1 (oerling@fb.com)
• 5a143aa Export round trip count for update/info/status (Wenlei Xie)
• 78f2470 Adding presto-pinot with pushdown (Devesh Agrawal)
• 6a7c4c8 Add common InternalCommunicationConfig to turn on HTTPS (Yi He)
• 0be3619 Fix table scan results with disjoint pushdown filters (Bhavani Hari)
• 2b79a58 Refactor SqlStageExecution creation (Rebecca Schlussel)
• ad92616 Small cleanups of SqlQueryScheduler (Rebecca Schlussel)
• 314dd12 Add FileOpener interface in Hive connector (Shixuan Fan)
• 6098d19 Remove unused static variable in TestBackgroundHiveSplitLoader (Shixuan Fan)
• 2de30ec Fail query with large taskUpdate size (Nikhil Collooru)
• 88416e5 Add peak memory distribution among tasks of each stage (Vic Zhang)
• a97dd94 Rename StageCpuDistribution to ResourceDistribution (Vic Zhang)
• 6dbfefc Add peak total memory to TaskStats (Vic Zhang)
• 947b5b9 Add PartitionObjectBuilder interface for Hive (Ajay George)
• 9350602 Support max_tasks_per_stage for scan (Cem Cayiroglu)
• a71977e Stop Parquet Dictionary Reads Eagerly in Predicate Pushdown (James Petty)
• d257b0c Fix Parquet Predicate Pushdown on Dictionaries (James Petty)
• fcd6f04 Do not overcommit memory in HashBuilderOperator (Andrii Rosa)
• a38dbfa Remove DeleteHandle from DeleteNode (Rebecca Schlussel)
• b581ffb Use TableHandle directly in StatisticsWriterNode (Rebecca Schlussel)
• 928a234 Move BeginTableWrite logic to scheduler (Rebecca Schlussel)
• 7ebe522 Simplify code dealing with WriterTargets (Rebecca Schlussel)
• 5b8ec08 Add ConnectorId to CreateName (Rebecca Schlussel)
• 94c73cc Make WriterTargets optional in plan nodes (Rebecca Schlussel)
• efbd394 Replace DeleteHandle with TableHandle for metadata delete (Rebecca Schlussel)
• 48c0693 Remove unused field from LocalExecutionPlanner (Rebecca Schlussel)
• a9fa7b6 Remove test usage of TableScan json constructor (Rebecca Schlussel)
• 44bc897 Fix Flaky TestTransactionManager.testExpiration (Cem Cayiroglu)
• fbb7794 Support using HDFS as storage in Raptor (Jiexi Lin)
• 9fb46ce Revert reserved words introduced in CREATE FUNCTION syntax support (Leiqing Cai)
• 0e7847b Fix SliceReader ArrayOutOfBounds when presentStream is null (Bhavani Hari)
• a6fd61b Move RowExpression translation above AddExchange (James Sun)
• 55c3f7c Rename SymbolResolver to VariableResolver (James Sun)
• 332d484 Move SetOperationNode to SPI (James Sun)
• 7491f20 Add session property for nested subfield filters (Bhavani Hari)
• f62dfdd Hanlde isNull filter for Array type (Bhavani Hari)
• 9092cb7 Add example for Jdbc Connector Row Expression Translation (Saksham Sachdev)
• e3694a0 Fix compareRanges logic (Bhavani Hari)
• 49df615 Move TranslateExpressions below AddExchange (Yi He)
• 4f74189 Move TranslateExpressions above last predicatePushdown (Yi He)
• 4343574 Include error message in SimpleHttpResonseHandler for SMILE (Nikhil Collooru)
• a51c1e3 Generate unique index for PartitionColumns (Bhavani Hari)
• 90cfbd2 Passing X_Forwarded_For from proxy (Swapnil Tailor)
• 8d5d5e6 Update to Facebook airlift (Wenlei Xie)
• 184fb50 Add ignored dependencies in presto-hive (Wenlei Xie)
• cff2383 Add more dependencies for HiveGcsModule (Wenlei Xie)
• 442a46f Fix undeclared dependency used for HTTP constants (David Phillips)
• 42e7b22 Use ValidationAssertions in TestPowerOfTwoValidator (David Phillips)
• 7e2f60d Reset DictionaryBlock values after getBlock (Bhavani Hari)
• 270a40f Fix selective readers output positions when reading null streams (Bhavani Hari)
• c1b2b2e Add cache for orc file tail information (Shixuan Fan)
• 418d63b Add RowExpressionPredicateExtractor (Yi He)
• 9a6d4de Intersect pushdown predicates (oerling@fb.com)
• 3c8584f prepare for next development iteration - 0.229-SNAPSHOT (Rongrong Zhong)

[jagill]

For commit c006c80 (it didn't seem to auto-extract :( )

General Changes

• Fix GeometryToBingTiles fails on certain degenerate geometries #13639: geometry_to_bing_tile fails on certain degenerate geometries.

[sachdevs]

For #13526: There is no release note needed, it does not add any new functionality at all. Will include NO RELEASE NOTE next time.

[caithagoras]

@rongrong I should probably make the auto-generated PR title to have [WIP] because it is not yet ready for review. I was waiting for the missing release notes before working on formatting it.

[caithagoras]

@sachdevs The correct format is shown as below. Thanks!

== NO RELEASE NOTE ==

[caithagoras]

@jagill Is this updated release note accurate? Does the "fail" you mentioned means "query failure"?

[rongrong]

@rongrong I should probably make the auto-generated PR title to have [WIP] because it is not yet ready for review. I was waiting for the missing release notes before working on formatting it.

That's fine. Just want to speed up the process since we are on a tight schedule. The review is free anyways. 😛

[caithagoras]

@rongrong Ready for final review.

[rongrong]

Add configuration property ...

[caithagoras]

The configuration property already exists and is being used. We're respecting its value in a different circumstance.

[rongrong]

Maybe Add support to respect xxx then?

[caithagoras]

Changed to

Add support to respect configuration property
``stage.max-tasks-per-stage`` for limiting the number of tasks per scan.

[caithagoras]

The CLA check seems to have passed.

[rongrong]

LGTM. @rschlussel do you want to take a look? I don't trust my English as much. 😂