Skip to content

Add release notes for 0.271 #17352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cocozianu wants to merge 24 commits into from
Closed

Add release notes for 0.271 #17352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
953f44d
Prepare for next development iteration - 0.272-SNAPSHOT
Feb 18, 2022
4d8aad1
Fix parquet data page offset calculation in parquet writer
v-jizhang Feb 14, 2022
6b5da9a
Support json type in PinotSegmentPageSource
xiangfu0 Feb 18, 2022
c07106b
ThreadResource supports JSON serde at present.
alanxu Feb 16, 2022
c911689
Add unit test for ThreadResource
alanxu Feb 18, 2022
3cb1ee2
Skip reading Parquet pages using Column Indexes feature
shangxinli Jan 5, 2021
fe1e84d
Fixing Distributed Query Runner
swapsmagic Feb 17, 2022
59d092c
Support iceberg connector concurrent insertion
liupan664021 Nov 12, 2021
afd4d2f
fixing presto pinot doc for pinot catalog configs and add session pro…
xiangfu0 Feb 22, 2022
39d20a8
Add ability to disable splitting file in hive connector
kewang1024 Feb 5, 2022
f66c85b
Add ability to do streaming aggregation for hive table scans
kewang1024 Feb 8, 2022
2fb2c31
Add ability to do streaming for partial aggregation
kewang1024 Feb 9, 2022
5edaef8
check field runtimeStats to prevent accessing undefined properties
ahouzheng Feb 22, 2022
ca90676
Improve memory accounting for broadcast table
pgupta2 Feb 19, 2022
e3c9af6
Enable cost based optimization by default
rongrong Feb 11, 2022
b1402dc
Create an abstract class for metadata of multiple iceberg catalogs
Feb 10, 2022
39b6cea
Remove the junit dependency to fix the build
beinan Feb 22, 2022
7b83d6a
Skip synthetic members to enable code coverage
g1y Feb 9, 2022
aab1acf
Add default implementation to HiveMetastore to fix the build
beinan Feb 23, 2022
3509718
Support optional row numbers in OrcSelectiveReader
harsharastogi Feb 2, 2022
d8323f5
Preserve quotedness for Identifier when formatting Query
v-jizhang Jan 13, 2022
d370d62
Tighten allowed characters for non-delimited identifiers
v-jizhang Jan 17, 2022
79370de
Add a new JDBC driver parameter timeZoneId
v-jizhang Sep 17, 2021
71fd1c6
Add release notes for 0.271
cocozianu Feb 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
<packaging>pom</packaging>

<name>presto-root</name>
Expand Down
2 changes: 1 addition & 1 deletion presto-accumulo/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-accumulo</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-atop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-atop</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-base-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-base-jdbc</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark-driver/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark-driver</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark-runner/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark-runner</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchto-benchmarks/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-benchto-benchmarks</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-bigquery/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-bigquery</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-blackhole/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-blackhole</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-bytecode/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-bytecode</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cache/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-cache</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cassandra/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-cassandra</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cli/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-cli</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-client</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cluster-ttl-providers/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion presto-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-common</artifactId>
Expand Down
3 changes: 2 additions & 1 deletion presto-common/src/main/java/com/facebook/presto/common/block/BlockBuilderStatus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ private static int deepInstanceSize(Class<?> clazz)

int size = ClassLayout.parseClass(clazz).instanceSize();
for (Field field : clazz.getDeclaredFields()) {
if (!field.getType().isPrimitive()) {
// The !field.isSynthetic() condition is specifically for jacoco which adds synthetic members $jacocoData and $jacocoInit() to classes.
if (!field.getType().isPrimitive() && !field.isSynthetic()) {
size += deepInstanceSize(field.getType());
}
}
Expand Down
2 changes: 1 addition & 1 deletion presto-delta/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-delta</artifactId>
Expand Down
19 changes: 15 additions & 4 deletions presto-delta/src/main/java/com/facebook/presto/delta/DeltaPageSourceProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import com.facebook.presto.parquet.RichColumnDescriptor;
import com.facebook.presto.parquet.cache.MetadataReader;
import com.facebook.presto.parquet.predicate.Predicate;
import com.facebook.presto.parquet.reader.ColumnIndexFilterUtils;
import com.facebook.presto.parquet.reader.ParquetReader;
import com.facebook.presto.spi.ColumnHandle;
import com.facebook.presto.spi.ConnectorPageSource;
Expand All @@ -55,6 +56,7 @@
import org.apache.parquet.hadoop.metadata.BlockMetaData;
import org.apache.parquet.hadoop.metadata.FileMetaData;
import org.apache.parquet.hadoop.metadata.ParquetMetadata;
import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore;
import org.apache.parquet.io.ColumnIO;
import org.apache.parquet.io.MessageColumnIO;
import org.apache.parquet.schema.GroupType;
Expand All @@ -64,6 +66,7 @@

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand Down Expand Up @@ -164,7 +167,8 @@ public ConnectorPageSource createPageSource(
isParquetBatchReaderVerificationEnabled(session),
typeManager,
deltaTableLayoutHandle.getPredicate(),
fileFormatDataSourceStats);
fileFormatDataSourceStats,
false);

return new DeltaPageSource(
deltaColumnHandles,
Expand Down Expand Up @@ -207,7 +211,8 @@ private static ConnectorPageSource createParquetPageSource(
boolean verificationEnabled,
TypeManager typeManager,
TupleDomain<DeltaColumnHandle> effectivePredicate,
FileFormatDataSourceStats stats)
FileFormatDataSourceStats stats,
boolean columnIndexFilterEnabled)
{
AggregatedMemoryContext systemMemoryContext = newSimpleAggregatedMemoryContext();

Expand Down Expand Up @@ -243,9 +248,12 @@ private static ConnectorPageSource createParquetPageSource(
Predicate parquetPredicate = buildPredicate(requestedSchema, parquetTupleDomain, descriptorsByPath);
final ParquetDataSource finalDataSource = dataSource;
ImmutableList.Builder<BlockMetaData> blocks = ImmutableList.builder();
List<ColumnIndexStore> blockIndexStores = new ArrayList<>();
for (BlockMetaData block : footerBlocks.build()) {
if (predicateMatches(parquetPredicate, block, finalDataSource, descriptorsByPath, parquetTupleDomain)) {
Optional<ColumnIndexStore> columnIndexStore = ColumnIndexFilterUtils.getColumnIndexStore(parquetPredicate, finalDataSource, block, descriptorsByPath, columnIndexFilterEnabled);
if (predicateMatches(parquetPredicate, block, finalDataSource, descriptorsByPath, parquetTupleDomain, columnIndexStore, columnIndexFilterEnabled)) {
blocks.add(block);
blockIndexStores.add(columnIndexStore.orElse(null));
}
}
MessageColumnIO messageColumnIO = getColumnIO(fileSchema, requestedSchema);
Expand All @@ -256,7 +264,10 @@ private static ConnectorPageSource createParquetPageSource(
systemMemoryContext,
maxReadBlockSize,
batchReaderEnabled,
verificationEnabled);
verificationEnabled,
parquetPredicate,
blockIndexStores,
columnIndexFilterEnabled);

ImmutableList.Builder<String> namesBuilder = ImmutableList.builder();
ImmutableList.Builder<Type> typesBuilder = ImmutableList.builder();
Expand Down
2 changes: 1 addition & 1 deletion presto-docs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.271-SNAPSHOT</version>
<version>0.272-SNAPSHOT</version>
</parent>

<artifactId>presto-docs</artifactId>
Expand Down
Loading