Skip to content

PARQUET-1310: Column indexes: Filtering #509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gszadovszky merged 10 commits into from
Aug 17, 2018
Merged

PARQUET-1310: Column indexes: Filtering #509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
465cca6
PARQUET-1310: New options for column index based filtering
gszadovszky Jun 5, 2018
c00b2c0
PARQUET-1310: Prepare for skipping pages at reading
gszadovszky Jun 21, 2018
3c9fe3e
PARQUET-1310: Simple implementation of page filtering
gszadovszky Jul 3, 2018
c471c72
PARQUET-1310: Glue the whole stuff together; implement proper tests
gszadovszky Jul 26, 2018
c0ce0f3
PARQUET-1310: Modifications according to zi's comments
gszadovszky Aug 8, 2018
7d5faea
PARQUET-1310: Implement binary search for ASCENDING/DESCENDING orders
gszadovszky Aug 15, 2018
f9b6ecc
PARQUET-1310: Modifications according to zi's comments
gszadovszky Aug 16, 2018
e82b9d5
PARQUET-1310: Update terminology of column index based filtering
gszadovszky Aug 16, 2018
325e5f6
PARQUET-1310: Fix exception naming
gszadovszky Aug 17, 2018
659c1a0
PARQUET-1310: Fix issue introduced at exception renaming
gszadovszky Aug 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions parquet-column/src/main/java/org/apache/parquet/column/ColumnReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,10 @@ public interface ColumnReader {

/**
* @return the totalCount of values to be consumed
* @deprecated will be removed in 2.0.0; Total values might not be able to be counted before reading the values (e.g.
* in case of column index based filtering)
*/
@Deprecated
long getTotalValueCount();

/**
Expand Down
11 changes: 6 additions & 5 deletions parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnReadStoreImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,13 @@ public ColumnReadStoreImpl(PageReadStore pageReadStore,

@Override
public ColumnReader getColumnReader(ColumnDescriptor path) {
return newMemColumnReader(path, pageReadStore.getPageReader(path));
}

private ColumnReaderImpl newMemColumnReader(ColumnDescriptor path, PageReader pageReader) {
PrimitiveConverter converter = getPrimitiveConverter(path);
return new ColumnReaderImpl(path, pageReader, converter, writerVersion);
PageReader pageReader = pageReadStore.getPageReader(path);
if (pageReadStore.isInPageFilteringMode()) {
return new SynchronizingColumnReader(path, pageReader, converter, writerVersion, pageReadStore.getRowIndexes());
} else {
return new ColumnReaderImpl(path, pageReader, converter, writerVersion);
}
}

private PrimitiveConverter getPrimitiveConverter(ColumnDescriptor path) {
Expand Down
Loading