Skip to content

PARQUET-41: add bloom filter #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chenjunjiedada wants to merge 4 commits into from
Closed

PARQUET-41: add bloom filter #99

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
691cb88
PARQUET-41: add bloom filter
Jun 3, 2018
cc8985b
PARQUET-41: update comments
Jul 9, 2018
4948425
PARQUET-41: Add Bloom filter to PageType and PageHeader
Jul 12, 2018
71d5b93
PARQUET-41: fix indentation
Jul 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions src/main/thrift/parquet.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,47 @@ struct ColumnMetaData {
* This information can be used to determine if all data pages are
* dictionary encoded for example **/
13: optional list<PageEncodingStats> encoding_stats;

/** Byte offset from beginning of file to Bloom filter data. **/
14: optional i64 bloom_filter_offset;
}

/** Block-based algorithm type annotation. **/
struct SplitBlockAlgorithm {}

/** The algorithm used in Bloom filter. **/
union BloomFilterAlgorithm {
/** Block-based Bloom filter. **/
1: SplitBlockAlgorithm BLOCK;
}

/** Hash strategy type annotation. It uses Murmur3Hash_x64_128 from the original SMHasher
* repo by Austin Appleby.
**/
struct Murmur3 {}

/**
* The hash function used in Bloom filter. This function takes the hash of a column value
* using plain encoding.
**/
union BloomFilterHash {
/** Murmur3 Hash Strategy. **/
1: Murmur3 MURMUR3;
}

/**
* Bloom filter header is stored at beginning of Bloom filter data of each column
* and followed by its bitset.
**/
struct BloomFilterHeader {

@jbapple-cloudera jbapple-cloudera Jun 25, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this also need a mention in PageType and PageHeader?

@jbapple-cloudera jbapple-cloudera Jul 9, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any thoughts on this?

@chenjunjiedada chenjunjiedada Jul 9, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to remind again.

It depends on how we treat Bloom filter data, it is natural if we abstract Bloom filter data as a specific page. Previously, I didn't read or write it as page since the PageReader/PageWriter APIs in parquet-mr looks specific to data and dictionary page. While I think it is easy to integrate if we want to do this.

@chenjunjiedada chenjunjiedada Jul 12, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

/** The size of bitset in bytes **/
1: required i32 numBytes;

/** The algorithm for setting bits. **/
2: required BloomFilterAlgorithm algorithm;

/** The hash function used for Bloom filter. **/
3: required BloomFilterHash hash;
}

struct ColumnChunk {
Expand Down