PARQUET-41: Add bloom filter for parquet #62
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -511,6 +511,33 @@ struct ColumnMetaData { | |
| * This information can be used to determine if all data pages are | ||
| * dictionary encoded for example **/ | ||
| 13: optional list<PageEncodingStats> encoding_stats; | ||
|
|
||
| /** Byte offset from beginning of file to bloom filter data**/ | ||
| 14: optional i64 bloom_filter_offset; | ||
| } | ||
|
|
||
| enum BloomFilterAlgorithm { | ||
|
||
| /** Block based bloom filter. **/ | ||
| BLOCK = 0; | ||
| } | ||
|
|
||
| enum BloomFilterHash { | ||
|
||
| /** The hash function used to compute hash of column value, | ||
| * murmur3 has 32 bits and 128 bits hash, we use lower 64 bits from | ||
| * murmur3 128 bits function murmur3hash_x64_128 | ||
| **/ | ||
| MURMUR3 = 0; | ||
|
||
| } | ||
|
|
||
| struct BloomFilterHeader { | ||
|
Contributor
There was a problem hiding this comment. Where is this header written? I'm assuming that when you seek to a bloom filter's offset, you should be able to read the header? Then the filter bytes follow the header? I think this should be more clear about where structures are placed in the file. |
||
| /** The size of bitset in bytes, must be a power of 2 and larger than 32**/ | ||
| 1: required i32 numBytes; | ||
|
|
||
| /** The algorithm for setting bits. **/ | ||
| 2: required BloomFilterAlgorithm bloomFilterAlgorithm; | ||
|
||
|
|
||
| /** The hash function used for bloom filter. **/ | ||
| 3: required BloomFilterHash bloomFilterHash; | ||
| } | ||
|
|
||
| struct ColumnChunk { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should specify where the bloom filter data should be located for each column. I think the conclusion we came to was to put bloom filters for all columns together before the start of the row group the filters describe. That way we can avoid a seek if a bloom filter and row group both need to be read.