Skip to content

ARROW-18437: [C++][Parquet] Fix encoder for DELTA_BINARY_PACKED when flushing more than once #14959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rok merged 6 commits into from
Dec 22, 2022
Merged

ARROW-18437: [C++][Parquet] Fix encoder for DELTA_BINARY_PACKED when flushing more than once #14959

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
68eadec
[Init] Fixing DeltaBitPack bug
mapleFU Dec 15, 2022
836db71
Fixing flushing when size == 0
mapleFU Dec 15, 2022
5bf770e
Merge branch 'master' into fixing-delta-bit-pack
mapleFU Dec 17, 2022
c894924
[Update] Enlarge ROUND_TRIP_TIME
mapleFU Dec 17, 2022
0736419
Merge branch 'master' into fixing-delta-bit-pack
mapleFU Dec 19, 2022
aca32f1
address comment: rename variable and make code-comment concrete
mapleFU Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions cpp/src/parquet/encoding.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2138,8 +2138,6 @@ class DeltaBitPackEncoder : public EncoderImpl, virtual public TypedEncoder<DTyp
"but it's " +
std::to_string(values_per_block % mini_blocks_per_block));
}
// Reserve enough space at the beginning of the buffer for largest possible header.
PARQUET_THROW_NOT_OK(sink_.Advance(kMaxPageHeaderWriterSize));
Comment thread
mapleFU marked this conversation as resolved.
}

std::shared_ptr<Buffer> FlushValues() override;
Expand Down Expand Up @@ -2177,6 +2175,8 @@ void DeltaBitPackEncoder<DType>::Put(const T* src, int num_values) {

int idx = 0;
if (total_value_count_ == 0) {
// Reserve enough space at the beginning of the buffer for largest possible header.
PARQUET_THROW_NOT_OK(sink_.Advance(kMaxPageHeaderWriterSize));
current_value_ = src[0];
first_value_ = current_value_;
idx = 1;
Expand All @@ -2186,7 +2186,7 @@ void DeltaBitPackEncoder<DType>::Put(const T* src, int num_values) {
while (idx < num_values) {
UT value = static_cast<UT>(src[idx]);
// Calculate deltas. The possible overflow is handled by use of unsigned integers
// making subtraction operations well defined and correct even in case of overflow.
// making subtraction operations well-defined and correct even in case of overflow.
// Encoded integers will wrap back around on decoding.
// See http://en.wikipedia.org/wiki/Modular_arithmetic#Integers_modulo_n
deltas_[values_current_block_] = value - current_value_;
Expand Down Expand Up @@ -2274,6 +2274,8 @@ std::shared_ptr<Buffer> DeltaBitPackEncoder<DType>::FlushValues() {
throw ParquetException("header writing error");
}
header_writer.Flush();
// reset counter
total_value_count_ = 0;

// We reserved enough space at the beginning of the buffer for largest possible header
// and data was written immediately after. We now write the header data immediately
Expand Down