Skip to content

Refactor: extract FooterTail from ParquetMetadataReader #8437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 6 commits into from
Sep 25, 2025
Merged

Refactor: extract FooterTail from ParquetMetadataReader #8437

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
17f9972
Refactor: extract FooterTail from ParquetMetadataReader
alamb Sep 24, 2025
f1e37bf
Update parquet/src/file/metadata/footer_tail.rs
alamb Sep 24, 2025
b5a2739
Update parquet/src/file/metadata/footer_tail.rs
alamb Sep 25, 2025
eab5073
Update parquet/src/file/metadata/footer_tail.rs
alamb Sep 25, 2025
3f94668
check metadata len
alamb Sep 25, 2025
729b4a2
Merge remote-tracking branch 'apache/main' into alamb/extract_footer_…
alamb Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions parquet/src/file/metadata/footer_tail.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

use crate::errors::{ParquetError, Result};
use crate::file::{FOOTER_SIZE, PARQUET_MAGIC, PARQUET_MAGIC_ENCR_FOOTER};

/// Parsed Parquet footer tail (last 8 bytes of a Parquet file)
///
/// There are 8 bytes at the end of the Parquet footer with the following layout:
/// * 4 bytes for the metadata length
/// * 4 bytes for the magic bytes 'PAR1' or 'PARE' (encrypted footer)
///
/// ```text
/// +-----+------------------+
/// | len | 'PAR1' or 'PARE' |
/// +-----+------------------+
/// ```
///
/// # Examples
/// ```
/// # use parquet::file::metadata::FooterTail;
/// // a non encrypted footer with 28 bytes of metadata
/// let last_8_bytes: [u8; 8] = [0x1C, 0x00, 0x00, 0x00, b'P', b'A', b'R', b'1'];
/// let footer_tail = FooterTail::try_from(last_8_bytes).unwrap();
/// assert_eq!(footer_tail.metadata_length(), 28);
/// assert_eq!(footer_tail.is_encrypted_footer(), false);
/// ```
///
/// ```
/// # use parquet::file::metadata::FooterTail;
/// // an encrypted footer with 512 bytes of metadata
/// let last_8_bytes = vec![0x00, 0x02, 0x00, 0x00, b'P', b'A', b'R', b'E'];
/// let footer_tail = FooterTail::try_from(&last_8_bytes[..]).unwrap();
/// assert_eq!(footer_tail.metadata_length(), 512);
/// assert_eq!(footer_tail.is_encrypted_footer(), true);
/// ```
///
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FooterTail {
metadata_length: usize,
encrypted_footer: bool,
}

impl FooterTail {
/// Try to decode the footer tail from the given 8 bytes
pub fn try_new(slice: &[u8; FOOTER_SIZE]) -> Result<FooterTail> {
let magic = &slice[4..];
let encrypted_footer = if magic == PARQUET_MAGIC_ENCR_FOOTER {
true
} else if magic == PARQUET_MAGIC {
false
} else {
return Err(general_err!("Invalid Parquet file. Corrupt footer"));
};
// get the metadata length from the footer
let metadata_len = u32::from_le_bytes(slice[..4].try_into().unwrap());

Ok(FooterTail {
// u32 won't be larger than usize in most cases
metadata_length: metadata_len.try_into()?,
encrypted_footer,
})
}

/// The length of the footer metadata in bytes
pub fn metadata_length(&self) -> usize {
self.metadata_length
}

/// Whether the footer metadata is encrypted
pub fn is_encrypted_footer(&self) -> bool {
self.encrypted_footer
}
}

impl TryFrom<[u8; FOOTER_SIZE]> for FooterTail {
type Error = ParquetError;

fn try_from(value: [u8; FOOTER_SIZE]) -> Result<Self> {
Self::try_new(&value)
}
}

impl TryFrom<&[u8]> for FooterTail {
type Error = ParquetError;

fn try_from(value: &[u8]) -> Result<Self> {
if value.len() != FOOTER_SIZE {
return Err(general_err!(
"Invalid footer length {}, expected {FOOTER_SIZE}",
value.len()
));
}
let slice: &[u8; FOOTER_SIZE] = value.try_into().unwrap();
Self::try_new(slice)
}
}
4 changes: 3 additions & 1 deletion parquet/src/file/metadata/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@
//!
//! * Same name, different struct
//! ```
mod footer_tail;
mod memory;
mod parser;
mod push_decoder;
Expand Down Expand Up @@ -121,8 +122,9 @@ use crate::schema::types::{
};
#[cfg(feature = "encryption")]
use crate::thrift::{TCompactSliceInputProtocol, TSerializable};
pub use footer_tail::FooterTail;
Copy link
Contributor Author

@alamb alamb Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the maintains the same public API

pub use push_decoder::ParquetMetaDataPushDecoder;
pub use reader::{FooterTail, PageIndexPolicy, ParquetMetaDataReader};
pub use reader::{PageIndexPolicy, ParquetMetaDataReader};
use std::ops::Range;
use std::sync::Arc;
pub use writer::ParquetMetaDataWriter;
Expand Down
54 changes: 5 additions & 49 deletions parquet/src/file/metadata/reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ use std::{io::Read, ops::Range};
#[cfg(feature = "encryption")]
use crate::encryption::decrypt::FileDecryptionProperties;
use crate::errors::{ParquetError, Result};
use crate::file::metadata::ParquetMetaData;
use crate::file::metadata::{FooterTail, ParquetMetaData};
use crate::file::page_index::index_reader::acc_range;
use crate::file::reader::ChunkReader;
use crate::file::{FOOTER_SIZE, PARQUET_MAGIC, PARQUET_MAGIC_ENCR_FOOTER};
use crate::file::FOOTER_SIZE;

#[cfg(all(feature = "async", feature = "arrow"))]
use crate::arrow::async_reader::{MetadataFetch, MetadataSuffixFetch};
Expand Down Expand Up @@ -100,26 +100,6 @@ impl From<bool> for PageIndexPolicy {
}
}

/// Describes how the footer metadata is stored
///
/// This is parsed from the last 8 bytes of the Parquet file
pub struct FooterTail {
metadata_length: usize,
encrypted_footer: bool,
}

impl FooterTail {
/// The length of the footer metadata in bytes
pub fn metadata_length(&self) -> usize {
self.metadata_length
}

/// Whether the footer metadata is encrypted
pub fn is_encrypted_footer(&self) -> bool {
self.encrypted_footer
}
}

impl ParquetMetaDataReader {
/// Create a new [`ParquetMetaDataReader`]
pub fn new() -> Self {
Expand Down Expand Up @@ -720,39 +700,15 @@ impl ParquetMetaDataReader {
}
}

/// Decodes the end of the Parquet footer
///
/// There are 8 bytes at the end of the Parquet footer with the following layout:
/// * 4 bytes for the metadata length
/// * 4 bytes for the magic bytes 'PAR1' or 'PARE' (encrypted footer)
///
/// ```text
/// +-----+------------------+
/// | len | 'PAR1' or 'PARE' |
/// +-----+------------------+
/// ```
/// Decodes a [`FooterTail`] from the provided 8-byte slice.
pub fn decode_footer_tail(slice: &[u8; FOOTER_SIZE]) -> Result<FooterTail> {
let magic = &slice[4..];
Copy link
Contributor Author

@alamb alamb Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code to parse the footer was previously a member function of ParquetMetadataReader, but I need to use it in the ParquetMetadataPushDecoder, so I pulled it into its own function

let encrypted_footer = if magic == PARQUET_MAGIC_ENCR_FOOTER {
true
} else if magic == PARQUET_MAGIC {
false
} else {
return Err(general_err!("Invalid Parquet file. Corrupt footer"));
};
// get the metadata length from the footer
let metadata_len = u32::from_le_bytes(slice[..4].try_into().unwrap());
Ok(FooterTail {
// u32 won't be larger than usize in most cases
metadata_length: metadata_len as usize,
encrypted_footer,
})
FooterTail::try_new(slice)
}

/// Decodes the Parquet footer, returning the metadata length in bytes
#[deprecated(since = "54.3.0", note = "Use decode_footer_tail instead")]
pub fn decode_footer(slice: &[u8; FOOTER_SIZE]) -> Result<usize> {
Self::decode_footer_tail(slice).map(|f| f.metadata_length)
Self::decode_footer_tail(slice).map(|f| f.metadata_length())
}

/// Decodes [`ParquetMetaData`] from the provided bytes.
Expand Down
Loading