Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions shredded_variant/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

# Variant Shredding

This directory contains binary files used to verify shredded variant readers.

## Contents

`cases.json` - a JSON list of test cases. Each case is an error case, a single record variant case, or a multi-record variant case.

Each JSON object in the list represents a single case and includes:
* `case_number` - a number to identify the case and its data files
* `test` - name of the test from which the case was generated. Multiple cases can be generated from a single test. For instance, `testShreddedVariantPrimitives` is used to generate a case for each variant primitive.

Binary files for each case are named using the case number. Variant binary files are also named using the row number.

Error cases have the following fields:
* `error_message` - a message describing why the case is an error

Single record cases have the following fields:
* `parquet_file` - path of the Parquet file to be read for the case
* `variant_file` - path of the binary variant file to be read for the case
* `variant` - string representation of the variant for the case

Multi-record cases have the following fields:
* `parquet_file` - path of the Parquet file to be read for the case, containing multiple records
* `variant_files` - path of each binary variant file, one for each record in the Parquet file (may be null for a null variant)
* `variants` - string representation of the variants for the case

## Variant file encoding

Each `*.variant.bin` file contains a single variant serialized by concatenating the serialized bytes of the variant metadata followed by the serialized bytes of the variant value.

## Parquet file encoding

Each Parquet file contains one or more rows. Each row corresponds to a variant file (by ID) for the test case and consists of an `id` field and a `var` field.

## Source

For more information, see the [original test cases](https://github.com/apache/iceberg/blob/main/parquet/src/test/java/org/apache/iceberg/parquet/TestVariantReaders.java).
Binary file added shredded_variant/case-001.parquet
Binary file not shown.
Binary file added shredded_variant/case-001_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-002.parquet
Binary file not shown.
Binary file added shredded_variant/case-002_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-004.parquet
Binary file not shown.
Binary file added shredded_variant/case-004_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-005.parquet
Binary file not shown.
Binary file added shredded_variant/case-005_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-006.parquet
Binary file not shown.
Binary file added shredded_variant/case-006_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-007.parquet
Binary file not shown.
Binary file added shredded_variant/case-007_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-008.parquet
Binary file not shown.
Binary file added shredded_variant/case-008_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-009.parquet
Binary file not shown.
Binary file added shredded_variant/case-009_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-010.parquet
Binary file not shown.
Binary file added shredded_variant/case-010_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-011.parquet
Binary file not shown.
Binary file added shredded_variant/case-011_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-012.parquet
Binary file not shown.
Binary file added shredded_variant/case-012_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-013.parquet
Binary file not shown.
Binary file added shredded_variant/case-013_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-014.parquet
Binary file not shown.
Binary file added shredded_variant/case-014_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-015.parquet
Binary file not shown.
Binary file added shredded_variant/case-015_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-016.parquet
Binary file not shown.
Binary file added shredded_variant/case-016_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-017.parquet
Binary file not shown.
Binary file added shredded_variant/case-017_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-018.parquet
Binary file not shown.
Binary file added shredded_variant/case-018_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-019.parquet
Binary file not shown.
Binary file added shredded_variant/case-019_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-020.parquet
Binary file not shown.
Binary file added shredded_variant/case-020_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-021.parquet
Binary file not shown.
Binary file added shredded_variant/case-021_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-022.parquet
Binary file not shown.
Binary file added shredded_variant/case-022_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-023.parquet
Binary file not shown.
Binary file added shredded_variant/case-023_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-024.parquet
Binary file not shown.
Binary file added shredded_variant/case-024_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-025.parquet
Binary file not shown.
Binary file added shredded_variant/case-025_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-026.parquet
Binary file not shown.
Binary file added shredded_variant/case-026_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-027.parquet
Binary file not shown.
Binary file added shredded_variant/case-027_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-028.parquet
Binary file not shown.
Binary file added shredded_variant/case-028_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-029.parquet
Binary file not shown.
Binary file added shredded_variant/case-029_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-030.parquet
Binary file not shown.
Binary file added shredded_variant/case-030_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-031.parquet
Binary file not shown.
Binary file added shredded_variant/case-031_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-032.parquet
Binary file not shown.
Binary file added shredded_variant/case-032_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-033.parquet
Binary file not shown.
Binary file added shredded_variant/case-033_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-034.parquet
Binary file not shown.
Binary file added shredded_variant/case-034_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-035.parquet
Binary file not shown.
Binary file added shredded_variant/case-035_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-036.parquet
Binary file not shown.
Binary file added shredded_variant/case-036_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-037.parquet
Binary file not shown.
Binary file added shredded_variant/case-037_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-038.parquet
Binary file not shown.
Binary file added shredded_variant/case-038_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-039.parquet
Binary file not shown.
Binary file added shredded_variant/case-039_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-040.parquet
Binary file not shown.
Binary file added shredded_variant/case-041.parquet
Binary file not shown.
Binary file added shredded_variant/case-041_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-042.parquet
Binary file not shown.
Binary file added shredded_variant/case-043-INVALID.parquet
Binary file not shown.
Binary file added shredded_variant/case-043-INVALID_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-044.parquet
Binary file not shown.
Binary file added shredded_variant/case-044_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-045.parquet
Binary file not shown.
Binary file added shredded_variant/case-045_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-045_row-1.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-045_row-2.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-045_row-3.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-046.parquet
Binary file not shown.
Binary file added shredded_variant/case-046_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-047.parquet
Binary file not shown.
Binary file added shredded_variant/case-047_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-048.parquet
Binary file not shown.
Binary file added shredded_variant/case-048_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-049.parquet
Binary file not shown.
Binary file added shredded_variant/case-049_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-050.parquet
Binary file not shown.
Binary file added shredded_variant/case-050_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-051.parquet
Binary file not shown.
Binary file added shredded_variant/case-051_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-052.parquet
Binary file not shown.
Binary file added shredded_variant/case-052_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-053.parquet
Binary file not shown.
Binary file added shredded_variant/case-053_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-054.parquet
Binary file not shown.
Binary file added shredded_variant/case-054_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-055.parquet
Binary file not shown.
Binary file added shredded_variant/case-055_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-056.parquet
Binary file not shown.
Binary file added shredded_variant/case-056_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-057.parquet
Binary file not shown.
Binary file added shredded_variant/case-057_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-058.parquet
Binary file not shown.
Binary file added shredded_variant/case-058_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-059.parquet
Binary file not shown.
Binary file added shredded_variant/case-059_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-060.parquet
Binary file not shown.
Binary file added shredded_variant/case-060_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-061.parquet
Binary file not shown.
Binary file added shredded_variant/case-061_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-062.parquet
Binary file not shown.
Binary file added shredded_variant/case-062_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-063.parquet
Binary file not shown.
Binary file added shredded_variant/case-063_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-064.parquet
Binary file not shown.
Binary file added shredded_variant/case-064_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-065.parquet
Binary file not shown.
Binary file added shredded_variant/case-065_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-066.parquet
Binary file not shown.
Binary file added shredded_variant/case-066_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-067.parquet
Binary file not shown.
Binary file added shredded_variant/case-067_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-068.parquet
Binary file not shown.
Binary file added shredded_variant/case-068_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-069.parquet
Binary file not shown.
Binary file added shredded_variant/case-069_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-070.parquet
Binary file not shown.
Binary file added shredded_variant/case-070_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-071.parquet
Binary file not shown.
Binary file added shredded_variant/case-071_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-072.parquet
Binary file not shown.
Binary file added shredded_variant/case-072_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-073.parquet
Binary file not shown.
Binary file added shredded_variant/case-073_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-074.parquet
Binary file not shown.
Binary file added shredded_variant/case-074_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-075.parquet
Binary file not shown.
Binary file added shredded_variant/case-075_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-076.parquet
Binary file not shown.
Binary file added shredded_variant/case-076_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-077.parquet
Binary file not shown.
Binary file added shredded_variant/case-077_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-078.parquet
Binary file not shown.
Binary file added shredded_variant/case-078_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-079.parquet
Binary file not shown.
Binary file added shredded_variant/case-079_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-080.parquet
Binary file not shown.
Binary file added shredded_variant/case-080_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-081.parquet
Binary file not shown.
Binary file added shredded_variant/case-081_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-082.parquet
Binary file not shown.
Binary file added shredded_variant/case-082_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-083.parquet
Binary file not shown.
Binary file added shredded_variant/case-083_row-1.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-083_row-2.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-083_row-3.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-084-INVALID.parquet
Binary file not shown.
Binary file added shredded_variant/case-084-INVALID_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-085.parquet
Binary file not shown.
Binary file added shredded_variant/case-085_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-086.parquet
Binary file not shown.
Binary file added shredded_variant/case-086_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-087.parquet
Binary file not shown.
Binary file added shredded_variant/case-088.parquet
Binary file not shown.
Binary file added shredded_variant/case-088_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-089.parquet
Binary file not shown.
Binary file added shredded_variant/case-089_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-090.parquet
Binary file not shown.
Binary file added shredded_variant/case-090_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-091.parquet
Binary file not shown.
Binary file added shredded_variant/case-091_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-092.parquet
Binary file not shown.
Binary file added shredded_variant/case-092_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-093.parquet
Binary file not shown.
Binary file added shredded_variant/case-093_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-094.parquet
Binary file not shown.
Binary file added shredded_variant/case-094_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-095.parquet
Binary file not shown.
Binary file added shredded_variant/case-095_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-096.parquet
Binary file not shown.
Binary file added shredded_variant/case-096_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-097.parquet
Binary file not shown.
Binary file added shredded_variant/case-097_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-098.parquet
Binary file not shown.
Binary file added shredded_variant/case-098_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-099.parquet
Binary file not shown.
Binary file added shredded_variant/case-099_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-100.parquet
Binary file not shown.
Binary file added shredded_variant/case-100_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-101.parquet
Binary file not shown.
Binary file added shredded_variant/case-101_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-102.parquet
Binary file not shown.
Binary file added shredded_variant/case-102_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-103.parquet
Binary file not shown.
Binary file added shredded_variant/case-103_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-104.parquet
Binary file not shown.
Binary file added shredded_variant/case-104_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-105.parquet
Binary file not shown.
Binary file added shredded_variant/case-105_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-106.parquet
Binary file not shown.
Binary file added shredded_variant/case-106_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-107.parquet
Binary file not shown.
Binary file added shredded_variant/case-107_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-108.parquet
Binary file not shown.
Binary file added shredded_variant/case-108_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-109.parquet
Binary file not shown.
Binary file added shredded_variant/case-109_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-110.parquet
Binary file not shown.
Binary file added shredded_variant/case-110_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-111.parquet
Binary file not shown.
Binary file added shredded_variant/case-111_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-112.parquet
Binary file not shown.
Binary file added shredded_variant/case-112_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-113.parquet
Binary file not shown.
Binary file added shredded_variant/case-113_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-114.parquet
Binary file not shown.
Binary file added shredded_variant/case-114_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-115.parquet
Binary file not shown.
Binary file added shredded_variant/case-115_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-116.parquet
Binary file not shown.
Binary file added shredded_variant/case-116_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-117.parquet
Binary file not shown.
Binary file added shredded_variant/case-117_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-118.parquet
Binary file not shown.
Binary file added shredded_variant/case-118_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-119.parquet
Binary file not shown.
Binary file added shredded_variant/case-119_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-120.parquet
Binary file not shown.
Binary file added shredded_variant/case-120_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-121.parquet
Binary file not shown.
Binary file added shredded_variant/case-121_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-122.parquet
Binary file not shown.
Binary file added shredded_variant/case-122_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-123.parquet
Binary file not shown.
Binary file added shredded_variant/case-123_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-124.parquet
Binary file not shown.
Binary file added shredded_variant/case-124_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-125-INVALID.parquet
Binary file not shown.
Binary file added shredded_variant/case-125-INVALID_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-126.parquet
Binary file not shown.
Binary file added shredded_variant/case-126_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-126_row-1.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-127.parquet
Binary file not shown.
Binary file added shredded_variant/case-128.parquet
Binary file not shown.
Binary file added shredded_variant/case-129.parquet
Binary file not shown.
Binary file added shredded_variant/case-129_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-130.parquet
Binary file not shown.
Binary file added shredded_variant/case-130_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-131.parquet
Binary file not shown.
Binary file added shredded_variant/case-131_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-132.parquet
Binary file not shown.
Binary file added shredded_variant/case-132_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-133.parquet
Binary file not shown.
Binary file added shredded_variant/case-133_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-134.parquet
Binary file not shown.
Binary file added shredded_variant/case-134_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-135.parquet
Binary file not shown.
Binary file added shredded_variant/case-135_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-136.parquet
Binary file not shown.
Binary file added shredded_variant/case-136_row-0.variant.bin
Binary file not shown.
Binary file added shredded_variant/case-137.parquet
Binary file not shown.
Binary file added shredded_variant/case-138.parquet
Binary file not shown.
Binary file added shredded_variant/case-138_row-0.variant.bin
Binary file not shown.
822 changes: 822 additions & 0 deletions shredded_variant/cases.json

Large diffs are not rendered by default.