feat(metadata): Improve Logical Type Handling on Col Stats#13711

Merged

yihua merged 92 commits intoapache:masterfrom

jonvex:col_stats_new_schema

Sep 23, 2025

Commits on Aug 12, 2025

just schema change
Jonathan Vexler
committed
current progress
Jonathan Vexler
committed

Commits on Aug 13, 2025

builds
Jonathan Vexler
committed
revert avro refactor not needed
Jonathan Vexler
committed
add parquet type parsing
Jonathan Vexler
committed
fix another test
Jonathan Vexler
committed
fix another test
Jonathan Vexler
committed
fix more tests
Jonathan Vexler
committed
fix issue with reading col stats record
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed

Commits on Aug 14, 2025

current progress. Timestamp millis seem to be converted to micros in the reader
Jonathan Vexler
committed

Commits on Aug 16, 2025

tests mostly working now. small precision decimals giving problems
Jonathan Vexler
committed
testing working
Jonathan Vexler
committed

Commits on Aug 17, 2025

add versioning and refactor
Jonathan Vexler
committed
refactor value metadata
Jonathan Vexler
committed
minor cleanup
Jonathan Vexler
committed
refactor the wrapping
Jonathan Vexler
committed
builds
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
clean up a few things
Jonathan Vexler
committed
use null type instead of the v1 type
Jonathan Vexler
committed

Commits on Aug 18, 2025

more cleanup
Jonathan Vexler
committed
add updgrade/downgrade handling
Jonathan Vexler
committed

Commits on Aug 25, 2025

Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed

Commits on Sep 3, 2025

Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
fix test failure datagen
Jonathan Vexler
committed

Commits on Sep 8, 2025

current progress on testing
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
disable tests not ready
Jonathan Vexler
committed
build for spark 3.4
Jonathan Vexler
committed
builds on spark3.3
Jonathan Vexler
committed
test passes for 3.3
Jonathan Vexler
committed
fix spark 3.4 and hopefully 3.5
Jonathan Vexler
committed
fix index version test
Jonathan Vexler
committed

Commits on Sep 9, 2025

fix some more failing tests
Jonathan Vexler
committed
fix array logic
Jonathan Vexler
committed
fix validation for supported types
Jonathan Vexler
committed

Commits on Sep 10, 2025

fix some more test failures
Jonathan Vexler
committed
fix data type issue with timestamp partition cols
Jonathan Vexler
committed
fix single array value not using arraycomparable
Jonathan Vexler
committed
disable test that doesn't work without row writer
Jonathan Vexler
committed
allow row writer again, but disable it for the ds tests with the logical types
Jonathan Vexler
committed
fix schema evolution
Jonathan Vexler
committed

Commits on Sep 11, 2025

fix expression index
Jonathan Vexler
committed
make decimal fixed by default in internal schema
Jonathan Vexler
committed
rever disabled tests
Jonathan Vexler
committed

Commits on Sep 12, 2025

fix expression index
Jonathan Vexler
committed
fix and disable a few more tests
Jonathan Vexler
committed
fix partition stats testing and test value type enums so they don't get changed
Jonathan Vexler
committed
revert throw exception
Jonathan Vexler
committed

Commits on Sep 13, 2025

switch to using localdate as the type for date since online research says that is newer and better. Also fix some more tests
Jonathan Vexler
committed
fix flink array support
Jonathan Vexler
committed
fix failing tests
Jonathan Vexler
committed
fix the type processor things
Jonathan Vexler
committed
fix spark 33 issue
Jonathan Vexler
committed
actually fix spark
Jonathan Vexler
committed

Commits on Sep 15, 2025

move spark value metadata to its own file, revert the gh actions change, and fix one bug in the eight to nine upgrade handler
Jonathan Vexler
committed
allow compare between arrays of different length
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
fix test import
Jonathan Vexler
committed
finish the backwards compat testing
Jonathan Vexler
committed
fix schema because nulls need to come first in unions and only null default for that case. Now spark uses localdatetime so in the assert equals we need to convert, found bug in the write support
Jonathan Vexler
committed

Commits on Sep 16, 2025

add fix for potential npe
Jonathan Vexler
committed
adjust spark adapter bug for 3.3
Jonathan Vexler
committed
remove array support for col stats
Jonathan Vexler
committed
refactor collect col stats to break apart v1 and v2 stuff
Jonathan Vexler
committed

Commits on Sep 17, 2025

address some minor comments
Jonathan Vexler
committed
fix validation for .zip, and also get rid of failing bundle validation
Jonathan Vexler
committed

Commits on Sep 18, 2025

switch to original type for 1.10.1 parquet compat
Jonathan Vexler
committed
revert bot.yml
Jonathan Vexler
committed
remove another ref
Jonathan Vexler
committed
get more debug from validation
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
try to print out the logs again, also make the value metadata spark a utils class instead of extending and make some things package private
Jonathan Vexler
committed
another try to get more debugging
Jonathan Vexler
committed
add parquet 10 adapter to prevent logical annotation issues
Jonathan Vexler
committed
split adapter into 2 parts, and try to load class because checking version seems not to work
Jonathan Vexler
committed
another try to get it working
Jonathan Vexler
committed
use instanceof instead of those enums
Jonathan Vexler
committed
restore validate script
Jonathan Vexler
committed

Commits on Sep 19, 2025

add spark conf flags to the validation
Jonathan Vexler
committed
disable the hive bundle validation
Jonathan Vexler
committed
Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
directly use the class so it will fail when loaded
Jonathan Vexler
committed
addressing review comments
Jonathan Vexler
committed

Commits on Sep 22, 2025

Merge branch 'master' into col_stats_new_schema
Jonathan Vexler
committed
catch all exception
Jonathan Vexler
committed
catch throwable instead of exception
Jonathan Vexler
committed
enable upgrade tests
Jonathan Vexler
committed

Commits on Sep 23, 2025

address review comments
Jonathan Vexler
committed
Fix nits
yihua
committed
Merge branch 'master' into col_stats_new_schema
yihua
committed