fix(ingest): Repair affected logical timestamp milli tables#14161

Merged

yihua merged 84 commits intoapache:masterfrom

jonvex:timestamp_logical_final

Nov 11, 2025

Commits on Nov 8, 2025

current progress

Jonathan Vexler
authored and
yihua
committed
seems to be working for spark non vectorized and avro

Jonathan Vexler
authored and
yihua
committed
filters working

Jonathan Vexler
authored and
yihua
committed
prevent overflow

Jonathan Vexler
authored and
yihua
committed
use read support instead of mapping function

Jonathan Vexler
authored and
yihua
committed
use repaired schema instead of doing operations after we read the data

Jonathan Vexler
authored and
yihua
committed
add spark log support

Jonathan Vexler
authored and
yihua
committed
remove find cols to multiply class

Jonathan Vexler
authored and
yihua
committed
log file changes as requested and set up the read supports for spark and avro

Jonathan Vexler
authored and
yihua
committed
hive working

Jonathan Vexler
authored and
yihua
committed
add individual test and fix issue with dropping messagetype logical

Jonathan Vexler
authored and
yihua
committed
revert calls for rewrite avro with extra param

Jonathan Vexler
authored and
yihua
committed
revert config to prevent timestamp evolutions

Jonathan Vexler
authored and
yihua
committed
A few fixes

Jonathan Vexler
authored and
yihua
committed
fix bug with field reuse in avro schema repair

Jonathan Vexler
authored and
yihua
committed
fix read parquet log block timestamp ntz

Jonathan Vexler
authored and
yihua
committed
allow long to timestampntz without cast

Jonathan Vexler
authored and
yihua
committed
refactor AvroSchemaRepair for performance and add unit tests

Jonathan Vexler
authored and
yihua
committed
refactor schema repair for performance and add testing

Jonathan Vexler
authored and
yihua
committed
try fix other spark versions

Jonathan Vexler
authored and
yihua
committed
fix spark 3.3 build

Jonathan Vexler
authored and
yihua
committed
fix spark3.4 build

Jonathan Vexler
authored and
yihua
committed
hopefully fix spark4

Jonathan Vexler
authored and
yihua
committed
fix issue with union schema, add table schema to cdc in missing place

Jonathan Vexler
authored and
yihua
committed
add spark cow read testing for repair

Jonathan Vexler
authored and
yihua
committed
building, and add spark mor tests

Jonathan Vexler
authored and
yihua
committed
forgot to add the zips

Jonathan Vexler
authored and
yihua
committed
cow write testing

Jonathan Vexler
authored and
yihua
committed
add mor testing

Jonathan Vexler
authored and
yihua
committed
disable tests for spark 3.3

Jonathan Vexler
authored and
yihua
committed
fix for spark 3.4

Jonathan Vexler
authored and
yihua
committed
fix spark33 for real

Jonathan Vexler
authored and
yihua
committed
remove fg reader test

Jonathan Vexler
authored and
yihua
committed
remove unneeded avro utils change

Jonathan Vexler
authored and
yihua
committed
fix spark 4

Jonathan Vexler
authored and
yihua
committed
fix timestamps in deltastreamer test

Jonathan Vexler
authored and
yihua
committed
fix failing test

Jonathan Vexler
authored and
yihua
committed
vectorized fallback for 3.3 and 3.4

Jonathan Vexler
authored and
yihua
committed
fix vectorized fallback

Jonathan Vexler
authored and
yihua
committed
add testing, and also fallback for local timestamp millis

Jonathan Vexler
authored and
yihua
committed
add tests from java-parquet

Jonathan Vexler
authored and
yihua
committed
replace import with hardcode

Jonathan Vexler
authored and
yihua
committed
fix long import names

Jonathan Vexler
authored and
yihua
committed
address most review comments

Jonathan Vexler
authored and
yihua
committed
Fixing spark3.3 reads

nsivabalan
authored and
yihua
committed
minor renames

nsivabalan
authored and
yihua
committed
Adding java docs

nsivabalan
authored and
yihua
committed
java docs

nsivabalan
authored and
yihua
committed
refactor repairFooterSchema to a common module

Vamsi
authored and
yihua
committed
rename resolveNullableSchema method

Vamsi
authored and
yihua
committed
add check for no compactions and clustering on test table

Vamsi
authored and
yihua
committed
Add checks for enableLogicalTimestampFieldRepair

Vamsi
authored and
yihua
committed
Revert pom changes
yihua
committed
Revert "refactor repairFooterSchema to a common module"
yihua
committed
Fix scalastyle
yihua
committed
Fix build on Spark 3.3 and 3.4
yihua
committed
Fix TestHoodieDeltaStreamer
yihua
committed
Fix licences
yihua
committed
Fix Spark 4
yihua
committed
Fix Spark 4 and renames
yihua
committed
Fix Spark40LegacyHoodieParquetFileFormat
yihua
committed
Add flag for repairing loogical timestamp in avro log file reader

Vamsi
authored and
yihua
committed
Fix checkstyle
yihua
committed
Fix bug in config passing
yihua
committed
Disable TestAvroSchemaConverter to validate other tests
yihua
committed
Update testNestedTypeVectorizedReadWithTypeChange test behavior
yihua
committed
Another fix
yihua
committed
Fix the testLogicalTypesReadRepair

linliu-code
authored and
yihua
committed
Make schema fetching and parsing lazy for adjusting column and partition stats definition
yihua
committed
Fix testReadIndexDefFromStorage
yihua
committed
Enhance test
yihua
committed
Disable failed tests for investigation
yihua
committed
Fix test logic
yihua
committed

Commits on Nov 10, 2025

Fix naming of configs
yihua
committed
fix col stats test failures

Vamsi
authored and
nsivabalan
committed
Addressing Siva's feedback on fixing col stats for v1 index defn
nsivabalan
committed