Skip to content

Timestamp logical final #14233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vamsikarnika wants to merge 62 commits into from
Closed

Timestamp logical final #14233

vamsikarnika wants to merge 62 commits into from

Conversation

@vamsikarnika
Copy link
Collaborator

@vamsikarnika vamsikarnika commented Nov 7, 2025

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

Jonathan Vexler added 30 commits November 6, 2025 14:29
@yihua
current progress
ccf5dd3
@yihua
seems to be working for spark non vectorized and avro
add1a90
@yihua
filters working
3b2ceca
@yihua
prevent overflow
eaad2b1
@yihua
use read support instead of mapping function
5600b37
@yihua
use repaired schema instead of doing operations after we read the data
018a11e
@yihua
add spark log support
c9bdd39
@yihua
remove find cols to multiply class
a8430cf
@yihua
log file changes as requested and set up the read supports for spark …
2dde0fc 
…and avro
@yihua
hive working
36c0d1c
@yihua
add individual test and fix issue with dropping messagetype logical
d7d6dd5
@yihua
revert calls for rewrite avro with extra param
04c6ce5
@yihua
revert config to prevent timestamp evolutions
00c85e5
@yihua
A few fixes
195f90e
@yihua
fix bug with field reuse in avro schema repair
3f6b4a8
@yihua
fix read parquet log block timestamp ntz
93d005a
@yihua
allow long to timestampntz without cast
47ce37e
@yihua
refactor AvroSchemaRepair for performance and add unit tests
036d9cc
@yihua
refactor schema repair for performance and add testing
bdff8ff
@yihua
try fix other spark versions
b8f889e
@yihua
fix spark 3.3 build
3c401e5
@yihua
fix spark3.4 build
2d6d1ea
@yihua
hopefully fix spark4
6a47cc9
@yihua
fix issue with union schema, add table schema to cdc in missing place
11e0422
@yihua
add spark cow read testing for repair
ead216c
@yihua
building, and add spark mor tests
5f7512c
@yihua
forgot to add the zips
6ec7eb5
@yihua
cow write testing
7bf05a9
@yihua
add mor testing
5167319
@yihua
disable tests for spark 3.3
31e5a59
Jonathan Vexler and others added 26 commits November 6, 2025 14:34
@yihua
fix failing test
d0d502a
@yihua
vectorized fallback for 3.3 and 3.4
52fced7
@yihua
fix vectorized fallback
f0cb0c9
@yihua
add testing, and also fallback for local timestamp millis
0bd08f0
@yihua
add tests from java-parquet
fb711a0
@yihua
replace import with hardcode
bab0c7a
@yihua
fix long import names
e6fba1b
@yihua
address most review comments
7deced3
@nsivabalan @yihua
Fixing spark3.3 reads
3a95184
@nsivabalan @yihua
minor renames
58322eb
@nsivabalan @yihua
Adding java docs
38398f1
@nsivabalan @yihua
java docs
6aafc13
@yihua
refactor repairFooterSchema to a common module
5153d4e
@yihua
rename resolveNullableSchema method
8a0759c
@yihua
add check for no compactions and clustering on test table
2911d07
@yihua
Add checks for enableLogicalTimestampFieldRepair
d23db24
@yihua
Revert pom changes
a32494f
@yihua
Revert "refactor repairFooterSchema to a common module"
6eec8a7 
This reverts commit 5153d4e.
@yihua
Fix scalastyle
8ccd313
@yihua
Fix build on Spark 3.3 and 3.4
3456728
@yihua
Fix TestHoodieDeltaStreamer
bee84af
@yihua
Fix licences
9f40c75
@yihua
Fix Spark 4
11a2dce
@yihua
Fix Spark 4 and renames
eabdc23
@yihua
Fix Spark40LegacyHoodieParquetFileFormat
cdf8624
Add flag for repairing loogical timestamp in avro log file reader
36ac25f
@github-actions github-actions bot added the size:XL PR with lines of changes > 1000 label Nov 7, 2025
@hudi-bot
Copy link
Collaborator

hudi-bot commented Nov 7, 2025

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

yihua
yihua reviewed Nov 7, 2025
View reviewed changes
Copy link
Contributor

@yihua yihua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vamsikarnika thanks for the change. Closing this one as the additional changes are cherrypicked into #14161.

@yihua yihua closed this Nov 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yihua yihua yihua left review comments

Assignees

No one assigned

Labels

size:XL PR with lines of changes > 1000

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vamsikarnika @hudi-bot @yihua @nsivabalan