[HUDI-6787] Integrate the new file group reader with Hive query engine#10422

Merged

yihua merged 43 commits intoapache:masterfrom

jonvex:use_fg_reader_hive

Jun 9, 2024

Commits on May 31, 2024

squash commits

Jonathan Vexler
authored and
codope
committed
refactor a bit and add some comments

Jonathan Vexler
authored and
codope
committed
make build properly

Jonathan Vexler
authored and
codope
committed
fix some of the failing tests

Jonathan Vexler
authored and
codope
committed
revert to old impl when schema evolution enabled

Jonathan Vexler
authored and
codope
committed
disable fg reader for stupid test

Jonathan Vexler
authored and
codope
committed
fix some failing tests

Jonathan Vexler
authored and
codope
committed
assigned the ports backwards

Jonathan Vexler
authored and
codope
committed
verbose output bundle validation

Jonathan Vexler
authored and
codope
committed
add volume to docker compose

Jonathan Vexler
authored and
codope
committed
need to lowercase the field names omg

Jonathan Vexler
authored and
codope
committed
don't remove partition if it's listed in the schema

Jonathan Vexler
authored and
codope
committed
put partition cols at end of output

Jonathan Vexler
authored and
codope
committed
invert filter

Jonathan Vexler
authored and
codope
committed
support no base file, only log. support read from non-hudi table

Jonathan Vexler
authored and
codope
committed
disable for skip merge as well

Jonathan Vexler
authored and
codope
committed
fix non hoodie path read

Jonathan Vexler
authored and
codope
committed
revert setting precombine

Jonathan Vexler
authored and
codope
committed
fix no meta cols table

Jonathan Vexler
authored and
codope
committed
check if no requested fields

Jonathan Vexler
authored and
codope
committed
create empty schema properly

Jonathan Vexler
authored and
codope
committed
check if metadata folder exists

Jonathan Vexler
authored and
codope
committed
handle mor with no meta fields

Jonathan Vexler
authored and
codope
committed
disable reader for a test because mor seems to work different

Jonathan Vexler
authored and
codope
committed
delete partition column from the jobconf if it is written in the file

Jonathan Vexler
authored and
codope
committed
modify data schema due to partition column madness

Jonathan Vexler
authored and
codope
committed
remove unused import

Jonathan Vexler
authored and
codope
committed
add some comments

Jonathan Vexler
authored and
codope
committed
don't add partition fields when the data schema doesn't have them

Jonathan Vexler
authored and
codope
committed
address review feedback

Jonathan Vexler
authored and
codope
committed
accidently put remove in for loop for combine reader

Jonathan Vexler
authored and
codope
committed
get building again

Jonathan Vexler
authored and
codope
committed
address some review comments

Jonathan Vexler
authored and
codope
committed
add reviewer suggested change

Jonathan Vexler
authored and
codope
committed
add missing params fg reader

Jonathan Vexler
authored and
codope
committed
address some comments

Jonathan Vexler
authored and
codope
committed
Fix post rebase
codope
committed
fix hudi table check
codope
committed
Address self feedback
codope
committed

Commits on Jun 7, 2024

Merge branch 'master' into use_fg_reader_hive
Jonathan Vexler
committed
fix failing test
Jonathan Vexler
committed

Commits on Jun 8, 2024

Merge branch 'master' into use_fg_reader_hive
Jonathan Vexler
committed
fix failing test
Jonathan Vexler
committed