[HUDI-5785] Enhance Spark Datasource tests #7938
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yihua
added
priority:critical
yihua
force-pushed
the
HUDI-5785-enhance-mdt-spark-datasource-test
branch
from
9223732 to
4e7da70
Compare
alexeykudinkin
previously requested changes
Feb 14, 2023
Contributor
alexeykudinkin
left a comment
alexeykudinkin
left a comment
There was a problem hiding this comment.
Let's start the process of unifying all of the utilities to make sure we're not getting bitten by the same thing again
https://github.com/apache/hudi/pull/7702/files#diff-93d5c78a2db3470cef4a643a3b41b8b97876f411310a5653d232525c87a6d749
Contributor
Author
I created this to unify all APIs to construct Spark configs: HUDI-5788 |
nsivabalan
force-pushed
the
HUDI-5785-enhance-mdt-spark-datasource-test
branch
from
4e7da70 to
1bf2b8c
Compare
Contributor
|
Since this has been approved already, will go ahead and merge once CI is green. |
Collaborator
nsivabalan
approved these changes
Mar 18, 2023
nsivabalan
dismissed
alexeykudinkin’s stale review
We got 2 approvals and moving ahead
nsivabalan
pushed a commit
to nsivabalan/hudi
that referenced
this pull request
Mar 18, 2023
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
nsivabalan
pushed a commit
to nsivabalan/hudi
that referenced
this pull request
Mar 22, 2023
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
fengjian428
pushed a commit
to fengjian428/hudi
that referenced
this pull request
Apr 5, 2023
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
stayrascal
pushed a commit
to stayrascal/hudi
that referenced
this pull request
Apr 20, 2023
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
KnightChess
pushed a commit
to KnightChess/hudi
that referenced
this pull request
Jan 2, 2024
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Previously, we found that Spark Datasource read of metadata table was broken and the issue is fixed by #7924. However, the
TestMetadataTableWithSparkDataSourceguarding the exact same functionality did not fail in CI or local mvn command below. After investigation, the Hudi Spark configs (spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog,spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension) are not properly added to the Spark session in the test environment.This PR sets the proper Hudi Spark configs for Spark Datasource tests and adds one more test on reading metadata table through Spark Datasource.
Impact
After this change, without the fix #7924, the following test fails which is consistent with the behavior of spark-shell (previously it passed without raising the alarm).
Risk level
low
Documentation Update
N/A
Contributor's checklist