[HUDI-5975] Release 0.12.3 prep triage flaky test #8288
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pache#5610) - Alter table drop partition will not add schema to instant. If using delete sql, will get latest instant to get schema, which is "". This PR fixes the parsing of null or empty schema. Co-authored-by: Sagar Sumit <[email protected]>
) Co-authored-by: Nicholas Jiang <[email protected]>
…dieFlinkWriteClient (apache#7509) Different with other write clients, HoodieFlinkWriteClient invokes the dataset writing methods(#upsert or #insert) for each batch of new data set in the long running task. In current impl, a engine-specific hoodie table would be created before performing these actions, and before the table creation, some table bootstrapping operations are performed(such as table upgrade/downgrade, the metadata table bootstrap). These bootstrapping operations are guarded by a trasanction lock. In Flink, these bootstrapping operations can be avoided because they are all performed only once on the coordinator. The changes: - Make BaseHoodieWriteClient#doInitTable non abstract, it now only performs the bootstrapping operations - Add a default impl BaseHoodieWriteClient#initMetadataTable for metadata table bootstrap specifically - Add a new abstract method for creating engine-specific hoodie table
…ut format (apache#7493)" (apache#7526) This reverts commit cc1c1e7.
…pache#7527) Push virtual key fetch inside createFileStatusUnchecked for MOR input format
…iles due to retry (apache#7517) When a write transaction writes uncommitted log files in a delta commit, e.g., due to Spark task retries, these log files stay in the file system after the successful delta commit for some time (unlike uncommitted base files, which are deleted based on the markers). The delta commit metadata does not contain these log files, and the metadata table does not contain these entries either. This is a valid case where the metadata-table-based file listing (providing committed data files) is different from the file system (providing committed data files + uncommited log files in this case). In such a case, before this PR, the metadata table validator throws an exception for the mismatch, because the log blocks are checked based on the commit time, not validated against the commit metadata. This PR fixes the logic of the metadata table validator to check whether the difference in the list of log files between metadata table and direct file system is due to committed log files, based on the commit metadata.
To match the date with announcement email sent on 2022-12-28
…pache#7588) In some of the execution modes, the execution env can only handle single job, so instantiates a fresh new execution env instead of a global singleton in service mode.
Before this change, the Hudi archived timeline is always loaded during the metastore sync process if the last sync time is given. Besides, the archived timeline is not cached inside the meta client if the start instant time is given. These cause performance issues and read timeout on cloud storage due to rate limiting on requests because of loading archived timeline from the storage, when the archived timeline is huge, e.g., hundreds of log files in .hoodie/archived folder. This change improves the timeline loading by (1) only reading active timeline if the last sync time is the same as or after the start of the active timeline; (2) caching the archived timeline based on the start instant time in the meta client, to avoid unnecessary repeated loading of the same archived timeline.
…han earliest pending commit (apache#7568)
…ial uncommitted write metadata event (apache#7611)
Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Jonathan Vexler <=>
…ache#7620) In the beginning, we bootstrap the ckp metadata by cleaning all the messages. This introduces some corner case like 'the write task cannot fetch the pending instant correctly when restarting the job', if a checkpoint succeeds and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engine thinks the checkpoint/instant is successful. Q: Why we clean the messages? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant), 2. the timeline instant is pending while the ckp message does not start (for starting a new instant). For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. Keeping the ckp messages as it is can actually preserve correctness.
…latency marker (apache#7609) Co-authored-by: coder_wang <[email protected]>
…apache#7573) Avoid using GenericRecord in ColumnStatMetadata. HoodieMetadataPayload is constructed using GenericRecord with reflection, and columnStatMetadata stores minValue and maxValue, both of which are GenericRecord types. Once spill is generated, kryo deserialization fails. Root cause AVRO-2377 1.9.2 Modified the type of FIELD_RESERVED to Collections.unmodifiableSet. https://github.com/apache/avro/blame/master/lang/java/avro/src/main/java/org/apache/avro/Schema.java#L483 SPARK-27733 SPARK-34778 (Spark3.2.0) avro version upgraded from 1.8.2 to 1.10.2. As a result, Hudi may encounter UnsupportedOperationException when running Spark3.2.0 or later.
…cy marker (apache#7625) Co-authored-by: coder_wang <[email protected]>
…ollback or clean in data table (apache#7580) Before this change, the archival for the metadata table uses the earliest instant of all actions from the active timeline of the data table. In the archival process, CLEAN and ROLLBACK instants are archived separately apart from commits (check HoodieTimelineArchiver#getCleanInstantsToArchive). Because of this, a very old completed CLEAN or ROLLBACK instant in the data table can block the archive of the metadata table timeline and causes the active timeline of the metadata table to be extremely long, leading to performance issues for loading the timeline. This commit changes the archival in metadata table to not rely on completed rollback or clean in data table, by archiving the metadata table's instants after the earliest commit (COMMIT, DELTA_COMMIT, and REPLACE_COMMIT only, considering non-savepoint commit only if enabling archive beyond savepoint) and the earliest inflight instant (all actions) in the data table's active timeline.
…orts DAY_ROLLING strategy (apache#7656)
…edFileSystemView (apache#7387) Co-authored-by: chenzhiming <[email protected]>
…pache#8150) The HoodieRetryWrapperFileSystem should override all the necessary methods.
- Enhancing spark ds tests to ensure tests for MDT spark datasource read tests are robust
… MDT (apache#8223) - Fixing a corner case bug where compaction in MDT could get triggered w/ partially failed commit in DT.
…e#8248) - Adding timeline server support for integ test suite
(cherry picked from commit 05fc359) # Conflicts: # hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java # hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/format/TestInputFormat.java
…dieFlinkWriteClient (apache#7509) Different with other write clients, HoodieFlinkWriteClient invokes the dataset writing methods(#upsert or #insert) for each batch of new data set in the long-running task. In current impl, an engine-specific hoodie table would be created before performing these actions, and before the table creation, some table bootstrapping operations are performed(such as table upgrade/downgrade, the metadata table bootstrap). These bootstrapping operations are guarded by a transaction lock. In Flink, these bootstrapping operations can be avoided because they are all performed only once on the coordinator. The changes: - Make BaseHoodieWriteClient#doInitTable non abstract, it now only performs the bootstrapping operations - Add a default impl BaseHoodieWriteClient#initMetadataTable for metadata table bootstrap specifically - Add a new abstract method for creating engine-specific hoodie table (cherry picked from commit fd62a14)
(cherry picked from commit d439fab)
…annot be read normally by spark (apache#8026) (cherry picked from commit 31e94ab)
Contributor
Author
|
@hudi-bot run azure |
nsivabalan
force-pushed
the
release-0.12.3-prep-triage-flaky-test
branch
from
f1187da to
8101540
Compare
Contributor
Author
|
@hudi-bot run azure |
nsivabalan
force-pushed
the
release-0.12.3-prep-triage-flaky-test
branch
from
6cc4481 to
a5482d7
Compare
Collaborator
Member
|
@nsivabalan HUDI-5822 requires HUDI-5862 for the test that was added in HUDI-5822 to succeed. Just a headsup. If you pick one without the other, it'll shift the bug appearing in COW tables back to MOR tables. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Release 0.12.3 prep triage flaky test
Impact
Release 0.12.3 prep triage flaky test
Risk level (write none, low medium or high below)
low.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist