[HUDI-2285] Metadata Table synchronous design #3426

prashantwason · 2021-08-07T08:52:13Z

What is the purpose of the pull request

Implementation of the metadata table synchronous design.

Brief change log

Please see HUDI-2285 for changes and how they are implemented. There are different commits that handle each change.

Verify this pull request

Metadata table unit tests have been updated.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hudi-bot · 2021-08-07T08:56:30Z

CI report:

66366b9884f4ae533751e7a379d23e306d61ac94 UNKNOWN
21e8d70 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

1. Removed code which calls syncTableMetadata 2. Unit tests are broken because not all functionality have been implemented yet.

Reader does not need to merge the instants in memory. It simply opens the base and log files, validates which log blocks to read (should have completed instants on dataset timeline).

…inst file listing. Validation does not work in several cases especially with multi-writer. So its best to remove it.

We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.

1. There will be fixed number of shards for each Metadata Table partition. 2. Shards are implemented using filenames of format fileId00ABCD where ABCD is the shard number. This allows easy identification of the files and their order while still keeping the names unique. 3. Shards are pre-allocation during the time of bootstrap. 4. Currently only files partition has 1 shard. But this code is required for record-level-index so implemented here.

…han latest compaction on metadata table. LogBlocks written to the log file of Metadata Table need to be validated - they are used only if they correspond to a completed action on the dataset.

…ing the table and re-bootstrapping. The two versions differ in schema and its complicated to check whether the table is in sync. So its simpler to re-bootstrap as its only the file listing which needs to be re-bootstrapped.

Since each operation on metadata table writes to the same files (file-listing partition has a single FileSlice), we can only allow single-writer access to the metadata table. For this, the Transaction Manager is used to lock the table before any updates.

leesf · 2021-08-09T15:04:24Z

...lient/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java

      HoodieTimelineArchiveLog archiveLog = new HoodieTimelineArchiveLog(config, table);
      archiveLog.archiveIfRequired(context);
      autoCleanOnCommit();
-      syncTableMetadata();


why remove the metadata table sync?

there is no more additional sync process, with this re-design.

leesf · 2021-08-09T15:13:32Z

...t-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackActionExecutor.java


  protected void finishRollback(HoodieRollbackMetadata rollbackMetadata) throws HoodieIOException {
    try {
+      // TODO: Potential error here - rollbacks have already completed here so if the syncTableMetadata fails,


here how can we handle the case, re-bootstrap?

sorry, @leesf can you please clarify your question.

leesf · 2021-08-09T15:15:11Z

...-client/src/main/java/org/apache/hudi/table/action/commit/BaseFlinkCommitActionExecutor.java

      HoodieCommitMetadata metadata = CommitUtils.buildMetadata(writeStats, result.getPartitionToReplaceFileIds(),
          extraMetadata, operationType, getSchemaToStoreInCommit(), getCommitActionType());

+      syncTableMetadata(metadata);


here means we take syncing to metadata table into a commit. more reasonable than making sync table metadata in postCommit

For Flink, this code is still executed at the driver, right?

...ent/hudi-flink-client/src/main/java/org/apache/hudi/table/upgrade/FlinkUpgradeDowngrade.java

nbalajee

LGTM.

Before completing the dataset commit, writing the changes to the metadata table makes sense. a) Makes the dataset commit consistent with metadata table commit b) writes are scalable for future metadata operations, like record level index.

Could you update the description section, to reflect that (a) we are writing to the metadata table from the in-memory states available, as part of the original/dataset commit. (b) dataset commit will be complete, only after the metadata write is completed/successful.

...client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java

nbalajee · 2021-08-10T19:26:20Z

...i-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java

+    //  Dataset:   C1        C2         C3.inflight[failed]  C4.clean     C5   R6[rolls back C3]
+    //  Metadata:  C1.delta  C2.delta
+    //
+    // When R6 completes, C3.xxx will be deleted. When C5 completes, C4, C5 and R6 will be committed to Metadata Table


If R5 completed, then metadata table can be updated with C4, C5 etc.

Corner case: R6 rolled back C3 deleting all associated data/metadata files. But before R6.commit was written, Rollback failed (eg, failure related to writing rollback metadata - in other words c3.inflight was replaced by R6.inflight).

nbalajee

LGTM.

nbalajee · 2021-08-10T22:52:51Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

      Option<HoodieInstant> lastInstant = metaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
      String latestMetaInstantTimestamp = lastInstant.map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);


latestMetaInstantTimestamp is same as the last instant on completedInstants of the dataset, isn't?

nbalajee

LGTM.

nbalajee

LGTM.

nbalajee · 2021-08-10T23:48:35Z

...park-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java

+  private void compactIfNecessary(SparkRDDWriteClient writeClient, String instantTime) {
+    List<HoodieInstant> pendingInstants = datasetMetaClient.reloadActiveTimeline().filterInflightsAndRequested().findInstantsBefore(instantTime)
+        .getInstants().collect(Collectors.toList());
+    if (!pendingInstants.isEmpty()) {


If the dataset timeline has an incomplete commit (multiple parallel commits C1, C2 were started, C2 succeeded but C1 failed leaving C1.inflight). Dataset commits and delta commits on metadata table will be successful, but compaction could fall behind, until manual intervention.

With current approach, manual intervention would be required to clean up the inflight to allow compaction to make progress or would ingestion/dataset commits fail due to maxArchivalLimit on metadata table is reached(due to delta commits created, but not compacted)?

@nbalajee : are you talking about compaction of data table or metadata table. We expect that if compaction of data table fails, there will be continuous retries. If not, liveness will not be guaranteed. But in general, we need to think through this and see if we can relax this constraint. We will take this up as a follow up.

nbalajee

LGTM.

nbalajee · 2021-08-11T00:20:12Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+   *    fc9f18eb-6049-4f47-bc51-23884bef0001
+   *    fc9f18eb-6049-4f47-bc51-23884bef0002


When an executor receives more than one file's worth of records, we end up creating f1-0_wt_cx.parquet, f1-1_wt2_cx.parquet etc. Should the same naming convention used here?

In other words, should you use the 36 byte fileId prefix + "-ABCD"?

if we use the 0000-9999 as a hash partition, then we cannot reuse that?

@nbalajee : these shards/buckets are instantiated by driver based on configured partitions and bucket counts per partition. Not sure how executor is involved here. can you help me understand.

@vinothchandar : sorry, I don't understand your point on hash partition. can you help me understand please.

nbalajee · 2021-08-11T00:22:01Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+            .withFs(datasetMetaClient.getFs())
+            .withRolloverLogWriteToken(FSUtils.makeWriteToken(0, 0, 0))
+            .withLogWriteToken(FSUtils.makeWriteToken(0, 0, 0))
+            .withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();


Are there any advantages to creating a log file vs the base file here?

as you might have known, we fence compaction with any inflght commits in data table. And so any new writes for a new shard or bucket has to start with log file and not go into base file. If we don't create a log file here, a new write to a new bucket might start creating a base file first. And due to the fact that we fence compaction, we can't create base file here.

nbalajee · 2021-08-11T00:46:55Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

+   * @return An integer hash of the given string
+   */
+  public static int keyToShard(String str, int numShards) {
+    int h = 0;


With a 6+ character key, int would overflow. Should this be a long?

Also, should keyToShard be customizable with other hash functions?

but this is what I found in java sdk source code as well for String.hashcode().
guess it should be fine. thats why we take absolute value below and calculate the right bucket.

leesf · 2021-08-13T14:12:40Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+    // The lock location should be different from the dataset
+    Properties properties = new Properties();
+    properties.putAll(datasetWriteConfig.getProps());
+    properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, properties.getProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, datasetWriteConfig.getBasePath() + "/.hoodie/.locks") + "/metadata");


make the '/.hoodie/.locks' constant?

filesystem based lock may not work on cloud storage. not sure if we can assume this.

So, how do you suggest we go about this.
I am thinking if we can do something like this.
Determine the type of lock acquired and automatically derive the properties accordingly.
for FileSystemBasedLock:
User has to set "hoodie.write.lock.filesystem.path" for data table.
we can add "/metadata" to the value set for this.
for eg: if someone sets data table config to "hudi_path/.locks", we infer path for metadata as "hudi_path/.locks/metadata".

for metastore based locks:
I could not think of a way to auto derive metadata table configs. bcoz, we have 3 configs for data table locks. database name, table name and metastore uris. don't think we can do much from these configs. we might have to add 3 new props (something like below) and expect users to set them as well.
"hoodie.write.lock.hivemetastore.database" -> "hoodie.metadata.write.lock.hivemetastore.database"
"hoodie.write.lock.hivemetastore.table" - "hoodie.metadata.write.lock.hivemetastore.table"
"hoodie.write.lock.hivemetastore.uris" - "hoodie.metadata.write.lock.hivemetastore.uris"

for zookeeper based locks:
we can infer from "hoodie.write.lock.zookeeper.lock_key". we can suffix "_metadata" to the value set of this by the user.

I am not addressing this issue right now. once we have some consensus, will work on the fix.

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

vinothchandar

Made one pass. @nsivabalan is going to take a stab at rebasing , addressing some stuff.

vinothchandar · 2021-09-01T19:38:12Z

...lient/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java

        lastCompletedTxnAndMetadata.isPresent() ? Option.of(lastCompletedTxnAndMetadata.get().getLeft()) : Option.empty());
    try {
      preCommit(instantTime, metadata);
+      table.getMetadataWriter().ifPresent(w -> ((HoodieTableMetadataWriter)w).update(metadata, instantTime));


nts: first committing to metadata table

vinothchandar · 2021-09-01T19:38:53Z

...lient/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java

      HoodieTimelineArchiveLog archiveLog = new HoodieTimelineArchiveLog(config, table);
      archiveLog.archiveIfRequired(context);
      autoCleanOnCommit();
-      syncTableMetadata();


there is no more additional sync process, with this re-design.

vinothchandar · 2021-09-01T19:41:05Z

...lient/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java

      HoodieTableMetaClient metaClient) {
    setOperationType(writeOperationType);
    this.lastCompletedTxnAndMetadata = TransactionUtils.getLastCompletedTxnInstantAndMetadata(metaClient);
-    this.txnManager.beginTransaction(Option.of(new HoodieInstant(State.INFLIGHT, metaClient.getCommitActionType(), instantTime)), lastCompletedTxnAndMetadata


nts: this lock was only being taken for purposes of syncing. So removing this is fine.

vinothchandar · 2021-09-01T19:42:37Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

-        syncFromInstants(datasetMetaClient);
-        metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.SYNC_STR, timer.endTimer()));
-      }
+      this.datasetMetaClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(datasetWriteConfig.getBasePath()).build();


nts: loading this afresh here.

vinothchandar · 2021-09-01T19:43:15Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+    // The lock location should be different from the dataset
+    Properties properties = new Properties();
+    properties.putAll(datasetWriteConfig.getProps());
+    properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, properties.getProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, datasetWriteConfig.getBasePath() + "/.hoodie/.locks") + "/metadata");


hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

vinothchandar · 2021-09-01T20:29:29Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, lastSyncTs);
+    if (!wasSynced) {


nts: need to revisit again with rollback/restore issues fixed

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

vinothchandar · 2021-09-01T20:39:13Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

+
+        // If the base file is present then create a reader
+        Option<HoodieBaseFile> basefile = slice.getBaseFile();
+        if (basefile.isPresent()) {


do we send initial data to log files? without any base? is this why we are creating the log files with empty delete block upfront?

vinothchandar · 2021-09-01T20:42:07Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

+            .withSpillableMapBasePath(spillableMapDirectory)
+            .withDiskMapType(commonConfig.getSpillableDiskMapType())
+            .withBitCaskDiskMapCompressionEnabled(commonConfig.isBitCaskDiskMapCompressionEnabled())
+            .withLogBlockTimestamps(validInstantTimestamps)


this is what fences all uncommitted data from being read out of metadata table

vinothchandar · 2021-09-03T21:34:02Z

Closing in favor for #3590

vinothchandar · 2021-09-03T21:35:14Z

@nbalajee @nsivabalan lets resolve the CR feedback there and land.

nsivabalan · 2021-09-06T17:04:12Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, lastSyncTs);
+    if (!wasSynced) {


nts on processRollbackMetadata(...).

with multi-writer, lastsyncTs could be mis-leading. So, just calling it out for understanding purpose.
lets say,
dataset timeline:
C1.complete, C2.complete, C3.inflight, C4.complete
metadata timeline:
C1.complete, C2.complete, C3.inflight, C4.complete.

rollback is triggered for C3.
Case1: C3 is committed to metadata.
Case2: C3 is not committed to metadata yet.

Lets see what happens when we processRollbackMetadata() R5 for rollback of C3.
As per code, 2 cases will be ignored while trying to process rollback metadata.
Case A: if instant to rollback > lastSyncTs. // metadata table is not yet caught up. skip processing.
Case B: If instant to rollback was never committed to metadata and is part of active timeline. refers to failed commit. skip processing.
For any other case, we will process the rollback metadata and add/delete appropriate files.

Case1: C3 is committed to metadata.
As per logic above,
Case A is false.
Case B true. since C3 is part of active timeline, we can't skip processing.
And so we will process the rollback since C3 was committed to metadata table.

Case2: C3 not committed to metadata.
As per logic above,
Case A is false.
Case B false. since C3 is committed to metadata, we can skip processing.

nsivabalan · 2021-09-20T11:06:41Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+    // The lock location should be different from the dataset
+    Properties properties = new Properties();
+    properties.putAll(datasetWriteConfig.getProps());
+    properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, properties.getProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, datasetWriteConfig.getBasePath() + "/.hoodie/.locks") + "/metadata");


So, how do you suggest we go about this.
I am thinking if we can do something like this.
Determine the type of lock acquired and automatically derive the properties accordingly.
for FileSystemBasedLock:
User has to set "hoodie.write.lock.filesystem.path" for data table.
we can add "/metadata" to the value set for this.
for eg: if someone sets data table config to "hudi_path/.locks", we infer path for metadata as "hudi_path/.locks/metadata".

for metastore based locks:
I could not think of a way to auto derive metadata table configs. bcoz, we have 3 configs for data table locks. database name, table name and metastore uris. don't think we can do much from these configs. we might have to add 3 new props (something like below) and expect users to set them as well.
"hoodie.write.lock.hivemetastore.database" -> "hoodie.metadata.write.lock.hivemetastore.database"
"hoodie.write.lock.hivemetastore.table" - "hoodie.metadata.write.lock.hivemetastore.table"
"hoodie.write.lock.hivemetastore.uris" - "hoodie.metadata.write.lock.hivemetastore.uris"

for zookeeper based locks:
we can infer from "hoodie.write.lock.zookeeper.lock_key". we can suffix "_metadata" to the value set of this by the user.

I am not addressing this issue right now. once we have some consensus, will work on the fix.

nsivabalan · 2021-09-20T11:08:49Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

      .initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());

    initTableMetadata();
+    initializeShards(datasetMetaClient, MetadataPartitionType.FILES.partitionPath(), createInstantTime, 1);


Guess the intention here is for creating buckets for record level index and any other such partitions we might have. Will change it to buckets.

nsivabalan · 2021-09-20T11:20:04Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+   *    fc9f18eb-6049-4f47-bc51-23884bef0001
+   *    fc9f18eb-6049-4f47-bc51-23884bef0002


@nbalajee : these shards/buckets are instantiated by driver based on configured partitions and bucket counts per partition. Not sure how executor is involved here. can you help me understand.

@vinothchandar : sorry, I don't understand your point on hash partition. can you help me understand please.

nsivabalan · 2021-09-20T11:20:29Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+    final String newFileIdPrefix = newFileId.substring(0, 32);
+    final HashMap<HeaderMetadataType, String> blockHeader = new HashMap<>();
+    blockHeader.put(HeaderMetadataType.INSTANT_TIME, instantTime);
+    final HoodieDeleteBlock block = new HoodieDeleteBlock(new HoodieKey[0], blockHeader);


yes. you are right.

nsivabalan · 2021-09-20T11:23:05Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+            .onParentPath(FSUtils.getPartitionPath(metadataWriteConfig.getBasePath(), partition))
+            .withFileId(shardFileId).overBaseCommit(instantTime)
+            .withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
+            .withFileSize(0L)


yes, for first log block (even in regular flow), we do the same.

return HoodieLogFormat.newWriterBuilder() .onParentPath(FSUtils.getPartitionPath(hoodieTable.getMetaClient().getBasePath(), partitionPath)) .... .withFileSize(latestLogFile.map(HoodieLogFile::getFileSize).orElse(0L)) ....

for when there is no latestLogFile for a given file group, we set the size to 0.

nsivabalan · 2021-09-20T12:18:47Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

+            .withFs(datasetMetaClient.getFs())
+            .withRolloverLogWriteToken(FSUtils.makeWriteToken(0, 0, 0))
+            .withLogWriteToken(FSUtils.makeWriteToken(0, 0, 0))
+            .withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();


as you might have known, we fence compaction with any inflght commits in data table. And so any new writes for a new shard or bucket has to start with log file and not go into base file. If we don't create a log file here, a new write to a new bucket might start creating a base file first. And due to the fact that we fence compaction, we can't create base file here.

nsivabalan · 2021-09-20T12:23:47Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

+
+        // If the base file is present then create a reader
+        Option<HoodieBaseFile> basefile = slice.getBaseFile();
+        if (basefile.isPresent()) {


nsivabalan · 2021-09-20T12:24:22Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

+            .withSpillableMapBasePath(spillableMapDirectory)
+            .withDiskMapType(commonConfig.getSpillableDiskMapType())
+            .withBitCaskDiskMapCompressionEnabled(commonConfig.isBitCaskDiskMapCompressionEnabled())
+            .withLogBlockTimestamps(validInstantTimestamps)


nsivabalan · 2021-09-20T12:28:41Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java

+  @Override
+  public Option<String> getLatestCompactionTime() {
+    if (metaClient != null) {
+      Option<HoodieInstant> latestCompaction = metaClient.getActiveTimeline().getCommitTimeline().filterCompletedInstants().lastInstant();


nts: we filter for commitTimeline bcoz, clean, clustering, delta commits, etc may not result in a "commit", but respective ones (like clean, replace commit, delta commit). and so only compaction will result in a "commit instant".

nsivabalan · 2021-09-20T12:41:40Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

+   * @return An integer hash of the given string
+   */
+  public static int keyToShard(String str, int numShards) {
+    int h = 0;


but this is what I found in java sdk source code as well for String.hashcode().
guess it should be fine. thats why we take absolute value below and calculate the right bucket.

prashantwason requested a review from vinothchandar August 7, 2021 08:52

prashantwason added 10 commits August 7, 2021 02:35

[HUDI-848] Synchronous commits before completion of instants.

985ae86

1. Removed code which calls syncTableMetadata 2. Unit tests are broken because not all functionality have been implemented yet.

[HUDI-852] Reader side logic with unit test.

e85c712

Reader does not need to merge the instants in memory. It simply opens the base and log files, validates which log blocks to read (should have completed instants on dataset timeline).

[HUDI-XXX] Remove option for runtime validation of metadata table aga…

d683122

…inst file listing. Validation does not work in several cases especially with multi-writer. So its best to remove it.

[HUDI-859] Fixed compactions on the Metadata Table.

690ea20

We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.

[HUDI-850] Do not archive instants on the dataset if they are newer t…

b658779

…han latest compaction on metadata table. LogBlocks written to the log file of Metadata Table need to be validated - they are used only if they correspond to a completed action on the dataset.

[HUDI-1191] Fixed all unit tests in TestHoodieBackedMetadata.

fdc267c

Fixed checkstyle

59cd841

prashantwason force-pushed the pw_metadata_table_v2_oss branch from 66366b9 to 59cd841 Compare August 7, 2021 09:35

leesf reviewed Aug 9, 2021

View reviewed changes

...ent/hudi-flink-client/src/main/java/org/apache/hudi/table/upgrade/FlinkUpgradeDowngrade.java Show resolved Hide resolved

vinothchandar self-assigned this Aug 9, 2021

nbalajee suggested changes Aug 10, 2021

View reviewed changes

nbalajee reviewed Aug 10, 2021

View reviewed changes

nbalajee approved these changes Aug 10, 2021

View reviewed changes

nbalajee reviewed Aug 11, 2021

View reviewed changes

vinothchandar added the big-needle-movers label Aug 12, 2021

leesf reviewed Aug 13, 2021

View reviewed changes

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Show resolved Hide resolved

prashantwason mentioned this pull request Aug 19, 2021

[HUDI-53] Adding a record level index based on the Metadata Table v2 #3508

Closed

5 tasks

Enabling metadata table by default.

21e8d70

vinothchandar reviewed Sep 1, 2021

View reviewed changes

nsivabalan mentioned this pull request Sep 3, 2021

[HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426 #3590

Merged

5 tasks

vinothchandar closed this Sep 3, 2021

nsivabalan reviewed Sep 6, 2021

View reviewed changes

nsivabalan reviewed Sep 20, 2021

View reviewed changes

		Option<HoodieInstant> lastInstant = metaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
		String latestMetaInstantTimestamp = lastInstant.map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);

		* fc9f18eb-6049-4f47-bc51-23884bef0001
		* fc9f18eb-6049-4f47-bc51-23884bef0002

[HUDI-2285] Metadata Table synchronous design #3426

[HUDI-2285] Metadata Table synchronous design #3426

Uh oh!

Conversation

prashantwason commented Aug 7, 2021

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

hudi-bot commented Aug 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

leesf Aug 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leesf Aug 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nbalajee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbalajee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbalajee left a comment

Choose a reason for hiding this comment

Uh oh!

nbalajee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbalajee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vinothchandar left a comment

Choose a reason for hiding this comment

hudi-bot commented Aug 7, 2021 •

edited

Loading

leesf Aug 9, 2021 •

edited

Loading

leesf Aug 9, 2021 •

edited

Loading