[HUDI-6784] Support deletion logic in merger #9593

linliu-code · 2023-09-01T01:56:59Z

Change Logs

The goal of this PR is to support custom logic during data write through Merger API. Our current solution is to add Option wrapper for older and newer parameters in merge function. In such way, all of update, delete, combine logics are merged into one api. Meanwhile, we apply this merge function into HoodieMergeHandle to handle both update and insert write path.

TESTS:
Unit tests are added for existing merger implementations.

Impact

Users could implement merge api to support their own logic about deletion now. Previously deletion is not supported.

Risk level (write none, low medium or high below)

Low.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

linliu-code · 2023-09-01T01:57:23Z

@yihua @danny0405

beyond1920

Thanks for contribution. I left some comments.
There are some places missed to call new merge API.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

beyond1920 · 2023-09-01T07:22:26Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala

  // NOTE: This have to stay lazy to make sure it's initialized only at the point where it's
  //       going to be used, since we modify `logRecords` before that and therefore can't do it any earlier
-  private lazy val logRecordsIterator: Iterator[Option[HoodieRecord[_]]] =
+  private lazy val logRecordsIterator: Iterator[scala.Option[HoodieRecord[_]]] =


Why introduce scala prefix?

Because Hudi uses its own Option keyword; and scala has its own. There are two ways to differentiate them: one to add scala prefix to scala Option; one to using alias for Hudi Option, like HOption. I will probably do the later one and will update.

beyond1920 · 2023-09-01T07:30:20Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala

+  protected def removeLogRecord(key: String): scala.Option[HoodieRecord[_]] = logRecords.remove(key)

  protected def doHasNext: Boolean = hasNextInternal



We should also call merger API here to check whether the record need to be dropped before load it as result of query.

Hi @beyond1920, not quiet understand why we need to use merge here since the function tends to remove the record. Can you explain a bit?

beyond1920 · 2023-09-01T07:34:57Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala

    serializer.serialize(curRowRecord).asInstanceOf[GenericRecord]

-  private def merge(curRow: InternalRow, newRecord: HoodieRecord[_]): Option[InternalRow] = {
+  private def merge(curRow: InternalRow, newRecord: HoodieRecord[_]): scala.Option[InternalRow] = {


Don't forget to call this merge method even if updatedRecordOpt.isEmpty is true.

@beyond1920, I modified the logic in the function hasNextInternal. Please see if it works correctly.

linliu-code · 2023-09-01T22:59:12Z

@beyond1920 Thanks for the comments. I will update soon.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

yihua · 2023-09-05T21:31:14Z

...park-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java

-  public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
-    ValidationUtils.checkArgument(older.getRecordType() == HoodieRecordType.SPARK);
-    ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecordType.SPARK);
+  public Option<Pair<HoodieRecord, Schema>> merge(


As a follow-up, could you implement the logic of DefaultHoodieRecordPayload and a custom merging strategy mentioned in #9430 in new merger strategies and make sure the new merge API covers all the functionality we support with HoodieRecordPayload?

Sounds good. Will do.

yihua · 2023-09-05T21:47:07Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala

-      } else {
-       val mergedRecordOpt = merge(curRow, updatedRecordOpt.get)
-        if (mergedRecordOpt.isEmpty) {
+      val mergedRecordOpt = merge(curRow, updatedRecordOpt)


I think you still need to check updatedRecordOpt.isEmpty and if there is no update for the record key, return the record from the base file.

Will update.

The original code did that. So I will remove my change here.

yihua · 2023-09-14T00:46:23Z

@linliu-code could you check the test failure in GH actions?

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

yihua · 2023-09-14T00:54:33Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

  public void write(HoodieRecord<T> oldRecord) {
    Schema oldSchema = config.populateMetaFields() ? writeSchemaWithMetaFields : writeSchema;
    Schema newSchema = useWriterSchemaForCompaction ? writeSchemaWithMetaFields : writeSchema;
+


nit: let's avoid unnecessary cosmetic changes

Yeah, I will start another PR for these changes.

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

yihua · 2023-09-14T01:05:39Z

hudi-common/src/test/java/org/apache/hudi/model/TestHoodieAvroRecordMerger.java

+
+    for (HoodieAvroRecord record : records) {
+      Option<Pair<HoodieRecord, Schema>> r = MERGER.merge(
+          Option.empty(),


Could you also add test cases where the invalid record is represented as HoodieEmptyRecord or HoodieRecord.SENTINEL used by HoodieAvroRecord?

yihua · 2023-09-14T01:10:21Z

hudi-common/src/test/java/org/apache/hudi/model/TestHoodieAvroRecordMerger.java

+import static org.apache.hudi.model.TestUtil.SCHEMA;
+import static org.apache.hudi.model.TestUtil.generateData;
+
+public class TestHoodieAvroRecordMerger {


We should also add an functional/integration test which uses the Avro/Spark record merger to implement the same payload/merging logic as @beyond1920 mentioned and make sure that the upsert operation generates the same set of records for inserts, updates, and deletes (which is the criteria of approval on this PR), covering all new functionality of the new merge API.

I will add that in a separate PR.

yihua · 2023-09-14T01:14:36Z

...park-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java

+      Option<HoodieRecord> newerIgnored,
+      Schema newSchemaIgnored,
+      TypedProperties propsIgnored) throws IOException {
+    return Option.empty();


For the delete case, should the HoodieEmptyRecord be returned?

Either one should work, but checking option.isPresent() should be more efficient than instanceOf. Wdyt?

It depends on the merging case. When merging the records from log records with the ordering field, we have to use HoodieEmptyRecord to include the record key and ordering field to indicate that this is a delete record and used for merging with deletes/updates from other log files.

linliu-code · 2023-09-14T20:13:20Z

@linliu-code could you check the test failure in GH actions?

I could not reproduce some issues locally; e.g., some failing tests cannot be found. I think there should be some tricks to use "-pl" option in mvn test command in Hudi. Anyways, I will try to figure it out.

yihua · 2023-09-19T18:00:46Z

@linliu-code are you still resolving the test failures? Are they related to production code or just test issues?

yihua · 2023-09-19T19:54:12Z

@linliu-code could you check the test failure in GH actions?

I could not reproduce some issues locally; e.g., some failing tests cannot be found. I think there should be some tricks to use "-pl" option in mvn test command in Hudi. Anyways, I will try to figure it out.

In GH actions, some tests run in a specific Spark version so you need to check the exact setup where the test failed. Or it may be flakiness.

linliu-code · 2023-09-20T21:24:39Z

@linliu-code could you check the test failure in GH actions?

I could not reproduce some issues locally; e.g., some failing tests cannot be found. I think there should be some tricks to use "-pl" option in mvn test command in Hudi. Anyways, I will try to figure it out.

In GH actions, some tests run in a specific Spark version so you need to check the exact setup where the test failed. Or it may be flakiness.

Right, I have to check the exact mvn options to reproduce the issues. Generally I can reproduce so far; now I am still chasing these bugs.

yihua · 2023-09-21T19:10:32Z

@hudi-bot run azure

danny0405

Block on me for a final review, cc @beyond1920 to review if you have spare time.

yihua · 2023-09-22T18:11:09Z

@hudi-bot run azure

yihua · 2023-09-22T18:11:22Z

@linliu-code there's conflict with master. Could you resolve it?

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

yihua · 2023-09-22T18:15:52Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+    return writeRecord(newRecord, Option.of(processedRecord.get().getLeft()), writerSchema, config.getPayloadConfig().getProps(), isDelete);
+  }
+
+  protected boolean is_delete_record(HoodieRecord record, Schema schema, TypedProperties props) throws IOException {


Suggested change

protected boolean is_delete_record(HoodieRecord record, Schema schema, TypedProperties props) throws IOException {

protected boolean isDeleteRecord(HoodieRecord record, Schema schema, TypedProperties props) throws IOException {

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

yihua · 2023-09-22T18:33:06Z

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

+      Option<IndexedRecord> newData = newer.toIndexedRecord(schema, props).map(HoodieAvroIndexedRecord::getData);
+      return newData;


Is this a bug before?

Yeah I think this covers HoodieRecordPayload#getInsertValue because previousAvroData is not present means no such record exists on storage. Ideally, we shouldn't even be inside this if-block because the older record is empty (not valid) and hence the code will return from line 60 of merge method.

@yihua, @codope, based on the old logic, when the old record is not valid, the merge result is empty, then the merge logic in MergeHandle will try to keep the old data, that is, the invalid old data will be written to the disk again. So I updated it as: when the old data is invalid, but the new one is valid, we should output the new data; the mergeHandle will check if the new data is a delete or not.

hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePreCombineAvroRecordMerger.java

yihua · 2023-09-22T18:51:09Z

Pasting my deep-diving of the code based on release 0.12.2 (pre-landing of RFC-46 PR #7345 ) here. Here is how different APIs, #preCombine, #combineAndGetUpdateValue, and #getInsertValue, of HoodieRecordPayload, responsible for handling record-level inserts, updates, and deletes, are used:

T preCombine(T oldValue, Properties props): deduplicate the records within the same batch, or merge records from the log files in the same file slice, for the same record key
- HoodieWriteHelper#deduplicateRecords: deduplicate records of the same record key within the same batch from the data source (similar for Flink and Java write helpers)
- HoodieMergedLogRecordScanner#processNextRecord: when scanning records from the log files in the same file slice, merge the records of the same record key from the log files and save the result in a spillable map containing the mapping of the record key to record mapping
- HoodieBackedTableMetadata#readFromBaseAndMergeWithLogRecords: merging records of the same record key from log files for the metadata table
Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties): merge the update record with the record from base file
- HoodieMergeHandle.write: Merge new record to the one on storage
- HoodieMergeOnReadRDD.ecordMergingFileIterator.merge: merge record from the base file with the record from the log files (results from the merged log scanner) for MOR in spark reader
- MergeOnReadInputFormat.MergeIterator.mergeRowWithLog (Flink): merge record from base file with the record from log files (after merged log scanner)
Option<IndexedRecord> getInsertValue(Schema schema, Properties properties): get record to insert when there is no existing record on storage to combine with
- HoodieFileSliceReader.getFileSliceReader: transform records from the log files with getInsertValue
- HoodieLazyInsertIterable , HoodieAppendHandle , HoodieCreateHandle , HoodieMergeHandle , HoodieSortedMergeHandle: get the insert value before writing
- ColumnStatsIndices.readColumnStatsIndexByColumns , ColumnStatsIndexSupport: get insert value of col stats
- RealtimeCompactedRecordReader , RealtimeUnmergedRecordReader: get records from the log files
- HoodieMergeOnReadRDD: transform records read from the log files with getInsertValue
Comparable<?> getOrderingValue() : returns ordering value
- HoodieAppendHandle.writeToBuffer: get the ordering value for delete records
- HoodieMergedLogRecordScanner.processNextDeletedRecord: get the ordering value for processing deletes
Option<Map<String, String>> getMetadata(): returns record metadata, used by HoodieAppendHandle, HoodieCreateHandle, HoodieMergeHandle, and HoodieWriteHandle

yihua · 2023-09-22T19:00:12Z

To be fully compatible with before, the places where HoodieRecordPayload APIs (#preCombine, #combineAndGetUpdateValue, and #getInsertValue) are called should call merge API too.

To make sure the handling of inserts, updates, and deletes are correct, here are a few scenarios you should test end-to-end with spark datasource read:

Base file only
Base file + one log file with updates (Avro type for default record and merger, and parquet type for Spark record and merger)
Base file + one log file with deletes (Avro type for default record and merger, and parquet type for Spark record and merger)
Base file + two log files with updates on the same record key, different precombine field ordering (Avro type for default record and merger, and parquet type for Spark record and merger, merging logic: overwrite with latest, event-time based ordering)
Base file + two log file with deletes on the same record key, different precombine field ordering (Default record and merger, Spark record and merger, merging logic: overwrite with latest, event-time based ordering)
Base file + one log file with updates + one log file with deletes + one log file with updates, on the same record key, different precombine field ordering (Avro type for default record and merger, and parquet type for Spark record and merger, merging logic: overwrite with latest, event-time based ordering)

The goal is to make sure all complex logic around deletes and updates is handled correctly now. Note that some of these scenarios may not be covered by unit/functional tests.

linliu-code · 2023-09-22T19:28:21Z

Ok, working on the comments now.

danny0405 · 2023-09-27T04:01:23Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+    return record.isDelete(schema, props)
+        || record instanceof HoodieEmptyRecord
+        || (record.getData() != null && record.getData() instanceof EmptyHoodieRecordPayload)
+        || HoodieOperation.isDelete(record.getOperation());


What are the gains we bind all these check together ? Empty record and delete record are meant to be dropped but are they handled in the same logic originally?

Based on my understanding, Hudi uses both empty record and delete record interchangeably to mean delete. So I combine here. I will modify this PR based on our discussion so this logic should be gone. But we should find a way to standardize these logics; otherwise, the scattered logic is really confusing.

Revert all the changes to HoodieMergeHandle, then add a new interface to the Merger API:

/** * Checks the merged record valility before flushing into dist, if returns false, the given record would be ignored. * In some scenarios, the bussiness logic needs to check the validity of the merged record, so this interface give * a chance for the user to do a sanity check. * * <p> This interface is experimental and might got evolved in the future. **/ @Experimental default boolean isValid(HoodieRecord record, Schema schema) { return true; }

This interface would be invoked before each merged record flushing. Only merged record needs this check currently.

danny0405

I kind of feel the isDelete check is now scattered every where which is error prone to maintain, and the complexity are all handled over to the merger API, if you take a look at the original HoodieAvroRecordMerger it looks much simpler, so can we keep that as it is now?

It just does not make sense we check for validity in each merge call, a separate interface like #insert makes more sense to me.

Remove the Option.of from the param or add a new interface.

codope · 2023-09-28T05:06:55Z

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java

   * of the single record, both orders of operations applications have to yield the same result)
   */
-  Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+  Option<Pair<HoodieRecord, Schema>> merge(Option<HoodieRecord> older, Schema oldSchema, Option<HoodieRecord> newer, Schema newSchema, TypedProperties props) throws IOException;


Let's update the javadoc to define the parameters as well.

Will revert the change here, but will add the javadoc.

codope · 2023-09-28T05:19:19Z

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

+  /**
+   * Check if a DELETE operation is intended.
+   */
+  private boolean isDelete(Option<HoodieRecord> record, Schema schema, TypedProperties props) throws IOException {


Can we move this method and the above to a suitable util class in hudi-common, or maybe define s static methods in HoodieRecordMerger interface? We can reuse across mergers and/or write handle as the implementation does not differ.

When we add "insert" method in the interface, the logic of the merger will be as simple as before. So we don't need these checks at all.

codope · 2023-09-28T05:55:01Z

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

+      Option<IndexedRecord> newData = newer.toIndexedRecord(schema, props).map(HoodieAvroIndexedRecord::getData);
+      return newData;


Yeah I think this covers HoodieRecordPayload#getInsertValue because previousAvroData is not present means no such record exists on storage. Ideally, we shouldn't even be inside this if-block because the older record is empty (not valid) and hence the code will return from line 60 of merge method.

codope · 2023-09-28T06:50:51Z

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java

+    if (!isValidOld && !isValidNew) { // No meaningful information found.
+      return Option.empty();
+    } else if (isValidOld && !isValidNew) { // Return old record for data safety.
+      return Option.of(Pair.of(older.get(), oldSchema));
+    } else if (!isValidOld) { // Either insert or delete case, return the new record.
+      return Option.of(Pair.of(newer.get(), newSchema));
+    } else {
+      if (isDeleteNew) { // delete case
+        return Option.of(Pair.of(newer.get(), newSchema));
+      }


I guess this is all the complication that @danny0405 mentioned. I think we should still check for validity before calling combineAndGetUpdateValue. However, we can keep the handling of indert/delete to a separate method that can be overridden. Default would be to return the new record.

IMO, one method is cleaner and easy-to-use. Also, I checked what Kafka and Flink do. They also provide one method and leave the handling of empty/null keys to the concrete implementation.
Kafka Merger - https://kafka.apache.org/35/javadoc/org/apache/kafka/streams/kstream/Merger.html
Flink InternalRowMerger - https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/org/apache/flink/table/examples/java/functions/InternalRowMergerFunction.html

@danny0405 What do you think?

If you look at the APIs from Kafka and Flink, they either uses a T or specific RowData there, and I guess both of the old and new values are not nullable, but take a look of our API:

we make it as option so the user needs to take care whether they are empty and they even needs to disinduish UPSERT and DELETES, which is totally unnecessary.

We have a HoodieRecord there, but there is many specific logic for hadling deletes inside Hudi which should not explose to user totally.

We have many ramifications for write paths for different representations of delete HoodieRecord, like a HoodieRecord with D as operation, a HoodieAvroEmptyRecord, a IGNORE record, I don't think any normal user should see that complexity, even we make a default impls there, the user still needs to undertand it and have a good comprehension to use it correctly.

danny0405 · 2023-09-29T01:14:48Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java

+        // Inject custom logic for the record.
+        Option<Pair<HoodieRecord, Schema>> processedRecord = config.getRecordMerger().insert(record, schema, config.getProps());
+        if (!processedRecord.isPresent()
+            || HoodieOperation.isDelete(processedRecord.get().getLeft().getOperation())


We can revert all the changes to create handle, the delete messages already got handled in line 137.

Should we allow the custom logic?

danny0405 · 2023-09-29T01:21:30Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+    return record.isDelete(schema, props)
+        || record instanceof HoodieEmptyRecord
+        || (record.getData() != null && record.getData() instanceof EmptyHoodieRecordPayload)
+        || HoodieOperation.isDelete(record.getOperation());


Revert all the changes to HoodieMergeHandle, then add a new interface to the Merger API:

/** * Checks the merged record valility before flushing into dist, if returns false, the given record would be ignored. * In some scenarios, the bussiness logic needs to check the validity of the merged record, so this interface give * a chance for the user to do a sanity check. * * <p> This interface is experimental and might got evolved in the future. **/ @Experimental default boolean isValid(HoodieRecord record, Schema schema) { return true; }

This interface would be invoked before each merged record flushing. Only merged record needs this check currently.

The solution is (1) to add a new interface method called "insert" in the HoodieRecordMerger to support custom logic. (2) to apply merge function in create/merge handles to enable potential custom logic. (3) to add various unit tests to confirm the current merger api works.

hudi-bot · 2023-09-29T11:05:21Z

CI report:

f9651ab Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

beyond1920 reviewed Sep 1, 2023

View reviewed changes

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from f906f3b to d1fa8c6 Compare September 1, 2023 14:03

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from d1fa8c6 to 6d4fb04 Compare September 5, 2023 18:21

yihua reviewed Sep 5, 2023

View reviewed changes

linliu-code force-pushed the add_optional_parameters_for_merge_api branch 4 times, most recently from b8c3581 to 1fa522e Compare September 7, 2023 22:20

linliu-code mentioned this pull request Sep 8, 2023

[HUDI-6702] Remove unnecessary calls of getInsertValue api from HoodieRecordPayload #9572

Closed

4 tasks

linliu-code force-pushed the add_optional_parameters_for_merge_api branch 2 times, most recently from 2c341b9 to 6f810f4 Compare September 11, 2023 20:15

linliu-code requested review from beyond1920, danny0405 and yihua September 11, 2023 20:25

yihua self-assigned this Sep 14, 2023

yihua reviewed Sep 14, 2023

View reviewed changes

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from 6f810f4 to 05d9424 Compare September 14, 2023 20:16

linliu-code requested a review from yihua September 14, 2023 20:19

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from 05d9424 to bb4e1c9 Compare September 20, 2023 21:20

linliu-code force-pushed the add_optional_parameters_for_merge_api branch 2 times, most recently from 8b1fb2c to d98bb35 Compare September 21, 2023 08:44

danny0405 requested changes Sep 22, 2023

View reviewed changes

yihua changed the title ~~[HUDI-6784][RFC-46] Support deletion logic in merger~~ [HUDI-6784] Support deletion logic in merger Sep 22, 2023

yihua reviewed Sep 22, 2023

View reviewed changes

danny0405 reviewed Sep 27, 2023

View reviewed changes

danny0405 requested changes Sep 27, 2023

View reviewed changes

codope reviewed Sep 28, 2023

View reviewed changes

linliu-code force-pushed the add_optional_parameters_for_merge_api branch 3 times, most recently from 88a73cb to f85abc4 Compare September 29, 2023 00:42

danny0405 reviewed Sep 29, 2023

View reviewed changes

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from f85abc4 to 4f59993 Compare September 29, 2023 06:17

linliu-code force-pushed the add_optional_parameters_for_merge_api branch from 4f59993 to f9651ab Compare September 29, 2023 08:17

linliu-code closed this Oct 6, 2023

		protected def removeLogRecord(key: String): scala.Option[HoodieRecord[_]] = logRecords.remove(key)

		protected def doHasNext: Boolean = hasNextInternal

	protected boolean is_delete_record(HoodieRecord record, Schema schema, TypedProperties props) throws IOException {
	protected boolean isDeleteRecord(HoodieRecord record, Schema schema, TypedProperties props) throws IOException {

		Option<IndexedRecord> newData = newer.toIndexedRecord(schema, props).map(HoodieAvroIndexedRecord::getData);
		return newData;

[HUDI-6784] Support deletion logic in merger #9593

[HUDI-6784] Support deletion logic in merger #9593

Uh oh!

Conversation

linliu-code commented Sep 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Contributor's checklist

Uh oh!

linliu-code commented Sep 1, 2023

Uh oh!

beyond1920 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linliu-code commented Sep 1, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua commented Sep 14, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linliu-code commented Sep 14, 2023

Uh oh!

yihua commented Sep 19, 2023

Uh oh!

yihua commented Sep 19, 2023

Uh oh!

linliu-code commented Sep 20, 2023

Uh oh!

yihua commented Sep 21, 2023

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

yihua commented Sep 22, 2023

linliu-code commented Sep 1, 2023 •

edited

Loading

linliu-code Sep 28, 2023 •

edited

Loading

linliu-code Sep 28, 2023 •

edited

Loading