[HUDI-3478] Support CDC for Spark in Hudi #6476

YannByron · 2022-08-23T04:25:20Z

Change Logs

This PR is going to support CDC on Spark.

The changes are listed:

add a new field named cdcPath in HoodieWriteStat to hold the cdc file.
upgrade HooideMergeHandle and its sub-classes to persist the cdc data to the cdc log file if needed.
extract some cdc common classes in hudi-common that will be used to implement CDCReader for different engines;
CDCReader to response the CDC query on spark.

Impact

Low

XuQianJin-Stars · 2022-08-23T10:44:02Z

nice work!

YannByron · 2022-08-23T14:21:58Z

@prasannarajaperumal @xushiyan
This pr has been updated according to the updated RFC: #6256
please help to review this.

xushiyan

will review a few more rounds. can you please also populate the PR description as per the PR template

...cala/org/apache/spark/sql/execution/datasources/parquet/Spark31HoodieParquetFileFormat.scala

...-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCBase.scala

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieSortedMergeHandle.java

hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java

hudi-common/src/main/java/org/apache/hudi/common/table/cdc/CDCFileTypeEnum.java

hudi-common/src/main/java/org/apache/hudi/common/table/cdc/CDCOperationEnum.java

xushiyan · 2022-08-29T07:20:27Z

hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordReader.java

        totalLogBlocks.incrementAndGet();
+        if (logBlock.getBlockType() == CDC_DATA_BLOCK) {
+          // hit a cdc block, just skip.
+          continue;


get instant time above should be skipped if skipping here. also, should totalLogBlocks increment? pls confirm

according to the following logic, even the block doesn't be read, totalLogBlocks also should increment.

hudi-common/src/main/java/org/apache/hudi/common/table/log/CDCLogRecordReader.java

YannByron · 2022-08-31T02:22:07Z

@xushiyan have updated this, and solved the comments above. pls review again.

prasannarajaperumal

@YannByron - Thanks for making the implementation changes. I made an initial pass and suggested some changes.

I really would like this PR to be split into 2 parts

All changes related to writing the CDC log blocks, We will try to land this first. When I enable CDC and set a logging mode - the CDC data is written but its not queryable yet.
Changes related to constructing the CDC stream (Identifying cdc splits)

Doing 1 as the first step will make the abstractions we want to have clear in the writing path. For e.g. I am not too convinced about moving the log writer code into HoodieMergeHandle, I am still thinking what an alternative here would be.

Lets please split this PR and we can discuss more there.

Does that make sense? @YannByron
cc fyi @xushiyan @vinothchandar @yihua

prasannarajaperumal · 2022-09-05T05:27:25Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+  protected HoodieLogFormat.Writer cdcWriter;
+  // the cdc data
+  protected Map<String, SerializableRecord> cdcData;
+  //


Nit: doc missing

yep, miss it.

hudi-common/src/main/java/org/apache/hudi/avro/SerializableRecord.java

prasannarajaperumal · 2022-09-05T06:13:47Z

hudi-common/src/main/java/org/apache/hudi/common/table/cdc/CDCExtractor.java

+import static org.apache.hudi.common.table.timeline.HoodieTimeline.isInRange;
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.REPLACE_COMMIT_ACTION;
+
+public class CDCExtractor {


Please add Java doc

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieCommitMetadata.java

prasannarajaperumal · 2022-09-05T06:22:27Z

hudi-common/src/main/java/org/apache/hudi/common/table/cdc/CDCExtractor.java

+    if (StringUtils.isNullOrEmpty(writeStat.getCdcPath())) {
+      // no cdc log files can be used directly. we reuse the existing data file to retrieve the change data.
+      String path = writeStat.getPath();
+      if (path.endsWith(HoodieFileFormat.PARQUET.getFileExtension())) {


Not extensible to other file types

let me upgrade this to support other base file types.

to support other base file types like orc in spark is blocked by https://issues.apache.org/jira/browse/HUDI-4496.

YannByron · 2022-09-05T15:23:51Z

@prasannarajaperumal Thank you for reviewing this pr.

maybe I prefer to one single pr. i think these changes are easy to judgement which related to write or read. and one pr is better to run the whole process tests.

I am not too convinced about moving the log writer code into HoodieMergeHandle, I am still thinking what an alternative here would be.

One key reason that put the log writer code in HoodieMergeHandler is that: persisting CDC data need to get the old value and the current value, these values can be get directly in HoodieMergeHandler without any other overhead.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

xushiyan · 2022-09-08T00:55:30Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+                                         GenericRecord oldRecord, GenericRecord newRecord) {
+    GenericData.Record record;
+    if (cdcSupplementalLoggingMode.equals(HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE_WITH_BEFORE_AFTER)) {
+      record = CDCUtils.cdcRecord(operation.getValue(), instantTime,


can we prefix classes with Hoodie? like HoodieCDCUtils , which is the convention in the codebase

xushiyan · 2022-09-08T01:02:15Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+  protected Option<AppendResult> writeCDCData() {
+    if (!cdcEnabled || cdcData.isEmpty() || recordsWritten == 0L || (recordsWritten == insertRecordsWritten)) {
+      // the following cases where we do not need to write out the cdc file:
+      // case 1: all the data from the previous file slice are deleted. and no new data is inserted;
+      // case 2: all the data are new-coming,
+      return Option.empty();
+    }


can we encapsulate CDC writer logic within dedicated CDC writer as much as possible? the concern here is we don't want to spread the condition checkings across different classes, which would be hard to maintain, in case any of these assumption changed in future. Grouping all the CDC logic together really helps with evolving/maintenance in future

see: #5885 (comment)

If inheritance hierarchy is complicated, lets us not do that. Instead can we create a HoodieCDCLogger as a helper class and move all code related to writing CDC data into this abstraction. You can just initialize a HoodieCDCLogger from HoodieMergeHandle?

Would that work?

I think that should work. I will do this asap.

Second that: CDC is non-trivial logic that will be proliferated across Writing Handles, therefore we should encapsulate its state along w/ well-defined API w/in single component

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieSortedMergeHandle.java

xushiyan · 2022-09-08T01:10:29Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java

+    this.keyField = config.populateMetaFields() ? HoodieRecord.RECORD_KEY_METADATA_FIELD
+        : hoodieTable.getMetaClient().getTableConfig().getRecordKeyFieldProp();


this could be multiple keys that user configured. we might need a keyFields as a list

hudi-common/src/main/java/org/apache/hudi/common/table/log/CDCLogRecordReader.java

hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieCDCDataBlock.java

xushiyan · 2022-09-08T01:38:18Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java

+  public List<HoodieInstant> getInstantsAsList() {
+    return instants;


should have a clean up ticket to call this getInstants and the other getInstantsAsStream. Public API names should not cause confusion

Where's this method used? Can't find any usage

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java

xushiyan · 2022-09-08T01:42:20Z

...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala

-    reusableRecordBuilder.build()
-  }
-
-  private def getPartitionPath(split: HoodieMergeOnReadFileSplit): Path = {


can you clarify this big chunk of code removal? a result of refactoring? if that's the case, let's keep change minimal for CDC-related. try to achieve the same functionalities without moving code around. note that there can be other refactoring work going on around this and conflicts can be tricky

HoodieCDCRDD need to use LogFileIterator and RecordMergingFileIterator to extract the cdc data from the log files of MOR tables. So I just extend the arguments of the construct method to let them can be reused and move them to the common place. That's all.

YannByron · 2022-09-09T09:44:37Z

@xushiyan @prasannarajaperumal update pr to solve the comments above. please continue to review, thanks.

YannByron · 2022-09-09T16:11:33Z

@hudi-bot run azure

prasannarajaperumal

I made a pass. Looks good to me except for a small abstraction comment. Let us make sure we create jiras for all follow up work and leave a TODO comment in code. This is a big feature and thanks for driving this.

prasannarajaperumal · 2022-09-13T09:09:56Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+  protected Option<AppendResult> writeCDCData() {
+    if (!cdcEnabled || cdcData.isEmpty() || recordsWritten == 0L || (recordsWritten == insertRecordsWritten)) {
+      // the following cases where we do not need to write out the cdc file:
+      // case 1: all the data from the previous file slice are deleted. and no new data is inserted;
+      // case 2: all the data are new-coming,
+      return Option.empty();
+    }


If inheritance hierarchy is complicated, lets us not do that. Instead can we create a HoodieCDCLogger as a helper class and move all code related to writing CDC data into this abstraction. You can just initialize a HoodieCDCLogger from HoodieMergeHandle?

Would that work?

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieCommitMetadata.java

…gging

alexeykudinkin · 2022-09-14T03:10:08Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

-    return writeRecord(hoodieRecord, indexedRecord, isDelete);
+    boolean result = writeRecord(hoodieRecord, indexedRecord, isDelete);
+    if (cdcEnabled) {
+      String recordKey = StringUtils.join(


This might not be matching the record key. We should be using proper record-key in here.

this is a custom key just for cdc when there is more that one record key which users provide.
and this key will be parsed and used in HoodieCDCRDD.

alexeykudinkin · 2022-09-14T03:10:59Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

  protected HoodieFileWriter<IndexedRecord> fileWriter;
+  // a flag that indicate whether allow the change data to write out a cdc log file.
+  protected boolean cdcEnabled = false;
+  protected String cdcSupplementalLoggingMode;


Why is this a String?

do you suggest that should use an enum?

Was reviewing this before i caught up w/ an updated version of the RFC so got confused.
Yeah, let's use enum for this one

alexeykudinkin · 2022-09-14T03:21:43Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+    return new SerializableRecord(record);
+  }
+
+  protected GenericRecord addCommitMetadata(GenericRecord record, String recordKey, String partitionPath) {


Why are we adding the metadata to CDC payload?

if the before value or after value is persisted in CDC Block, we want to make the cdc record with the all fields.

Meta fields carry purely semantical information related to their persistence by Hudi.
These aren't the part of the record's payload and we shouldn't be carrying them w/in CDC payload.

that makes sense. so we don't add the meta fields in it, and should remove meta fields from the old record. has updated in this pr and the separate writing-related pr.

alexeykudinkin · 2022-09-14T03:28:12Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+  protected Option<AppendResult> writeCDCData() {
+    if (!cdcEnabled || cdcData.isEmpty() || recordsWritten == 0L || (recordsWritten == insertRecordsWritten)) {
+      // the following cases where we do not need to write out the cdc file:
+      // case 1: all the data from the previous file slice are deleted. and no new data is inserted;
+      // case 2: all the data are new-coming,
+      return Option.empty();
+    }


Second that: CDC is non-trivial logic that will be proliferated across Writing Handles, therefore we should encapsulate its state along w/ well-defined API w/in single component

alexeykudinkin · 2022-09-14T03:29:26Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

+          : hoodieTable.getMetaClient().getTableConfig().getRecordKeyFieldProp();
+      Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
+      header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, instantTime);
+      if (cdcSupplementalLoggingMode.equals(HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE_WITH_BEFORE_AFTER)) {


Let's abstract this conditional as a method (i believe you already have one below)

alexeykudinkin · 2022-09-14T03:42:49Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java

+        .withFs(fs)
+        .withRolloverLogWriteToken(writeToken)
+        .withLogWriteToken(logWriteToken)
+        .withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();


Are we mixing CDCBlocks w/ normal Delta Data Blocks?

I don't think we can do that as this will severely affect performance for pure non-CDC queries for MOR tables

the cdc log file with CDCBlock is a normal log file, same with others. But when read mor in non-cdc mode, we will skip the cdc block directly only with the cost of reading the first few bytes from InputStream of the log files.

So one of the pre-requisites of the CDC is:

When we're issuing normal Data query (and not a CDC one), there should be no performance impact to it

Moreover, we should clearly disambiguate the CDC infra from the Data infra w/o the need to even fetch the first block of the file (we can still use the same Log format, but we should definitely create separate naming scheme for CDC Log files to not mix these up w/ the Data Delta Log files)

alexeykudinkin

@YannByron this is gigantic effort. Thanks for taking this forward!

I'd really like us to split this PR in 2 parts to accelerate our convergence and make sure we can iterate and move forward faster -- it's already 4k LOC and it's quite hard to digest all o it at once, let alone iterate on it.

It seems very natural to me to split it up into

CDC writing part
CDC reading part

WDYT?

alexeykudinkin · 2022-09-14T22:53:03Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCDCLogger.java

+      if (oldRecord == null) {
+        recordKey = hoodieRecord.getRecordKey();
+      } else {
+        recordKey = StringUtils.join(


Please check my previous comment that we have to use proper KeyGen here

has fixed this. use KeyGen.

alexeykudinkin · 2022-09-14T22:56:41Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

      fileWriter.writeAvroWithMetadata(key, rewriteRecord(avroRecord));
    }
+    if (cdcEnabled) {
+      cdcLogger.getAndIncrement();


This should not be needed

alexeykudinkin · 2022-09-14T23:00:21Z

hudi-common/src/main/java/org/apache/hudi/avro/SerializableRecord.java

+ *
+ * This class wraps [[GenericData.Record]].
+ */
+public class SerializableRecord implements Serializable {


This is not going to work -- Avro's Schema is not serializable by Java's default serialization protocol

The serialization of external map is done by Kryo here. Is there a Serializer registered for Avro Schema with kyro?. I dont see the test for this with ExternalSpillableMap anymore - @YannByron - Can you check?

i will use HoodieAvroPayload as the value type of ExternalSpillableMap. This pr has updated but hasn't remove the SerializableRecord code.
and i am going to split this pr to two according to your suggestion. @alexeykudinkin @prasannarajaperumal

alexeykudinkin · 2022-09-14T23:03:23Z

hudi-common/src/main/java/org/apache/hudi/common/table/cdc/HoodieCDCUtils.java

+   */
+  public static GenericData.Record cdcRecord(
+      String op, String commitTime, GenericRecord before, GenericRecord after) {
+    String beforeJsonStr = recordToJson(before);


Why are we using JSON and not Avro for ex?

alexeykudinkin · 2022-09-14T23:23:49Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCDCLogger.java

+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class HoodieCDCLogger<T extends HoodieRecordPayload> implements Closeable {


I don't think we need to parameterize this by HoodieRecordPayload

yes. updated.

alexeykudinkin · 2022-09-14T23:46:13Z

hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieCDCLogRecordReader.java

+
+  @Override
+  public boolean hasNext() {
+    if (null == itr || !itr.hasNext()) {


Let's keep code style consistent (itr == null)

alexeykudinkin · 2022-09-14T23:48:59Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java

+  public List<HoodieInstant> getInstantsAsList() {
+    return instants;


Where's this method used? Can't find any usage

alexeykudinkin · 2022-09-14T23:53:05Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala

-      }
-    (shortName(), sqlSchema)
+
+    if (CDCRelation.isCDCTable(metaClient) &&


isCDCEnabled ("CDCTable" reference is confusing)

Let's also limit this conditional to fetching schema:

val schema = if (cdc) { // cdc-schema } else { // ... }

alexeykudinkin · 2022-09-14T23:53:24Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala

-    (shortName(), sqlSchema)
+
+    if (CDCRelation.isCDCTable(metaClient) &&
+        parameters.get(QUERY_TYPE.key).contains(QUERY_TYPE_INCREMENTAL_OPT_VAL) &&


We can re-use isCdcQuery

alexeykudinkin · 2022-09-14T23:54:39Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/EmptyRelation.scala

 * @param sqlContext instance of SqlContext.
 */
-class EmptyRelation(val sqlContext: SQLContext, metaClient: HoodieTableMetaClient) extends BaseRelation with TableScan {
+class EmptyRelation(val sqlContext: SQLContext, metaClient: HoodieTableMetaClient, isCDCQuery: Boolean) extends BaseRelation with TableScan {


Instead, let's modify EmptyRelation to accept schema

hudi-bot · 2022-09-15T16:37:22Z

CI report:

5e320dd Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

xushiyan · 2022-09-26T01:10:37Z

Split into read and write PRs

YannByron mentioned this pull request Aug 23, 2022

[HUDI-3478] Support CDC for Spark in Hudi #5885

Closed

5 tasks

YannByron changed the title ~~[RFC-51][HUDI-3478] Support CDC for Spark in Hudi~~ [HUDI-3478] Support CDC for Spark in Hudi Aug 23, 2022

YannByron force-pushed the master_cdc_code branch from 1fd639d to 0bc3211 Compare August 23, 2022 08:50

xushiyan assigned xushiyan and prasannarajaperumal Aug 26, 2022

xushiyan added the priority:blocker Production down; release blocker label Aug 26, 2022

xushiyan reviewed Aug 29, 2022

View reviewed changes

prasannarajaperumal requested changes Sep 5, 2022

View reviewed changes

yihua added the big-needle-movers label Sep 6, 2022

xushiyan reviewed Sep 8, 2022

View reviewed changes

YannByron force-pushed the master_cdc_code branch 2 times, most recently from afde1d3 to 2de30cf Compare September 9, 2022 09:27

YannByron force-pushed the master_cdc_code branch from 2de30cf to 6bbe3fb Compare September 10, 2022 08:11

prasannarajaperumal approved these changes Sep 13, 2022

View reviewed changes

YannByron added 10 commits September 14, 2022 09:58

[RFC-51][HUDI-3478] Hudi CDC

3a7ecca

update

d0903c6

[RFC-51][HUDI-3478] support the case that disable cdc supplemental lo…

399b3f5

…gging

update

0c987c5

solve comments

adb1a8d

update spark3.3 avro serde

bcdba4a

fix streaming ut

d28c42e

cdc quey type -> incremental query table + cdc output format

9e4a620

support three supplemental.logging.modes

9c4b0b7

rename configs

0286c11

YannByron added 3 commits September 14, 2022 09:58

rebase master

f303178

solve comments

853a59b

solve comments

ff8f82f

alexeykudinkin reviewed Sep 14, 2022

View reviewed changes

YannByron force-pushed the master_cdc_code branch from 6bbe3fb to 621d2bf Compare September 14, 2022 07:52

abstract HoodieCDCLogger and leave a TODO to link jira

34088ae

YannByron force-pushed the master_cdc_code branch from 621d2bf to 34088ae Compare September 14, 2022 14:31

alexeykudinkin reviewed Sep 15, 2022

View reviewed changes

YannByron added 6 commits September 15, 2022 16:35

remove SerializableRecord

dfe4136

remove unused code

0947d8c

enum HoodieCDCSupplementalLoggingMode

a274deb

update

c24f150

remove meta fields for cdc data

eef1ba6

update

5e320dd

YannByron force-pushed the master_cdc_code branch from 6fb2368 to 5e320dd Compare September 15, 2022 14:11

This was referenced Sep 20, 2022

[HUDI-3478] Implement CDC Write in Spark #6697

Merged

[HUDI-3478] Implement CDC Read in Spark #6727

Merged

xushiyan closed this Sep 26, 2022

		this.keyField = config.populateMetaFields() ? HoodieRecord.RECORD_KEY_METADATA_FIELD
		: hoodieTable.getMetaClient().getTableConfig().getRecordKeyFieldProp();

		public List<HoodieInstant> getInstantsAsList() {
		return instants;

[HUDI-3478] Support CDC for Spark in Hudi #6476

[HUDI-3478] Support CDC for Spark in Hudi #6476

Uh oh!

Conversation

YannByron commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Uh oh!

XuQianJin-Stars commented Aug 23, 2022

Uh oh!

YannByron commented Aug 23, 2022

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

YannByron commented Aug 31, 2022

Uh oh!

prasannarajaperumal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

prasannarajaperumal Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YannByron commented Sep 5, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YannByron commented Aug 23, 2022 •

edited

Loading

prasannarajaperumal Sep 5, 2022 •

edited

Loading

prasannarajaperumal left a comment •

edited

Loading