[HUDI-7486] Classify schema exceptions when converting from avro to spark row representation #10778

jonvex · 2024-02-28T21:59:59Z

Change Logs

All issues related to schema should throw exceptions that are HoodieSchemaExceptions. Classify exceptions when converting from avro to spark row format as schema compatibility exceptions because they are due to illegal schema, or the records are incompatible with the provided schema.

Additionally, introduce a new exception type HoodieRecordCreationException which is thrown when turning an engine specific record into a HoodieRecord

Impact

Exceptions can be classified easier

Risk level (write none, low medium or high below)

low

Documentation Update

N/A

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java

...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java

hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala

hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java

yihua · 2024-03-27T21:49:55Z

hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala


+  def maybeWrapDataFrameWithException(df: DataFrame, exceptionClass: String, msg: String, shouldWrap: Boolean): DataFrame = {
+    if (shouldWrap) {
+      HoodieUnsafeUtils.createDataFrameFromRDD(df.sparkSession, injectSQLConf(df.queryExecution.toRdd.mapPartitions {


Is SQLConf injection necessary here?

#6352 alexey says that "Yes, it will propagate to all RDDs in the execution chain (up to a shuffling point)". So I guess the question is if there is a shuffling point?

My doubt is if wrapping the df with exception needs the SQL conf injection. So you're saying this needs SQL conf injection so the configs can still be propagated to the executors.

Let's keep an eye on the DAG and Spark execution when this is rolled out. I think this is OK for now.

...client/hudi-spark-client/src/main/scala/org/apache/hudi/util/ExceptionWrappingIterator.scala

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java

hudi-utilities/src/main/java/org/apache/hudi/utilities/config/HoodieStreamerConfig.java

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/RowSource.java

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SourceFormatAdapter.java

hudi-utilities/src/main/java/org/apache/hudi/utilities/config/HoodieStreamerConfig.java

yihua · 2024-03-28T17:30:40Z

hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala


+  def maybeWrapDataFrameWithException(df: DataFrame, exceptionClass: String, msg: String, shouldWrap: Boolean): DataFrame = {
+    if (shouldWrap) {
+      HoodieUnsafeUtils.createDataFrameFromRDD(df.sparkSession, injectSQLConf(df.queryExecution.toRdd.mapPartitions {


Let's keep an eye on the DAG and Spark execution when this is rolled out. I think this is OK for now.

yihua

Block merging for a couple of comments on docs to address

yihua · 2024-04-02T23:17:00Z

...client/hudi-spark-client/src/main/scala/org/apache/hudi/util/ExceptionWrappingIterator.scala

+package org.apache.hudi.util
+
+import org.apache.hudi.common.util.ReflectionUtils
+


hudi-common/src/main/java/org/apache/hudi/exception/HoodieRecordCreationException.java

hudi-common/src/main/java/org/apache/hudi/exception/HoodieAvroSchemaException.java

…rdCreationException.java Co-authored-by: Y Ethan Guo <[email protected]>

…SchemaException.java Co-authored-by: Y Ethan Guo <[email protected]>

hudi-bot · 2024-04-03T04:00:11Z

CI report:

1770608 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

yihua

LGTM

…park row representation (#10778) * make exceptions more specific * use hudi avro exception * Address review comments * fix unnecessary changes * add exception wrapping * style * address review comments * remove . from config * address review comments * fix merge * fix checkstyle * Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieRecordCreationException.java Co-authored-by: Y Ethan Guo <[email protected]> * Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieAvroSchemaException.java Co-authored-by: Y Ethan Guo <[email protected]> * add javadoc to exception wrapper --------- Co-authored-by: Jonathan Vexler <=> Co-authored-by: Y Ethan Guo <[email protected]>

make exceptions more specific

5916630

github-actions bot added the size:M PR with lines of changes in (100, 300] label Feb 28, 2024

Merge branch 'master' into classify_record_rewrite_exceptions

63c8182

jonvex changed the title ~~make exceptions more specific~~ Classify schema exceptions when converting from avro to spark row representation Mar 6, 2024

jonvex changed the title ~~Classify schema exceptions when converting from avro to spark row representation~~ [HUDI-7486] Classify schema exceptions when converting from avro to spark row representation Mar 6, 2024

jonvex marked this pull request as ready for review March 6, 2024 22:09

danny0405 reviewed Mar 7, 2024

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java Outdated Show resolved Hide resolved

use hudi avro exception

a5ccaf5

danny0405 approved these changes Mar 8, 2024

View reviewed changes

danny0405 self-assigned this Mar 8, 2024

danny0405 added the area:sql SQL interfaces label Mar 8, 2024

apache deleted a comment from hudi-bot Mar 11, 2024

yihua requested changes Mar 11, 2024

View reviewed changes

...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala Outdated Show resolved Hide resolved

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java Outdated Show resolved Hide resolved

Address review comments

ce3b208

jonvex requested a review from yihua March 13, 2024 18:48

danny0405 reviewed Mar 14, 2024

View reviewed changes

hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala Show resolved Hide resolved

fix unnecessary changes

5808ef2

yihua reviewed Mar 18, 2024

View reviewed changes

jonvex and others added 3 commits March 27, 2024 09:38

Merge branch 'apache:master' into classify_record_rewrite_exceptions

d12ec39

add exception wrapping

6369eb5

style

42d6607

jonvex requested a review from yihua March 27, 2024 16:18

yihua reviewed Mar 27, 2024

View reviewed changes

address review comments

90a3006

jonvex requested a review from yihua March 28, 2024 16:56

yihua approved these changes Mar 28, 2024

View reviewed changes

remove . from config

521ae79

yihua requested changes Mar 28, 2024

View reviewed changes

address review comments

0e2e1d8

github-actions bot removed the size:M PR with lines of changes in (100, 300] label Apr 1, 2024

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Apr 1, 2024

Jonathan Vexler added 3 commits April 1, 2024 10:00

Merge branch 'master' into classify_record_rewrite_exceptions

204bffd

fix merge

26afcf3

fix checkstyle

5138020

jonvex requested a review from yihua April 1, 2024 14:07

github-actions bot added size:M PR with lines of changes in (100, 300] and removed size:L PR with lines of changes in (300, 1000] labels Apr 1, 2024

yihua reviewed Apr 2, 2024

View reviewed changes

jonvex and others added 3 commits April 2, 2024 21:31

Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieReco…

f7e2e1b

…rdCreationException.java Co-authored-by: Y Ethan Guo <[email protected]>

Update hudi-common/src/main/java/org/apache/hudi/exception/HoodieAvro…

ea630b5

…SchemaException.java Co-authored-by: Y Ethan Guo <[email protected]>

add javadoc to exception wrapper

1770608

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:M PR with lines of changes in (100, 300] labels Apr 3, 2024

yihua approved these changes Apr 3, 2024

View reviewed changes

jonvex merged commit bf723f5 into apache:master Apr 3, 2024

nsivabalan added the release-0.15.0 label May 1, 2024

		package org.apache.hudi.util

		import org.apache.hudi.common.util.ReflectionUtils

[HUDI-7486] Classify schema exceptions when converting from avro to spark row representation #10778

[HUDI-7486] Classify schema exceptions when converting from avro to spark row representation #10778

Uh oh!

Conversation

jonvex commented Feb 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yihua Mar 27, 2024

Choose a reason for hiding this comment

Uh oh!

jonvex Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

yihua Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

yihua Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yihua Mar 28, 2024

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

yihua Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hudi-bot commented Apr 3, 2024

CI report:

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jonvex commented Feb 28, 2024 •

edited

Loading