Compaction/Clustering job not fail when throw HoodieException

[0xDTE]

Clustering/Compaction job throw follow exception, the final result returns -1 and the job's state is success.

ERROR UtilHelpers: Cluster failed
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 2 times, most recent failure: Lost task 0.1 in stage 4.0 (TID 5) : org.apache.hudi.exception.HoodieException: unable to read next record from parquet file
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:54)
at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39)
at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39)
at org.apache.hudi.client.utils.ConcatenatingIterator.hasNext(ConcatenatingIterator.java:45)
Issue Links
links to
Web Link GitHub Pull Request #10050
Activity
All
Comments
Work Log
History
Activity
Transitions

danny0405
Danny Chen added a comment - 02/Dec/23 03:43
Fixed via master branch: 77e2f42

People
Assignee:
Unassigned
Unassigned
Reporter:
ksmou
kwang
Votes:
0 Vote for this issue
Watchers:
2 Start watching this issue
Dates
Created:
10/Nov/23 08:36
Updated:
Yesterday 03:43

[torvalds-dev]

torvalds.dev is analyzing the ticket

[torvalds-dev]

The issue you're referring to is related to Apache Hudi, a data lake framework that provides stream processing and incremental data pipeline capabilities. The specific problem is with the compaction and clustering jobs in Hudi.

From the error message, it seems that the compaction/clustering job is failing due to an exception (HoodieException) being thrown. This exception is caused by an inability to read the next record from a Parquet file. Parquet is a columnar storage file format that is optimized for use with Apache Hadoop due to its superior query performance and lower storage cost.

Despite this failure, the job's state is reported as successful, and the final result returns -1. This is likely a bug, as a job that encounters a failure should not report a successful state.

The issue was reported by a user named ksmou and was not assigned to anyone. It was created on 10/Nov/23 and last updated yesterday.

A user named Danny Chen commented that the issue was fixed via the master branch with the commit 77e2f42. You can view the changes made in this commit by visiting the provided GitHub link.

As for the code you provided, it seems to be a test case for the Hudi framework. It's creating a Parquet table, simulating inserts, setting up incremental reads, and verifying that incremental reads do not show inserts after a compaction timestamp. It also tests the conversion of Avro records to ArrayWritable objects.

If you have any specific questions about the code or the issue, feel free to ask!