[SPARK-32910][SS]Remove UninterruptibleThread usage from KafkaOffsetReaderAdmin #30668
Conversation
| val (topic, timestamps) = prepareTimestampRelatedUnitTest | ||
|
|
||
| val e = intercept[SparkException] { | ||
| // KafkaOffsetReaderConsumer and KafkaOffsetReaderAdmin both throws AssertionError |
There was a problem hiding this comment.
In order to help reviewers I've created a small extract how the exception difference look like.
With UninterruptibleThread test execution:
org.apache.spark.SparkException(Exception thrown in awaitResult: )
java.util.concurrent.ExecutionException(Boxed Error)
java.lang.AssertionError(assertion failed: No offset matched from request of topic-partition topic-0-2 and timestamp 9223372036854775807.)
Without UninterruptibleThread test execution:
java.lang.AssertionError(assertion failed: No offset matched from request of topic-partition topic-0-2 and timestamp 9223372036854775807.)
|
Test build #132418 has finished for PR 30668 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
HeartSaVioR
left a comment
HeartSaVioR
left a comment
There was a problem hiding this comment.
+1
I'll wait for a couple of days to see whether there's additional review comment, and merge once there's no further comment.
|
retest this, please |
|
Will merge either Jenkins or Github Action passes. |
|
Test build #132622 has finished for PR 30668 at commit
|
|
Thanks @gaborgsomogyi for the contribution! Merged into master (3.2.0). |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Extremely small step for the community but huge step for me. This was my 100th commit, thank you all! 🙂 |
|
@gaborgsomogyi it's a extremely large step for the community :-). |
7 participants
What changes were proposed in this pull request?
The Kafka offset reader which uses
AdminClientstill usesUninterruptibleThreadto call it. Since there is no evidence thatAdminClientsuffers from similar issues like KAFKA-1894 I'm removingUninterruptibleThreadusage. In order to put theAdminClientunder stress and make sure it works I've created the following standalone application: https://github.com/gaborgsomogyi/kafka-admin-interruptionWhat this PR contains:
UninterruptibleThreadfromKafkaOffsetReaderAdminKafkaRelationSuitepartitionsAssignedToConsumertopartitionsAssignedToAdminWhy are the changes needed?
KafkaOffsetReaderAdmindoesn't needUninterruptibleThreadusage.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing unit tests + manually with simple Kafka to Kafka query.