[SPARK-28142][SS] Use CaseInsensitiveStringMap for KafkaContinuousStream#24942
Conversation
|
I skipped adding UT as similar changes for other data source are not covered by UT as well. |
|
cc. @cloud-fan |
|
Test build #106811 has finished for PR 24942 at commit
|
|
Actually, @HeartSaVioR, can we file a JIRA since it fixes an actual bug? |
|
Sure, just filed SPARK-28142 and changed the title. Thanks! |
gaborgsomogyi
left a comment
There was a problem hiding this comment.
LGTM. This lower/natural case is a constant problem.
|
As I'm implementing DSv2 batch source/sink I've pinpointed similar problems, so creating a PR... |
|
Merged to master. |
|
@gaborgsomogyi @HyukjinKwon Could you have a look for #22282? It is related to this PR. |
|
Thanks for reviewing and merging! |
|
@dongjinleekr I've some major items on my plate but coming back to the mentioned PR. |
|
@HeartSaVioR @gaborfeher @HyukjinKwon We should not skip adding the unit test cases. Please add one. Thanks! |
|
@gatorsmile Thanks for the suggestion. I'll see whether I can do it with small changes, and raise follow-up PR. |
|
#24999 addressed UT. Please review the new patch. Thanks! |
What changes were proposed in this pull request?
This patch addresses a missing spot which Map should be passed as CaseInsensitiveStringMap - KafkaContinuousStream seems to be only the missed one.
Before this fix, it has a relevant bug where
pollTimeoutMsis always set to default value, as the value ofKafkaSourceProvider.CONSUMER_POLL_TIMEOUTiskafkaConsumer.pollTimeoutMswhich key-lowercased map has been provided assourceOptions.How was this patch tested?
N/A.