Skip to content

Conversation

@dilipbiswal
Copy link
Contributor

What changes were proposed in this pull request?

Set a reasonable poll timeout thats used while consuming topics/partitions from kafka. In the
absence of it, a default of 2 minute is used as the timeout values. And all the negative tests take a minimum of 2 minute to execute.

After this change, we save about 4 minutes in this suite.

How was this patch tested?

Test fix.

@SparkQA
Copy link

SparkQA commented Oct 8, 2018

Test build #97102 has finished for PR 22670 at commit 47e0940.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dilipbiswal
Copy link
Contributor Author

cc @gatorsmile

@srowen
Copy link
Member

srowen commented Oct 12, 2018

@dilipbiswal I like this change too. The suite goes from 4:34 to 0:53. I wonder if we can make this change elsewhere in general Kafka test config? This kind of setting seems useful everywhere.

@dilipbiswal
Copy link
Contributor Author

@srowen Thanks. Did you mean, the test cases should extend a shared spark context (SharedKafkaSparkContext) which would have this property set ?

Actually Sean, there are 3 suites in this directory. DirectKafkaStreamSuite.scala KafkaDataConsumerSuite.scala and KafkaRDDSuite.scala. Given only two tests which are affected by this timeout (which we are fixing here), do you think we need to take on this refactoring work as part of this PR ? Please let me know.

@srowen
Copy link
Member

srowen commented Oct 12, 2018

I don't so much mean that much refactoring. I wonder if there are 1-2 other places where common Kafka params are set in tests that we could add this to for now, that kind of thing. This change is OK by itself too though.

@dilipbiswal
Copy link
Contributor Author

@srowen OK, Let me look.

@SparkQA
Copy link

SparkQA commented Oct 16, 2018

Test build #97473 has finished for PR 22670 at commit 609daba.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Oct 16, 2018

Merged to master

@asfgit asfgit closed this in 9d4dd79 Oct 16, 2018
@dilipbiswal
Copy link
Contributor Author

@srowen Thank you very much.

jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
…kaRDDSuite

## What changes were proposed in this pull request?
Set a reasonable poll timeout thats used while consuming topics/partitions from kafka. In the
absence of it, a default of 2 minute is used as the timeout values. And all the negative tests take a minimum of 2 minute to execute.

After this change, we save about 4 minutes in this suite.

## How was this patch tested?
Test fix.

Closes apache#22670 from dilipbiswal/SPARK-25631.

Authored-by: Dilip Biswal <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants