Skip to content

[SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated #7961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tdas wants to merge 11 commits into from
Closed

[SPARK-9640][Streaming][Test] Do not run Python Kinesis tests when the Kinesis assembly JAR has not been generated #7961

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e292e64
Added filters for Kinesis python tests
tdas Aug 5, 2015
b858196
Fixed conditions and few other things based on PR comments.
tdas Aug 6, 2015
94be631
Fixed style
tdas Aug 7, 2015
871fcc8
Fixed SparkBuild
tdas Aug 7, 2015
88d59bd
updated test modules
tdas Aug 7, 2015
ba9cfc7
Improved kinesis test selection logic
tdas Aug 7, 2015
ca90719
Removed extra line
tdas Aug 10, 2015
e6a677e
Merge remote-tracking branch 'apache-github/master' into SPARK-9640
tdas Aug 10, 2015
9bbb97d
Minor style fies
tdas Aug 10, 2015
004ae26
style fixes
tdas Aug 10, 2015
974ce19
Undo changes related to SPARK-9727
tdas Aug 11, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions python/pyspark/streaming/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -997,11 +997,7 @@ def search_kinesis_asl_assembly_jar():
os.path.join(kinesis_asl_assembly_dir,
"target/scala-*/spark-streaming-kinesis-asl-assembly-*.jar"))
if not jars:
raise Exception(
("Failed to find Spark Streaming Kinesis ASL assembly jar in %s. " %
kinesis_asl_assembly_dir) + "You need to build Spark with "
"'build/sbt -Pkinesis-asl assembly/assembly streaming-kinesis-asl-assembly/assembly' "
"or 'build/mvn -Pkinesis-asl package' before running this test")
return None
elif len(jars) > 1:
raise Exception(("Found multiple Spark Streaming Kinesis ASL assembly JARs in %s; please "
"remove all but one") % kinesis_asl_assembly_dir)
Expand All @@ -1013,7 +1009,22 @@ def search_kinesis_asl_assembly_jar():
kafka_assembly_jar = search_kafka_assembly_jar()
flume_assembly_jar = search_flume_assembly_jar()
kinesis_asl_assembly_jar = search_kinesis_asl_assembly_jar()
jars = "%s,%s,%s" % (kafka_assembly_jar, flume_assembly_jar, kinesis_asl_assembly_jar)
if kinesis_asl_assembly_jar is None:
kinesis_jar_present = False

@zsxwing zsxwing Aug 5, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ENABLE_KINESIS_TESTS is set to 1 but kinesis_jar_present is false, I think we should fail the test because it should be a bug that the assembly jar cannot be generated correctly.

@tdas tdas Aug 5, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. If that flag is set, then the use definitely wants to run ALL the kinesis tests. In that case its best to fail.

jars = "%s,%s" % (kafka_assembly_jar, flume_assembly_jar)
else:
kinesis_jar_present = True
jars = "%s,%s,%s" % (kafka_assembly_jar, flume_assembly_jar, kinesis_asl_assembly_jar)

print kinesis_jar_present, jars
os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars %s pyspark-shell" % jars
unittest.main()
testcases = [BasicOperationTests, WindowFunctionTests, StreamingContextTests, \
CheckpointTests, KafkaStreamTests, FlumeStreamTests, FlumePollingStreamTests]
if kinesis_jar_present is True:
testcases.append(KinesisStreamTests)

@zsxwing zsxwing Aug 7, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra empty line

sys.stderr.write("Running tests %s\n" % (str(testcases)))
for testcase in testcases:
print "[", testcase, "]"
tests = unittest.TestLoader().loadTestsFromTestCase(testcase)
unittest.TextTestRunner(verbosity=2).run(tests)