Skip to content

Conversation

@markgrover
Copy link
Member

What changes were proposed in this pull request?

Removing references to assembly jar in documentation.
Adding an additional (previously undocumented) usage of spark-submit to run examples.

How was this patch tested?

Ran spark-submit usage to ensure formatting was fine. Ran examples using SparkSubmit.

@srowen
Copy link
Member

srowen commented Apr 13, 2016

Looks good though a quick search showed many more instances of "assembly jar" -- you might want to zap them all here

@markgrover
Copy link
Member Author

Thanks! I only looked through docs, but let me take a closer second look.

@markgrover
Copy link
Member Author

Ok, I have replaced the usage at a few more places.

And, for those curious, the only places that are left now are:

>grep -ri "assembly jar" *
docs/submitting-applications.md:create an assembly jar (or "uber" jar) containing your code and its dependencies. Both
docs/submitting-applications.md:have assembly plugins. When creating assembly jars, list Spark and Hadoop
python/pyspark/streaming/tests.py:            ("Failed to find Spark Streaming kafka assembly jar in %s. " % kafka_assembly_dir) +
python/pyspark/streaming/tests.py:        raise Exception(("Found multiple Spark Streaming Kafka assembly JARs: %s; please "
python/pyspark/streaming/tests.py:            ("Failed to find Spark Streaming Flume assembly jar in %s. " % flume_assembly_dir) +
python/pyspark/streaming/tests.py:        raise Exception(("Found multiple Spark Streaming Flume assembly JARs: %s; please "
python/pyspark/streaming/tests.py:        raise Exception(("Found multiple Spark Streaming Kinesis ASL assembly JARs: %s; please "
python/pyspark/streaming/tests.py:            ("Failed to find Spark Streaming Kinesis assembly jar in %s. "
python/run-tests.py:# Find out where the assembly jars are located.

seem legit usages because they either talk about streaming backends where assembly jars are still the norm, or how users run applications as assembly/uber jars.

@SparkQA
Copy link

SparkQA commented Apr 13, 2016

Test build #55732 has finished for PR 12365 at commit 7bd16cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 13, 2016

Test build #55738 has finished for PR 12365 at commit a6e08b8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 13, 2016

Test build #55737 has finished for PR 12365 at commit 5b06dc2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • // Build up a PYTHONPATH that includes the Spark assembly (where this class is), the

@srowen
Copy link
Member

srowen commented Apr 14, 2016

LGTM

@rxin
Copy link
Contributor

rxin commented Apr 15, 2016

Thanks - merging this.

@asfgit asfgit closed this in ff9ae61 Apr 15, 2016
@markgrover
Copy link
Member Author

Thanks Reynold and Sean!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants