Skip to content

[Minor][SS][DOCS] Minor doc change for kafka integration #18550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 1 commit into from
Closed

[Minor][SS][DOCS] Minor doc change for kafka integration #18550

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/structured-streaming-kafka-integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ For Scala/Java applications using SBT/Maven project definitions, link your appli
For Python applications, you need to add this above library and its dependencies when deploying your
application. See the [Deploying](#deploying) subsection below.

For experimenting on `spark-shell`, you need to add this above library and its dependencies too when invoking `spark-shell`. Also see the [Deploying](#deploying) subsection below.

## Reading Data from Kafka

### Creating a Kafka Source for Streaming Queries
Expand Down Expand Up @@ -607,5 +609,9 @@ and its dependencies can be directly added to `spark-submit` using `--packages`,

./bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION_SHORT}} ...

For experimenting on `spark-shell`, you can also use `--packages` to add `spark-sql-kafka-0-10_{{site.SCALA_BINARY_VERSION}}` and its dependencies directly,
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun Jul 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @viirya
For spark-shell, is you need to use better than you can also use because there is no other way?

Copy link
Copy Markdown
Member Author

@viirya viirya Jul 6, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh. the you can also use is relative to spark-submit. Above this, spark-submit uses --packages to add library, so here it says you can also use this option to add library.

Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun Jul 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. In the Linking section, there is a warning need to add. So, I think we had better to be more clear here.

For experimenting on `spark-shell`, you need to add this above library and its dependencies too when invoking `spark-shell`. Also see the [Deploying](#deploying) subsection below.

For the Deploy section, it looks like that as you said.


./bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION_SHORT}} ...

See [Application Submission Guide](submitting-applications.html) for more details about submitting
applications with external dependencies.