Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
01848dc
[SPARK-37558][DOC] Improve spark sql cli document
AngersZhuuuu Dec 6, 2021
7a9ecbf
update
AngersZhuuuu Dec 7, 2021
a0218f8
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 7, 2021
1c59308
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 7, 2021
cea5be4
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 7, 2021
8eb04bf
Merge branch 'master' into SPARK-37558
AngersZhuuuu Dec 8, 2021
36ce29f
follow comment
AngersZhuuuu Dec 8, 2021
21fb349
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 10, 2021
0c77217
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 10, 2021
e274700
Update sql-distributed-sql-engine-spark-sql-cli.md
cloud-fan Dec 13, 2021
6d6b0cf
Update docs/sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 13, 2021
a06b166
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 13, 2021
8d2fa9a
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 13, 2021
4a7ca98
Update docs/sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 14, 2021
45ec6a0
Update docs/sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 14, 2021
403d966
Merge branch 'master' into SPARK-37558
AngersZhuuuu Dec 14, 2021
d9f150c
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 14, 2021
453f260
Update sql-distributed-sql-engine-spark-sql-cli.md
cloud-fan Dec 14, 2021
d4f6778
Update docs/sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 15, 2021
29f3b45
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 15, 2021
2b6f130
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 15, 2021
147d8b0
Update sql-distributed-sql-engine-spark-sql-cli.md
AngersZhuuuu Dec 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ options for deployment:
* [GraphX](graphx-programming-guide.html): processing graphs
* [SparkR](sparkr.html): processing data with Spark in R
* [PySpark](api/python/getting_started/index.html): processing data with Spark in Python
* [Spark SQL CLI](sql-distributed-sql-engine-spark-sql-cli.html): process data use SQL with command line interface
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated

**API Docs:**

Expand Down
169 changes: 169 additions & 0 deletions docs/sql-distributed-sql-engine-spark-sql-cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
---
layout: global
title: Spark SQL CLI
displayTitle: Spark SQL CLI
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---

* Table of contents
{:toc}


The Spark SQL CLI is a convenient tool to run the Hive metastore service in local mode and execute
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated
queries input from the command line. Note that the Spark SQL CLI cannot talk to the Thrift JDBC server.

To start the Spark SQL CLI, run the following in the Spark directory:

./bin/spark-sql

Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.
Comment thread
AngersZhuuuu marked this conversation as resolved.

## Spark SQL Command Line Options

You may run `./bin/spark-sql --help` for a complete list of all available options.

CLI options:
-d,--define <key=value> Variable substitution to apply to Hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
-e <quoted-query-string> SQL from command line
-f <filename> SQL from files
-H,--help Print help information
--hiveconf <property=value> Use value for given property
--hivevar <key=value> Variable substitution to apply to Hive
Comment thread
cloud-fan marked this conversation as resolved.
commands. e.g. --hivevar A=B
-i <filename> Initialization SQL file
-S,--silent Silent mode in interactive shell
-v,--verbose Verbose mode (echo executed SQL to the
console)

## The hiverc File

The Spark SQL CLI when invoked without the `-i` option will attempt to load `$HIVE_HOME/bin/.hiverc` and `$HOME/.hiverc` as initialization files.
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated

## Spark SQL CLI Interactive Shell Commands

When `$SPARK__HOME/bin/spark-sql` is run without either the `-e` or `-f` option, it enters interactive shell mode.
Use `;` (semicolon) to terminate commands, but user can escape `;` by `\\;`. Comments in scripts can be specified using the `--` prefix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if ; is not the end of the line? does it still terminate commands?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should explain more about ;

  • what if it's inside a comment? SELECT /* xxx; */ 1;
  • is ; the only way to terminate commands? what if I type SELECT 1 and then press enter?
  • can we put multiple commands in a single line? SELECT 1; SELECT 2;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about current?


<table class="table">
<tr><th>Command</th><th>Description</th></tr>
<tr>
<td><code>quit</code> <code>exit</code></td>
<td>Use <code>quit</code> or <code>exit</code> to leave the interactive shell.</td>
</tr>
<tr>
<td><code>!&lt;command&gt;</code></td>
<td>Executes a shell command from the Spark SQL CLI shell.</td>
</tr>
<tr>
<td><code>dfs &lt;dfs command&gt;</code></td>
<td>Executes a dfs command from the Hive shell.</td>
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated
</tr>
<tr>
<td><code>&lt;query string&gt;</code></td>
<td>Executes a Spark SQL query and prints results to standard output.</td>
</tr>
<tr>
<td><code>source &lt;filepath&gt;</code></td>
<td>Executes a script file inside the CLI.</td>
</tr>
</table>

## Supported comment type
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated

<table class="table">
<tr><th>Comment</th><th>Example</th></tr>
<tr>
<td>simple comment</td>
<td>
<code>
-- This is a simple comment.
<br>
SELECT 1;
</code>
</td>
</tr>
<tr>
<td>bracketed comment</td>
<td>
<code>
/* This is a bracketed comment. */
<br>
SELECT 1;
</code>
</td>
</tr>
<tr>
<td>nested bracketed comment</td>
<td>
<code>
/* This is a /* nested bracketed comment*/ .*/
<br>
SELECT 1;
</code>
</td>
</tr>
</table>

## Examples

See Variable Substitution for examples of using the hiveconf option.
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated


Example of running a query from the command line
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated

./bin/spark-sql -e 'SELECT COL FROM TBL'

Example of setting Hive configuration variables

./bin/spark-sql -e 'SELECT COL FROM TBL' --hiveconf hive.exec.scratchdir=/home/my/hive_scratch --hiveconf mapred.reduce.tasks=32

Example of dumping data out from a query into a file using silent mode

./bin/spark-sql -S -e 'SELECT COL FROM TBL' > result.txt

Example of running a script non-interactively from local disk

./bin/spark-sql -f /path/to/spark-sql-script.sql
Comment thread
cloud-fan marked this conversation as resolved.

Example of running a script non-interactively from a Hadoop supported filesystem

./bin/spark-sql -f hdfs://<namenode>:<port>/spark-sql-script.sql
./bin/spark-sql -f s3://mys3bucket/spark-sql-script.sql

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this apply to the init file as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this apply to the init file as well?

Yes, they call same method processFile()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then shall we have a individual section about path interpretation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then shall we have a individual section about path interpretation?

Yea, necessary, it's not same as normal usage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about current.


Example of running an initialization script before entering interactive mode

./bin/spark-sql -i /path/to/spark-sql-init.sql

Example of entering interactive mode

./bin/spark-sql
spark-sql> SELECT 1;
1
spark-sql> -- This is a simple comment.
spark-sql> SELECT 1;
1

Example of entering interactive mode with escape `;` in comment

./bin/spark-sql
spark-sql>/* This is a comment contains \\;
> It won't be terminaled by \\; */
> SELECT 1;
1

12 changes: 4 additions & 8 deletions docs/sql-distributed-sql-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,8 @@ See more details in [[SPARK-21067]](https://issues.apache.org/jira/browse/SPARK-

## Running the Spark SQL CLI

The Spark SQL CLI is a convenient tool to run the Hive metastore service in local mode and execute
queries input from the command line. Note that the Spark SQL CLI cannot talk to the Thrift JDBC server.
To use the Spark SQL command line interface (CLI) from the shell:

To start the Spark SQL CLI, run the following in the Spark directory:

./bin/spark-sql

Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.
You may run `./bin/spark-sql --help` for a complete list of all available options.
./bin/hive
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated

Details please refer to [Spark SQL CLI](sql-distributed-sql-engine-spark-sql-cli.html)
Comment thread
AngersZhuuuu marked this conversation as resolved.
Outdated