Skip to content

Commit

Permalink
docs: split catalog (databendlabs#13394)
Browse files Browse the repository at this point in the history
* updates

* Update docusaurus.config.js

* add index pages

* add index pages

* add index pages

* Update dml-copy-into-table.md

* fixed links
  • Loading branch information
soyeric128 authored and andylokandy committed Nov 27, 2023
1 parent cccc7e4 commit c50e86b
Show file tree
Hide file tree
Showing 34 changed files with 367 additions and 284 deletions.
2 changes: 1 addition & 1 deletion docs/doc/12-load-data/00-stage/02-stage-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Staging Files
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Databend recommends two file upload methods for stages: [PRESIGN](/14-sql-commands/00-ddl/80-presign/presign.md) and PUT/GET commands. These methods enable direct data transfer between the client and your storage, eliminating intermediaries and resulting in cost savings by reducing traffic between Databend and your storage.
Databend recommends two file upload methods for stages: [PRESIGN](../../14-sql-commands/00-ddl/80-presign.md) and PUT/GET commands. These methods enable direct data transfer between the client and your storage, eliminating intermediaries and resulting in cost savings by reducing traffic between Databend and your storage.

![Alt text](../../../public/img/load/staging-file.png)

Expand Down
2 changes: 1 addition & 1 deletion docs/doc/12-load-data/00-transform/05-querying-stage.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ This example shows how to query data in a Parquet file stored in different locat
<Tabs groupId="query2stage">
<TabItem value="Stages" label="Stages">

Let's assume you have a sample file named [books.parquet](https://datafuse-1253727613.cos.ap-hongkong.myqcloud.com/data/books.parquet) and you have uploaded it to your user stage, an internal stage named *my_internal_stage*, and an external stage named *my_external_stage*. To upload files to a stage, use the [PRESIGN](/14-sql-commands/00-ddl/80-presign/presign.md) method.
Let's assume you have a sample file named [books.parquet](https://datafuse-1253727613.cos.ap-hongkong.myqcloud.com/data/books.parquet) and you have uploaded it to your user stage, an internal stage named *my_internal_stage*, and an external stage named *my_external_stage*. To upload files to a stage, use the [PRESIGN](../../14-sql-commands/00-ddl/80-presign.md) method.

```sql
-- Query file in user stage
Expand Down
6 changes: 1 addition & 5 deletions docs/doc/14-sql-commands/00-ddl/10-database/_category_.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"label": "Database",
"link": {
"type": "generated-index",
"slug": "/sql-commands/ddl/database"
}
"label": "Database"
}
8 changes: 8 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/10-database/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: Database
---
import IndexOverviewList from '@site/src/components/IndexOverviewList';

This page provides reference information for the database-related commands in Databend.

<IndexOverviewList />
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"label": "File Format",
"link": {
"type": "generated-index",
"slug": "/sql-commands/ddl/file-format"
}
"label": "File Format"
}
8 changes: 8 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/100-file-format/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: File Format
---
import IndexOverviewList from '@site/src/components/IndexOverviewList';

This page provides reference information for the file format-related commands in Databend.

<IndexOverviewList />
6 changes: 1 addition & 5 deletions docs/doc/14-sql-commands/00-ddl/20-table/_category_.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"label": "Table",
"link": {
"type": "generated-index",
"slug": "/sql-commands/ddl/table"
}
"label": "Table"
}
28 changes: 28 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/20-table/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: Table
---

This page provides reference information for the table-related commands in Databend.

## Table Management:

- [CREATE TABLE](10-ddl-create-table.md)
- [DROP TABLE](20-ddl-drop-table.md)
- [UNDROP TABLE](21-ddl-undrop-table.md)
- [RENAME TABLE](30-ddl-rename-table.md)
- [TRUNCATE TABLE](40-ddl-truncate-table.md)
- [VACUUM DROP TABLE](91-vacuum-drop-table.md)
- [VACUUM TABLE](91-vacuum-table.md)
- [ATTACH TABLE](92-attach-table.md)

## Table Information:

- [DESCRIBE TABLE](50-describe-table.md)
- [ANALYZE TABLE](80-analyze-table.md)

## Table Modification:

- [ALTER TABLE COLUMN](90-alter-table-column.md)
- [ALTER TABLE OPTION](90-alter-table-option.md)
- [OPTIMIZE TABLE](60-optimize-table.md)
- [FLASHBACK TABLE](70-flashback-table.md)
6 changes: 1 addition & 5 deletions docs/doc/14-sql-commands/00-ddl/30-user/_category_.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"label": "User",
"link": {
"type": "generated-index",
"slug": "/sql-commands/ddl/user"
}
"label": "User & Role"
}
28 changes: 28 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/30-user/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: User
---

This page provides reference information for the user and role related commands in Databend.

## User and Role Management:

- [CREATE USER](01-user-create-user.md)
- [DROP USER](02-user-drop-user.md)
- [CREATE ROLE](04-user-create-role.md)
- [DROP ROLE](05-user-drop-role.md)

## Permissions Management:

- [GRANT privileges](10-grant-privileges.md)
- [REVOKE privileges](11-revoke-privileges.md)
- [GRANT ROLE](20-grant-role.md)
- [REVOKE ROLE](21-revoke-role.md)

## Information Retrieval:

- [SHOW USERS](02-user-show-users.md)
- [SHOW GRANTS](22-show-grants.md)

## User Modification:

- [ALTER USER](03-user-alter-user.md)
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Lists files in a stage.
See also:

- [LIST_STAGE](../../../15-sql-functions/112-table-functions/list-stage.md): This function lists files in a stage and allows you to filter files in a stage based on their extensions and obtain comprehensive details about each file.
- [PRESIGN](../80-presign/presign.md): Databend recommends using the Presigned URL method to upload files to the stage.
- [PRESIGN](../80-presign.md): Databend recommends using the Presigned URL method to upload files to the stage.
- [REMOVE STAGE FILES](05-ddl-remove-stage.md): Removes files from a stage.

## Syntax
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Removes files from a stage.
See also:

- [LIST STAGE FILES](04-ddl-list-stage.md): Lists files in a stage.
- [PRESIGN](../80-presign/presign.md): Databend recommends using the Presigned URL method to upload files to the stage.
- [PRESIGN](../80-presign.md): Databend recommends using the Presigned URL method to upload files to the stage.

## Syntax

Expand Down
6 changes: 1 addition & 5 deletions docs/doc/14-sql-commands/00-ddl/60-view/_category_.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"label": "View",
"link": {
"type": "generated-index",
"slug": "/sql-commands/ddl/view"
}
"label": "View"
}
8 changes: 8 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/60-view/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: View
---
import IndexOverviewList from '@site/src/components/IndexOverviewList';

This page provides reference information for the view-related commands in Databend.

<IndexOverviewList />
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
---
title: PRESIGN
description:
Generates the pre-signed URL for a staged file.
title: Presign
---

Generates the pre-signed URL for a staged file by the stage name and file path you provide. The pre-signed URL enables you to access the file through a web browser or an API request.
Expand All @@ -19,8 +17,8 @@ curl -X PUT -T books.csv -H "<header-generated-by-presign>" <presigned-url>
See also:

- [Presign RFC](https://databend.rs/doc/contributing/rfcs/presign)
- [LIST STAGE FILES](../40-stage/04-ddl-list-stage.md): Lists files in a stage.
- [REMOVE STAGE FILES](../40-stage/05-ddl-remove-stage.md): Removes files from a stage.
- [LIST STAGE FILES](40-stage/04-ddl-list-stage.md): Lists files in a stage.
- [REMOVE STAGE FILES](40-stage/05-ddl-remove-stage.md): Removes files from a stage.

## Syntax

Expand Down
7 changes: 0 additions & 7 deletions docs/doc/14-sql-commands/00-ddl/80-presign/_category_.json

This file was deleted.

70 changes: 0 additions & 70 deletions docs/doc/14-sql-commands/00-ddl/91-catalog/create-catalog.md

This file was deleted.

83 changes: 83 additions & 0 deletions docs/doc/14-sql-commands/00-ddl/91-catalog/hive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
title: Apache Hive
---
import FunctionDescription from '@site/src/components/FunctionDescription';

<FunctionDescription description="Introduced or updated: v1.2.83"/>

Databend supports the integration of an [Apache Hive](https://hive.apache.org/) catalog, enhancing its compatibility and versatility for data management and analytics. This extends Databend's capabilities by seamlessly incorporating the powerful metadata and storage management capabilities of Apache Hive into the platform.

## Managing Apache Hive Catalogs

Databend provides you the following commands to manage Apache Hive catalogs:

- [CREATE CATALOG](#create-catalog)
- [SHOW CREATE CATALOG](#show-create-catalog)

### CREATE CATALOG

Defines and establishes a new catalog in the Databend query engine.

#### Syntax

```sql
CREATE CATALOG <catalog_name>
TYPE = <catalog_type>
CONNECTION = (
METASTORE_ADDRESS = '<hive_metastore_address>'
URL = '<data_storage_path>'
<connection_parameter> = '<connection_parameter_value>'
<connection_parameter> = '<connection_parameter_value>'
...
)
```

| Parameter | Required? | Description |
|-----------------------|-----------|---------------------------------------------------------------------------------------------------------------------------|
| TYPE | Yes | Type of the catalog: 'HIVE' for Hive catalog or 'ICEBERG' for Iceberg catalog. |
| METASTORE_ADDRESS | No | Hive Metastore address. Required for Hive catalog only.|
| URL | Yes | Location of the external storage linked to this catalog. This could be a bucket or a folder within a bucket. For example, 's3://databend-toronto/'. |
| connection_parameter | Yes | Connection parameters to establish connections with external storage. The required parameters vary based on the specific storage service and authentication methods. Refer to [Connection Parameters](../../../13-sql-reference/51-connect-parameters.md) for detailed information. |

:::note
To read data from HDFS, you need to set the following environment variables before starting Databend. These environment variables ensure that Databend can access the necessary Java and Hadoop dependencies to interact with HDFS effectively. Make sure to replace "/path/to/java" and "/path/to/hadoop" with the actual paths to your Java and Hadoop installations, and adjust the CLASSPATH to include all the required Hadoop JAR files.
```shell
export JAVA_HOME=/path/to/java
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
export HADOOP_HOME=/path/to/hadoop
export CLASSPATH=/all/hadoop/jar/files
```
:::

### SHOW CREATE CATALOG

Returns the detailed configuration of a specified catalog, including its type and storage parameters.

#### Syntax

```sql
SHOW CREATE CATALOG <catalog_name>;
```

## Usage Examples

This example demonstrates the creation of a catalog configured to interact with the Hive Metastore and access data stored on Amazon S3, located at 's3://databend-toronto/'.

```sql
CREATE CATALOG hive_ctl
TYPE = HIVE
CONNECTION =(
METASTORE_ADDRESS = '127.0.0.1:9083'
URL = 's3://databend-toronto/'
AWS_KEY_ID = '<your_key_id>'
AWS_SECRET_KEY = '<your_secret_key>'
);

SHOW CREATE CATALOG hive_ctl;

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Catalog │ Type │ Option │
├──────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ hive_ctl │ hive │ METASTORE ADDRESS\n127.0.0.1:9083\nSTORAGE PARAMS\ns3 | bucket=databend-toronto,root=/,endpoint=https://s3.amazonaws.com │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Loading

0 comments on commit c50e86b

Please sign in to comment.