Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/ingest/etl/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,10 @@ Use visual data flow and integration frameworks and platforms.
MySQL and MariaDB are well-known free and open-source relational database management
systems (RDBMS), available as standalone and managed variants.

- {ref}`postgresql`

PostgreSQL is the world's most advanced open source relational database.

- {ref}`sql-server`

Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL
Expand Down Expand Up @@ -237,6 +241,7 @@ Load data from datasets and open table formats.
- {ref}`n8n`
- {ref}`nifi`
- {ref}`node-red`
- {ref}`postgresql`
- {ref}`risingwave`
- {ref}`sql-server`
- {ref}`streamsets`
Expand Down
1 change: 1 addition & 0 deletions docs/integrate/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ n8n/index
nifi/index
node-red/index
plotly/index
postgresql/index
Power BI <powerbi/index>
prometheus/index
pyviz/index
Expand Down
45 changes: 45 additions & 0 deletions docs/integrate/postgresql/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
(postgresql)=
# PostgreSQL

```{div} .float-right
[![postgresql-logo](https://www.postgresql.org/media/img/about/press/elephant.png){height=60px loading=lazy}][PostgreSQL]
```
```{div} .clearfix
```

:::{rubric} About
:::

[PostgreSQL] is the world's most advanced open source relational database.

:::{rubric} Synopsis
:::

```shell
uvx 'cratedb-toolkit[io-ingestr]' load table \
"postgresql://postgres:postgres@localhost:5432/test?table=public.demo" \
--cluster-url="crate://crate:crate@localhost:4200/doc/postgresql_demo"
```

:::{rubric} Learn
:::

::::{grid}

:::{grid-item-card} Tutorial: Use CrateDB Toolkit
:link: postgresql-tutorial
:link-type: ref
Load data from PostgreSQL into CrateDB using CrateDB Toolkit.
:::

::::


:::{toctree}
:maxdepth: 1
:hidden:
Tutorial <tutorial>
:::


[PostgreSQL]: https://www.postgresql.org/
95 changes: 95 additions & 0 deletions docs/integrate/postgresql/tutorial.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
(postgresql-tutorial)=
# Load data from PostgreSQL into CrateDB

The tutorial will walk you through starting [PostgreSQL] and CrateDB,
inserting a record into PostgreSQL, loading data into a CrateDB table,
and validating that the data has been stored successfully.
The data transfer is supported by the
{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements.

## Prerequisites

Docker is used for running all components. This approach works consistently
across Linux, macOS, and Windows. Alternatively, you can use Podman.

Create a shared network.
```shell
docker network create cratedb-demo
```

Start CrateDB.
```shell
docker run --rm --name=cratedb --network=cratedb-demo \
--publish=4200:4200 --publish=5432:5432 --env=CRATE_HEAP_SIZE=2g \
docker.io/crate -Cdiscovery.type=single-node
```

Start PostgreSQL.
```shell
docker run --rm --name=postgresql --network=cratedb-demo \
--publish=6432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \
docker.io/postgres postgres -c log_statement=all
```
:::{note}
Because CrateDB is configured to listen on port `5432` with its PostgreSQL
interface, let's use a different port for PostgreSQL itself.
:::

Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client
programs.

::::{tab-set}

:::{tab-item} Linux and macOS
To make the settings persistent, add them to your shell profile (`~/.profile`).
```shell
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk"
alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql"
```
:::
:::{tab-item} Windows PowerShell
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
```powershell
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args }
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
```
:::
:::{tab-item} Windows Command
```shell
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $*
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
```
:::

::::

## Usage

Write a few sample records to PostgreSQL.
```shell
psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL
CREATE DATABASE test;
\connect test;
CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB);
INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}');
INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}');
SQL
```

Invoke the data transfer pipeline.
```shell
ctk-ingest load table \
"postgresql://postgres:postgres@postgresql:5432/test?table=public.demo" \
--cluster-url="crate://crate:crate@cratedb:4200/doc/postgresql_demo"
```

Inspect data stored in CrateDB.
```shell
crash --hosts cratedb -c "SELECT * FROM doc.postgresql_demo"
```


[PostgreSQL]: https://www.postgresql.org/
Loading