Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 57 additions & 2 deletions docs/integrate/telegraf/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ a very minimal memory footprint.
:::{rubric} Overview
:::

::::{grid}

:::{grid-item}
- **IoT sensors**: Collect critical stateful data (pressure levels, temperature
levels, etc.) with popular protocols like MQTT, ModBus, OPC-UA, and Kafka.

Expand All @@ -31,16 +34,68 @@ a very minimal memory footprint.

- **System telemetry**: Metrics from system telemetry like iptables, Netstat,
NGINX, and HAProxy help provide a full stack view of your apps.
:::

:::{grid-item}
![Telegraf architecture overview](https://www.influxdata.com/wp-content/uploads/Main-Diagram_06.01.2022v1.png){loading=lazy}
:::

::::


:::{rubric} Synopsis
:::

Telegraf output plugin configuration snippet for CrateDB.
```toml
# Configuration for CrateDB to send metrics to.
[[outputs.cratedb]]

# A github.com/jackc/pgx/v4 connection string.
# See https://pkg.go.dev/github.com/jackc/pgx/v4#ParseConfig
url = "postgres://crate@localhost/doc?sslmode=disable"

# Timeout for all CrateDB queries.
timeout = "5s"

![Telegraf architecture overview](https://www.influxdata.com/wp-content/uploads/Main-Diagram_06.01.2022v1.png){height=200px loading=lazy}
# Name of the table to store metrics in.
table = "metrics"

# If true, and the metrics table does not exist, create it automatically.
table_create = true

# The character(s) to replace any '.' in an object key with
key_separator = "_"
```


:::{rubric} Learn
:::

- [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics]
::::{grid}

:::{grid-item-card} Tutorial: Use Telegraf with CrateDB
:link: telegraf-tutorial
:link-type: ref
How to configure Telegraf to submit metrics to CrateDB.
:::

:::{grid-item-card} Blog: Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics
:link: https://cratedb.com/blog/use-cratedb-with-telegraf-an-agent-for-collecting-reporting-metrics
:link-type: url
Learn how to set up Telegraf, have Telegraf send metrics data to CrateDB,
and visualize the collected data with Grafana.
:::

::::


:::{toctree}
:maxdepth: 1
:hidden:
Tutorial <tutorial>
:::

```{seealso}
[CrateDB and Telegraf]
```
Expand Down
129 changes: 129 additions & 0 deletions docs/integrate/telegraf/tutorial.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
(telegraf-tutorial)=
# Load data into CrateDB using Telegraf

This tutorial walks you through starting the [Telegraf] agent and CrateDB,
then configuring Telegraf to submit system metrics to CrateDB.

## Prerequisites

Docker is used for running all components. This approach works consistently
across Linux, macOS, and Windows. Alternatively, you can use Podman.

Create a shared network.
```shell
docker network create cratedb-demo
```

Start CrateDB.
```shell
docker run --name=cratedb --rm -it --network=cratedb-demo \
--publish=4200:4200 --publish=5432:5432 \
--env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node
```

Prepare shortcut for the CrateDB shell and the Telegraf command.

::::{tab-set}

:::{tab-item} Linux and macOS
To make the settings persistent, add them to your shell profile (`~/.profile`).
```shell
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
alias telegraf="docker run --rm --network=cratedb-demo docker.io/telegraf"
```
:::
:::{tab-item} Windows PowerShell
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
```powershell
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
function telegraf { docker run --rm --network=cratedb-demo docker.io/telegraf @args }
```
:::
:::{tab-item} Windows Command
```shell
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
doskey telegraf=docker run --rm --network=cratedb-demo docker.io/telegraf $*
```
:::

::::


## Usage

Configure Telegraf by creating a configuration blueprint and adjusting it.
Telegraf is a plugin-driven tool and has plugins to collect many different types
of metrics. Because we just want to test things out, we are using `--input-filter cpu`
to limit input plugins so that Telegraf only collects readings about CPU usage
on the local computer. To send the collected data to CrateDB, use
`--output-filter cratedb`.
```shell
telegraf \
--input-filter cpu \
--output-filter cratedb \
config > telegraf.conf
```

The database connection URL is a `pgx/v4` connection string. Configure
`table_create = true` to automatically let Telegraf create the metrics table
if it doesn't exist.
::::{tab-set}
:::{tab-item} Linux
```shell
sed -i 's!postgres://user:password@localhost/schema?sslmode=disable!postgres://crate@cratedb/doc?sslmode=disable!g' telegraf.conf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BSD sed and GNU sed do not have the same semantics for option -i. I suggest to use -i '' to cover both cases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbiai already suggested the same, thanks!

In docs/integrate/telegraf/tutorial.md, we recommend to...

71-84: Split Linux vs macOS for sed; BSD sed needs a different -i form.

The current “Linux and macOS” block will fail on macOS because BSD sed requires -i ''. Also, consider setting the table name explicitly to avoid relying on implicit defaults.

sed -i 's!# table_create = false!table_create = true!' telegraf.conf
```
:::
:::{tab-item} macOS and BSD
```shell
sed -i '' 's!postgres://user:password@localhost/schema?sslmode=disable!postgres://crate@cratedb/doc?sslmode=disable!g' telegraf.conf
sed -i '' 's!# table_create = false!table_create = true!' telegraf.conf
```
:::
:::{tab-item} Windows PowerShell
```powershell
(Get-Content telegraf.conf) -replace 'postgres://user:password@localhost/schema\?sslmode=disable','postgres://crate@cratedb/doc?sslmode=disable' |
ForEach-Object { $_ -replace '# table_create = false','table_create = true' } |
Set-Content telegraf.conf
```
:::
::::


Start Telegraf.
::::{tab-set}
:::{tab-item} Linux and macOS
```shell
docker run --name=telegraf --rm -it --network=cratedb-demo \
--volume "$(pwd)"/telegraf.conf:/etc/telegraf/telegraf.conf \
docker.io/telegraf
```
:::
:::{tab-item} Windows PowerShell
```powershell
docker run --name=telegraf --rm -it --network=cratedb-demo `
--volume "${PWD}\telegraf.conf:/etc/telegraf/telegraf.conf" `
docker.io/telegraf
```
:::
:::{tab-item} Windows Command
```shell
docker run --name=telegraf --rm -it --network=cratedb-demo ^
--volume "%cd%\telegraf.conf:/etc/telegraf/telegraf.conf" ^
docker.io/telegraf
```
:::
::::


After 10 seconds, which is the default output flush interval of Telegraf, the first
metrics will appear in the `metrics` table in CrateDB. To adjust the value, navigate
to `flush_interval = "10s"` in `telegraf.conf`.

Inspect data stored in CrateDB.
```shell
crash --hosts cratedb -c "SELECT * FROM doc.metrics LIMIT 5;"
```


[Telegraf]: https://www.influxdata.com/time-series-platform/telegraf/