Skip to content

Commit 2d49faf

Browse files
amotlamotl
committed
collectd: Index page and two tutorials
1 parent 26db150 commit 2d49faf

File tree

8 files changed

+494
-0
lines changed

8 files changed

+494
-0
lines changed

‎docs/ingest/telemetry/index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,12 @@ scenarios. CrateDB offers corresponding integration adapters.
5252
:padding: 0
5353
:gutter: 2
5454

55+
::::{grid-item-card} collectd
56+
:link: collectd
57+
:link-type: ref
58+
Send metrics with collectd, a system and application metrics collection daemon.
59+
::::
60+
5561
::::{grid-item-card} Prometheus
5662
:link: prometheus
5763
:link-type: ref

‎docs/integrate/collectd/Dockerfile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM docker.io/debian:13-slim
2+
3+
# Configure system environment.
4+
ENV DEBIAN_FRONTEND=noninteractive
5+
6+
# Install collectd.
7+
RUN apt update --yes
8+
RUN apt install --yes collectd
9+
10+
# Run collectd in foreground.
11+
ENTRYPOINT ["collectd", "-f"]

‎docs/integrate/collectd/collectd-cratedb.conf‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# collectd configuration for storing metrics into CrateDB.
2+
# https://collectd.org/documentation/manpages/collectd.conf.html#plugin-postgresql
3+
4+
LoadPlugin postgresql
5+
6+
<Plugin postgresql>
7+
<Writer sqlstore>
8+
Statement "INSERT INTO doc.collectd_data (p_time, p_host, p_plugin, p_plugin_instance, p_type, p_type_instance, p_value_names, p_type_names, p_values) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9);"
9+
StoreRates true
10+
</Writer>
11+
<Database cratedb>
12+
Writer sqlstore
13+
CommitInterval 30
14+
Host cratedb
15+
User crate
16+
Password crate
17+
</Database>
18+
</Plugin>

‎docs/integrate/collectd/collectd-telegraf.conf‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# collectd configuration for sending metrics to Telegraf.
2+
# https://collectd.org/documentation/manpages/collectd.conf.html#plugin-network
3+
4+
<Plugin "network">
5+
Server "telegraf"
6+
</Plugin>

‎docs/integrate/collectd/index.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
(collectd)=
2+
# collectd
3+
4+
```{div} .float-right
5+
[![collectd example graph](https://collectd.org/images/graph-cpu.png){width=180px loading=lazy}][collectd]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[collectd] is a system statistics collection daemon suitable for application
14+
performance metrics periodically and provides mechanisms to store the values
15+
in a variety of ways.
16+
17+
collectd gathers metrics from various sources, e.g. the operating system,
18+
applications, logfiles and external devices, and stores this information or
19+
makes it available over the network. Those statistics can be used to monitor
20+
systems, find performance bottlenecks (i.e. performance analysis) and predict
21+
future system load (i.e. capacity planning).
22+
23+
:::{rubric} Synopsis
24+
:::
25+
26+
Either use the `postgresql` plugin to store metrics into CrateDB,
27+
28+
:::{literalinclude} collectd-cratedb.conf
29+
:::
30+
31+
or use the `network` plugin to forward metrics to Telegraf.
32+
33+
:::{literalinclude} collectd-telegraf.conf
34+
:::
35+
36+
37+
:::{rubric} Learn
38+
:::
39+
40+
::::{grid}
41+
42+
:::{grid-item-card} Tutorial: Use collectd with CrateDB
43+
:link: collectd-tutorial-base
44+
:link-type: ref
45+
How to configure collectd to submit metrics to CrateDB.
46+
:::
47+
48+
:::{grid-item-card} Tutorial: Use collectd with Telegraf and CrateDB
49+
:link: collectd-tutorial-telegraf
50+
:link-type: ref
51+
How to configure collectd and Telegraf to submit metrics to CrateDB.
52+
:::
53+
54+
::::
55+
56+
57+
:::{toctree}
58+
:maxdepth: 1
59+
:hidden:
60+
Tutorial with collectd <tutorial-collectd>
61+
Tutorial with Telegraf <tutorial-telegraf>
62+
:::
63+
64+
65+
[collectd]: https://collectd.org/

‎docs/integrate/collectd/telegraf.conf‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
[[inputs.socket_listener]]
2+
service_address = "udp://:25826"
3+
4+
## Data format to consume.
5+
## Each data format has its own unique set of configuration options, read
6+
## more about them here:
7+
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
8+
data_format = "collectd"
9+
10+
## Authentication file for cryptographic security levels
11+
# collectd_auth_file = "/etc/collectd/auth_file"
12+
## One of none (default), sign, or encrypt
13+
# collectd_security_level = "encrypt"
14+
## Path of to TypesDB specifications
15+
# collectd_typesdb = ["/usr/share/collectd/types.db"]
16+
17+
## Multi-value plugins can be handled two ways.
18+
## "split" will parse and store the multi-value plugin data into separate measurements
19+
## "join" will parse and store the multi-value plugin as a single multi-value measurement.
20+
## "split" is the default behavior for backward compatibility with previous versions of InfluxDB.
21+
collectd_parse_multivalue = "split"
22+
23+
# Configuration for CrateDB to send metrics to.
24+
[[outputs.cratedb]]
25+
## Connection parameters for accessing the database see
26+
## https://pkg.go.dev/github.com/jackc/pgx/v4#ParseConfig
27+
## for available options
28+
url = "postgres://crate:crate@cratedb/doc?sslmode=disable"
29+
30+
## Timeout for all CrateDB queries.
31+
# timeout = "5s"
32+
33+
## Name of the table to store metrics in.
34+
# table = "metrics"
35+
36+
## If true, and the metrics table does not exist, create it automatically.
37+
table_create = true
38+
39+
## The character(s) to replace any '.' in an object key with
40+
# key_separator = "_"
41+
42+
# Read metrics about cpu usage
43+
[[inputs.cpu]]
44+
## Whether to report per-cpu stats or not
45+
percpu = true
46+
## Whether to report total system cpu stats or not
47+
totalcpu = true
48+
## If true, collect raw CPU time metrics
49+
collect_cpu_time = false
50+
## If true, compute and report the sum of all non-idle CPU states
51+
## NOTE: The resulting 'time_active' field INCLUDES 'iowait'!
52+
report_active = false
53+
## If true and the info is available then add core_id and physical_id tags
54+
core_tags = false

‎docs/integrate/collectd/tutorial-collectd.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
(collectd-tutorial-base)=
2+
# Load data into CrateDB using collectd
3+
4+
This tutorial walks you through configuring and starting the [collectd]
5+
agent and daemon, and CrateDB, to submit and store system metrics into
6+
CrateDB.
7+
8+
## Prerequisites
9+
10+
Docker is used for running all components. This approach works consistently
11+
across Linux, macOS, and Windows. Alternatively, you can use Podman.
12+
13+
### CLI
14+
15+
Prepare shortcut for the psql command.
16+
17+
::::{tab-set}
18+
:sync-group: os
19+
20+
:::{tab-item} Linux and macOS
21+
:sync: unix
22+
To make the settings persistent, add them to your shell profile (`~/.profile`).
23+
```shell
24+
alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql"
25+
```
26+
:::
27+
:::{tab-item} Windows PowerShell
28+
:sync: powershell
29+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
30+
```powershell
31+
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
32+
```
33+
:::
34+
:::{tab-item} Windows Command
35+
:sync: dos
36+
```shell
37+
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
38+
```
39+
:::
40+
41+
::::
42+
43+
### CrateDB
44+
45+
Create a shared network.
46+
```shell
47+
docker network create cratedb-demo
48+
```
49+
50+
Start CrateDB.
51+
```shell
52+
docker run --name=cratedb --rm -it --network=cratedb-demo \
53+
--publish=4200:4200 --publish=5432:5432 \
54+
--env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node
55+
```
56+
57+
## Configure
58+
59+
### Provision database
60+
61+
Create a database table that stores collected metrics.
62+
```shell
63+
psql "postgresql://crate:crate@cratedb:5432/" <<SQL
64+
CREATE TABLE doc.collectd_data (
65+
p_time timestamp with time zone,
66+
p_host TEXT,
67+
p_plugin TEXT,
68+
p_plugin_instance TEXT,
69+
p_type TEXT,
70+
p_type_instance TEXT,
71+
p_value_names TEXT,
72+
p_type_names TEXT,
73+
p_values TEXT,
74+
month GENERATED ALWAYS AS date_trunc('month',p_time)
75+
) PARTITIONED BY (month);
76+
SQL
77+
```
78+
79+
### Build collectd OCI
80+
81+
collectd is not available per OCI image, so either install standalone,
82+
or use these instructions for building an OCI to invoke on Docker or Podman.
83+
Store this file under the name `Dockerfile`, then invoke the command
84+
displayed below.
85+
86+
:::{literalinclude} Dockerfile
87+
:::
88+
```shell
89+
docker build -t local/collectd -f Dockerfile .
90+
```
91+
92+
### Configure collectd
93+
94+
To send the collected data to CrateDB, configure collectd by loading its
95+
[`postgresql` plugin] and supplying settings. Store this file under
96+
the name `collectd-cratedb.conf`.
97+
98+
:::{literalinclude} collectd-cratedb.conf
99+
:::
100+
101+
## Start services
102+
103+
### collectd
104+
105+
Start collectd.
106+
::::{tab-set}
107+
:sync-group: os
108+
109+
:::{tab-item} Linux and macOS
110+
:sync: unix
111+
```shell
112+
docker run --name=collectd --rm -it --network=cratedb-demo \
113+
--volume ${PWD}/collectd-cratedb.conf:/etc/collectd/collectd.conf.d/collectd-cratedb.conf \
114+
local/collectd
115+
```
116+
:::
117+
:::{tab-item} Windows PowerShell
118+
:sync: powershell
119+
```powershell
120+
docker run --name=collectd --rm -it --network=cratedb-demo `
121+
--volume "${PWD}\collectd-cratedb.conf:/etc/collectd/collectd.conf.d/collectd-cratedb.conf" `
122+
local/collectd
123+
```
124+
:::
125+
:::{tab-item} Windows Command
126+
:sync: dos
127+
```shell
128+
docker run --name=collectd --rm -it --network=cratedb-demo ^
129+
--volume "%cd%\collectd-cratedb.conf:/etc/collectd/collectd.conf.d/collectd-cratedb.conf" ^
130+
local/collectd
131+
```
132+
:::
133+
::::
134+
135+
## Explore data
136+
137+
After starting the daemon, the first metrics will appear in the designated table in
138+
CrateDB, ready to be inspected.
139+
```shell
140+
psql "postgresql://crate:crate@cratedb:5432/" -c "SELECT * FROM doc.collectd_data LIMIT 5;"
141+
```
142+
```psql
143+
p_time | p_host | p_plugin | p_plugin_instance | p_type | p_type_instance | p_value_names | p_type_names | p_values | month
144+
----------------------------+--------------+-----------+-------------------+------------+-----------------+---------------+-------------------+--------------+----------------------------
145+
2025-09-20 13:57:12.822+00 | 9cde293016c2 | interface | gre0 | if_errors | | {'rx','tx'} | {'gauge','gauge'} | {nan,nan} | 2025-09-01 00:00:00.000+00
146+
2025-09-20 13:57:12.822+00 | 9cde293016c2 | memory | | memory | cached | {'value'} | {'gauge'} | {4600500224} | 2025-09-01 00:00:00.000+00
147+
2025-09-20 13:57:12.822+00 | 9cde293016c2 | interface | gre0 | if_dropped | | {'rx','tx'} | {'gauge','gauge'} | {nan,nan} | 2025-09-01 00:00:00.000+00
148+
2025-09-20 13:57:12.822+00 | 9cde293016c2 | interface | erspan0 | if_octets | | {'rx','tx'} | {'gauge','gauge'} | {nan,nan} | 2025-09-01 00:00:00.000+00
149+
2025-09-20 13:57:12.822+00 | 9cde293016c2 | interface | ip_vti0 | if_dropped | | {'rx','tx'} | {'gauge','gauge'} | {nan,nan} | 2025-09-01 00:00:00.000+00
150+
(5 rows)
151+
```
152+
153+
154+
[collectd]: https://collectd.org/
155+
[`postgresql` plugin]: https://collectd.org/documentation/manpages/collectd.conf.html#plugin-postgresql

0 commit comments

Comments
 (0)