Feedback on using Otel + GreptimeDB + Grafana

[spihiker]

What problem does the new feature solve?

Feedback on using Otel + GreptimeDB + Grafana. Hope GreptimeDB can be improved.

1.When collecting metric data with Otel and storing it in GreptimeDB, the automatically generated tables all have names starting with "system_". Moreover, the official example for integrating with Grafana doesn’t work at all. The tables stored in GreptimeDB don’t even have fields like labels. The YAML configuration needs to be rewritten from scratch.

2.When visualizing data with Grafana, the templates all start with "node_". There are no ready-to-use dashboards; everything has to be rewritten. This makes it very difficult to promote adoption. Currently, many monitoring platforms are built with Grafana + Prometheus or ClickHouse. I hope the integration with Grafana can be made more user-friendly, making it easier for users to migrate to GreptimeDB.

What does the feature do?

I hope the integration with Grafana can be made more user-friendly, making it easier for users to migrate to GreptimeDB.

Implementation challenges

No response

[killme2008]

Thanks for the feedback. Just to clarify: GreptimeDB does not append any prefix to metric names — it maps the metric name directly to the table name.

Could you provide more details about your scenario? For example, what system are the metrics collected from, which agent/collector you are using, and how the data is being written into GreptimeDB. If you can share some sample metrics or configuration, that would be very helpful.

[killme2008]

Based on your description, the system_ prefix most likely comes from the OpenTelemetry Host Metrics Receiver (hostmetricsreceiver), which follows the OTel semantic conventions and produces metrics like system.cpu.time, system.memory.usage, etc. The . is converted to _ when stored in GreptimeDB, resulting in table names like system_cpu_time.

This is fundamentally different from the Prometheus node_exporter naming convention (node_cpu_seconds_total, etc.), which is why existing Grafana dashboards designed for node_exporter don't work out of the box.

Regarding the missing labels field — GreptimeDB maps OTel resource attributes and metric attributes into individual Tag columns rather than a single labels map. So the labels are there, just represented as separate columns (e.g., cpu, state, host_name). You can use them directly in Grafana queries.

Two ways to bridge the gap:

1. Use the OTel Collector Transform Processor (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) to rename metrics to Prometheus conventions before writing to GreptimeDB, so existing Grafana dashboards work unchanged.
2. Use the Prometheus collector and send data via Prometheus Remote Write: https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus/ ， then you can reuse existing Prometheus-based Grafana dashboards. This way metric names and label structures stay fully compatible with standard Grafana templates.

[killme2008]

Forgot to add:

3 . Use OTel-native Grafana dashboards — If you prefer to stay with the OTel pipeline, there are community dashboards designed for OTel hostmetrics naming, e.g.,
https://grafana.com/grafana/dashboards/20376-opentelemetry-collector-hostmetrics-node-exporter/ . These can work as a drop-in replacement for node_exporter dashboards.

The core issue here is the difference between the two ecosystems — OTel and Prometheus have their own metric naming conventions and data models, which leads to incompatible dashboard templates.

[spihiker]

Thank you! When using it, I adjusted the Otel configuration file myself and was able to write labels such as host, job, and instance into the table. However, the official dashboards provided by Grafana, such as Node Exporter Full, use tables prefixed with node_ for metric queries and display. As a result, none of the charts can be displayed.

Create Table: CREATE TABLE IF NOT EXISTS system_cpu_load_average_1m (
greptime_timestamp TIMESTAMP(3) NOT NULL,
greptime_value DOUBLE NULL,
host STRING NULL,
instance STRING NULL,
job STRING NULL,
otel_scope_name STRING NULL,
otel_scope_version STRING NULL,
TIME INDEX (greptime_timestamp),
PRIMARY KEY (host, instance, job, otel_scope_name, otel_scope_version)
)

ENGINE=metric
WITH(
'comment' = 'Created on insertion',
on_physical_table = 'greptime_physical_table'

otelcol-contrib version 0.108.1
grafana-v9.5.18
greptime
branch: main
commit: 13cdfa9
clean: false
version: 1.0.0-rc.2

[killme2008]

Got you.

As mentioned above, the core issue is that OTel and Prometheus use different metric naming conventions and data models, so Prometheus-based Grafana dashboards can’t be used directly with OTel-collected metrics.

1. Use Prometheus Remote Write (recommended if you want to reuse existing Prometheus dashboards)
2. Use OTel-native Grafana dashboards, such as https://grafana.com/grafana/dashboards/23319-otel-hostmetrics/.
3. Use the OTel Transform Processor

[spihiker]

I tried.
1.Exporter configuration in otel
exporters:
debug:
verbosity: detailed

prometheusremotewrite:
endpoint: "http://192.168.1.30:4000/v1/prometheus/write?db=mom"
tls:
insecure: true
headers:
X-Greptime-DB-Name: mom
2.Configuration in grafana
http://192.168.1.30:4000/v1/prometheus?db=mom

But PromQL query statements used in Node exporter full.
node_load1{instance="$node",job="$job"}
node_load5{instance="$node",job="$job"}
node_load15{instance="$node",job="$job"}

So there are no ready-to-use dashboards; everything has to be rewritten.It is hoped that when using Grafana for visualization, dashboards that support GreptimeDB as the backend storage can be shared, such as commonly used dashboards for Linux operating system, MySQL, PostgreSQL, MongoDB, Nginx, Redis, Kafka, etc.

[killme2008]

I think something may be wrong here.

We have many use cases relying on node exporter dashboards, including our cloud service.

Could you share the error reported by Grafana? Also, please check whether the metrics are being written via MySQL(SHOW TABLES) or the embedded GreptimeDB dashboard (http://db-host:4000/dashboard/).

[killme2008]

E.g.

I Just ran the grafana Node Exporter Full on my machine:

• Prometheus config:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9090", "localhost:9100"]

# Prometheus config using Grafana (read-only) account
remote_write:
  - url: http://localhost:4000/v1/prometheus/write?db=public

localhost:9090 refers to the node exporter service.

[spihiker]

I am not using Prometheus. I am collecting metrics via Otel and writing them directly to GreptimeDB, then displaying them through Grafana. Will there be any issues with this setup?

[killme2008]

I am not using Prometheus. I am collecting metrics via Otel and writing them directly to GreptimeDB, then displaying them through Grafana. Will there be any issues with this setup?

As I said in the comment #7887 (comment) and #7887 (comment)

Prometheus-based Grafana dashboards are incompatible with metrics collected via OpenTelemetry due to differing metric names and models.

[shuiyisong]

Hi,
I've managed to view the Otel dashboard from here with the following config:

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      disk: {}
      load: {}
      filesystem: {}
      memory: {}
      network: {}

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
  resourcedetection:
    detectors: [system]
    system:
      hostname_sources: ["os"]

exporters:
  otlphttp:
    endpoint: "http://<$HOST>:4000/v1/otlp"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlphttp]

The setup is simple:

1. Start GreptimeDB
2. Run Otel collector with the above config, which collects host metrics and send them to the GreptimeDB through Otel metrics protocol
3. Run a Grafana and import the dashboard. Should see the charts (some of them).

As @killme2008 mentioned earlier, most dashboards are designed for Prometheus naming conventions, which is not exactly the same as the Otel conventions. But the 24638 Grafana dashboard should give you a good start to modify yours.