MQTT Topic Exporter

Custom Prometheus exporter for MQTT messages. Uses regular expressions to describe how topic and payload should be translated into Prometheus metric labels and value.

Installing and running

Clone repository and install dependencies by running this in project directory:

pip3 install -r requirements.txt

Exporter uses paho-mqtt to interact with MQTT broker and WSGI server from Python standard library for serving metrics.

All settings are taken from configuration file, with only command line options being path to config and verbosity settings:

usage: mqtt_topic_exporter.py [-h] [--config PATH] [--verbose]

Serve MQTT messages on specific topics as Prometheus metrics

optional arguments:
  -h, --help            show this help message and exit
  --config PATH, -c PATH
                        Path to configuration file, default is mqtt_topic_exporter.ini
  --verbose, -v         Show debug output

Rename and adjust configuration template mqtt_topic_exporter.example.ini or consult it for making new config from scratch.

Configuration

Configuration file is build on INI format with [section]s containing multiple key = value pairs. There are two sections with pre-defined names, used to configure connection to MQTT broker and WSGI server providing metrics, the rest can be named arbitrary and is used to describe rules, used to construct metrics from MQTT messages, one per section.

Broker connection section

Must be named [mqtt] and has following parameters:

• host - MQTT broker IP
• port - MQTT broker port, default 1883
• keepalive - MQTT connection keepalive setting
• client_id - must be unique within broker, autogenerated if omitted
• username and password - credentials used to connect to broker. Leave empty for anonimous access
• loglevel - specifies loglevel for paho library, default INFO

Exporter web-server section

Must be named [exporter] and has following parameters:

• bind_address - interface for server to listen on. Set to 0.0.0.0 to allow all interfaces (default) or 127.0.0.1 to only accept connections from localhost
• port - web-server port, default 8842
• metrics_path - url on server to access metrics, default /metrics
• log_path - path to file with web-server access logs. Default is exporter.log in current directory

Since all settings in this section has default values it can be left empty or fully omitted.

Metrics sections

There can be multiple sections with arbitrary names, as long as they are unique within config. Usage of topic wildcards allows one section to set template for multiple topics. Metrics name is static, but one topic can be used in multiple sections with different templates to produce different metrics.

When new MQTT message received, topic gets concatenated with payload separated by value of topic_payload_separator, a single whitespace by default, and then matched with regular expression from topic_payload_pattern. Match groups then used to build labels string (comma-separated list of label_name = "label_value" pairs) from labels_template and topic value from value_template strings, which can contain any text with match groups placeholders (\1, \2 and so on), as long as they form valid labels and value.

By default exporter will generate metric after corresponding MQTT message was received and then continue to serve it until next message on topic arrive. If absense of new messages for certain amount of time means that metric is no longer up to date, no_activity_timeout value in seconds can be set to make metrics that did not receive update within given time to be removed from list.

Generated metrics are not preserved between restarts, so if exporter web-server gets metrics requests before first MQTT message for given topic was received, corresponding metric will not be in response to them.

Metric section properties:

• topic - single topic or topic template with + and # wildcards

• metric_name - Prometheus metric name that will be generated by this section

• metric_type - Prometheus metric type, such as COUNTER or GAUGE

• metric_help - description message for metric, empty if omitted

• topic_payload_separator - used to separate concatenated topic and payload strings

• topic_payload_pattern - regular expression combined topic-separator-payload string will be matched against to extract requred pieces of data

• labels_template and value_template - strings with match groups placeholders that will be used as labels (a single string placed inside {}) and value of metric

• no_activity_timeout - do not generate metric if last message on topic was more than no_activity_timeout seconds ago

• only_float_values - do not generate metric if after applying value_template metric value can't be parsed as float, true by default

Example: replacing ESPHome prometheus component

Sensors values

[mqtt]
host = 127.0.0.1

[exporter]

[sensors state]
topic = esphome_device_topic_prefix/sensor/+/state

metric_name = esphome_sensor_value
metric_type = gauge

topic_payload_pattern = esphome_device_topic_prefix/sensor/([^/]+)/state (.*)
labels_template = id="\1", name="\1"
value_template = \2

no_activity_timeout = 180

Sensors state messages typically has a single floating point value as payload. For sensors with non-numeric values metrics will not be generated.

Switches

There is currently no good way to configure multiple payload substitutions. For replacing values ON/OFF used for switch and binary sensors with 1/0, following workaround can be used: for each of possible values separate copy of section is created, with substituted value hardcoded in topic_payload_pattern and substituting value placed in value_template.

# common [mqtt] and [exporter] sections omitted

[switches ON]
topic = esphome_device_topic_prefix/switch/+/state
metric_name = esphome_switch_value
metric_type = gauge
topic_payload_pattern = esphome_device_topic_prefix/switch/([^/]+)/state ON
labels_template = id="\1", name="\1"
value_template = 1
no_activity_timeout = 180

[switches OFF]
topic = esphome_device_topic_prefix/switch/+/state
metric_name = esphome_switch_value
metric_type = gauge
topic_payload_pattern = esphome_device_topic_prefix/switch/([^/]+)/state OFF
labels_template = id="\1", name="\1"
value_template = 0
no_activity_timeout = 180

mqtt_cmd

Utility script, using too much of mqtt_topic_exporter code to move it in separate repository. Used to monitor MQTT topic and execute shell command if payload matches template.

Usage

usage: mqtt_cmd.py [-h] [--config PATH] [--verbose]

Run pre-defined command when receiving MQTT messages on specific topics

optional arguments:
  -h, --help            show this help message and exit
  --config PATH, -c PATH
                        Path to configuration file, default is mqtt_cmd.ini
  --verbose, -v         Show debug output

Configuration

Uses same format with single section for broker connection details and one section per topic + payload to command mapping.

Broker connection section

Same as for MQTT Topic Exporter: [mqtt]

Command section

• topic - topic that will be subscribed and processed in this section
• payload - regular expression which payload should match in order to be processed. Can be omitted to make any value match
• cmd - shell command to execute