-
Notifications
You must be signed in to change notification settings - Fork 1.3k
msgpack schema for msgpack payloads of cmetrics and ctraces
This is the first specification of observability contexts that are leveraged with cmetrics and ctraces. They are used in observability contexts on Fluent Bit.
This is the first version of observability contexts of specification.
- November 7, 2023: the first release of this spec.
This specification describes the recently implemented observability contexts in Fluent Bit. Fluent Bit v2 implemented capabilities for metrics and traces.
To implement them in Fluent Bit, cmetrics and ctraces provide the functionalities to implement observability. These are used for storing and updating values of observable quantities which are collated by the running Fluent Bit agents.
The keywords "MUST", "MUST NOT", "SHOULD", "SHOULD NOT" and "MAY" in this document are to be interpreted as described in RFC 2119. The following terms are also used:
- msgpack: a light weight binary representation of serialized objects.
- Prometheus: An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database, and modern alerting approach.
- OpenTelemetry: High-quality, ubiquitous, and portable telemetry to enable effective observability.
- cmetrics: a standalone library to create and manipulate metrics in C.
- ctraces: Library to create and manipulate traces in C.
- Hash: HashMap data structure.
Cmetrics is built with C and it provides functionalities to handle quantities that are typed as counter, gauge, untyped, summary, and histogram types. They are based on Prometheus' type system of quantities.
Originally, Prometheus' quantity type system provides counter, gauge, summary, and histogram types.
In more detail, please refer to the documentation for the type system: https://prometheus.io/docs/tutorials/understanding_metric_types/.
The untyped type of metrics is the cmetrics specific type system that provides a least common set of operations for counter and gauge types.
This quantity handling system brings to Prometheus compatible operation of observabilities. Then, this specification will focus the msgpack encoding format on the current cmetrics.
From the cmetrics' source, cmetrics uses to encode the following schema.
In short, the cmetrics' msgpack encoding is compounded by two parts, header and metrics.
The header is always added and metrics MAY able to zero or more than one.
{
HEADER,
METRICS*
}
The header is encoded as:
'meta' => {
'cmetrics' => {
'producer': STRING
},
'external' => { ... },
'processing' => {
'static_labels' => [
[STRING, STRING], ...
]
}
},
This indicates that the header section in meta is composed with three elements:
| name | type | content |
|---|---|---|
| cmetrics | Hash | internal metadata |
| external | Hash | external metadata |
| processing | Hash | Hash contains static labels |
Metrics is built by two parts which are a header part that includes the meta key and a metrics part that includes the values key:
'metrics' => [
{
'meta' => {
'ver' => INTEGER
'type' => INTEGER
'0' = counter
'1' = gauge
'2' = histogram
'3' = summary
'4' = untyped
'opts' => {
'ns' => ns
'subsystem' => subsystem
'name' => name
'description' => description
},
'label_keys' => [STRING, ...],
'buckets' => [n, ...],
'quantiles' => [n, ...]
},
'values' => [
{
'ts' : nanosec timestamp,
'value': float64 value,
'label_values': [STRING, ...],
'histogram':{
'sum': float64,
'count': uint64,
'buckets': [n, ...]
},
'summary': {
'sum': float64,
'count': uint64,
'quantiles': [n, ...],
'quantiles_set': uint64
},
'hash': uint64 value
}
]
}, ...
]
This indicates that the metrics contains actual metrics:
| name | type | content |
|---|---|---|
| metrics | Array | metrics data |
metrics can contains zero or more actual metrics:
| name | type | content |
|---|---|---|
| meta | Hash | metadata of the actual metrics |
| values | Hash | The actual data of metrics |
meta has the following fields:
| name | type | content |
|---|---|---|
| ver | int | metadata version of the actual metrics |
| type | int | show the type of metrics. (counter, gauge, histogram, summary, untyped) |
| opts | Hash | The description for Prometheus compatible metrics |
| label_keys | Hash | The actual data of key for labels |
| buckets | [n, ...] | An array of count and associated upper bounds. meta.type = 2. |
| quantiles | [n, ...] | An array of count and associated upper bounds. meta.type = 3. |
A msgpack payloads of cmetrics MAY contain other keys in meta part in metrics key.
values has the following fields. Ond of the value, histogram, and summary is required.
| name | type | content |
|---|---|---|
| ts | uint64 | timestamp of a metrics |
| value | double | The double number of the actual data of metrics. This field is used for counter, gauge, and untyped type of metrics. |
| label_values | Array | The values of dynamic labels for the actual data of metrics |
| hash | uint64 | The Hash value for the actual data of metrics |
| histogram | Hash | A histogram values. Only used for summary type of metrics. meta.type = 2. |
| summary | Hash | A summary values Only used for histogram type of metrics. meta.type = 3. |
histogram contains the three fields to represent its values:
| name | type | content |
|---|---|---|
| sum | double | A sum of summary |
| count | uint64 | a count of summary values |
| buckets | [n, ...] | An array of count and associated upper bounds. |
summary contains the three fields to represent its values:
| name | type | content |
|---|---|---|
| sum | double | A sum of summary |
| count | uint64 | a count of summary values |
| quantiles | [n, ...] | An array of count and associated upper bounds. |
| quantiles_set | uint64 | Quantive values |
Then, the final combined format is:
{
'meta' => {
'cmetrics' => {
'producer': STRING
},
'external' => { ... },
'processing' => {
'static_labels' => [
[STRING, STRING], ...
]
}
},
'metrics' => [
{
'meta' => {
'ver' => INTEGER
'type' => INTEGER
'0' = counter
'1' = gauge
'2' = histogram
'3' = summary
'4' = untyped
'opts' => {
'ns' => ns
'subsystem' => subsystem
'name' => name
'description' => description
},
'label_keys' => [STRING, ...],
'buckets' => [n, ...],
'quantiles' => [n, ...]
},
'values' => [
{
'ts' : nanosec timestamp,
'value': float64 value,
'label_values': [STRING, ...],
'histogram':{
'sum': float64,
'count': uint64,
'buckets': [n, ...]
},
'summary': {
'sum': float64,
'count': uint64,
'quantiles': [n, ...],
'quantiles_set': uint64
},
'hash': uint64 value
}
]
}, ...
]
}
/*
*
*
* The following fields are metric type specific and are only
* included for histograms :
* meta->buckets
* values[n]->buckets
* values[n]->count
* values[n]->sum
*
* The following fields are metric type specific and are only
* included for summary :
* meta->quantiles
* values[n]->quantiles
* values[n]->quantiles_set
* values[n]->count
* values[n]->sum
*/
This specification is derived from the implementation of msgpack encoder in cmetrics v0.6.4.
Ctraces is built with C and it provides functionalities to handle traces of applications that are used in OpenTelemetry, In more detail, please refer to the documentation for the traces on OpenTelemetry: https://opentelemetry.io/docs/concepts/signals/traces/
These traces handling system brings to OpenTelemerty compatible operation of observabilities. Then, this specification will focus the msgpack encoding format on the current ctraces.
From the ctraces' source, ctraces uses to encode the following schema. ctraces' msgpack schema refers to the definition of protocol buffers on OpenTelemetry: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto
Fluent Bit uses the msgpack format as an internal representation for re-encoding to JSON.
This schema is automatically generated from the protobuf definition. Fluent Bit uses the generated C sources from that definition.
Currently, the generated source was used this revision of protobuf of OpenTelemetry: https://github.com/open-telemetry/opentelemetry-proto/commit/d1468b7700309cec0a3fdfffbfba4e84acf94072