Skip to content

Commit

Permalink
[Refactor] Complete metrics overhaul
Browse files Browse the repository at this point in the history
Metrics got an entire overhaul. Instead of relying on a broken
prometheus library to publish our metrics, we now use the
`tracing` library and with OpenTelemetry that we bind together
then publish into a prometheus library.

Metrics are now mostly derive-macros. This means that the struct
can express what it wants to export and a help text. The library
will choose if it is able to export it.

Tracing now works by calling `.publish()` on the parent structs,
those structs need to call `.publish()` on all the child members
it wishes to publish data about. If a "group" is requested, use
the `group!()` macro, which under-the-hood calls `tracing::span`
with some special labels. At primitive layers, it will call the
`publish!()` macro, which will call `tracing::event!()` macro
under-the-hood with some special fields set. A custom
`tracing::Subscriber` will intercept all the events and spans
and convert them into a json-like object. This object can then
be exported as real json or encoded into other formats like
otel/prometheus.

closes: #1164, #650, #384, #209
towards: #206
  • Loading branch information
allada committed Jul 26, 2024
1 parent 3574149 commit 9711458
Show file tree
Hide file tree
Showing 79 changed files with 2,702 additions and 1,478 deletions.
8 changes: 8 additions & 0 deletions BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ rust_binary(
deps = [
"//nativelink-config",
"//nativelink-error",
"//nativelink-metric",
"//nativelink-metric-collector",
"//nativelink-proto",
"//nativelink-scheduler",
"//nativelink-service",
Expand All @@ -28,16 +30,22 @@ rust_binary(
"@crates//:futures",
"@crates//:hyper",
"@crates//:mimalloc",
"@crates//:opentelemetry",
"@crates//:opentelemetry-prometheus",
"@crates//:opentelemetry_sdk",
"@crates//:parking_lot",
"@crates//:prometheus",
"@crates//:prometheus-client",
"@crates//:rustls-pemfile",
"@crates//:scopeguard",
"@crates//:serde_json",
"@crates//:serde_json5",
"@crates//:tokio",
"@crates//:tokio-rustls",
"@crates//:tonic",
"@crates//:tower",
"@crates//:tracing",
"@crates//:tracing-subscriber",
],
)

Expand Down
210 changes: 203 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 9711458

Please sign in to comment.