Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d029359
chore: remove leftover for output type
lean-apple Oct 14, 2025
b44284e
style: rename output
lean-apple Oct 14, 2025
045e6aa
feat: add new trace org to enable grpc export
lean-apple Oct 14, 2025
2300a24
chore: udpate book
lean-apple Oct 14, 2025
632e481
Merge branch 'otlp-grpc-endpoint' of github.com:lean-apple/reth into …
lean-apple Oct 14, 2025
2d0bdbc
chore: remove unused dep
lean-apple Oct 14, 2025
dd5dc7c
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 14, 2025
af092eb
fix: use async block for grpc export
lean-apple Oct 16, 2025
9c4f13f
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 16, 2025
319e993
fix: fix conflicts on parsing endpoint
lean-apple Oct 16, 2025
4342317
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 16, 2025
6333e50
chore: zepter
lean-apple Oct 16, 2025
3f5efb2
chore: dprint
lean-apple Oct 16, 2025
71db3f0
fix: fix import
lean-apple Oct 16, 2025
ea38253
refactor: use valueenum directly
lean-apple Oct 16, 2025
32a938f
chore: update book
lean-apple Oct 16, 2025
2c28c20
chore: add derive feature to clap
lean-apple Oct 16, 2025
98a9b58
chore: fix merge conflicts on doc
lean-apple Oct 16, 2025
d832e0d
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 17, 2025
679c119
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 20, 2025
311d7d4
refactor: remove opt runner arg
lean-apple Oct 21, 2025
39406c2
Merge branch 'main' into otlp-grpc-endpoint
lean-apple Oct 21, 2025
be76ab5
feat: add env var for traces protocol choice arg
lean-apple Oct 23, 2025
8a28678
docs: explain grpc runtime requirement in init_otlp_export
lean-apple Oct 23, 2025
81097fa
chore: update book
lean-apple Oct 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions crates/ethereum/cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,13 @@ reth-node-ethereum.workspace = true
reth-node-metrics.workspace = true
reth-rpc-server-types.workspace = true
reth-tracing.workspace = true
reth-tracing-otlp.workspace = true
reth-node-api.workspace = true

# misc
clap.workspace = true
eyre.workspace = true
url.workspace = true
tracing.workspace = true

[dev-dependencies]
Expand Down
51 changes: 42 additions & 9 deletions crates/ethereum/cli/src/app.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ use reth_node_ethereum::{consensus::EthBeaconConsensus, EthEvmConfig, EthereumNo
use reth_node_metrics::recorder::install_prometheus_recorder;
use reth_rpc_server_types::RpcModuleValidator;
use reth_tracing::{FileWorkerGuard, Layers};
use reth_tracing_otlp::OtlpProtocol;
use std::{fmt, sync::Arc};
use tracing::info;
use url::Url;

/// A wrapper around a parsed CLI that handles command execution.
#[derive(Debug)]
Expand Down Expand Up @@ -96,7 +98,8 @@ where
self.cli.logs.log_file_directory.join(chain_spec.chain().to_string());
}

self.init_tracing()?;
self.init_tracing(&runner)?;

// Install the prometheus recorder to be sure to record all metrics
let _ = install_prometheus_recorder();

Expand All @@ -106,25 +109,55 @@ where
/// Initializes tracing with the configured options.
///
/// If file logging is enabled, this function stores guard to the struct.
pub fn init_tracing(&mut self) -> Result<()> {
/// For gRPC OTLP, it requires tokio runtime context.
pub fn init_tracing(&mut self, runner: &CliRunner) -> Result<()> {
if self.guard.is_none() {
let mut layers = self.layers.take().unwrap_or_default();

#[cfg(feature = "otlp")]
if let Some(output_type) = &self.cli.traces.otlp {
info!(target: "reth::cli", "Starting OTLP tracing export to {:?}", output_type);
layers.with_span_layer(
"reth".to_string(),
output_type.clone(),
self.cli.traces.otlp_filter.clone(),
)?;
{
self.cli.traces.validate()?;

if let Some(endpoint) = &self.cli.traces.otlp {
info!(target: "reth::cli", "Starting OTLP tracing export to {:?}", endpoint);
self.init_otlp_export(&mut layers, endpoint, runner)?;
}
}

self.guard = self.cli.logs.init_tracing_with_layers(layers)?;
info!(target: "reth::cli", "Initialized tracing, debug log directory: {}", self.cli.logs.log_file_directory);
}
Ok(())
}

/// Initialize OTLP tracing export based on protocol type.
///
/// For gRPC, `block_on` is required because tonic's channel initialization needs
/// a tokio runtime context, even though `with_span_layer` itself is not async.
#[cfg(feature = "otlp")]
fn init_otlp_export(
&self,
layers: &mut Layers,
endpoint: &Url,
runner: &CliRunner,
) -> Result<()> {
let endpoint = endpoint.clone();
let protocol = self.cli.traces.protocol;
let filter_level = self.cli.traces.otlp_filter.clone();

match protocol {
OtlpProtocol::Grpc => {
runner.block_on(async {
layers.with_span_layer("reth".to_string(), endpoint, filter_level, protocol)
})?;
}
OtlpProtocol::Http => {
layers.with_span_layer("reth".to_string(), endpoint, filter_level, protocol)?;
}
}

Ok(())
}
}

/// Run CLI commands with the provided runner, components and launcher.
Expand Down
3 changes: 2 additions & 1 deletion crates/node/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,9 @@ url.workspace = true
dirs-next.workspace = true
shellexpand.workspace = true

# tracing
# obs
tracing.workspace = true
reth-tracing-otlp.workspace = true

# crypto
secp256k1 = { workspace = true, features = ["global-context", "std", "recovery"] }
Expand Down
58 changes: 38 additions & 20 deletions crates/node/core/src/args/trace.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
//! Opentelemetry tracing configuration through CLI args.

use clap::Parser;
use eyre::{ensure, WrapErr};
use eyre::WrapErr;
use reth_tracing::tracing_subscriber::EnvFilter;
use reth_tracing_otlp::OtlpProtocol;
use url::Url;

/// CLI arguments for configuring `Opentelemetry` trace and span export.
#[derive(Debug, Clone, Parser)]
pub struct TraceArgs {
/// Enable `Opentelemetry` tracing export to an OTLP endpoint. Currently
/// only http exporting is supported.
/// Enable `Opentelemetry` tracing export to an OTLP endpoint.
///
/// If no value provided, defaults to `http://localhost:4318/v1/traces`.
/// If no value provided, defaults based on protocol:
/// - HTTP: `http://localhost:4318/v1/traces`
/// - gRPC: `http://localhost:4317`
///
/// Example: --tracing-otlp=http://collector:4318/v1/traces
#[arg(
Expand All @@ -28,6 +30,22 @@ pub struct TraceArgs {
)]
pub otlp: Option<Url>,

/// OTLP transport protocol to use for exporting traces.
///
/// - `http`: expects endpoint path to end with `/v1/traces`
/// - `grpc`: expects endpoint without a path
///
/// Defaults to HTTP if not specified.
#[arg(
long = "tracing-otlp-protocol",
env = "OTEL_EXPORTER_OTLP_PROTOCOL",
global = true,
value_name = "PROTOCOL",
default_value = "http",
help_heading = "Tracing"
)]
pub protocol: OtlpProtocol,

/// Set a filter directive for the OTLP tracer. This controls the verbosity
/// of spans and events sent to the OTLP endpoint. It follows the same
/// syntax as the `RUST_LOG` environment variable.
Expand All @@ -47,25 +65,25 @@ pub struct TraceArgs {

impl Default for TraceArgs {
fn default() -> Self {
Self { otlp: None, otlp_filter: EnvFilter::from_default_env() }
Self {
otlp: None,
protocol: OtlpProtocol::Http,
otlp_filter: EnvFilter::from_default_env(),
}
}
}

// Parses and validates an OTLP endpoint url.
fn parse_otlp_endpoint(arg: &str) -> eyre::Result<Url> {
let mut url = Url::parse(arg).wrap_err("Invalid URL for OTLP trace output")?;

// If the path is empty, we set the path.
if url.path() == "/" {
url.set_path("/v1/traces")
impl TraceArgs {
/// Validate the configuration
pub fn validate(&mut self) -> eyre::Result<()> {
if let Some(url) = &mut self.otlp {
self.protocol.validate_endpoint(url)?;
}
Ok(())
}
}

// OTLP url must end with `/v1/traces` per the OTLP specification.
ensure!(
url.path().ends_with("/v1/traces"),
"OTLP trace endpoint must end with /v1/traces, got path: {}",
url.path()
);

Ok(url)
// Parses an OTLP endpoint url.
fn parse_otlp_endpoint(arg: &str) -> eyre::Result<Url> {
Url::parse(arg).wrap_err("Invalid URL for OTLP trace output")
}
3 changes: 3 additions & 0 deletions crates/optimism/cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ reth-optimism-evm.workspace = true
reth-cli-runner.workspace = true
reth-node-builder = { workspace = true, features = ["op"] }
reth-tracing.workspace = true
reth-tracing-otlp.workspace = true

# eth
alloy-eips.workspace = true
Expand All @@ -55,6 +56,7 @@ alloy-rlp.workspace = true
futures-util.workspace = true
derive_more.workspace = true
serde.workspace = true
url.workspace = true
clap = { workspace = true, features = ["derive", "env"] }

tokio = { workspace = true, features = ["sync", "macros", "time", "rt-multi-thread"] }
Expand Down Expand Up @@ -105,4 +107,5 @@ serde = [
"reth-optimism-primitives/serde",
"reth-primitives-traits/serde",
"reth-optimism-chainspec/serde",
"url/serde",
]
50 changes: 41 additions & 9 deletions crates/optimism/cli/src/app.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ use reth_optimism_consensus::OpBeaconConsensus;
use reth_optimism_node::{OpExecutorProvider, OpNode};
use reth_rpc_server_types::RpcModuleValidator;
use reth_tracing::{FileWorkerGuard, Layers};
use reth_tracing_otlp::OtlpProtocol;
use std::{fmt, sync::Arc};
use tracing::info;
use url::Url;

/// A wrapper around a parsed CLI that handles command execution.
#[derive(Debug)]
Expand Down Expand Up @@ -63,7 +65,8 @@ where
self.cli.logs.log_file_directory.join(chain_spec.chain.to_string());
}

self.init_tracing()?;
self.init_tracing(&runner)?;

// Install the prometheus recorder to be sure to record all metrics
let _ = install_prometheus_recorder();

Expand Down Expand Up @@ -114,23 +117,52 @@ where
/// Initializes tracing with the configured options.
///
/// If file logging is enabled, this function stores guard to the struct.
pub fn init_tracing(&mut self) -> Result<()> {
/// For gRPC OTLP, it requires tokio runtime context.
pub fn init_tracing(&mut self, runner: &CliRunner) -> Result<()> {
if self.guard.is_none() {
let mut layers = self.layers.take().unwrap_or_default();

#[cfg(feature = "otlp")]
if let Some(output_type) = &self.cli.traces.otlp {
info!(target: "reth::cli", "Starting OTLP tracing export to {:?}", output_type);
layers.with_span_layer(
"reth".to_string(),
output_type.clone(),
self.cli.traces.otlp_filter.clone(),
)?;
{
self.cli.traces.validate()?;
if let Some(endpoint) = &self.cli.traces.otlp {
info!(target: "reth::cli", "Starting OTLP tracing export to {:?}", endpoint);
self.init_otlp_export(&mut layers, endpoint, runner)?;
}
}

self.guard = self.cli.logs.init_tracing_with_layers(layers)?;
info!(target: "reth::cli", "Initialized tracing, debug log directory: {}", self.cli.logs.log_file_directory);
}
Ok(())
}

/// Initialize OTLP tracing export based on protocol type.
///
/// For gRPC, `block_on` is required because tonic's channel initialization needs
/// a tokio runtime context, even though `with_span_layer` itself is not async.
#[cfg(feature = "otlp")]
fn init_otlp_export(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure where to put it to avoid the duplicate with op-cli app code

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO we can figure out how to properly dedup tracing + otlp init for OP / eth in a followup

&self,
layers: &mut Layers,
endpoint: &Url,
runner: &CliRunner,
) -> Result<()> {
let endpoint = endpoint.clone();
let protocol = self.cli.traces.protocol;
let level_filter = self.cli.traces.otlp_filter.clone();

match protocol {
OtlpProtocol::Grpc => {
runner.block_on(async {
layers.with_span_layer("reth".to_string(), endpoint, level_filter, protocol)
})?;
Comment on lines +157 to +159
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this isn't an async fn (ie, no await), then why do we need to use the runner and call with block_on? Does the build fn for tonic need to be called in the context of a runtime?

Copy link
Contributor Author

@lean-apple lean-apple Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit tricky, it's linked to the tonic exporter (there is one per observability event type), and its builder.

For example for the span one, when we activate with_tonic() on the span builder, it will call
build_span_exporter,

and then in this function, it will call build_channel, and finally among this function ,

I would say this part Channel::from_shared(), needs a runtime for the tokio executor :
--> Endpoint::new_uri at this level set up.
--> SharedExec::tokio(),

There is also probably the connect_lazy inside build_channel that requires a runtime.

Copy link
Contributor Author

@lean-apple lean-apple Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yet I mainly discovered it was needed because of this error there is no reactor running, must be called from the context of a Tokio 1.x runtime when testing it, with nearly the same error path mentioned just before among the opentelemetry and tonic deps

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, then yeah we can keep this, mind just adding a doc comment explaining this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment in top of init_otlp_export 8a28678

}
OtlpProtocol::Http => {
layers.with_span_layer("reth".to_string(), endpoint, level_filter, protocol)?;
}
}

Ok(())
}
}
Loading
Loading