Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions crates/ourios-server/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -780,6 +780,18 @@ fn resolve_config(config_path: Option<&Path>) -> Result<ServerConfig, String> {
Ok(config)
}

/// Honor `OTEL_TRACES_EXPORTER` as an on/off switch: `none` disables the
/// traces pipeline (the OTel-standard per-signal off switch); any other value,
/// including unset, leaves it on. Ourios does **not** implement the full
/// exporter *selector* — when traces are on it always exports over OTLP, so
/// e.g. `OTEL_TRACES_EXPORTER=console` is treated as "on" (OTLP), not as a
/// console exporter. We lean on this universal env var rather than a bespoke
/// Ourios config knob (RFC 0038 §3.4); the sampler is likewise the standard
/// `OTEL_TRACES_SAMPLER`, resolved by the SDK.
fn traces_enabled(otel_traces_exporter: Option<&str>) -> bool {
!otel_traces_exporter.is_some_and(|value| value.trim().eq_ignore_ascii_case("none"))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// `--config <path>` selects the RFC 0020 file front-end; without it the
Expand All @@ -800,8 +812,12 @@ async fn main() -> Result<(), Box<dyn Error>> {

// Boot OpenTelemetry first so the compactor's instruments export
// (RFC 0001 §6.8). The guard flushes pending metrics on shutdown;
// OTEL_EXPORTER_OTLP_ENDPOINT et al. tune the exporter.
let telemetry = ourios_telemetry::init(&TelemetryConfig::new("ourios-server"))?;
// OTEL_EXPORTER_OTLP_ENDPOINT et al. tune the exporter, and
// OTEL_TRACES_SAMPLER/_ARG tune the trace sampler (both read by the SDK).
let mut telemetry_config = TelemetryConfig::new("ourios-server");
telemetry_config.traces_enabled =
traces_enabled(std::env::var("OTEL_TRACES_EXPORTER").ok().as_deref());
let telemetry = ourios_telemetry::init(&telemetry_config)?;

#[cfg(unix)]
let sigterm = startup_guards(&config);
Expand Down Expand Up @@ -933,6 +949,23 @@ mod tests {

use ourios_server::config::file::parse;

/// RFC0038.4 — `OTEL_TRACES_EXPORTER` is honored as an on/off switch, not a
/// full exporter selector: `none` (any casing/whitespace) is the only value
/// that disables; unset and every other value (`otlp`, `console`, …) leave
/// traces on — always exported over OTLP.
#[test]
fn rfc0038_4_otel_traces_exporter_none_disables_traces() {
assert!(traces_enabled(None), "unset → on (OTLP)");
assert!(traces_enabled(Some("otlp")), "otlp → on");
assert!(
traces_enabled(Some("console")),
"unsupported selector → still on (OTLP), not off",
);
assert!(!traces_enabled(Some("none")), "none → off");
assert!(!traces_enabled(Some(" none ")), "trimmed none → off");
assert!(!traces_enabled(Some("None")), "case-insensitive none → off");
}

/// A `local` [`StoreConfig`] for `path`, the common test fixture.
fn local(path: &str) -> StoreConfig {
StoreConfig::Local(PathBuf::from(path))
Expand Down
35 changes: 13 additions & 22 deletions crates/ourios-telemetry/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ use opentelemetry_otlp::{LogExporter, MetricExporter, SpanExporter, WithExportCo
use opentelemetry_sdk::Resource;
use opentelemetry_sdk::logs::SdkLoggerProvider;
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider};
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing_subscriber::layer::SubscriberExt as _;
use tracing_subscriber::util::SubscriberInitExt as _;
use tracing_subscriber::{EnvFilter, Layer as _};
Expand Down Expand Up @@ -72,26 +72,24 @@ pub struct TelemetryConfig {
/// `TracerProvider` + `tracing-opentelemetry` layer, so `tracing`
/// spans become `OTel` spans and every log record carries the active
/// span's `trace_id`/`span_id`. `false` restores the logs+metrics-only
/// posture (no tracer, no `trace_id` on logs).
/// posture (no tracer, no `trace_id` on logs). Operators disable via the
/// standard `OTEL_TRACES_EXPORTER=none` (mapped to this flag by the
/// server); the trace **sampler** is the standard `OTEL_TRACES_SAMPLER`
/// env var, resolved by the SDK — not a field here (RFC 0038 §3.4).
pub traces_enabled: bool,
/// Trace sampler (RFC 0038 §3.4). `None` → `parentbased_always_on`
/// (the `OTel` default; the disciplined span count sits far below the
/// ~1000 traces/sec threshold `OTel` says to sample at). `Some(r)` →
/// `parentbased_traceidratio` at ratio `r` in `[0.0, 1.0]`.
pub trace_sample_ratio: Option<f64>,
}

impl TelemetryConfig {
/// Config for `service_name` with spec defaults (default endpoint,
/// [`DEFAULT_EXPORT_INTERVAL`], traces on, always-on sampler).
/// [`DEFAULT_EXPORT_INTERVAL`], traces on; the sampler comes from the
/// SDK's `OTEL_TRACES_SAMPLER` env resolution, default `parentbased_always_on`).
#[must_use]
pub fn new(service_name: impl Into<String>) -> Self {
Self {
service_name: service_name.into(),
otlp_endpoint: None,
export_interval: DEFAULT_EXPORT_INTERVAL,
traces_enabled: true,
trace_sample_ratio: None,
}
}
}
Expand Down Expand Up @@ -293,28 +291,21 @@ pub fn init(config: &TelemetryConfig) -> Result<TelemetryGuard, TelemetryError>
// record through the appender bridge. Also built before installing any
// global (the span exporter is the last fallible step). `None` when
// traces are disabled, which keeps today's logs+metrics posture exactly.
// The sampler is `parentbased_always_on` by default, or
// `parentbased_traceidratio` when a ratio is configured (RFC 0038 §3.4).
// We deliberately do NOT set a sampler: the SDK resolves it from the
// standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` env vars
// (default `parentbased_always_on`), so operators tune sampling with the
// universal OTel knob instead of a bespoke Ourios one (RFC 0038 §3.4). The
// disciplined span count sits far below the ~1000 traces/sec threshold
// OTel says to sample at, so the always-on default is the right baseline.
let (tracer, otel_layer): (Option<SdkTracerProvider>, Option<BoxedLayer>) =
if config.traces_enabled {
let mut builder = SpanExporter::builder().with_tonic();
if let Some(endpoint) = &config.otlp_endpoint {
builder = builder.with_endpoint(endpoint.clone());
}
let span_exporter = builder.build()?;
let sampler = match config.trace_sample_ratio {
// Clamp defensively to the documented `[0.0, 1.0]`. The
// RFC 0038 §3.4 config-file layer rejects an out-of-range
// ratio at startup before it reaches here; this only guards a
// direct library caller from a surprising sampler.
Some(ratio) => Sampler::ParentBased(Box::new(Sampler::TraceIdRatioBased(
ratio.clamp(0.0, 1.0),
))),
None => Sampler::ParentBased(Box::new(Sampler::AlwaysOn)),
};
let tracer_provider = SdkTracerProvider::builder()
.with_batch_exporter(span_exporter)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
.with_sampler(sampler)
.with_resource(resource)
.build();
// Strip `tracing`'s synthetic per-span attributes: `busy_ns`/`idle_ns`
Expand Down
106 changes: 53 additions & 53 deletions docs/rfcs/0038-self-tracing.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
rfc: 0038
title: Self-tracing — the OTel traces signal, disciplined to request scope
status: specified
status: green
author: Jens Holdgaard Pedersen <jens@holdgaard.org>
drafting-assistance: Claude
created: 2026-07-23
Expand All @@ -11,11 +11,15 @@ superseded-by: —

# RFC 0038 — Self-tracing — the OTel traces signal, disciplined to request scope

> **Status: `specified` (2026-07-23).** Sections §§1–4 and the §5 acceptance
> criteria are complete — every hazard/invariant the RFC touches carries a
> scenario, testable in principle (per the `docs/rfcs/README.md` lifecycle,
> a written §5 is the `specified` stage). No implementation has landed — not
> yet `red`/`green`.
> **Status: `green` (2026-07-24).** All six §5 acceptance criteria are
> implemented and pass: RFC0038.1 (request-scope spans + log correlation,
> #614/#615/#616/#617), RFC0038.2 (ingest O(1), #615), RFC0038.3 (spawn-boundary
> context, #617), RFC0038.4 (traces configured via the universal OTel SDK env
> vars — no bespoke Ourios surface — with the `OTEL_TRACES_EXPORTER=none`
> disable mapping tested), RFC0038.5 (loop guard, #614), RFC0038.6 (flush on
> shutdown, #614). §3.4 was amended to lean on the universal OTel env vars
> instead of a bespoke config-file sampler surface (maintainer decision,
> 2026-07-24).

## 1. Summary

Expand Down Expand Up @@ -151,32 +155,29 @@ volume-sensitive span is the per-Export-batch one under heavy ingest, and the
standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` knob (e.g.
`parentbased_traceidratio` at `0.1`) is the operator's lever for exactly that —
Export batches are independent root traces, so ratio-sampling them loses no
cross-request correlation. `TelemetryConfig` gains `traces_enabled: bool`
(default on) and an optional sample ratio; a new `telemetry.*` section in the
RFC 0020 config file
(`traces.enabled`, `traces.sample_ratio`, `otlp.endpoint`) exposes it in the
file front-end — the **first** telemetry config section (telemetry is env-only
today). Traces can be disabled wholesale (`traces.enabled: false`), which
restores today's logs-plus-metrics posture exactly.

**Precedence and failure (the sampler resolution is authoritative and
fail-fast):**

- `telemetry.traces.sample_ratio`, when set in the config file, **selects and
configures** a `parentbased_traceidratio` sampler at that ratio — it does not
merely tune an already-chosen one. It **takes precedence** over the env
sampler (RFC 0020: the file is authoritative when `--config` is given). The
file surface is a ratio only, which maps unambiguously to that one sampler.
- When the file sets no ratio, the standard OTel `OTEL_TRACES_SAMPLER` /
`OTEL_TRACES_SAMPLER_ARG` env vars resolve the sampler (the SDK's own
handling — any standard sampler name). When neither file nor env sets one,
the default is `parentbased_always_on`.
- **Validation.** Ourios validates its own file value: `traces.sample_ratio`
must be a number in `[0.0, 1.0]`; out-of-range or non-numeric **rejects
startup** with a config error, consistent with Ourios's fail-fast validation
of every other config field. The OTel env vars stay the SDK's domain — an
invalid `OTEL_TRACES_SAMPLER`/`_ARG` is logged and ignored by the SDK, which
falls back to its default; Ourios does not alter that behaviour.
cross-request correlation.

**Lean on the universal OTel SDK env vars — no bespoke Ourios config.** These
env vars are the config contract operators already know; inventing a parallel
Ourios surface for the same thing is drift and a second way to configure one
knob. So Ourios configures traces entirely through the standard SDK vars and
does not couple a unique config to them:

- **Sampler:** Ourios does **not** call `.with_sampler(...)`. The SDK resolves
the sampler from `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` (any
standard sampler name; default `parentbased_always_on`). Invalid values are
logged and ignored by the SDK per the env-var spec — Ourios does not add its
own validation or precedence layer.
- **Disable:** the standard per-signal switch `OTEL_TRACES_EXPORTER=none` turns
the traces pipeline off, restoring today's logs-plus-metrics posture exactly
(no tracer, no `trace_id` on logs). The server maps it to
`TelemetryConfig.traces_enabled` (default on), the one programmatic flag the
library keeps; `OTEL_SDK_DISABLED` still disables all three signals together.
- **Endpoint / transport:** `OTEL_EXPORTER_OTLP_ENDPOINT` and the other
`OTEL_EXPORTER_OTLP_*` vars, already read by the SDK exporter.

There is no `telemetry.traces.*` config-file section and no file-vs-env
precedence: the SDK's own env resolution is authoritative.

### 3.5 Span names and attributes

Expand Down Expand Up @@ -262,21 +263,21 @@ Collector expects, and Ourios's whole posture is OTel-native.
> trace — verified by asserting the emitted log's `trace_id` equals the span's
> (the `tokio::spawn` context-loss trap is closed).

> **Scenario RFC0038.4 — sampling is configurable, resolves by a defined
> precedence, and defaults sane.**
> **Given** the standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG`
> knobs and the config file's `telemetry.traces.{enabled,sample_ratio}`,
> **When** the sampler is left unset; set via env `parentbased_traceidratio`
> at a ratio; set via the file `sample_ratio` *and* a conflicting env sampler;
> given an out-of-range file `sample_ratio`; and disabled
> (`traces.enabled: false`),
> **Then** the default samples (root) traces; the env ratio sampler exports the
> configured fraction deterministically; the **file `sample_ratio` wins** over
> the env sampler, selecting `parentbased_traceidratio` at that ratio (§3.4
> precedence); an out-of-range file `sample_ratio` **rejects startup** with a
> config error; and disabling installs **no** tracer and stamps **no**
> `trace_id`/`span_id` on log records — the observable, runtime
> logs-plus-metrics-only behaviour (no throughput change).
> **Scenario RFC0038.4 — traces configure through the universal OTel SDK env
> vars, and disabling is the standard per-signal switch.**
> **Given** the standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` and
> `OTEL_TRACES_EXPORTER` env vars (no bespoke Ourios config surface),
> **When** the sampler is left unset; set via env `parentbased_traceidratio` at
> a ratio; and `OTEL_TRACES_EXPORTER=none`,
> **Then** the default samples (root) traces (`parentbased_always_on`, the SDK
> default — Ourios does **not** override the sampler); the env ratio sampler
> exports the configured fraction (the SDK's own resolution, which Ourios does
> not alter); and `OTEL_TRACES_EXPORTER=none` maps to `traces_enabled=false`, so
> **no** tracer is installed and **no** `trace_id`/`span_id` is stamped on log
> records — the observable, runtime logs-plus-metrics-only behaviour (no
> throughput change). (Sampler resolution and invalid-value handling are the
> SDK's universal, upstream-tested behaviour; Ourios tests only its own mapping
> of `OTEL_TRACES_EXPORTER=none` to the disable path.)

> **Scenario RFC0038.5 — no telemetry-induced-telemetry loop.**
> **Given** the OTLP span exporter's own transport stack (`tonic`/`hyper`/…)
Expand Down Expand Up @@ -307,12 +308,11 @@ Mapped to `CLAUDE.md` §6.2:
(spans are O(1) in N), plus a `criterion` guard on the ingest
(`OTLP → WAL`, `WAL → Parquet`) hot-path benchmarks confirming no
per-record tracing cost — a regression there blocks merge (§6.2 benchmarks).
- **RFC0038.4** — unit/integration over the sampler configuration surface:
default; ratio (deterministic fraction over a fixed set of trace-ids); the
§3.4 precedence (file `sample_ratio` overriding a conflicting env sampler);
invalid-value rejection (an out-of-range file `sample_ratio` fails config
validation at startup); and the `traces.enabled: false` disable path
(asserting no tracer and no `trace_id` on logs).
- **RFC0038.4** — a unit test over Ourios's own mapping: `OTEL_TRACES_EXPORTER`
→ whether the traces pipeline installs (`none` → off; unset / `otlp` / any
other → on). Sampler resolution (`OTEL_TRACES_SAMPLER`/`_ARG`) is the SDK's
universal, upstream-tested behaviour that Ourios no longer overrides — there
is nothing Ourios-specific left to test there.

## 7. Open questions

Expand Down