docs(rfc-0003): draft RFC 0003 — OTLP receiver - #24
Merged
Conversation
Wave-2 follow-up to the OTLP-alignment trajectory (#20 finding, #21 §6.1 amendment, #22 roadmap split, #23 §6.2 algorithm rewrite). The miner now consumes a structured `OtlpLogRecord`; this RFC specifies the wire-decode layer that produces those records. Status: `drafted`. Per `docs/rfcs/README.md` lifecycle, sections §§1-4 + §§7-8 are filled (the requirement for the drafted stage); §5 acceptance criteria are sketched as a stub for the specified-stage move PR; §6 design is filled at proposal level (8 subsections covering the overall shape, wire stack defaults, tenant fan-out, AnyValue canonicalisation boundary, WAL-before-ack sequencing, the `OtlpLogRecord` in-memory shape, backpressure deferral, and out-of-scope concerns). Key design decisions baked into the draft: - Default wire stack is `tonic` (gRPC) + `axum`/`hyper` (HTTP) against the official `opentelemetry-proto` Rust crate. - Both transports converge on the same in-memory `LogsData` representation; business logic is shared. HTTP supports both `application/x-protobuf` and `application/json`. - AnyValue → OTLP-canonical JSON canonicalisation happens at the receiver, not the miner — the miner sees ready-to-template bytes (rationale: hot-path constant-time write, transport knowledge stays at the boundary). - Tenant derivation is per-`ResourceLogs` (matching RFC 0001 §6.1 *Tenant derivation*); a single export can fan out to multiple tenants. All-or-nothing batch rejection on unresolved tenant. - WAL-before-ack sequence is explicit: append → fsync → mine → ack. Crash semantics covered for each gap. Alternatives considered and rejected (§7): - Embed `rotel` as a library — its public API is collector- shaped, not receiver-shaped; embedding brings the full pipeline machinery we don't want. - Out-of-process OTel Collector forwarding — breaks WAL-before-ack at the Collector ack point; remains a *deployment option* but not the default. - Hand-roll protobuf without `opentelemetry-proto` — drift risk vs upstream. - HTTP-only or gRPC-only — the OTel emitter ecosystem is split. - Synchronous canonicalisation in the miner — layering inversion, hot-path cost. Open questions (§9, 9 items): HTTP/JSON support at MVP, canonicalisation cost placement, `dropped_attributes_count` semantics, receiver process model, partial-success response shape, authentication/tenancy interaction, multi-line/non-UTF-8 body handling, compression support, receiver-side telemetry placement. Verification: - `mdbook build` clean (RFC renders under RFCs section) - Doc-only change; no code, no `cargo`/`clippy`/`test` impact Wave 3 (next turn): §5 acceptance-criteria additions to RFC 0001 for the new OTLP fields. The §5 acceptance criteria for *this* RFC (the receiver) come at the specified-stage move PR for RFC 0003 itself, after design review.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a drafted RFC that specifies the OTLP receiver wire-decode layer (gRPC + HTTP) producing OtlpLogRecords for the miner’s OTLP-aligned ingest contract, and links it into the mdBook navigation.
Changes:
- Add new drafted RFC 0003 describing receiver scope, wire formats, design, alternatives, testing strategy, and open questions.
- Add RFC 0003 to the docs table of contents (
docs/SUMMARY.md).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| docs/SUMMARY.md | Adds mdBook navigation link for the new RFC 0003 document. |
| docs/rfcs/0003-otlp-receiver.md | New drafted RFC specifying the OTLP receiver design and its contract with the miner/WAL. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- §1 summary: clarify only `body.kind != AnyValue::String` records are JSON-canonicalised; string bodies pass through as `L_raw`. - §3.1, §6.1, §6.2, §6.5, §4.3: standardise on `ExportLogsServiceRequest` as the post-decode in-memory type; note `LogsData` is the structurally-equivalent file-format message. - §5: renumber acceptance-criteria scenario IDs to the RFC0003.<m> form per `docs/rfcs/README.md` (the §3.4.RFC0003.x compound form was non-conformant). §8 mapping updated to match. - §6.5: drop the misleading `docs/hazards.md` H3 idempotency citation — neither H3 nor H8 covers the single-replica retry case directly. Promoted as an open question in §9. - §6.5: rephrase the "WAL no-ops in MVP" paragraph; the receiver is post-MVP per `roadmap.md` §5 and there is no MVP code path that acks before durability — the §3.4 contract has no escape hatch. - §6.8: schema URLs (`ResourceLogs.schema_url`, `ScopeLogs.schema_url`) are deliberately dropped (RFC 0001 §6.1 has no column for them, no consumer references them); folded the question of adding fields later into §9. - §9: drop the HTTP/JSON open question (resolved — §3.2 is committal); add the retry-dedup and schema-URL questions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 14, 2026
jensholdgaard
added a commit
that referenced
this pull request
May 14, 2026
The §6.1 amendment (PR #21) and §6.2 algorithm rewrite (PR #23) moved the miner's contract to consume `OtlpLogRecord` and added the `body.kind` fork plus the `(severity_number, scope_name, ...)` template-key tuple — but no §5 acceptance criteria covered the new commitments. RFC 0003 (PR #24) settled the receiver side; this PR closes the corresponding gap on the miner side. Added scenarios: - §5.1 H1.4 — `severity_number` is part of the template key; no silent INFO/ERROR merge. - §5.1 H1.5 — `scope_name` is part of the template key; no cross-scope merge; `None` is its own bucket. - §5.2 §3.7.3 — Tenant derivation runs per `ResourceLogs`, not per export batch. - §5.3 RFC0001.9 — `body_kind = Structured` short-circuits to a structured-template id (locks the §6.2 step-0 fork). - §5.3 RFC0001.10 — `time_unix_nano` preserved verbatim from the wire (gates `docs/benchmarks.md` B1). - §5.3 RFC0001.11 — `severity_number = 0` and `scope_name = None` are distinct key buckets. Scenario IDs follow the existing `H<n>.<m>` / `§3.<n>.<m>` / `RFC0001.<m>` grammar from `docs/rfcs/README.md` (no new namespace introduced). §6.x design content unchanged; this is purely §5 acceptance criteria. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jensholdgaard
added a commit
that referenced
this pull request
May 14, 2026
* docs(rfc-0001): extend §5 with OTLP-receiver acceptance criteria The §6.1 amendment (PR #21) and §6.2 algorithm rewrite (PR #23) moved the miner's contract to consume `OtlpLogRecord` and added the `body.kind` fork plus the `(severity_number, scope_name, ...)` template-key tuple — but no §5 acceptance criteria covered the new commitments. RFC 0003 (PR #24) settled the receiver side; this PR closes the corresponding gap on the miner side. Added scenarios: - §5.1 H1.4 — `severity_number` is part of the template key; no silent INFO/ERROR merge. - §5.1 H1.5 — `scope_name` is part of the template key; no cross-scope merge; `None` is its own bucket. - §5.2 §3.7.3 — Tenant derivation runs per `ResourceLogs`, not per export batch. - §5.3 RFC0001.9 — `body_kind = Structured` short-circuits to a structured-template id (locks the §6.2 step-0 fork). - §5.3 RFC0001.10 — `time_unix_nano` preserved verbatim from the wire (gates `docs/benchmarks.md` B1). - §5.3 RFC0001.11 — `severity_number = 0` and `scope_name = None` are distinct key buckets. Scenario IDs follow the existing `H<n>.<m>` / `§3.<n>.<m>` / `RFC0001.<m>` grammar from `docs/rfcs/README.md` (no new namespace introduced). §6.x design content unchanged; this is purely §5 acceptance criteria. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(miner): add ignored stubs for new §5 OTLP scenarios + §8 mapping Addresses PR #25 review (verification.md §2.3 greppability): every scenario id must appear in a test doc comment so `grep -R "<id>" .` resolves bidirectionally between RFC and tests. New stubs (all `#[ignore]` per the existing Red-gate pattern, each with a `todo!()` pointing at the §6 section the implementation lives in): - crates/ourios-miner/tests/hazards.rs - H1.4 — `severity_number` is part of the template key - H1.5 — `scope_name` is part of the template key - crates/ourios-miner/tests/invariants.rs - §3.7.3 — Tenant derivation runs per `ResourceLogs`. The test's doc comment also clarifies the layer split: this stub asserts the miner-side contract (per-record `tenant_id` correctness once records arrive); the receiver-side derivation test belongs to RFC 0003 §6.3. - crates/ourios-miner/tests/rfc_internal.rs - RFC0001.9 — body_kind = Structured short-circuit - RFC0001.10 — time_unix_nano preserved verbatim - RFC0001.11 — severity = 0 / scope_name = None edge buckets RFC 0001 §8 (Testing strategy) extended with two bullets so the new ids are mapped to techniques in the same place §3.7.1/§3.7.2 and the existing scenarios are mapped: - "Per-`ResourceLogs` tenant derivation (miner-side stub)" — covers §3.7.3, calls out the receiver-side ownership split. - "OTLP-aligned template-key tests" — covers H1.4, H1.5, RFC0001.9, RFC0001.10, RFC0001.11. Verified locally: `cargo fmt --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `cargo test --all-features` (3 passed, 10 ignored, 0 failed), `mdbook build`, and `grep -RIn <id>` resolves to RFC + test + §8 mapping for each new id. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave-2 of the OTLP-alignment trajectory (after #20 finding, #21 §6.1, #22 roadmap, #23 §6.2). The miner now consumes a structured
OtlpLogRecord; this RFC specifies the wire-decode layer that produces those records — the missing producer for the contract the §6.1 amendment and §6.2 algorithm rewrite both require.Status:
drafted. Perdocs/rfcs/README.mdlifecycle this fills §§1–4 + §§7–8 (the drafted-stage requirement); §5 acceptance criteria are sketched as a stub for the specified-stage move PR; §6 design is filled at proposal level (8 subsections).Key design decisions baked into the draft
tonic(gRPC) +axum/hyper(HTTP) against the officialopentelemetry-protoRust crate. Both transports converge on the same in-memoryLogsData; business logic is shared. HTTP supports bothapplication/x-protobufandapplication/json.ResourceLogs(matching RFC 0001 §6.1); a single export can fan out to multiple tenants. All-or-nothing batch rejection on unresolved tenant (per-Resource partial acceptance reserved for a future RFC).partial_success.rejected_log_records. Partial-success reserved for a future RFC.Alternatives considered and rejected (§7)
rotelas a library — its public API is collector-shaped, not receiver-shaped; embedding brings full pipeline machinery we don't want.opentelemetry-proto— drift risk vs upstream.Open questions (§9)
9 items for review/discussion: HTTP/JSON at MVP scope, canonicalisation cost placement,
dropped_attributes_countsemantics, receiver process model, partial-success response shape, auth ↔ tenancy interaction, multi-line/non-UTF-8 body handling, compression, receiver-side telemetry placement.Files
docs/rfcs/0003-otlp-receiver.md(new, ~670 lines)docs/SUMMARY.md— link added under the RFCs sectionVerification
mdbook buildclean (RFC renders under the RFCs section)cargo/clippy/testimpactdocs/rfcs/README.mdschemaWave-plan context
This is Wave 2 of the three-wave sequence the user approved:
Wave 1 (merged): docs(roadmap): split OTLP scope — record shape in MVP, wire endpoints post-MVP #22 (roadmap split) + docs(rfc-0001): rewrite §6.2 algorithm for the OTLP body.kind fork #23 (§6.2 algorithm rewrite)time_unix_nano,severity_number,scope_name,body_kindfork, etc.). Benefits from Wave 1's §6.2 shape having landed.The §5 acceptance criteria for this RFC (the receiver itself) come at the specified-stage move PR for RFC 0003 — after design review.
🤖 Generated with Claude Code