docs(roadmap): split OTLP scope — record shape in MVP, wire endpoints post-MVP - #22
Merged
Merged
Conversation
… post-MVP Operationalises the recommendation from PR #20's investigation finding (`docs/architecture/otlp-log-format.md`) and the §6.1 amendment from PR #21. The pre-amendment roadmap listed "OTLP receiver (gRPC + HTTP)" as a single post-MVP item; the investigation surfaced that the **record shape** is a thesis-gate correctness concern (C2's template-count convergence has to be measured over real OTel record shape, not flat text), while the **wire endpoints** are an independent post-MVP shipping concern. What changed: - §3 last-updated framing reflects PR #20/#21 landing and the §4/§5 OTLP scope split. - §3 missing-pieces table: B1 blocker now mentions `time_unix_nano` carry-through; C2 blocker now includes "miner consumes structured `OtlpLogRecord`". - Phase 1 capabilities now include `MinerCluster::ingest` consuming structured `OtlpLogRecord` per the amended §6.1 (the body_kind = String / Structured fork lands with the §6.2 algorithm rewrite — a follow-on PR). - Phase 2 "Record schema" bullet rewritten to enumerate the amended §6.1 columns (identity + OTLP-derived + body / miner-derived groups). - Phase 3 "ourios-bench" capability clarifies it reads pre-recorded OTLP `LogsData` from disk into an `OtlpLogRecord` stream — no network receiver in MVP. `testdata/corpus/` clarified as serialised OTLP, not flat text. - §5 OTLP row split: replaced "OTLP receiver (gRPC + HTTP)" with "OTLP wire endpoints (gRPC + HTTP listeners)" — the wire-decode layer specifically. Added a "Note on OTLP scope" paragraph explaining the split rationale. Verification: - `mdbook build` clean - Doc-only change; no code, no `cargo`/`clippy`/`test` impact
2 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Doc-only roadmap update splitting the prior single "OTLP receiver" post-MVP item into two: the OTLP record shape (now MVP, required for C2 thesis validity) and OTLP wire endpoints (gRPC/HTTP listeners, remaining post-MVP). Updates §3 missing pieces, §4 phase capabilities, and §5 deferred-capability table accordingly, consistent with PR #20's finding and PR #21's §6.1 amendment.
Changes:
- §3: B1 mentions
time_unix_nano; C2 adds structuredOtlpLogRecordconsumption requirement. - §4: Phase 1 adds
MinerCluster::ingest(OtlpLogRecord); Phase 2 schema enumerates amended §6.1 columns; Phase 3 clarifiesourios-benchreads OTLP from disk andtestdata/corpus/is serialised OTLP. - §5: Replaces "OTLP receiver" row with "OTLP wire endpoints" and adds an explanatory "Note on OTLP scope" paragraph.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
4 tasks
jensholdgaard
added a commit
that referenced
this pull request
May 14, 2026
* docs(rfc-0003): draft RFC 0003 — OTLP receiver Wave-2 follow-up to the OTLP-alignment trajectory (#20 finding, #21 §6.1 amendment, #22 roadmap split, #23 §6.2 algorithm rewrite). The miner now consumes a structured `OtlpLogRecord`; this RFC specifies the wire-decode layer that produces those records. Status: `drafted`. Per `docs/rfcs/README.md` lifecycle, sections §§1-4 + §§7-8 are filled (the requirement for the drafted stage); §5 acceptance criteria are sketched as a stub for the specified-stage move PR; §6 design is filled at proposal level (8 subsections covering the overall shape, wire stack defaults, tenant fan-out, AnyValue canonicalisation boundary, WAL-before-ack sequencing, the `OtlpLogRecord` in-memory shape, backpressure deferral, and out-of-scope concerns). Key design decisions baked into the draft: - Default wire stack is `tonic` (gRPC) + `axum`/`hyper` (HTTP) against the official `opentelemetry-proto` Rust crate. - Both transports converge on the same in-memory `LogsData` representation; business logic is shared. HTTP supports both `application/x-protobuf` and `application/json`. - AnyValue → OTLP-canonical JSON canonicalisation happens at the receiver, not the miner — the miner sees ready-to-template bytes (rationale: hot-path constant-time write, transport knowledge stays at the boundary). - Tenant derivation is per-`ResourceLogs` (matching RFC 0001 §6.1 *Tenant derivation*); a single export can fan out to multiple tenants. All-or-nothing batch rejection on unresolved tenant. - WAL-before-ack sequence is explicit: append → fsync → mine → ack. Crash semantics covered for each gap. Alternatives considered and rejected (§7): - Embed `rotel` as a library — its public API is collector- shaped, not receiver-shaped; embedding brings the full pipeline machinery we don't want. - Out-of-process OTel Collector forwarding — breaks WAL-before-ack at the Collector ack point; remains a *deployment option* but not the default. - Hand-roll protobuf without `opentelemetry-proto` — drift risk vs upstream. - HTTP-only or gRPC-only — the OTel emitter ecosystem is split. - Synchronous canonicalisation in the miner — layering inversion, hot-path cost. Open questions (§9, 9 items): HTTP/JSON support at MVP, canonicalisation cost placement, `dropped_attributes_count` semantics, receiver process model, partial-success response shape, authentication/tenancy interaction, multi-line/non-UTF-8 body handling, compression support, receiver-side telemetry placement. Verification: - `mdbook build` clean (RFC renders under RFCs section) - Doc-only change; no code, no `cargo`/`clippy`/`test` impact Wave 3 (next turn): §5 acceptance-criteria additions to RFC 0001 for the new OTLP fields. The §5 acceptance criteria for *this* RFC (the receiver) come at the specified-stage move PR for RFC 0003 itself, after design review. * docs(rfc-0003): address Copilot review feedback - §1 summary: clarify only `body.kind != AnyValue::String` records are JSON-canonicalised; string bodies pass through as `L_raw`. - §3.1, §6.1, §6.2, §6.5, §4.3: standardise on `ExportLogsServiceRequest` as the post-decode in-memory type; note `LogsData` is the structurally-equivalent file-format message. - §5: renumber acceptance-criteria scenario IDs to the RFC0003.<m> form per `docs/rfcs/README.md` (the §3.4.RFC0003.x compound form was non-conformant). §8 mapping updated to match. - §6.5: drop the misleading `docs/hazards.md` H3 idempotency citation — neither H3 nor H8 covers the single-replica retry case directly. Promoted as an open question in §9. - §6.5: rephrase the "WAL no-ops in MVP" paragraph; the receiver is post-MVP per `roadmap.md` §5 and there is no MVP code path that acks before durability — the §3.4 contract has no escape hatch. - §6.8: schema URLs (`ResourceLogs.schema_url`, `ScopeLogs.schema_url`) are deliberately dropped (RFC 0001 §6.1 has no column for them, no consumer references them); folded the question of adding fields later into §9. - §9: drop the HTTP/JSON open question (resolved — §3.2 is committal); add the retry-dedup and schema-URL questions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
3 tasks
This was referenced Jul 14, 2026
jensholdgaard
added a commit
that referenced
this pull request
Jul 18, 2026
Issue #538 item 2: the per-PR loki-interop job's fixture was stamped ~30s ago, so it never exercised the query-routing regime the real dispatch runs in — a frozen corpus whose entire time range is days old, where Loki's ingester-vs-store routing decides whether unflushed rows are visible. Both characterized dispatch failure modes live there: the L3 trace pair's 0-of-N flicker (runs #20/#22) and the L4 wide-range completeness loss. The new rfc0031_backdated_wide_range_interop test pins that regime per-PR in ~1 minute: nine records spanning ~4 days (12h apart, one trace, two peer-param values of one template), pushed to a Loki running the dispatch's exact flags, then (a) the L3-shaped trace query polled to all 9 rows — a plateau is the routing flake at fixture scale — and (b) the L4-shaped count_over_time matrix polled to completeness and held EXACTLY equal to Ourios's grouped counts (no margin at fixture scale; if the corpus-scale loss ever reproduces down here, the failure is the discovery). The dispatch's Loki flag list is extracted into LOKI_DISPATCH_FLAGS, shared by the dispatch run and this test — the config the cheap test pins is by construction the config the expensive run uses, so drift between them is unrepresentable. Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
jensholdgaard
added a commit
that referenced
this pull request
Jul 18, 2026
Both findings valid, and the better fix for the second was recovering the data: run 29608796312's L4 report DID print before the unrelated L3 panic (storage 3.698x, processed 86.469x, from its job log), so the row now carries real ratios instead of dashes — which also corrects the processed band's floor to 86.5x. Header casing workflow id -> ID. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y
jensholdgaard
added a commit
that referenced
this pull request
Jul 18, 2026
* docs(bench): §9.17 — the L4 frequency-aggregation measurement record The PR #536 arc's outcome folded into the thesis-gates doc: the completeness-shortfall investigation summary (upstream grafana/loki #10658, every harness-side mechanism ruled out, RFC 0031 §7 margin amendment), the measured pair, and the 4-run 3.69-3.73× storage / 86.6-87.1× processed band with per-run completeness. M_L4 stays §7-deferred; the proposed freeze shape lives on #498. Closes the §9-record step for the last unmeasured must-win class. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y * docs(bench): §9.17 review fixes — code-span template, run id, L2 precedent All three review findings were genuine: the template renders as inline code instead of escaped angle brackets; the first measurement row now carries its workflow id (29573249312) so the four-run series is fully auditable; and the M_L4 freeze framing mis-stated the L2 precedent — L2's freeze is processed-primary at 10x PLUS a frozen 1.1x storage-side floor, not storage-informational, and L4's 3.69-3.73x storage band would clear a similar floor with headroom. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y * docs(bench): §9.17 — single-date header, first run's completeness figure Both Copilot findings valid: every listed run is 2026-07-17 (the 07-18 work was follow-on hardening, not measurement), and the first row now carries its real completeness — 1167/1197 = 97.5%, pulled from run 29573249312's job log — instead of a status word in a metric column. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y * docs(bench): §9.17 — attribute the mutation-check evidence to PR #539 "Mutation-tested" read as if a standing mutation-testing harness exists in-repo; the check was a one-time manual verification during #539 (re-introduce each historical comparator bug, confirm a property catches it) whose evidence trail is that PR's record. Reworded to say exactly that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y * docs(bench): §9.17 — run #22's ratios recovered from its log, ID casing Both findings valid, and the better fix for the second was recovering the data: run 29608796312's L4 report DID print before the unrelated L3 panic (storage 3.698x, processed 86.469x, from its job log), so the row now carries real ratios instead of dashes — which also corrects the processed band's floor to 86.5x. Header casing workflow id -> ID. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave-1 follow-up to the OTLP-alignment trajectory (#20 finding + #21 §6.1 amendment). The pre-amendment roadmap treated "OTLP receiver" as a single post-MVP line; the investigation surfaced that the record shape is a thesis-gate correctness concern while the wire endpoints are an independent shipping concern. This PR splits them in §4 and §5.
What changed
time_unix_nanocarry-through; C2 now includes "miner consumes structuredOtlpLogRecord" (the corpus must exercise OTel record shape for C2 to validly measure the thesis).MinerCluster::ingestconsuming structuredOtlpLogRecordper amended §6.1.ourios-bench— clarifies it reads pre-recorded OTLPLogsDatafrom disk into anOtlpLogRecordstream; no network receiver in MVP.testdata/corpus/clarified as serialised OTLP.Verification
mdbook buildcleancargo/clippy/testimpactWave-plan context
This is Wave 1, PR 1 of a three-wave sequence the user approved:
PR 4 (§6.2 algorithm rewrite) opens in the same turn as this PR.
🤖 Generated with Claude Code