feat(ingester): materialize LogRecord into OtlpLogRecord (RFC0003.7–.10) by jensholdgaard · Pull Request #132 · jensholdgaard/ourios

jensholdgaard · 2026-06-06T14:50:26Z

What

Third green slice of the OTLP receiver (RFC 0003 §6.1 steps 2–3): record materialisation, flipping RFC0003.7–.10 live. Builds on the decode layer (#129, #131).

receiver::materialize::materialize_record maps a decoded LogRecord → the flat OtlpLogRecord the miner consumes, inheriting the Resource attributes and InstrumentationScope name/version so downstream never walks the OTLP hierarchy. It consumes the record so the body AnyValue + per-record attributes move (no deep clone of structured trees, per the §6.4 amendment).

Scenarios (now live)

.7 — non-string_value AnyValue → Body::Structured verbatim (table over bool/int/double/bytes/array/kvlist), via Body::from_any_value.
.8 — string_value → Body::String(s) unwrapped byte-for-byte; absent body → None.
.9 — edge fields: severity_number = 0 (UNSPECIFIED) kept as 0, time_unix_nano = 0 (unknown) kept as the u64 0; wire observed_time_unix_nano = 0 and empty scope_name/scope_version narrow to None; nothing coalesced/substituted; inherited resource attributes pass through verbatim.
.10 — dropped_attributes_count reflected verbatim, never recomputed.

Scope

API grown incrementally per the maintainer decision: only the per-record unit the four scenarios exercise. Tenant derivation is a parameter — fan-out + grouping over ResourceLogs (RFC0003.3/.4) build on this next. No new dependencies.

The narrowing helpers (nonzero, nonempty, fixed_len) collapse proto's "empty value = absence" into a single Option/None at the boundary; out-of-range severity_number (invalid OTLP) narrows to 0/UNSPECIFIED.

Verification

cargo test -p ourios-ingester ✓ — RFC0003.5–.10 live; 9 scenarios remain ignored (.1/.2/.3/.4/.11–.15).
cargo fmt --all --check ✓ · cargo clippy --all-targets --all-features -- -D warnings ✓ (workspace)

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Introduced a log record materialization layer that transforms and normalizes incoming data with schema boundary conversions and tenant routing support.
Tests
- Activated comprehensive test suites verifying correct handling of log attributes, body content, and edge case values during materialization.
Documentation
- Updated ingester documentation describing processing stages and maturity levels.

Third green slice of the OTLP receiver (RFC 0003 §6.1 steps 2–3). Adds `receiver::materialize::materialize_record`: the per-record mapping from a decoded `LogRecord` to the flat `OtlpLogRecord` the miner consumes, and flips RFC0003.7–.10 live. The mapping: - body fork via `ourios_core::otlp::Body::from_any_value` — `string_value` → `Body::String` (unwrapped `L_raw`, RFC0003.8), every other variant → `Body::Structured` verbatim (RFC0003.7); absent body → `None`; - narrows proto's empty-value sentinels into `Option`/`None` at the wire boundary (RFC0003.9): wire `observed_time_unix_nano`/empty scope-name/version → `None`, while `severity_number = 0` (UNSPECIFIED) and `time_unix_nano = 0` (unknown) are explicit values kept as-is; - reflects `dropped_attributes_count` verbatim, never recomputed (RFC0003.10); - inherits the `Resource` attributes and `InstrumentationScope` name/version onto each record so downstream never walks the hierarchy. Consumes the `LogRecord` so the body `AnyValue` + per-record attributes move (no deep clone of structured trees, per the §6.4 amendment). API grown incrementally: this slice adds only the per-record unit the four scenarios exercise. Tenant derivation is taken as a parameter — fan-out + grouping over `ResourceLogs` (RFC0003.3/.4) build on this next. No new dependencies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-06-06T14:50:32Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d2bde304-154d-4b30-a2c7-261f64d8198c

📥 Commits

Reviewing files that changed from the base of the PR and between 10c2fc5 and fa1f7b9.

📒 Files selected for processing (7)

crates/ourios-ingester/src/lib.rs
crates/ourios-ingester/src/receiver.rs
crates/ourios-ingester/src/receiver/materialize.rs
crates/ourios-ingester/tests/rfc0003_10_dropped_attributes_count.rs
crates/ourios-ingester/tests/rfc0003_7_body_structured_verbatim.rs
crates/ourios-ingester/tests/rfc0003_8_body_string_lraw.rs
crates/ourios-ingester/tests/rfc0003_9_edge_otlp_fields.rs

📝 Walkthrough

Walkthrough

This PR implements the materialize_record function to convert decoded OTLP LogRecord messages into the flat OtlpLogRecord schema with sentinel conversions and validation, exposes it via the public receiver module, and activates four RFC0003 acceptance tests that verify field mapping, body encoding, and edge-case handling.

Changes

Receiver Materialization

Layer / File(s)	Summary
Core materialize_record implementation and public surface `crates/ourios-ingester/src/lib.rs`, `crates/ourios-ingester/src/receiver.rs`, `crates/ourios-ingester/src/receiver/materialize.rs`	Updated crate documentation to name the materialize step; exposed `materialize` module and `materialize_record` function; implemented field-by-field conversion from OTLP `LogRecord` to `OtlpLogRecord` with schema boundary conversions (observed time `0` → `None`, severity out-of-range → `0`, empty strings → `None`, fixed-length ID validation); added helper functions for proto-to-schema sentinel conversions.
RFC0003 acceptance tests for materialize behavior `crates/ourios-ingester/tests/rfc0003_7_body_structured_verbatim.rs`, `crates/ourios-ingester/tests/rfc0003_8_body_string_lraw.rs`, `crates/ourios-ingester/tests/rfc0003_9_edge_otlp_fields.rs`, `crates/ourios-ingester/tests/rfc0003_10_dropped_attributes_count.rs`	Converted four previously ignored test stubs into active tests: RFC0003.7 verifies non-string `AnyValue` bodies carry verbatim in `Body::Structured`; RFC0003.8 verifies string bodies → `Body::String` and absent bodies → `None`; RFC0003.9 tests severity normalization and time/scope field coercion; RFC0003.10 verifies `dropped_attributes_count` reflected unchanged.

Sequence Diagram

sequenceDiagram
  participant Input as LogRecord
  participant Materialize as materialize_record
  participant Helpers as sentinels
  participant Output as OtlpLogRecord

  Input->>Materialize: LogRecord with metadata
  Materialize->>Helpers: observed_time_unix_nano
  Helpers-->>Materialize: None if zero
  Materialize->>Helpers: severity_number
  Helpers-->>Materialize: u8 clamped at boundaries
  Materialize->>Helpers: scope name and version
  Helpers-->>Materialize: None if empty string
  Materialize->>Helpers: body AnyValue
  Helpers-->>Materialize: Body variant
  Materialize->>Helpers: trace and span ids
  Helpers-->>Materialize: validated fixed length
  Materialize->>Output: fully coerced record

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

jensholdgaard/ourios#128: Main green-lift PR that implements receiver::materialize::materialize_record and converts RFC0003.7/8/9/10 red-gate stubs into active acceptance tests.
jensholdgaard/ourios#101: Establishes the ourios-ingester scaffold that this PR extends by adding the materialize module and public re-exports.

Poem

🐰 Records now flow through materialize's gate,
Converting OTLP to the schema state—
Sentinels coerce with careful care,
While tests confirm the mappings fair.
RFC0003 accepts what engineers declare! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: adding the materialize layer that maps LogRecord to OtlpLogRecord, with RFC references (RFC0003.7–.10) indicating the scope.
Description check	✅ Passed	The description covers the PR purpose, builds on prior work, explains the four scenarios (RFC0003.7–.10), clarifies API scope, mentions verification steps, and includes testing status. However, it does not explicitly check off the provided template items (cargo fmt, clippy, tests, docs/changelog, RFC link).
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/otlp-receiver-materialize

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jensholdgaard · 2026-06-06T14:51:35Z

@coderabbitai review

coderabbitai · 2026-06-06T14:51:40Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

Implements the “record materialisation” slice of the OTLP receiver by mapping decoded OTLP LogRecord values into the flattened ourios_core::otlp::OtlpLogRecord shape consumed by the miner, and turns RFC0003.7–.10 acceptance tests from ignored stubs into live tests.

Changes:

Added receiver::materialize::materialize_record to materialize a decoded LogRecord into OtlpLogRecord (body fork, sentinel narrowing, passthrough fields).
Updated receiver module exports and crate docs to reflect the new materialization layer.
Replaced RFC0003.7–.10 red-gate test stubs with active assertions.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/ourios-ingester/src/receiver/materialize.rs	New materialization mapping from decoded OTLP `LogRecord` to flat `OtlpLogRecord`.
crates/ourios-ingester/src/receiver.rs	Exposes the new `materialize` module and re-exports `materialize_record`.
crates/ourios-ingester/src/lib.rs	Updates crate-level docs to list materialization as landed functionality.
crates/ourios-ingester/tests/rfc0003_7_body_structured_verbatim.rs	Makes RFC0003.7 assertions live for non-string `AnyValue` → `Body::Structured`.
crates/ourios-ingester/tests/rfc0003_8_body_string_lraw.rs	Makes RFC0003.8 assertions live for string body unwrapping + absent body.
crates/ourios-ingester/tests/rfc0003_9_edge_otlp_fields.rs	Makes RFC0003.9 assertions live for edge field narrowing/passthrough.
crates/ourios-ingester/tests/rfc0003_10_dropped_attributes_count.rs	Makes RFC0003.10 assertion live for `dropped_attributes_count` passthrough.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jensholdgaard · 2026-06-06T15:08:47Z

+        // `UNSPECIFIED` (`0`) is an explicit OTLP value, kept as `0`
+        // (RFC0003.9). Proto's `i32` is narrowed to the schema's `u8`;
+        // an out-of-range/invalid value narrows to `0`/UNSPECIFIED.
+        severity_number: u8::try_from(record.severity_number).unwrap_or(0),
+        severity_text: nonempty(record.severity_text),


Good catch — fixed. The narrowing now enforces the OtlpLogRecord 0..=24 contract: severity_to_u8 maps anything outside 0..=24 (the u8-representable 25..=255, negatives, and >255) to 0/UNSPECIFIED, while valid values incl. 0 and 24 are preserved. The misleading "out-of-range/invalid narrows to 0" comment is corrected, and a new test pins 0/24/25/1000/-5. This keeps the downstream template-key + Parquet assumptions valid, and is consistent with RFC0003.9 (it normalises invalid input rather than coalescing valid values).

The prior narrowing (`u8::try_from(i32).unwrap_or(0)`) only mapped values that don't fit a `u8` to 0; invalid-but-representable `25..=255` passed through, violating the documented `OtlpLogRecord` severity contract (`0..=24`) that the miner's template key and the Parquet schema rely on. Add `severity_to_u8`, which narrows anything outside `0..=24` — `25..=255`, negative, or `> 255` — to `0`/UNSPECIFIED. Valid values (incl. `0` UNSPECIFIED and `24` FATAL4) are preserved. New test covers `0/24/25/1000/-5`. Surfaced in review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

jensholdgaard · 2026-06-06T15:16:17Z

+        scope_name: scope.and_then(|s| nonempty(s.name.clone())),
+        scope_version: scope.and_then(|s| nonempty(s.version.clone())),


Fixed in fa1f7b9 — scope_name/scope_version now clone only on the non-empty path ((!s.name.is_empty()).then(|| s.name.clone())), so an empty scope name no longer allocates a string just to discard it.

jensholdgaard · 2026-06-06T15:16:18Z

+/// Proto scalar `0` → `None`, else `Some` — the §6.9-style narrowing of
+/// a "0 = unset" wire sentinel.


Fixed in fa1f7b9 — the nonzero doc now references RFC0003.9 instead of the stray "§6.9".

… ref Two review nits, non-behavioral: - `scope_name`/`scope_version` checked `is_empty` on a clone then discarded it when empty; now clone only on the non-empty path. - `nonzero`'s doc referenced "§6.9-style" narrowing; the relevant reference is RFC0003.9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

jensholdgaard requested a review from Copilot June 6, 2026 14:51

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 14:51 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard requested a review from Copilot June 6, 2026 15:08

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 15:08 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard requested a review from Copilot June 6, 2026 15:16

Copilot started reviewing on behalf of jensholdgaard June 6, 2026 15:16 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jensholdgaard merged commit 5ae5435 into main Jun 6, 2026
12 checks passed

jensholdgaard deleted the feat/otlp-receiver-materialize branch June 6, 2026 15:24

This was referenced Jun 6, 2026

feat(ingester): tenant derivation + fan-out (RFC0003.3/.4) #133

Merged

docs(rfc-0003): advance status specified → green #138

Merged

This was referenced Jun 20, 2026

feat: rfc0018.1/.2 green — persist scope attributes + schema_url #268

Merged

feat: rfc0018.6 green — preserve out-of-range severity + error.type #269

Merged

feat(rfc0017.3): query-time body rendering (LogBody three zones) #275

Merged

Navigation Menu

feat(ingester): materialize LogRecord into OtlpLogRecord (RFC0003.7–.10) - #132

feat(ingester): materialize LogRecord into OtlpLogRecord (RFC0003.7–.10)#132
jensholdgaard merged 3 commits into
mainfrom
feat/otlp-receiver-materialize

jensholdgaard commented Jun 6, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Poem

Uh oh!

jensholdgaard commented Jun 6, 2026

Uh oh!

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

jensholdgaard Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		scope_name: scope.and_then(\|s\| nonempty(s.name.clone())),
		scope_version: scope.and_then(\|s\| nonempty(s.version.clone())),

		/// Proto scalar `0` → `None`, else `Some` — the §6.9-style narrowing of
		/// a "0 = unset" wire sentinel.

Conversation

jensholdgaard commented Jun 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Scenarios (now live)

Scope

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Poem

Uh oh!

jensholdgaard commented Jun 6, 2026

Uh oh!

coderabbitai Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

jensholdgaard Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jensholdgaard commented Jun 6, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading

coderabbitai Bot commented Jun 6, 2026 •

edited

Loading