feat(otlp-transformer): add custom logs protobuf serializer

[pichlermarc]

Which problem is this PR solving?

Currently we run into warnings that CSPs are being violated by protobuf.js (#4987). Also, to convert to protobuf.js' format, we do have to go through an intermediate representation, which is costly in terms of allocations.

This PR moves toward fixing this by introducing a custom protobuf serializer that can be used instead.

Note: my previous PR #6228 had a simple dynamic growing approach. However, since there is a risk that memory behavior will change for the worse from the current protobuf.js-based implementation, which does a double-pass to determine the size of the final message, I opted to also implement a double-pass approach. The first pass determines the buffer size and the second pass actually writes to said buffer. This makes the approach slightly slower than what I proposed in #6228 (also slower than JSON serialization, but faster than the protobuf.js-based approach), but more predictable than the doubling the buffer size or always allocating 64KB, which may not be needed.

Towards fixing:

• use of eval is strongly discouraged #4987 (removing protobuf.js as dependency will get rid of the warning, and will enable use of protobuf exporters in the browser)
  • I also have a draft for a custom deserializer already, I'll open that one once this PR has merged. Since the response is fairly simple compared to the export request, it's relatively easy to parse.
• Stalled large POST requests in opentelemetry-js for web/browsers using exporter-trace-otlp-http #6274 (doing serialization ourselves will eventually allow us to split messages into chunks to avoid hitting client-side and server-side size limits)

Part of:

• Performance improvements #6100 (we can also continue improving with a custom implementation)

Supersedes #6228

Relevant benchmark results:

Old:

transform 512 logs (protobuf) x 502 ops/sec ±0.14% (96 runs sampled)
transform 512 logs (json) x 732 ops/sec ±0.19% (96 runs sampled)

New:

transform 512 logs (protobuf) x 719 ops/sec ±0.20% (97 runs sampled)
transform 512 logs (json) x 734 ops/sec ±0.33% (97 runs sampled)

Disclosure of AI use: I generated the initial prototype of this with GitHub Copilot and Claude Sonnet 4.5, but then applied a whole bunch of optimizations and changes to it to make it fit what we're trying to do here.

Short description of the changes

Type of change

• New feature

How Has This Been Tested?

• Added unit tests
• Existing unit tests
• Existing E2E tests
• Existing benchmarks

[codecov]

Codecov Report

❌ Patch coverage is 98.96104% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.71%. Comparing base (84bfa90) to head (d152412).
⚠️ Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
...ansformer/src/common/protobuf/common-serializer.ts	97.87%	2 Missing ⚠️
...p-transformer/src/logs/protobuf/logs-serializer.ts	98.18%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6390      +/-   ##
==========================================
- Coverage   95.75%   95.71%   -0.04%     
==========================================
  Files         364      369       +5     
  Lines       12095    12461     +366     
  Branches     2884     2945      +61     
==========================================
+ Hits        11581    11927     +346     
- Misses        514      534      +20     

Files with missing lines	Coverage Δ
...mer/src/common/protobuf/protobuf-size-estimator.ts	100.00% <100.00%> (ø)
...transformer/src/common/protobuf/protobuf-writer.ts	100.00% <100.00%> (ø)
...ages/otlp-transformer/src/common/protobuf/utils.ts	100.00% <100.00%> (ø)
...ackages/otlp-transformer/src/logs/protobuf/logs.ts	100.00% <100.00%> (ø)
...ansformer/src/common/protobuf/common-serializer.ts	97.87% <97.87%> (ø)
...p-transformer/src/logs/protobuf/logs-serializer.ts	98.18% <98.18%> (ø)

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[dyladan]

Future enhancements not for this PR

• some things like scope and resource never change. We can cache the results of their serialization and avoid redoing work on future passes
• time_unix_nano, observed_time_unix_nano, severity_number, dropped_attributes_count, flags, trace_id, span_id are always fixed sizes. We can skip them in the size estimation or shortcut them (e.g. no need to calculate the size of a trace_id when we know it is always 16 bytes + 1 byte tag + 1 byte len).

[pichlermarc]

thanks @dyladan for the review. I'll go over the remaining comments on monday :)
the things that I applied for now already yielded additional perf improvements, now we're almost back at JSON levels even though we're doing the double-pass. 🙌

avoiding the string operations for preorganizing the log records is a good call. In practice we do always put the same object on there so it can be used as a key. I changed the comment in sdk-logs to reflect that. people may construct ReadableLogRecords themselves, but we can require them put the same object on there too; it will be needed for #6413 anyway if we want to have any hope of implementing scopeAttributes in a usable way.

[dyladan]

Should we @deprecate or even fully remove this function now?

[pichlermarc]

We still use it during JSON serialization so we still need to keep it around for now.

With this PR some opportunity of improving the JSON path open up since we don't need to support protobuf on the same code-path anymore (removing Encoder comes to mind, it's primary purpose is to facilitate protobuf and JSON on the same code-path), but I'd keep this PR scoped to just protobuf for now.

It'll also be a lot easier to do once we've migrated all signals to a hand-rolled proto implementation; all signals share some common code that can be simplified at once.