prototype(protobuf): extend toBinaryFast with map + oneof (+ real OTel fixture)#4
Closed
intech wants to merge 2 commits intofeat/prototype-size-estimatorfrom
Closed
prototype(protobuf): extend toBinaryFast with map + oneof (+ real OTel fixture)#4intech wants to merge 2 commits intofeat/prototype-size-estimatorfrom
intech wants to merge 2 commits intofeat/prototype-size-estimatorfrom
Conversation
Removes map fields and oneof groups from the fast-path fallback blacklist
and encodes both directly using the same two-pass size-estimate-then-write
pattern as the rest of the fast path. The only remaining fallback to the
reflective toBinary is proto2 delimited (group) encoding.
Map fields iterate Object.keys on the runtime plain-object representation
and parse integer/bool string keys back to their typed value before
running the scalar-size / scalar-write helpers. Entry bodies are not
cached — recomputing key+value size per entry is cheap because only the
submessage branch already has a cached size in the sizes map.
Oneof groups are dispatched via desc.oneofs after the regular-field loop.
The ADT shape (message[oneof.localName] = { case, value }) is read
directly; fields with `field.oneof !== undefined` are skipped in the
regular loop so they can't be encoded twice. Crucially, zero-valued
oneof cases are always emitted because presence is carried by the
discriminator, not by the value (new test covers this).
Benchmark fixture updated to the full OTel shape so the measurements
reflect real workload:
- KeyValue.value is now an AnyValue oneof (string / bool / int / bytes /
double), matching opentelemetry.proto.common.v1.AnyValue
- Resource.labels is a map<string,string>, exercising the new map path
- fixture AnyValue distribution: mostly string, some int, some bool,
matching what a real OTLP exporter batches
Measurements (Node 25.8, OTel 100-span full-shape fixture):
| Variant | ops/s | bytes/op |
|--------------------------------|-------|----------|
| create+toBinary (reflective) | 436 | 21,465 |
| create+toBinaryFast | 455 | 19,501 |
| protobufjs create+encode | 2,570 | 47,457 |
| Pre-built encode | ops/s |
|-------------------------------|-------|
| toBinary | 488 |
| toBinaryFast | 494 |
| protobufjs | 2,689 |
Correctness: byte-identical output verified against toBinary on the full
OTel fixture (32,926 bytes). 2,823 existing tests pass plus 16 new
tests covering every legal map K/V combination and every oneof-member
kind (scalar, message, enum) including the zero-valued case.
The throughput gap vs protobufjs on this shape (~5x) is larger than on
the simpler pre-H2 fixture. The richer shape exposes per-entry map
overhead and oneof dispatch that protobufjs amortizes in codegen. Next
hypothesis: codegen an encoder per schema so the field walk disappears
from the hot path. Tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 19, 2026
The package compiles with target ES2017; BigInt literal syntax (`0n`,
`-9007199254740993n`) requires ES2020 and triggers TS2737. Materialize
the bigint zero once at module load with a /*@__PURE__*/ annotation so
tree-shakers can drop it, and construct the 64-bit test literal via
`BigInt("...")` string parse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supported field types after this PR
Scalars (all 15 types), enums, nested messages, repeated scalar (packed + unpacked), repeated message, map (any legal K/V), oneof groups. Only delimited (group) encoding falls through to toBinary.
Correctness
verify-correctness.ts.Results on full-shape fixture (Node 25.8, 100 spans, OTel-like)
The full-shape fixture exposes per-entry map dispatch and oneof walk overhead that the simpler pre-H2 fixture didn't. Fast path still beats reflective on throughput and on allocation; the remaining ~5x gap vs protobufjs is the next hypothesis to test (per-schema codegen vs two-pass reflective walk).
Test plan
Scope
Internal PR within the Connectum-Framework fork, stacked on #3 (toBinaryFast H2 prototype). Upstream submission to bufbuild/protobuf-es is gated on user approval.