Add benchmark matrix with realistic fixtures

benchmarks/.gitignore

@@ -2,3 +2,4 @@ src/gen
 src/gen-protobufjs
 node_modules
 dist
+bench-results.json

benchmarks/README.md

@@ -24,6 +24,7 @@ npm run bench:create-toBinary -w @bufbuild/protobuf-benchmarks
 npm run bench:fromBinary -w @bufbuild/protobuf-benchmarks
 npm run bench:fromJson-path -w @bufbuild/protobuf-benchmarks
 npm run bench:comparison -w @bufbuild/protobuf-benchmarks
+npm run bench:matrix -w @bufbuild/protobuf-benchmarks
 npm run bench:memory -w @bufbuild/protobuf-benchmarks
 ```
 
@@ -37,6 +38,7 @@ npm run bench:memory -w @bufbuild/protobuf-benchmarks
 | `bench-fromBinary.ts` | Cost of `fromBinary(Schema, bytes)` on pre-encoded payloads — reflective decoder walk in isolation |
 | `bench-fromJson-path.ts` | `fromJsonString + toBinary` and `fromJson + toBinary` paths on the same fixture. The first one is the #6221 regression shape; the second is the partial-fix midpoint |
 | `bench-comparison-protobufjs.ts` | Cross-library comparison: protobuf-es vs `protobufjs` (pbjs static codegen) on the same `.proto` fixture. Covers full roundtrip, encode-only, decode-only |
+| `bench-matrix.ts` | `toBinary` + `fromBinary` across the full realistic-fixture matrix (OTel traces/metrics/logs, K8s Pod list, GraphQL request/response, RPC envelope, stress). Emits a JSON summary on stdout for CI diffing |
 | `bench-memory.ts` | Heap allocations per operation (`heapUsed` delta after forced GC) for both libraries. Requires `--expose-gc` |
 
 ## Methodology
@@ -51,9 +53,76 @@ npm run bench:memory -w @bufbuild/protobuf-benchmarks
 
 ## Fixtures
 
-The fixture in `proto/nested.proto` is a simplified subset of the [OTLP `ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto) — enough shape (bytes fields, fixed64 timestamps, repeated nested `KeyValue`, two levels of grouping) to be representative of a real export hot path without dragging in the full OpenTelemetry proto dependency graph. The default payload is 100 spans, each with 10 attributes.
-
-`proto/small.proto` is a 3-scalar-field message that isolates per-call overhead (create/toBinary) without allocation noise.
+The suite runs across a matrix of payload shapes so a regression can be
+attributed to a class of workload rather than lumped into a single "encoder
+is slower" result. All fixtures live under `proto/` and are built by
+helpers in `src/fixtures.ts`.
+
+| Fixture | `.proto` | Shape | Typical encoded size | Notes |
+|---------|----------|-------|---------------------:|-------|
+| `SimpleMessage` | `small.proto` | 3 scalar fields | ~19 B | per-call overhead baseline |
+| `ExportTraceRequest` | `nested.proto` | OTel traces: 100 spans × 10 attrs, fixed64 timestamps, bytes IDs | ~35 KB | repro of [open-telemetry/opentelemetry-js#6221](https://github.com/open-telemetry/opentelemetry-js/issues/6221) |
+| `ExportMetricsRequest` | `otel-metrics.proto` | OTel metrics: 50 series with Gauge/Sum/Histogram mix, explicit bucket bounds | ~17 KB | exercises the `oneof data` dispatch + repeated doubles/uint64s |
+| `ExportLogsRequest` | `otel-logs.proto` | OTel logs: 100 LogRecords, severity, string body, trace/span IDs | ~21 KB | string-heavy with attribute maps |
+| `K8sPodList` | `k8s-pod.proto` | 20 Pods with labels/annotations, 2 containers each, ports + env + resource limits | ~29 KB | map-dominant config payload |
+| `GraphQLRequest` | `graphql.proto` | Long query string + JSON-encoded variables map | ~0.6 KB | mixes a large string with `map<string,bytes>` |
+| `GraphQLResponse` | `graphql.proto` | JSON-encoded `data` + structured errors | ~1.4 KB | bytes + repeated messages with string paths |
+| `RpcRequest` | `rpc-simple.proto` | Routing fields + header map + 256-byte payload | ~0.5 KB | baseline RPC envelope |
+| `RpcResponse` | `rpc-simple.proto` | Status + header map + 512-byte payload | ~0.6 KB | baseline RPC response |
+| `StressMessage` | `stress.proto` | Depth-8 self-nested + 200-wide int32/string/attr arrays + 4KB blob + every scalar type | ~13 KB | synthetic — surfaces per-scalar-type regressions |
+
+### Design notes
+
+- **Map-heavy vs. list-heavy.** Kubernetes payloads stress `map<string,string>`
+  encode paths; OTel payloads stress repeated nested messages. Both show up in
+  production consumers and the encoder walks differ.
+- **Deep nesting.** The stress fixture recurses through `StressMessage.child`
+  eight levels deep. The encoder pays a length-prefix per level (fork buffer +
+  measure + prefix), so depth is a distinct failure mode from total size.
+- **All scalar types exactly once.** `StressMessage` declares each proto3
+  scalar in a fixed slot so a regression specific to `sfixed64` or `sint32`
+  varint zig-zag is visible in this fixture but not in realistic ones.
+- **GraphQL/RPC payloads use `bytes` for opaque data** rather than structured
+  sub-messages because real clients carry JSON-encoded variables and
+  opaque RPC payloads as bytes on the wire; the benchmark reflects that.
+
+### Future work
+
+- `bench-matrix` currently measures `toBinary` + `fromBinary` on pre-built
+  messages. A follow-up pass should add `create + toBinary` (full roundtrip)
+  and `fromJsonString + toBinary` paths across the matrix to catch
+  regressions in the JSON-input code paths that the existing
+  `bench-fromJson-path` only exercises on the OTLP traces fixture.
+- GraphQL variables are currently modelled as `map<string,bytes>` with JSON
+  blobs per value. A richer fixture using a `google.protobuf.Struct`-like
+  shape would exercise the same code paths the `@bufbuild/protobuf/wkt`
+  `Value` type uses in real services.
+
+## Report snapshot
+
+Generated by `npm run bench:report -w @bufbuild/protobuf-benchmarks`. The
+script writes `bench-results.json`, `chart.svg`, and the table below. See
+`src/report.ts` for the generator and `src/report-helpers.ts` for the
+rendering helpers.
+
+![chart](./chart.svg)
+
+<!--BENCHMARK_TABLE_START-->
+
+| Fixture                            |  Bytes | toBinary | toBinaryFast | protobufjs | Best                 |
+| ---------------------------------- | -----: | -------: | -----------: | ---------: | -------------------- |
+| SimpleMessage                      |     19 |    1.06M |        1.23M |          - | toBinaryFast (1.16x) |
+| ExportTraceRequest (100 spans)     | 32,926 |    1,778 |        3,218 |      2,676 | toBinaryFast (1.20x) |
+| ExportMetricsRequest (50 series)   | 17,696 |    3,316 |        6,429 |          - | toBinaryFast (1.94x) |
+| ExportLogsRequest (100 records)    | 21,319 |    3,257 |        6,032 |          - | toBinaryFast (1.85x) |
+| K8sPodList (20 pods)               | 28,900 |    3,458 |        5,444 |          - | toBinaryFast (1.57x) |
+| GraphQLRequest                     |    624 |  256,431 |      312,765 |          - | toBinaryFast (1.22x) |
+| GraphQLResponse                    |  1,366 |  293,549 |      402,902 |          - | toBinaryFast (1.37x) |
+| RpcRequest                         |    501 |  379,472 |      455,529 |          - | toBinaryFast (1.20x) |
+| RpcResponse                        |    602 |  579,153 |      766,432 |          - | toBinaryFast (1.32x) |
+| StressMessage (depth=8, width=200) | 12,868 |   10,848 |       18,034 |          - | toBinaryFast (1.66x) |
+
+<!--BENCHMARK_TABLE_END-->
 
 ## Current results
 

benchmarks/package.json

@@ -10,10 +10,12 @@
     "bench:fromJson-path": "tsx src/bench-fromJson-path.ts",
     "bench:fromBinary": "tsx src/bench-fromBinary.ts",
     "bench:comparison": "tsx src/bench-comparison-protobufjs.ts",
+    "bench:matrix": "tsx src/bench-matrix.ts",
     "bench:memory": "node --expose-gc --import tsx src/bench-memory.ts",
+    "bench:report": "tsx src/report.ts",
     "build": "../node_modules/typescript/bin/tsc --noEmit",
     "generate": "buf generate && npm run generate:protobufjs",
-    "generate:protobufjs": "pbjs -t static-module -w commonjs --force-long -o src/gen-protobufjs/nested.cjs proto/nested.proto",
+    "generate:protobufjs": "mkdir -p src/gen-protobufjs && pbjs -t static-module -w commonjs --force-long -o src/gen-protobufjs/nested.cjs proto/nested.proto",
     "format": "biome format --write",
     "license-header": "license-header",
     "lint": "biome lint --error-on-warnings"

benchmarks/proto/graphql.proto

@@ -0,0 +1,57 @@
+// Copyright 2021-2026 Buf Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+package bench.v1;
+
+// GraphQL request/response envelope wrapped in protobuf. This is the shape
+// of a GraphQL-over-gRPC gateway: `query` + `operation_name` + variables
+// (stored as JSON-encoded bytes so the schema can carry arbitrary value
+// types without leaking the GraphQL scalar system into protobuf).
+//
+// Benchmark value: large string payloads (query text), map<string,bytes>
+// (variables), and a nested error array with paths (repeated string).
+
+message GraphQLSourceLocation {
+  uint32 line = 1;
+  uint32 column = 2;
+}
+
+message GraphQLError {
+  string message = 1;
+  repeated GraphQLSourceLocation locations = 2;
+  // GraphQL errors carry a JSON `path` that alternates string keys and
+  // int indices. We flatten to repeated string here (numeric indices are
+  // stringified) because protobuf doesn't have a heterogeneous list type
+  // and this matches what a typed client would serialize over the wire.
+  repeated string path = 3;
+  map<string, bytes> extensions = 4;
+}
+
+message GraphQLRequest {
+  string query = 1;
+  string operation_name = 2;
+  // Variables are JSON-encoded per key so that numbers, strings, nulls,
+  // and nested objects all fit without inventing a sum type. This
+  // matches the pattern used by apollo-rs and graphql-go gateways.
+  map<string, bytes> variables = 3;
+  map<string, string> extensions = 4;
+}
+
+message GraphQLResponse {
+  // `data` is the top-level GraphQL `data` field, JSON-encoded.
+  bytes data = 1;
+  repeated GraphQLError errors = 2;
+  map<string, bytes> extensions = 3;
+}