Add benchmark matrix with realistic fixtures#7
Closed
intech wants to merge 3 commits intofeat/prototype-estimator-map-oneoffrom
Closed
Add benchmark matrix with realistic fixtures#7intech wants to merge 3 commits intofeat/prototype-estimator-map-oneoffrom
intech wants to merge 3 commits intofeat/prototype-estimator-map-oneoffrom
Conversation
5 tasks
intech
changed the title
…logs, stress fixtures Prototype series measured only the OTLP traces fixture plus a 3-field SimpleMessage. Phase 2 of the investigation needs performance readings across the payload classes protobuf-es actually sees in production, so that a regression (or improvement) can be attributed to a class of workload rather than lumped into a single OTLP traces number. New .proto fixtures under benchmarks/proto/: - otel-metrics.proto — OTLP metrics export: Gauge/Sum/Histogram oneof, NumberDataPoint, HistogramDataPoint with explicit bucket bounds. - otel-logs.proto — OTLP logs export: LogRecord batch with severity, string body, trace/span correlation IDs. - k8s-pod.proto — Kubernetes Pod list subset: ObjectMeta with labels/annotations maps, containers with env/ports/resources, container statuses. Map-dominant config payload. - graphql.proto — GraphQL request/response envelope: long query string, map<string,bytes> variables, JSON-in-bytes data + structured errors with paths. - rpc-simple.proto — baseline RPC envelope: routing fields, small headers map, opaque bytes payload. Lower bound on per-call overhead. - stress.proto — synthetic payload: depth-8 self-nesting + 200-wide scalar/string/message arrays + 4 KB blob + every proto3 scalar type exactly once. Surfaces type-specific regressions that OTLP-only fixtures don't hit. New runner in benchmarks/src/bench-matrix.ts: - Runs toBinary + fromBinary across all 10 fixtures (including the existing SimpleMessage and ExportTraceRequest). - Emits the standard tinybench table plus a machine-readable JSON summary on stdout so CI can diff runs without scraping ANSI output. Extends benchmarks/src/fixtures.ts with realistic builders for each shape. Scale knobs (METRICS_SERIES_COUNT, LOGS_RECORD_COUNT, K8S_POD_COUNT, STRESS_DEPTH, STRESS_ARRAY_WIDTH) are exported so a follow-up pass can parameterize across multiple sizes without editing the fixture code. No changes to existing bench-*.ts files or to the protobuf-es runtime. The matrix runner is opt-in via `npm run bench:matrix` and the existing `npm run bench` aggregate is unchanged. Smoke run confirms every fixture encodes and decodes successfully on Node v25.8.1 / linux/x64. Encoded sizes span 19 B (SimpleMessage) to 35 KB (OTLP traces).
Adds report.ts entry + helpers for generating comparative benchmark reports. Inspired by packages/bundle-size/report.ts pattern. - benchmarks/src/report.ts - runs matrix, generates outputs - benchmarks/src/report-helpers.ts - SVG chart + markdown table + README injector - benchmarks/chart.svg - generated comparative chart - benchmarks/README.md - injected table section with markers Run via: npm run bench:report Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
intech
force-pushed
the
feat/benchmark-matrix
branch
from
0852010 to
63b86ef
Compare
- Apply biome format fixes to benchmarks and protobuf files - Prepend `mkdir -p src/gen-protobufjs` to generate:protobufjs so the output directory exists on fresh CI checkouts (pbjs does not create parent directories and failed with ENOENT) Unblocks format, license-header, lint, and attw jobs which all depend on @bufbuild/protobuf-benchmarks#generate succeeding. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
|
Contents merged via PR #6 (head branch was re-pointed to feat/benchmark-matrix before merge). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands benchmark fixture matrix with 6 new realistic schemas. Enables cross-workload performance measurement per tech-lead execution plan Phase 2.
New fixtures (6)
otel-metrics.proto— Gauge / Sum / Histogram with DataPoints + AnyValue attributesotel-logs.proto— LogRecord batch export (100 records)k8s-pod.proto— Pod + PodList subset (20 pods, labels, containers, env vars, resource limits)graphql.proto— Request/Response/Error with extensions maprpc-simple.proto— baseline key-value RPC envelopestress.proto— synthetic edge cases (deep nesting, all scalar types, large blobs)Smoke measurements
Base
Stacked on
feat/prototype-estimator-map-oneof(PR #4) because fixtures reference AnyValue schema introduced there.Scope
Internal PR within the Connectum-Framework fork. Pure additive — existing benchmarks unchanged.
🤖 Generated with Claude Code