Skip to content

Document CI pinning and median-of-5 in benchmarks README#16

Merged
intech merged 1 commit into
mainfrom
docs/benchmarks-ci-pinning-note
Apr 20, 2026
Merged

Document CI pinning and median-of-5 in benchmarks README#16
intech merged 1 commit into
mainfrom
docs/benchmarks-ci-pinning-note

Conversation

@intech
Copy link
Copy Markdown

@intech intech commented Apr 20, 2026

Summary

Adds a short paragraph to `benchmarks/README.md` describing the CI wrapper landed in #15 (run-matrix-ci.sh: taskset -c 0 + median-of-5 + flat 5%/10% gates).

Secondary purpose

Triggers `benchmark.yaml` workflow end-to-end so we verify the post-#15 pinned baseline artifact against a pinned PR run. Expected: all fixtures within ±5% throughput, ±10% memory — previous runs had 6-7 false positives from unpinned runner noise.

🤖 Generated with Claude Code

Adds a one-paragraph note covering the CI wrapper landed in PR #15:
run-matrix-ci.sh wraps bench-matrix in taskset -c 0, captures 5 runs,
and compares the per-fixture median against bench-baseline-main at
flat 5% / 10% gates.

Also serves as a trigger for the benchmark workflow so we can verify
the refreshed pinned baseline artifact against a pinned PR run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

Benchmark: 1 regression(s)

Thresholds: throughput regression >5%, memory regression >10%. Runner pinned to CPU 0 via taskset. Current run on linux/x64, Node v22.22.2, captured 2026-04-20T17:09:46.760Z.
Baseline captured 2026-04-20T15:56:46.953Z on linux/x64, Node v22.22.2.

Summary: 1 regressed, 7 improved, 0 new, 12 unchanged.

Fixture Baseline ops/s PR ops/s Δ ops Baseline B/op PR B/op Δ mem Status
SimpleMessage :: toBinary (pre-built, 19 B) 925,979 895,244 -3.3% ok
ExportTraceRequest (100 spans) :: toBinary (pre-built, 32926 B) 1,137 1,249 +9.9% improved
ExportMetricsRequest (50 series) :: toBinary (pre-built, 17696 B) 2,045 2,192 +7.2% improved
ExportLogsRequest (100 records) :: toBinary (pre-built, 21319 B) 2,009 2,170 +8.0% improved
K8sPodList (20 pods) :: toBinary (pre-built, 28900 B) 2,241 2,373 +5.9% improved
GraphQLRequest :: toBinary (pre-built, 624 B) 186,483 176,832 -5.2% REGRESSION
GraphQLResponse :: toBinary (pre-built, 1366 B) 245,970 244,497 -0.6% ok
RpcRequest :: toBinary (pre-built, 501 B) 311,769 300,665 -3.6% ok
RpcResponse :: toBinary (pre-built, 602 B) 458,123 445,231 -2.8% ok
StressMessage (depth=8, width=200) :: toBinary (pre-built, 12868 B) 7,515 7,909 +5.2% improved
SimpleMessage :: fromBinary (19 B) 1,076,345 1,034,031 -3.9% ok
ExportTraceRequest (100 spans) :: fromBinary (32926 B) 615.3 631.0 +2.6% ok
ExportMetricsRequest (50 series) :: fromBinary (17696 B) 1,173 1,200 +2.3% ok
ExportLogsRequest (100 records) :: fromBinary (21319 B) 1,120 1,120 +0.0% ok
K8sPodList (20 pods) :: fromBinary (28900 B) 1,304 1,418 +8.7% improved
GraphQLRequest :: fromBinary (624 B) 297,876 306,930 +3.0% ok
GraphQLResponse :: fromBinary (1366 B) 276,432 280,420 +1.4% ok
RpcRequest :: fromBinary (501 B) 264,482 272,305 +3.0% ok
RpcResponse :: fromBinary (602 B) 378,898 382,882 +1.1% ok
StressMessage (depth=8, width=200) :: fromBinary (12868 B) 3,817 4,104 +7.5% improved

Produced by benchmarks/scripts/compare-results.ts. Artifacts: bench-results-<pr> (current), bench-baseline-main (baseline).

@intech intech merged commit 91d2544 into main Apr 20, 2026
28 of 29 checks passed
@intech intech deleted the docs/benchmarks-ci-pinning-note branch April 20, 2026 17:25
@intech intech self-assigned this Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant