Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature] Add automated benchmarks in ci pipeline #79

Draft
wants to merge 30 commits into
base: main
Choose a base branch
from
Draft

[feature] Add automated benchmarks in ci pipeline #79

wants to merge 30 commits into from

Conversation

fako1024
Copy link
Owner

@fako1024 fako1024 commented Oct 21, 2023 •
edited
Loading

Things done / still TODO:

  • Setup basic dedicated device in personal infrastructure (Alpine + Docker)
  • Update BIOS of device and disable APU turbo mode
  • Enable kernel level CPU isolation (and isolate cores 1-3 ; leave 0 for OS etc.)
  • Enable realtime scheduler RT_RR and maximize priority for benchmark binary
  • Setup GitHub Runner Docker image for easy deployment on physical host(s)
  • Setup benchmark logic to run benchmarks on isolated cores only (in combination with previous step)
  • Adapt CPU frequency setting to aggressively stay on constant (high) frequency during benchmarks
  • Setup comparative benchmark logic using benchstat
  • Setup pipeline to run benchmarks on PR (and changes to PR)
  • Auto-post benchmark results in PR
  • Handle deviations of benchmark comparison (but only if worse than base branch)
  • Disable logging on host machine & Docker
  • Adapt statistical significance / p-value logic to minimize noise
  • Check / optimize GC impact
  • Check / optimize CPU frequency governor / setting impact

@fako1024
Add automated benchdiff job to CI
89c62d0
@fako1024
Fix and improve logging of benchmark CI job
cbf93be
@fako1024
Increase benchmark significance for CI
1f9dac6
@fako1024
Adapt CI benchmark arguments to avoid timeouts
80b11f4
@fako1024
Hopefully fix incorrect benchmark syntax
21707df
@fako1024
Use self-hosted Github runner to improve benchmark consistency
bef51c8
@fako1024
Reduce scope of benchmark to simplify testing
bb33a18
@fako1024
Improve runner and notification limits
3b6f06b
@fako1024
Attempt to speed up and run all benchmarks again
b85732d
@fako1024
Switch to self-managed benchmark & comparison
71c36e9
@fako1024
Fix Go environment availability
6f28393
@fako1024
Second attempt at preserving the Go environment for benchmark script
77ca9fe
@fako1024
Third attempt at preserving environment
1741bb2
@fako1024
Ensure Go env variables are added
2cc6f4d
@fako1024
Try to preserve env via shell
857b0c0
@fako1024
Use template notation for shell
69bf085
@fako1024
Update GitHub actions and add further debugging info
2dd4d05
@fako1024
Attempt to explicitly export PATH env variable
814a0a2
@fako1024
Store PATH in file to be read in benchmark script
ba45725
@fako1024
Reduce scope of benchmarks for faster debugging
e990c9f
@fako1024
Increase test runtime for another test
5f95b80
@fako1024
Add automatic sticky PR comment
8eae24e
@fako1024 fako1024 added the enhancement New feature or request label Oct 21, 2023
@fako1024 fako1024 self-assigned this Oct 21, 2023
@fako1024 fako1024 linked an issue Oct 21, 2023 that may be closed by this pull request
Add automated benchmarks in CI pipeline #78
Open
3 tasks
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions
Copy link

Benchmark Result

Benchmark diff with base branch 
goos: linux
goarch: amd64
pkg: github.com/fako1024/slimcap/capture/afpacket/afpacket
cpu: AMD GX-412TC SOC                               
                                     │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                     │             sec/op              │   sec/op     vs base                │
CaptureMethods/NextPacket-3                                49.40µ ± 0%   86.60µ ± 1%  +75.30% (p=0.000 n=15)
CaptureMethods/NextPacketInPlace-3                         47.30µ ± 0%   80.74µ ± 1%  +70.69% (p=0.000 n=15)
CaptureMethods/NextIPPacket-3                              50.09µ ± 1%   41.25µ ± 0%  -17.64% (p=0.000 n=15)
CaptureMethods/NextIPPacketInPlace-3                       47.71µ ± 0%   80.82µ ± 1%  +69.38% (p=0.000 n=15)
CaptureMethods/NextPacketFn-3                              47.49µ ± 0%   77.72µ ± 3%  +63.65% (p=0.000 n=15)
geomean                                                    48.39µ        71.06µ       +46.86%

                                     │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                     │              B/op               │    B/op     vs base                 │
CaptureMethods/NextPacket-3                               48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket-3                             32.00 ± 0%     32.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace-3                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn-3                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
geomean                                                              ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                     │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                     │            allocs/op            │ allocs/op   vs base                 │
CaptureMethods/NextPacket-3                               1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket-3                             1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace-3                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn-3                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
geomean                                                              ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/fako1024/slimcap/capture/afpacket/afring
                                                 │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                                 │             sec/op              │   sec/op     vs base                │
CaptureMethods/NextPacket_10485kiBx4-3                                517.9n ±  1%   551.8n ± 1%   +6.55% (p=0.000 n=15)
CaptureMethods/NextPacketInPlace_10485kiBx4-3                         275.2n ± 11%   231.7n ± 6%  -15.81% (p=0.000 n=15)
CaptureMethods/NextPayload_10485kiBx4-3                               454.6n ±  1%   485.6n ± 1%   +6.82% (p=0.000 n=15)
CaptureMethods/NextPayloadInPlace_10485kiBx4-3                        216.8n ± 15%   170.7n ± 0%  -21.26% (p=0.000 n=15)
CaptureMethods/NextPayloadZeroCopy_10485kiBx4-3                      100.30n ± 12%   90.07n ± 1%  -10.20% (p=0.000 n=15)
CaptureMethods/NextIPPacket_10485kiBx4-3                              460.0n ±  2%   488.7n ± 0%   +6.24% (p=0.000 n=15)
CaptureMethods/NextIPPacketInPlace_10485kiBx4-3                       205.8n ±  7%   174.4n ± 0%  -15.26% (p=0.000 n=15)
CaptureMethods/NextIPPacketZeroCopy_10485kiBx4-3                     104.50n ± 12%   91.72n ± 1%  -12.23% (p=0.000 n=15)
CaptureMethods/NextPacketFn_10485kiBx4-3                              143.3n ± 14%   124.1n ± 1%  -13.40% (p=0.000 n=15)
CaptureMethods/NextPacket_10kiBx512-3                                 1.450µ ±  1%   1.540µ ± 1%   +6.21% (p=0.000 n=15)
CaptureMethods/NextPacketInPlace_10kiBx512-3                          1.418µ ±  0%   1.556µ ± 1%   +9.73% (p=0.000 n=15)
CaptureMethods/NextPayload_10kiBx512-3                                1.444µ ±  0%   1.522µ ± 1%   +5.40% (p=0.000 n=15)
CaptureMethods/NextPayloadInPlace_10kiBx512-3                         1.405µ ±  0%   1.526µ ± 2%   +8.61% (p=0.000 n=15)
CaptureMethods/NextPayloadZeroCopy_10kiBx512-3                        1.400µ ±  0%   1.532µ ± 1%   +9.43% (p=0.000 n=15)
CaptureMethods/NextIPPacket_10kiBx512-3                               1.441µ ±  1%   1.542µ ± 1%   +7.01% (p=0.000 n=15)
CaptureMethods/NextIPPacketInPlace_10kiBx512-3                        1.405µ ±  0%   1.541µ ± 1%   +9.68% (p=0.000 n=15)
CaptureMethods/NextIPPacketZeroCopy_10kiBx512-3                       1.397µ ±  0%   1.536µ ± 1%   +9.95% (p=0.000 n=15)
CaptureMethods/NextPacketFn_10kiBx512-3                               1.399µ ±  0%   1.536µ ± 1%   +9.79% (p=0.000 n=15)
geomean                                                               574.3n         572.9n        -0.23%

                                                 │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                                 │              B/op               │    B/op     vs base                 │
CaptureMethods/NextPacket_10485kiBx4-3                                64.00 ± 0%     64.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace_10485kiBx4-3                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayload_10485kiBx4-3                               48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadInPlace_10485kiBx4-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadZeroCopy_10485kiBx4-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket_10485kiBx4-3                              48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace_10485kiBx4-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketZeroCopy_10485kiBx4-3                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn_10485kiBx4-3                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacket_10kiBx512-3                                 64.00 ± 0%     64.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace_10kiBx512-3                          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayload_10kiBx512-3                                48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadInPlace_10kiBx512-3                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadZeroCopy_10kiBx512-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket_10kiBx512-3                               48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace_10kiBx512-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketZeroCopy_10kiBx512-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn_10kiBx512-3                               0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
geomean                                                                          ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                 │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                                                 │            allocs/op            │ allocs/op   vs base                 │
CaptureMethods/NextPacket_10485kiBx4-3                                1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace_10485kiBx4-3                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayload_10485kiBx4-3                               1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadInPlace_10485kiBx4-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadZeroCopy_10485kiBx4-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket_10485kiBx4-3                              1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace_10485kiBx4-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketZeroCopy_10485kiBx4-3                      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn_10485kiBx4-3                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacket_10kiBx512-3                                 1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketInPlace_10kiBx512-3                          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayload_10kiBx512-3                                1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadInPlace_10kiBx512-3                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPayloadZeroCopy_10kiBx512-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacket_10kiBx512-3                               1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketInPlace_10kiBx512-3                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextIPPacketZeroCopy_10kiBx512-3                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
CaptureMethods/NextPacketFn_10kiBx512-3                               0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=15) ¹
geomean                                                                          ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/fako1024/slimcap/link
                        │ /tmp/base_branch_1698413281.out │  /tmp/this_branch_1698413281.out   │
                        │             sec/op              │   sec/op     vs base               │
NewLink/slimcap-3                             426.5µ ± 1%   429.1µ ± 2%       ~ (p=0.935 n=15)
NewLink/net.Interface-3                       276.1µ ± 1%   276.8µ ± 1%       ~ (p=0.838 n=15)
geomean                                       343.2µ        344.6µ       +0.43%

                        │ /tmp/base_branch_1698413281.out │    /tmp/this_branch_1698413281.out    │
                        │              B/op               │     B/op      vs base                 │
NewLink/slimcap-3                            10.22Ki ± 0%   10.22Ki ± 0%       ~ (p=1.000 n=15) ¹
NewLink/net.Interface-3                      9.898Ki ± 0%   9.899Ki ± 0%       ~ (p=0.355 n=15)
geomean                                      10.06Ki        10.06Ki       +0.00%
¹ all samples are equal

                        │ /tmp/base_branch_1698413281.out │   /tmp/this_branch_1698413281.out   │
                        │            allocs/op            │ allocs/op   vs base                 │
NewLink/slimcap-3                              12.00 ± 0%   12.00 ± 0%       ~ (p=1.000 n=15) ¹
NewLink/net.Interface-3                        32.00 ± 0%   32.00 ± 0%       ~ (p=1.000 n=15) ¹
geomean                                        19.60        19.60       +0.00%
¹ all samples are equal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@fako1024 fako1024

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add automated benchmarks in CI pipeline
1 participant
@fako1024