Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SuperNova CompressedSNARK benchmark #167

Closed
porcuquine opened this issue Dec 7, 2023 · 2 comments
Closed

SuperNova CompressedSNARK benchmark #167

porcuquine opened this issue Dec 7, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@porcuquine
Copy link
Contributor

This is to track the benchmark introduced in #131 and was run at this commit: 1e1658b

Note that a verification error and/or failure occurred on the largest circuit size below.

➜  arecibo git:(batched_spartan) cargo criterion --bench compressed-snark-supernova
   Compiling nova-snark v0.31.0 (/Users/clwk/fil/arecibo)
    Finished bench [optimized] target(s) in 19.00s
Gnuplot not found, using plotters backend
Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-0/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.1s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-0/Prove                                                                          
                        time:   [804.57 ms 812.42 ms 822.02 ms]
CompressedSNARKSuperNova-1circuit-StepCircuitSize-0/Verify                                                                           
                        time:   [32.799 ms 33.233 ms 33.534 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-6565/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 11.6s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-6565/Prove                                                                          
                        time:   [1.1734 s 1.1825 s 1.1928 s]
CompressedSNARKSuperNova-1circuit-StepCircuitSize-6565/Verify                                                                           
                        time:   [47.941 ms 48.262 ms 48.729 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-22949/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 19.4s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-22949/Prove                                                                          
                        time:   [1.9019 s 1.9187 s 1.9358 s]
CompressedSNARKSuperNova-1circuit-StepCircuitSize-22949/Verify                                                                           
                        time:   [64.626 ms 65.357 ms 66.119 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-55717/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 33.7s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-55717/Prove                                                                          
                        time:   [3.3087 s 3.3360 s 3.3650 s]
Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-55717/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 5.7s or enable flat sampling.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-55717/Verify                                                                          
                        time:   [102.73 ms 103.94 ms 104.77 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-121253/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 62.4s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-121253/Prove                                                                          
                        time:   [6.2778 s 6.3212 s 6.3741 s]
CompressedSNARKSuperNova-1circuit-StepCircuitSize-121253/Verify                                                                          
                        time:   [185.32 ms 187.87 ms 190.58 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-252325/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 121.6s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-252325/Prove                                                                          
                        time:   [12.183 s 12.275 s 12.359 s]
CompressedSNARKSuperNova-1circuit-StepCircuitSize-252325/Verify                                                                          
                        time:   [367.57 ms 371.06 ms 376.51 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-514469/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 244.5s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-514469/Prove                                                                          
                        time:   [24.369 s 24.774 s 25.280 s]
Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-514469/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 6.5s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-514469/Verify                                                                          
                        time:   [633.70 ms 652.49 ms 672.07 ms]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-1038757/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 464.0s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-1038757/Prove                                                                          
                        time:   [46.263 s 46.423 s 46.674 s]

Benchmarking CompressedSNARKSuperNova-1circuit-StepCircuitSize-1038757/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 13.0s.
CompressedSNARKSuperNova-1circuit-StepCircuitSize-1038757/Verify                                                                          
                        time:   [1.2770 s 1.2878 s 1.3057 s]

res failed NovaError(ProofVerifyError)
thread 'main' panicked at benches/compressed-snark-supernova.rs:251:9:
assertion failed: res.is_ok()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Criterion.rs benchmark target compressed-snark-supernova exited with exit status: 101
@porcuquine porcuquine added the bug Something isn't working label Dec 7, 2023
@porcuquine
Copy link
Contributor Author

For comparison, here is the regular Nova CompressedSNARK benchmark. As expected, we see slightly better performance than in the SuperNova case on an equivalent single-circuit proof.

This comparison is primarily intended as a sanity check that the SuperNova CompressedSNARK does not add catastrophic overhead or otherwise tank proving/verification performance.

➜  arecibo git:(batched_spartan) cargo criterion --bench compressed-snark          
   Compiling nova-snark v0.31.0 (/Users/clwk/fil/arecibo)
    Finished bench [optimized] target(s) in 9.25s
Gnuplot not found, using plotters backend
Benchmarking CompressedSNARK-StepCircuitSize-0/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 7.5s.
CompressedSNARK-StepCircuitSize-0/Prove                                                                          
                        time:   [746.97 ms 753.80 ms 764.46 ms]
                        change: [-6.4100% -4.8859% -3.1560%] (p = 0.00 < 0.05)
                        Performance has improved.
CompressedSNARK-StepCircuitSize-0/Verify                                                                           
                        time:   [27.983 ms 28.055 ms 28.142 ms]
                        change: [-4.0002% -3.1284% -2.2740%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking CompressedSNARK-StepCircuitSize-6565/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 11.3s.
CompressedSNARK-StepCircuitSize-6565/Prove                                                                          
                        time:   [1.1193 s 1.1261 s 1.1334 s]
                        change: [-6.7602% -4.9133% -3.1794%] (p = 0.00 < 0.05)
                        Performance has improved.
CompressedSNARK-StepCircuitSize-6565/Verify                                                                           
                        time:   [41.404 ms 41.881 ms 42.832 ms]
                        change: [-6.1499% -3.7336% -0.9702%] (p = 0.02 < 0.05)
                        Change within noise threshold.

Benchmarking CompressedSNARK-StepCircuitSize-22949/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 18.3s.
CompressedSNARK-StepCircuitSize-22949/Prove                                                                          
                        time:   [1.8469 s 1.8765 s 1.9117 s]
                        change: [-6.2170% -2.7697% +0.6235%] (p = 0.16 > 0.05)
                        No change in performance detected.
CompressedSNARK-StepCircuitSize-22949/Verify                                                                           
                        time:   [60.269 ms 60.630 ms 61.237 ms]
                        change: [-6.0960% -4.1554% -2.0358%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking CompressedSNARK-StepCircuitSize-55717/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 34.7s.
CompressedSNARK-StepCircuitSize-55717/Prove                                                                          
                        time:   [3.3189 s 3.3660 s 3.4138 s]
                        change: [-6.4090% -3.3243% -0.1846%] (p = 0.07 > 0.05)
                        No change in performance detected.
Benchmarking CompressedSNARK-StepCircuitSize-55717/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 5.6s or enable flat sampling.
CompressedSNARK-StepCircuitSize-55717/Verify                                                                          
                        time:   [100.16 ms 101.00 ms 102.30 ms]
                        change: [-8.0060% -6.3649% -4.7889%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking CompressedSNARK-StepCircuitSize-121253/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 68.9s.
CompressedSNARK-StepCircuitSize-121253/Prove                                                                          
                        time:   [6.2498 s 6.2791 s 6.3138 s]
                        change: [-5.0538% -3.5036% -2.2909%] (p = 0.00 < 0.05)
                        Performance has improved.
CompressedSNARK-StepCircuitSize-121253/Verify                                                                          
                        time:   [180.18 ms 182.69 ms 185.41 ms]
                        change: [-6.5866% -3.7248% -0.7407%] (p = 0.03 < 0.05)
                        Change within noise threshold.

Benchmarking CompressedSNARK-StepCircuitSize-252325/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 121.5s.
CompressedSNARK-StepCircuitSize-252325/Prove                                                                          
                        time:   [12.140 s 12.317 s 12.539 s]
                        change: [-6.0110% -3.9390% -1.6900%] (p = 0.00 < 0.05)
                        Performance has improved.
CompressedSNARK-StepCircuitSize-252325/Verify                                                                          
                        time:   [370.96 ms 374.40 ms 378.00 ms]

Benchmarking CompressedSNARK-StepCircuitSize-514469/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 244.4s.
CompressedSNARK-StepCircuitSize-514469/Prove                                                                          
                        time:   [24.024 s 24.374 s 24.883 s]
Benchmarking CompressedSNARK-StepCircuitSize-514469/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 6.2s.
CompressedSNARK-StepCircuitSize-514469/Verify                                                                          
                        time:   [617.19 ms 629.14 ms 643.18 ms]

Benchmarking CompressedSNARK-StepCircuitSize-1038757/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 466.6s.
CompressedSNARK-StepCircuitSize-1038757/Prove                                                                          
                        time:   [46.797 s 47.949 s 49.284 s]
Benchmarking CompressedSNARK-StepCircuitSize-1038757/Verify: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 13.3s.
CompressedSNARK-StepCircuitSize-1038757/Verify                                                                          
                        time:   [1.3096 s 1.3383 s 1.3696 s]

Benchmarking CompressedSNARK-Commitments-StepCircuitSize-0/Prove: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 66.9s.
CompressedSNARK-Commitments-StepCircuitSize-0/Prove                                                                          
                        time:   [6.6110 s 6.6417 s 6.6750 s]
CompressedSNARK-Commitments-StepCircuitSize-0/Verify                                                                          
                        time:   [158.35 ms 160.13 ms 161.80 ms]

@mpenciak mpenciak mentioned this issue Dec 8, 2023
Merged
@huitseeker
Copy link
Contributor

Fixed in #168

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@huitseeker @porcuquine