Speedup tests

[FrancoGiachetta]

Speedup tests

This PR adds cargo-nextest as the tool used to run tests. It also parallelizes how tests are run in cairo-native-test and scarb-native-test which were run rather sequentially.

Improvements:

• Parallelizing cairo-native-test gives a ~100% improvement over make test-cairo command.

Introduces Breaking Changes?

No.

• Created PR in sequencer
• Created PR in starknet-replay
• Updated the starknet-blocks.yml workflow to use these PRs.

These PRs should be merged after this one right away, in that order.

Checklist

• Linked to Github Issue.
• Unit tests added.
• Integration tests added.
• This change requires new documentation.
  • Documentation has been added/updated.

[github-actions]

Benchmark results Main vs HEAD.

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base dict_insert.cairo (JIT)	2.474 ± 0.017	2.444	2.500	1.04 ± 0.01
base dict_insert.cairo (AOT)	2.387 ± 0.012	2.373	2.410	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head dict_insert.cairo (JIT)	2.712 ± 0.067	2.624	2.805	1.04 ± 0.03
head dict_insert.cairo (AOT)	2.612 ± 0.047	2.554	2.714	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base dict_snapshot.cairo (JIT)	2.187 ± 0.010	2.173	2.207	1.03 ± 0.01
base dict_snapshot.cairo (AOT)	2.131 ± 0.022	2.079	2.161	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head dict_snapshot.cairo (JIT)	2.336 ± 0.027	2.297	2.390	1.05 ± 0.02
head dict_snapshot.cairo (AOT)	2.232 ± 0.022	2.195	2.260	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base factorial_2M.cairo (JIT)	2.610 ± 0.022	2.572	2.640	1.01 ± 0.02
base factorial_2M.cairo (AOT)	2.581 ± 0.032	2.544	2.635	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head factorial_2M.cairo (JIT)	2.709 ± 0.021	2.684	2.737	1.01 ± 0.01
head factorial_2M.cairo (AOT)	2.673 ± 0.028	2.630	2.736	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base fib_2M.cairo (JIT)	2.190 ± 0.065	2.112	2.271	1.06 ± 0.04
base fib_2M.cairo (AOT)	2.064 ± 0.031	2.035	2.132	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head fib_2M.cairo (JIT)	2.229 ± 0.031	2.186	2.274	1.02 ± 0.02
head fib_2M.cairo (AOT)	2.191 ± 0.023	2.165	2.230	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base heavy_circuit.cairo (JIT)	13.915 ± 0.200	13.675	14.161	1.05 ± 0.02
base heavy_circuit.cairo (AOT)	13.276 ± 0.085	13.145	13.407	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head heavy_circuit.cairo (JIT)	14.434 ± 0.176	14.192	14.771	1.04 ± 0.01
head heavy_circuit.cairo (AOT)	13.913 ± 0.099	13.776	14.074	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base linear_search.cairo (JIT)	2.285 ± 0.022	2.257	2.317	1.05 ± 0.02
base linear_search.cairo (AOT)	2.185 ± 0.037	2.122	2.234	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head linear_search.cairo (JIT)	2.431 ± 0.033	2.394	2.491	1.07 ± 0.02
head linear_search.cairo (AOT)	2.273 ± 0.022	2.245	2.301	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base logistic_map.cairo (JIT)	2.416 ± 0.031	2.389	2.488	1.09 ± 0.02
base logistic_map.cairo (AOT)	2.219 ± 0.013	2.202	2.241	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head logistic_map.cairo (JIT)	2.573 ± 0.043	2.511	2.650	1.08 ± 0.03
head logistic_map.cairo (AOT)	2.390 ± 0.049	2.338	2.501	1.00

[github-actions]

Benchmarking results

Benchmark for program dict_insert

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	11.173 ± 0.046	11.098	11.237	4.43 ± 0.06
cairo-native (embedded AOT)	2.523 ± 0.033	2.469	2.583	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.611 ± 0.015	2.588	2.638	1.03 ± 0.01

Benchmark for program dict_snapshot

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	526.9 ± 6.5	516.7	537.8	1.00
cairo-native (embedded AOT)	2217.8 ± 21.4	2194.5	2273.5	4.21 ± 0.07
cairo-native (embedded JIT using LLVM's ORC Engine)	2341.0 ± 33.3	2303.9	2394.2	4.44 ± 0.08

Benchmark for program factorial_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	4.878 ± 0.021	4.845	4.918	1.84 ± 0.02
cairo-native (embedded AOT)	2.654 ± 0.023	2.626	2.701	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.689 ± 0.018	2.659	2.717	1.01 ± 0.01

Benchmark for program fib_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	4.815 ± 0.029	4.782	4.875	2.20 ± 0.02
cairo-native (embedded AOT)	2.184 ± 0.019	2.161	2.222	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.215 ± 0.023	2.181	2.260	1.01 ± 0.01

Benchmark for program heavy_circuit

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	9.613 ± 0.070	9.486	9.720	1.00
cairo-native (embedded AOT)	14.758 ± 0.075	14.633	14.860	1.54 ± 0.01
cairo-native (embedded JIT using LLVM's ORC Engine)	15.070 ± 0.086	14.950	15.225	1.57 ± 0.01

Benchmark for program linear_search

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	572.5 ± 10.7	559.8	598.2	1.00
cairo-native (embedded AOT)	2242.1 ± 25.5	2208.0	2304.5	3.92 ± 0.09
cairo-native (embedded JIT using LLVM's ORC Engine)	2388.3 ± 27.9	2343.9	2430.2	4.17 ± 0.09

Benchmark for program logistic_map

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	382.3 ± 5.0	375.5	391.1	1.00
cairo-native (embedded AOT)	2359.3 ± 32.5	2331.5	2443.7	6.17 ± 0.12
cairo-native (embedded JIT using LLVM's ORC Engine)	2530.5 ± 24.5	2494.5	2582.5	6.62 ± 0.11

[JulianGCalderon]

Running make test prints the following warning. Is the config file required?

warning: ignoring unknown configuration keys in config file /Users/julian/projects/cairo_native/.config/nextest.toml:
  - profile.ci.debug
  - profile.ci.inherits
  - profile.ci.opt-level

[FrancoGiachetta]

Oh, I may have missed that message when I ran it. I'll check it.

[FrancoGiachetta]

I ended up removing the config file.

[JulianGCalderon]

LGTM!