Table of Content

• Overview
  • Measurements
• Test Cases
  • Brainfuck
    • bench.b
    • mandel.b
  • Base64
  • Json
  • Matmul
  • Primes
• Tests Execution
  • Environment
  • Using Docker
  • Manual Execution
    • Prerequisites
• Contribution
  • Makefile guide
    • Binary executables
    • Compiled artifacts
    • Scripting language
  • README update
  • Docker image update

Overview

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2023-10-16

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
Scala (Staged)	0.500±0.011	214.99±04.56 + 21.33±01.86	28.38±02.06
Racket (Staged)	1.300±0.000	100.79±00.40 + 0.00±00.00	47.18±00.05
C++/g++	1.323±0.000	1.84±00.03 + 0.00±00.00	52.88±00.06
Java	1.582±0.000	39.81±00.08 + 1.19±00.04	61.57±00.09
Nim/gcc	1.612±0.000	0.92±00.01 + 0.00±00.00	64.40±00.05
C++/clang++	1.642±0.000	1.62±00.02 + 0.00±00.00	63.90±00.13
V/gcc	1.649±0.002	1.84±00.03 + 0.00±00.00	63.31±00.62
Rust	1.658±0.000	0.92±00.03 + 0.00±00.00	64.18±00.04
D/gdc	1.658±0.000	6.34±00.02 + 0.00±00.00	68.09±00.23
D/ldc2	1.664±0.000	1.39±00.01 + 0.00±00.00	66.39±00.05
C/gcc	1.667±0.000	0.85±00.03 + 0.00±00.00	64.65±00.11
C/clang	1.674±0.000	0.89±00.02 + 0.00±00.00	64.93±00.16
Kotlin/JVM	1.689±0.000	43.17±00.06 + 0.79±00.27	65.90±00.17
Zig	1.790±0.001	0.92±00.02 + 0.00±00.00	70.28±00.46
Go	1.900±0.000	3.08±00.03 + 0.00±00.00	73.52±00.12
Chez Scheme	1.932±0.001	24.74±00.03 + 4.47±00.11	79.80±00.06
F#/.NET Core	1.959±0.003	36.38±00.11 + 0.58±00.00	80.39±00.23
C#/.NET Core	1.979±0.000	31.68±00.15 + 0.20±00.00	81.21±00.02
OCaml	1.997±0.001	3.24±00.04 + 2.40±00.03	91.15±00.11
Racket	2.027±0.022	93.25±00.28 + 20.64±00.63	81.62±00.91
Nim/clang	2.050±0.001	1.14±00.03 + 0.00±00.00	80.91±00.74
Vala/gcc	2.083±0.000	4.56±00.06 + 0.00±00.00	78.05±00.05
Vala/clang	2.259±0.000	4.58±00.03 + 0.00±00.00	84.70±00.11
Go/gccgo	2.331±0.000	23.56±00.03 + 0.00±00.00	94.42±00.99
Crystal	2.345±0.001	2.95±00.01 + 0.00±00.00	93.56±00.08
V/clang	2.511±0.038	1.84±00.01 + 0.00±00.00	106.19±00.94
MLton	2.608±0.000	1.61±00.05 + 0.25±00.00	104.24±01.38
C#/Mono	2.685±0.000	25.41±00.08 + 0.00±00.00	108.49±01.15
Julia	2.858±0.002	249.20±00.09 + 0.35±00.03	110.50±00.26
D/dmd	3.128±0.001	3.40±00.04 + 0.00±00.00	117.91±00.51
Scala	3.277±0.004	72.46±00.13 + 186.43±00.26	138.10±00.61
Haskell (MArray)	3.861±0.001	5.81±00.03 + 4.84±00.00	157.16±01.03
Node.js	4.160±0.004	38.00±00.02 + 3.94±00.03	166.25±01.10
Haskell (FP)	4.183±0.002	5.99±00.02 + 4.99±00.03	177.11±00.22
Swift	5.479±0.000	16.14±00.09 + 0.00±00.00	205.37±00.21
Ruby/truffleruby	5.733±0.233	224.91±04.25 + 647.13±70.73	278.60±11.01
Lua/luajit	6.489±0.005	2.44±00.01 + 0.00±00.00	259.08±00.13
Ruby/truffleruby (JVM)	6.637±0.177	398.30±18.36 + 602.85±86.95	317.74±07.84
Python/pypy	12.214±0.008	59.50±00.13 + 29.66±00.18	510.88±01.26
Idris	15.676±0.024	20.65±00.06 + 8.82±00.05	686.28±01.00
Elixir	23.946±0.021	70.45±00.70 + 0.00±00.00	942.72±02.80
Lua	51.056±0.422	2.23±00.04 + 0.00±00.00	1973.41±15.02
Ruby (--jit)	54.186±0.425	16.16±00.02 + 1.79±00.01	2302.73±18.18
PHP	55.973±0.024	17.81±00.08 + 0.00±00.00	2181.36±02.17
Ruby	78.429±0.569	14.95±00.02 + 0.00±00.00	3302.47±21.59
Ruby/jruby	85.643±1.154	197.30±04.27 + 212.10±11.71	3804.72±62.43
Python	119.026±0.454	10.14±00.03 + 0.00±00.00	4666.45±30.38
Tcl (FP)	270.164±2.350	3.93±00.04 + 0.00±00.00	11304.53±116.31
Perl	316.357±2.409	7.05±00.06 + 0.00±00.00	12793.56±88.59
Tcl (OOP)	530.477±4.818	3.93±00.03 + 0.00±00.00	22275.06±348.47

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C++/g++	10.158±0.026	1.80±00.02 + 2.29±00.04	414.88±04.38
Java	14.106±0.016	39.81±00.09 + 2.27±00.07	561.52±04.28
C/gcc	14.185±0.004	0.86±00.01 + 0.83±00.05	563.74±02.93
Racket (Staged)	14.379±0.428	100.46±00.16 + 76.88±02.04	572.08±14.26
Kotlin/JVM	14.396±0.014	42.93±00.21 + 2.18±00.22	592.87±01.55
Rust	15.057±0.014	0.90±00.01 + 1.10±00.02	594.38±02.79
Scala (Staged)	15.065±0.646	216.61±02.00 + 103.20±07.08	742.04±18.57
D/gdc	15.070±0.006	6.27±00.02 + 1.44±00.02	636.33±01.52
Zig	15.107±0.007	0.90±00.02 + 1.41±00.01	622.38±01.04
C/clang	15.113±0.005	0.86±00.01 + 0.88±00.03	648.78±04.04
C++/clang++	15.135±0.011	1.59±00.02 + 1.96±00.01	621.55±02.98
D/ldc2	15.319±0.005	3.01±00.03 + 0.79±00.02	613.95±00.68
Go	15.413±0.141	3.04±00.07 + 1.26±00.00	616.56±06.53
C#/.NET Core	16.331±0.009	31.53±00.09 + 0.88±00.00	691.29±02.05
Crystal	16.589±0.353	2.87±00.04 + 0.72±00.04	689.62±13.84
Vala/gcc	18.058±0.007	4.40±00.04 + 1.19±00.03	698.11±00.81
V/gcc	18.122±0.068	1.80±00.02 + 1.16±00.01	702.84±01.94
Swift	18.141±0.044	15.91±00.04 + 0.00±00.00	745.20±01.74
Nim/gcc	18.170±0.031	2.00±00.01 + 1.29±00.00	768.06±03.91
Vala/clang	19.705±0.003	4.39±00.04 + 1.21±00.02	763.62±00.28
Nim/clang	19.914±0.585	2.26±00.04 + 1.29±00.00	819.31±20.88
Go/gccgo	20.479±0.021	23.90±00.09 + 1.28±00.01	867.17±00.59
Scala	20.991±0.024	72.57±00.12 + 137.46±00.42	885.00±01.03
V/clang	22.603±0.245	1.83±00.01 + 1.14±00.02	985.37±15.29
OCaml	26.617±0.004	3.97±00.01 + 3.47±00.06	1284.03±06.98
Chez Scheme	27.772±0.070	25.50±00.05 + 3.68±00.01	1221.71±03.21
Node.js	28.483±0.392	38.93±00.06 + 6.70±00.12	1209.63±15.82
Julia	30.379±0.066	250.43±00.03 + 0.41±00.01	1160.03±01.94
C#/Mono	31.446±0.014	25.27±00.06 + 0.82±00.00	1337.71±01.79
F#/.NET Core	34.935±0.027	36.02±00.12 + 2.14±00.02	1480.47±04.19
Lua/luajit	35.086±0.061	2.42±00.02 + 0.43±00.00	1406.88±05.61
Racket	37.036±0.513	92.86±00.08 + 22.57±00.15	1640.19±13.64
Haskell (MArray)	37.162±0.014	5.61±00.01 + 5.82±00.00	1523.81±05.62
D/dmd	38.423±0.006	3.25±00.04 + 0.86±00.03	1412.70±09.18
MLton	44.897±0.052	1.65±00.03 + 4.11±00.00	1924.64±09.32
Python/pypy	48.246±0.092	59.57±00.07 + 30.36±00.08	2077.33±19.41
Ruby/truffleruby	48.288±0.912	222.51±09.39 + 557.98±63.76	2344.82±60.98
Ruby/truffleruby (JVM)	54.733±0.355	388.41±10.42 + 460.00±43.85	2492.14±26.26
Idris	67.241±0.176	21.86±00.06 + 9.54±00.00	2893.54±10.95
Haskell (FP)	80.301±0.045	5.75±00.03 + 75.70±00.00	3338.80±16.07

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/clang (aklomp)	0.096±0.000	2.07±00.04 + 0.00±00.00	4.59±00.07
C/gcc (aklomp)	0.099±0.000	2.08±00.02 + 0.00±00.00	4.69±00.05
PHP	0.106±0.000	18.51±00.04 + 0.00±00.00	4.98±00.02
Go	0.306±0.006	6.29±00.05 + 4.62±00.35	14.18±00.25
Rust	0.956±0.000	2.31±00.03 + 0.00±00.00	39.05±00.21
C/clang	0.997±0.000	1.98±00.07 + 0.00±00.00	36.78±00.05
D/ldc2	1.095±0.009	3.58±00.04 + 3.41±00.00	46.71±00.59
C/gcc	1.098±0.001	2.00±00.04 + 0.00±00.00	40.28±00.05
Crystal	1.102±0.001	3.57±00.02 + 1.27±00.03	45.33±00.27
Nim/clang	1.471±0.001	1.92±00.03 + 5.83±00.06	58.38±00.37
Java	1.508±0.002	40.71±00.09 + 209.37±12.91	60.15±00.52
V/clang	1.563±0.000	2.36±00.04 + 2388.26±00.77	59.06±00.12
Nim/gcc	1.575±0.001	1.66±00.02 + 5.25±00.03	64.12±00.20
Scala	1.578±0.001	68.33±00.19 + 318.32±05.49	64.35±00.22
V/gcc	1.584±0.001	2.32±00.04 + 2386.10±00.91	58.12±00.34
Kotlin/JVM	1.635±0.009	43.84±00.04 + 250.75±01.85	66.17±00.82
Vala/gcc	1.644±0.002	5.69±00.01 + 0.01±00.00	62.83±00.12
Vala/clang	1.645±0.001	5.61±00.04 + 0.00±00.00	63.55±00.56
Ruby (--jit)	1.670±0.002	16.79±00.02 + 41.32±02.06	65.08±00.30
Ruby	1.678±0.001	15.57±00.03 + 40.68±00.33	65.41±00.13
C++/clang++ (libcrypto)	1.717±0.002	5.03±00.05 + 0.68±00.04	69.64±00.50
C++/g++ (libcrypto)	1.717±0.003	5.54±00.08 + 0.70±00.03	69.18±00.27
Node.js	1.729±0.005	38.73±00.03 + 37.05±00.10	71.17±00.74
Go	1.784±0.002	4.17±00.04 + 4.48±00.39	75.34±00.29
Perl (MIME::Base64)	1.880±0.004	14.81±00.06 + 0.12±00.05	74.45±00.50
F#/.NET Core	2.339±0.035	36.42±00.08 + 43.54±01.89	87.52±01.18
D/gdc	2.383±0.001	7.29±00.04 + 3.35±00.00	105.29±00.61
C#/.NET Core	2.605±0.016	31.59±00.09 + 50.83±09.00	95.85±00.78
D/dmd	2.733±0.001	3.53±00.04 + 3.35±00.00	119.71±00.60
Python	3.186±0.013	10.23±00.02 + 0.09±00.00	125.50±00.65
Zig	3.196±0.003	1.52±00.04 + 0.00±00.00	125.26±00.49
Python/pypy	3.580±0.001	59.43±00.07 + 31.58±00.10	157.31±00.19
Tcl	3.624±0.003	5.05±00.04 + 0.00±00.00	148.02±00.31
Go/gccgo	3.674±0.001	24.81±00.12 + 8.08±00.25	166.56±00.23
Racket	3.903±0.036	91.21±00.25 + 19.37±00.50	155.49±01.84
Ruby/truffleruby (JVM)	3.967±0.036	391.81±10.87 + 298.87±50.79	201.52±02.09
C#/Mono	4.775±0.001	26.15±00.06 + 18.66±00.07	196.42±00.82
Julia	4.827±0.001	266.49±00.12 + 43.51±00.12	181.97±00.63
Ruby/jruby	10.655±0.215	190.12±01.70 + 91.63±06.55	428.25±09.08
Ruby/truffleruby	12.337±0.012	226.13±03.32 + 492.67±49.16	546.38±03.47
Perl (MIME::Base64::Perl)	13.588±0.098	16.08±00.04 + 0.28±00.09	574.28±06.82

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

• gason mutates input strings;
• simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
• DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
• DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:
  • DAW JSON Link's number parsing issue
  • gason's number parsing issue
  • D stdlib number parsing issue

Json

Language	Time, s	Memory, MiB	Energy, J
C++/clang++ (simdjson On-Demand)	0.059±0.000	112.62±00.29 + 59.80±00.25	2.48±00.01
C++/g++ (simdjson On-Demand)	0.060±0.000	113.51±00.06 + 59.81±00.00	2.56±00.02
C++/clang++ (DAW JSON Link NoCheck)	0.083±0.000	112.36±00.07 + 0.00±00.00	3.38±00.02
C++/g++ (DAW JSON Link NoCheck)	0.083±0.000	113.16±00.06 + 0.00±00.00	3.41±00.03
C++/clang++ (DAW JSON Link)	0.084±0.000	112.42±00.01 + 0.00±00.00	3.53±00.01
C++/g++ (DAW JSON Link)	0.087±0.000	113.11±00.07 + 0.00±00.00	3.66±00.02
Rust (Serde Custom)	0.096±0.000	111.31±00.03 + 0.00±00.00	4.11±00.04
Rust (Serde Typed)	0.104±0.000	111.67±00.03 + 11.77±00.00	4.43±00.01
C++/clang++ (simdjson DOM)	0.123±0.000	112.83±00.13 + 176.09±00.13	5.33±00.03
C++/g++ (simdjson DOM)	0.128±0.000	113.51±00.04 + 176.60±00.00	5.60±00.07
C++/clang++ (gason)	0.138±0.000	112.37±00.04 + 96.97±00.06	5.62±00.03
D/ldc2 (Mir Asdf DOM)	0.138±0.000	112.85±00.03 + 61.10±00.00	5.78±00.06
C++/g++ (gason)	0.138±0.000	113.13±00.05 + 96.93±00.05	5.50±00.02
C++/g++ (RapidJSON)	0.157±0.000	113.11±00.02 + 128.90±00.02	6.70±00.06
Scala (jsoniter-scala)	0.158±0.002	291.61±00.13 + 16.65±00.51	8.50±00.11
Go (rjson custom)	0.196±0.000	115.18±00.04 + 0.00±00.00	7.63±00.02
Zig	0.221±0.000	110.91±00.01 + 39.02±00.03	9.87±00.04
Go (rjson)	0.221±0.000	115.23±00.07 + 68.03±00.00	8.68±00.02
Go (Sonic)	0.225±0.002	123.66±00.07 + 111.83±00.23	9.79±00.15
C++/clang++ (RapidJSON)	0.226±0.000	112.42±00.03 + 128.97±00.03	9.46±00.06
D/ldc2 (Mir Amazon's Ion DOM)	0.231±0.000	112.87±00.02 + 80.70±00.00	9.62±00.05
C++/g++ (RapidJSON Precise)	0.233±0.000	113.17±00.05 + 128.33±00.59	10.02±00.06
Go (goccy/go-json)	0.269±0.000	115.61±00.15 + 112.20±00.14	10.66±00.04
C++/clang++ (RapidJSON Precise)	0.302±0.000	112.46±00.04 + 128.97±00.03	13.00±00.06
C/gcc (yajl)	0.374±0.001	110.83±00.03 + 0.00±00.00	16.15±00.07
C/clang (yajl)	0.374±0.000	110.89±00.04 + 0.00±00.00	16.29±00.09
C++/g++ (Boost.JSON)	0.380±0.000	113.27±00.01 + 308.09±00.05	15.91±00.04
C++/g++ (RapidJSON SAX)	0.388±0.000	112.91±00.03 + 0.00±00.00	16.51±00.02
Nim/clang (jsony)	0.391±0.000	111.38±00.02 + 146.18±00.03	16.73±00.07
C++/clang++ (Boost.JSON)	0.395±0.001	112.52±00.03 + 308.18±00.00	16.62±00.04
Nim/gcc (jsony)	0.407±0.000	111.10±00.01 + 156.78±01.00	17.43±00.16
C++/g++ (RapidJSON SAX Precise)	0.445±0.001	112.97±00.01 + 0.00±00.00	19.60±00.06
Go (jsoniter)	0.520±0.001	227.75±00.08 + 1.78±00.21	22.21±00.23
Rust (Serde Untyped)	0.547±0.001	111.65±00.01 + 839.98±00.00	23.09±00.15
C#/.NET Core (System.Text.Json)	0.573±0.001	488.19±00.29 + 139.73±00.02	25.40±00.12
C++/clang++ (RapidJSON SAX)	0.585±0.000	194.66±00.03 + 0.00±00.00	23.44±00.03
Julia (JSON3)	0.615±0.003	462.74±00.18 + 224.58±01.29	26.04±00.29
Java (DSL-JSON)	0.633±0.022	262.57±00.10 + 198.23±00.93	32.50±00.48
V/clang	0.634±0.000	111.43±00.01 + 496.21±00.06	26.61±00.19
V/gcc	0.638±0.000	111.40±00.03 + 496.21±00.00	26.68±00.26
Node.js	0.646±0.004	150.07±00.04 + 196.56±01.49	31.69±00.18
Crystal (Pull)	0.648±0.002	113.25±00.01 + 18.39±00.03	29.04±00.18
Python/pypy	0.668±0.001	279.67±00.01 + 125.79±00.07	28.63±00.11
Crystal (Schema)	0.669±0.002	113.24±00.04 + 48.84±00.06	30.10±00.21
C++/clang++ (RapidJSON SAX Precise)	0.695±0.001	194.65±00.05 + 0.00±00.00	28.76±00.07
Nim/gcc (Packedjson)	0.791±0.001	111.83±00.02 + 294.16±00.00	32.60±00.13
Perl (Cpanel::JSON::XS)	0.796±0.008	125.50±00.07 + 402.87±00.02	33.22±00.36
PHP	0.844±0.000	127.86±00.21 + 517.86±00.00	36.27±00.19
Go	0.887±0.002	115.35±00.10 + 104.95±00.22	36.93±00.10
Nim/clang (Packedjson)	0.899±0.002	112.14±00.01 + 294.16±00.00	36.99±00.21
Crystal	0.950±0.011	113.24±00.01 + 392.50±00.00	41.49±00.82
C#/.NET Core	1.084±0.012	496.41±00.12 + 271.70±00.01	48.39±00.47
Nim/gcc	1.116±0.002	111.86±00.03 + 1001.34±00.00	46.65±00.27
C++/clang++ (json-c)	1.163±0.004	112.67±00.02 + 1216.08±00.00	48.79±00.78
C++/g++ (json-c)	1.164±0.005	113.30±00.06 + 1216.02±00.06	49.22±00.81
Clojure	1.246±0.014	460.37±07.51 + 546.75±16.98	66.52±01.33
Nim/clang	1.277±0.001	112.14±00.01 + 999.02±00.00	52.22±00.36
Go/gccgo	1.286±0.001	138.88±00.10 + 83.51±00.03	55.05±00.31
CPython (UltraJSON)	1.337±0.001	122.63±00.05 + 495.71±02.58	49.85±00.19
C++/clang++ (Nlohmann)	1.343±0.006	112.63±00.05 + 360.14±00.00	57.01±00.32
Python	1.391±0.002	120.04±00.02 + 326.36±00.04	55.39±00.55
Ruby	1.452±0.002	125.12±00.04 + 261.61±00.07	61.45±00.35
Ruby (--jit)	1.539±0.008	126.19±00.04 + 263.91±00.01	65.79±00.58
F#/.NET Core (System.Text.Json)	1.576±0.004	498.64±00.26 + 229.56±04.48	69.97±00.45
C++/g++ (Nlohmann)	1.611±0.005	113.26±00.07 + 448.03±00.03	66.24±00.72
C#/Mono	1.802±0.028	253.15±00.17 + 31.50±00.00	78.74±01.13
Ruby (YAJL)	1.808±0.004	125.06±00.04 + 276.09±00.04	75.84±00.58
D/ldc2	2.044±0.006	112.68±00.01 + 680.39±00.04	85.05±00.13
Haskell	2.257±0.006	117.34±00.01 + 725.38±00.23	94.48±00.60
Rust (jq)	2.596±0.001	113.53±00.04 + 903.76±01.16	108.96±00.28
Ruby/jruby	2.986±0.024	456.73±06.29 + 913.88±75.59	154.96±02.12
C++/g++ (Boost.PropertyTree)	3.065±0.003	113.12±00.04 + 1440.12±00.00	128.54±01.53
C++/clang++ (Boost.PropertyTree)	3.092±0.005	194.90±00.04 + 1232.84±00.00	128.35±00.98
Vala/clang	3.326±0.012	114.98±00.01 + 980.04±00.01	144.32±01.14
Vala/gcc	3.327±0.003	114.98±00.08 + 980.04±00.01	143.32±01.85
D/gdc	3.612±0.016	116.49±00.05 + 681.07±00.10	155.27±00.75
Racket	3.779±0.036	220.86±00.19 + 263.51±25.06	156.87±01.63
D/dmd	4.453±0.006	113.07±00.03 + 680.43±00.05	179.34±00.49
Perl (JSON::Tiny)	9.657±0.113	126.14±00.09 + 528.61±00.05	423.18±04.31
Ruby/truffleruby	10.672±0.121	466.47±23.50 + 1900.14±89.47	619.64±07.46
Ruby/truffleruby (JVM)	10.982±0.124	498.64±10.98 + 2558.62±197.73	687.46±08.96

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.042±0.000	6.08±00.05 + 57.76±00.05	4.44±00.02
V/gcc (VSL + CBLAS)	0.046±0.000	6.62±00.02 + 58.28±00.00	4.65±00.03
V/clang (VSL + CBLAS)	0.046±0.000	6.66±00.01 + 58.29±00.00	4.65±00.03
Nim/gcc (Arraymancer)	0.063±0.004	5.46±00.06 + 57.49±00.12	5.32±00.22
Python (NumPy)	0.064±0.000	31.77±00.13 + 58.54±00.10	6.19±00.02
C++/g++ (Eigen)	0.065±0.003	28.29±10.15 + 61.41±10.18	4.90±00.13
C++/clang++ (Eigen)	0.065±0.003	30.17±09.25 + 60.15±09.15	5.48±00.18
Nim/clang (Arraymancer)	0.068±0.001	6.04±00.07 + 57.60±00.18	5.90±00.06
Java (ND4J)	0.076±0.001	112.18±02.20 + 92.14±00.02	5.98±00.05
Julia (threads: 2)	0.083±0.000	283.67±00.03 + 57.16±00.06	5.25±00.02
Rust (ndarray)	0.085±0.000	2.37±00.05 + 68.47±00.00	5.99±00.04
Julia (threads: 1)	0.133±0.000	283.77±00.18 + 56.76±00.06	6.62±00.04
V/clang (VSL)	0.272±0.004	7.33±00.07 + 51.56±00.00	19.00±00.23
V/gcc (VSL)	0.505±0.002	7.05±00.07 + 51.82±00.00	36.74±00.18
Julia (no BLAS)	1.018±0.004	267.67±00.06 + 51.50±00.00	45.38±00.67
D/ldc2	1.717±0.001	3.25±00.01 + 70.47±00.02	63.31±00.18
D/gdc	1.869±0.001	7.30±00.03 + 70.16±00.01	73.12±00.03
D/dmd	1.882±0.002	3.17±00.02 + 70.49±00.02	70.93±00.15
C/gcc	3.026±0.001	1.50±00.05 + 68.66±00.02	111.93±00.58
V/gcc	3.038±0.001	2.49±00.06 + 68.58±00.00	112.89±00.08
Vala/clang	3.056±0.000	5.48±00.04 + 68.32±00.00	104.92±00.17
V/clang	3.058±0.000	2.79±00.05 + 68.58±00.00	104.95±00.09
C/clang	3.060±0.000	1.49±00.01 + 68.70±00.00	104.77±00.04
Rust	3.062±0.000	2.10±00.02 + 68.57±00.00	105.83±00.47
Zig	3.070±0.001	1.79±00.05 + 68.58±00.00	108.93±00.04
Swift	3.090±0.000	7.91±00.03 + 68.75±00.01	110.70±00.57
Nim/gcc	3.093±0.001	2.54±00.04 + 57.75±00.00	114.70±00.43
Nim/clang	3.117±0.001	2.79±00.01 + 64.97±03.74	107.58±00.51
Vala/gcc	3.121±0.000	5.32±00.20 + 68.32±00.00	114.54±00.07
Go	3.150±0.001	3.62±00.19 + 72.15±00.13	115.35±00.13
Go/gccgo	3.157±0.001	24.18±00.07 + 73.45±00.04	112.37±00.18
Crystal	3.159±0.000	3.52±00.04 + 60.05±00.06	115.85±00.21
Java	3.175±0.001	40.65±00.08 + 69.13±00.15	124.54±00.09
Kotlin/JVM	3.207±0.005	41.86±00.02 + 68.71±00.41	130.79±00.32
Node.js	3.215±0.004	44.73±00.03 + 72.69±00.11	129.83±00.12
Python/pypy	3.275±0.001	60.40±00.21 + 68.76±00.17	136.34±00.07
Scala	3.313±0.003	68.84±00.22 + 160.83±00.27	121.22±00.21
C#/.NET Core	4.382±0.001	33.31±00.06 + 69.05±00.00	176.43±00.07
C#/Mono	7.411±0.000	26.02±00.06 + 69.47±00.00	304.12±02.00
Ruby/truffleruby	19.501±1.622	424.02±38.01 + 560.75±63.85	702.98±52.89
Ruby/truffleruby (JVM)	24.995±0.199	442.91±19.85 + 454.12±55.17	894.29±06.62
Ruby (--jit)	152.114±0.134	17.91±00.07 + 69.69±00.04	6574.42±25.26
Ruby	178.566±0.786	15.79±00.04 + 69.15±00.05	7779.16±40.92
Perl	223.173±1.482	9.54±00.08 + 599.64±00.09	9122.71±66.22
Python	238.744±0.471	10.51±00.02 + 68.84±00.00	9445.43±16.40
Tcl	332.625±1.772	7.31±00.05 + 400.44±00.00	13948.04±55.34
Ruby/jruby	445.418±12.396	276.55±04.29 + 1023.72±133.69	18883.42±466.58

Primes

Testing:

• generating primes using the optimized sieve of Atkin;
• prefix search for their decimal numbers using Trie data structure.

Notes:

• All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
• The results are always sorted (could be unstable or stable though).

Primes

Language	Time, s	Memory, MiB	Energy, J
Zig	0.058±0.000	0.90±00.03 + 50.20±00.26	2.44±00.04
C++/g++	0.068±0.000	3.61±00.01 + 79.45±00.38	2.63±00.02
C++/clang++	0.070±0.000	3.05±00.00 + 59.19±00.13	2.62±00.04
Go	0.075±0.001	3.00±00.06 + 0.00±00.00	3.13±00.07
V/clang	0.106±0.000	1.86±00.01 + 211.02±00.55	4.27±00.05
V/gcc	0.108±0.000	1.83±00.02 + 207.57±00.39	4.40±00.03
Rust	0.125±0.000	2.00±00.05 + 72.94±00.00	4.85±00.01
Crystal	0.146±0.000	3.71±00.02 + 88.43±00.00	5.94±00.07
Java	0.157±0.003	39.69±00.06 + 150.56±01.21	8.72±00.13
Scala	0.226±0.005	73.14±00.21 + 210.71±00.50	12.73±00.18
Node.js	0.256±0.002	38.33±00.02 + 151.44±00.14	12.60±00.07
Nim/clang	0.296±0.000	2.01±00.02 + 601.22±01.55	11.45±00.02
Nim/gcc	0.298±0.001	1.74±00.05 + 615.91±00.00	11.25±00.07
Lua/luajit	0.334±0.000	2.58±00.06 + 157.98±00.68	13.11±00.13
Julia	0.675±0.002	267.69±00.07 + 342.19±00.25	26.19±00.26
Racket	0.730±0.003	102.20±00.12 + 237.07±00.09	29.01±00.31
Python/pypy	0.793±0.003	59.10±00.02 + 249.23±00.08	31.40±00.13
Ruby/truffleruby	0.912±0.014	225.50±01.85 + 779.85±19.22	61.61±01.20
Ruby/truffleruby (JVM)	1.359±0.037	388.78±12.63 + 567.15±29.85	88.28±02.40
Lua	1.487±0.003	2.25±00.02 + 284.27±00.80	58.31±00.46
Ruby (--jit)	1.775±0.020	15.95±00.02 + 172.01±01.28	80.44±00.60
Ruby	1.887±0.003	14.91±00.02 + 142.89±00.03	77.16±00.57
Ruby/jruby	2.484±0.033	192.08±00.89 + 464.73±41.25	134.20±02.35
Python	3.101±0.013	10.14±00.04 + 181.56±00.77	125.18±00.55

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language	Version
.NET Core	7.0.402
C#/.NET Core	4.7.0-3.23416.9 (43b0b05c)
C#/Mono	6.12.0.200
Chez Scheme	9.5.8
Clojure	"1.11.1"
Crystal	1.10.1
D/dmd	v2.105.2
D/gdc	13.2.0
D/ldc2	1.34.0
Elixir	1.14.0
F#/.NET Core	12.7.0.0 for F# 7.0
Go	go1.21.3
Go/gccgo	13.2.0
Haskell	9.4.7
Idris 2	0.6.0
Java	21
Julia	v"1.9.3"
Kotlin	1.9.10
Lua	5.4.4
Lua/luajit	2.1.1696562864
MLton	20210117
Nim	2.0.0
Node.js	v20.8.1
OCaml	5.1.0
PHP	8.2.10
Perl	v5.36.0
Python	3.11.6
Python/pypy	7.3.13-final0 for Python 3.10.13
Racket	"8.10"
Ruby	3.2.2p53
Ruby/jruby	9.4.3.0
Ruby/truffleruby	23.1.0
Rust	1.73.0
Scala	3.3.1
Swift	5.9
Tcl	8.6
V	0.4.2 2332c17
Vala	0.56.13
Zig	0.11.0
clang/clang++	16.0.6 (15)
gcc/g++	13.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

• versions (print installed language versions);
• shell (start the shell);
• brainfuck bench (build and run Brainfuck bench.b benchmarks);
• brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
• base64 (build and run Base64 benchmarks);
• json (build and run Json benchmarks);
• matmul (build and run Matmul benchmarks);
• primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

• ./run.sh build (build the image);
• ./run.sh make versions (run the image with the versions command);
• sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

• Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

• NumPy for matmul tests (Debian package python3-numpy).
• UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

• Boost for JSON tests (Debian package libboost-dev).
• JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

• libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

• Coursier for downloading Maven artifacts.

For Haskell:

• network for TCP connectivity between the tests and the test runner.
• raw-strings-qq for raw string literals used in tests.

For Perl:

• cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

• JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

• The test runner listens on localhost:9001;
• All messages are sent using TCP sockets closed immediately after the message has been sent;
• There are two messages sent from a test (it establishes the measurement boundary):
  1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
  2. The end message with any content (mostly it's "stop" for consistency).
• The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

• append a new target (the final binary location) into executables variable;
• append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

• append a new target (the final artifact location) into artifacts variable;
• append the proper target rule to compile the test;
• append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

• append run[<script_file>] into all_runners variable;
• append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc