Switch to Golang 1.13.1 #1496

fkaleo · 2019-10-16T19:32:40Z

replay

👍

robert-milan · 2019-10-17T11:38:36Z

Can we get some benchmark differences?

fkaleo · 2019-10-17T14:22:40Z

Can we get some benchmark differences?

It's better for most cases. Some seem worse and require investigation.
Using make benchmark from #1498 with golang 1.11.13 and 1.13.1 then benchcmp:

diff.txt

fkaleo · 2019-10-17T19:00:03Z

Will use benchstat to get a measurement of the variance of those benchmarks.

fkaleo · 2019-10-17T20:14:34Z

> benchstat -sort delta -html benchmark_1.11.1.txt benchmark_1.13.1.txt > benchmark.html

results in:
ORDER WORST TO BEST: https://19a8a3c9-eff6-48cd-a552-2a5286db54da.htmlpasta.com/
ORDER BY PACKAGE: https://1cae3fc6-88e2-42b3-bed2-ddce264b3646.htmlpasta.com/

fkaleo · 2019-10-18T08:58:24Z

BenchmarkSerializeMetricPoint regressed in performance however it benchmarks func (m *MetricPoint) Marshal(b []byte) that we do not use.

fkaleo · 2019-10-18T09:45:30Z

The following regressions need to be looked into:

pkg:github.com/grafana/metrictank/consolidation goos:linux goarch:amd64
ConsolidateAvgRand1M_1-8	6.22ms ± 2%	5.92ms ± 3%	−4.81%	(p=0.000 n=9+9)
ConsolidateAvgRandWithNulls1M_1-8	5.54ms ± 1%	5.76ms ± 2%	+4.08%	(p=0.000 n=8+8)
ConsolidateAvgRand1M_2-8	3.98ms ± 2%	3.51ms ± 3%	−11.62%	(p=0.000 n=9+8)
ConsolidateAvgRandWithNulls1M_2-8	3.28ms ± 2%	3.30ms ± 8%	~	(p=0.258 n=9+9)
ConsolidateAvgRand1M_25-8	1.42ms ± 1%	1.65ms ± 0%	+16.28%	(p=0.000 n=9+8)
ConsolidateAvgRandWithNulls1M_25-8	1.26ms ± 0%	1.78ms ± 1%	+41.49%	(p=0.000 n=7+9)
ConsolidateAvgRand1M_100-8	1.40ms ± 2%	1.65ms ± 1%	+18.06%	(p=0.000 n=8+10)
ConsolidateAvgRandWithNulls1M_100-8	1.25ms ± 5%	1.73ms ± 1%	+38.93%	(p=0.000 n=8+9)
ConsolidateMinRand1M_1-8	5.61ms ± 1%	5.64ms ± 0%	~	(p=0.321 n=8+9)
ConsolidateMinRandWithNulls1M_1-8	5.43ms ± 1%	6.21ms ± 1%	+14.26%	(p=0.000 n=9+8)
ConsolidateMinRand1M_2-8	6.90ms ± 8%	6.88ms ± 7%	~	(p=0.460 n=8+10)
ConsolidateMinRandWithNulls1M_2-8	3.10ms ± 2%	3.58ms ±12%	+15.41%	(p=0.000 n=9+10)
ConsolidateMinRand1M_25-8	3.81ms ± 0%	4.04ms ± 0%	+5.89%	(p=0.000 n=10+9)
ConsolidateMinRandWithNulls1M_25-8	3.06ms ± 0%	3.26ms ± 0%	+6.57%	(p=0.000 n=9+8)
ConsolidateMinRand1M_100-8	2.98ms ± 1%	3.11ms ± 1%	+4.30%	(p=0.000 n=10+9)
ConsolidateMinRandWithNulls1M_100-8	2.49ms ± 1%	2.70ms ± 2%	+8.41%	(p=0.000 n=10+10)
ConsolidateMaxRand1M_1-8	5.58ms ± 1%	6.19ms ± 0%	+11.02%	(p=0.000 n=9+8)
ConsolidateMaxRandWithNulls1M_1-8	5.46ms ± 4%	5.89ms ± 7%	+7.88%	(p=0.000 n=9+10)
ConsolidateMaxRand1M_2-8	6.97ms ± 1%	7.04ms ± 4%	~	(p=0.321 n=8+9)
ConsolidateMaxRandWithNulls1M_2-8	3.10ms ± 0%	3.14ms ± 2%	+1.45%	(p=0.003 n=9+9)
ConsolidateMaxRand1M_25-8	4.23ms ± 2%	3.91ms ± 0%	−7.46%	(p=0.000 n=9+8)
ConsolidateMaxRandWithNulls1M_25-8	2.88ms ± 2%	2.81ms ± 7%	~	(p=0.447 n=9+10)
ConsolidateMaxRand1M_100-8	3.39ms ± 1%	3.39ms ± 0%	~	(p=0.436 n=9+9)
ConsolidateMaxRandWithNulls1M_100-8	2.26ms ± 1%	2.28ms ± 1%	+0.94%	(p=0.003 n=9+10)
ConsolidateSumRand1M_1-8	5.41ms ± 3%	6.07ms ± 0%	+12.20%	(p=0.000 n=9+10)
ConsolidateSumRandWithNulls1M_1-8	5.37ms ± 1%	5.44ms ± 2%	+1.32%	(p=0.050 n=8+8)
ConsolidateSumRand1M_2-8	3.24ms ± 1%	3.38ms ± 1%	+4.30%	(p=0.000 n=10+8)
ConsolidateSumRandWithNulls1M_2-8	3.19ms ± 6%	3.02ms ± 1%	−5.22%	(p=0.010 n=10+9)
ConsolidateSumRand1M_25-8	1.33ms ± 1%	1.51ms ± 1%	+13.16%	(p=0.000 n=9+8)
ConsolidateSumRandWithNulls1M_25-8	1.22ms ± 1%	1.53ms ± 7%	+25.14%	(p=0.000 n=9+10)
ConsolidateSumRand1M_100-8	1.43ms ± 2%	1.67ms ± 1%	+16.23%	(p=0.000 n=10+9)
ConsolidateSumRandWithNulls1M_100-8	1.34ms ± 1%	1.62ms ± 1%	+20.77%	(p=0.000 n=7+9)

ROB10Add-8	5.93ns ± 1%	6.17ns ± 1%	+4.03%	(p=0.000 n=9+8)
ROB120Add-8	5.92ns ± 2%	6.29ns ± 3%	+6.28%	(p=0.000 n=10+10)
ROB600Add-8	5.91ns ± 1%	6.20ns ± 2%	+4.96%	(p=0.000 n=9+10)
ROB10AddShuffled5-8	5.91ns ± 1%	6.19ns ± 1%	+4.79%	(p=0.000 n=10+10)
ROB120AddShuffled5-8	5.90ns ± 1%	6.17ns ± 0%	+4.60%	(p=0.000 n=8+9)
ROB600AddShuffled5-8	6.03ns ± 4%	6.18ns ± 1%	~	(p=0.068 n=10+9)
ROB10AddShuffled50-8	19.3ns ± 3%	17.5ns ± 1%	−9.27%	(p=0.000 n=9+9)
ROB120AddShuffled50-8	5.93ns ± 2%	6.25ns ± 3%	+5.46%	(p=0.000 n=9+10)
ROB600AddShuffled50-8	5.99ns ± 3%	6.19ns ± 2%	+3.32%	(p=0.003 n=10+10)
ROB10AddShuffled500-8	7.31ns ± 1%	8.28ns ±10%	+13.30%	(p=0.000 n=8+10)
ROB120AddShuffled500-8	18.5ns ± 4%	16.9ns ± 2%	−8.82%	(p=0.000 n=10+10)
ROB600AddShuffled500-8	5.98ns ± 3%	6.24ns ± 2%	+4.46%	(p=0.000 n=9+10)

ConsolidateAvgRandWithNulls1M_25-8	1.26ms ± 0%	1.78ms ± 1%	+41.49%
ConsolidateAvgRand1M_1-8	6.22ms ± 2%	5.92ms ± 3%	−4.81%
ConsolidateAvgRand1M_2-8	3.98ms ± 2%	3.51ms ± 3%	−11.62%

DivideSeriesLists10k_1AllSeriesHalfNulls-8	126µs ±12%	157µs ±13%	+24.06%	(p=0.000 n=9+10)
DivideSeriesLists10k_10AllSeriesHalfNulls-8	1.10ms ± 9%	1.51ms ± 6%	+37.44%	(p=0.000 n=10+10)
DivideSeriesLists10k_100AllSeriesHalfNulls-8	9.33ms ± 1%	14.43ms ± 3%	+54.57%	(p=0.000 n=10+10)
DivideSeriesLists10k_1000AllSeriesHalfNulls-8	91.4ms ± 2%	149.1ms ± 5%	+63.05%	(p=0.000 n=8+9) 

SerializeMetricPoint-8	28.3ns ± 8%	45.5ns ±17%	+60.79%	(p=0.000 n=9+10)


[REGRESSION ONLY WHEN USING `-benchtime=1ms`]
pkg:github.com/grafana/metrictank/idx goos:linux goarch:amd64
MarshalMsgArchive-8	160ns ± 1%	167ns ± 5%	+4.71%	(p=0.000 n=8+10)
AppendMsgArchive-8	118ns ± 1%	116ns ± 0%	−0.90%	(p=0.008 n=10+9)
UnmarshalArchive-8	369ns ± 0%	446ns ± 1%	+20.87%	(p=0.000 n=9+8)
EncodeArchive-8	335ns ± 0%	413ns ± 0%	+23.19%	(p=0.000 n=9+9)
DecodeArchive-8	604ns ± 1%	751ns ± 1%	+24.33%	(p=0.000 n=8+10)
MarshalMsgNode-8	60.0ns ± 3%	72.6ns ± 2%	+21.08%	(p=0.000 n=10+9)
AppendMsgNode-8	30.7ns ± 1%	38.4ns ± 1%	+25.16%	(p=0.000 n=10+8)
UnmarshalNode-8	105ns ± 1%	130ns ± 1%	+23.10%	(p=0.000 n=10+8)
EncodeNode-8	105ns ± 0%	137ns ± 1%	+30.19%	(p=0.000 n=9+10)
DecodeNode-8	204ns ±18%	254ns ±20%	+24.40%	(p=0.013 n=10+10)

[REGRESSION ONLY WHEN USING `-benchtime=1ms`]
pkg:github.com/grafana/metrictank/expr/tagquery goos:linux goarch:amd64
ExpressionParsing-8	27.9µs ± 7%	11.9µs ± 4%	−57.30%	(p=0.000 n=8+9)
MarshalMsgTag-8	46.7ns ± 1%	52.8ns ± 2%	+13.11%	(p=0.000 n=8+9)
AppendMsgTag-8	19.9ns ± 0%	23.1ns ± 3%	+16.02%	(p=0.000 n=6+9)
UnmarshalTag-8	59.8ns ± 1%	59.3ns ± 2%	−0.81%	(p=0.020 n=8+9)
EncodeTag-8	55.6ns ± 0%	65.1ns ± 1%	+17.18%	(p=0.000 n=9+8)
DecodeTag-8	80.5ns ± 4%	104.2ns ±13%	+29.31%	(p=0.000 n=9+10)
MarshalMsgTags-8	34.2ns ± 4%	31.8ns ± 3%	−7.04%	(p=0.000 n=9+10)
AppendMsgTags-8	9.52ns ± 0%	10.66ns ± 4%	+11.97%	(p=0.000 n=9+10)
UnmarshalTags-8	9.26ns ± 0%	9.86ns ± 3%	+6.50%	(p=0.000 n=8+10)
EncodeTags-8	12.5ns ± 0%	14.3ns ± 0%	+14.40%	(p=0.000 n=9+10)
DecodeTags-8	15.8ns ± 1%	18.2ns ± 4%	+15.34%	(p=0.000 n=8+10)

[REGRESSION ONLY WHEN USING `-benchtime=100ms`]
AddRangeDesc4-8	168µs ± 0%	385µs ±10%	+129.57%	(p=0.000 n=8+10)
AddRangeDesc64-8	11.3µs ± 1%	20.0µs ± 2%	+76.51%	(p=0.000 n=9+10)

fkaleo · 2019-10-18T11:22:28Z

Raw benchmarks using -benchtime=1ms, -benchtime=100ms, -benchtime=1000ms:

benchmark_1.11.1_1ms.txt
benchmark_1.11.1_100ms.txt
benchmark_1.11.1_1000ms.txt

benchmark_1.13.1_1ms.txt
benchmark_1.13.1_100ms.txt
benchmark_1.13.1_1000ms.txt

robert-milan · 2020-03-03T17:11:35Z

We should just switch to 1.14 instead at this point.

Dieterbe · 2020-03-05T08:06:18Z

we probably should first get the performance testing framework running before we look into this again.

stale · 2020-06-03T08:23:59Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Switched to Golang 1.13.1

4b5f3be

replay approved these changes Oct 16, 2019

View reviewed changes

fkaleo changed the title ~~Switched to Golang 1.13.1~~ Switch to Golang 1.13.1 Oct 16, 2019

fkaleo requested a review from replay October 18, 2019 09:46

fkaleo mentioned this pull request Oct 18, 2019

Add 'benchmark' target to Makefile that runs all benchmarks #1498

Merged

stale bot added the stale label Jun 3, 2020

stale bot closed this Jun 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Golang 1.13.1 #1496

Switch to Golang 1.13.1 #1496

fkaleo commented Oct 16, 2019 •

edited

Loading

replay left a comment

robert-milan commented Oct 17, 2019

fkaleo commented Oct 17, 2019 •

edited

Loading

fkaleo commented Oct 17, 2019

fkaleo commented Oct 17, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading

robert-milan commented Mar 3, 2020

Dieterbe commented Mar 5, 2020

stale bot commented Jun 3, 2020

Switch to Golang 1.13.1 #1496

Switch to Golang 1.13.1 #1496

Conversation

fkaleo commented Oct 16, 2019 • edited Loading

replay left a comment

Choose a reason for hiding this comment

robert-milan commented Oct 17, 2019

fkaleo commented Oct 17, 2019 • edited Loading

fkaleo commented Oct 17, 2019

fkaleo commented Oct 17, 2019 • edited Loading

fkaleo commented Oct 18, 2019 • edited Loading

fkaleo commented Oct 18, 2019 • edited Loading

fkaleo commented Oct 18, 2019 • edited Loading

robert-milan commented Mar 3, 2020

Dieterbe commented Mar 5, 2020

stale bot commented Jun 3, 2020

fkaleo commented Oct 16, 2019 •

edited

Loading

fkaleo commented Oct 17, 2019 •

edited

Loading

fkaleo commented Oct 17, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading

fkaleo commented Oct 18, 2019 •

edited

Loading