groupByTags Performance improvements + fix setting consolidator per group

[shanson7]

Use pre-parsed tag for name, only get the summarizeCons once.

Benchmarks for just this set of changes

benchmark                                              old ns/op      new ns/op      delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      11799          10548          -10.60%
BenchmarkGroupByTags10Series_100Datapoints_half-8      15975          14753          -7.65%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      220073         217893         -0.99%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      295150         301724         +2.23%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     14854710       13961473       -6.01%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     305783470      16255546       -94.68%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     1739274046     1734320020     -0.28%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     685210547      411100140      -40.00%

benchmark                                              old MB/s     new MB/s     speedup
BenchmarkGroupByTags10Series_100Datapoints_full-8      1016.96      1137.60      1.12x
BenchmarkGroupByTags10Series_100Datapoints_half-8      751.16       813.37       1.08x
BenchmarkGroupByTags10Series_10kDatapoints_full-8      5452.73      5507.28      1.01x
BenchmarkGroupByTags10Series_10kDatapoints_half-8      4065.72      3977.13      0.98x
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     807.82       859.51       1.06x
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     39.24        738.21       18.81x
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     689.94       691.91       1.00x
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     1751.29      2919.00      1.67x

benchmark                                              old allocs     new allocs     delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      40             28             -30.00%
BenchmarkGroupByTags10Series_100Datapoints_half-8      64             52             -18.75%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      40             28             -30.00%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      64             52             -18.75%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     20056          10054          -49.87%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     52289          41880          -19.91%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     22037          12035          -45.39%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     54386          42968          -20.99%

benchmark                                              old bytes     new bytes     delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      5242          4761          -9.18%
BenchmarkGroupByTags10Series_100Datapoints_half-8      7310          6826          -6.62%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      6478          5870          -9.39%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      15868         13792         -13.08%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     5110153       4790036       -6.26%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     9540243       9124264       -4.36%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     5522092       5202708       -5.78%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     207434013     140563990     -32.24%

Combined with #1164 and #1158 :

benchmark                                              old ns/op      new ns/op     delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      11799          10599         -10.17%
BenchmarkGroupByTags10Series_100Datapoints_half-8      15975          14098         -11.75%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      220073         184092        -16.35%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      295150         226125        -23.39%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     14854710       9008327       -39.36%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     305783470      15733034      -94.85%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     1739274046     212949763     -87.76%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     685210547      325118333     -52.55%

benchmark                                              old MB/s     new MB/s     speedup
BenchmarkGroupByTags10Series_100Datapoints_full-8      1016.96      1132.15      1.11x
BenchmarkGroupByTags10Series_100Datapoints_half-8      751.16       851.15       1.13x
BenchmarkGroupByTags10Series_10kDatapoints_full-8      5452.73      6518.46      1.20x
BenchmarkGroupByTags10Series_10kDatapoints_half-8      4065.72      5306.78      1.31x
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     807.82       1332.10      1.65x
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     39.24        762.73       19.44x
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     689.94       5635.13      8.17x
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     1751.29      3690.96      2.11x

benchmark                                              old allocs     new allocs     delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      40             28             -30.00%
BenchmarkGroupByTags10Series_100Datapoints_half-8      64             52             -18.75%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      40             28             -30.00%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      64             52             -18.75%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     20056          10044          -49.92%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     52289          41968          -19.74%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     22037          10234          -53.56%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     54386          41832          -23.08%

benchmark                                              old bytes     new bytes     delta
BenchmarkGroupByTags10Series_100Datapoints_full-8      5242          4761          -9.18%
BenchmarkGroupByTags10Series_100Datapoints_half-8      7310          6825          -6.63%
BenchmarkGroupByTags10Series_10kDatapoints_full-8      6478          5838          -9.88%
BenchmarkGroupByTags10Series_10kDatapoints_half-8      15868         13513         -14.84%
BenchmarkGroupByTags10kSeries_100Datapoints_full-8     5110153       4788323       -6.30%
BenchmarkGroupByTags10kSeries_100Datapoints_half-8     9540243       9205551       -3.51%
BenchmarkGroupByTags10kSeries_10kDatapoints_full-8     5522092       4827670       -12.58%
BenchmarkGroupByTags10kSeries_10kDatapoints_half-8     207434013     87353387      -57.89%

[shanson7]

After deploying I found an issue with simply using the 'name' tag. The alias* were not setting this tag, though it should be universally available.

[Dieterbe]

hmm tags was never used and the compiler never complained? that's strange

[shanson7]

It was "used" in that it was populated. It was never used after that though. Enough to satisfy the compiler!

[Dieterbe]

ah yes, i've run into this a few times before :/

[Dieterbe]

the Clean up tests/benchmarks commit is confusing.

before we tested every combo of numseries 1/10/100/1000 and input series generation via 2x test.RandFloats10k / test.RandFloats10k,test.RandFloatsWithNulls10k / test.RandFloatsWithNulls10k,test.RandFloatsWithNulls10k

why do we now benchmark fewer amounts of series and none of the RandFloatsWithNulls variants? i suppose the reason for the latter is because we don't want to conflate performance of the aggregator with that of GBT, but if that's the case, why do we still test different numbers of datapoints per series? I suppose different numbers of points per series could be useful to see the effect of the data memory use on that of the metadata stuff that GBT is doing.
I suppose we shouldn't benchmark every combo of every possible parameter as that makes the output unwieldy and the runtime longer.

But if anything the number of input series seems like a parameter we should explore more, like we used to

[shanson7]

Yeah, at this point I was still trying to isolate the performance issues I was seeing with the aggregation. I think the two variables relevant to GBT are number of input series and output series. Maybe whether or not to include 'name'.

[Dieterbe]

right on.
it is probably not clear, but it looks like we do try to assure the name tag is always set for any series (FuncGet sets it initially and any other function that creates new series should set it as well I think)

[Dieterbe]

this seems incorrect. it's possible for input series to have different consolidators defined, or am I missing something?

[shanson7]

That's true, but I think that's true with many of the aggregation functions and it seems first come first serve.

In general, I think grouping here would be very awkward with mixing consolidators and the user is likely doing something...odd.

[Dieterbe]

actually the code in master also seems wrong.
seems to me the most correct way is for each group to do cons, queryCons := summarizeCons(groupSeries)

[shanson7]

I think this is more correct, though in the case that the groups are mixed, there is still an arbitrary consolidator chosen. But when 'name' is one of the groupByTags, I could this working better

[Dieterbe]

this PR needs some history rewriting.
i suggest you do a git rebase -i master (you know about fixup and squash in rebase -i right?)

• I have added a fix up commit in https://github.com/grafana/metrictank/commits/GBT_perf
• the allocation optimization introduced in "Only grab summarizeCons once and name only if needed", and is then later removed, can you squash those commits so the change never appears
• "Add another test function" needs squashing into "Clean up tests/benchmarks" as that function is used there
• move the alias test/fix commits first. I think it's a good practice to move fixes for stuff we rely on first, before introducing the real changes and their benchmarks.

[shanson7]

I rebased to be (I believe) what you were asking for, though the commit order shown here is different than I get with git log

[shanson7]

Regarding the benchmarks, what do you think of:

input series: 1, 10, 100, 10k, 100k
output series: 1, same as input, then if applicable: 10, 100, 10k

This would result in 15 benchmarks. So as to not overly test the series aggregations, I could use a small number of datapoints, like 100.

[Dieterbe]

yes, but let's also include 1k.

[shanson7]

For both input and output? That would bump the number of benchmarks to 21. Should be fine

[Dieterbe]

yeah, i guess.

[Dieterbe]

unlike usual, i didn't bother to reboot my laptop into benchmark mode, as you already posted pretty good results. but i wanted to run them anyway with the latest changes.

my observations:

• changes become more profound the more outbound series you have for a given input serie (makes sense). good call also of introducing this parameter into the mix.
• some significant sources of allocs remaining
  • FuncMock.Exec calls SetTags for each input series. perhaps we should SetTags() before running the code we're measuring.
  • adding to the cache. which is ironic, as it is intended to be able to reuse series (e.g. partially computed results), but i suspect in reality we don't do this much. and it isn't turned on yet anyway cause the cache mechanism isn't fully implemented yet.

benchstat pre.txt post.txt
name                            old time/op    new time/op      delta
GroupByTags1in1out-8              2.64µs ± 5%      2.43µs ± 2%      -7.89%  (p=0.002 n=6+6)
GroupByTags10in1out-8             10.8µs ± 3%       9.5µs ± 3%     -12.23%  (p=0.002 n=6+6)
GroupByTags10in10out-8            20.9µs ± 2%      19.1µs ± 3%      -8.55%  (p=0.002 n=6+6)
GroupByTags100in1out-8            79.1µs ± 2%      68.7µs ± 2%     -13.16%  (p=0.002 n=6+6)
GroupByTags100in10out-8            106µs ± 1%        93µs ± 3%     -11.96%  (p=0.002 n=6+6)
GroupByTags100in100out-8           264µs ± 2%       186µs ± 1%     -29.40%  (p=0.004 n=6+5)
GroupByTags1000in1out-8            755µs ± 3%       658µs ± 3%     -12.82%  (p=0.002 n=6+6)
GroupByTags1000in10out-8           910µs ± 1%       764µs ± 2%     -16.04%  (p=0.002 n=6+6)
GroupByTags1000in100out-8         1.66ms ± 2%      0.98ms ± 0%     -40.72%  (p=0.010 n=6+4)
GroupByTags1000in1000out-8        8.08ms ± 0%      2.05ms ± 3%     -74.57%  (p=0.002 n=6+6)
GroupByTags10000in1out-8          7.93ms ± 1%      7.06ms ± 1%     -11.06%  (p=0.002 n=6+6)
GroupByTags10000in10out-8         8.94ms ± 2%      7.39ms ± 3%     -17.27%  (p=0.002 n=6+6)
GroupByTags10000in100out-8        15.9ms ± 3%       8.7ms ± 1%     -45.36%  (p=0.004 n=6+5)
GroupByTags10000in1000out-8       73.0ms ± 1%      10.3ms ± 1%     -85.91%  (p=0.002 n=6+6)
GroupByTags10000in10000out-8       642ms ± 1%        22ms ± 1%     -96.51%  (p=0.004 n=5+6)
GroupByTags100000in1out-8         85.4ms ± 1%      77.5ms ± 2%      -9.21%  (p=0.008 n=5+5)
GroupByTags100000in10out-8         103ms ± 2%        88ms ± 0%     -15.41%  (p=0.010 n=6+4)
GroupByTags100000in100out-8        177ms ± 3%        87ms ± 2%     -51.01%  (p=0.004 n=6+5)
GroupByTags100000in1000out-8       874ms ± 4%       103ms ± 2%     -88.19%  (p=0.002 n=6+6)
GroupByTags100000in10000out-8      7.32s ± 3%       0.13s ± 4%     -98.19%  (p=0.002 n=6+6)
GroupByTags100000in100000out-8     71.2s ± 3%        0.3s ± 1%     -99.63%  (p=0.004 n=6+5)

name                            old speed      new speed        delta
GroupByTags1in1out-8             454MB/s ± 4%     493MB/s ± 2%      +8.52%  (p=0.002 n=6+6)
GroupByTags10in1out-8           1.11GB/s ± 3%    1.26GB/s ± 3%     +13.91%  (p=0.002 n=6+6)
GroupByTags10in10out-8           574MB/s ± 2%     627MB/s ± 3%      +9.36%  (p=0.002 n=6+6)
GroupByTags100in1out-8          1.52GB/s ± 2%    1.75GB/s ± 2%     +15.17%  (p=0.002 n=6+6)
GroupByTags100in10out-8         1.13GB/s ± 1%    1.29GB/s ± 3%     +13.61%  (p=0.002 n=6+6)
GroupByTags100in100out-8         455MB/s ± 2%     644MB/s ± 1%     +41.64%  (p=0.004 n=6+5)
GroupByTags1000in1out-8         1.59GB/s ± 3%    1.82GB/s ± 3%     +14.69%  (p=0.002 n=6+6)
GroupByTags1000in10out-8        1.32GB/s ± 1%    1.57GB/s ± 2%     +19.11%  (p=0.002 n=6+6)
GroupByTags1000in100out-8        725MB/s ± 2%    1223MB/s ± 0%     +68.66%  (p=0.010 n=6+4)
GroupByTags1000in1000out-8       149MB/s ± 0%     584MB/s ± 3%    +293.41%  (p=0.002 n=6+6)
GroupByTags10000in1out-8        1.51GB/s ± 1%    1.70GB/s ± 1%     +12.44%  (p=0.002 n=6+6)
GroupByTags10000in10out-8       1.34GB/s ± 2%    1.62GB/s ± 3%     +20.88%  (p=0.002 n=6+6)
GroupByTags10000in100out-8       757MB/s ± 3%    1385MB/s ± 1%     +82.97%  (p=0.004 n=6+5)
GroupByTags10000in1000out-8      164MB/s ± 1%    1167MB/s ± 1%    +609.83%  (p=0.002 n=6+6)
GroupByTags10000in10000out-8    18.7MB/s ± 1%   535.3MB/s ± 1%   +2762.83%  (p=0.004 n=5+6)
GroupByTags100000in1out-8       1.41GB/s ± 1%    1.55GB/s ± 2%     +10.14%  (p=0.008 n=5+5)
GroupByTags100000in10out-8      1.16GB/s ± 2%    1.37GB/s ± 0%     +18.19%  (p=0.010 n=6+4)
GroupByTags100000in100out-8      678MB/s ± 3%    1384MB/s ± 2%    +104.15%  (p=0.004 n=6+5)
GroupByTags100000in1000out-8     137MB/s ± 4%    1164MB/s ± 2%    +746.93%  (p=0.002 n=6+6)
GroupByTags100000in10000out-8   16.4MB/s ± 2%   908.2MB/s ± 4%   +5435.45%  (p=0.002 n=6+6)
GroupByTags100000in100000out-8  1.68MB/s ± 3%  450.75MB/s ± 1%  +26677.11%  (p=0.004 n=6+5)

name                            old alloc/op   new alloc/op     delta
GroupByTags1in1out-8              1.68kB ± 0%      1.57kB ± 0%      -6.66%  (p=0.002 n=6+6)
GroupByTags10in1out-8             5.24kB ± 0%      4.76kB ± 0%      -9.18%  (p=0.002 n=6+6)
GroupByTags10in10out-8            12.0kB ± 0%      11.5kB ± 0%      -4.18%  (p=0.004 n=6+5)
GroupByTags100in1out-8            32.5kB ± 0%      29.2kB ± 0%     -10.34%  (p=0.002 n=6+6)
GroupByTags100in10out-8           46.7kB ± 0%      43.4kB ± 0%      -7.24%  (p=0.002 n=6+6)
GroupByTags100in100out-8           116kB ± 0%       111kB ± 0%      -4.50%  (p=0.002 n=6+6)
GroupByTags1000in1out-8            263kB ± 0%       230kB ± 0%     -12.26%  (p=0.002 n=6+6)
GroupByTags1000in10out-8           322kB ± 0%       289kB ± 0%     -10.10%  (p=0.002 n=6+6)
GroupByTags1000in100out-8          487kB ± 0%       450kB ± 0%      -7.50%  (p=0.002 n=6+6)
GroupByTags1000in1000out-8        1.51MB ± 3%      1.44MB ± 2%      -4.08%  (p=0.009 n=6+6)
GroupByTags10000in1out-8          5.12MB ± 0%      4.80MB ± 0%      -6.25%  (p=0.000 n=6+5)
GroupByTags10000in10out-8         2.63MB ± 0%      2.31MB ± 0%     -12.16%  (p=0.002 n=6+6)
GroupByTags10000in100out-8        3.24MB ± 0%      2.90MB ± 0%     -10.54%  (p=0.002 n=6+6)
GroupByTags10000in1000out-8       5.12MB ± 0%      4.61MB ± 1%      -9.87%  (p=0.004 n=5+6)
GroupByTags10000in10000out-8      34.0MB ± 0%      17.8MB ±10%     -47.68%  (p=0.004 n=5+6)
GroupByTags100000in1out-8         64.5MB ± 0%      61.3MB ± 0%      -4.97%  (p=0.000 n=5+6)
GroupByTags100000in10out-8        54.4MB ± 0%      49.5MB ± 0%      -8.97%  (p=0.000 n=6+5)
GroupByTags100000in100out-8       29.5MB ± 0%      24.7MB ± 0%     -16.55%  (p=0.002 n=6+6)
GroupByTags100000in1000out-8      50.0MB ± 1%      32.4MB ± 0%     -35.16%  (p=0.002 n=6+6)
GroupByTags100000in10000out-8      100MB ± 0%        54MB ± 1%     -46.65%  (p=0.008 n=5+5)
GroupByTags100000in100000out-8     367MB ± 0%       198MB ±12%     -45.99%  (p=0.002 n=6+6)

name                            old allocs/op  new allocs/op    delta
GroupByTags1in1out-8                17.0 ± 0%        15.0 ± 0%     -11.76%  (p=0.002 n=6+6)
GroupByTags10in1out-8               40.0 ± 0%        28.0 ± 0%     -30.00%  (p=0.002 n=6+6)
GroupByTags10in10out-8              86.0 ± 0%        74.0 ± 0%     -13.95%  (p=0.002 n=6+6)
GroupByTags100in1out-8               223 ± 0%         121 ± 0%     -45.74%  (p=0.002 n=6+6)
GroupByTags100in10out-8              306 ± 0%         204 ± 0%     -33.33%  (p=0.002 n=6+6)
GroupByTags100in100out-8             733 ± 0%         629 ± 0%     -14.19%  (p=0.002 n=6+6)
GroupByTags1000in1out-8            2.03k ± 0%       1.02k ± 0%     -49.43%  (p=0.002 n=6+6)
GroupByTags1000in10out-8           2.14k ± 0%       1.14k ± 0%     -46.87%  (p=0.002 n=6+6)
GroupByTags1000in100out-8          2.96k ± 0%       1.96k ± 0%     -34.01%  (p=0.000 n=6+4)
GroupByTags1000in1000out-8         7.47k ± 1%       6.43k ± 0%     -13.92%  (p=0.002 n=6+6)
GroupByTags10000in1out-8           20.1k ± 0%       10.1k ± 0%     -49.67%  (p=0.002 n=6+6)
GroupByTags10000in10out-8          20.3k ± 0%       10.3k ± 0%     -49.35%  (p=0.002 n=6+6)
GroupByTags10000in100out-8         21.5k ± 0%       11.3k ± 0%     -47.11%  (p=0.002 n=6+6)
GroupByTags10000in1000out-8        30.5k ± 0%       19.6k ± 0%     -35.66%  (p=0.000 n=5+6)
GroupByTags10000in10000out-8        100k ± 0%         66k ± 3%     -34.63%  (p=0.004 n=5+6)
GroupByTags100000in1out-8           210k ± 0%        110k ± 0%     -47.61%  (p=0.002 n=6+6)
GroupByTags100000in10out-8          220k ± 0%        110k ± 0%     -49.94%  (p=0.002 n=6+6)
GroupByTags100000in100out-8         222k ± 0%        112k ± 0%     -49.65%  (p=0.002 n=6+6)
GroupByTags100000in1000out-8        314k ± 0%        133k ± 0%     -57.74%  (p=0.002 n=6+6)
GroupByTags100000in10000out-8       510k ± 0%        216k ± 0%     -57.63%  (p=0.004 n=6+5)
GroupByTags100000in100000out-8     1.10M ± 0%       0.70M ± 4%     -36.78%  (p=0.002 n=6+6)