groupByNodes() does not return series in a consistent order, Graphite does

[replay]

When using groupByNodes() in Metrictank the order of the returned series is randomized because the order is determined by the order in which keys get read from this map, and Go randomizes the map key order:

metrictank/expr/func_groupbynodes.go

Line 73 in 487e2d4

for key, group := range groups {

In graphite the order is guaranteed to be the order in which keys got generated, to do this it is using a list here:
https://github.com/graphite-project/graphite-web/blob/4ce727593e949554cf1653ec7a42280a4b4578fd/webapp/graphite/render/functions.py#L4891

This can lead to issues if the return value of groupByNodes() is used as input for another function for which the order is relevant. We have observed a user who's using the output of groupByNodes() as input for divideSeriesLists(), which has lead to wrong results once we switched to Metrictank function processing.

[Dieterbe]

it's not clear to me what the order should be to always match Graphite.
do you know how the sorting works in graphite exactly?

[replay]

the main reason why it should always match graphite is for backwards compatibility, because otherwise queries may end up returning wrong results in a situation like divideSeriesLists(groupByNodes(...), <some other list of metrics>).

graphite iterates over the given series in the order they were given, for each series it generates the metric name of the resulting aggregate, this metric name gets added to a list (keys https://github.com/graphite-project/graphite-web/blob/4ce727593e949554cf1653ec7a42280a4b4578fd/webapp/graphite/render/functions.py#L4891). the order in which the aggregated metrics then get returned from groupByNodes() is the order in which the metric names got added to keys, so it is the order of the first occurrence of each aggregate metric name.

f.e. if you call groupByNodes() on 2 metrics and it returns 2 metrics (so no aggregation has been done), then the returned 2 metrics are going to be in the same order as the input metrics were passed into groupByNodes().