Use String.join() to describe a list of tasks

[DaveCTurner]

This change replaces the use of string concatenation with a call to
String.join(). String concatenation might be quadratic, unless the compiler can
optimise it away, whereas String.join() is more reliably linear. There can
sometimes be a large number of pending ClusterState update tasks and #28920
includes a report that this operation sometimes takes a long time.

[DaveCTurner]

NB I haven't been able to reproduce the slowness reported in #28920, because I don't have easy access to a partitionable cluster with 10k shards. @danielmitterdorfer do you have any opinions about this change and/or good ideas about how to quantify the benefit?

If this change looks ok, I can ask the OP of #28920 to try it.

[danielmitterdorfer]

do you have any opinions about this change

I am not familiar with this part of the code. But if this is the bottleneck then your change makes sense to me.

good ideas about how to quantify the benefit?

I think this would be a good candidate for a microbenchmark? We have some infrastructure for that in place in the benchmarks module. You can also have a look at an example benchmark that I did recently in the context of #28702. Happy to help out here as well and we can also run it in our microbenchmarking environment (which is tuned to reduce measurement noise).

[bleskes]

LGTM

[bleskes]

out of curiosity - didn't T::toString work?

[DaveCTurner]

No, apparently Iterator<String> cannot be converted to Iterator<CharSequence>:

> Task :server:compileJava 
/Users/davidturner/src/elasticsearch-master/server/src/main/java/org/elasticsearch/cluster/ClusterStateTaskExecutor.java:59: error: no suitable method found for join(String,tasks.stre[...]rator)
        return String.join(", ", tasks.stream().map(T::toString).filter(t -> t.length() == 0)::iterator);
                     ^
    method String.join(CharSequence,CharSequence...) is not applicable
      (varargs mismatch; CharSequence is not a functional interface
          multiple non-overriding abstract methods found in interface CharSequence)
    method String.join(CharSequence,Iterable<? extends CharSequence>) is not applicable
      (argument mismatch; bad return type in method reference
          Iterator<String> cannot be converted to Iterator<CharSequence>)

[DaveCTurner]

Thanks @danielmitterdorfer for the pointer to the benchmarks module, that's just what I was after. I just overwrote the existing benchmark with this one: https://gist.github.com/DaveCTurner/de8763bc791d860e4fb0c9a9f98df7cd. At 10,000 tasks, each with a 100-byte description, the improvement is from ~500ms to ~120µs:

Result "org.elasticsearch.benchmark.routing.allocation.AllocationBenchmark.measureStreamReduce":
  578.282 ±(99.9%) 14.884 ms/op [Average]
  (min, avg, max) = (534.191, 578.282, 633.771), stdev = 22.278
  CI (99.9%): [563.398, 593.166] (assumes normal distribution)

...


Result "org.elasticsearch.benchmark.routing.allocation.AllocationBenchmark.measureStringJoin":
  0.119 ±(99.9%) 0.004 ms/op [Average]
  (min, avg, max) = (0.114, 0.119, 0.138), stdev = 0.006
  CI (99.9%): [0.115, 0.123] (assumes normal distribution)


# Run complete. Total time: 00:02:14

Benchmark                                Mode  Cnt    Score    Error  Units
AllocationBenchmark.measureStreamReduce  avgt   30  578.282 ± 14.884  ms/op
AllocationBenchmark.measureStringJoin    avgt   30    0.119 ±  0.004  ms/op

This seems worth doing.

[danielmitterdorfer]

You're welcome. That's quite a significant difference indeed.