Fix the performance problem of InternalResourceGroup.getActiveWorkCount

[dongshengbc]

During an investigation of "missing metrics" problem, we traced it back to a performance problem with InternalResourceGroup.getActiveWorkCount.

The evidences:

1. We took a JFR recording and we can see all the fb303 threads are blocked.
2. By checking the thread dump, the lock is often held by this call InternalResourceGroup.getActiveWorkCount
3. We did a small benchmark, and it shows a simple Sets.diff(Sets.diff) is 30 times slower than HashSet.size(), which should be O(1).

Change

To fix the problem, we calculate the value in AllNodes' contr once, and then read many times. Since all the node sets in AllNodes are immutable, this is safe to do.

Problem

The benchmark result is here. avgt = average time.

https://gist.github.com/dongshengbc/be7eee51a4d98b0dc3363b294266424b

Benchmark	Mode	Samples	Score	Score error	Units
o.s.ImmutableSetBenchmark.testImmutableDiffSize	avgt	5	8.605	0.241	ns/op
o.s.ImmutableSetBenchmark.testImmutableSize	avgt	5	0.865	0.012	ns/op
o.s.ImmutableSetBenchmark.testTwoImmutableDiffSize	avgt	5	22.483	0.923	ns/op

== RELEASE NOTES ==

General Changes
* Improve performance of getting resource group metrics

[patzar]

Overall looks good, thanks for the fix.

[swapsmagic]

LGTM. Thanks for fixing it.

[ajaygeorge]

LGTM % a comment

[agrawaldevesh]

Requesting changes because: (a) Does this optimization actually help in terms of reducing the cost of the expensive call or the num of its invocations. (b) If it can be done in NodeManager proper

[agrawaldevesh]

Forgot to request changes in the prev round of reviews

[rschlussel]

@dongshengbc can you add a release note with a user facing description? Maybe something like Improve performance of fetching resource group metrics?

[dongshengbc]

@dongshengbc can you add a release note with a user facing description? Maybe something like Improve performance of fetching resource group metrics?

Sure, how to do that? link to the instruction?

[rschlussel]

should have been in the PR template when you created the PR. But basically, add it in a code block at the end of the description following the format in this example: #19814 (comment)