allocator: evaluate lower IO overload thresholds for blocking rebalancing / lease transfers

[kvoli]

Is your feature request related to a problem? Please describe.
Two settings control blocking replica rebalancing, and lease transfers to stores depending on their IO overload score:

rebalancing:: kv.allocator.replica_io_overload_threshold
lease transfers:: kv.allocator.lease_io_overload_threshold

These settings both take into account the mean, as well as the threshold define in the setting, requiring a store be above the 110% of the mean and the threshold:

cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go

Lines 2387 to 2391 in c8df48c

	func ioOverloadCheck(
	score, avg, absThreshold, meanThreshold float64,
	enforcement IOOverloadEnforcementLevel,
	disallowed ...IOOverloadEnforcementLevel,
	) (ok bool, reason string) {

We have seen instances where too many recovery + rebalancing snapshots can itself cause IO overload to occur, as disk resources are saturated.

Describe the solution you'd like
Re-evaluate the default values for kv.allocator.replica_io_overload_threshold (0.8) ¹ and kv.allocator.lease_io_overload_threshold (0.5) ², considering lower values.

Additional context
See scale testing for 23.2 (internal) slack.

Jira issue: CRDB-32448

Footnotes

1. https://github.com/cockroachdb/cockroach/blob/c8df48cb85f40c10940dfbc3efa6281dfe5c9701/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L74-L77 ↩

2. https://github.com/cockroachdb/cockroach/blob/c8df48cb85f40c10940dfbc3efa6281dfe5c9701/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L82-L82 ↩

[kvoli]

We could apply this threshold to blocking all replication as well. This currently only blocks rebalancing for the replica threshold:

cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go

Lines 168 to 172 in cddcddd

	"block_rebalance_to",
	map[int64]string{
	int64(IOOverloadThresholdIgnore): "ignore",
	int64(IOOverloadThresholdBlockTransfers): "block_rebalance_to",
	int64(IOOverloadThresholdBlockAll): "block_all",

In the linked thread, it was recovery snapshots, which aren't affected with the default (block_rebalance_to).

[kvoli]

@sumeerbhola noted that the previous defaults were set when AC began throttling at a higher threshold, this changed in 23.2:

These were set when AC attempted to stabilize the store at a threshold of 1.0, so the allocator was not adding load before AC started aggressively shaping incoming traffic (modulo the fact that AC sometimes didn't work, e.g. for follower writes). As of v23.2 AC will typically stabilize a store to a threshold of 0.5 (~10 L0 sub-levels).

[kvoli]

Will be closed on 23.2 backport #115129