Merge #113667

113667: allocator: lower default IO overload threshold r=sumeerbhola a=kvoli

The IO overload threshold determines whether a store will be excluded as a target for rebalancing replicas, or leases onto it. The previous thresholds were `0.8` for replica rebalancing
(`kv.allocator.replica_io_overload_threshold`) and `0.5` for lease transfers (`kv.allocator.lease_io_overload_threshold`).

The previous settings were selected when Admission Control (AC) attempted to stabilize IO overload at a threshold of 1.0. AC will now typically stabilize a store to a threshold of 0.5.

Lower the thresholds to `0.4` for replica rebalancing and `0.3` for lease transfers. Note that a store needs to both exceed the absolute threshold mentioned above, as well as be greater than `+10% * mean` of equivalent rebalancing candidates.

Resolves: #112497
Release note: None

Co-authored-by: Austen McClernon <[email protected]>

Author: craig[bot]

pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go

@@ -74,12 +74,12 @@ const (
 	// DefaultReplicaIOOverloadThreshold is used to avoid allocating to stores with an
 	// IO overload score greater than what's set. This is typically used in
 	// conjunction with IOOverloadMeanThreshold below.
-	DefaultReplicaIOOverloadThreshold = 0.8
+	DefaultReplicaIOOverloadThreshold = 0.4
 
-	// DefaultLeaseIOOverloadThreshold is used to shed leases from stores with an
-	// IO overload score greater than this threshold. This is typically used in
-	// conjunction with IOOverloadMeanThreshold below.
-	DefaultLeaseIOOverloadThreshold = 0.5
+	// DefaultLeaseIOOverloadThreshold is used to block lease transfers to stores
+	// with an IO overload score greater than this threshold. This is typically
+	// used in conjunction with IOOverloadMeanThreshold below.
+	DefaultLeaseIOOverloadThreshold = 0.3
 
 	// DefaultLeaseIOOverloadShedThreshold is used to shed leases from stores
 	// with an IO overload score greater than the this threshold. This is