kv/kvserver: TestFlowControlRangeSplitMergeV2 failed

[cockroach-teamcity]

kv/kvserver.TestFlowControlRangeSplitMergeV2 failed on release-24.3.21-rc @ 1b38d62fc62e6a54d9f6aa92a085421b5d4b5032:

        -  kvflowcontrol.tokens.eval.elastic.returned                        | 30 MiB  
        -  kvflowcontrol.tokens.eval.elastic.returned.disconnect             | 0 B     
        -  kvflowcontrol.tokens.eval.elastic.unaccounted                     | 0 B     
        -  kvflowcontrol.tokens.eval.regular.available                       | 48 MiB  
        -  kvflowcontrol.tokens.eval.regular.deducted                        | 0 B     
        -  kvflowcontrol.tokens.eval.regular.returned                        | 0 B     
        -  kvflowcontrol.tokens.eval.regular.returned.disconnect             | 0 B     
        -  kvflowcontrol.tokens.eval.regular.unaccounted                     | 0 B     
        -  kvflowcontrol.tokens.send.elastic.available                       | 24 MiB  
        -  kvflowcontrol.tokens.send.elastic.deducted                        | 30 MiB  
        -  kvflowcontrol.tokens.send.elastic.deducted.force_flush_send_queue | 0 B     
        -  kvflowcontrol.tokens.send.elastic.deducted.prevent_send_queue     | 0 B     
        -  kvflowcontrol.tokens.send.elastic.returned                        | 30 MiB  
        -  kvflowcontrol.tokens.send.elastic.returned.disconnect             | 0 B     
        -  kvflowcontrol.tokens.send.elastic.unaccounted                     | 0 B     
        -  kvflowcontrol.tokens.send.regular.available                       | 48 MiB  
        -  kvflowcontrol.tokens.send.regular.deducted                        | 0 B     
        -  kvflowcontrol.tokens.send.regular.deducted.prevent_send_queue     | 0 B     
        -  kvflowcontrol.tokens.send.regular.returned                        | 0 B     
        -  kvflowcontrol.tokens.send.regular.returned.disconnect             | 0 B     
        -  kvflowcontrol.tokens.send.regular.unaccounted                     | 0 B     
        +  kvflowcontrol.tokens.eval.elastic.available                       | 24 MiB   
        +  kvflowcontrol.tokens.eval.elastic.deducted                        | 30 MiB   
        +  kvflowcontrol.tokens.eval.elastic.returned                        | 30 MiB   
        +  kvflowcontrol.tokens.eval.elastic.returned.disconnect             | 8.0 MiB  
        +  kvflowcontrol.tokens.eval.elastic.unaccounted                     | 0 B      
        +  kvflowcontrol.tokens.eval.regular.available                       | 48 MiB   
        +  kvflowcontrol.tokens.eval.regular.deducted                        | 0 B      
        +  kvflowcontrol.tokens.eval.regular.returned                        | 0 B      
        +  kvflowcontrol.tokens.eval.regular.returned.disconnect             | 0 B      
        +  kvflowcontrol.tokens.eval.regular.unaccounted                     | 0 B      
        +  kvflowcontrol.tokens.send.elastic.available                       | 24 MiB   
        +  kvflowcontrol.tokens.send.elastic.deducted                        | 30 MiB   
        +  kvflowcontrol.tokens.send.elastic.deducted.force_flush_send_queue | 0 B      
        +  kvflowcontrol.tokens.send.elastic.deducted.prevent_send_queue     | 0 B      
        +  kvflowcontrol.tokens.send.elastic.returned                        | 30 MiB   
        +  kvflowcontrol.tokens.send.elastic.returned.disconnect             | 8.0 MiB  
        +  kvflowcontrol.tokens.send.elastic.unaccounted                     | 0 B      
        +  kvflowcontrol.tokens.send.regular.available                       | 48 MiB   
        +  kvflowcontrol.tokens.send.regular.deducted                        | 0 B      
        +  kvflowcontrol.tokens.send.regular.deducted.prevent_send_queue     | 0 B      
        +  kvflowcontrol.tokens.send.regular.returned                        | 0 B      
        +  kvflowcontrol.tokens.send.regular.returned.disconnect             | 0 B      
        +  kvflowcontrol.tokens.send.regular.unaccounted                     | 0 B      
         
         
         -- Observe only the merged replica with its own three streams.
         SELECT range_id, count(*) AS streams
             FROM crdb_internal.kv_flow_control_handles_v2
        --- FAIL: TestFlowControlRangeSplitMergeV2/v2_enabled_when_leader_level=2/kvadmission.flow_control.mode=apply_to_elastic (33.48s)

Parameters:

• attempt=1
• race=true
• run=3
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-55831

[tbg]

At first glance, it looks like the listings just aren't in the same order. Maybe a sort is missing.

[tbg]

kvflowcontrol.tokens.{eval,send}.elastic.returned.disconnect

is 8mb when it should be 0.

I'm out of time this week to do something substantial about the annoying flakiness inherent in all of these flow control tests, but it can't hurt to not stress them.

[tbg]

I've got a linked PR that skips all of these tests under duress. I won't backport it all the way back, since there was some code movement, so any future flakes on this particular should just be closed out.