Conditional breakpoints are slow

[viktor-ferenczi]

1. What version of Delve are you using (dlv version)?

Delve Debugger
Version: 1.2.0
Build: ac3b1c7

2. What version of Go are you using? (go version)?

go version go1.12.2 windows/amd64

3. What operating system and processor architecture are you using?

Windows 10 64bit
AMD Ryzen 1800X

4. What did you do?

Put a conditional breakpoint inside an inner loop running millions of times. The condition is on the integer loop variable.

5. What did you expect to see?

I expected the loop run up to a few times slower due to the conditional breakpoint.

6. What did you see instead?

The loop runs about 1000x slower, which makes the conditional breakpoint useless if the loop count is higher than a few thousand.

A speedy workaround is to put the same condition as an if statement and an unconditional breakpoint inside it, but it is inconvenient, since needs recompilation and re-running the binary.

I am using delve via GoLand as the GUI. JetBrains tested my code (see below) via delve (without GoLand) and it had the same performance issue. So this is not due to the GUI integration.

Binary inside GoLand: %AppData%\Local\JetBrains\Toolbox\apps\Goland\ch-0\191.6183.86\plugins\go\lib\dlv\windows>dlv

How to reproduce:

package main

import ( 
    "fmt" 
    "time" 
)

func main() { 
    sum := int64(0) 
    start := time.Now() 
    for value := int64(0); value < 10000000; value++ { 
        sum += value 
    } 
    elapsed := time.Since(start) 
    fmt.Printf("Sum: %d\nTook %s", sum, elapsed) 
}

Run without debugging:

Sum: 49999995000000
Took 6.0053ms

Run with debugger, without conditional breakpoint:

Sum: 49999995000000
Took 24.0209ms

Put breakpoint with condition value == -1 on code line sum += value.

Debug run with conditional breakpoint:

Paused it after a few minutes and checked the current value. It got only 36k iterations far.

So I slashed the loop count by 1000 from 10M to 10000 and run it again:

Sum: 49995000
Took 32.4708425s

Based on the above result 10M iterations would have taken 32471 seconds (9 hours). More than 1000 times slowdown due to the presence of the conditional breakpoint.

This is why conditional breakpoints cannot be used conveniently inside inner loops right now, because they are plainfully slow.

Workaround is to add the condition to the code as an if statement and put a breakpoint inside it:

if value == -1 { 
    fmt.Print("Put breakpoint here") 
} 

It is barely slower than running the original code in debug mode:

Sum: 49999995000000
Took 28.0243ms

There must be some crazy high overhead of calling out to check those breakpoint conditions. I guess it calls out to the debugger, extracts the variable values from the stack / heap, then somehow interprets the condition there.

Would it be possible to apply the above workaround automatically at least if such a conditional breakpoint is set before starting the debugged application? It would help a lot with catching corner cases in inner loops and analyzing them in their context.

Maybe there is a way to add a call between each pair of instructions, allowing the debugger to inject conditional checks there while the code is already running. Or some "NOP" instructions which can be replaced later at runtime with a debug trap. There must be a solution, in case of C/C++ it had good performance.

I understand if this is not possible, then I just stick with my workaround.

[redlus]

Experiencing this too. Is there any known workaround that does not require re-compilation?

[arthurlopes]

Any news on this issue?

[aarzilli]

All reasonable optimization that could be done about this was done at the time. We may revisit this issue in the future to assess if there has been any slippage. The remaining slowness is either inherent to the mechanism used to implement conditional breakpoints (ptrace &c) or caused by #21827 (see #49848 for a longer explanation).