Benchmark suite

[pepeiborra]

The benchmark suite is designed to be run with cabal bench. It does not attempt to compare with past performance, I think that's better tackled as a separate task.

The module comment in the benchmark module contains some details on the benchmark methodology and experiments, which currently cover the basics:

• hover
• go to definition
• completions
• code actions
• document symbols

I have not made any attempts to integrate with CI since I don't know how consistent the timings would be across runs.

Goes some way towards closing #242

[pepeiborra]

Example output:

$ cabal bench
Running 1 benchmarks...
Benchmark ghcide-bench: RUNNING...
starting test
Unpacking to bench/example/Cabal-3.2.0.0/
/Users/pepe/Dropbox/code/ghcide/dist-newstyle/build/x86_64-osx/ghc-8.10.1/ghcide-0.1.0/x/ghcide/build/ghcide/ghcide
    Alloc    Copied     Live     GC     GC      TOT      TOT  Page Flts
    bytes     bytes     bytes   user   elap     user     elap
ghcide version: 0.1.0 (GHC: 8.10.1) (PATH: /Users/pepe/Dropbox/code/ghcide/dist-newstyle/build/x86_64-osx/ghc-8.10.1/ghcide-0.1.0/x/ghcide/build/ghcide/ghcide) (GIT hash: b759359f457da747f5c4e8659cc2ce08c8441fcd)
Starting LSP server...
If you are seeing this in a terminal, you probably should have run ghcide WITHOUT the --lsp option!
...
...
TOTAL hover = 2.56s (10 repetitions)
TOTAL getDefinition = 2.67s (10 repetitions)
TOTAL documentSymbols = 1.43s (100 repetitions)
TOTAL documentSymbols after edit = 2.90s (100 repetitions)
TOTAL completions after edit = 4.53s (10 repetitions)
TOTAL code actions = 0.36s (10 repetitions)
TOTAL code actions after edit = 3.60s (10 repetitions)
Benchmark ghcide-bench: FINISH

[mpickering]

Looks useful @pepeiborra

Once my branch starts getting merged in then some of these tests will become trivial lookups in a HashMap so will probably have to be modified to show anything useful.

[ndmitchell]

This is awesome!

The one feature I'd have liked was the ability to do profiling. That requires installing ghcide with profiling (a user step), passing +RTS -p when starting ghcide (easy to change below) and then arranging for ghcide to kill itself rather than be forcibly terminated by lsp-test, which I didn't have a good sense on how to do. Of course, this feature could definitely come later.

[pepeiborra]

This is awesome!

The one feature I'd have liked was the ability to do profiling. That requires installing ghcide with profiling (a user step), passing +RTS -p when starting ghcide (easy to change below) and then arranging for ghcide to kill itself rather than be forcibly terminated by lsp-test, which I didn't have a good sense on how to do. Of course, this feature could definitely come later.

I implemented a nice way of allowing ghcide to exit cleanly which I think should work fine with profiling, although I have not checked that.

Another issue is that the benchmark invokes ghcide once per experiment, and afaik there is no way to redirect the profiling artifacts to a specific path, so the user would need to select only one experiment (or we change the test suite to run every experiment in a different folder).

[ndmitchell]

When I tried sending a close to ghcide it didn't exit, merely stopped responding to messages, so worth validating (I could well have screwed it up). Running ghcide on a single benchmark at a time seems what people would want to do, so no problem with that.

Note that you might need to roll the stack.yaml files or explicitly override to get the latest version of extra included.

[pepeiborra]

When I tried sending a close to ghcide it didn't exit, merely stopped responding to messages, so worth validating (I could well have screwed it up).

Maybe you didn't send a shutdown message followed by an exit message? That seems to work for me, at least the +RTS -s stats are being printed on exit. The only problem I found is that lsp-test kills the server before it gets a chance to actually exit, so I send the exit messages explicitly and then sleep for a bit.

Running ghcide on a single benchmark at a time seems what people would want to do, so no problem with that.

That's right, but then more user steps are needed, because the benchmark suite does not support selecting experiments to run.

[ndmitchell]

Ah, that would be it - I think I only sent a shutdown. Good to know, and even better that with this benchmark I don't need to know!

[mpickering]

Thanks Pepe. I left a few comments but nothing critical to stop the merge imo.

[cocreature]

Awesome, thanks!