Improve stack support for HLS

[fendor]

Hi!

Setting the stage, Haskell Language Server's support for stack projects is currently lacking.
There are many reasons why using HLS with stack is a subpar experience.

I am trying to list the most important issues here, in the hope of better collaboration, so we can improve the developer experience with both HLS and stack.

Finding the compilation options of a filepath

Essentially, HLS is a compiler. As the compiler, it needs to know how to compile a component / file. Stack knows how to compile a file part of a stack project, and we want to extract that information.

The status quo is that we call a variation of stack repl with a special GHC shim that intercepts the ghc options when stack repl tries to invoke ghc --interactive. Then we use these options to compile the project ourselves.
This works reasonably well, but has some shortcomings. Most notably, the issues:

• "stack repl <test suite>" fails if "stack test --no-run-tests" is not run in advance for test suite containing errors #5380
• Internal library component not available for REPL #4564
• False positive: Main.hs (executable) does not recognise function defined in Lib.hs (library) haskell/haskell-language-server#1822
• Allow setting the working-dir to absolute directories #6135

Some of these are full-blown blockers and stack users have suffered in the past.

You can work around this using implicit-hie, which is its own can of worms, and not a proper solution for the needs of HLS.

What does HLS need

It is not a requirement for us to keep using this ugly hack of stack repl.

What we fundamentally need is this:

Given a filepath, we need to be able to find the exact compilation options for it (preferably for the whole component it belongs to) in the quickest way possible, with as little work as possible. Additionally, the compilation options must be enough to actually compile the file. Potential configure hooks must have been run before HLS tries to compile the file.
In the light of multiple home units coming to cabal and ghc, we would like even a bit more, but I can't specify right now what exactly.

Going forward

To get this stuff sorted out, there are multiple ways. In Cabal, we've added a couple of months/years ago the --enable-build-info flag with writes out build-info.json files containing the build information required to compile a component.

If someone wants to collaborate on this, I'd be happy to give more information, maybe even in a sync call.

[fendor]

@mpilgrem if possible, I would like to know what you think :)

[mpilgrem]

@fendor, I support the objective.

Stack has a command stack ide with subcommands that are provided with the aim of helping IDEs; and a command stack query that aims to output 'general build information' in JSON format. If something can be added to Stack (perhaps a new subcommand of stack ide or an enhancement of stack query?) that helps HLS support Stack users, that is all to the good.

In terms of helping with the implementation of the idea, I am conscious of my own limitations but I would do what I can to help.

To help me understand what is required from Stack, it would help to make the abstract more concrete, with a simple example. For example, if you had a plain vanilla stack new foo project, does a 'filepath' mean something like app/Main.hs or src/Lib.hs?

In this context, what is meant by 'all the exact compilation options'? If you command stack --verbose build --cabal-verbose, you can see the detail of how Stack is commanding Cabal (the library) (with the configure and build commands) and how (in turn) Cabal is commanding GHC. In this simple example, the configure step looks something like (on Windows):

--verbose=2 
--builddir=.stack-work\dist\8a54c84f 
configure 
--with-ghc=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.2.8\bin\ghc-9.2.8.exe 
--with-ghc-pkg=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.2.8\bin\ghc-pkg-9.2.8.exe 
--user 
--package-db=clear 
--package-db=global 
--package-db=C:\sr\snapshots\ee96b439\pkgdb 
--package-db=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\pkgdb 
--libdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\lib 
--bindir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\bin 
--datadir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\share 
--libexecdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\libexec 
--sysconfdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\etc 
--docdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--htmldir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--haddockdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--dependency=base=base-4.16.4.0 
--extra-include-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\include 
--extra-lib-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\lib 
--extra-lib-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\bin 
--exact-configuration 
--ghc-option=-fhide-source-paths

and the build step looks something like:

--verbose=2
--builddir=.stack-work\dist\8a54c84f 
build 
lib:foo 
exe:foo-exe 
--ghc-options " -fdiagnostics-color=always"

You may already know this, but Stack uses the version of Cabal (the library) that ships with the version of GHC in question, which may limit the availability to Stack of information provided by Cabal's --enable-build-info flag (it seems to me that the flag was first documented in the guide for Cabal 3.8).

[hasufell]

I'm somewhat familiar with stack codebase and don't mind contributing in this effort.

[mpilgrem]

Thinking out loud:

• other than at the command line, Stack is configured by a 'global' file common to all Stack projects (config.yaml - perhaps there is more than one on Unix-like operating systems: see https://docs.haskellstack.org/en/stable/yaml_configuration/#yaml-configuration), a unique project-level file (stack.yaml) and, for local packages within a Stack project, their individual package.yaml (Hpack) or Cabal files describing the package. Package description can be conditional, depending on, in part, Cabal flags and Stack's own configuration can include the setting or unsetting of Cabal flags;
• so, given a relative file path to a *.hs file, you need a function that identifies the project-level Stack configuration file. If the file is assumed to be relative to the working directory, you can follow Stack's own approach to identifying the relevant project-level configuration (which will be the project-level configuration for the 'global' project, as a last resort) - https://docs.haskellstack.org/en/stable/yaml_configuration/#yaml-configuration;
• once you have the project-level configuration (and, hence, know all the local packages in the Stack project) you can work out which local package(s) the *.hs is a 'member' of, by reference to the package descriptions - which might be none, one, or more than one. If more than one package (unlikely, but possible), I suppose you need a basis for picking just one - otherwise the 'compilation options' will not be unique for the file path.
• once you have the unique project (and any Cabal flags set in the Stack configuration) and the (unique) package (with its conditions satisfied), you should know what 'options' Stack will pass on to Cabal (the library) in respect of the file and, I suppose, what Cabal will pass on to GHC.

[fendor]

Thank you for your extensive comments!

Let me add some clarifications:

For example, if you had a plain vanilla stack new foo project, does a 'filepath' mean something like app/Main.hs or src/Lib.hs?

Yes, that is exactly correct. When talking about filepaths, I mean haskell source files that are part of a stack project.

In this context, what is meant by 'all the exact compilation options'?

Yes, what you correctly inferred. The problem, for example, with stack build is: it does too much work. It compiles the component/project, even though it doesn't need to, since we are going to rebuild all of that. For big projects, the start-up time would likely explode.

(it seems to me that the flag was first documented in the guide for Cabal 3.8).

That is true, but let's not delay improvements just because some people cannot benefit from it immediately.

---

I think stack ide and stack query are both fine commands, if we can tweak them to our needs, that'd be great :)

[mpilgrem]

@fendor, does the following assist?

As noted above, recent versions of Cabal (the library) offer the --enable-build-info flag. The Cabal documentation says that the flag causes Cabal to add a file build-info.json to the root of Cabal's 'build' directory.

stack Setup.hs configure --help suggests that the flag relates to the Cabal configure command. So, a Stack user can currently specify it for, say, all targets by adding to Stack's configuration file:

configure-options:
  $targets:
  - --enable-build-info

After a stack build, Cabal's 'build' directory is found at .stack-work/dist/<hash>. The location of that directory (relative to the Stack project-level confiuration file) is provided by stack path --dist-dir. With the above Stack configuration option set, it will include the build-info.json file.

For example, with a 'plain vanilla' stack new foo project (which currently uses Stackage LTS Haskell LTS 21.1, GHC 9.4.6 and Cabal-3.8.1.0), the build-info.json file contains (formatted):

{
  "cabal-lib-version": "3.8.1.0",
  "compiler": {
    "flavour": "ghc",
    "compiler-id": "ghc-9.4.6",
    "path": "C:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.4.6\\bin\\ghc-9.4.6.exe"
  },
  "components": [
    {
      "type": "lib",
      "name": "lib",
      "unit-id": "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
      "compiler-args": [
        "-fbuilding-cabal-package",
        "-O",
        "-outputdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-odir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-hidir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-stubdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-i",
        "-i.stack-work\\dist\\51f21a8f\\build",
        "-isrc",
        "-i.stack-work\\dist\\51f21a8f\\build\\autogen",
        "-i.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build",
        "-IC:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\msys2-20230526\\mingw64\\include",
        "-optP-include",
        "-optP.stack-work\\dist\\51f21a8f\\build\\autogen\\cabal_macros.h",
        "-this-unit-id",
        "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
        "-hide-all-packages",
        "-Wmissing-home-modules",
        "-no-user-package-db",
        "-package-db",
        "C:\\sr\\snapshots\\3fe291b3\\pkgdb",
        "-package-db",
        "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\.stack-work\\install\\9bf47987\\pkgdb",
        "-package-id",
        "base-4.17.2.0",
        "-XHaskell2010",
        "-Wall",
        "-Wcompat",
        "-Widentities",
        "-Wincomplete-record-updates",
        "-Wincomplete-uni-patterns",
        "-Wmissing-export-lists",
        "-Wmissing-home-modules",
        "-Wpartial-fields",
        "-Wredundant-constraints"
      ],
      "modules": [
        "Lib",
        "Paths_foo"
      ],
      "src-files": [],
      "hs-src-dirs": [
        "src"
      ],
      "src-dir": "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\",
      "cabal-file": ".\\foo.cabal"
    },
    {
      "type": "exe",
      "name": "exe:foo-exe",
      "unit-id": "foo-0.1.0.0-8kxBowjWnorJiiEogpBLXn-foo-exe",
      "compiler-args": [
        "-fbuilding-cabal-package",
        "-O",
        "-outputdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-odir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-hidir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-stubdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-i",
        "-i.stack-work\\dist\\51f21a8f\\build",
        "-iapp",
        "-i.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen",
        "-i.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build",
        "-IC:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\msys2-20230526\\mingw64\\include",
        "-optP-include",
        "-optP.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen\\cabal_macros.h",
        "-hide-all-packages",
        "-Wmissing-home-modules",
        "-no-user-package-db",
        "-package-db",
        "C:\\sr\\snapshots\\3fe291b3\\pkgdb",
        "-package-db",
        "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\.stack-work\\install\\9bf47987\\pkgdb",
        "-package-id",
        "base-4.17.2.0",
        "-package-id",
        "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
        "-XHaskell2010",
        "-Wall",
        "-Wcompat",
        "-Widentities",
        "-Wincomplete-record-updates",
        "-Wincomplete-uni-patterns",
        "-Wmissing-export-lists",
        "-Wmissing-home-modules",
        "-Wpartial-fields",
        "-Wredundant-constraints",
        "-threaded",
        "-rtsopts",
        "-with-rtsopts=-N"
      ],
      "modules": [
        "Paths_foo"
      ],
      "src-files": [
        "Main.hs"
      ],
      "hs-src-dirs": [
        "app"
      ],
      "src-dir": "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\",
      "cabal-file": ".\\foo.cabal"
    }
  ]
}

There is, currently, no Stack command-line equivalent of the configuration option above. However, it would be (I think) trivial to add a stack build --enable-build-info flag that was equivalent to setting the configuration file.

[fendor]

I think that works in general...

The only disadvantage is that we can't load a component if one of its dependencies doesn't build due to type errors, I believe. It would be much nicer, if we had something like haskell/cabal#8726 in stack as well.

However, I will play around with it, time to polish https://github.com/fendor/cabal-build-info/

[mpilgrem]

@fendor, I did not follow what was the relevant part of haskell/cabal#8726. The initial post refers to a goal of starting GHCi with more than one package loaded. stack ghci already has a --package option that allows you to load GHCi with an additional package: https://docs.haskellstack.org/en/stable/ghci/#specifying-extra-packages-to-build-or-depend-on

[fendor]

In the process of that PR, the author implemented the concept of a promised-dependency. Ie a dependency that is promised to exist at compile time but not at configure time. Currently, if you configure a package and one of the dependencies is not already installed in the package-db, the build crashes.

The promised-dependency features allows us to configure all components and then load them into ghci immediately, without having to build anything from stack's perspective, speeding up the startup time of HLS immensely. As a nice side effect, it allows us to load a project, even if not all local dependencies typecheck at the moment, since the build cannot fail in that case.

[mpilgrem]

@fendor, thanks. Adding here a link to the blog post on that topic: https://well-typed.com/blog/2023/03/cabal-multi-unit/.