Unbreak ghcide and bump haskell-lsp version

[turion]

Motivation for this change

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

cc @

[cdepillabout]

In general, the Haskell stuff in nixpkgs tracks the latest LTS release. We don't override these package versions with our own updated versions (since this could break a lot of stuff).

If you look at the top of this list here, you can see that these packages are locked to versions from the latest LTS. (peti updates these weekly, you can watch live if you're interested: https://www.twitch.tv/peti343)

It appears that the latest version of ghcide requires haskell-lsp ==0.17.*, haskell-lsp-types ==0.17.*, which is later than the version from LTS-14.

However, hackage2nix generates the latest versions of packages in the LTS as well, so haskellPackages.haskell-lsp_0_18_0_0 and haskellPackages.haskell-lsp-types_0_18_0_0 should be available.

I suggest trying to override ghcide to use these two package versions and then jailbreaking the cabal file.

Here's an example of hnix overriding package versions as well as using doJailbreak:

nixpkgs/pkgs/development/haskell-modules/configuration-common.nix

Lines 230 to 233 in 63f6681

	hnix =
	generateOptparseApplicativeCompletion "hnix" (
	dontCheck (doJailbreak (super.hnix.override { these = self.these_0_7_6; }))
	);

Could you update this PR to remove the above lines from configuration-hackage2nix.yaml, and instead adding an entry for ghcide to configuration-common.nix?

[cdepillabout]

I suggest trying to override ghcide to use these two package versions and then jailbreaking the cabal file.

Oh, and let me know if this doesn't work. There is a way to force hackage2nix to generate arbitrary package versions (so we could make it generate haskellPackages.haskell-lsp_0_17_0_0 for ghcide).

[turion]

Thanks for your explanations! Do I have to run hackage2nix locally to test my changes?

[cdepillabout]

@turion No, peti runs hackage2nix locally, so you don't have to do it in this PR.

[turion]

Unfortunately ghcide does not build with haskell-lsp-0.18. We probably need to make a haskellPackages.haskell-lsp_0_17_0_0 , but I don't know how. In parallel, I'll start and (send a PR to) upgrade ghcide to build with a newer haskell-lsp.

[turion]

I see. I'm working towards 1. then first :)

[turion]

ghcide with haskell-lsp-0.18 is released. I don't really understand how this change gets here. Do I just wait and rebase?

[cdepillabout]

@turion

I don't really understand how this change gets here.

I think there is an automatic script that updates the haskell-updates branch with recent package versions from Hackage. (Or maybe it is semi-automatic and peti has to run it by hand)

You can see the current state of haskell-updates here (including a bunch of commits with the message hackage-packages.nix: automatic Haskell package set update):

#75429

At some point (maybe today or tomorrow?) one of these Haskell package set updates should contain the new version of ghcide. At that point you should be able to rebase this PR on haskell-updates and use the new version of haskell-lsp.

[turion]

Ok. Thanks for all your explanations!

Is this whole workflow documented somewhere? Just in case peti goes on a 2 month vacation and someone needs to pick up the reign ;)

[cdepillabout]

It is somewhat documented in the hackage2nix Readme:

https://github.com/NixOS/cabal2nix/tree/master/hackage2nix

There is also a youtube video showing the steps:

https://www.youtube.com/watch?v=qX0mgtSm360

Also, peti streams live these updates, so if you watch him doing it once or twice, you should get a good idea of how to do it:

https://discourse.nixos.org/t/please-join-the-new-weekly-haskell-updates-video-livestream/4420

[peti]

Can you please re-base this PR on top of a current version of haskell-updates to resolve the merge conflicts?

[turion]

Can you please re-base this PR on top of a current version of haskell-updates to resolve the merge conflicts?

@peti Yes, since the conflicts are only internal to haskell-updates. But it seems that you added a commit that obsoletes my PR. Also, it seems that it doesn't work anymore because haskell-lsp has advanced to 0.19.0.0 in the meantime.

[turion]

Another dependency, hie-bios, is broken on haskell-updates. Something with the test suite. I'll see whether I can fix it easily.

[turion]

Another dependency, hie-bios, is broken on haskell-updates. Something with the test suite. I'll see whether I can fix it easily.

Seems nontrivial to fix. I'll add dontCheck for the time being. This should be removed once there is an acceptable answer to haskell/hie-bios#25 (comment)

[turion]

Fixing hie-bios is separately in #75869.

[turion]

I'm a bit out of ideas. The upstream version for haskell-lsp is 0.19 by now, and ghcide is once again behind. It does not compile with the newer haskell-lsp, that's the latest thing I tried now. Do I have to pin the version of haskell-lsp at 0.18 somewhere? Is this a manual process that gets stuck every time upstream versions get out of sync?

[mpickering]

@turion You are discovering why people use haskell.nix.

[cdepillabout]

@turion What you can do is send a PR adding a line to this list:

nixpkgs/pkgs/development/haskell-modules/configuration-hackage2nix.yaml

Lines 2475 to 2536 in c8c7398

	extra-packages:
	- aeson < 0.8 # newer versions don't work with GHC 7.6.x or earlier
	- aeson-pretty < 0.8 # required by elm compiler
	- apply-refact < 0.4 # newer versions don't work with GHC 8.0.x
	- aws ^>= 0.18 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- binary > 0.7 && < 0.8 # keep a 7.x major release around for older compilers
	- binary > 0.8 && < 0.9 # keep a 8.x major release around for older compilers
	- blank-canvas < 0.6.3 # more recent versions depend on base-compat-batteries == 0.10.* but we're on base-compat-0.9.*
	- Cabal == 2.2.* # required for jailbreak-cabal etc.
	- Cabal == 2.4.* # required for cabal-install etc.
	- colour < 2.3.4 # newer versions don't support GHC 7.10.x
	- conduit >=1.1 && <1.3 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- conduit-extra >=1.1 && <1.3 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- containers < 0.5 # required to build alex with GHC 6.12.3
	- control-monad-free < 0.6 # newer versions don't compile with anything but GHC 7.8.x
	- dbus <1 # for xmonad-0.26
	- deepseq == 1.3.0.1 # required to build Cabal with GHC 6.12.3
	- generic-deriving == 1.10.5.* # new versions don't compile with GHC 7.10.x
	- gloss < 1.9.3 # new versions don't compile with GHC 7.8.x
	- haddock == 2.22.* # required on GHC 8.0.x
	- haddock-api == 2.22.* # required on GHC 7.8.x
	- happy <1.19.6 # newer versions break Agda
	- happy == 1.19.9 # for purescript
	- haskell-gi-overloading == 0.0 # gi-* packages use this dependency to disable overloading support
	- haskell-src-exts == 1.19.* # required by hindent and structured-haskell-mode
	- hinotify == 0.3.9 # for xmonad-0.26: https://github.com/kolmodin/hinotify/issues/29
	- hoogle == 5.0.14 # required by hie-hoogle
	- html-conduit ^>= 1.2 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- http-conduit ^>= 2.2 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- inline-c < 0.6 # required on GHC 8.0.x
	- inline-c-cpp < 0.2 # required on GHC 8.0.x
	- lens-labels == 0.1.* # required for proto-lens-descriptors
	- mainland-pretty == 0.6.2.* # required for tensorflow-opgen-0.1.0.0
	- mtl < 2.2 # newer versions require transformers > 0.4.x, which we cannot provide in GHC 7.8.x
	- mtl-prelude < 2 # required for to build postgrest on mtl 2.1.x platforms
	- network == 2.6.3.1 # newer versions don't compile with GHC 7.4.x and below
	- network == 3.0.* # required by network-bsd, HTTP, and many others (2019-04-30)
	- parallel == 3.2.0.3 # newer versions don't work with GHC 6.12.3
	- patience ^>= 0.1 # required by chell-0.4.x
	- persistent >=2.5 && <2.8 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- persistent-sqlite < 2.7 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- primitive == 0.5.1.* # required to build alex with GHC 6.12.3
	- proto-lens == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
	- proto-lens-protobuf-types == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
	- proto-lens-protoc == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
	- QuickCheck < 2 # required by test-framework-quickcheck and its users
	- resolv == 0.1.1.2 # required to build cabal-install-3.0.0.0 with pre ghc-8.8.x
	- resourcet ==1.1.* # pre-lts-11.x versions neeed by git-annex 6.20180227
	- seqid < 0.2 # newer versions depend on transformers 0.4.x which we cannot provide in GHC 7.8.x
	- seqid-streams < 0.2 # newer versions depend on transformers 0.4.x which we cannot provide in GHC 7.8.x
	- split < 0.2 # newer versions don't work with GHC 6.12.3
	- tar < 0.4.2.0 # later versions don't work with GHC < 7.6.x
	- these == 0.7.6 # required by hnix 0.6.1
	- transformers == 0.4.3.* # the latest version isn't supported by mtl yet
	- vector < 0.10.10 # newer versions don't work with GHC 6.12.3
	- xml-conduit ^>= 1.7 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod ^>= 1.4 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod-core < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod-form < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod-persistent < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod-static ^>= 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
	- yesod-test ^>= 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227

like

- haskell-lsp == 0.18.0.0 # for ghcide-0.4 (or whatever is the correct ghcide version)

Once this gets merge into haskell-updates and the update script run, it will create a haskell-lsp_0_18_0_0 derivation. This will exist as long as the haskell-lsp == 0.18.0.0 line exists.

You should be able to use this package in an override to ghcide. That will hopefully get ghcide building.

[turion]

@turion You are discovering why people use haskell.nix.

Hehe, yes ;) but I haven't given up on nixpkgs becoming good enough for standard usage yet.

[turion]

@cdepillabout I guess I could, but this is quite a manual and tiresome cycle. I'll just wait until upstream has released a new version. https://github.com/digital-asset/ghcide/issues/277

This whole workflow feels like it's going to break every time e.g. haskell-lsp releases a newer version than ghcide can handle. Is that correct? How would this situation ever stabilize?

[cdepillabout]

@turion Sorry this is somewhat of a manual/annoying process.

This whole workflow feels like it's going to break every time e.g. haskell-lsp releases a newer version than ghcide can handle. Is that correct?

For the most part, yes.

How would this situation ever stabilize?

There are some cases where the situation would stabilize:

• On release branches of nixpkgs.

  release branches of nixpkgs don't get updates to the Haskell package versions. So if you get ghcide working on one of the release branches, it will probably stay working forever.

• If ghcide and haskell-lsp are released to stackage (and end up in an LTS release).

  nixpkgs tracks the latest LTS release, so packages from the LTS releases are almost always working, because they are all "known compatible".

  If the upstream authors of ghcide and haskell-lsp don't want to get compatible versions into stackage, maybe you could?

• If you add specific versions of haskell-lsp and ghcide to extra-packages in configuration-hackage2nix.nix as suggested above.

  This will generate versions of haskell-lsp and ghcide like haskell-lsp_0_18_0_0 and ghcide_0_4_0_0 (or whatever) that you can use in overrides and are known to work together.

  These should continue working at least until other dependencies get too new.

• If ghcide and haskell-lsp stop frequently releasing new versions to Hackage.

---

Unfortunately, the Haskell stuff in nixpkgs doesn't work too well for new packages that aren't in Stackage and have frequent releases to Hackage.

As suggested by mpickering, creating an overlay with known package versions (or using a tool like haskell.nix to do this for you) might be the easiest solution.

Creating a ghcide.nix github repo that other people could also use might be helpful for the community as well.

[turion]

@cdepillabout Thank again you for your detailed explanations! It certainly helps me to understand the process and to more effectively contribute.

If ghcide and haskell-lsp are released to stackage (and end up in an LTS release).

Ah, good point! So LTS in Haskell on nixpkgs always refers to stackage LTS, I'm assuming now. There is a ticket tracking that: https://github.com/digital-asset/ghcide/issues/144
This sounds like the best option for me then. I'll see whether I can help with that.

[peti]

I have attempted to fix the ghcide build in Nixpkgs, too, but it is quite a mess. Honestly, I think it's much easier to work with upstream to fix ghcide's build with recent versions of its dependencies than trying to build it in Nixpkgs with old ones.

[cdepillabout]

I think @mdorman might have gotten ghcide working in #76103.

[domenkozar]

This is why I maintain https://github.com/hercules-ci/ghcide-nix

[peti]

@turion, do you still intend to update this PR? Or can it be closed?