Dune support

[ejgallego]

Hi folks,

the Coq mode for Dune is getting more mature, and a big blocker now is integration with Proof General. I am unsure how to best approach this; basically PG should add the right flags to make libraries compiled in _build into scope.

A possibility is indeed to let dune handle this; so PG could call dune coqtop foo.v and indeed coqtop would be started with the right settings. WDYT?

[cpitclaudel]

A possibility is indeed to let dune handle this; so PG could call dune coqtop foo.v and indeed coqtop would be started with the right settings. WDYT?

Sounds reasonable, but we'd need to be able to determine when dune is in use.

Could dune generate a _CoqProject file for us? In my projects I just write it by hand:

-R _build/default/coq/ Xyz
-Q _build/default/tests/ Tests
-Q _build/default/examples/ Examples
-R _build/default/examples/rv RV

Also, we need some indication of where each .v file came from, so that jump-to-definition can go to the right place.

[ejgallego]

Could dune generate a _CoqProject file for us?

It could, but I'm not sure I'm willing to do this; notably the _CoqProject format will have trouble keeping with Dune features; also it pollutes the source tree. Experience with generating .merlin files in OCaml indicates that this is a path we don't want to follow.

Also, we need some indication of where each .v file came from, so that jump-to-definition can go to the right place.

What do you concretely mean? If you use dune coqtop foo.v you would be using the right file already; otherwise you could implement something like tuareg does [it understands _build]

dune provides some APIs to query projects, but I guess I'd need to see the concrete use case. I'm not sure if this index functionality is something to have in PG, or instead PG should rely on an external too such as ocp-index for OCaml.

[ejgallego]

notably the _CoqProject format will have trouble keeping with Dune features

Yeah, already we cannot generate a _CoqProject that works for a simple case with two composed libraries, and those with different flags.

[cpitclaudel]

It could, but I'm not sure I'm willing to do this; notably the _CoqProject format will have trouble keeping with Dune features; also it pollutes the source tree. Experience with generating .merlin files in OCaml indicates that this is a path we don't want to follow.

OK. Then running dune coq sounds good. How do we detect dune projects reliably? Do we just search for dune-project files?

What do you concretely mean?

When I press M-. in company-coq it issues a Locate query for the current symbol. Then it uses Locate Library to find where the library came from. For example M-. on nat yields this:

<prompt>Coq < 17 || 0 < </prompt>Timeout 1 Locate nat.
Inductive Coq.Init.Datatypes.nat
<prompt>Coq < 19 || 0 < </prompt>Timeout 1 Locate Library Coq.Init.Datatypes.
Coq.Init.Datatypes has been loaded from file
/home/clement/.opam/default/lib/coq/theories/Init/Datatypes.vo

Then it jumps to the corresponding .v file: /home/clement/.opam/default/lib/coq/theories/Init/Datatypes.v

With dune this doesn't work: instead of going to the original .v file it jumps to the copy that dune made in _build. For OCaml merlin detects this. I need to detect it too in company-coq. So the question is: how do I find which original file a .v file in _build is a copy of?

(We discussed this on gitter a while ago:

@cpitclaudel
Hey @ejgallego, we've been using dune to build our latest Coq project, and it's working very nice. Thanks again!
There's one thing I need to tweak in company-coq for it to work perfectly, though: currently jump-to-definition goes to teh .v file in _build/default because that's where the .vo and .v file are. Do you have ideas on how company-coq could learn where the original .v file was?

@SkySkimmer
remove the _build/default prefix?

@cpitclaudel
Yes, but that feels a bit ah-hod, and IIUC dune doesn't always use that prefix.

@SkySkimmer
how does merlin do it? I know dune generates some .merlin files for it but don't know how it links the B and S lines together

@cpitclaudel
I don't know either :) But you're onto something. Maybe dune could generate a _CoqProject (I think it currently doesn't?) that would include the right info
Emilio Jesús Gallego Arias

@ejgallego
Yup merlin + dune is an open problem, upstream devs are working on it, it is a bit painful as you have to call dune at every check.
What I am thinking is to provide a dune info command that editors could use, given a file foo.v, dune info foo.v would provide:

• flags
• list of dependencies
• original location
  as indeed you are correct @cpitclaudel the location of the files is highly dependent on quite a few factors
  as for merlin, it will basically do something like that in the future, likely keeping a pipe open to dune, as it is expensive sometimes, think of the whole duniverse

@cpitclaudel
Very interesting, thanks a lot. But since Dune already generates a merlin file, could it not enhance that with extra info so that it doesn't have to be called each time? (this might be naive)
And in the Coq case, couldn't we put the relevant info in the _CoqProject?
I'm thinking something like a map file that would say where each .vo file came from

Emilio Jesús Gallego Arias
@ejgallego
@cpitclaudel both .merlin and _CoqProject have quite serious problems, for example they cannot do per-file flags, etc... So indeed Dune will stop generating .merlin files pretty soon I heard.

I'm thinking something like a map file that would sya where each .vo file came from

Yup, dune info should provide that, however I'm curious why do you need to find the information about the .vo files instead of the .v file, which is what the editor uses?

@cpitclaudel
To locate a symbol I do Locate xyz.and then Locate Library A.B.C, which points me to a .vo file

@ejgallego

To locate a symbol I do Locate xyz.and then Locate Library A.B.C, which points me to a .vo file

I see, indeed that's a bit tricky, as imagine A.vo is coming from a generated A.v file which is not in the source tree, I'll have a look how merlin does handle locate, but indeed we could provide a reverse-map

[ejgallego]

OK. Then running dune coq sounds good. How do we detect dune projects reliably? Do we just search for dune-project files?

That's the standard method, we could add a variable in case the user wants to override?

We discussed this on gitter a while ago

Oh yes I remember that, I thought it had happened in an issue but I couldn't locate it.

Actually OCaml has the same problem sometimes [you use M-. but it sends you to the _build dir] but tuareg got a hack for it; I'll have a look and discuss with dune / OCaml upstream to see what the plan regarding this is.

ocp-index seems to use a hack, see https://github.com/OCamlPro/ocp-index/blob/master/libs/indexBuild.ml#L276

Actually that doesn't bother me too much; for now the large majority of users will be satisfied by that; we could add a locate-build-prefix var for projects that use a special prefix.

The medium term plan is to have something similar to the LSP protocol locate call; still the relation between the lsp-server and the build tool has not been fully determined tho.

[Matafou]

Isn't it the dune file that we want to detect here? dune-project is not necessary is it?
Be careful, we use the _CoqProject informations for other actions than just launching coqtop.
For instanceIt is used by "auto-compilation on Require" (which detects if dependencies of a file needs recompilation and preforms it), and by company-coq as already remarked. @cpitclaudel how many other of your features need the _CoqProject infos?

So if we accept dune project files we need to either parse it or be able to query dune for several things.

[cpitclaudel]

@cpitclaudel how many other of your features need the _CoqProject infos?

It's not so much that company-coq needs _CoqProject; it's that dune copies source files into a separate directory, so it adds extra work to find out where things came from.

Maybe dune could add a comment at the end of the file, or something. Or maybe it could have a command to figure out where a file came from.

[Matafou]

OK and no other feature of company-coq needs the compile-options?

[cpitclaudel]

Everything is queried through Coq, so they do need the options, but indirectly ^^

[ejgallego]

Note that dune coqtop foo.v will update the dependencies of foo.v automatically, as it does for the OCaml version. It will also set the right flags for foo.v.

[erikmd]

@ejgallego IIUC you suggest that the main place from PG codebase where coqtop is called to run the REPL, we should replace for example (if I open a file from ValidSDP library) the command:

coqtop -topfile …/validsdp/theories/validsdp.v -emacs -I …/validsdp/plugins/soswitness/src -R …/validsdp/plugins/soswitness/theories ValidSDP -R …/validsdp/plugins/soswitness/src ValidSDP -R …/validsdp/plugins/soswitness/test-suite ValidSDP -R …/validsdp/test-suite ValidSDP -R …/validsdp/theories ValidSDP

with just dune coqtop -topfile …/validsdp/theories/validsdp.v -emacs ?

If yes, would this work as well with dune coqidetop …? (to be able to use Coq's XML protocol)

[ejgallego]

Not even -topfile should be needed IMO, but indeed yes, dune knows how to compute the flags and includes for a particular file so IMO it is better not to duplicate that logic in emacs; the same could be applied to coqidetop, good suggestion indeed!

[erikmd]

BTW I have the same question as
@Matafou

Isn't it the dune file that we want to detect here? dune-project is not necessary is it?

also we may want to detect if dune itself is in the path.

anyway correct me if I'm wrong, but it seems we'll also need to keep _CoqProject support functional (say, as a fallback if the dune binary and project file is not found)

[ejgallego]

dune-project indicates the root, IMHO it is better to issue locate-dominating-file buffer-file-name for dune-project and assume that's the root of the project. dune files can appear in many subdirs but are really small pieces.

I guess it boils down to whether you want to do dune coqtop theories/foo.v or cd theories && dune coqtop foo.v

I tend to prefer working from the root; keep in mind that dune coqtop foo.v would compile foo's deps, so IMHO reported paths in case of errors would look more natural.

Keeping _CoqProject is fine, whether a project wants to support both build systems is IMHO down to them.

Regarding locate I suggest we follow fow now the "hack" approach as done in ocp-index ; the issue is being discussed upstream so it seems to me that waiting to see what happens there may be sensible vs rolling our own complex solution.

[cpitclaudel]

dune-project indicates the root, IMHO it is better to issue locate-dominating-file buffer-file-name for dune-project and assume that's the root of the project. dune files can appear in many subdirs but are really small pieces.

That sounds good, though we should use default-directory, not buffer-file-name. Also, this will break for some projects using dune, so we should be careful to add a setting to disable it (my Coq projects using dune have multiple libraries depending on each other, and these dependencies are captured in a _CoqProject but not in a dune file. Maybe the latest changes make that unnecessary?

I guess it boils down to whether you want to do dune coqtop theories/foo.v or cd theories && dune coqtop foo.v

Note that doing the former will change the way commands producing files behave, I think (not necessarily a bad thing, just pointing it out).

Keeping _CoqProject is fine, whether a project wants to support both build systems is IMHO down to them.

I think it's also that we want to support projects with just one of these systems.