Feature request: Support for executable-only projects for dune top/utop

[kiranandcode]

Desired Behavior

This follows on from the discussion on discourse: Request for feedback about a proper way to auto-require packages

Currently, dune provides commands to automatically setup a toplevel for a project when it consists of one or more libraries, with commands like dune top returning the sequence of directives to import the external libraries used in the project, dune utop starting a utop instance with all the libraries used in the project preloaded.

When the dune project only contains an executable, these commands do not work as expected, and do nothing: dune top returns an empty sequence of directives, and dune utop starts a utop instance with no additional libraries loaded.

The user interface of tools like dune top and dune utop would be improved if they behaved consistently, irrespective of the particular structure of the dune project (only executable, libraries etc.).

Edit: following on from further discussions on discuss.ocaml.org, @rgrinberg mentioned that loading executable files may run side-effects that the user may not want - as a workaround to still provide a consistent experience to the user, it would be good if dune still loaded the external libraries used by the executable, and only loaded the executable itself if requested by the user.

Example

Given a dune project as follows:

(executable 
      (name example) 
      (libraries containers))

Running dune top should return:

$ dune top
#directory "/home/<user>/.opam/4.12.0/lib/containers";;
#load "/home/<user>/.opam/4.12.0/lib/containers/monomorphic/containers_monomorphic.cma";;
#load "/home/<user>.opam/4.12.0/lib/containers/containers.cma";;

Note that the directives above don't load the executable itself thereby avoiding any side-effects that the executable may have run. The user can then separately explicitly request the executable be loaded by passing an additional flag --load-executables

[ghost]

That's a nice idea

[rgrinberg]

@Gopiandcode would you like to send a PR for this? The solution is quite localized so it should be approachable for a new contributor. We can guide you through the relevant parts of the codebase (should be more than a few modules).

[kiranandcode]

@rgrinberg Certainly, it would be my pleasure. Where would you suggest I start? I guess most of the the changes would be localised within bin/top.ml and bin/utop.ml, no?

I assume the functionality could be implemented for dune top by checking if libs is empty at line 49 in bin/top.ml

dune/bin/top.ml

Line 49 in 3cd240e

let* libs =

          let* libs =
            Dune_rules.Utop.libs_under_dir sctx ~db ~dir:(Path.build dir)
          in

but this only changes the behaviour of dune top and not dune utop, so I'm guessing changes are required in bin/utop.ml as well?

[rgrinberg]

Certainly, it would be my pleasure. Where would you suggest I start?

Looks like you're on the right track already. The libs_under_dir ~dir function returns all the libraries (and preprocessors) that should be loaded into a toplevel invoked with $ dune top dir. Therefore, you only need to modify to consider executables as well as libraries. In this function, Dune_file.Library is the constructor for library stanzas. We need an analogous case for Dune_file.Executables. To get the list of Lib.t for an executable, you will need to go through Lib.DB.resolve_user_written_deps_for_exes instead of Lib.DB.resolve_when_exists.

I assume the functionality could be implemented for dune top by checking if libs is empty at line 49 in bin/top.ml

I'm not sure why we need this. I thought the feature we agreed on would love the libraries of executables regardless.

but this only changes the behaviour of dune top and not dune utop, so I'm guessing changes are required in bin/utop.ml as well?

The change I described above will work for both top and utop.

Lastly, I recommend that you start by writing a test case demonstrating the intended behavior. See test/blackbox-tests/test-cases/utop/ for some examples.

[kiranandcode]

I'm not sure why we need this. I thought the feature we agreed on would love the libraries of executables regardless.

Right, I thought Dune_rules.Utop.libs_under_dir was used as part of other parts of the build system, so didn't want to change their behaviour to avoid causing unrelated builds to fail, but having grepped the codebase again, indeed, I guess that function is used purely for top and utop.

The libs_under_dir ~dir function returns all the libraries (and preprocessors) that should be loaded into a toplevel invoked with $ dune top dir. Therefore, you only need to modify to consider executables as well as libraries. In this function, Dune_file.Library is the constructor for library stanzas. We need an analogous case for Dune_file.Executables. To get the list of Lib.t for an executable, you will need to go through Lib.DB.resolve_user_written_deps_for_exes instead of Lib.DB.resolve_when_exists.

Perfect, thanks for the suggestion - I had looked at modifying libs_under_dir myself previously to add a case for Executables, but had been stumped with regards as to how to convert it to a list of libraries - Lib.Db.resolve_user_written_deps_for_exes was exactly what I needed.

Created a pull request.

[rgrinberg]

Thanks. This is now fixed in master.