Allow dependencies for executables

[talex5]

Desired Behavior

Dune should provide a way to say that an executable depends on another build target (at runtime). Without this, running a program with dune exec may run with a stale build of the dependency (or fail completely).

Example

main.ml:

let () =
  print_endline "Hello from main";
  Unix.execvp "stage2" [| "stage2" |]

stage2.ml:

let () =
  print_endline "Hello from stage 2"

dune

(executable
 (name main)
 (modules main)
 (libraries unix))

(executable
 (public_name stage2)
 (modules stage2))

Results:

$ dune exec -- ./main.exe
Hello from main      
Fatal error: exception Unix.Unix_error(Unix.ENOENT, "execvp", "stage2")
$ dune build @install
$ dune exec -- ./main.exe
Hello from main      
Hello from stage 2

Real projects where this is a problem:

• In ocaml-ci I have a separate process for the solver. The main process spawns this as a subprocess when testing locally. When I test it with dune exec, I may be testing the old version of the solver.
• In the CI unit-tests, I need to make sure the executables are built before running the tests. I use:

  (rule
    (alias runtest)
    (package ocaml-ci-service)
    (deps (alias ../install))
    (action (run ./test.exe)))
  

  This seems to ensure that the executables are built on the first run, but doesn't always run the tests again when the executables change.
• @avsm said on Slack:

  I was discussing this same issue with @NathanReb yesterday, as ocaml-mdx has the same issue (it needs to exec ocaml-mdx-test. I think this is a pretty good feature request to file on the Dune issue tracker -- inter-binary dependencies are really useful and missing.

• @emillon notes that this is related to dune exec for adverbial commands (time, afl, ...) #3171
• I test 0install using dune exec -- 0install, but this doesn't ensure that the GUI plugin is up-to-date.

[bobot]

#3104 also gives examples where you want all the ressource directories to be up-to-date.

[ghost]

This seems to ensure that the executables are built on the first run, but doesn't always run the tests again when the executables change.

If dune doesn't re-run the test when the executable changes, that is definitely a bug

[rgrinberg]

If dune doesn't re-run the test when the executable changes, that is definitely a bug

I wonder if this is related to the alias expansion bug @aalekseyev and @snowleopard fixed recently

[ghost]

It is. So there were two issues:

• if rule R depended on alias A and A depended on file F, then Dune didn't thought that R depended on the contents of F. So if F changed, then R wasn't re-executed
• if R was sandboxed, then F wasn't copied in the sandbox

the first fix feels like something we should backport in the 2.x branch. The second I'm less sure about because there is a performance degradation we should look more into before Dune 3.0.

[jchavarri]

For the record, I have been using libraries are "poor man deps" for this kind of case with some success. It's clunky,... but it helps workaround the problem until there is a better solution.

E.g.:

(executable
 (name main)
 (modules main)
 (libraries stage2_hook unix))

(executable
 (public_name stage2)
 (modules stage2))

(library
 (name stage2_hook)
 (modules stage2_hook))

;;; Dummy file just to be able to add a custom dependency to the executable

(rule
 (targets stage2_hook.ml)
 (deps %{bin:stage2})
 (action
  (with-stdout-to
   %{targets}
   (run echo ""))))

[gasche]

What is the current best approach for this problem? I have two public executables in my Dune project, foo and bar, and bar calls foo at runtime, and I would like to write a test for bar. I don't know how to specify, in the test for bar, that the test also depends on foo. So when I run the test, it passes or fails depending on the build order.

I wrote a synthetic repro case here: https://gitlab.com/gasche/dune-executable-dependencies-repro-case

# start from a clean state
$ dune clean

# this run of "dune runtest" fails
$ dune runtest
sh: line 1: foo: command not found
File "test/bar.expected", line 1, characters 0-0:
diff --git 1/_build/default/test/bar.expected 2/_build/default/test/bar.out
index 31f77a6..e69de29 100644
--- 1/_build/default/test/bar.expected
+++ 2/_build/default/test/bar.out
@@ -1 +0,0 @@
-Foo !

# dune build now also fails
$ dune build
sh: line 1: foo: command not found  
File "test/bar.expected", line 1, characters 0-0:
diff --git 1/_build/default/test/bar.expected 2/_build/default/test/bar.out
index 31f77a6..e69de29 100644
--- 1/_build/default/test/bar.expected
+++ 2/_build/default/test/bar.out
@@ -1 +0,0 @@
-Foo !

# but a second run (different ordering) succeeds
$ dune build

# and now the test passes
$ dune runtest

[gasche]

I just pushed a fix suggested by @nojb, which is to use (deps %{bin:foo} %{bin:bar}) in the test rule.