[RFC] Formalise support for directory targets

[ghost]

This issue is to track support for directory targets. The first step is to describe exactly what directory targets mean. @snowleopard pointed to this document which explains how the Microsoft tools handle directory targets. This seems like a good starting point.

[snowleopard]

So, here is a draft proposal, which we can turn into a proper RFC at some point.

Directory targets

A directory target corresponds to a file tree rooted at a specified path, for example, docs/*. Typical examples of rules producing directory targets are: unzipping an archive, running make in a vendored package, and building files with non-deterministic names (e.g. including the current date).

There are two ways to depend on a directory target:

• An opaque dependency on the whole file tree docs/*. Opaque dependencies are invalidated if the contents of the tree is changed in any way.
• A projection dependency on a specific file in the tree, e.g. docs/html/index.html. A projection dependency works like a usual file dependency and supports early cutoff. For example, if the docs/* directory is rebuilt and only docs/html/logo.png is modified, then the dependency on docs/html/index.html is considered to be up-to-date. Note that it is easy to make a mistake with such projection dependencies, for example, by forgetting that index.html actually does include the image docs/html/logo.png. In such cases, sandboxing will help since only the requested projection dependencies will be copied.

Constraints:

• It is not allowed to have two rules with the same directory target, i.e. like file targets, directory targets are exclusive (but see shared directory targets below).

• Directory targets cannot have nested file or directory targets, i.e. other rules are not allowed to declare targets within the file tree of a directory target.

Extensions

Here are some possible extensions of the above basic version to consider:

• Opaque directory targets: a rule may declare that its directory target is opaque, in which case projection dependencies on its content will be disallowed. One can also consider only partially opaque directory targets, where the contents of the directory is only partially visible.

• Shared directory targets: it is possible to allow multiple rules to write to the same directory target, as long as they do not write to the same files. In this case, depending on a directory target would mean depending on all of the rules that declare it as a target. Conceptually, this is more complex but it can be added as an extension.

I will be editing this comment to include other alternatives as the discussion goes, so that we can keep everything in one place.

[ghost]

I like this proposal :)

One thought: when a sandboxed rule depends on an opaque directory, we could produce a single symlink to the directory. That would save doing one symlink for each file. Though it's a bit dodgy as one could escape the sandbox by accident with ...

[rgrinberg]

@snowleopard

An opaque dependency on the whole file tree docs/*. Opaque dependencies are invalidated if the contents of the tree is changed in any way.

We currently interpret $ dune build dir/ as $ dune build @dir/default. I think it would make sense to change the interpretation of this build command to build the directory tree.

[snowleopard]

@rgrinberg Good point: I haven't thought about the user interface. I expect that directory targets would be relatively uncommon, so not sure it's worth changing the existing convention.

Perhaps, we could overload dune build dir/?

• If dir/ is a directory target, we build it. In this case, the alias @dir/default shouldn't be allowed to exist anyway.
• Otherwise, we build the alias @dir/default.

[rgrinberg]

The overloading you propose doesn't sound very intuitive to me. I would prefer an interpretation for $ dune build dir/ that matched building (deps dir/).

But then again, I also don't really find the current behavior as intuitive so I'm certainly not looking for reasons to preserve it.

[snowleopard]

Status update: I've started working on a draft implementation.

[rgrinberg]

Status update: Andrey implemented (almost) everything and the feature lives under an experimental extension.