Improve UX around portable lockdirs

[gridbugs]

Using this issue to collect cases of UX confusion around portable lockdirs.

When solving dependencies dune prints all packages in the solution, which may include multiple different versions of the same package

For example:

- sexp_pretty.v0.15.1
- sexp_pretty.v0.17.0
- sexplib.v0.15.1
- sexplib.v0.17.0
- sexplib0.v0.15.1
- sexplib0.v0.17.0
- spawn.v0.17.0
- splittable_random.v0.15.0
- splittable_random.v0.17.0
- stdio.v0.15.0
- stdio.v0.17.0
- stdlib-shims.0.3.0
- system-mingw.1
- textutils.v0.15.0
- textutils.v0.17.0
- textutils_kernel.v0.15.0

Only one version of each package will be built on any given platform, but different platforms may require different versions of the same package. One possibility for making this less confusing would be to annotate packages in the output of dune pkg lock with the condition under which they will be chosen if they aren't chosen on all platforms.

[rgrinberg]

I discussed this with @maiste and @art-w offline and issue with the current implementation is that multiple versions for the same package are going to be selected far more often than the user expects. It is much more common for the user to want the following additional constraint:

If a package is selected for more than one configuration, the solver should try to satisfy both configurations with the same package version.

Or put another way, multiple versions of a package should be opt-in and never offered by default.

We've discussed how we might want to express this, and it seems like we'd really want to express this constraint in the solver and only solve a single SAT problem for the all configurations at once. I don't have concrete ideas how to do this so far, but this direction seems interesting to me.

[gridbugs]

we'd really want to express this constraint in the solver and only solve a single SAT problem for the all configurations at once

I couldn't agree more. Solving a single SAT problem rather than one problem per platform is the better option for many reasons. The choice to implement it with one problem per platform came from internal discussions at Tarides where we concluded that we'd do it this way first as it's easier/faster to implement, but format lockfiles in such a way that the solver algorithm can be changed at a later time.

[gridbugs]

Regarding the original issue of Dune printing all the deps in one list, leading to confusion when a single package appears multiple times with different versions, I'm working on an alternative way of printing the dependencies when portable lockdirs is enabled which should make this clearer. Dependencies common to all platforms are printed separately from those which are platform-specific, and each supported platform with specific dependencies gets printed along with the dependencies only installed on that platform.

Here's an example:

$ dune pkg lock
Solution for dune.lock

This solution is supports the following platforms:
- arch = arm64; os = linux
- arch = arm64; os = macos
- arch = arm64; os = win32
- arch = x86_64; os = linux
- arch = x86_64; os = macos
- arch = x86_64; os = win32

Dependencies on all supported platforms:
- base.v0.17.3
- base-unix.base
- csexp.1.5.2
- dune-configurator.3.20.2
- jane-street-headers.v0.17.0
- jst-config.v0.17.0
- ocaml.5.3.0
- ocaml-base-compiler.5.3.0
- ocaml-compiler.5.3.0
- ocaml-compiler-libs.v0.17.0
- ocaml-config.3
- ocaml_intrinsics_kernel.v0.17.1
- ppx_assert.v0.17.0
- ppx_base.v0.17.0
- ppx_cold.v0.17.0
- ppx_compare.v0.17.0
- ppx_derivers.1.2.1
- ppx_enumerate.v0.17.0
- ppx_expect.v0.17.3
- ppx_globalize.v0.17.2
- ppx_hash.v0.17.0
- ppx_here.v0.17.0
- ppx_inline_test.v0.17.1
- ppx_optcomp.v0.17.1
- ppx_sexp_conv.v0.17.1
- ppxlib.0.37.0
- ppxlib_jane.v0.17.4
- sexplib0.v0.17.0
- stdio.v0.17.0
- stdlib-shims.0.3.0
- time_now.v0.17.0

Additionally, some packages will only be built on specific platforms.

arch = arm64; os = macos:
- cmdliner.1.3.0

arch = arm64; os = win32:
- cmdliner.2.0.0
- conf-mingw-w64-gcc-i686.1
- flexdll.0.44
- ocaml-env-mingw32.1
- system-mingw.1

arch = x86_64; os = macos:
- cmdliner.1.3.0

arch = x86_64; os = win32:
- arch-x86_64.1
- cmdliner.2.0.0
- conf-mingw-w64-gcc-x86_64.1
- flexdll.0.44
- ocaml-env-mingw64.1
- system-mingw.1

I think this addresses the primary usability issue with portable lockdirs, which is that users were confused about why multiple versions of the same package are included in a solution.

[gridbugs]

It's possible to take this further and make it an error if a solution contains multiple versions of the same package (possibly surpressed by some configuration). Either way I think clarifying which platforms are included in the solution and which packages are specific to those platforms is a better message to print after solving, so I'll just start with changing the message.

[rgrinberg]

It's possible to take this further and make it an error if a solution contains multiple versions of the same package (possibly surpressed by some configuration)

Given that we can't guide the solver towards such a solution, this is probably premature. Stay tuned though, there might be a way forward with a new solver.

[gridbugs]

In the meantime, how do you feel about enabling portable lockdirs by default once #12620 is merged?

[rgrinberg]

What would be the default set of problems that a user would solve for?

[gridbugs]

It will solve for platforms Linux (x64, arm64), MacOS (x64, arm64) and Windows x64.

[rgrinberg]

Which set of distros does that include? Anyhow, It's a reasonable set of platforms but I'm not sure how good default it would be. Would it be too intrusive to just ask the user to specify the set of platforms they intend to support?

EDIT: Actually, this set of platforms isn't reasonable. There's no need to include windows on this list until package management can actually build things on Windows.

[gridbugs]

Which set of distros does that include?

The default platforms don't specify distros, though this can be configured in the custom list of platforms in dune-workspace. If someone needs a package solution which is distro specific they're expected to add that platform to their dune-workspace. Note that this doesn't cause problems for depexts because lockfiles contain a copy of the entire depext section from the opam file for its package.

Actually, this set of platforms isn't reasonable. There's no need to include windows on this list until package management can actually build things on Windows.

Ok that's fair, I can remove windows from the set of default platforms.

[Alizter]

I believe this issue has almost been fully addressed. There is still however the remaining question of which default set of platforms do we target.

Let me recall the proposals thusfar (please feel free to expand on these points):

1. Target the platform of the user running the lock

Advantages:

• We only solve once
• Works out of the box for each platform

Disadvantages

• Expanding the platform set has to be done by configuring it manually
• Not immediately shareable in a repo with the guarantee it will work on the platforms we are interested in.

2. Target a predefined default set of platforms

Advantages:

• Works out of the box for each supported platform
• Everybody produces the same lock file

Disadvantages:

• We solve multiple times (though could be combined into single SAT or use different solver)
• Might not work for platforms not in the target platform, even if they are locking

3. Mutate the list of supported platforms for each new platform

Advantages:

• Works out of the box for each platform
• Platforms targeted grow as potential users grow

Disadvantages:

• Unintuitive
• Not reproducible across platforms
• Relies on order
• Not everyone wants random platforms appended

---

I suggested 3 a few meetings back, but I don't like it anymore. My own feeling has been between 1 and 2. Right now, 1 seems sensible.

[shonfeder]

(3) seems like the best option, IMO.

Addressing the list of supposed disavantages,

• What is unintuitive about it? It fits my intuition.
• In what sense is it "Not reproducible across platforms"? IIUC, each platform should be able to reproduce the locked dependency plan checked in for the platform. If there's isn't one, it can add a new solution. Relocking is not "reproducible" generally in any case, unless all the inputs are fixed (e.g., opam repo has, same platform etc.). So what is not provided here?
• What does "Relies on order" mean?
• "Not everyone wants random platforms appended": I don't understand this either. Why would random platforms be appended?

Regarding the other two options:

(1) seems to miss the whole point of the portable format, since we could already do this without the portable format. "Not immediately shareable in a repo with the guarantee it will work on the platforms we are interested in." this seems like it should be avoided as much as possible. Requiring esoteric configurations to build a project on each new platform when we have an automated solution available seems to have upside for users, afaict.

(2) Is preferable to (1), in that it is at least what we already have (so requires no extra work), and it is compatible with (3). But I understand the critiques here and agree that (3) is preferable in its simplicity, and in that locks then certify solvability (and if they are checked in, buildability) on actual platforms.