package alternatives in Pkg3?

[stevengj]

I wonder whether we want Pkg3 to contain something like Debian alternatives: a package A can depend on package B or package C or package D.

For example, the Plots.jl package might want to depend on having at least one plotting package installed (cc @tbreloff).

Or, now that FFTW.jl is being split off into its own package, and that package in turn will probably depend on an AbstractFFTs package (JuliaMath/FFTW.jl#2), other packages may want to depend on having some package installed that implements FFTs following the AbstractFFT interface (e.g. FFTW, or a native Julia FFT, or FFTPACK, or MKL, etcetera).

[simonbyrne]

Similarly, there should be a way for users to specify which option they want to use when multiple are available, presumably via a TOML file?

[oxinabox]

The situtation perhaps is more complex.
Though we might be able to ignore the complex case and get the low hanging fruit.

• JuMP requires you to have a Solver package installed. of which there are 15
• If you want to solve a MIP then JuMP requires a MIP solver. Of which there are 8
• If you want to solve a MIP with lazy (callback) constrains then you need a MIP solver with that support. Of which there are 3 or 4
• if it is a NLMIP with lazy constraints you need such a solver. Of which there are none.
• If you both want to solve LP and MIP you need either two different solvers or one of 6 solvers that support both

Etc etc,
the constraints are fairly complex.

[stevengj]

@oxinabox, that doesn't seem too much more complex to me. It just means:

• A package can have multiple sets of optional dependencies.

• Optional dependencies should include "none" as an option (in which case the package should disable the corresponding functionality).

[tkelman]

Would this be mutually exclusive, with only one package allowed to actively satisfy the dependency at a time?

[stevengj]

@tkelman, I don't think so. At Pkg.build time, the package should be able to find out which (if any) of the optional dependencies were installed, and can then decide to use all of them or just one of them depending on how the package works.

But there should be a way for the user to indicate a preference if the package is just going to use one of the options.

[stevengj]

I'm thinking of an interface something like the following. Suppose MyPackage wants at least one of three different FFT packages, and optionally can use a plotting package or can disable plotting if no such package is present. All of these optional packages are declared as optional=true packages in the Pkg3 TOML file. In addition, you add section declaring the alternatives preference:

[option.fft]
packages = ["FFTW", "MKL", "FFTPACK"]

[option.plots]
packages = ["", "PyPlot", "Gadfly"]  # "" means it is okay to install nothing

Then, in Pkg.add("MyPackage"), if one of the options is not already present, it will install the default ("FFTW" for fft and nothing for plots).

Pkg.option("MyPackage", :plots) will return a list of all installed plots options (or an empty list if none), in the default order ["PyPlot", "Gadfly"] unless the user has changed the default. The build script for MyPackage will typically call this function to decide which option to configure with (if any).

Pkg.option("MyPackage", fft="MKL") will change the fft option to "MKL", installing "MKL" if needed, and rebuilding MyPackage if the option has changed; Pkg.option("MyPackage", :fft) will now list "MKL" first (returning ["MKL", "FFTW"] if FFTW is also installed).

You can specify more than one package, e.g. Pkg.option("MyPackage", fft=("MKL","FFTPACK")), and it will install both of these dependencies. Pkg.option("MyPackage", :fft) will return the dependencies with the user-specified ones first: ["MKL", "FFTPACK", "FFTW"]

You can also specify the option as keywords in Pkg.add or Pkg.build. e.g. Pkg.add("MyPackage", fft="MKL").

[StefanKarpinski]

@KristofferC and I have been brainstorming and debating this for a while and so far the best scheme we've come up with is something like this made up example:

[deps]
ABC   = "<uuid>"
Gtk   = "<uuid>"
TclTk = "<uuid>"
Qt    = "<uuid>"
XYZ   = "<uuid>"

[alts]
graphics = ["Gtk", "TclTk", "Qt"]

[build.deps]
Builder = "<uuid>"
Clang   = "<uuid>"
GCC     = "<uuid>"
MSVC    = "<uuid>"

[build.alts.compiler]
linux   = ["GCC", "Clang"]
macos   = ["Clang"]
windows = ["MSVC"]

[test.deps]
Test       = "<uuid>"
QuickCheck = "<uuid>"

There are two different things at play here – alternatives and targets:

• build and test are targets which add dependencies on top of the main set of dependencies.
• graphics is a set of alternatives for the default target, letting you choose between one of three graphics backends – Gtk, TclTk and Qt
• compiler is a set of alternatives for the build target that lets you choose a compiler in a platform-specific way – between GCC and Clang on Linux, Clang on macOS and MSVC on Windows.

The general rules are that:

• Every package in deps is requires by the default target unless it appears on the RHS of a set an entry in the atls section.
• The LHS of every entry in the alts section is required in place of its RHS alternatives.
• Thus, the default target on the example requires ABC, XYZ and graphics which may be one of Gtk, TclTk or Qt

Each target is a microcosm of the top-level with its own deps and alts sections. These can be merged as dicts, as in:

target_deps = merge(project["deps"], project[target]["deps"])
target_alts = merge(project["alts"], project[target]["alts"])

If there are any key collisions it should be an error – targets allowed to override the identity of a package. A generalization of this maybe this:

target_project = recursive_merge(project, project[target])

where recursive_merge recurses into dictionary values and errors on key collisions. The resulting target_project can be saved as a project file and used for executing the target – build, test, etc.

[StefanKarpinski]

The motivation for naming sets of alternatives is that it allows configuration files to specify them as in:

[config.SomePackage]
graphics = "Qt"
compiler = "Clang"

[mauro3]

Maybe its worth spelling alts out as I think it wouldn't be used that much? Unlike deps it's not a "word". Another word would be choices or choiceof, which reads well in above example.

[StefanKarpinski]

I was originally writing out alternatives but I got tired of typing it over again.

[stevengj]

I find the compiler to be a bit out of place … how is that fundamentally different from any other non-Julia dependency? Hopefully, the vast majority of projects will use something like BinaryProvider.jl and won't require a compiler.

[StefanKarpinski]

It's just an example of an system-specific alternative. Replace it with

[build.alts.glam]
linux   = ["Glow", "Sparkle"]
macos   = ["Sparkle"]
windows = ["Glint"]

if the specifics are bothering you.

[stevengj]

How does the user choose the alternative to be used?

It would be nice to support keywords in Pkg3.add, e.g. Pkg3.add("Foo", graphics="Tk", glam="Sparkle"), in addition to any kind of terminal-specific interactive menu.

[stevengj]

There should also be a way for the config file to specify a default choice, so that if you have a bunch of complicated options it will automatically select a reasonable one and only sophisticated users can opt to select a different option.

You also want to be able to change the option later, e.g. by calling Pkg.build again, and the options should also be exposed to the package itself so that it can query them at build time. See also my proposals in #38.

[StefanKarpinski]

I would expect that earlier alternatives are preferred so the ordering gives the preference. I don't see any reason why we can't support keyword arguments for picking alternatives. We can also interactively prompt people for which one they want.

[stevengj]

Should the config file be able to specify whether the first one is picked silently if the user doesn't specify (i.e. if there is a reasonable default), or if the user is always prompted for a choice?

[stevengj]

I also find it confusing how you are specifying platform-dependent options. What determines which options have platform-specific choices (build.alts.glam) and which ones don't (graphics)? Shouldn't all options potentially be either platform-specific or platform-agnostic?

And why is it [alts] and [build.alts.glam]? Shouldn't it be [alts.graphics] for consistency?

[StefanKarpinski]

If the dict value for an alternative is an array then it's platform-agnostic, if it's a dict of arrays, then it's platform-specific and the keys are used to choose which set of options apply.