Pkg3: Runtime Configuration - Specification of Installed Packages?

[ChrisRackauckas]

First of all, really nice work!

As noted in the DifferentialEquations plan for modularization (SciML/DifferentialEquations.jl#59), the only form of "conditional dependencies" that I really need is the ability to add/remove functionality at compile-time by knowing which packages are installed on the user's system. It seems like this can be done by the user specifying a flag in the Runtime Configuration file, but is there any way that this can be computed? As in, can this flag instead be determined a code which sees if packages X, Y, Z are installed, if so, allow me to use add functionality?

An example is that, at the top of the module, I'd want to:

if SUNDIALS_INSTALLED
  using Sundials
end

and then inside some higher level API functions have a branch which is allowed if SUNDIALS_INSTALLED (and error if not). If this is a possibility, then nothing would actually need to be changing at runtime, and thus the current conditional module problems that I have (which are not too difficult! I know there are more difficult conditional module problems which don't apply here) would be solved seamlessly to the user.

[tkelman]

Since this repo will contain additional juleps as time goes on, best to preface issue titles with which julep they're about.

[StefanKarpinski]

Here are some of the latest thoughts on this: JuliaLang/julia#15705 (comment). Basically, your package would provide a primary package module and "interaction modules" that are loaded when all of the packages that the interaction depends on are present. These are effectively separate packages but may live in the same repository as the main package. Does that address your issue?

[ChrisRackauckas]

I'm not entirely seeing it, so maybe a specific example helps. I have an alg keyword argument for a symbol for which algorithm to choose (ex: alg = :dopri5), and it checks vs dictionaries of algorithms to know which algorithm to call:

function solve(....)
  # Does some preliminary stuff
  if alg in DIFFEQ_ALGS
     #Run solve using one of these algs
     solve_using_diffeq(...)
  if alg in ODEINTERFACE_ALGS
     #Run solve using one of these algs
     solve_using_odeinterface(...)
  ### Etc, wrapping many packages
  end 
  # Build solution obj
end

In this case, the ODEINTERFACE_ALGS are conditional dependencies since they are a possible choice, but require a working Fortran compiler and I'd expect most users to not use those options.

Would this type of conditional dependency usage be supported by the interaction modules? For example, would it recompile the main module after loading so that way solve_using_odeinterface (which would live in the interaction module) will be available? Or is there a different design that will be able to do this effectively?

[tbreloff]

These are effectively separate packages but may live in the same repository as the main package

This would solve a bunch of existing problems, but not everything... particularly the goal of lazily initializing/loading a package when a user requires it, even if it was already installed.

Much of the Plots conditionals could be resolved like:

module Plots

# don't load this automatically
@optional module Plots_GR
    import GR
    # GR-specific definitions
end

function plot(...)
    ...
    if backend() == GR
        using Plots_GR
    end
    ...
end

Now, I'm pretty sure this could be done today, but it would be really important to find a way to do this without breaking precompilation. Many of the most prominent frameworks need to solve this design pattern, and none of the options are really good enough.

[ChrisRackauckas]

What are you gaining by being able to load packages lazily? It seems like if you let to precompile dependent on the currently installed backends, the "first time to plot" would be quicker.

[tbreloff]

It's not just timing... sometimes optional dependencies don't play well together (loading C libraries with global state and such) and so you don't want to load more than what you're using for stability.

[ChrisRackauckas]

That's a compelling enough reason to require a solution.