Next step for AbstractMCMC.jl

[torfjelde]

Recently there has been lots of discussions on whether AbstractMCMC is a good choice for "one interface to rule them all" in the sample-space in Julia. During these discussions it's become clear that the current implementation isn't quite sufficient.

The aim of this discussion is therefore to answer two questions:

1. What is missing or should be changed in AbstractMCMC.jl for it to become the go-to?
2. Can we address the issues identified in (1)?

If you have a gripe or something that you want addressed, make a new thread below rather than commenting on some other thread. This makes it much easier to identify the issues and get an overview of the discussions for the different issues.

[torfjelde]

Generalize scope of AbstractMCMC.jl

In the world of probabilistic programming there are more sampling-techinques than just Markov Chain Monte Carlo (MCMC).

Here we should discuss examples of what these can be and how the interface might look.

Identified issues

1. Rename AbstractMCMC.jl to AbstractSampling.jl?
2. Are the current step and sample sufficient?

We will update the identified issues as progress is made and/or new ones arise.

[torfjelde]

True. That's a point for renaming, isn't it?

[devmotion]

Yes, I guess we just never paid too much attention to the name. Since importance sampling was one of the earliest examples of the interface (IIRC), the name has never been complelety correct.

[devmotion]

However, I also think the name AbstractSampling is too broad and too general, probably.

[cscherrer]

What else would you call it?

[devmotion]

Weighted sampling with and without sampling, sampling of random variables etc. are all some sort of sampling, so I think AbstractSampling implies an unintended generality. So I think a name that makes it clear that it is sampling in a (Bayesian?) inference setting would be more appropriate. Maybe AbstractSamplingInference? Maybe one should also use the opportunity to get rid of the term Abstract - there are some abstract type definitions but it also contains concrete types for ensemble algorithms and a concrete implementation for standard and ensemble sampling. Maybe one could rather use the Base suffix idiom, e.g., something like SamplingInferenceBase.

[torfjelde]

How should packages hook into AbtractMCMC.jl? Glue-packages vs. not.

As per usual with such "interface" packages, a major question is: should PackageX.jl implement the AMCMC interface in PackageX.jl, or should the implementation of the AMCMC interface for PackageX.jl go in a glue-package AbstractMCMCPackageX.jl?

[cscherrer]

As long as package authors are willing to add support for AbstractMCMC, this is a non-issue. My only concern is that we shouldn't depend so strongly on this assumption, in case it sometimes might break.

[torfjelde]

As long as package authors are willing to add support for AbstractMCMC, this is a non-issue. My only concern is that we shouldn't depend so strongly on this assumption, in case it sometimes might break.

In the cases where the assumption fails, it seems like it can be addressed by adding a glue-package if needed? Or is there something that could go wrong with that approach?

[cscherrer]

No, I think that's enough. My early impression was that very few packages use AbstractMCMC, so glue packages are the norm. But it would be great if that can change

[devmotion]

Actually, I think so far AbstractMCMC is mainly used as an interface, I don't know any actual glue packages (Turing is not a proper one since it only defines sampling for its/DynamicPPLs samplers).

[cscherrer]

Just remembered another concern re: glue packages
#72 (reply in thread)

[torfjelde]

Chains-interface: go here or somewhere else?

There are currently several different implementations of chain-structures used in the context of sampling, e.g. MCMCChains.jl, SampleChains.jl.

As of right now [2021-05-18], we have a AbstractChain type in AbstractMCMC.jl, but there is no interface to speak of. This is mostly an artefact of how the Turing.jl ecosystem is set up, and is something we want to move away from.

Identified issues

1. Should there be any mention of chains in AbstractMCMC.jl at all?
2. If so, what are the bare-minimum behavior we expect from a chain-type? I.e. what interface should we enforce?

We will update the identified issues as progress is made and/or new ones arise.

[cscherrer]

If this becomes AbstractSampling and the AbstractChain is really generic enough to handle the cases in #72 (reply in thread) (and others the may arise) then having it here could make sense.

[cpfiffer]

I'm fine with it, actually. Some earlier ideas I had for AbstractMCMC included very general interfaces for chains, parameters, etc. but it wasn't totally fleshed out or carefully done.

I would expect any implementation of chains to do the following

• Something giving you an iterator across samples, in whatever shape they are in (rows in MCMCChains, named tuples in SampleChains, etc.).
• Something to get the available parameters in the chain (in MCMCChains this is params)
• getindex and/or getproperty methods with common behavior, though I'm less interested in unifying this. I think each chain can just index however they want.

Other than that, I really don't think you need too much else, though I'm sure we'd find good candidates for inclusion if we started adding this stuff.

[torfjelde]

Features lacking in AbstractMCMC.jl

AbstractMCMC.jl is not feature-complete in any sense. Here misisng features should go so we can have a discussion about which ones should make it into the package and which shouldn't.

Identified issues

1. Functionality for resuming sampling from existing chain.

We will update the identified issues as progress is made and/or new ones arise.

[torfjelde]

Strongly disagree on this one. I want to be able to swap samplers without needing to entirely rewrite my code. So the internal representation is a continuous trajectory, but one (mutable) field is the dt for discretization. The result needs to be iterable (in the domain space), with a show method using this for summarization.

But this is a chain-issue though, not a AbstractMCMC-interface issue?

Why would the back-end tell the front-end what to do? Seems to me the back-end should only worry about matching the interface.

I guess this is another reason for glue packages - the internal representation across back ends can be very different.

What is backend here? The sampler? And the front-end is what? The chain?

I want to use SampleChains with all of them in a uniform way, and I'd guess you want the same for MCMCChains.

I don't think we do. It's fine if there are certain types of samplers, e.g. ZigZag that just doesn't make sense with the current impl of MCMCChains. Maybe we can make changes to accomodate that, maybe it's not worth it. I don't think MCMCChains has an ambition of supporting any types of chains; AFAIK it's only meant to handle "standard" ones.

And we should really stop bringing MCMCChains up in this discussion 😅 It's only relevant IMO for the "Chains-interface"-thread above, not the rest of AbstractMCMC.

[devmotion]

No we deviate to the other thread 😛

I guess this is another reason for glue packages - the internal representation across back ends can be very different. I want to use SampleChains with all of them in a uniform way, and I'd guess you want the same for MCMCChains. But I also don't want SampleChains to have a huge number of dependencies. And the representation should not be embedded in Soss or Turing.

My suggestion would be roughly the same as written in TuringLang/DynamicPPL.jl#170 (comment): Add an interface for iterating over samples (AbstractMCMC.eachsample?) and parameters (AbstractMCMC.eachparam?) and then chain structures such as MCMCChains can just implement a very generic bundle_samples(::Type{Chains}, ....) that would be able to deal with all internal representations that implement this interface. Similar to Tables.jl, AbstractMCMC would just define these for common representations such as AbstractVector and NamedTuple.

Of course, one could still implement special version for e.g. custom samplers or without eachsample etc. if needed but this should cover many common cases, I assume.

BTW we also use Requires in Turing but considering all the problems with it I think it would be very unfortunate if this would be the recommended approach.

[cscherrer]

But this is a chain-issue though, not a AbstractMCMC-interface issue?

The suggestion I heard was that AbstractMCMC would have an AbstractChain, and that a given sampler connecting to AbstractMCMC would need to choose a concrete representation to hold those samples. This forces strong dependencies between the two.

What is backend here? The sampler? And the front-end is what? The chain?

Sorry I wasn't clear, yes this is what I meant.

I don't think MCMCChains has an ambition of supporting any types of chains; AFAIK it's only meant to handle "standard" ones.

That's fine, but it's not what I want. I'd like a uniform interface across lots of concrete representations, that's the idea with SampleChains.

And we should really stop bringing MCMCChains up in this discussion 😅 It's only relevant IMO for the "Chains-interface"-thread above, not the rest of AbstractMCMC.

I'd really like to have these untangled, but we can't pretend this isn't an issue.

Add an interface for iterating over samples (AbstractMCMC.eachsample?) and parameters (AbstractMCMC.eachparam?)

Interesting, how would eachparam work for nested namespaces, or array-valued parameters?

chain structures such as MCMCChains can just implement a very generic bundle_samples(::Type{Chains}, ....) that would be able to deal with all internal representations that implement this interface. Similar to Tables.jl, AbstractMCMC would just define these for common representations such as AbstractVector and NamedTuple.

This would be nice to have as a fallback, but it would be nice to avoid the overhead of, for example, first building a vector of NamedTuples and copying it into a TupleVector. Going directly to a TupleVector can be a lot more efficient.

BTW we also use Requires in Turing but considering all the problems with it I think it would be very unfortunate if this would be the recommended approach.

Requires has issues, but for some applications it seems to work fine. But maybe I've just been lucky so far?

[torfjelde]

The suggestion I heard was that AbstractMCMC would have an AbstractChain, and that a given sampler connecting to AbstractMCMC would need to choose a concrete representation to hold those samples. This forces strong dependencies between the two.

There's a thread further up which is questioning whether we want this or not. As specified in that thread, we don't need to have the AbstractChain in AbstractMCMC.jl at all.

That's fine, but it's not what I want. I'd like a uniform interface across lots of concrete representations, that's the idea with SampleChains.

Sure! But again, this is a chain-representation issue, not a sample-issue. Plleeeeease discuss this in the other thread above. I made these separate because all the previous discussions end up mixing issues and we get nowhere as a result. We need to keep things separate here so we can actually reason about which parts work just fine and which doesn't.

I'd really like to have these untangled, but we can't pretend this isn't an issue.

Wat. I'm saying: AbstractMCMC can live just fine without any notion of chain, so don't conflate the two. What you're talking about is an issue, but it's an issue related to the chains representation, not the sample-interface.

[cscherrer]

Gotcha. @torfjelde I appreciate your effort to organize this discussion. I'd love to completely separate discussions of chains from AbstractMCMC, but things still feel very conflated to me. But I agree it's better to try to keep these subtopics specific to each thread. Thanks :)

[phipsgabler]

Relation to AbstractPPL.jl

Because it had existed previously and is ingrained in a lot of places, AbstractPPL.jl currently depends on AbstractMCMC.jl and defines its AbstractProbabilisticProgram interface such that a probabilistic program should be an AbstractModel, and have sample working in the AbstractMCMC sense and interact sensibly with the rest of the functions, insofar as they exist.

That a PP is not necessarily sampled from via MCMC was something that has already bothered me, so an AbstractSampling "parent package" for AbstractPPL would be ideal. It has to stay lightweight enough, though, including the requirements for the AbstractModel.

[devmotion]

Seems to belong to this thread?

[yebai]

The name ‘AbstractMCMC’ was known to be inaccurate from day one. Nevertheless, we stil used the name because most people knows MCMC is associated with Bayesian computation. To that extent, it’s an acronym for being Bayesian for me. I don’t see how we can make it descriptive and accurate at the same time. Maybe we should just add a line in the README explaining its scope instead of changing the name, I think.

[cscherrer]

It's very confusing to call something MCMC if that's not a good description of its capability