Positive Elbo (evidence lower bound)

[organic-chemistry]

I would like to use the elbo (evidence lower bound) to perform model selection using variationnal inference.
However I started to notice that sometimes I get positive values for the elbo.
To my knowledge it is a lower bound of the log of a probability, so it should not be higher than 0.

using Turing
using AdvancedVI
@model function estimate_mu(data)
    n = length(data)
    mu ~ Normal(0,1)
    for i = 1:n
        data[i] ~ Normal(mu,0.1)
    end
end

data=[0,0,0,0]
model = estimate_mu(data)

advi = ADVI(20, 1000)
q = vi(model, advi);
elbo = AdvancedVI.elbo(advi, q, model, 1000)  # gives 2.5

Is it a mistake from my part and it can be higer than 0 or is there an implementation problem?
Best

[torfjelde]

log of a probability, so it should not be higher than 0.

This is sensible when talking about discrete distributions, but not when we're working with continuous distributions. Simple example:

julia> using Distributions

julia> pdf(Normal(0, 1e-3), 0)
398.9422804014327

julia> logpdf(Normal(0, 1e-3), 0)
5.988816745777464

And given that you have data which is just a bunch of zeros, this doesn't seem unreasonable:)

[organic-chemistry]

I get your point.
However I thought that the Elbo was defined as the opposite of the Kulback liebler divergence between the variational posterior and the joint probability of the model. Which mean that it should be negative.

[torfjelde]

It's not quite the "opposite" of the KL. It's true that maximizing the ELBO minimizes the KL, yes, but it's not a bound on the KL.

The following is a copy-paste from https://en.wikipedia.org/wiki/Evidence_lower_bound

Here you see that the ELBO is equal to the "evidence" (which can take any real value) - KL (which is strictly non-negative), and so the ELBO can indeed be positive.

The reason why maximizing the ELBO minimizes the KL is because the "evidence" is constant wrt. parameters, and so does not affect our optimization wrt. parameters.

[organic-chemistry]

If I look a bit lower on wikipedia it says:

which to me is -KL(q(z|x),p(x,z)) and again which should be negative

[torfjelde]

$p(x,z)$ is not a density of $z$, it's a density of both $z$ and $x$; that is, $\int p(x, z) dz = p(x)$, and so that's not a KL.

[organic-chemistry]

Oh, I see. Thanks a lot for taking the time to clarify this for me.
Have a good day!

[torfjelde]

Glad I could help!:)