fixed mixup between conf and modelMCMCConf

[danielturek]

Fixes a mixup between the conf and the modelMCMCConf variables, which was introduced in PR #1068.

[paciorek]

How about adding a numerical test for the default MSE loss? I think the lack of that is why this slipped through.

[danielturek]

Updated the test using the dyes model (which was added in the faulty bug fix #1068) to match patched results.

Also added a test (again using the dyes model) using the default "MSE" loss function.

[danielturek]

@paciorek Second set of eyes welcome on this.

[paciorek]

It seems weird the loss for dyes would be larger (97.9) without the bug than with it (63.9). My (perhaps incomplete) understanding of the effect of the bug was that it would compute the loss sampling only the held-out data nodes and not the model parameters hence conditioning on the inits for parameters, and I would think that should lead to a higher loss than when actually doing MCMC on the parameters.

Any thoughts?

[danielturek]

@paciorek This took me a few minutes of investigation, but it does make sense. Your high-level instinct is right, that by fixing the model parameters at their initial values (the buggy behaviour), we would perhaps expect a higher CV value. However, the initial values use 1 as the precision & standard deviation values, which when fixed at this value constraints sampled values of the predictive nodes (originally data nodes, which were set to non-data via a CV fold) to a very tight region. So although that region is centered around a less-than-optimal (initial) value, it's centered there very tightly, leading to an overall lower CV value. In contrast, the correct (fixed) case has the sampled values of the predictive nodes (the CV fold nodes) centered around a better value, but it uses the posterior sampled value of the standard deviation term, thus the spread of the samples of the predictive nodes is much larger, leaving to a larger CV value. That's what leads to the larger CV value in the fixed case.

[paciorek]

Ok, thanks for investigating. I'm still surprised that the fixed values are as good as they are (perhaps they are reasonable by accident), but as long as you're satisfied then ok to proceed by me.