Add Probabilistic Matrix Factorization model with analysis on Jester data in IPython NB.

[macks22]

I've implemented the Probabilistic Matrix Factorization (PMF) model in pymc3 and validated it on a subset of the Jester dataset. The notebook I've included in this pull request is essentially a polished log of an analysis done using PMF for user rating prediction on unseen jokes in the Jester dataset. According to @twiecki there are currently few multivariate examples, so this seems like a useful contribution (ref).

I've also included the data I used, and a backup of the MAP estimate calculated in the model. The NB contains a function for reloading this, which facilitates quick reproducibility. The analysis includes both intuitive and formal explanations of the model. Also presented is a new traceplot implementation for high-dimensional data that may be useful to incorporate into the codebase. Results are compared against 3 baselines. They can also be compared with related studies in the literature.

If there is interest from the pymc3 developers, I'd be interested in knowing how I might be able to get this model added to the examples in the docs.

Cheers!

[twiecki]

Probabilistic Matrix Factorization (PMF) [3] is a probabilistic to the collaborative Is there a word missing?

[twiecki]

Minor: for the N of a normal distrubiton, you can use \mathcal{N}.

[twiecki]

Just read through the whole thing and all I can is wow, this really exceeded my expectations. Clear and understandable writing that really provides a nice intuition about the problem and solution. The model comparison also really highlights the benefits of PMF.

I do wonder though why MCMC sampling produced worse results in this case but we probably shouldn't get hung up on that. Is the mean of the posterior close to the MAP?

[macks22]

I added the mathcal font for the normal distribution notation and fixed that error in the PMF intro.

I also edited the post to make it clearer that degradation in performance is actually expected from the sampler using this model. The authors of PMF never actually approximate the posterior. They simply compute the MAP using gradient descent. With the fixed precision parameters, sampling doesn't have the freedom to improve upon the MAP estimate. While I haven't seen it done before, it makes sense that MCMC sampling for this model would lead to overfitting.

Some of the 90:10 train:test splits actually produced small decreases in RMSE (< 0.1), but it's not really worth the sampling time to get that measly and unreliable gain. I wish I had some better intuition for this, but I don't fully understand when or why the sampler would be able to produce gains. It may be that sparser datasets than this would exhibit different characteristics. Perhaps I'll experiment later with a smaller training set and larger test set.

[twiecki]

Well it's more about MAP vs Expectation and especially with higher dimensions Expectations should be preferred. Anyway, this is awesome and I'm going to merge this as well as add it to the online docs. Thank you, this is a great contribution.

[twiecki]

Here we go: http://pymc-devs.github.io/pymc3/pmf-pymc/

Btw. feel free to add your name to the top of that file. You might also want to consider starting a blog :).

[twiecki]

Just saw you already have one :).

[macks22]

Awesome! Thanks Thomas. I'm glad this ended up being useful. The pymc project is a great contribution to the Python community, and I plan on using it often in the coming years.

Regarding the docs formatting: It seems a few of the equations are not rendered properly. In particular, see the "Probabilistic Matrix Factorization" section, and near the end of the "Evaluation" section. Any idea why this might be happening? They are rendered fine when I view the NB locally.

Also, can you explain what you mean about MAP vs Expectation in a little more detail? Or perhaps point me towards a resource or two on the subject? A clearer picture of why the MCMC results increase error would make for a much more interesting conclusion to the analysis. Thanks!

[twiecki]

The point is made here: https://www.youtube.com/watch?v=pHsuIaPbNbY

[twiecki]

\begin{equation} E = \frac{1}{2} \sum_{i=1}^N \sum_{j=1}^M I_{ij} (R_{ij} - U_i V_j^T)^2 + \frac{\lambda_U}{2} \sum_{i=1}^N \|U\|{Fro}^2 + \frac{\lambda_V}{2} \sum^M \|V\|_{Fro}^2, \end{equation}

I think this has to be done with $math$ instead.

[macks22]

Thanks for the link. That video was very enlightening. I've never been able to visualize the posterior as a shell being explored like that. It makes a lot of sense. The point about MAP vs. Expectation also makes sense. I updated the NB to remove my (probably quite wrong) statements about why MAP should do better. In reality, the sampler actually does improve the results, but only those on the training set. I believe the overfitting is facilitated by setting the precision parameters using point estimates from the training data.

I also fixed the equation rendering. When I use the $math$ syntax, mkdocs renders them fine. I updated the headers to produce a better ToC as well.

What is the easiest way to update the NB?

Also, I noticed the absence of the BEST.ipynb and hierarchical.ipynb NBs in the examples directory is undocumented in howto_docs.md. This might be useful for those wanting to build the docs locally as I was doing. Without these files present, nothing can be built using the current version of mkdocs.

[twiecki]

Thanks for pointing that out. Added with 6bf01be.

As to updating the NB, PR is probably easiest.

[macks22]

Submitted. #698

[twiecki]

Merged, compiled and updated. Thanks!

[twiecki]

maybe we should think of adding the Frobenius norm to pymc3. Perhaps as a flag to traceplot?

[macks22]

I'm sure that would come in handy. Another feature that might be useful is traceplots of perplexity.

Something else that occurred to me: it would be nice to have a simple, efficient way of generating samples from the posterior for the observed variable. Everything you need to generate the sample is contained in the model definition for the observed variable and the sampled values from the trace. All you really need to specify is optional burn-in size.

[twiecki]

Agreed on perplexity.

Also agreed on generating PPC samples. stan also does this and it's quite nice. Requires #442 though.

[macks22]

I see. What's left to do with #442?

Also, one last thought on traceplots: it might be interesting to plot traces of several samples drawn randomly from all sampling dimensions. So for matrices and arrays, just take a random sampling of the entries and make a traceplot of those. This would give a better picture of the sign fluctuation as opposed to the simple picture of magnitude provided by the Frobenius norms. The output could be one plot per random variable, with traces for each of the randomly drawn entries as separate lines.

[twiecki]

The main thing is figure out the pickling problem.

I like that idea of randomly choosing a subset.