CFA and SEM with PyMC by NathanielF · Pull Request #698 · pymc-devs/pymc-examples

NathanielF · 2024-09-01T22:18:08Z

CFA and SEM with PyMC

Related to this issue here I'm adding a PR to demonstrate the functionality of CFA and SEM models using PyMC
#695

Notebook follows style guide https://docs.pymc.io/en/latest/contributing/jupyter_style.html
PR description contains a link to the relevant issue:
- a tracker one for existing notebooks (tracker issues have the "tracker id" label)
- or a proposal one for new notebooks
Check the notebook is not excluded from any pre-commit check: https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml

Helpful links

https://github.com/pymc-devs/pymc-examples/blob/main/CONTRIBUTING.md

📚 Documentation preview 📚: https://pymc-examples--698.org.readthedocs.build/en/698/

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

review-notebook-app · 2024-09-01T22:18:12Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

NathanielF · 2024-09-08T16:22:08Z

Think this is ready for review @drbenvincent , @cluhmann , @AlexAndorra . Any feedback/pushback welcome.

NathanielF · 2024-09-16T09:32:06Z

Giving this another nudge!

Thanks!

drbenvincent · 2024-09-17T17:34:19Z

FYI: I'm totally overloaded at the moment, so going to struggle with a review. Will see if things ease up a bit next week.

NathanielF · 2024-09-17T20:14:34Z

No worries.

review-notebook-app · 2024-09-25T03:08:46Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:46Z
----------------------------------------------------------------

Perhaps add reference to psychometrics in the title to make it more specific, and aid in discoverability.

NathanielF commented on 2024-09-25T11:38:08Z
----------------------------------------------------------------

Good idea.

review-notebook-app · 2024-09-25T03:08:47Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:47Z
----------------------------------------------------------------

nitpick: data set should not have a hyphen; data set or dataset, no?

NathanielF commented on 2024-09-25T11:38:41Z
----------------------------------------------------------------

changed.

review-notebook-app · 2024-09-25T03:08:48Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:47Z
----------------------------------------------------------------

Maybe hyperlink SEM and CFA to some introductory materials on the topics (even a Wikipedia link would be good)

NathanielF commented on 2024-09-25T11:38:54Z
----------------------------------------------------------------

Added links to wikipedia.

review-notebook-app · 2024-09-25T03:08:49Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:48Z
----------------------------------------------------------------

There are lots of UserWarnings in the notebook. Consider filtering them out.

review-notebook-app · 2024-09-25T03:08:49Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:49Z
----------------------------------------------------------------

The last line of the equation set contains a typo -- should be \psi_n rather than \psi_3

NathanielF commented on 2024-09-25T11:39:29Z
----------------------------------------------------------------

Fixed. Thanks, good catch!

review-notebook-app · 2024-09-25T03:08:50Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:50Z
----------------------------------------------------------------

Line #29.        chol, _, _ = pm.LKJCholeskyCov("chol_cov", n=2, eta=2, sd_dist=sd_dist, compute_corr=True)

Why do you set compute_corr=True and discard the correlations?

NathanielF commented on 2024-09-25T11:40:37Z
----------------------------------------------------------------

It's not used here in this model, but useful to have computed for analysing the correlations between the constructs post model fit. I demonstrate this below for the full measurement model....

review-notebook-app · 2024-09-25T03:08:51Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:50Z
----------------------------------------------------------------

Cool plots!

NathanielF commented on 2024-09-25T11:45:43Z
----------------------------------------------------------------

Thanks!

review-notebook-app · 2024-09-25T03:08:52Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:51Z
----------------------------------------------------------------

Line #81.            draws=10000,

Why so many draws? Bad autocorrelation?

NathanielF commented on 2024-09-25T11:42:40Z
----------------------------------------------------------------

Yes, it seemed to help generally boost the ESS. The model samples pretty fast so it was easy enough to just boost the samples.

review-notebook-app · 2024-09-25T03:08:53Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:52Z
----------------------------------------------------------------

typo: reflected

NathanielF commented on 2024-09-25T11:42:50Z
----------------------------------------------------------------

Fixed.

review-notebook-app · 2024-09-25T03:08:53Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:53Z
----------------------------------------------------------------

typo in first sentence: individual

NathanielF commented on 2024-09-25T11:42:59Z
----------------------------------------------------------------

Fixed.

review-notebook-app · 2024-09-25T03:08:54Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:53Z
----------------------------------------------------------------

Typo: each

NathanielF commented on 2024-09-25T11:43:11Z
----------------------------------------------------------------

Fixed

review-notebook-app · 2024-09-25T03:08:55Z

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:54Z
----------------------------------------------------------------

What about measures of relative fit among the models? What do the LOO/WAIC values look like, and are they helpful in selecting the appropriate level of complexity?

NathanielF commented on 2024-09-25T11:45:26Z
----------------------------------------------------------------

Added the global fit comparison for the SEM and full measurement models. The LOO metrics are quite close, but the full measurement model "wins" on this score. I added a note to say we need to consider the value of the trade-off between expressive power of the SEMs and the questions we can answer in the context of simple global fit comparisons.

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

NathanielF · 2024-09-25T11:38:09Z

Good idea.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:38:43Z

changed.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:38:55Z

Added links to wikipedia.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:39:30Z

Fixed. Thanks, good catch!

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:40:38Z

It's not used here in this model, but useful to have computed for analysing the correlations between the constructs post model fit. I demonstrate this below for the full measurement model....

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:42:41Z

Yes, it seemed to help generally boost the ESS. The model samples pretty fast so it was easy enough to just boost the samples.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:42:51Z

Fixed.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:43:01Z

Fixed.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:43:12Z

Fixed

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:45:27Z

Added the global fit comparison for the SEM and full measurement models. The LOO metrics are quite close, but the full measurement model "wins" on this score. I added a note to say we need to consider the value of the trade-off between expressive power of the SEMs and the questions we can answer in the context of simple global fit comparisons.

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:45:44Z

Thanks!

View entire conversation on ReviewNB

NathanielF · 2024-09-25T11:48:31Z

Thanks @fonnesbeck I think i've addressed most of the points you raised. But let me know if you think I should add anything else.

Note I also re-parameterised the tau terms in the final SEM model which seemed to fix the divergences i was getting.

fonnesbeck

LGTM!

NathanielF · 2024-09-25T14:26:42Z

Sweet!

NathanielF · 2024-09-25T17:27:26Z

Think you'll have to merge that for me @fonnesbeck, i don't have that power.

NathanielF · 2024-09-25T17:43:04Z

Woo! Thanks @fonnesbeck

* First commit CFA and SEM Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * test pre-commit hooks Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * updated for pre-commit Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * updated git index Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * update with full run Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * add some text write up and a pair plot Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * update with full run and text Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * Run with explained variance calculations Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * Add sem structure section Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * full run with lavaan syntax noted Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * update with draft conclusion Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * tigher writing Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * tidying Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * emphasis on exchangeability Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * update plot sizes Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * add pearl reference Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> * updating with Chris's comments and better parameterisation Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com> --------- Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

An external review raised two blockers; both reproduce. First, `approx.hist` is not the full-data negative ELBO the notebook said it was. PyMC normalizes twice: MinibatchRV scales the observed logp up by N/b, then OPVI divides the objective by the same constant because `scale_cost_to_minibatch` defaults on. What is recorded is the batch's own log-likelihood sum plus (b/N) KL, which scales linearly with b — measured directly by alternating 1000- and 400-row blocks and reading the ratio of mean recorded loss: 0.4126 against the 0.400 the batch sizes predict. Multiplying by N/b restores the full-data scale (0.9998 against a full-data reference over 1200 steps). The notebook now derives this, plots and analyses `hist * N/b`, and reports 188 nats of stage-2 reduction rather than the 0.60 that the batch scale produced. Second, the shard geometry made blocks ragged: 288 of 1024 rows plus eight of 636, which put a deterministic sawtooth into the loss and left `len(loader)` matching neither the block count nor the yields. Ten shards of 30,000 rows with row groups equal to the batch size give exactly 300 equal blocks, `len(loader) == 300`, and a loss series whose only structure is optimization and noise. The per-step standardized contrast now has unit spread (0.99, from 1.07) and the epoch-alignment result survives on the clean series: the spread at epoch-aligned horizons is more than twenty times smaller than at 1.5 epochs. Third, provenance. The stored outputs were produced in an environment whose pymc-extras metadata predated the merge and whose PyMC violated the merged package's own floor. Everything here was re-executed against pymc-extras at 8db1880 (the pymc-devs#698 merge commit), PyMC 6.2.0, PyTensor 3.2.4, and the install note now gives readers that exact command. Also corrected: the CUSUM warning overstated its case (the recursion sits at zero while improvement exceeds the allowance; the real failure is that it cannot tell convergence from improvement too slow to resolve); the fixed replay is described as cyclic finite-sum optimization rather than the unbiased N/B identity; the hurdle alone does not prevent sufficient- statistic collapse; the nu floor is a statement about support, not prior belief (P(nu<=2) is 4e-6); the row-scale generator arrays are released before fitting and the in-memory preprocessing is labelled as such; the shrinkage claim in the model section is withdrawn; and the predictive check is described as one batch, one step ahead. Cold runs in the pinned environment: 11.0 / 10.0 / 9.6 s, peak RSS 619 MB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

First commit CFA and SEM

ba79db8

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

NathanielF added 15 commits September 1, 2024 23:30

test pre-commit hooks

194a6f5

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

updated for pre-commit

8083c9c

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

updated git index

5cb1c42

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

update with full run

6f6d880

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

add some text write up and a pair plot

e9a5beb

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

update with full run and text

d15639a

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

Run with explained variance calculations

47da13d

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

Add sem structure section

62d788b

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

full run with lavaan syntax noted

64799e9

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

update with draft conclusion

a69f76d

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

tigher writing

aeda1be

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

tidying

198c814

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

emphasis on exchangeability

8ec7863

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

update plot sizes

eb4ed98

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

add pearl reference

2247d19

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

NathanielF marked this pull request as ready for review September 8, 2024 16:21

NathanielF requested review from AlexAndorra, cluhmann and drbenvincent September 8, 2024 16:22

updating with Chris's comments and better parameterisation

a0f13b1

Signed-off-by: Nathaniel <NathanielF@users.noreply.github.com>

fonnesbeck self-requested a review September 25, 2024 13:40

fonnesbeck approved these changes Sep 25, 2024

View reviewed changes

fonnesbeck merged commit ba7a936 into pymc-devs:main Sep 25, 2024

YichengYang-Ethan mentioned this pull request Aug 17, 2026

Out-of-core minibatch ADVI on a financial tick stream #892

Open

Uh oh!

Conversation

NathanielF commented Sep 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CFA and SEM with PyMC

Helpful links

Uh oh!

review-notebook-app Bot commented Sep 1, 2024

Uh oh!

NathanielF commented Sep 8, 2024

Uh oh!

NathanielF commented Sep 16, 2024

Uh oh!

drbenvincent commented Sep 17, 2024

Uh oh!

NathanielF commented Sep 17, 2024

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

NathanielF commented Sep 25, 2024

Uh oh!

fonnesbeck left a comment

Choose a reason for hiding this comment

Uh oh!

NathanielF commented Sep 1, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading

review-notebook-app Bot commented Sep 25, 2024 •

edited

Loading