Skip to content

Replace the network-wide init scheme with per-parameter initializers - #85

Merged
jessegrabowski merged 3 commits into
pymc-devs:mainfrom
jessegrabowski:per-parameter-initializers
Aug 11, 2026
Merged

Replace the network-wide init scheme with per-parameter initializers#85
jessegrabowski merged 3 commits into
pymc-devs:mainfrom
jessegrabowski:per-parameter-initializers

Conversation

@jessegrabowski

Copy link
Copy Markdown
Member

initialize_params and Model.initialize take initializers={parameter: HeNormal()}, keyed by the parameter object rather than its name, which is the only way to reach something no constructor keyword exposes — a bias inside a transformer block, or a weight in a network you loaded rather than built.

Then scheme= goes. It was the one API that acted on parameters the caller hadn't named, and it was paying for itself with everything around it: Linear.W had to withhold its own default so a scheme could reach it, which made a recorded declaration a silent library-wide breakage; a constant had to be fenced off from it; precedence had to be three-way. All of that is gone, and Linear and Embedding now declare the Xavier draw they were already using. initialize(seed=0) means redraw every parameter from its own initializer.

model.initialize(seed=0, initializers={block.ff.fc_in.b: NormalInitializer(0.0, 0.02)})

One behavior change worth a look: a parameter that declares no initializer now raises rather than being silently left alone. Only reachable by building a trainable() by hand — every layer declares one per parameter — but leaving it alone would put an all-zero weight back in a graph that trains, which is what #84 just fixed.

Also worth knowing: he_normal across a whole network is now unsayable, and permanently, since once W declares there is no set of undeclared weights left to address. Per-layer weight_initializer= is the replacement, as in torch, flax and keras.

Every parameter now declares the law it is drawn from, so `initialize` means redraw from those declarations under one seed. Linear and Embedding could not declare their own default while a scheme existed, since a declaration shadowed it -- that trap, the refusal of a constant scheme, and the three-way precedence all go with it.
@codecov-commenter

codecov-commenter commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.48%. Comparing base (5c54d4a) to head (91952ca).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
- Coverage   95.49%   95.48%   -0.01%     
==========================================
  Files          45       45              
  Lines        1819     1818       -1     
==========================================
- Hits         1737     1736       -1     
  Misses         82       82              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jessegrabowski
jessegrabowski merged commit 5e64a44 into pymc-devs:main Aug 11, 2026
7 checks passed
@jessegrabowski
jessegrabowski deleted the per-parameter-initializers branch August 11, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants