Skip to content

Draw every parameter at construction, and refuse a constant scheme - #84

Merged
jessegrabowski merged 2 commits into
pymc-devs:mainfrom
jessegrabowski:construction-time-draws
Aug 11, 2026
Merged

Draw every parameter at construction, and refuse a constant scheme#84
jessegrabowski merged 2 commits into
pymc-devs:mainfrom
jessegrabowski:construction-time-draws

Conversation

@jessegrabowski

Copy link
Copy Markdown
Member

A network built and never .initialize()d couldn't learn. Every weight matrix was created as zeros, so nothing propagated below the last layer and only the output bias moved — the loss still crept down, which reads as slow convergence rather than a bug. Parameters are now drawn from their own initializer when the layer builds them, which incidentally makes Linear("fc", 4, 8, bias_initializer=NormalInitializer()) give you that bias instead of silently keeping zeros until someone calls initialize().

Second commit closes the other route to the same dead network: initialize("zeros") now raises. A constant is the right initializer for one bias or norm scale and never for every weight matrix in a network, so it stays available as a declaration on the parameter that wants it.

Construction draws from fresh entropy, so initialize(seed=...) is still the only thing needed for reproducibility and still overwrites everything.

@codecov-commenter

codecov-commenter commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.49%. Comparing base (620fbd1) to head (6748496).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #84      +/-   ##
==========================================
+ Coverage   95.47%   95.49%   +0.01%     
==========================================
  Files          45       45              
  Lines        1811     1819       +8     
==========================================
+ Hits         1729     1737       +8     
  Misses         82       82              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jessegrabowski
jessegrabowski merged commit 5c54d4a into pymc-devs:main Aug 11, 2026
7 checks passed
@jessegrabowski
jessegrabowski deleted the construction-time-draws branch August 11, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants