Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(createpackages): use jinja for mf6 module code generation #2333

Merged
merged 3 commits into from
Mar 7, 2025

Conversation

wpbonelli
Copy link
Member

@wpbonelli wpbonelli commented Oct 10, 2024

Carved out of #2317. Just move to Jinja as a first step. Type hints can come afterwards as they will take some effort to get right. The aim here is to match the old createpackages.py capabilities without relying on mfstructure.py. This is an initial step to trim mfstructure.py in favor of a leaner representation of the input spec. mfstructure.py is still used at runtime which will need to be unraveled in followup work.

Introduce a flopy.mf6.utils.codegen module with some new machinery for generating code with Jinja. This includes a bunch of filters handling quirks of the generated classes which we can aim to gradually eliminate. The templates should also get simpler over time.

The codegen module consumes TOML rather than legacy DFNs. Devtools is used at code generation time to first convert DFNs to TOML. This happens under the hood, with no change in apparent behavior to the user of the codegen utility.

Jinja2 and modflow-devtools are added to a new optional dependency group codegen. These are required to use the code generation utilities. These as well as transitive dependencies are added to environment.yml.

Miscellaneous:

@wpbonelli wpbonelli force-pushed the jinja branch 2 times, most recently from a3b7c2e to af1e7a5 Compare October 10, 2024 17:49
@wpbonelli wpbonelli added modflow 6 dependencies Pull requests that update a dependency file maintenance labels Oct 10, 2024
@wpbonelli wpbonelli force-pushed the jinja branch 4 times, most recently from d1e8a21 to a7f2644 Compare October 10, 2024 18:30
@wpbonelli wpbonelli added this to the 3.9 milestone Oct 10, 2024
Copy link
Contributor

@deltamarnix deltamarnix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked through everything, but I left you a PR on your own branch with indications on how I would probably approach this. Let's take a look at it together and see if we can get rid of the shim :)

@wpbonelli wpbonelli force-pushed the jinja branch 2 times, most recently from bab83ac to cf68432 Compare October 18, 2024 00:32
Copy link

codecov bot commented Oct 18, 2024

Codecov Report

Attention: Patch coverage is 4.91803% with 406 lines in your changes missing coverage. Please review.

Project coverage is 75.9%. Comparing base (bb9824e) to head (ad9e2e2).
Report is 67 commits behind head on develop.

Files with missing lines Patch % Lines
flopy/mf6/utils/codegen/filters.py 0.0% 308 Missing ⚠️
flopy/mf6/utils/codegen/__init__.py 0.0% 70 Missing ⚠️
flopy/mf6/utils/codegen/component.py 0.0% 20 Missing ⚠️
flopy/mf6/utils/generate_classes.py 40.0% 6 Missing ⚠️
flopy/mf6/utils/createpackages.py 50.0% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##           develop   #2333     +/-   ##
=========================================
+ Coverage     68.4%   75.9%   +7.5%     
=========================================
  Files          294     296      +2     
  Lines        59390   63918   +4528     
=========================================
+ Hits         40652   48577   +7925     
+ Misses       18738   15341   -3397     
Files with missing lines Coverage Δ
flopy/mf6/data/mfdatastorage.py 75.4% <ø> (+6.5%) ⬆️
flopy/mf6/data/mfdatautil.py 77.9% <ø> (+23.3%) ⬆️
flopy/mf6/data/mfstructure.py 73.6% <100.0%> (+9.5%) ⬆️
flopy/mf6/mfbase.py 87.6% <100.0%> (+12.1%) ⬆️
flopy/mf6/mfpackage.py 81.9% <ø> (+17.3%) ⬆️
flopy/mf6/mfsimbase.py 75.6% <100.0%> (+13.4%) ⬆️
flopy/mf6/utils/createpackages.py 60.0% <50.0%> (+53.5%) ⬆️
flopy/mf6/utils/generate_classes.py 18.6% <40.0%> (+0.8%) ⬆️
flopy/mf6/utils/codegen/component.py 0.0% <0.0%> (ø)
flopy/mf6/utils/codegen/__init__.py 0.0% <0.0%> (ø)
... and 1 more

... and 249 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wpbonelli wpbonelli marked this pull request as ready for review February 4, 2025 13:01
@wpbonelli
Copy link
Member Author

IMO this should wait til this week's release goes out, then it can get some exercise on the develop branch for a while

Copy link

@christianlangevin christianlangevin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice big step forward, @wpbonelli. I don't have anything to add other than I think this is the right direction toward a leaner and more sustainable implementation.

Copy link
Contributor

@deltamarnix deltamarnix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have suggested some improvements that I think could make the code a bit cleaner. The general concept is great!

Copy link
Contributor

@deltamarnix deltamarnix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only looked through some of the output files, but I have found some remarkable differences that we should check.

@wpbonelli
Copy link
Member Author

Thanks. Yeah, this is why I want to wait to merge til after release. I think it'll take some time to ferret out all the minute differences and make sure no behavior has changed, even after merge.

@wpbonelli
Copy link
Member Author

here goes nothing..

@wpbonelli wpbonelli merged commit c5b5a41 into modflowpy:develop Mar 7, 2025
20 checks passed
@wpbonelli wpbonelli deleted the jinja branch March 7, 2025 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file maintenance modflow 6
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants