2025 Software Engineering meeting notes

March 31, 2025

Machine status and notices 🖥️

CISL: Tue 1 April
NERSC: Wed 16 April

Upcoming meetings and events 🗓️

FATES modeling meeting: TBD
- Probably make this office hours

PR board status update

Integrated

restart fixes for two-stream (#1321), FATES two-stream restart fixes (ctsm#2949)

Testing

Johnson & Berry electron transpot (#1350): Alistair to review. Greg walked him through the process last week.
Greg to use his discretion on how to integrate the upcoming parameter file changes

Reviews

Prescribed Fire (#1360)
- Ryan: is there another way that we could capture these prescribed parameters?
  - Charlie: we could put it on the landuse file or make a specific prescribed fire file
    - Consensus is that this might be a good idea as prescribed fire is site/region specific
    - Greg: this would require new/updated infrastucture. What does Xiulin's timeline look like; can she wait?
- Marcos: combine upper/lower along a fire dimension
- Ryan: Worthwhile to invite Xiulin speak on this? Consensus is yes. Charlie will reach out to her and Lara.

New

Nocomp seed localization (#1371)
- The localization number can be integer for now, but in future for landuse specific nocomp, it should a real number.

Latest issues and discussions of note

NEON-FATES-NIWO failing run (ctsm#3038)
- Seeing negative coszen value, which is causing crash. Possibly valid coszen due to sun being below zenith if site is on ridge?
- Likely due to recent two-stream fix with ctsm5.3.034.
FATES errsol balance error with cam7 (ctsm#3043)
- Some part of the model thinks that there is no reflected radiation (fsr = 0). Likely due to recent two-stream fix with ctsm5.3.034.
potential logging bug (#1367)
change use_fates_potentialveg (#1318)

Calibration progress update

Rosie: Working on hand tuning.
Adrianna: currently working on other deliverables not related to global calibration

Walk on topics

Adrianna: slamax, slatop being the same ok?
- Marcos: this means there will be no plasticity.
  - Charlie: Typically for grasses/shrubs
Rosie: SLA unit conversion sheet and paper results comparison
- Marcos: qualitatively the numbers look correct relative to his recent TRY analysis
  - Be mindful of where in the canopy the measurement was made, which can skew results.
- Jennifer: recommending making a synthesis table of all the SLA/LMA results that group has

March 24, 2025

Machine status and notices 🖥️

CISL: Tue 1 April
NERSC: Wed 26 March

Upcoming meetings and events 🗓️

FATES modeling meeting: TBD
- Qing Zhu moved to later date
- April 10 Eastern regional DVGM overlap.
- Greg to cancel this week.

PR board status update

Integrated

GetNutrientTargetCNP update (#1348)

Testing

restart fixes for two-stream (#1321): Ryan running final tests
- FATES two-stream restart fixes (ctsm#2949): associated ctsm pull request
- Running into testing issues on izumi. We troubleshot this during standup today. Issue appeared to be license server needing a restart.
- Ryan: clean up misleading text in future PR. Make comment
- Charlie: make two-stream the default?
  - Ryan: Conduct a performance test. Greg made an issue #136?
NCL max (#1315): smoke tests run, but results need review

Reviews

Johnson & Berry electron transpot (#1350): Greg to reach out to Alistair about reviewing
Prescribed Fire (#1360): Xiulin has responded to Sam's review notes. Comments need resolving.
Leaf phenology parameter simplification (#1355): Greg has approved. To be coordinated with other parameter file pull requests.
Default logging parameters (#1359): reviewer?
Ryan to double check the age-class comments. DONE. Good to go.

New

update LUH2 tests (ctsm#3022): simple update to add output to LUH2 tests
vegetation bin indexing (#1364): discussion with visual aids next week wrt trimming porttion

Latest issues and discussions of note

Error calculating carbon flux due to fire mortality (#1365): Charlie to review and respond
Reconcile parameter and scales driving leaf phenology and mortality (#1363)
- TL;DR: Difference in timescales for hydro mort and enviro mort could result in deciduous pfts dying in conditions that they should be protected against
  - From an ideal, simplified case deciduous should be nearly perfectly sheilded from hydraulic failure. Not so for evergreens that can "take more risk."
- Ryan: would it be hard to kill deciduous plants?
- Rosie: Hydraulic disassociation from the soil?
- Marcos: calibration results for drought deciduous pfts?
  - Rosie seeing that they are dying, but not sure about the mortality type
- Charlie:
- Marcos: change deciduosness and see how this fairs
- Aside: Marcos also seeing a lot of termination mortality. Likely unrelated.

Calibration progress update

deferred due to time

FATES specific surface dataset discussion

deferred until next week (missing to many folks)

March 17, 2025

Machine status and notices 🖥️

CISL: none planned
NERSC: 26 March, perlmutter

Upcoming meetings and events 🗓️

FATES modeling meeting: Qing Zhu, Modeling fire risks and impacts at climate scale
- Nothing planned April 10
- Marcos: will reach out to author: https://www.science.org/doi/10.1126/science.adl5414
- Ryan: Buckley?
  - Rosie: waiting until Danica is back in May/June

PR board status update

New

Crown fire Scottt & Reinhardt
- Adrianna: Xiulin took old PR and bring it up to date with main
  - Recommending closing the two older PRs: closed with comment.
Prescribed Fire (#1360): Sam R and Adrianna are reviewing.
Competitive Exclusion to -1 (#1358): Rosie to review and sign off.
Default logging parameters (#1359): Main update: Change default logging direct fraction to be one to make sure secondary forest generated is more distinct from primary.
Spitfire refactor on mortality (#1357): This should come in after prescribed fire #1360.

Next

GetNutrienTargetCNP for Deciduous (#1348): Ryan testing now.

Updated

Johnson & Berry electron transpot (#1350): Greg to reach out to Alistair about reviewing
Seed bank (#1338): Jessie reviewed. Looks ok. Needs deconflict. Ryan to help.
NCL max (#1315): Length of smoke test? Greg and Ryan to discuss.
harvesting unoccupied canopy area bugfix (#1339): Reviewed, ready for testing.

In-Progress

dynamic L2FR (#1353): Needs elm-specific calls reworked: Greg self-assigned

Latest issues and discussions of note

unify complexity mode namelist option (#1361)
- Ryan: Is prescribe physiology exclusive to reduced complexity modes?
  - Jessie: theoretically no, they could be used together

Calibration progress update

Rosie: no new update since Thursday modeling meeting updates. Kicking off new tests. Working on dry deposition as well.
Adrianna: getting ready to kick off another "one-at-a-time" calibration run using new fnps parameter
Jessie: Kasey (ORNL) different allometries for pfts depending on regional (i.e. continent)
- Implementation idea: bounds on where seeds for specific pfts can not recruit. Potential ForestGEO collab.
- Adrianna: this seems like it would benefit GPP calibration as well (seeing region balance shift of GPP for tropical evergreen trees, Asian vs Amazon)
- Rosie: related paper: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2024EF005225

Walk-on topics

Rosie: fates-specific surface dataset (to facilitate the above region specific pft discussion)
- Charlie: move LAI data out of surface dataset and let fates use directly (similar to land use data)

March 10, 2025

Machine status and notices 🖥️

CISL: none planned
NERSC: 26 March, perlmutter

Upcoming meetings and events 🗓️

FATES modeling meeting: 13 March, project report back

PR board status update

Integrated

leaf biophysics refactor (#1262), spitfire fire behavior refactor (#1326), fix FATES branch runs (CTSM#2955)

Updated

ncl max increase (#1315): run long term test status
luh reorder update (E3SM#7032): tested, waiting integration
E3SM API37.1 (E3SM#7027): need issue #1354 resolved

New

Johnson & Berry electron transpot (#1350)
dynamic L2FR (#1353): Jessie as reviewer?
- Needs elm-specific calls reworked

Latest issues and discussions of note

Calibration progress update

XML parameter file discussion

Pros:
- CIME makes use of python XML base functionality
- FOX as existing HLM fortran
- fortran code on fates-side that reads in parameters is complicated (since we avoid HLM IO), would simplify any necessary maintenance
- allows us to expand the features that we could control in the parameter file (e.g. metadata) Cons:
- netcdf provides dimensionality fail-safes that we currently "get for free"
  - we'd have to take on this functionality ourselves
- We'd need to pull FoX into FATES to make available to all HLMs
Questions
- parameter ensembles?
  - xarray allows for modification of netcdf emsembles easily
- Would the developer's experience in add parameters be simpler (i.e. RegisterParam)
ED2 (via Mike Dietz) implemented XML fortran

Walk-on topic

March 3, 2025

Wins (1 min)
Machine status and meeting notices (5 min)
Upcoming Meetings (5 min)
PR board status (10 min)
Latest issues (5 min)
Report in on LMWG and E3SM all hands
Discussion on tracking progress in calibrations and benchmarking
Calibration update (10 min)

Wins and Whimsy 🎉

API38 integrated and tagged (still waiting on e3sm side)

Machine status and notices 🖥️

CISL: none planned
NERSC: 26 March, perlmutter

Upcoming meetings and events 🗓️

FATES modeling meeting: 13 March, project report back

PR board status update

Integrated

parameter migration to namelist (CTSM#2904), two-stream patch area conservation (#1310), johnson & berry namelist and parameter file update (#1344)

Updates

leaf biophysics refactor (#1262): Needs deconflict and testing. Next tag.

New

harvesting unoccupied canopy area bugfix (#1339): Greg to review
GetNutrientTargetCNP update (#1348)

Latest issues and discussions of note

E3SM DEBUG floating invalid (e3sm#7049): Non-FATES problem, holding up fates integration and debug usage, fix in progress
Add electron transport formulation (#1337)
Cold-deciduous climate enveloping (#1345)

E3SM and LMWG report back

E3SM: focusing on V3
- Utilizing E3SM Diags, the model diagnostics package more, in order to report performance progress of FATES. Can ask for tutorial from Jill or Xylar.
  - Includes benchmarking as well as diagnostics (includes ILAMB)
  - Related: NorESM looking at utilizing cupid.
- Ryan: GPU push
  - E3SM is not concerned with FATES converting to C++. There currently exists multiple ways to integrate existing fortran code with C+
LMWG
- FATES crop model work seen as necessary for full CLM adoption
  - Concerns about fates CPU performance cost

Calibration progress update

Ryan: centralized work tracking and data sharing repository?
- Initially attempted to centralize, but diverged due to different focus.
Rosie: no-comp GPP with Medlyn in amazon issue discussion
- Marcos: driest parts of the amazon seems to diverge the most

February 24, 2025

Wins and Whimsy 🎉

Machine status and notices 🖥️

CISL: none planned
NERSC: 19 March, perlmutter

Upcoming meetings and events 🗓️

FATES modeling meeting: 13 March, project report back
- Greg to send out reply-all to previous notice about this weeks usual meeting being canceled
- Marcos: suggested to move Steven's presentation to April 24 to encourage attendance (avoid scheduleing conflicts with Eastern DVGM meeting)
- Greg to send out notice for API 38 update. Bring this up during

PR board status update

Integrated

10 pull requests merged in the last two weeks
- Parameter file migration (#1300): integrated and associated escomp#ctsm2904 to be integrated soon

Update

luh reorder update (E3SM#7032): under test, should be b4b
- perlmutter python create_test
API37.1 update (E3SM#7027): ready for testing
parameter switch migration (E3SM#6918): ready for testing (post e3sm#7027)
leaf biophysics refactor (#1262): Greg reviewed. To be tested.

New

Seedbank by pft output (#1338)A
- Jessie to take a look into this. Recommending that we wait until after #1262.
Harvesting bugfix (#1339)

Latest issues and discussions of note

file write error (#1333)
- Jessie: Possibly climate forcing data issue?
ncvarsort.py not preserving order (#1335)
Add electron transport formulation (#1337)

Calibration progress update

Walk-on topic

February 17, 2025

Canceled due to Federal Holiday

February 10, 2025

Wins and Whimsy 🎉

Adrianna has been given expanded fates repo priviledges
New github issue templates

Machine status and notices 🖥️

CISL: 11-12 February, derecho
NERSC: 19 February

Upcoming meetings and events 🗓️

FATES modeling meeting: 13 February, TBD
- Consensus: cancel, Greg to send out notice
- Rosie has reached out to Tom Buckley

PR board status update

Integrated

Meier2022 fix (CTSM#2934), AllVars update (CTSM#2936), use_fates_luh order (CTSM#2898)

Update

cohort insertion and sort refactor (#1317): pulled out non-b4b part and created #1327
ncl max increase (#1315): Ryan to run long term test status
leaf biophysics refactor (#1262): Greg reviewed. To be tested.
nocomp patch cleanup (#1226): Greg reviewed. So minor cleanup TBD. To be tested.

New

sp mode data driver (CTSM#2952): fixes ESCOMP/ctsm#2945 (htop issue)

Latest issues and discussions of note

tree damage module test update (#2959)
- Related: Adrianna to reach out to Jessie to ask about the associated/deprecated damage site-level variables

Multi-layer canopy planning

Ryan: stakeholders?
- Gordon
- Gautam
- Adrianna and Danica: grant funded work, connect Sam L's ctsm branch to FATES. Working on site-level data collection for validation.
  - Timeline: through to 2028
  - Includes NCAR NEON stuff as well
goal: FATES changes to be model agnostic
action: Ryan and Adrianna to meet to discuss

Calibration progress update

Adrianna: once SP bug fixed (via escomp/ctsm#2952), will rerun calibration
- three paper ideas based on this work
Rosie: new post-doc to help with FATES calibration
- Adrianna to share calibration script location

Walk-on topic

Rosie: Matvey thinks that Leung dust scheme (new scheme) might have been built on top of Meier2022
- Adrianna: Zender is the old dust scheme
  - We should check in with Erik about Leung

February 03, 2025

Wins and Whimsy 🎉

Land use fixes integrated

Machine status and notices 🖥️

UCAR HPC: izumi experiencing issues since Friday
CISL: 11-12 February, derecho
NERSC: 19 February

Upcoming meetings and events 🗓️

FATES modeling meeting: 13 February, TBD
- Rosie to reach out to Tom Buckley
- Ryan: project coordination meeting update?
  - Jennifer suggested doing this later as a report back from the CESM-LMWG and E3SM all hands
- Charlie will be out next week

PR board status update

Integrated

land use fixes (#1273)

Update

Meier2022 fix (CTSM#2934): Waiting on izumi to come back up to check test results, to be tagged ctsm5.3.022
Grazing (#1140): Fixes restested and passing now. Still needs baseline comparison. Jennifer reviewed and Charlie addressed comments.
parameter migration to namelist (CTSM#2904): waiting on ctsm5.3.022
- Includes parameter file updates PRs
Refactor per-ageclass history (#1252): scortch height question

New

Update fates_turb_z0mr (#1319): addresses #1316
restart fixes for two-stream (#1321): addresses ESCOMP/ctsm#2325
spitfire fire behavior refactor (#1326): should be b4b

Latest issues and discussions of note

Add NBP into bc_out (#1320)
- The carbon from this is only partially handled. We need to update this to pass the the HLM to the flux coupler.
- Came up in the context of Grazing (#1140)
- Not high priority, need to coordinate with fire emissions
htop issue: https://github.com/ESCOMP/CTSM/issues/2932#issuecomment-2605781603
- Adrianna to get to this. No questions currently.
FDI calculation (#1298)
- Marcos: FDI flag for different approaches?
  - Adrianna: yes, there is a Canadian FDI calculation, but Thonicke hardcode
- Action: Adrianna to test the update to match Thonicke

Calibration progress update

no updates

Walk-on topic

Rosie: Land use tool needs to wor for unstructed grids. Rosie has someone to work on this.

Janurary 27, 2025

Wins and Whimsy 🎉

Machine status and notices 🖥️

CISL: 11-12 February
NERSC: 27-31 January, Full site power down

Upcoming meetings and events 🗓️

FATES modeling meeting: TBD
- FATES tool poll results
  - modify_params had highest uptake, batchpatch, consistent with Ryan's expectations
  - immediate take away: don't change anything, look
- office hours: Greg to send out notice
Mingjie and Jessie wind mortality meeting this Wednesday 11.30 - others welcome

PR board status update

Integrated

Crown depth checks (#1314)
- Marcos: should we add more alllometry checks?
  - Greg: definitely worthwhile to add more checks for user conveience.
    - One option would be to move convienence checks (i.e. not things that could crash the model) upstream to fates tooling that changes the parameter file.
- Charlie: sapwood allometry?

Update

Meier2022 fix (CTSM#2934): derecho testing looks good, waiting on izumi and ctsm5.3.021 tag completion
- land use fixes (#1273), AllVars update (CTSM#2936), and use_fates_luh order (CTSM#2898) to be combined into #2934
Grazing (#1140): Charlie update with PRT2 build fix. Greg retesting.
nocomp patch cleanup (#1226): Needs review.

New

ncl max increase (#1315): Ryan to run 100-year smoke test to make sure things don't break

Latest issues and discussions of note

z0mr parameters (#1316)
- Adrianna is calibrating this, but consensus is that we should update them

Calibration progress update

Jessica: ran CLM calibration parameter file in ELM. At preliminary review, it looks good.
Adrianna: rerunning following htop issue fix (Adrianna to make formal issue).

Walk-on topic

Charlie: land use spin up
- Default decomp value in ELM is very agressive, Charlie adjusted and spin up works much better
- Might be due to of the decomp method "scheme"
- Default value in CLM is more reasonable relative to updated value
Rosie: Meier2022
- ELM
- Ryan: making the code more "defensive"
  - Initializing fates code with "nan"
  - Don't allocate if not used
  - Adrianna: API that doesn't allow HLM to have access to data structures
  - Greg: look for API update issue/discussion

January 20, 2025

Canceled due to Federal Holiday

January 13, 2025

Wins (1 min)
Machine status and meeting notices (5 min)
PR board status (10 min)
Latest issues (5 min)
Synthetic patches functional testing update (10 min) - deferred
Calibration update (10 min)

Wins and Whimsy 🎉

CTSM temp branch merged into master
ctsm5.0.18 has history time being on the middle of time bounds
- Marcos: will this be reflected in e3sm?
  - Greg will reach out to e3sm folks
Rosie: New FATES post-doc hire. Will be working on dwarf shrub parameterization.

Machine status and notices 🖥️

CISL: 11-12 February
NERSC: 15 January

Upcoming meetings and events 🗓️

FATES modeling meeting: 16 January, Annemarie Eckes-Shephard
- Ryan to conduct informal poll on using patches.

PR board status update

Integrated

e3sm API37 (E3SM#6762), ctsm hydro test update (CTSM#2882)

Update

fates hydro solver param update (#1263): regression testing underway
- Greg to create xml patch file
veg temp weight fix (#1306): Needs a second reviewer
leaf biophys refactor (#1262): Greg and Marcos reviewing
Grazing: Greg to deconflict, ok to push directly per Charlie
- Jennifer to quick review this as well.
- Discussion about reasonableness of original values. Consider these highly uncertain for now. Bring these in with zero values now.
nocomp patch cleanup: TL;DR, de-scoped a lot of changes. Mostly cleanup and "hardening" to make sure everything is consistent.
use_fates_luh passing order (CTSM#2898): discussion about use_fates_potentialveg name change
Large LAI crash fix (#1269): Ryan refactoring updates
two-stream finidat fix (#1286): WIP, needs cleanup

New

fix veg temp weighting (#1306): reviewed, Greg to test
two-stream area adjustment calculation (#1310): WIP

Latest issues and discussions of note

allom_h2cd check during read (#1308)

Synthetic patches for functional testing update (Adrianna)

insert_cohort not preserving sorted order inherently - deferred

Calibration progress update

deferred

Walk-on topic

Rosie: https://github.com/ESCOMP/CTSM/issues/2932
- Boundary conditions are different between clm6 and fates
- Displacement height needs to be fixed, biasing SP results. Is Meier2022 the default? Check in with Adrianna.
- Charlie: ELM using ZengWang? Rosie: yes
- Ryan: create quick PR to make sure ZengWang is default if fates is on? Greg to address.
- Jessie will reach out to Adrianna for latest parameter file and create a new smoke test

January 6, 2025

Wins and Whimsy 🎉

Machine status and notices 🖥️

CISL: 11-12 February
NERSC: 15 January

Upcoming meetings and events 🗓️

FATES modeling meeting: 16 January, Annemarie Eckes-Shephard

PR board status update

Integrated

modify_fates_paramfile fix (#1275), Indexing python tool fix (#1287), cmake functional test fix (#1303), ROS refactor (#1297), frac_burnt zero fix (#1302)

Update

Land use fixes (#1273): Deconflicted and reviewed. Build test failing.
e3sm API37 (E3SM#6762): Waiting on integration
ctsm hydro test update (CTSM#2882): On hold due to izumi nag compiler test fails (CTSM#2924)

Latest issues and discussions of note

FDI calculation (#1298)
- Xuilin, NGEE-tropics phase 3 has a bunch of work planned (Lara)
- Adrianna to bring this to the CA meeting
  - Jennifer will give Lara a heads up about discussing this at a future meeting
fire calculations regardless of event (#1299)
- Adrianna: do we want to avoid calculating values when there is no fire?
  - Sam: not doing so will improve computation. Maybe improve history output.
  - Sam: some of the fire outputs were weighted by burned area, but not all outputs.
  - Rosie: you might want to know the potential effects regardless of fire active or not (are you looking "upstream" or "downstream") .
PVT run FAIL (CTSM#2919)

Synthetic (aka "dummy") patches for functional testing (Adrianna)

Adrianna developing "dummy" patches for use with funtional and unit tests
Idealized biomes with simplistic, well-defined values
Motivated by future work to conduct with Danica as well as creating unit tests for cohort sorting, patch insertion, etc.