FATES-Hydro with drought-deciduous PFT?

[jennykowalcz]

Hi all! Has anyone run simulations with FATES-Hydro using a drought-deciduous PFT? My case ran without error for 10 years but when I tried to continue the run, it fails with an error: "Could not find a stable solution for hydro 1D solve" for the drought-deciduous PFT with LAI = 0. E.g:

42:  Could not find a stable solution for hydro 1D solve
 42:  
 42:  error code:            1
173:  error diag:   0.000000000000000E+000  0.000000000000000E+000
173:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
131:  lat:   -8.52631578947377      longitidue:   290.000000000000     
131:  is recruitment:  F
131:  layer:            4
131:  wb_step_err =                      NaN
131:  q_top_eff*dt_step =   0.000000000000000E+000
131:  w_tot_beg =                      NaN
131:  w_tot_end =                      NaN
131:  leaf water:                      NaN  kg/plant
131:  stem_water:   0.000000000000000E+000  kg/plant
131:  troot_water:   0.000000000000000E+000
131:  aroot_water:   0.000000000000000E+000
131:  LWP:                      NaN
131:  dbh:    3.22212417194120     

UPDATE: I am also encountering this error in a similar simulation using only evergreen PFTs, except now it occurs towards the end of year 2. I will try using the 2D hydro solver and see if that helps...

[jennykowalcz]

So it turns out that my question should not be specific to drought-deciduous PFTs, but rather just asking if anyone has seen similar errors using FATES-Hydro in no-comp mode in regional or global simulations?

Here are the cases I've tried and the errors I'm getting. These are all no-comp runs, regional-scale across/around the Amazon, using tropical tree PFTs (early, mid, and late successional evergreens and a drought-deciduous) based on Marcos's TRY+ analysis with the hydro-specific parameters unchanged from the values in the default parameter file for broadleaf evergreen and hydro-deciduous tropical trees (PFTs 1 and 5).

1. With TRY+ 4 PFTs:
1D solver: runs for 10 years successfully than fails upon resubmit with error:

151:  Could not find a stable solution for hydro 1D solve
151:  
151:  error code:            1
 14:  r1:  0.247079893141669       0.319417740613632      -1.984612688924937E-002
 14:                          0.000000000000000E+000
 14:  r2:  0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 14:                          0.000000000000000E+000
 14:  r3:                     NaN  0.000000000000000E+000  0.000000000000000E+000
154:                          0.000000000000000E+000
154:  r4:  -5551.14531024558       0.000000000000000E+000  0.000000000000000E+000
154:                          0.000000000000000E+000
154:  r5:  -5551.14531024558                          NaN  0.000000000000000E+000
154:  kmax_aroot_radial_out:   1.200780452984725E-004
154:  surf area of root:    1.20078045298473     
154:  aroot_frac_plant:   2.122536047336938E-004   1911.10144660632     
154:    9003858.61057154     
154:  kmax_upper_shell:    7912.61013237662     
154:  kmax_lower_shell:    1527.37710486970     
154:  
154:  tree lai:   0.000000000000000E+000  m2/m2 crown
154:  area and area to volume ratios
154:  
154:  a:  1.528356297635699E-004
154:                           1.20078045298473     
154:  r1:  0.262286191370948     
154:                           79.3750874482490     
154:  r2:  0.000000000000000E+000
154:                        
154:  r3:                     NaN
154:                        
154:  r4:  -5551.14531024558     
154:                        
154:  r5:  -5551.14531024558     
154:  inner shell kmaxs:    9941.22152546376        7392.79341159929     
 14:                          0.000000000000000E+000
 14:  r4:  -5551.14531024558       0.000000000000000E+000  0.000000000000000E+000
 14:                          0.000000000000000E+000
 14:  r5:  -5551.14531024558                          NaN  0.000000000000000E+000
 14:  kmax_aroot_radial_out:   1.422231235782002E-004
 14:  surf area of root:    1.42223123578200     
 14:  aroot_frac_plant:   1.999597691308903E-004   2263.55131394386     
 14:    11320033.6436785     
 14:  kmax_upper_shell:    10138.2282819373     
 14:  kmax_lower_shell:    2016.44098699463     
 14:  
 14:  tree lai:   0.000000000000000E+000  m2/m2 crown
 14:  area and area to volume ratios
 14:  
 14:  a:  2.122190451818226E-004
 14:                           1.42223123578200     
 14:  r1:  0.247079893141669     
 14:                           83.8457964174904     
 14:  r2:  0.000000000000000E+000
 14:                        
 14:  r3:                     NaN
 14:                        
 14:  r4:  -5551.14531024558     
 14:                        
 14:  r5:  -5551.14531024558     
 14:  inner shell kmaxs:    9955.24096706401        7792.63096400720     
154:    8427.41991550738        7912.61013237662        6090.49280228983     
154:    4328.28103606282        3966.93352940172        2730.30719805020     
154:    1216.12985213546     
 14:    9440.41426106515        10138.2282819373        9601.04945084937     
 14:    6277.92071677222        6204.27096253882        5377.99325412841     
 14:    2166.11678123531     
132:  pft:            4
132:  z nodes:    3.56056261455281        1.75528130727640     
132:  -0.190993887372315      -0.227329413955693      -0.227329413955693     
132:  psi_z:                      NaN  1.720175681111868E-002 -1.871740096248686E-003
132:  -2.227828256764042E-003 -2.227828256765797E-003
132:  vol,    theta,   H,  Psi,     kmax-
132:  flux:            0.000000000000000E+000
132:  l:  0.000000000000000E+000                     NaN                     NaN
132:                      NaN
132:                          4.794417216847331E-003
132:  s:  1.178730872639167E-003  0.000000000000000E+000  -5551.12810848877     
132:   -5551.14531024558     
132:                          1.788344028471106E-002
132:  t:  7.460786158210913E-004  0.000000000000000E+000  -5551.14718198568     
132:                          9.452025392294483E-003
132:  a:  1.310621744446665E-004  0.000000000000000E+000  -1845.22052352817     
132:                     in:  1.888766363526109E-002
132:                    out:  1.100216626556518E-004

Same with 2D solver: runs for 10 years than fails upon resubmit with error:

53:  Warning: number of iteraction reaches 100 for bisect_pv
 53:  Initialized plant compartments with positive pressure?
 53:  psi troot:                 Infinity
 53:  psi ag(:):                 Infinity                Infinity
 53:  psi_aroot(:):                      NaN                     NaN
 53:                      NaN                     NaN                     NaN
 53:                      NaN                     NaN                     NaN
 53:                      NaN
 53:  ENDRUN:
 53:  ERROR in FatesPlantHydraulicsMod.F90 at line 669      

2. With 3 evergreen PFTs and no drought-deciduous:
1D solver: fails towards end of year 2 with error:

167:  Could not find a stable solution for hydro 1D solve
167:  
167:  error code:            1
167:  error diag:   0.000000000000000E+000  0.000000000000000E+000
167:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
167:  lat:   -4.73684210526324      longitidue:   280.000000000000     
167:  is recruitment:  T
167:  layer:            4
167:  wb_step_err =   6.231370315263201E-005
167:  q_top_eff*dt_step =   1.278546902632006E-006
167:  w_tot_beg =    392231909620.569     
167:  w_tot_end =    392231909620.569     
167:  leaf water:   1.977119271375372E-002  kg/plant
167:  stem_water:   1.307778893556870E-002  kg/plant
167:  troot_water:   1.012960357648249E-002
167:  aroot_water:   2.423778333540367E-002
167:  LWP:   -11.4704920165053     
167:  dbh:   0.764072176312517     
167:  pft:            2
167:  z nodes:    1.25000000000000       0.600000000000000     
167:  -0.222826201934367      -0.227329413955693      -0.227329413955693     
167:  psi_z:   1.224999999999987E-002  5.879999999999441E-003 -2.183696778956801E-003
167:  -2.227828256765818E-003 -2.227828256765818E-003
167:  vol,    theta,   H,  Psi,     kmax-
167:  flux:            1.278546902632006E-006
167:  l:  8.911960606182126E-005  0.221850090989391       -11.4582420165053     
167:   -11.4704920165053     
167:                          7.288414898907708E-004
167:  s:  5.149745376445012E-005  0.253950204905016       -11.4382420068961     
167:   -11.4441220068961     
167:                          1.938993985079932E-003
167:  t:  3.987109367687730E-005  0.254058834166293       -11.4182420060674     
167:                          1.324312610523329E-003
167:  a:  5.650707462205153E-006  0.713138515214688      -0.705970633341333     
167:                     in:  2.645558340631056E-003
167:                    out:  1.268879133417446E-005
167:  r1:   1473234852.71776       0.266238549065649      -0.706057093286554     
167:                          0.000000000000000E+000
167:  r2:  0.000000000000000E+000  0.000000000000000E+000   299.249193326083     
167:                          0.000000000000000E+000
167:  r3:  -8.00830603553817       0.000000000000000E+000  4.743030200075967E-322
167:                          0.000000000000000E+000
167:  r4:  -8.00121554768493       0.000000000000000E+000  8.744961931390064E-322
167:                          0.000000000000000E+000
167:  r5:  -7.99276885173250       0.248867590478379       0.000000000000000E+000
167:  kmax_aroot_radial_out:   1.274987965542320E-005
167:  surf area of root:   0.127498796554232     
167:  aroot_frac_plant:    1191934.06224864        202.920637098726     
167:   1.702448512260032E-004
167:  kmax_upper_shell:    502.128225197214     
167:  kmax_lower_shell:    21.9675255422892     
167:  
167:  tree lai:   0.756667509231287       m2/m2 crown
167:  area and area to volume ratios
167:  
167:  a:  5.650707462205153E-006
167:                          0.127498796554232     
167:  r1:   1473234852.71776     
167:                           1938224.29436465     
167:  r2:  0.000000000000000E+000
167:                        
167:  r3:  -8.00830603553817     
167:                        
167:  r4:  -8.00121554768493     
167:                        
167:  r5:  -7.99276885173250     
167:  inner shell kmaxs:    649.823728610446        488.471638384210     
167:    546.597463408998        502.128225197214        410.895270938480     
167:    323.642125620401        354.118419449814        329.681453045857     
167:    139.079050444771     
167:  ENDRUN:
167:  ERROR in FatesPlantHydraulicsMod.F90 at line 3456  

2D solver: fails after 20 years with error:

12:  Site plant water balance does not close
 12:  delta plant storage:   0.178159998792918       [kg/m2]
 12:  integrated root flux:   0.174079358831601       [kg/m2]
 12:  transpiration flux:   9.561243103577813E-008  [kg/m2]
 12:  end storage:    2.34503585071237     
 12:  pre_h2oveg   2.16687585191945     
 12:  ENDRUN:
 12:  ERROR in FatesPlantHydraulicsMod.F90 at line 2808

[xuchongang]

@jennykowalcz , looks like the soil is really dry (-11MPa) and leaf water potential is also really dry (-11MPa). This could potentially that there is not much water available in the soil for the plant to pull the water. I think I will need to make a pull request to fix this edge case.

[jennykowalcz]

Okay, thank you!

[jennykowalcz]

Hi @xuchongang! I tried the 3 PFT case with the 1D hydro solver again with this branch where I have merged your branch xu/xuchongang/master-hydro-stability and fixed that typo I mentioned. Unfortunately it fails again at the same spot with the same error:

167:  Could not find a stable solution for hydro 1D solve
167:  
167:  error code:            1
167:  error diag:   0.000000000000000E+000  0.000000000000000E+000
167:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
167:  lat:   -4.73684210526324      longitidue:   280.000000000000     
167:  is recruitment:  T
167:  layer:            4
167:  wb_step_err =   6.231370315263201E-005
167:  q_top_eff*dt_step =   1.278546902632006E-006
167:  w_tot_beg =    392231909620.569     
167:  w_tot_end =    392231909620.569     
167:  leaf water:   1.977119271375372E-002  kg/plant
167:  stem_water:   1.307778893556870E-002  kg/plant
167:  troot_water:   1.012960357648249E-002
167:  aroot_water:   2.423778333540367E-002
167:  LWP:   -11.4704920165053     
167:  dbh:   0.764072176312517     
167:  pft:            2
167:  z nodes:    1.25000000000000       0.600000000000000     
167:  -0.222826201934367      -0.227329413955693      -0.227329413955693     
167:  psi_z:   1.224999999999987E-002  5.879999999999441E-003 -2.183696778956801E-003
167:  -2.227828256765818E-003 -2.227828256765818E-003
167:  vol,    theta,   H,  Psi,     kmax-
167:  flux:            1.278546902632006E-006
167:  l:  8.911960606182126E-005  0.221850090989391       -11.4582420165053     
167:   -11.4704920165053     
167:                          7.288414898907708E-004
167:  s:  5.149745376445012E-005  0.253950204905016       -11.4382420068961     
167:   -11.4441220068961     
167:                          1.938993985079932E-003
167:  t:  3.987109367687730E-005  0.254058834166293       -11.4182420060674     
167:                          1.324312610523329E-003
167:  a:  5.650707462205153E-006  0.713138515214688      -0.705970633341333     
167:                     in:  2.645558340631056E-003
167:                    out:  1.268879133417446E-005
167:  r1:   1473234852.71776       0.266238549065649      -0.706057093286554     
167:                          0.000000000000000E+000
167:  r2:  0.000000000000000E+000  0.000000000000000E+000   299.249193326083     
167:                          0.000000000000000E+000
167:  r3:  -8.00830603553817       0.000000000000000E+000  1.263441270150936E-315
167:                          0.000000000000000E+000
167:  r4:  -8.00121554768493       0.000000000000000E+000  1.262391321365646E-315
167:                          0.000000000000000E+000
167:  r5:  -7.99276885173250       0.248867590478379       4.743030200075967E-322
167:  kmax_aroot_radial_out:   1.274987965542320E-005
167:  surf area of root:   0.127498796554232     
167:  aroot_frac_plant:    1191934.06224864        202.920637098726     
167:   1.702448512260032E-004
167:  kmax_upper_shell:    502.128225197214     
167:  kmax_lower_shell:    21.9675255422892     
167:  
167:  tree lai:   0.756667509231287       m2/m2 crown
167:  area and area to volume ratios
167:  
167:  a:  5.650707462205153E-006
167:                          0.127498796554232     
167:  r1:   1473234852.71776     
167:                           1938224.29436465     
167:  r2:  0.000000000000000E+000
167:                        
167:  r3:  -8.00830603553817     
167:                        
167:  r4:  -8.00121554768493     
167:                        
167:  r5:  -7.99276885173250     
167:  inner shell kmaxs:    649.823728610446        488.471638384210     
167:    546.597463408998        502.128225197214        410.895270938480     
167:    323.642125620401        354.118419449814        329.681453045857     
167:    139.079050444771     
167:  ENDRUN:
167:  ERROR in FatesPlantHydraulicsMod.F90 at line 3476    

(only the line number is different because of the changes to FatesPlantHydraulicsMod)
What variables could I look at to help with diagnostics?

[jennykowalcz]

@xuchongang The 4 PFT case with the 1D solver and this branch also fails again in the same way it did before, i.e. it ran successfully for 10 years (STOP_N value) and then crashed upon restart. I think like @ckoven said this is probably a separate error than the one I posted about above with the 3 PFT case. In this case, with the error on resubmit, the error messages all have PFT = 4 (drought deciduous), is recruitment = F, LAI = 0, stem_water = 0, w_tot_* = NaN, and LWP = NaN. For example:

209:  Could not find a stable solution for hydro 1D solve
209:
209:  error code:            1
 15:  lat:   -23.6842105263159      longitidue:   302.500000000000
 15:  is recruitment:  F
 15:  layer:            4
 15:  wb_step_err =                      NaN
 15:  q_top_eff*dt_step =   0.000000000000000E+000
 15:  w_tot_beg =                      NaN
 15:  w_tot_end =                      NaN
 15:  leaf water:                      NaN  kg/plant
 15:  stem_water:   0.000000000000000E+000  kg/plant
 15:  troot_water:   0.000000000000000E+000
 15:  aroot_water:   0.000000000000000E+000
 15:  LWP:                      NaN
 15:  dbh:    3.49675824928844
 15:  pft:            4
 15:  z nodes:    4.24889198027262        2.09944599013631
 15:  -0.190993887372315      -0.227329413955693      -0.227329413955693
 15:  psi_z:                      NaN  2.057457070350210E-002 -1.871740096248686E-003
 15:  -2.227828256764042E-003 -2.227828256765796E-003
 15:  vol,    theta,   H,  Psi,     kmax-
 15:  flux:            0.000000000000000E+000
 15:  l:  0.000000000000000E+000                     NaN                     NaN
 15:                      NaN
 15:                          6.053417126477062E-003
 15:  s:  1.888056051911535E-003  0.000000000000000E+000  -5551.12473567488
 15:   -5551.14531024558
...
153:  ERROR: Unknown error submitted to shr_abort_abort.
237:  ENDRUN:
237:  ERROR in FatesPlantHydraulicsMod.F90 at line 3476

@mpaiao how is a restart handled with FATES-Hydro if the leaves are off a drought deciduous PFT?

[jennykowalcz]

This issue is resolved by PR 1156 and PR 1164 with fates_hydro_solver=2