Hi,
I'm running single-point simulations at Barro Colorado Island, Panama using ctsm5.3.077 with FATES and the environmentally sensitive Tree Recruitment Scheme (TRS).
While debugging, I noticed that soil matric potential (SMP) from the host model spikes to ~-115 trillion on the first model day (see diagnostic output below). I've describe the issue on the FATES GitHub (First-day host model SMP cause 100% seedling mortality when using the Tree Recruitment Scheme with seedling dynamics · Issue #1500 · NGEET/fates) and in the short term I can work around this by skipping the exponential moving average updates of moisture deficit on model day one. However, since this values is coming from the host model, I wanted to raise the issue here as well.
Is this SMP spike a known initialization artifact in CTSM/CLM?
While I have a work around for my specific case, I imagine the SMP spike could generate other problems for modelers.
What version of the code are you using?
ctsm5.3.077 with FATES (sci.1.84.0_api.40.0.0)
Have you made any changes to files in the source tree?
I've fixed a couple of minor bugs and added diagnostic print statements to track SMP and moisture deficit accumulation in `FatesInterfaceMod.F90` and `EDPhysiologyMod.F90` in my `debug_trs` branch of FATES (GitHub - r-ward/fates at trs_debug)
Describe every step you took leading up to the problem:
- created a single-point case for Barro Colorado Island (I use observed meteorology and a surface file created with the ctsm subset tool).
- edited the user_nl_clm to run with fates_regeneration_model = 'trs'
- model crashes when the seed germination pool goes negative, in part due to extreme moisture deficit on day one which is 'remembered' by an exponential moving average of moisture deficit for the seedling layer which causes 100% mortality of the seed germination pool
- log shows the SMP spiking on the second substep (c_index) of model day 1, and returning to more reasonable values by step 9 or 10,
Please attach any log files showing error messages or other useful information.
------------------------------------------------------------
Successfully initialized the land model
begin initial run at:
nstep= 1 year= 1985 month= 1 day= 1 seconds= 1800
************************************************************
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: 0.0000000000000000
new_seedling_layer_smp: 0.0000000000000000
seedling_psi_crit: -251995.70000000001
Deficit check val for check 251995.70000000001
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 0.0000000000000000
Current c_index: 1
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -115997489906016.48
new_seedling_layer_smp: -115997489906016.48
seedling_psi_crit: -251995.70000000001
Deficit check val for check -115997489654020.78
new_seedling_mdd (after zero check): 14615683696406618.
Current EMA c_mean: 0.0000000000000000
Current c_index: 2
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -30257502454.039650
new_seedling_layer_smp: -30257502454.039650
seedling_psi_crit: -251995.70000000001
Deficit check val for check -30257250458.339649
new_seedling_mdd (after zero check): 3812413557750.7959
Current EMA c_mean: 4871894565468872.0
Current c_index: 3
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -332845219.35745889
new_seedling_layer_smp: -332845219.35745889
seedling_psi_crit: -251995.70000000001
Deficit check val for check -332593223.65745890
new_seedling_mdd (after zero check): 41906746180.839821
Current EMA c_mean: 3654874027491091.5
Current c_index: 4
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -38398293.236233741
new_seedling_layer_smp: -38398293.236233741
seedling_psi_crit: -251995.70000000001
Deficit check val for check -38146297.536233738
new_seedling_mdd (after zero check): 4806433489.5654507
Current EMA c_mean: 2923907603342109.5
Current c_index: 5
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -8543591.0642576218
new_seedling_layer_smp: -8543591.0642576218
seedling_psi_crit: -251995.70000000001
Deficit check val for check -8291595.3642576216
new_seedling_mdd (after zero check): 1044741015.8964603
Current EMA c_mean: 2436590470524006.5
Current c_index: 6
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -2752552.4450957361
new_seedling_layer_smp: -2752552.4450957361
seedling_psi_crit: -251995.70000000001
Deficit check val for check -2500556.7450957359
new_seedling_mdd (after zero check): 315070149.88206273
Current EMA c_mean: 2088506266840722.2
Current c_index: 7
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -1023203.0811228864
new_seedling_layer_smp: -1023203.0811228864
seedling_psi_crit: -251995.70000000001
Deficit check val for check -771207.38112288644
new_seedling_mdd (after zero check): 97172130.021483690
Current EMA c_mean: 1827443022869400.8
Current c_index: 8
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -440411.51312879863
new_seedling_layer_smp: -440411.51312879863
seedling_psi_crit: -251995.70000000001
Deficit check val for check -188415.81312879862
new_seedling_mdd (after zero check): 23740392.454228625
Current EMA c_mean: 1624393808903037.2
Current c_index: 9
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -213703.35025217672
new_seedling_layer_smp: -213703.35025217672
seedling_psi_crit: -251995.70000000001
Deficit check val for check 38292.349747823289
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 1461954430386772.8
Current c_index: 10
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -135027.13194443836
new_seedling_layer_smp: -135027.13194443836
seedling_psi_crit: -251995.70000000001
Deficit check val for check 116968.56805556166
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 1329049482169793.2
Current c_index: 11
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
I'm running single-point simulations at Barro Colorado Island, Panama using ctsm5.3.077 with FATES and the environmentally sensitive Tree Recruitment Scheme (TRS).
While debugging, I noticed that soil matric potential (SMP) from the host model spikes to ~-115 trillion on the first model day (see diagnostic output below). I've describe the issue on the FATES GitHub (First-day host model SMP cause 100% seedling mortality when using the Tree Recruitment Scheme with seedling dynamics · Issue #1500 · NGEET/fates) and in the short term I can work around this by skipping the exponential moving average updates of moisture deficit on model day one. However, since this values is coming from the host model, I wanted to raise the issue here as well.
Is this SMP spike a known initialization artifact in CTSM/CLM?
While I have a work around for my specific case, I imagine the SMP spike could generate other problems for modelers.
What version of the code are you using?
ctsm5.3.077 with FATES (sci.1.84.0_api.40.0.0)
Have you made any changes to files in the source tree?
I've fixed a couple of minor bugs and added diagnostic print statements to track SMP and moisture deficit accumulation in `FatesInterfaceMod.F90` and `EDPhysiologyMod.F90` in my `debug_trs` branch of FATES (GitHub - r-ward/fates at trs_debug)
Describe every step you took leading up to the problem:
- created a single-point case for Barro Colorado Island (I use observed meteorology and a surface file created with the ctsm subset tool).
- edited the user_nl_clm to run with fates_regeneration_model = 'trs'
- model crashes when the seed germination pool goes negative, in part due to extreme moisture deficit on day one which is 'remembered' by an exponential moving average of moisture deficit for the seedling layer which causes 100% mortality of the seed germination pool
- log shows the SMP spiking on the second substep (c_index) of model day 1, and returning to more reasonable values by step 9 or 10,
Please attach any log files showing error messages or other useful information.
------------------------------------------------------------
Successfully initialized the land model
begin initial run at:
nstep= 1 year= 1985 month= 1 day= 1 seconds= 1800
************************************************************
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: 0.0000000000000000
new_seedling_layer_smp: 0.0000000000000000
seedling_psi_crit: -251995.70000000001
Deficit check val for check 251995.70000000001
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 0.0000000000000000
Current c_index: 1
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -115997489906016.48
new_seedling_layer_smp: -115997489906016.48
seedling_psi_crit: -251995.70000000001
Deficit check val for check -115997489654020.78
new_seedling_mdd (after zero check): 14615683696406618.
Current EMA c_mean: 0.0000000000000000
Current c_index: 2
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -30257502454.039650
new_seedling_layer_smp: -30257502454.039650
seedling_psi_crit: -251995.70000000001
Deficit check val for check -30257250458.339649
new_seedling_mdd (after zero check): 3812413557750.7959
Current EMA c_mean: 4871894565468872.0
Current c_index: 3
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
--WARNING-- skipping CN balance check for first timesteps after startup or data assimilation
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -332845219.35745889
new_seedling_layer_smp: -332845219.35745889
seedling_psi_crit: -251995.70000000001
Deficit check val for check -332593223.65745890
new_seedling_mdd (after zero check): 41906746180.839821
Current EMA c_mean: 3654874027491091.5
Current c_index: 4
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -38398293.236233741
new_seedling_layer_smp: -38398293.236233741
seedling_psi_crit: -251995.70000000001
Deficit check val for check -38146297.536233738
new_seedling_mdd (after zero check): 4806433489.5654507
Current EMA c_mean: 2923907603342109.5
Current c_index: 5
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -8543591.0642576218
new_seedling_layer_smp: -8543591.0642576218
seedling_psi_crit: -251995.70000000001
Deficit check val for check -8291595.3642576216
new_seedling_mdd (after zero check): 1044741015.8964603
Current EMA c_mean: 2436590470524006.5
Current c_index: 6
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -2752552.4450957361
new_seedling_layer_smp: -2752552.4450957361
seedling_psi_crit: -251995.70000000001
Deficit check val for check -2500556.7450957359
new_seedling_mdd (after zero check): 315070149.88206273
Current EMA c_mean: 2088506266840722.2
Current c_index: 7
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -1023203.0811228864
new_seedling_layer_smp: -1023203.0811228864
seedling_psi_crit: -251995.70000000001
Deficit check val for check -771207.38112288644
new_seedling_mdd (after zero check): 97172130.021483690
Current EMA c_mean: 1827443022869400.8
Current c_index: 8
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -440411.51312879863
new_seedling_layer_smp: -440411.51312879863
seedling_psi_crit: -251995.70000000001
Deficit check val for check -188415.81312879862
new_seedling_mdd (after zero check): 23740392.454228625
Current EMA c_mean: 1624393808903037.2
Current c_index: 9
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -213703.35025217672
new_seedling_layer_smp: -213703.35025217672
seedling_psi_crit: -251995.70000000001
Deficit check val for check 38292.349747823289
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 1461954430386772.8
Current c_index: 10
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000
========================
=== MDD DEBUG Day: 1.0208333333333333
PFT: 1 Soil layer: 2
Raw SMP from host: -135027.13194443836
new_seedling_layer_smp: -135027.13194443836
seedling_psi_crit: -251995.70000000001
Deficit check val for check 116968.56805556166
new_seedling_mdd (after zero check): 0.0000000000000000
Current EMA c_mean: 1329049482169793.2
Current c_index: 11
n_mem: 6048
sdlng_mdd_timescale: 126.00000000000000