ERROR: lnd_export ERROR: One or more of the output from CLM to the coupler are NaN

[Yi Yao]

Hello, recently I started three FHIST runs which start in 1990 and last 30 years, however, all three simulations stopped in 2005 with the error message: ERROR: lnd_export ERROR: One or more of the output from CLM to the coupler are NaN .
My version is release-2.2.0, and the resolution is 0.9x1.25.
I found that you have the same problem while developing this version: Bug in snow aerosol numerics can cause aerosol masses to blow up, starting with ctsm1.0.dev065 · Issue #965 · ESCOMP/CTSM . I checked the source code and found that the SnowHydrologyMod.F90 is changed as Sacks did, so it is a little weird for me.
Do you have any idea?

 

[oleson]

Yes, that bug should be fixed in release-cesm2.2.0.
There should be some output in the logs (cesm and lnd) that will point to the specific field being passed from CLM to the coupler that has NaNs.

 

[Yi Yao]

Hello, here is the screenshot of my log file:
Is that meaning that Sl_t is the variable?
Thanks in advance.

 

[oleson]

Yes. Sl_t is surface or radiative temperature. It is derived from outgoing longwave, i.e., in components/clm/src/main/lnd2atmMod.F90:

lnd2atm_inst%t_rad_grc(g) = sqrt(sqrt(lnd2atm_inst%eflx_lwrad_out_grc(g)/sb))

Sl_t is assigned to t_rad_grc.

So, this is likely pointing to an unrealistic outgoing longwave (eflx_lwrad_out_grc).
In my experience, if this is a coupled simulation (you noted that it was an FHIST simulation, although the case name seems to indicate an IHIST simulation) then the problem could be associated with an instability in the atmosphere creating bad atmospheric forcing to the land. It could also be an undiscovered bug in the land model.
There may be more information/clues in the log file files. Can you post all of your log files?
Otherwise, you'll need to trace the presumably erroneous eflx_lwrad_out_grc back through the code to see what is causing it (after verifying that eflx_lwrad_out_grc is indeed erroneous).
The troubleshooting chapter in the CLM User's Guide may be able to help with this:

1.8.1. Troubleshooting — ctsm CTSM master documentation

 

[Yi Yao]

Hey Keith, I checked again and it should be I compset, sorry for the mistake. Thanks for your information, I'll try to solve it next week.

 

[Yi Yao]

Hi Keith, I'm sure that this problem was caused by a cracked atm forcing file.

 

[JIAA]

Yes. Sl_t is surface or radiative temperature. It is derived from outgoing longwave, i.e., in components/clm/src/main/lnd2atmMod.F90:

lnd2atm_inst%t_rad_grc(g) = sqrt(sqrt(lnd2atm_inst%eflx_lwrad_out_grc(g)/sb))

Sl_t is assigned to t_rad_grc.

So, this is likely pointing to an unrealistic outgoing longwave (eflx_lwrad_out_grc).
In my experience, if this is a coupled simulation (you noted that it was an FHIST simulation, although the case name seems to indicate an IHIST simulation) then the problem could be associated with an instability in the atmosphere creating bad atmospheric forcing to the land. It could also be an undiscovered bug in the land model.
There may be more information/clues in the log file files. Can you post all of your log files?
Otherwise, you'll need to trace the presumably erroneous eflx_lwrad_out_grc back through the code to see what is causing it (after verifying that eflx_lwrad_out_grc is indeed erroneous).
The troubleshooting chapter in the CLM User's Guide may be able to help with this:

1.8.1. Troubleshooting — ctsm CTSM master documentation

Hello, Oleson! I meet the same question. my cesm version is 2.2.0 and case is I1850Clm50SpCru. When running to 1905, the error occurs. Some logs are the in annexes.
Some changes I made are below:
NTASKS=60, RUN_STARTDATE=1905-01-01,DATM_CLMNCEP_YR_END=1931

user_nl_clm:

user_nl_datm:

I have checked all stream files above, and they are the right size.

Thanks for your help!