ERROR: One or more of the output from CLM to t he coupler are NaN

[zqzq]

Dear all,
When I modified the thermodynamic roughness (z0mv) parameterization scheme, the following problems appeared:
cesm.log:
ERROR:
lnd_export ERROR: One or more of the output from CLM to t
he coupler are NaN
Image PC Routine Line Source
cesm.exe 00000000012A0CC6 Unknown Unknown Unknown
cesm.exe 0000000000DE2B15 shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 0000000000528286 abortutils_mp_end 50 abortutils.F90
cesm.exe 000000000052748E lnd_import_export 424 lnd_import_export.F90
cesm.exe 000000000051D7B8 lnd_comp_mct_mp_l 463 lnd_comp_mct.F90
cesm.exe 0000000000446E2A component_mod_mp_ 737 component_mod.F90
cesm.exe 0000000000427C7B cime_comp_mod_mp_ 2626 cime_comp_mod.F90
cesm.exe 0000000000446A05 MAIN__ 133 cime_driver.F90
cesm.exe 0000000000425A62 Unknown Unknown Unknown
libc-2.17.so 00002ADB1DD163D5 __libc_start_main Unknown Unknown
cesm.exe 0000000000425969 Unknown Unknown Unknown
Abort(1001) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 0

The missing values are shown in the attached picture，Hope to get your help, thank you.

 

[erik]

Moving to the CLM forum.

 

[oleson]

Examine the modifications you made to the code and add write statements to make sure you incorporated your changes correctly.

 

[zqzq]

Examine the modifications you made to the code and add write statements to make sure you incorporated your changes correctly.

thank you for your reply，the variable I changed is the thermal roughness length（z0hv） under the vegetation canopy,

The file path is my_cesm_sandbox/components/clm/src/biogeophys/CanopyFluxesMod.F90,The modified content is shown in the attached picture.CLM5.0 defaults to Z0MV=Z0HV. However, there is a certain relationship between Z0MV and Z0HV in my modification, but this leads to the addition of a parameter friction speed (ustar) in the relationship, I am not sure I will use it directly “call statement” to introduce the module, I hope you can give me some suggestions.

 

[oleson]

You could try to re-build and run the model with DEBUG set to TRUE in env_build.xml. This will turn on the debug compiler flags which should catch if a value has turned into a NaN or infinity or possibly point out a problem with the code in the build stage.

 

[ZYue]

Dear all,
When I modified the thermodynamic roughness (z0mv) parameterization scheme, the following problems appeared:
cesm.log:
ERROR:
lnd_export ERROR: One or more of the output from CLM to t
he coupler are NaN
Image PC Routine Line Source
cesm.exe 00000000012A0CC6 Unknown Unknown Unknown
cesm.exe 0000000000DE2B15 shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 0000000000528286 abortutils_mp_end 50 abortutils.F90
cesm.exe 000000000052748E lnd_import_export 424 lnd_import_export.F90
cesm.exe 000000000051D7B8 lnd_comp_mct_mp_l 463 lnd_comp_mct.F90
cesm.exe 0000000000446E2A component_mod_mp_ 737 component_mod.F90
cesm.exe 0000000000427C7B cime_comp_mod_mp_ 2626 cime_comp_mod.F90
cesm.exe 0000000000446A05 MAIN__ 133 cime_driver.F90
cesm.exe 0000000000425A62 Unknown Unknown Unknown
libc-2.17.so 00002ADB1DD163D5 __libc_start_main Unknown Unknown
cesm.exe 0000000000425969 Unknown Unknown Unknown
Abort(1001) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 0

The missing values are shown in the attached picture，Hope to get your help, thank you.

Hi zq,
Have you solved your problem? I have the same problem, can you give me some guidance and advice?
Thanks!
ZYue

 

[Xueli Huo]

You could try to re-build and run the model with DEBUG set to TRUE in env_build.xml. This will turn on the debug compiler flags which should catch if a value has turned into a NaN or infinity or possibly point out a problem with the code in the build stage.

Hi,

I have the same NaN values error. I tried turning on the DEBUG mode, and set the INFO_LEVEL to be 3 (much). However, CLM aborted right away, throwing out a data floating error which otherwise didn't show up when the DEBUG mode was not turned on. This means when the DEBUG mode is turned on, the error tolerance decreases? Any option that can be tuned to increase the error tolerance if the DEBUG mode is turned on?

Thanks!

 

[slevis]

Yes, DEBUG usually decreases the error tolerance and increases the verbosity of error messages. It is meant to help you "debug" your code. Other than turning off DEBUG, I don't know how to increase error tolerance. In general, it is best to address rather than mask errors so as to have confidence in the model output.

 

[Xueli Huo]

Examine the modifications you made to the code and add write statements to make sure you incorporated your changes correctly.

Hi,

I am still debugging this issue and I am considering to add some write statements. It's unclear to me for a particular column what the local column index corresponding to the global column index is so that only variable values corresponding to that column are printed out in the log file.
For example, I know the NaN values occurred in one particular column with the global index of 1378 from analyzing the restart files. And the section that I would like to add some write statements is in the source file biogeophys/SnowCoverFractionSwensonLawrence2012Mod.F90
The related codes in the file are:

do fc = 1, num_c
c = filter_c(fc)

!====================== FSCA PARAMETERIZATIONS ======================
! fsca parameterization based on *changes* in swe
if (h2osno_total(c) == 0._r8) then
if (newsnow(c) > 0._r8) then
frac_sno(c) = tanh(this%accum_factor * newsnow(c))
else
! NOTE(wjs, 2019-08-07) This resetting of frac_sno to 0 when h2osno_total is 0
! may already be done elsewhere; if it isn't, it possibly *should* be done
! elsewhere rather than here.
frac_sno(c) = 0._r8
end if

else ! h2osno_total(c) > 0
if (snowmelt(c) > 0._r8) then
! first compute change from melt during previous time step
frac_sno(c) = this%FracSnowDuringMelt( &
c = c, &
h2osno_total = h2osno_total(c), &
int_snow = int_snow(c))
end if

if (newsnow(c) > 0._r8) then

And I would like to write the variable values on the particular column whose global column index is 1378. But I have no idea what the local column index c is so that I can add an if statement to output only that column values.

Any suggestion?

Sincerely,
Xueli

 

[slevis]

A suggestion might be to look around the code for guidance. It's often the case that a template for what you're trying to accomplish exists elsewhere in the code already.

But another suggestion from looking at the code that you shared is this:
Think about any possible divisions by zero or log(0) or any syntax that may generate NaNs in your modified code.

 

[oleson]

There is some discussion about local and global indices here:

1.8.1. Troubleshooting — ctsm CTSM master documentation

 

[Xueli Huo]

Hi,

Thank you both sincerely for the suggestions. I ended up with adding the write statements using the GetGlobalIndex and GetGlobalWrite in the GetGlobalValuesMod to print out the global index of the local index for pft, column and gridcell. This helped me to trace down to the original. The NaN values are caused by the difference in temperatures of two adjacent snow layers divided by 0.


Sincerely,
Xueli