What is the cause of the error: ERROR: One or more of the output from CLM to the coupler are NaN?

[LiwenWu]

Dear Scientists,
I was running a I2000Clm50SpGs case for spinning up, but after submitting the case, an error occured and when I cat the cesm log file it shows:

---

# of NaNs = 7
Which are NaNs = F F F F F F T F F F F F F F F F F F F F F F F F F F F F F F F
F F F F F T T T T F F F F F T T F F F F F F F F F F F F F F
NaN found in field at gridcell index
169501
NaN found in field at gridcell index
169531
NaN found in field at gridcell index
169532
NaN found in field at gridcell index
169533
NaN found in field at gridcell index
169534
NaN found in field at gridcell index
169540
NaN found in field at gridcell index
169541
ERROR: ERROR: One or more of the output from CLM to the coupler are NaN
ERROR: ERROR: One or more of the output from CLM to the coupler are NaN
ERROR: ERROR: One or more of the output from CLM to the coupler are NaN
ERROR: ERROR: One or more of the output from CLM to the coupler are NaN
ERROR: ERROR: One or more of the output from CLM to the coupler are NaN

---

I wonder the reason for this error, there is no mention in the logfile why the NaN value occurred. Is this due to my atm forcing data or surfdata that contain the NaN value? Or due to my modificiation on the surfdata file? Note that I have changed the land use, PFT, LAI and soil texture data in the surfdata file but used the default SAI data. Maybe the LAI and SAI data is not corresponding in some gridcell so the NaN value occurred?
I attached the cesm.log file behind. Could you please give me some advice? Any advice would be greatly appreciated. Thank you.

 

[oleson]

Typically, a NaN in this case means that a calculation for some variable passed to the coupler has gone bad. This could come from either bad or missing atm forcing data or bad or missing input data, e.g., surface data. The log will usually tell you which variable it is and what the latitude/longitude is:

write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)

but I don't see that information in your log file. It could be appearing in one of your other log files. Look at src/cpl/utils/lnd_import_export_utils.F90 and see if you have that line in your file. If not, maybe you can add it and get the lat/lon information.

 

[LiwenWu]

Typically, a NaN in this case means that a calculation for some variable passed to the coupler has gone bad. This could come from either bad or missing atm forcing data or bad or missing input data, e.g., surface data. The log will usually tell you which variable it is and what the latitude/longitude is:

write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)

but I don't see that information in your log file. It could be appearing in one of your other log files. Look at src/cpl/utils/lnd_import_export_utils.F90 and see if you have that line in your file. If not, maybe you can add it and get the lat/lon information.

Dear Oleson,
Thanks for your timely and patient reply.
1) I followed your suggestion and modified the src/cpl/utils/lnd_import_export_utils.F90 like this:

---

! Check if any input from mediator or output to mediator is NaN

if (any(isnan(array))) then
write(iulog,*) '# of NaNs = ', count(isnan(array))
write(iulog,*) 'Which are NaNs = ', isnan(array)
do i = 1, size(array)
if (isnan(array(i))) then
write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)
end if
end do
call shr_sys_abort(' ERROR: One or more of the output from CLM to the coupler are NaN ' )
end if
end subroutine check_for_nans

---

then I ran ./case.submit and met the same error mentioned above. I checked the newly generated log files but there were still no information about which variable is bad and what the latitude/longitude is. I attached the log files behind. Could you please help me check if there is any relevant information in the logfiles?


2) What's more, I recreated a new case after modifying the src/cpl/utils/lnd_import_export_utils.F90, but encountered an error when ran ./case.build, opening the log file it shows:

---

err:

cat: Srcfiles: No such file or directory
/home/xunbin/CTSM-ctsm5.1.dev038/src/init_interp/initInterpMultilevelCopy.F90(47): warning #6178: The return value of this FUNCTION has not been defined. [CONSTRUCTOR]
type(interp_multilevel_copy_type) function constructor()
---------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/utils/restUtilMod.F90.in(104): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value. [READVAR]
long_name, units, interpinic_flag, data, readvar, &
------------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/biogeophys/SoilWaterRetentionCurveClappHornberg1978Mod.F90(35): warning #6178: The return value of this FUNCTION has not been defined. [CONSTRUCTOR]
type(soil_water_retention_curve_clapp_hornberg_1978_type) function constructor()
---------------------------------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/biogeophys/SoilWaterRetentionCurveVanGenuchten1980Mod.F90(35): warning #6178: The return value of this FUNCTION has not been defined. [CONSTRUCTOR]
type(soil_water_retention_curve_vangenuchten_1980_type) function constructor()
-------------------------------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/cpl/utils/lnd_import_export_utils.F90(161): error #6404: This name does not have a type, and must have an explicit type. [GRC]
write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)
--------------------------------------------------------------------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/cpl/utils/lnd_import_export_utils.F90(161): error #6460: This is not a field name that is defined in the encompassing structure. [LONDEG]
write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)
------------------------------------------------------------------------------------------------------------^
/home/xunbin/CTSM-ctsm5.1.dev038/src/cpl/utils/lnd_import_export_utils.F90(161): error #6460: This is not a field name that is defined in the encompassing structure. [LATDEG]
write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)
---------------------------------------------------------------------------------------------------------------------------------^
compilation aborted for /home/xunbin/CTSM-ctsm5.1.dev038/src/cpl/utils/lnd_import_export_utils.F90 (code 1)
make: *** [lnd_import_export_utils.o] Error 1
make: *** Waiting for unfinished jobs....

ERROR: Command make complib -j 8 MODEL=clm COMPLIB=/home/xunbin/cesm/scratch/shanxispinup5/bld/intel/impi/nodebug/nothreads/mct/mct/noesmf/lib/libclm.a -f /home/xunbin/CTSM-ctsm5.1.dev038/cime/scripts/shanxispinup5/Tools/Makefile CIME_MODEL=cesm SMP=FALSE CASEROOT="/home/xunbin/CTSM-ctsm5.1.dev038/cime/scripts/shanxispinup5" CASETOOLS="/home/xunbin/CTSM-ctsm5.1.dev038/cime/scripts/shanxispinup5/Tools" CIMEROOT="/home/xunbin/CTSM-ctsm5.1.dev038/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="FALSE" EXEROOT="/home/xunbin/cesm/scratch/shanxispinup5/bld" INCROOT="/home/xunbin/cesm/scratch/shanxispinup5/bld/lib/include" LIBROOT="/home/xunbin/cesm/scratch/shanxispinup5/bld/lib" LILAC_MODE="off" MACH="myintel" MPILIB="impi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="2" SHAREDLIBROOT="/home/xunbin/cesm/scratch/shanxispinup5/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" COMP_LND="clm" COMPARE_TO_NUOPC="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE" failed with rc=2

---

Seems to be related to the code: write(iulog,*) "NaN found in field ", trim(fname), ' at gridcell index/lon/lat: ',begg+i-1,grc%londeg(begg+i-1),grc%latdeg(begg+i-1)? Did I enter the wrong code?

 

[oleson]

You're going to have to define the variables that are associated with that line since you are using older code. Or use a newer version of the code. For example, grc%londeg and grc%latdeg are not defined in the version of the code you are using (presumably ctsm5.1.dev038). So you'll have add this in that subroutine:

use GridcellType , only : grc

There may be other things you need to add.

 

[LiwenWu]

You're going to have to define the variables that are associated with that line since you are using older code. Or use a newer version of the code. For example, grc%londeg and grc%latdeg are not defined in the version of the code you are using (presumably ctsm5.1.dev038). So you'll have add this in that subroutine:

use GridcellType , only : grc

There may be other things you need to add.

Thanks for your timely reply Oleson. Since I am not familiar with the code, may I ask which subroutine it is? Sorry to bother you again, sincerely looking forward to your reply.

 

[slevis]

Hi @LiwenWu,

I may have misunderstood, but I think @oleson is referring to the same subroutine that you referred to earlier in this thread.