CESM 2.2 producing NaNs

O

olderandcolder

Conn O'Rourke
New Member
Hi!

I have been trying to get a port of CESM 2.2 set-up to run on an unsupported machine.

v2.1.5 works fine, but I have a problem with 2.2. The config seems to work fine, and it happily builds the executable, but when I run a case it produces nonsensical values. e.g:

create_newcase --case fw2000climo_f19 --compset F2000climo --res f19_f19_mg17 --run-unsupported --compiler gnu

ncdump -v T fw2000climo_f19.cam.h0.0001-01-01-00000.nc

shows:

3.663985, -5.519625e-21, 3.664102, -0.02896234, 3.664015, -5.836616e+16,
3.66395, -0.02510962, 3.662463, -0.005411599, 3.662345, 1.198804e+19,
3.662009, 4.791464e-10, 3.661701, 2.938234e-07, 3.662174, 2.715558e+31,
3.661859, 7.122214e-18, 3.661371, -7.863312e+12, 3.661163, 1.058136e+34,
3.661381, -1.088261e-29, 3.661909, 1.093692e+12, 3.662251, -3868253,
3.66252, -1.976378e-28, 3.662592, 4.912823e+14, 3.662512, -8.644941e-14,
3.66259, -0.0005397396, 3.662619, -1.338264e+37, 3.662555, 6.146299e-15,
3.6625, -5.022401e-12, 3.662601, -0.1192986, 3.662822, 2.227761e+25,
3.663461, -8.781988e-24, 3.66398, 2.277065e+22, 3.664523, -0.0001048699,

I have set it up to use both the intel and gnu toolchains, and both show the same problem were the temperatures have nonsensical data.

I would like some advice on tracking down the root of the problem and getting a working version of 2.2.

Any help is appreciated!
 
Sort by date Sort by votes
F

fischer

CSEG and Liaisons
Staff member
Hi Conn,

I'm using CESM2.2.2, and was unable to reproduce the nonsensical temperatures. You can turn on debugging to try and catch any floating point exceptions.

./xmlchange DEBUG=TRUE

The CESM 2.2 releases are technical release to support CAM-SE and variable mesh grids. If you intend to do any science runs you should use CESM 2.1.5.

Chris
 
Vote Upvote 0 Downvote
O

olderandcolder

Conn O'Rourke
New Member
Hi @fischer

Thanks for getting back to me.

I actually don't use CESM, I'm an RSE and have been asked to install it on our HPC system for @Jakub_Sypek

The port of 2.1.5 seems to be fine, but I have been unable to get 2.2 to work, either with the Intel compilers or the GNU compilers. As to CESM 2.2 not to be used for science runs, I'll leave that up to @Jakub_Sypek to decide on - the requirement is to get it installed and running correctly.

Having switched on debug=True i get errors with the gnu toolchain:
Code: 
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_4'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_0'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'

Component atm build complete with 331 warnings
cam built in 7.003403 seconds
Component ice build complete with 756 warnings
copying /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/ice/obj/ice_comp_mct.mod -> /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/lib/include
cice built in 12.350160 seconds
ERROR: BUILD FAIL: cam.buildlib failed, cat /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/atm.bldlog.260604-091114

  611 |   call l_bndry_loc%make_left(grid_spacing, seed, &
      |                                           1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_4'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:620:44:

  620 |   call r_bndry_loc%make_right(grid_spacing, seed, &
      |                                            1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_0'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:555:40:

  555 |   op = deriv_op_from_seed(grid_spacing, second_derivative_seed, &
      |                                        1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:525:40:

  525 |   op = deriv_op_from_seed(grid_spacing, first_derivative_seed, &
      |                                        1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'
gmake: *** [/iridisfs/scratch/co1f23/CESM/gcc_version/fw2000climo_f19/Tools/Makefile:999: linear_1d_operators.o] Error 1

I have the following modules loaded:

Currently Loaded Modules:
1) netcdf-fortran/4.6.2.gcc 3) hdf5/1.14.6.gcc15 5) gcc/15.1.0 7) openmpi/5.0.8_gcc15 9) tbb/2022.3 11) mkl/2025.3
2) szip/2.1.1 4) binutils/2.42 6) pmix/5.0.2 8) netcdf-c/4.9.2.gcc 10) compiler-rt/2025.3

What GCC compiler version would you recommend for v2.2 - from looking at the config_machines.xml some of the compiler modules loaded on various machines look to be much older.

Thanks again!
 
Vote Upvote 0 Downvote
J

Jakub_Sypek

Jakub
New Member
Hi Chris,

I've been working with Dr Christopher Maloney at UoC on WACCM/CARMA simulations and CESM v2.2.2 is what they use in his lab. It has compsets such as FWma2000climo that are not available in 2.1.5 and has the specific CARMA coupling used in their recent satellite emissions paper, which is why we are interested in porting this specific version to our supercomputer.

Best wishes,
Jakub
 
Vote Upvote 0 Downvote
J

Jakub_Sypek

Jakub
New Member
Hello again,

I just had a meeting with Chris and he taught be a way to get the satellite stuff running on CESM 2.1.5 so there’s no need for us to port 2.2.2 anymore.

He did mention that 2.2.2 is difficult to port so if anyone knows why the model might have been producing NaNs, a response may still be useful for anyone in the future experiencing this issue. We have been unable to find out why ourselves.

Best wishes,
Jakub
 
Vote Upvote 0 Downvote
T

Tianliang

Tianliang Yang
New Member
olderandcolder said:
Hi @fischer

Thanks for getting back to me.

I actually don't use CESM, I'm an RSE and have been asked to install it on our HPC system for @Jakub_Sypek

The port of 2.1.5 seems to be fine, but I have been unable to get 2.2 to work, either with the Intel compilers or the GNU compilers. As to CESM 2.2 not to be used for science runs, I'll leave that up to @Jakub_Sypek to decide on - the requirement is to get it installed and running correctly.

Having switched on debug=True i get errors with the gnu toolchain:
Code: 
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_4'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_0'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'

Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'

Component atm build complete with 331 warnings
cam built in 7.003403 seconds
Component ice build complete with 756 warnings
copying /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/ice/obj/ice_comp_mct.mod -> /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/lib/include
cice built in 12.350160 seconds
ERROR: BUILD FAIL: cam.buildlib failed, cat /scratch/co1f23/CESM_OUTPUTS/fw2000climo_f19/bld/atm.bldlog.260604-091114

  611 |   call l_bndry_loc%make_left(grid_spacing, seed, &
      |                                           1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_4'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:620:44:

  620 |   call r_bndry_loc%make_right(grid_spacing, seed, &
      |                                            1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_0'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:555:40:

  555 |   op = deriv_op_from_seed(grid_spacing, second_derivative_seed, &
      |                                        1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'
/iridisfs/scratch/co1f23/CESM/gcc_version/cesm.2.2.2/components/cam/src/utils/linear_1d_operators.F90:525:40:

  525 |   op = deriv_op_from_seed(grid_spacing, first_derivative_seed, &
      |                                        1
Error: Interface mismatch in dummy procedure ‘seed’ at (1): Shape mismatch in argument '_formal_8'
gmake: *** [/iridisfs/scratch/co1f23/CESM/gcc_version/fw2000climo_f19/Tools/Makefile:999: linear_1d_operators.o] Error 1

I have the following modules loaded:

Currently Loaded Modules:
1) netcdf-fortran/4.6.2.gcc 3) hdf5/1.14.6.gcc15 5) gcc/15.1.0 7) openmpi/5.0.8_gcc15 9) tbb/2022.3 11) mkl/2025.3
2) szip/2.1.1 4) binutils/2.42 6) pmix/5.0.2 8) netcdf-c/4.9.2.gcc 10) compiler-rt/2025.3

What GCC compiler version would you recommend for v2.2 - from looking at the config_machines.xml some of the compiler modules loaded on various machines look to be much older.

Thanks again!
Click to expand...
Hi Conn,

I had also encountered this issue and nailed it. So I realized the lack of pnetcdf in your loaded modules and the NaN output should be gone with this module.

Hope this can help,
Tianliang
 
Vote Upvote 0 Downvote
You must log in or register to reply here.
Back
Top