Please fill in all relevant information below, deleting the red text after you have read it.
Before submitting a help request, please check to see if your question is already answered:
- Search the forums for similar issues
- Check the CIME troubleshooting guide to see if any suggestions there solve your problem
- Check any other relevant CESM documentation
What version of the code are you using?
CESM VERSION: release-cesm2.1.4-0-g4003996
> ./manage_externals/checkout_externals --status --verbose
Checking local status of required & optional components: cam, chem_proc, carma, clubb, cosp2, cice, cime, cism, source_cism, clm, fates, ptclm, mosart, pop, cvmix, marbl, rtm, ww3,
./cime
clean sandbox, on cime5.6.44
./components/cam
clean sandbox, on cam_cesm2_1_rel_r58
./components/cam/chem_proc
clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel
./components/cam/src/physics/carma/base
clean sandbox, on carma/release_tags/carma3_49_rel
./components/cam/src/physics/clubb
clean sandbox, on vendor_clubb_r8099_n03
./components/cam/src/physics/cosp2/src
clean sandbox, on CFMIP/COSPv2.0/tags/v2.1.4cesm/src
./components/cice
clean sandbox, on cice5_cesm2_1_1_20190321
./components/cism
clean sandbox, on cism-release-cesm2.1.2_02
M ./components/cism/source_cism
modified sandbox, on release-cism2.1.03
./components/clm
clean sandbox, on release-clm5.0.35
./components/clm/src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_20200902
./components/mosart
clean sandbox, on release-cesm2.0.04
./components/pop
clean sandbox, on pop2_cesm2_1_rel_n14
./components/pop/externals/CVMix
clean sandbox, on v0.93-beta
./components/pop/externals/MARBL
clean sandbox, on cesm2.1-n00
./components/rtm
clean sandbox, on release-cesm2.0.04
./components/ww3
clean sandbox, on ww3_181001
Describe every step you took leading up to the problem:
- Describe every step you took, starting with the create_newcase command and including any changes you made to xml files, user_nl files, etc. Please try to reproduce the problem first using your own instructions.
I am running B1850 hybrid experiment with some land grids turning to ocean grids.
I change user_nl_pop:
topography_file = "kmt.2.da"
region_mask_file = "region.2.da"
ebm_param_option = "internal"
ecosys_on = .false.
abio_dic_dic14_on = .false.
iage_on = .false.
s_interior_restore_tau = 365.
s_interior_restore_filename = ''
dt_count = 48
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
dec0147.hsn.de.hpc.ucar.edu 52: ERROR:
component_mod:check_fields NaN found in OCN instance: 1 field So_t 1d global
index: 19173
dec0147.hsn.de.hpc.ucar.edu 52: Image PC Routine Line Source
cesm.exe 0000000002DDDFFD shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 00000000004359AD component_type_mo 257 component_type_mod.F90
cesm.exe 0000000000431673 component_mod_mp_ 731 component_mod.F90
cesm.exe 0000000000418184 cime_comp_mod_mp_ 3452 cime_comp_mod.F90
cesm.exe 0000000000430F59 MAIN__ 125 cime_driver.F90
cesm.exe 00000000004145FD Unknown Unknown Unknown
libc-2.31.so 000014DB7770929D __libc_start_main Unknown Unknown
cesm.exe 000000000041452A Unknown Unknown Unknown
dec0147.hsn.de.hpc.ucar.edu 52: MPICH ERROR [Rank 52] [job id 75021d9b-b345-43c4-9e3d-e7120faf1289] [Fri May 23 13:24:33 2025] [dec0147] - Abort(1001) (rank 52 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 52
Before submitting a help request, please check to see if your question is already answered:
- Search the forums for similar issues
- Check the CIME troubleshooting guide to see if any suggestions there solve your problem
- Check any other relevant CESM documentation
What version of the code are you using?
CESM VERSION: release-cesm2.1.4-0-g4003996
> ./manage_externals/checkout_externals --status --verbose
Checking local status of required & optional components: cam, chem_proc, carma, clubb, cosp2, cice, cime, cism, source_cism, clm, fates, ptclm, mosart, pop, cvmix, marbl, rtm, ww3,
./cime
clean sandbox, on cime5.6.44
./components/cam
clean sandbox, on cam_cesm2_1_rel_r58
./components/cam/chem_proc
clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel
./components/cam/src/physics/carma/base
clean sandbox, on carma/release_tags/carma3_49_rel
./components/cam/src/physics/clubb
clean sandbox, on vendor_clubb_r8099_n03
./components/cam/src/physics/cosp2/src
clean sandbox, on CFMIP/COSPv2.0/tags/v2.1.4cesm/src
./components/cice
clean sandbox, on cice5_cesm2_1_1_20190321
./components/cism
clean sandbox, on cism-release-cesm2.1.2_02
M ./components/cism/source_cism
modified sandbox, on release-cism2.1.03
./components/clm
clean sandbox, on release-clm5.0.35
./components/clm/src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_20200902
./components/mosart
clean sandbox, on release-cesm2.0.04
./components/pop
clean sandbox, on pop2_cesm2_1_rel_n14
./components/pop/externals/CVMix
clean sandbox, on v0.93-beta
./components/pop/externals/MARBL
clean sandbox, on cesm2.1-n00
./components/rtm
clean sandbox, on release-cesm2.0.04
./components/ww3
clean sandbox, on ww3_181001
Describe every step you took leading up to the problem:
- Describe every step you took, starting with the create_newcase command and including any changes you made to xml files, user_nl files, etc. Please try to reproduce the problem first using your own instructions.
I am running B1850 hybrid experiment with some land grids turning to ocean grids.
I change user_nl_pop:
topography_file = "kmt.2.da"
region_mask_file = "region.2.da"
ebm_param_option = "internal"
ecosys_on = .false.
abio_dic_dic14_on = .false.
iage_on = .false.
s_interior_restore_tau = 365.
s_interior_restore_filename = ''
dt_count = 48
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
dec0147.hsn.de.hpc.ucar.edu 52: ERROR:
component_mod:check_fields NaN found in OCN instance: 1 field So_t 1d global
index: 19173
dec0147.hsn.de.hpc.ucar.edu 52: Image PC Routine Line Source
cesm.exe 0000000002DDDFFD shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 00000000004359AD component_type_mo 257 component_type_mod.F90
cesm.exe 0000000000431673 component_mod_mp_ 731 component_mod.F90
cesm.exe 0000000000418184 cime_comp_mod_mp_ 3452 cime_comp_mod.F90
cesm.exe 0000000000430F59 MAIN__ 125 cime_driver.F90
cesm.exe 00000000004145FD Unknown Unknown Unknown
libc-2.31.so 000014DB7770929D __libc_start_main Unknown Unknown
cesm.exe 000000000041452A Unknown Unknown Unknown
dec0147.hsn.de.hpc.ucar.edu 52: MPICH ERROR [Rank 52] [job id 75021d9b-b345-43c4-9e3d-e7120faf1289] [Fri May 23 13:24:33 2025] [dec0147] - Abort(1001) (rank 52 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 52