Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Error reading initial/restart file for CLM

Y

yixia

Yi Xia
New Member
Hi,

I'm running CESM2.1.5 using the FHIST compset on the DOE machine Perlmutter. When I used the CLM initial file from the CESM input data folder (clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc), the simulations crashed while reading the initial CLM files. I couldn't resolve this error. I would appreciate any insights on this problem.

The error message in cesm.log:

Code: 
  1: forrtl: severe (174): SIGSEGV, segmentation fault occurred
  1: Image              PC                Routine            Line        Source
  1: cesm.exe           000000000245242A  Unknown               Unknown  Unknown
  1: libpthread-2.31.s  00007FAFC6BA2910  Unknown               Unknown  Unknown
  1: libnetcdff.so.7.0  00007FAFC6CB3D5F  netcdf_mp_nf90_ge     Unknown  Unknown
  1: cesm.exe           00000000022F510B  pionfget_mod_mp_g         416  pionfget_mod.F90.in
  1: cesm.exe           00000000019092BC  ncdio_pio_mp_ncd_        1377  ncdio_pio.F90.in
  1: cesm.exe           00000000018C45A0  initinterpmod_mp_         920  initInterp.F90
  1: cesm.exe           00000000018C5328  initinterpmod_mp_         764  initInterp.F90
  1: cesm.exe           00000000018C8374  initinterpmod_mp_         354  initInterp.F90
  1: cesm.exe           000000000182A1F7  clm_initializemod         552  clm_initializeMod.F90
  1: cesm.exe           00000000018120C8  lnd_comp_mct_mp_l         233  lnd_comp_mct.F90
  1: cesm.exe           0000000000438B43  component_mod_mp_         267  component_mod.F90
  1: cesm.exe           0000000000427354  cime_comp_mod_mp_        1237  cime_comp_mod.F90
  1: cesm.exe           0000000000435B59  MAIN__                    114  cime_driver.F90
  1: cesm.exe           0000000000418152  Unknown               Unknown  Unknown
  1: libc-2.31.so       00007FAFC3AF21FD  __libc_start_main     Unknown  Unknown
  1: cesm.exe           000000000041806A  Unknown               Unknown  Unknown
srun: error: nid006383: task 1: Exited with exit code 174
srun: Terminating StepId=44600692.0
  0: slurmstepd: error: *** STEP 44600692.0 ON nid006383 CANCELLED AT 2025-10-30T15:46:46 ***

The bottom lines in the lnd.log:
Code: 
(GETFIL): attempting to find local file
 clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc
 (GETFIL): using
 /global/cfs/cdirs/m4373/yixia00/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000
 -01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc
 **** Mapping clm initial data from input
 /global/cfs/cdirs/m4373/yixia00/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000
 -01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc  to output finidat_interp_dest.nc
  ****
 input gridcells =        21013  output gridcells =         5663
 input landuntis =        62336  output landunits =        17701
 input columns   =       554509  output columns   =        39440
 input pfts      =       848691  output pfts      =       118722
 Glacier elevation classes same in input and output?:  T
 ipft_not_vegetated                      =            0
 icol_vegetated_or_bare_soil             =            1
 ilun_vegetated_or_bare_soil             =            1
 ilun_crop                               =            2
 ilun_landice_multiple_elevation_classes =            4
 create_glacier_mec_landunits            = true
 finding minimum distance for pfts
 calling set_subgrid_info for pft for input

I tried running another FHIST with a cold start for the CLM component by setting CLM_FORCE_COLDSTART=on. The simulation finished successfully. Then I took the restart file from the finished run to run another hybrid simulation. It crashed while initializing the land model again. The error messages go as follows.

The error message in cesm.log:

Code: 
127:  proc=          127  clump no =            1  clump id=          128
127:   beg cohort     =         5622  end cohort     =         5663
127:   total cohorts per clump     =           42
  1:  NetCDF: Invalid dimension ID or name
  1:  NetCDF: Invalid dimension ID or name
  1:  NetCDF: Invalid dimension ID or name
  1:  NetCDF: Invalid dimension ID or name
  1:  NetCDF: Invalid dimension ID or name
  1:  NetCDF: Variable not found
  1:  NetCDF: Variable not found
  1: forrtl: severe (174): SIGSEGV, segmentation fault occurred
  1: Image              PC                Routine            Line        Source             
  1: libpthread-2.31.s  00007F159C677910  Unknown               Unknown  Unknown
  1: libnetcdff_parall  00007F1599FE4A28  netcdf_mp_nf90_ge     Unknown  Unknown
  1: cesm.exe           00000000023E8675  pionfread_mod_mp_         172  pionfread_mod.F90.in
  1: cesm.exe           00000000023C0E84  piodarray_mp_read        1011  piodarray.F90.in
  1: cesm.exe           00000000023C729D  piodarray_mp_read         361  piodarray.F90.in
  1: cesm.exe           00000000019B86A5  ncdio_pio_mp_ncd_        1834  ncdio_pio.F90.in
  1: cesm.exe           0000000001979518  initinterp2dvar_m         185  initInterp2dvar.F90.in
  1: cesm.exe           000000000198120B  initinterpmultile         288  initInterpMultilevelContainer.F90
  1: cesm.exe           0000000001984371  initinterpmultile         117  initInterpMultilevelContainer.F90
  1: cesm.exe           00000000019723D2  initinterpmod_mp_         399  initInterp.F90
  1: cesm.exe           00000000018D2BD4  clm_initializemod         552  clm_initializeMod.F90
  1: cesm.exe           00000000018BC69C  lnd_comp_mct_mp_l         233  lnd_comp_mct.F90
  1: cesm.exe           0000000000437B98  component_mod_mp_         267  component_mod.F90
  1: cesm.exe           0000000000426B54  cime_comp_mod_mp_        1237  cime_comp_mod.F90
  1: cesm.exe           0000000000434D0B  MAIN__                    114  cime_driver.F90
  1: cesm.exe           00000000004183FD  Unknown               Unknown  Unknown
  1: libc-2.31.so       00007F15973401FD  __libc_start_main     Unknown  Unknown
  1: cesm.exe           000000000041832A  Unknown               Unknown  Unknown
srun: error: nid004788: task 1: Exited with exit code 174
srun: Terminating StepId=44598005.0
  0: slurmstepd: error: *** STEP 44598005.0 ON nid004788 CANCELLED AT 2025-10-30T14:01:32 ***
1

The error message in lnd.log:

Code: 
 **** Mapping clm initial data from input
 CESM2.SDCAM6.HIST.196001-196012.spinup.try2.1.clm2.r.1979-01-06-00000.nc
   to output finidat_interp_dest.nc ****
 input gridcells =         5663  output gridcells =         5663
 input landuntis =        17705  output landunits =        17701
 input columns   =        39460  output columns   =        39440
 input pfts      =       118742  output pfts      =       118722
 Glacier elevation classes same in input and output?:  T
 ipft_not_vegetated                      =            0
 icol_vegetated_or_bare_soil             =            1
 ilun_vegetated_or_bare_soil             =            1
 ilun_crop                               =            2
 ilun_landice_multiple_elevation_classes =            4
 create_glacier_mec_landunits            = true
 finding minimum distance for pfts
 calling set_subgrid_info for pft for input
 calling set_subgrid_info for pft for output
 calling set_mindist for pft
 finding minimum distance for columns
 calling set_subgrid_info for column for input
 calling set_subgrid_info for column for output
 calling set_mindist for column
 finding minimum distance for landunits
 calling set_subgrid_info for landunit for input
 calling set_subgrid_info for landunit for output
 calling set_mindist for landunit
 finding minimum distance for gridcells
 calling set_subgrid_info for gridcell for input
 calling set_subgrid_info for gridcell for output
 calling set_mindist for gridcell
 setting up interpolators for multi-level variables
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
What is your create_newcase command? Have you made any changes to the model?
I ran this on our own machine which uses the initial file you refer to and it seems to work fine:

./create_newcase --case FHIST_1deg --compset FHIST --res f09_g17 --run-unsupported

I assume cesm2.1.5 has been properly ported to perlmutter...
 
Y

yixia

Yi Xia
New Member
oleson said:
What is your create_newcase command? Have you made any changes to the model?
I ran this on our own machine which uses the initial file you refer to and it seems to work fine:

./create_newcase --case FHIST_1deg --compset FHIST --res f09_g17 --run-unsupported

I assume cesm2.1.5 has been properly ported to perlmutter...
Click to expand...
Thanks a lot for your response, Keith.

My create_newcase command is:
./create_newcase --case $CASEDIR --compset HIST_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV --res f19_f19_mg16 --machine perlmutter --run-unsupported

I ran the same simulation using the same version of CESM on derecho, and it's all good as well. So yes might be some problems with the machine config setting for case building. I attached my config files under $CESMROOT/cime/config/cesm/machines for reference. The machine name is perlmutter.
 

Attachments

  • config_batch.xml.txt
    28 KB · Views: 0
  • config_compilers.xml.txt
    45.7 KB · Views: 0
  • config_machines.xml.txt
    122.5 KB · Views: 0
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I'd suggest posting on the porting Forum for questions about running on perlmutter.
One other thing to check is whether the initial file you are using on perlmutter is the same as the one on Derecho, i.e., is not corrupted, is the same file type, etc.
 
Y

yixia

Yi Xia
New Member
Thanks for your suggestion, Keith!
I tried replacing the initial file I'm using on Perlmutter with the one from Derecho, and it still fails.
 
slevis

slevis

Moderator
Staff member
So the same case works on derecho but fails on perlmutter? I will move this thread to the "porting" forum as Keith suggested.
 
You must log in or register to reply here.
Top