Runtime/Timeout Issues 1.2.2 (Resolved)

[jmnicklas@gmail_com]

To those who may be able to help me,

I am currently trying to get CESM1.2.2 to run on Brown University's Oscar computer cluster. The system is SLURM-based, and currently I only have access to exploratory account which gives me access only 16 cpus for up to 24 hours for a single job. I have successfully compiled many cases, but somehow I have only succeeded at running the compset X testcase. All other attempts that made to run compset B in the past month have failed: either the f19g16 or f45g37 resolutions timeout, or the T31_gx3v7 resolution results in a cryptic MPI COMMUNICATOR 9 CREATE FROM 0 error. The former cases should not be timing out: I got them to run in a fraction of the 24 hours on my 4core, dual-processor laptop. I just want to get something, anything, to work at this point.
I have attached everything I think would be of relevance: configure files, and logs from one of the resolutions that timed out.  The failed log from the compset B res T31_gx3v7 resolution is copied below. I am happy to search for whatever else you might need to solve these problems.
Again, any help would be greatly appreciated.

Sincerely,
John Nicklas ...In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 9)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 10)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 11)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 12)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 13)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 14)
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 15)
   96.0000000000000       and that obtained from surface dataset
   100.000000000000       at g=          404
 ENDRUN: called without a message string
In: PMI_Abort(1, application called MPI_Abort(comm=0xC4000008, 1) - process 3)
slurmstepd: *** STEP 14873889.0 ON node454 CANCELLED AT 2017-02-17T01:58:31 ***
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
srun: error: node454: tasks 0-5,8-10,12-14: Killed
srun: error: node454: tasks 6-7,11,15: Exited with exit code 1

 

[jmnicklas@gmail_com]

Hello,I have succesfully run compsets A and C. I am now attempting to run compset B at res f45_g37. However, I keep getting this error:Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L2 6_c060608.nc       65536 Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002B7F584927E0  Unknown               Unknown  Unknowncesm.exe           0000000001B43520  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771EA66  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771E932  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57736FEA  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57737198  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5773F5D5  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F576A087A  Unknown               Unknown  Unknownlibmpifort.so.12   00002B7F57DE26E2  Unknown               Unknown  Unknowncesm.exe           000000000199CCCA  pio_spmd_utils_mp         297  pio_spmd_utils.F90.incesm.exe           000000000198B5AC  box_rearrange_mp_         571  box_rearrange.F90.incesm.exe           00000000019201FA  rearrange_mp_rear         108  rearrange.F90.incesm.exe           0000000001964589  piodarray_mp_read         390  piodarray.F90.incesm.exe           00000000009767BE  ncdio_atm_mp_infl         287  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002B7F586BED5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node808: task 1: Exited with exit code 174 From there, the code stops doing anything but fails to exit, and hence my earlier confusion about what was actually happening. I realize that this error looks identical to this one: https://bb.cgd.ucar.edu/cesm-122-segmentation-fault-when-running-model, but I tried everything listed on that post and nothing worked. The modification to the source code in diags_on_lat_aux_grid.F90 did not help, nor did setting the  -heap_arrays, nor  -O2 . I already uncommented limit coredumpsize 1000000 and limit stacksize unlimited. 

 

[jmnicklas@gmail_com]

Hello,I have succesfully run compsets A and C. I am now attempting to run compset B at res f45_g37. However, I keep getting this error:Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L2 6_c060608.nc       65536 Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002B7F584927E0  Unknown               Unknown  Unknowncesm.exe           0000000001B43520  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771EA66  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771E932  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57736FEA  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57737198  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5773F5D5  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F576A087A  Unknown               Unknown  Unknownlibmpifort.so.12   00002B7F57DE26E2  Unknown               Unknown  Unknowncesm.exe           000000000199CCCA  pio_spmd_utils_mp         297  pio_spmd_utils.F90.incesm.exe           000000000198B5AC  box_rearrange_mp_         571  box_rearrange.F90.incesm.exe           00000000019201FA  rearrange_mp_rear         108  rearrange.F90.incesm.exe           0000000001964589  piodarray_mp_read         390  piodarray.F90.incesm.exe           00000000009767BE  ncdio_atm_mp_infl         287  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002B7F586BED5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node808: task 1: Exited with exit code 174 From there, the code stops doing anything but fails to exit, and hence my earlier confusion about what was actually happening. I realize that this error looks identical to this one: https://bb.cgd.ucar.edu/cesm-122-segmentation-fault-when-running-model, but I tried everything listed on that post and nothing worked. The modification to the source code in diags_on_lat_aux_grid.F90 did not help, nor did setting the  -heap_arrays, nor  -O2 . I already uncommented limit coredumpsize 1000000 and limit stacksize unlimited. 

 

[jmnicklas@gmail_com]

Hello,I have succesfully run compsets A and C. I am now attempting to run compset B at res f45_g37. However, I keep getting this error:Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L2 6_c060608.nc       65536 Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002B7F584927E0  Unknown               Unknown  Unknowncesm.exe           0000000001B43520  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771EA66  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771E932  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57736FEA  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57737198  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5773F5D5  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F576A087A  Unknown               Unknown  Unknownlibmpifort.so.12   00002B7F57DE26E2  Unknown               Unknown  Unknowncesm.exe           000000000199CCCA  pio_spmd_utils_mp         297  pio_spmd_utils.F90.incesm.exe           000000000198B5AC  box_rearrange_mp_         571  box_rearrange.F90.incesm.exe           00000000019201FA  rearrange_mp_rear         108  rearrange.F90.incesm.exe           0000000001964589  piodarray_mp_read         390  piodarray.F90.incesm.exe           00000000009767BE  ncdio_atm_mp_infl         287  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002B7F586BED5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node808: task 1: Exited with exit code 174 From there, the code stops doing anything but fails to exit, and hence my earlier confusion about what was actually happening. I realize that this error looks identical to this one: https://bb.cgd.ucar.edu/cesm-122-segmentation-fault-when-running-model, but I tried everything listed on that post and nothing worked. The modification to the source code in diags_on_lat_aux_grid.F90 did not help, nor did setting the  -heap_arrays, nor  -O2 . I already uncommented limit coredumpsize 1000000 and limit stacksize unlimited. 

 

[jmnicklas@gmail_com]

Hello,I have succesfully run compsets A and C. I am now attempting to run compset B at res f45_g37. However, I keep getting this error:Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L2 6_c060608.nc       65536 Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002B7F584927E0  Unknown               Unknown  Unknowncesm.exe           0000000001B43520  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771EA66  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771E932  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57736FEA  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57737198  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5773F5D5  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F576A087A  Unknown               Unknown  Unknownlibmpifort.so.12   00002B7F57DE26E2  Unknown               Unknown  Unknowncesm.exe           000000000199CCCA  pio_spmd_utils_mp         297  pio_spmd_utils.F90.incesm.exe           000000000198B5AC  box_rearrange_mp_         571  box_rearrange.F90.incesm.exe           00000000019201FA  rearrange_mp_rear         108  rearrange.F90.incesm.exe           0000000001964589  piodarray_mp_read         390  piodarray.F90.incesm.exe           00000000009767BE  ncdio_atm_mp_infl         287  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002B7F586BED5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node808: task 1: Exited with exit code 174 From there, the code stops doing anything but fails to exit, and hence my earlier confusion about what was actually happening. I realize that this error looks identical to this one: https://bb.cgd.ucar.edu/cesm-122-segmentation-fault-when-running-model, but I tried everything listed on that post and nothing worked. The modification to the source code in diags_on_lat_aux_grid.F90 did not help, nor did setting the  -heap_arrays, nor  -O2 . I already uncommented limit coredumpsize 1000000 and limit stacksize unlimited. 

 

[jmnicklas@gmail_com]

Hello,I have succesfully run compsets A and C. I am now attempting to run compset B at res f45_g37. However, I keep getting this error:Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L2 6_c060608.nc       65536 Opened existing file /users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002B7F584927E0  Unknown               Unknown  Unknowncesm.exe           0000000001B43520  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771EA66  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5771E932  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57736FEA  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F57737198  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F5773F5D5  Unknown               Unknown  Unknownlibmpi.so.12       00002B7F576A087A  Unknown               Unknown  Unknownlibmpifort.so.12   00002B7F57DE26E2  Unknown               Unknown  Unknowncesm.exe           000000000199CCCA  pio_spmd_utils_mp         297  pio_spmd_utils.F90.incesm.exe           000000000198B5AC  box_rearrange_mp_         571  box_rearrange.F90.incesm.exe           00000000019201FA  rearrange_mp_rear         108  rearrange.F90.incesm.exe           0000000001964589  piodarray_mp_read         390  piodarray.F90.incesm.exe           00000000009767BE  ncdio_atm_mp_infl         287  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002B7F586BED5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node808: task 1: Exited with exit code 174 From there, the code stops doing anything but fails to exit, and hence my earlier confusion about what was actually happening. I realize that this error looks identical to this one: https://bb.cgd.ucar.edu/cesm-122-segmentation-fault-when-running-model, but I tried everything listed on that post and nothing worked. The modification to the source code in diags_on_lat_aux_grid.F90 did not help, nor did setting the  -heap_arrays, nor  -O2 . I already uncommented limit coredumpsize 1000000 and limit stacksize unlimited. 

 

[jmnicklas@gmail_com]

Alternatively, the middle part of the error messages spit out (when DEBUG is FALSE):/users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002BA33A4C97E0  Unknown               Unknown  Unknowncesm.exe           0000000001B41A5B  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799846  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799826  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339786577  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339783372  Unknown               Unknown  Unknownlibmpi.so.12       00002BA33978238D  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DECA4  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DEA23  Unknown               Unknown  Unknownlibmpifort.so.12   00002BA339E1982D  Unknown               Unknown  Unknowncesm.exe           000000000199E547  pio_spmd_utils_mp         345  pio_spmd_utils.F90.incesm.exe           0000000001997D87  box_rearrange_mp_        1372  box_rearrange.F90.incesm.exe           000000000198F57D  box_rearrange_mp_        1029  box_rearrange.F90.incesm.exe           0000000001885D19  piolib_mod_mp_pio        1166  piolib_mod.F90cesm.exe           00000000004DE341  cam_pio_utils_mp_         449  cam_pio_utils.F90cesm.exe           0000000000976715  ncdio_atm_mp_infl         279  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002BA33A6F5D5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node655: task 1: Exited with exit code 174

 

[jmnicklas@gmail_com]

Alternatively, the middle part of the error messages spit out (when DEBUG is FALSE):/users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002BA33A4C97E0  Unknown               Unknown  Unknowncesm.exe           0000000001B41A5B  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799846  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799826  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339786577  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339783372  Unknown               Unknown  Unknownlibmpi.so.12       00002BA33978238D  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DECA4  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DEA23  Unknown               Unknown  Unknownlibmpifort.so.12   00002BA339E1982D  Unknown               Unknown  Unknowncesm.exe           000000000199E547  pio_spmd_utils_mp         345  pio_spmd_utils.F90.incesm.exe           0000000001997D87  box_rearrange_mp_        1372  box_rearrange.F90.incesm.exe           000000000198F57D  box_rearrange_mp_        1029  box_rearrange.F90.incesm.exe           0000000001885D19  piolib_mod_mp_pio        1166  piolib_mod.F90cesm.exe           00000000004DE341  cam_pio_utils_mp_         449  cam_pio_utils.F90cesm.exe           0000000000976715  ncdio_atm_mp_infl         279  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002BA33A6F5D5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node655: task 1: Exited with exit code 174

 

[jmnicklas@gmail_com]

Alternatively, the middle part of the error messages spit out (when DEBUG is FALSE):/users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002BA33A4C97E0  Unknown               Unknown  Unknowncesm.exe           0000000001B41A5B  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799846  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799826  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339786577  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339783372  Unknown               Unknown  Unknownlibmpi.so.12       00002BA33978238D  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DECA4  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DEA23  Unknown               Unknown  Unknownlibmpifort.so.12   00002BA339E1982D  Unknown               Unknown  Unknowncesm.exe           000000000199E547  pio_spmd_utils_mp         345  pio_spmd_utils.F90.incesm.exe           0000000001997D87  box_rearrange_mp_        1372  box_rearrange.F90.incesm.exe           000000000198F57D  box_rearrange_mp_        1029  box_rearrange.F90.incesm.exe           0000000001885D19  piolib_mod_mp_pio        1166  piolib_mod.F90cesm.exe           00000000004DE341  cam_pio_utils_mp_         449  cam_pio_utils.F90cesm.exe           0000000000976715  ncdio_atm_mp_infl         279  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002BA33A6F5D5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node655: task 1: Exited with exit code 174

 

[jmnicklas@gmail_com]

Alternatively, the middle part of the error messages spit out (when DEBUG is FALSE):/users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002BA33A4C97E0  Unknown               Unknown  Unknowncesm.exe           0000000001B41A5B  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799846  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799826  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339786577  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339783372  Unknown               Unknown  Unknownlibmpi.so.12       00002BA33978238D  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DECA4  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DEA23  Unknown               Unknown  Unknownlibmpifort.so.12   00002BA339E1982D  Unknown               Unknown  Unknowncesm.exe           000000000199E547  pio_spmd_utils_mp         345  pio_spmd_utils.F90.incesm.exe           0000000001997D87  box_rearrange_mp_        1372  box_rearrange.F90.incesm.exe           000000000198F57D  box_rearrange_mp_        1029  box_rearrange.F90.incesm.exe           0000000001885D19  piolib_mod_mp_pio        1166  piolib_mod.F90cesm.exe           00000000004DE341  cam_pio_utils_mp_         449  cam_pio_utils.F90cesm.exe           0000000000976715  ncdio_atm_mp_infl         279  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002BA33A6F5D5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node655: task 1: Exited with exit code 174

 

[jmnicklas@gmail_com]

Alternatively, the middle part of the error messages spit out (when DEBUG is FALSE):/users/jnickla1/data/cesm1_2_2/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c0 50520.nc      131072 NetCDF: Invalid dimension ID or nameforrtl: severe (174): SIGSEGV, segmentation fault occurredImage              PC                Routine            Line        Sourcecesm.exe           0000000001B37969  Unknown               Unknown  Unknowncesm.exe           0000000001B362E0  Unknown               Unknown  Unknowncesm.exe           0000000001AD34C2  Unknown               Unknown  Unknowncesm.exe           0000000001A5F373  Unknown               Unknown  Unknowncesm.exe           0000000001A68D8B  Unknown               Unknown  Unknownlibpthread.so.0    00002BA33A4C97E0  Unknown               Unknown  Unknowncesm.exe           0000000001B41A5B  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799846  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339799826  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339786577  Unknown               Unknown  Unknownlibmpi.so.12       00002BA339783372  Unknown               Unknown  Unknownlibmpi.so.12       00002BA33978238D  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DECA4  Unknown               Unknown  Unknownlibmpi.so.12       00002BA3396DEA23  Unknown               Unknown  Unknownlibmpifort.so.12   00002BA339E1982D  Unknown               Unknown  Unknowncesm.exe           000000000199E547  pio_spmd_utils_mp         345  pio_spmd_utils.F90.incesm.exe           0000000001997D87  box_rearrange_mp_        1372  box_rearrange.F90.incesm.exe           000000000198F57D  box_rearrange_mp_        1029  box_rearrange.F90.incesm.exe           0000000001885D19  piolib_mod_mp_pio        1166  piolib_mod.F90cesm.exe           00000000004DE341  cam_pio_utils_mp_         449  cam_pio_utils.F90cesm.exe           0000000000976715  ncdio_atm_mp_infl         279  ncdio_atm.F90cesm.exe           00000000008799AA  inidat_mp_read_in         230  inidat.F90cesm.exe           0000000000604732  startup_initialco          54  startup_initialconds.F90cesm.exe           00000000005158A4  inital_mp_cam_ini          51  inital.F90cesm.exe           00000000004A71A1  cam_comp_mp_cam_i         164  cam_comp.F90cesm.exe           00000000004A30C0  atm_comp_mct_mp_a         276  atm_comp_mct.F90cesm.exe           0000000000432DFB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90cesm.exe           00000000004355F6  MAIN__                     90  ccsm_driver.F90cesm.exe           0000000000412066  Unknown               Unknown  Unknownlibc.so.6          00002BA33A6F5D5D  Unknown               Unknown  Unknowncesm.exe           0000000000411ED9  Unknown               Unknown  Unknownsrun: error: node655: task 1: Exited with exit code 174

 

[jmnicklas@gmail_com]

This issue has since been resolved. I believe that it may have been unique to my particular machine. Also, part of the problem could have been that I had insufficient permissions to install CESM in the appropriate runtime directories. Please contact me via email and I will be happy to refer you to the person at our institution who solved this.

 

[jmnicklas@gmail_com]

This issue has since been resolved. I believe that it may have been unique to my particular machine. Also, part of the problem could have been that I had insufficient permissions to install CESM in the appropriate runtime directories. Please contact me via email and I will be happy to refer you to the person at our institution who solved this.

 

[jmnicklas@gmail_com]

This issue has since been resolved. I believe that it may have been unique to my particular machine. Also, part of the problem could have been that I had insufficient permissions to install CESM in the appropriate runtime directories. Please contact me via email and I will be happy to refer you to the person at our institution who solved this.

 

[jmnicklas@gmail_com]

This issue has since been resolved. I believe that it may have been unique to my particular machine. Also, part of the problem could have been that I had insufficient permissions to install CESM in the appropriate runtime directories. Please contact me via email and I will be happy to refer you to the person at our institution who solved this.

 

[jmnicklas@gmail_com]

This issue has since been resolved. I believe that it may have been unique to my particular machine. Also, part of the problem could have been that I had insufficient permissions to install CESM in the appropriate runtime directories. Please contact me via email and I will be happy to refer you to the person at our institution who solved this.