What version of the code are you using?
release-cesm2.1.5-0-g7a6c5b0 (describe version attached)
Have you made any changes to files in the source tree?
No changes to the source tree.
Describe every step you took leading up to the problem:
/work/07644/oxygen/ls6/cesm-2.1.5/cime/scripts/create_newcase \
--case "${CASE}" \
--compset FHIST_BGC \
--res f09_f09_mg17 \
--machine ls6
./xmlchange NTASKS=-8
./xmlchange DOUT_S=TRUE
./xmlchange STOP_N=5
./xmlchange RESUBMIT=0
./xmlchange RUN_REFCASE=b.e21.B1850.f09_g17.CMIP6-piControl.001
./xmlchange RUN_REFDATE=1231-01-01
./xmlchange RUN_TYPE=hybrid
./xmlchange GET_REFCASE=TRUE
./xmlchange RUN_STARTDATE=1850-01-01
./xmlchange STOP_OPTION=ndays
./xmlchange GMAKE_J=64
./xmlchange SSTICE_YEAR_ALIGN=1
./xmlchange SSTICE_YEAR_START=0
./xmlchange SSTICE_YEAR_END=0
./xmlchange SSTICE_DATA_FILENAME="${SCRATCH}/cesm-input-data/SSTICE/sstice_cmip6_pi-Control_clim_40101_200012_diddled.nc"
./case.setup
./case.build --clean-all
./case.build
./case.submit
Namelists are also attached.
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
This is a port for TACC's Lonestar6. The config xmls are attached.
Describe your problem or question:
When scaling this case to multiple nodes (LS6 has 128-cores per node), CESM encounters a PIO error during initialization and crashes. This particular run failed during LND but a few runs failed during ATM as well:
Here is what I have tried that did not work and produced the same error (sometimes at different stages in the run):
release-cesm2.1.5-0-g7a6c5b0 (describe version attached)
Have you made any changes to files in the source tree?
No changes to the source tree.
Describe every step you took leading up to the problem:
/work/07644/oxygen/ls6/cesm-2.1.5/cime/scripts/create_newcase \
--case "${CASE}" \
--compset FHIST_BGC \
--res f09_f09_mg17 \
--machine ls6
./xmlchange NTASKS=-8
./xmlchange DOUT_S=TRUE
./xmlchange STOP_N=5
./xmlchange RESUBMIT=0
./xmlchange RUN_REFCASE=b.e21.B1850.f09_g17.CMIP6-piControl.001
./xmlchange RUN_REFDATE=1231-01-01
./xmlchange RUN_TYPE=hybrid
./xmlchange GET_REFCASE=TRUE
./xmlchange RUN_STARTDATE=1850-01-01
./xmlchange STOP_OPTION=ndays
./xmlchange GMAKE_J=64
./xmlchange SSTICE_YEAR_ALIGN=1
./xmlchange SSTICE_YEAR_START=0
./xmlchange SSTICE_YEAR_END=0
./xmlchange SSTICE_DATA_FILENAME="${SCRATCH}/cesm-input-data/SSTICE/sstice_cmip6_pi-Control_clim_40101_200012_diddled.nc"
./case.setup
./case.build --clean-all
./case.build
./case.submit
Namelists are also attached.
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
This is a port for TACC's Lonestar6. The config xmls are attached.
Describe your problem or question:
When scaling this case to multiple nodes (LS6 has 128-cores per node), CESM encounters a PIO error during initialization and crashes. This particular run failed during LND but a few runs failed during ATM as well:
Here is what I have tried that did not work and produced the same error (sometimes at different stages in the run):
- Tested on 1 node, 2 nodes, 4 nodes, and (attached) 8 nodes. I did test runs without namelist modifications (except for use_init_interp in user_nl_clm) and they passed once, but not twice. After rebuilding and setting namelists, only 1-node worked with 2 simulated years per day. However, one iteration of the 1-node run seemingly failed when writing out the restart files.
- Switching PIO_TYPENAME from the default 'pnetcdf' to 'netcdf' and 'netcdf4p'. Only 'netcdf' worked but performance tanked on the 1-node from 2 SYPD to 0.8 SYPD.
- Lowering the PIO_STRIDE
- Adjusting the PE layout to put LND on a single node
- Switching to PIO2
- Adjusting the disk striping in the run directory from default 72 targets to 8, 4, and 1 target, changing the chunksize as well.
- Adding I_MPI_IO_HINTS, I_MPI_EXTRA_FILESYSTEM, and I_MPI_EXTRA_FILESYSTEM_LIST
- Adding an explicit romio_hints file to the run directory
- Temporarily disabling grid interpolation in the LND model and copying finidat from successful case manually
- Re-downloading the source code
- Building/running in an apptainer container with GNU, AOCC compilers and switching to MVAPICH2 instead of IMPI
