Problem with regional CTSM5.3 simulation

[johanna_teresa]

Hello,

I am trying to run a regional simulation over Switzerland with ctsm5.3, which I had set up for a previous version of the model. The simulation keeps crashing after a few seconds, and the error logs point towards line 409 in CTSM/src/main/ncdio_pio.F90, which if I understand it correctly indicates a mismatch in dimension of some sort. In order to eliminate the possibility of a problem with my own previously generated (and now updated for version 5.3) surface data file I have re-created a test file over a similar region with:

./subset_data region --lat1 45.5 --lat2 48.5 --lon1 5.5 --lon2 10.5 --reg CH_in --create-surface --surf-year 2000 --create-mesh --create-domain --create-user-mods --inputdata-dir /capstor/scratch/cscs/jmalle/ctsm5_3/input --outdir /capstor/scratch/cscs/jmalle/ctsm5_3/input/regional_cases

and then used the generated surface datafile and mesh file.

Do you happen to know what the source for this error could be?

Logs and surfacedate/mesh files are attached, as are machine port related files. Steps I have taken during file creation are below.

Best,
Johanna

What version of the code are you using?
ctsm5.3.021

Describe every step you took leading up to the problem:
COMPILER=intel
COMPSET=I2000Clm50Sp
RES=CLM_USRDAT
./create_newcase --case ${CASEDIRPATH}${CASE} --compiler ${COMPILER} --compset ${COMPSET} --res ${RES} --mach eiger --driver nuopc --mpilib mpich --pecount 128 --run-unsupported
./xmlchange COMP_INTERFACE='nuopc'
./xmlchange ATM_DOMAIN_MESH='/capstor/scratch/cscs/jmalle/ctsm5_3/input/regional_cases/domain.lnd.fv0.9x1.25_gx1v7_CH_in_c250220_ESMF_UNSTRUCTURED_MESH.nc'
./xmlchange LND_DOMAIN_MESH='/capstor/scratch/cscs/jmalle/ctsm5_3/input/regional_cases/domain.lnd.fv0.9x1.25_gx1v7_CH_in_c250220_ESMF_UNSTRUCTURED_MESH.nc'
./xmlchange MASK_MESH='/capstor/scratch/cscs/jmalle/ctsm5_3/input/regional_cases/domain.lnd.fv0.9x1.25_gx1v7_CH_in_c250220_ESMF_UNSTRUCTURED_MESH.nc'
./xmlchange ATM_NX=4
./xmlchange LND_NX=4
./xmlchange ATM_NY=3
./xmlchange LND_NY=3

./case.setup
./preview_namelists
./case.build

./xmlchange ATM_NCPL=8
./xmlchange RUN_STARTDATE=$start_date
./xmlchange STOP_OPTION=$stop_opt
./xmlchange STOP_N=$stop_n
./xmlchange DATM_YR_ALIGN=$yr_start
./xmlchange DATM_YR_START=$yr_start
./xmlchange DATM_YR_END=$yr_end

./xmlchange JOB_WALLCLOCK_TIME=$time --subgroup case.run
./xmlchange JOB_QUEUE=$job_queue --force

./case.submit

 

[oleson]

I see this in your lnd log:

Reading initial conditions from file ./init_generated_files/finidat_interp_dest.nc
(GETFIL): attempting to find local file
finidat_interp_dest.nc
(GETFIL): using ./init_generated_files/finidat_interp_dest.nc
Reading restart dataset: ./init_generated_files/finidat_interp_dest.nc
check_dim_size ERROR: mismatch of input dimension 12
with expected value 221 for variable gridcell
Did you mean to set use_init_interp = .true. in user_nl_clm?
(Setting use_init_interp = .true. is needed when doing a
transient run using an initial conditions file from a non-transient run,
or a non-transient run using an initial conditions file from a transient run,
or when running a resolution or configuration that differs from the initial conditions.)

which indicates that your surface dataset has 12 (4x3?) gridcells, while the initial file that is being used (./init_generated_files/finidat_interp_dest.nc) has 221 gridcells.
That file is usually generated when use_init_interp is set to .true.
Did that file come from a previous simulation maybe? What is finidat set to in your lnd_in?

 

[johanna_teresa]

Explicitly setting use_init_interp = .false. solved this issue, thanks for the hint, very helpful.

I can now run with my own atmospheric dataset, but as soon as I try to subset global atmospheric data the model crashes with a "forrtl: severe (71): integer divide by zero" error. Basically the land model initializes successfully, but right at the first timestep it crashes (see attached log files). Do you happen to have an idea what this could be related to?

 

[oleson]

When you say "subset global atmospheric data", have you actually subset the data, e.g., "/capstor/scratch/cscs/jmalle/ctsm5_3/input/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.2013-01.nc". Is the mesh file consistent with that?
Another thought is that it seems to be dying in the accumulator module. I wonder if there is something wrong/inconsistent with the accumulator variables in the initial file you are using. Have you tried running with a cold start?

 

[johanna_teresa]

I have tried running with a cold start, which still gives me the same error.

I have not actually subset the atmospheric data myself, as it seems this does not work yet (if I understood this issue here correctly?). But theoretically I should still be able to run with global datm files correct?

I am a bit confused regarding the meshes and LND_NX/LND_NY and ATM_NX/ATM_NY settings, so it could very well be that this is where my errors are coming from. If I am aiming to run with global datm files, do I need to set my ATM_NX/ATM_NY to match the LND_NX/LND_NY values? There seems to be a Mesh independently set in the datm stream files for the global, in my case GSWP3 data. Does this have to match the setting of ATM_DOMAIN_MESH? And what about the setting of the MASK_MESH? I have played around with a few different configurations, but am still getting the same error.

Thanks a lot for any hints!

 

[oleson]

LND_NX/LND_NY and ATM_NX/ATM_NY should be set to the dimensions of your surface dataset.
ATM_DOMAIN_MESH and LND_DOMAIN_MESH should be set to the mesh file that corresponds to the surface dataset. MASK_MESH can also be set to this.
To use the default global GSWP3 forcing data, you wouldn't make any changes to user_nl_datm and user_nl_datm_streams. The mesh file should show up as clmforc.GSWP3.c2011.0.5x0.5.TPQWL.SCRIP.210520_ESMFmesh.nc

 

[johanna_teresa]

Thanks a lot for the clarification. This is exactly what I was doing, also the mesh file shows up correctly in the datm_streams.