BHIST stopped at "setting up interpolators for multi-level variables"

[Jeline]

Hi everyone,
I am running a BHIST case on CESM 2.1.5 using surfdata and land use time series data that I generated myself using the ctsm52landusedatatool-main and mksurfdata_map tools. I have ensured that these two datasets have the same variables and dimensions as the official data. However, after submitting the case, I encountered an error.

The cesm.log file stops at

Abort(697358) on node 193 (rank 193 in comm 0): Fatal error in internal_Wait: Message truncated,
error stack: internal_Wait(89): MPI_Wait(request=0x7ffdd5f93c34, status=0x9b83fe0) failed
MPIR_Wait (911)...: (unknown)(): Message truncated

The cpl.log file stops at

(seq_mct_drv) : Initialize each component: atm, lnd, rof, ocn, ice, glc, wav, esp
(component_init_cc:mct) : Initialize component atm
(component_init_cc:mct) : Initialize component lnd”

The lnd.log file stops at

finding minimum distance for gridcells
calling set_subgrid_info for gridcell for input
calling set_subgrid_info for gridcell for output
calling set_mindist for gridcell
setting up interpolators for multi-level variables

It looks like an issue between finidat and fsurdat? But I don’t know why this is happening. I’ve attached the cesm.log, cpl.log, and lnd.log files. If anyone can help resolve this, I’d be very grateful.

Here are the commands I used during the case compilation process:

cd cesm2.1.5/cime/scripts

./create_newcase --case $HOME/cases/BHIST0418 --compset BHIST --res f09_g17 --mach cli01-17-6 --compiler intel --mpilib mpich --run-unsupported
cd $HOME/cases/BHIST0418
./xmlchange NTASKS=1680,NTHRDS=1,ROOTPE=0
./xmlchange STOP_OPTION=nyears,STOP_N=10
./xmlchange REST_OPTION=nyears,REST_N=2
./case.setup
# user_nl_clm
fsurdat = '/home/xibnlkjdxue/Pengshzh/mksurfdata_map_output/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr1850_c260403.nc'
flanduse_timeseries = '/home/xibnlkjdxue/Pengshzh/mksurfdata_map_output/landuse.timeseries_0.9x1.25_hist_78pfts_CMIP6_simyr1850-2000_c260403.nc'
use_init_interp = .true.
./case.build
dsub -s "BHIST0418.sh" # I'm using the university's own supercomputing platform; the ./case.submit command and other queue-related commands are included in this script.

 

[slevis]

I looked more carefully at your cesm.log and found various errors. For example:
```
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/ozone_strataero/ozone_stratae
ro_WACCM_L70_zm5day_18500101-20150103_CMIP6ensAvg_c180923.nc 196608
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/waccm/lb/LBC_1750-2015_CMIP6_Glob
AnnAvg_c180926.nc 196608
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
[...]
NetCDF: Attribute not found
[...]
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/chem/emis/CMIP6_emissions_175
0_2015/emissions-cmip6_DMS_bb_surface_1750-2015_0.9x1.25_c20170322.nc
393216
NetCDF: Attribute not found
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/chem/emis/CMIP6_emissions_175
0_2015/emissions-cmip6_DMS_bb_surface_1750-2015_0.9x1.25_c20170322.nc
393216
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Invalid dimension ID or name
```
I do not have more information than what I see in your cesm.log, but this suggests problems in your custom data.

 

[Jeline]

I looked more carefully at your cesm.log and found various errors. For example:
```
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/ozone_strataero/ozone_stratae
ro_WACCM_L70_zm5day_18500101-20150103_CMIP6ensAvg_c180923.nc 196608
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/waccm/lb/LBC_1750-2015_CMIP6_Glob
AnnAvg_c180926.nc 196608
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Variable not found
[...]
NetCDF: Attribute not found
[...]
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/chem/emis/CMIP6_emissions_175
0_2015/emissions-cmip6_DMS_bb_surface_1750-2015_0.9x1.25_c20170322.nc
393216
NetCDF: Attribute not found
Opened existing file
/home/xibnlkjdxue/Pengshzh/CESM/inputdata/atm/cam/chem/emis/CMIP6_emissions_175
0_2015/emissions-cmip6_DMS_bb_surface_1750-2015_0.9x1.25_c20170322.nc
393216
NetCDF: Variable not found
NetCDF: Variable not found
NetCDF: Invalid dimension ID or name
```
I do not have more information than what I see in your cesm.log, but this suggests problems in your custom data.

Hi slevis, thank you so much for replying! In the logs from I compsrt runs I’ve successfully completed in the past, there are also entries similar to “NetCDF: Variable not found” and “NetCDF: Invalid dimension ID or name.” Therefore, I still don’t know what the problem is with my custom surface data; structurally, it looks exactly the same as the official surface data.

However, when I tried running BHIST with the default data in CESM 2.1.5, I received the error: “UrbanInput: parameter nlevurb=10 does not equal input dataset nlevurb= 5 ENDRUN: ERROR: ERROR in UrbanParamsType.F90 at line 479” And I was unable to find any suitable official surface data with nlevurb=10 at - Revision 70792: /trunk/inputdata.

I still have three questions:
Why does the out-of-the-box BHIST default input data produce this mismatch error? Where should I look for the correct surface data? How can I determine the correct direction to modify my custom surface data?

 

[slevis]

A workaround to try:
In your components/clm/src/main/clm_varpar.F90, change nlevurb from 5 to 10.
This mismatch may be happening because you're using an updated fsurdat file with nlevurb = 10. I'm surprised that nobody else encountered the error, but maybe cesm2.1.5 is not widely used...