"confirm_monotonically_increasing ERROR" in running BHIST

[fkc000@163_com]

Hello, everyone:
I've already successfully built the case BHIST with a compset of HIST_CAM60_CLM50%BGC-CROP_CICE_POP2%ECO%ABIO-DIC_MOSART_CISM2%NOEVOLVE_WW3_BGC%BDRD and resolution of f09_g17_gl4, but I have met some problem in submitting the job.
The intel and mpi intel compiler version is 2019.5.281, the netecdf version is 4.4.1

Here's the main error message:
proc= 1023 clump no = 1 clump id= 1024
beg patch = 883373 end patch = 884206
total patches per clump = 834
proc= 1023 clump no = 1 clump id= 1024
beg cohort = 20994 end cohort = 21013
total cohorts per clump = 20
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Variable not found
NetCDF: Variable not found
confirm_monotonically_increasing ERROR: array not monotonically increasing:
2.121995791459338E-314 2.121995791459338E-314
confirm_monotonically_increasing ERROR: array not monotonically increasing:
0.000000000000000E+000 0.000000000000000E+000
ENDRUN:
ERROR:
confirm_monotonically_increasing ERROR: array not monotonically increasingERROR
in initInterpMultilevelInterp.F90 at line 465
confirm_monotonically_increasing ERROR: array not monotonically increasing:
0.000000000000000E+000 0.000000000000000E+000
ENDRUN:

I've seen other posts about the problem, but their solution(which is updating the compiler) didn't seem work for me.
Log files and lnd_in file is attached.

 

[katec]

Hi there, can you tell me what machine you are running this on?

 

[fkc000@163_com]

Hi there, can you tell me what machine you are running this on?

It's on our institute's server(not on cheyenne or other NCAR's platform), and I can run CESM1.2.2 successfully on the server with the same compiler and library.

 

[katec]

Ok, this still looks to me like a CESM2 porting issue. There were a lot of infrastructure changes between CESM 1.2.2 and CESM2, so getting the newer model running may take different steps than the older version. I'm going to move this post to the Porting board for help debugging that.

 

[jedwards]

I don't see anything obvious - your netcdf library is pretty old, you can try updating that.
Check the md5 sum of file
/public/home/feikece/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x
1.25_gx1v7_simyr2000_c200728.nc
it should be: 87749130b1523befcdb85747ab8da62f

 

[fkc000@163_com]

Ok, this still looks to me like a CESM2 porting issue. There were a lot of infrastructure changes between CESM 1.2.2 and CESM2, so getting the newer model running may take different steps than the older version. I'm going to move this post to the Porting board for help debugging that.

The error seems to be related to "Interpolate finidat onto new template file", and I've tried another initial file but still didn't work.

 

[fkc000@163_com]

I don't see anything obvious - your netcdf library is pretty old, you can try updating that.
Check the md5 sum of file
/public/home/feikece/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x
1.25_gx1v7_simyr2000_c200728.nc
it should be: 87749130b1523befcdb85747ab8da62f

I've checked, the md5 sum is correct. As for the netcdf library, I read from CESM official website that the requirement is "netcdf need to be 4.3 or newer" while my netcdf lib is 4.4.1. Could that still be the problem? Anyway, I'll try updating netcdf. Thanks a lot.