Error in CLM interpolation: in initInterpMultilevelInterp.F90 at line 465

[xieyk]

Hi there,
I am running CESM2.1.3, B1850, f09_g17. The error message is " array not monotonically increasingERROR in initInterpMultilevelInterp.F90 at line 465".
I think it may be caused by the inconsistence between the CLM code "clm/src/init_interp/initInterpMultilevelInterp.F90" and the default inputdata "b.e20.B1850.f09_g17.pi_control.all.299_merge_v3.clm2.r.0134-01-01-00000.nc".
However, I do not know how to fix it.
Hope the professional in CLM could provide some help.
Thanks in advance.
Sincerely,
Yongkun XIE

Here is the detailed information:
Processes,
create_213_case --case B1850_test_1d --compset B1850 --res f09_g17
modfiy env_mach_pes.xml
./case.setup
modify env_build.xml "change DEBUG to TRUE"
modify env_run.xml
./case.build --skip-provenance-check
./.case.run
Attached please find the required files.
By the way, the model can successfully run for case FHIST. I also found a similar thread,

CESM2.1.3 - array not monotonically increasing in B1850

Error running B1850 with CESM2.1.3 - array not monotonically increasing in initInterpMultilevelInterp.F90 | DiscussCESM Forums (ucar.edu) Hi,everyone. This is the original post, but no one replied so I put it here. In order to exclude errors caused by the gcc version,I installed gcc 5.4.0 at...

bb.cgd.ucar.edu

 

[oleson]

I was not able to reproduce this error on our machine. The simulation ran successfully for 5 days. Here are the steps I took:

./create_newcase --case cesm213_B1850_test --compset B1850 --res f09_g17
./case.setup
./preview_namelists (this indicated that finidat = b.e20.B1850.f09_g17.pi_control.all.299_merge_v3.clm2.r.0134-01-01-00000.nc and use_init_interp=.true.)
./case.build
./case.submit

So I'm not sure what to suggest...
Do you have a successful port of cesm2.1.3 to your machine?

 

[xieyk]

The port is successful because I have finished running the compset FHIST from 1979-2014.

 

[oleson]

Since I can't reproduce the problem here, we'll need more specific information about how you've setup your case.
In particular, please provide more information about what you mean by "modify env_mach_pes.xml" and "modify env_run.xml"
I don't see any files attached.
It would be best if you provided all of the information requested here:

Information to include in help requests

When you are posting to the forums because you want help with an issue, please include as much of the following information as possible. Before submitting a help request, please check to see if your question is already answered: Search the forums for similar issues Check the CIME...

bb.cgd.ucar.edu

 

[xieyk]

Here I uploaded the required files including,
version_info.txt （this may be not helpful because git on our machine is not allowed to access overseas networks)
env_mach_pes.xml
env_run.xml
lnd_in
initInterpMultilevelInterp.F90
cesm.log.210812-235313
lnd.log.210812-235313
atm.log.210812-235313
cpl.log.210812-235313
Macros.make

 

[oleson]

Thanks for the information. Your setup looks the same as mine and I still can't replicate the error.
It's possible that your finidat file is corrupted:

b.e20.B1850.f09_g17.pi_control.all.299_merge_v3.clm2.r.0134-01-01-00000.nc

What's it's size? The one I'm using has this size (about 32 GB):

3204796892 Apr 3 2019 /glade/p/cesmdata/cseg/inputdata/cesm2_init/b.e20.B1850.f09_g17.pi_control.all.299_merge_v3/0134-01-01/b.e20.B1850.f09_g17.pi_control.all.299_merge_v3.clm2.r.0134-01-01-00000.nc

 

[xieyk]

Thanks. The size of my finidat file is the same as yours.
"3204796892 8月 11 18:07 inputdata/cesm2_init/b.e20.B1850.f09_g17.pi_control.all.299_merge_v3/0134-01-01/b.e20.B1850.f09_g17.pi_control.all.299_merge_v3.clm2.r.0134-01-01-00000.nc"
I downloaded it by using wget from https://svn-ccsm-inputdata.cgd.ucar...9_g17.pi_control.all.299_merge_v3/0134-01-01/

Could you compare whether the code "initInterpMultilevelInterp.F90" that I upload last time is exactly the same as yours?

 

[oleson]

Yes, it is exactly the same as mine.

 

[xieyk]

Hi Keith,
When I changed to another pe layout, the error became
" spvals or NaNs found in coordinate array where level_class /= ispval; this is c
urrently unhandled ERROR in initInterpMultilevelInterp.F90 at line 435"
Does this provide a new hint?

Additionally, I will try another version if I cannot fix this problem. Do you have suggestions on version, 2.2, or 2.1.0 - 2?

 

[oleson]

No, I don't know why the error would change with a different PE layout.
I've noticed that your PE layout is quite different from the one that comes out of the box. I've attached the env_mach_pes.xml that I've used. You could try that if possible on your machine. Although I don't know why it would cause these errors.
You could also try cesm2.2.0, which is the development version of CESM.

 

[xieyk]

Many thanks. But what you attached is env_run.xml rather than env_mach_pes.xml.

 

[oleson]

Apologies! Attached is the correct file.

 

[xieyk]

Hi Keith,
Changing to version 2.2.0 cannot solve this problem. Could you provide me the information about your Fortran compiler? I want to check whether it is a problem with the compiler.
Thanks.

 

[oleson]

I think it is intel/17.0.1.
I've attached my env_mach_specific.xml and software_environment.txt from my case that attempts to replicate your case, in case that helps.