the internal domain decomposition error for the land model

[xgao304]

I am running the cesm2 with intel compiler and mpi and netcdf on our own machine. I am able to run the model with 4 nodes (128 processors), but with 8 nodes I will get the error related to the internal domain decomposition for the land model. Here is the information for my case:
./create_newcase --case ../../../cases/test2000 --compset I2000Clm50SpGs --res hcru_hcru --machine svante --compiler intel --run-unsupportedI attach the cesm log file for error message.Any help is appreciated.Xiang

 

[jedwards]

NO attachement found.  What did you do to switch from 4 to 8 nodes?

 

[xgao304]

create a totally new case when I switch from 4 to 8. See the attachment.

 

[xgao304]

try again.

 

[xgao304]

I am wondering if you have any idea about this problem based on the attached log file. Any update is really appreciated.

 

[jedwards]

I have been unable to reproduce your problem locally.   What version of netcdf are you using?   Try changing the PIO_STRIDE value:./xmlchange PIO_STRIDE=32 (16, 8, ...) 

 

[xgao304]

Here is the information:intel/2017.0.1netcdf/4openmpi/2.0.2I will give it a try with PIO_STRIDE and let you know.Thanks.  

 

[xgao304]

I just found out that my default PIO_STRIDE is set as "32" as follows:    PIO_STRIDE: ['CPL:32', 'ATM:32', 'LND:32', 'ICE:32', 'OCN:32', 'ROF:32', 'GLC:32', 'WAV:32', 'ESP:32']Thanks,Xiang

 

[jedwards]

Experiment with changing the pio stride to different values, since you were successful with 4 nodes try PIO_STRIDE=64The netcdf version has three digits, you can get it with nc-config --version

 

[xgao304]

netcdf 4.4.1I will try with different PIO_STRIDE values. 

 

[simkins]

For anyone experiencing this issue still, I just solved it...

The problem had nothing to do with the land model, but rather that the INPUT files I generated were not NETCDF3 files. I had to convert them to NETCDF3 64 bit in order to solve this.