Problem with MPI-enabled CLM3 using intel 8.1 on linux cluster

[gulden@mail_utexas_edu]

Hi CCSM users,

I have successfully compiled MPI-enabled CLM3 using intel icc and ifort (8.1) on a linux cluster. The code runs without error in serial, but when I run the code in parallel (num. procs >=2), it stops on the first time step, giving me water balance errors (see output pasted below).

The same code compiled with pgf90 and worked like a charm in both serial and parallel (MPI enabled). (Unfortunately, that compiler is no longer installed on our machines!)

Has anyone experienced a similar problem?

I appreciate your time and help.

Lindsey Gulden
.
.
.

Attempting to initialize atm->land model grid interpolation .....
Initializing atm -> srf interpolation .....
Successfully made atm -> srf interpolation
Successfully initialized area-averaging interpolation

water balance nstep = 1 point = 3097 imbalance =16991.17 mm
clm model is stopping
ENDRUN: called without a message string
water balance nstep = 1 point = 4634 imbalance =******** mm
clm model is stopping
ENDRUN: called without a message string
water balance nstep = 1 point = 10813 imbalance =20289.60 mm
clm model is stopping
ENDRUN: called without a message string
.
.
.
Etc (depending on number of processors employed).

 

[songroy@fio_org_cn]

I think it is the error of the CLM model. Can you give the detailed report, for example, the log file of the CLM and CPL.

 

[gulden@mail_utexas_edu]

Songroy,

I should have mentioned in the post above that for now I am running CLM3 offline.

I've attached two text files. One ("output.txt") contains the output log for CLM when the job is run on two processors. The second ("output.serial.txt") contains the output log for CLM when the same job is run on one processor.

I used the same input surface datasets and meteorological forcings for both runs. Water balance problems exist from the first time step in the parallel run.

I think this is likely a bug in the divving up of cells into clumps, but I am not sufficiently practiced in MPI to know what's truly going on.

I appreciate your taking a look at this!

Lindsey

 

[songroy@fio_org_cn]

I think it is wrong with CLM2. But I do not know what about it is!
I only use ONE PE on the CLM2. I'll try TWE PE or more PE on the CLM2.
Perhaps that's a bug of the Intel Fortran Compiler?I use the Intel Fortran Compiler 9.0 now. If you like, you can upgrade your Fortran Compiler 8.1 to 9.0 .
My email is songroy@fio.org.cn, we can use email to communicate.

 

[songroy@fio_org_cn]

I have test TWO PE on CLM. There is no error occur. I think you should use the Intel Fortran Compiler 9.0 to try.

 

[gulden@mail_utexas_edu]

I appreciate your taking the time to look into this, Songroy. Assuming that our license covers ifort 9, I'll install it and try that.

leg

 

[gulden@mail_utexas_edu]

Thanks Songroy! You solved the problem. I really appreciate your help.

Folks, if you have a similar problem and your license coveres 9.0, you can get the source at
ftp://download.intel.com/software/products/compilers/downloads/eval/preview/l_fc_p_9.0.021.tar.gz