Running CESM in parallel

[TCNasa]

Anytime I try to specify NTHREADS greater than 1. My CESM code crashes with errors like this.
Any suggestions?

--------------------------------------------------------------------------


Primary job terminated normally, but 1 process returned


a non-zero exit code. Per user-direction, the job has been aborted.


--------------------------------------------------------------------------


--------------------------------------------------------------------------


orterun noticed that process rank 2 with PID 0 on node bn12 exited on signal 11 (Segmentation fault).


--------------------------------------------------------------------------


[bn12:10058] PMIX ERROR: NO-PERMISSIONS in file dstore_base.c at line 237


[bn12:10058] PMIX ERROR: NO-PERMISSIONS in file dstore_base.c at line 246

 

[jedwards]

A couple of answers
1. CESM does not perform as well with openmp threading as with mpi - whenever possible you should use mpi for parallelism instead of openmp
2. How threading is set up in the config_batch.xml file is very machine dependent, if possible seek help from a local system administrator.

Is this happening before any cesm.log file is written? Can you run a simple test with threading - for example - hello world?

 

[TCNasa]

Is there a setting to switch from openmp to mpi?
I don't know precisely when it happens. The error messages are from the cesm log file.

 

[jedwards]

The default is to use mpi, please attach the entire cesm.log file.
Please provide information as suggested here.

 

[TCNasa]

bn12:/CERES/sarb/caldwell/CESM2.2 {187} ./describe_version


------------------------------------------------------------------------


git describe:


cesm2.2.0-0-g332937b


------------------------------------------------------------------------




These commands are used to set the task and thread values:
./xmlchange --id MAX_TASKS_PER_NODE --val 2
./xmlchange --id MAX_MPITASKS_PER_NODE --val 2
./xmlchange NTHRDS=2,NTASKS=6

The browser won't let me choose the log files for upload

 

[TCNasa]

I tried tarring it up but apparently 21 MB is too big

 

[jedwards]

Can you try a run with no threading and make sure that that works?

 

[TCNasa]

Yes a run with NTHREADS=1 works, but I can't increase the number of TASKS over 12 without failure either.

 

[TCNasa]

I can vary tasks per node but I have to use NTASKS=12 and NTHREADS=1

 

[jedwards]

Again - I think that you need to discuss with your system administration.