CESM freeze issue on Perlmutter when running on multiple nodes

[jray]

Hi,

I'm currently working on porting CESM to the Perlmutter system at NERSC. While the model builds and runs successfully on a single node, it consistently freezes without any error message during runtime when I attempt to use multiple nodes. This issue persists across multiple CESM versions that I've tested, including:

• cesm2_3_beta17
• cesm3_0_beta3

Has anyone encountered similar problems when running CESM on Perlmutter with more than one node? I would greatly appreciate any suggestions, debugging tips, or working configurations known to the community.

Thanks,

 

[jedwards]

I was able to run on multiple nodes this morning with no issues. I am using cesm3.0-beta06 I am using a branch of the ccs_config repository that
you can find here:
GitHub - jedwards4b/ccs_config_cesm: CESM CIME Case Control System configuration files
* [new branch] perlmutter_update -> perlmutter_update

 

[jray]

Hi Jim, Thank you for your reply.

It seems that ESMF might be the key difference here. In your config_machines.xml, the module esmf/8.7.0 is listed, along with parallelio/2.6.3, but I had installed ESMF manually. When I try module load esmf/8.7.0 or module load parallelio/2.6.3, the system cannot find these modules.

Could you let me know how are these modules made available on your system?

 

[jray]

I attempted to access the module path /global/cfs/cdirs/ccsm1/modulefiles/perlmutter/, but it returned a permission denied error.

 

[jedwards]

Permissions corrected, sorry.

 

[jray]

In addition, it also appears that access to the following directory is required:
/global/cfs/cdirs/ccsm1/sw/perlmutter/

This is because ESMFMKFILE is defined as:
/global/cfs/cdirs/ccsm1/sw/perlmutter/modules/cray-mpich/8.1.28/intel/2023.2.0/esmf/8.7.0/lib/libO/Unicos.intel.64.mpi.default//esmf.mk

Thank you