Porting to Archer2: ESMFmesh / PIO/PNetCDF error = Unknown Error: Unrecognized error code -500

[marcinkupilas]

Hello

I am porting CESM to Archer2 and am running into what appears to be an ESMF error that's unable to read in an ESMFmesh file.

The steps used to produce the error, including model version, compset and resolution, are listed below in case workflow.

I have attached the logs and a code I used to troubleshoot the error. The error is found in the PET file. See Error details below.

I would appreciate any help.

Best wishes
Marcin

---

Case workflow

- Version: cam6_4_028 development tag
- Compset: F2000climo
- Resolution: ne30pg3_ne30pg3_mg17

Workflow:

module load PrgEnv-gnu
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load cray-parallel-netcdf
module load cray-libsci
module load cray-python

export ESMF_LIBDIR=/scratch-nvme/n02/n02/phyfs/esmf/lib/libO/Linux.gfortran.64.mpich.default
export ESMFMKFILE=${ESMF_LIBDIR}/esmf.mk
export CIMEROOT=/mnt/lustre/a2fs-work2/work/n02/n02/mmkupilas/tag_cam6_4_028/cime
export ESMF_INSTALL_PREFIX=/scratch-nvme/n02/n02/phyfs/esmf

export NETCDF_PATH=$NETCDF_DIR
export PNETCDF_PATH=$PNETCDF_DIR
export NETCDF_C_PATH=$NETCDF_DIR
export NETCDF_FORTRAN_PATH=$NETCDF_DIR

/work/n02/n02/mmkupilas/tag_cam6_4_028/cime/scripts/create_newcase --case /work/n02/n02/mmkupilas/cesm_sims/cam6_4_028/cam6_4_028.F2000climo.ne30pg3_ne30pg3_mg17 --compset F2000climo --res ne30pg3_ne30pg3_mg17 --run-unsupported --project n02-MESOS2D --machine archer2

cd /work/n02/n02/mmkupilas/cesm_sims/cam6_4_028/cam6_4_028.F2000climo.ne30pg3_ne30pg3_mg17

./case.setup
./case.build
./case.submit

---

Error details
In the PET file, I'm told I can't open the mesh file

ne30pg3_ESMFmesh_cdf5_c20211018.nc.

The full message is

z20251216 124137.705 ERROR PET384 /scratch-nvme/n02/n02/phyfs/esmf-8.9.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:428 ESMCI_mesh_create_from_ESMFMesh_ The NetCDF Library returned an error - Unable to open existing file: /scratch-nvme/n02/n02/mmkupilas/cam64028_input_data/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc, (PIO/PNetCDF error = Unknown Error: Unrecognized error code -500
This is strange, because I wrote a code that opens that file using the version of PNETCDF linked with ESMF, and I can open the file just fine.

> mpicc -O2 -o probe_pnetcdf probe_pnetcdf.c -I"$PNETCDF_DIR/include" -L"$PNETCDF_DIR/lib" -lpnetcdf
> ./probe_pnetcdf /scratch-nvme/n02/n02/mmkupilas/cam64028_input_data/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
Opened OK: /scratch-nvme/n02/n02/mmkupilas/cam64028_input_data/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
dims=4 vars=6 gatts=4 unlimdimid=-1