Error in previously functioning port of CESM2.2.2

[James King]

Hi all,

Some colleagues and I successfully ported CESM2.2.2 to the UK HPC service ARCHER2 a while back. I've just returned to using it and can build and run cases, however each case fails with the same mysterious error regardless of compset and other settings. In the cesm.log, we find:

Abort with message Unknown error in file operation in file /work/n02/n02/jking/cesm/CESM2.2.2/my_cesm_sandbox/cime/src/externals/pio2/src/clib/pioc_support.c at line 2843
Obtained 10 stack frames.
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x210fa28]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x210facc]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x210fe70]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x2113bf7]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x210eeec]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x20b2a04]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x1f11c08]
/work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe() [0x1ecec71]

There are no other error messages in any of the component logs [attached], even with debug mode on. The line referred to in the message is in this code block;

/* If there was an error, free allocated memory and deal with the error. */

if (ierr)

{

free(file);

return check_netcdf2(ios, NULL, ierr, __FILE__, __LINE__);

}

I'm wondering if something has changed on ARCHER2 with respect to libraries that might be causing this, but I'm not sure if this hunch is correct or how to fix it. Any suggestions of where to look would be much appreciated. I couldn't attach the cesm.log as the file was too large.

James

 

[jedwards]

I suggest that you check the file type and the integrity of the file in question. Sometimes you can get a traceback using
addr2line -e /work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe 0x1f11c08

 

[James King]

I resolved this for now by installing a clean build of CESM2.2.2 which seems to run OK. Running the traceback returned something in the river routing model:

addr2line -e /work/n02/n02/jking/cesm/CESM2.2.2/runs/FHIST_UK_JAK_MUSICA_test.001/bld/cesm.exe 0x1f11c08
/work/n02/n02/jking/cesm/CESM2.2.2/my_cesm_sandbox/cime/../components/mosart/src/riverroute/RtmIO.F90:179