Issue with building cases after porting CESM 2.2.2 onto new machine

[christopher_maloney@colorado_edu]

Hello,

I am porting CESM 2.2.2 onto our Hera machine and am running into an issue with building cases after successfully running "./cesm.setup". I have updated "config_machines.xml", "config_compilers.xml", and "config_batch.xml" for our machine. These individual files are attached to this post in the zipped file. The error message I am receiving at the end of the cesm.build step (copied below) suggest that I am missing a netcdf library. However, I am not sure which library I am missing as I never ran into this issue with porting over earlier versions of the model.

Error message:
/scratch1/BMC/chimera/cmaloney/model_development/CESM2_2_2_code/cime/src/externals/pio2/src/clib/pio_nc4.c:175: undefined reference to `nc_def_var_szip'

Relevant netcdf library call from "config_compilers.xml":
<SLIBS>
<base>-L$(NETCDF)/lib -L$(NETCDF)/include -lnetcdf -lnetcdff</base>
<append MPILIB="impi"> -mkl=sequential </append>
</SLIBS>

Important note: we just updated our operating system on our machine to Rocky 8 and updated our intel compilers. Below is the compiler call list from "config_machines.xml". I am not sure if the order of the "gnu" and "intel" call matters here, but this is the only way that I could get 'cesm.setup' to execute properly. Our intel compilers require gnu to be loaded first.

<modules>
<command name="purge"/>
</modules>
<modules compiler="intel">
<command name="load">gnu/13.2.0</command>
<command name="load">intel/2023.2.0</command>
</modules>
<modules mpilib="impi">
<command name="load">impi/2022.1.2</command>
<command name="load">netcdf-hdf5parallel/4.7.0</command>
<command name="load">pnetcdf/1.10.0</command>
</modules>

Also included in the zipped file is the bld.log. To navigate to my added code in "config_compilers.xml" and "config_batch.xml" search for the machine named "hera".

Main question: Is there a netcdf library that I am missing, or is this something potentially related to the change in our operating system?

Thank you in advance for the assistance!
- Chris Maloney

 

[slevis]

Moving to the Infrastructure forum.

 

[jedwards]

Your netcdf version does not support the full features of netcdf, you should update to at least 4.9.2
Also I do not recommend the cesm2.2.2 release if your aim is to track cesm development.
The latest version is cesm3_0_beta02.

 

[christopher_maloney@colorado_edu]

Thank you for the suggestion! I figured it was a simple fix. However, a new problem has occurred. The model compiles successfully now, but at around 1 min into my test runs, a segmentation fault occurs which kills the the run. This error has appeared regardless of the compset that I tried. The error message in the cesm.log file suggests something to do with the PIO library (see attached log files) and occurs either when the atmospheric model or land model is initializing (depending on the compset I am using). The log files that are attached to this post are for the F2000climo compset.

As for moving on to CESM3, for now we plan on sticking with CESM2 as I am not sure some of the modules that we use (like CARMA) have been validated on CESM3 yet.


Thanks again!
Chris

 

[jedwards]

This looks like an error in the impi mpi-io layer - do you have another mpi library that you can try?

 

[aherring]

FYI we dropped support for cesm2.2 before resolving performance issues in the 2.2.2 tag on Derecho (see here). Buyer beware.

 

[christopher_maloney@colorado_edu]

Thank you Jim and Adam for the comments. Per Jim's suggestions, I have successfully run the CESM 2.2.2 base model. Now I have been working on testing our more complex sets of simulations. Since CESM 2.2.2 performance issues weren't fully resolved, would you suggest going back to an older version of CESM2? For example, before our operating system switch, we were using CESM2.1.3.

Thanks,
Chris Maloney

 

[aherring]

CESM2.1 is still supported, and the latest tag is CESM2.1.5.

 

[christopher_maloney@colorado_edu]

Thanks Adam. I am finally getting back to this after being sidetracked with other work. I thought I would continue this thread instead of creating another. I have CESM 2.1.3 running on our new operating system, but only for the supported compsets. We often use unsupported compsets but every time I try to run a simulation with "-run-unsupported", the run times out during the writing of the restart files. Here is an example waccm_ma compset frequently used: 2000_CAM60%WCCM_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_CISM2%NOEVOLVE_SWAV.

This is the only error message that is appearing in my cesm.log file:

GPTLstopf thread 0: timer for "a:PIO:pre_pio_write_nf" had not been started

I looked around at old forum posts an the only similar error I found was from this post: errors occur when the cesm2.1.3 start running

The normal case compile and submit executables are being used, so could this just be an issue with needing to replace the old PIO1 and PIO2 libraries with the most up to date versions?

Thanks,
Chris Maloney

 

[aherring]

I think a software engineer will need to chime in.

 

[jedwards]

The lastest in the 2.1.x is 2.1.5, please try again with that and let us know if you are still having issues.

 

[christopher_maloney@colorado_edu]

Thanks Jim and Adam,

The oldest PIO2 version I see on the github is 2.4.0. Do you mean 2.5.1?

 

[jedwards]

I'm not talking about pio - I'm talking about the cesm tag you are using - you need to update from 2.1.3 to 2.1.5.

 

[TINGTING]

Hello, I also met same error as you :

Error message:/scratch1/BMC/chimera/cmaloney/model_development/CESM2_2_2_code/cime/src/externals/pio2/src/clib/pio_nc4.c:175: undefined reference to `nc_def_var_szip'

How did you solve it?
Thanks

 

[jedwards]

You need a newer version of netcdf.