Problem with netcdf and pio (porting CTSM 5.3) with intel compiler

We are successful building and running CTSM using gcc/gnu but stuck trying to get it to work with the intel compiler (2021) (which we would like to have running for other reasons). Porting to a fairly standard linux HPC cluster.

Our theory: the problem seems related to netcdf and pio (not using pnetcdf). On our cluster we have netcdf installed into the system (e.g. in /usr/bin etc.), ie. so that ncdump doesn't require any module loads, but for building cesm and other models users would typically need to load a module stack including compiler-specific netcdf and pio modules. When trying to build with intel however the build log reports it is using the netcdf system version (4.7), not the module version it claims to be using (4.9).

The specific build error seems suspicious of not using the proper netcdf/pio builds from the module stack:
pio.F90(90): error #6580: Name in only-list does not exist or is not accessible. [PIO_INQ_VAR_FILTER_IDS] PIO_inq_var_filter_ids , &

Does it seem like we are on the right track diagnosing this problem? Any suggestions?
(again, not clear why the gcc build, using a separate gcc-built modules for netcdf and pio, works for us...)

Thanks, Jeff

You didn't tell us what version of cesm you are trying to build. However you can provide an explicit
path to NETCDF using the env variable NETCDF_DIR. I think that because you have an install of netcdf
in /usr it's finding that one first and not using the one set in your module. If you show me the output of
module show netcdf/4.9 I may be able to provide a better suggestion.

We successfully run many cesm versions -- 1.x, 2.1, 2.2 using intel and do not have this problem, which is popping up with CTSM 5.3.
Perhaps this is a problem that might continue to occur with new CIME infrastructure (?)

We are not setting NETCDF_DIR, we'll try that.

Thanks, Jeff


$ module show netcdf/4_shared_libs

--------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/software/intel/2021.4.0/pkg/rhel8/modulefiles/netcdf/4_shared_libs:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
conflict("netcdf")
prepend_path("PATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/bin")
prepend_path("CPATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/include")
prepend_path("LIBRARY_PATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/lib")
prepend_path("LD_LIBRARY_PATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/lib")
prepend_path("MANPATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/share/man")
prepend_path("NETCDF","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/bin")
prepend_path("NETCDF_PATH","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/bin")
prepend_path("NETCDF_INC","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/include")
prepend_path("NETCDF_LIB","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs/lib")
prepend_path("NetCDF_ROOT","/home/software/intel/2021.4.0/pkg/rhel8/netcdf/4_shared_libs")

$ nc-config --version
netCDF 4.9.3