Problems running CESM1.0

[meir02ster@gmail_com]

Dear all,

I was trying to run CESM1.0 with a similar setting that I used to run CCSM4.0.
And I got this error message:

READINITIAL: model parameters do not match initial dataset parameters
Model Parameters: plev = 30 plon = 24 plat =
19
Dataset Parameters: dlev = 26 dlon = 144 dlat =
96

So I thought it's due to the resolution incompatibility. Then I changed the configure part of my jobscript to this:

$cfgdir/configure -phys cam5 -hgrid 96x144 -nlev 26 -dyn fv -ntasks 8

Then I got another error message:

[meir02@water bld]$ ./sgi2.csh: Set mpirun to /usr/local/mpi/bin/mpirun
set_horiz_grid: no match for dycore fv and hgrid 96x144
configure failed

So it seems this resolution is not a supported one. Could someone help me here?
And my other question is that if I want to run CCSM4.0 or CESM1.0 with active DGVM of CLM4? How could I achieve that? I assume without specification in the jobscript the CLM4.0 is ported without DGVM.

Thank you very much.

Rui

 

[meir02ster@gmail_com]

I knew where I was wrong. It should be 'configure -hgrid 1.9x2.5' and the nlev is required to be 30. But I still appreciate if someone can answer me the part of question regarding DGVM in CLM4.

And I am also wondering it looks like CAM4 is a transition version to CAM5. Is there big differences between CAM5 and CAM4 when simulating climatology of precipitation or annual variability of precipitation?

Thank you very much

Rui

 

[meir02ster@gmail_com]

Hi, guys,

I got a new error:

trcdata_init: data type: CYCLICAL file: oxid_1.9x2.5_L26_1850-2005_c091123.nc
(GETFIL): attempting to find local file oxid_1.9x2.5_L26_1850-2005_c091123.nc
(GETFIL): using
/home/meir02/ccsm4_0/inputdata/atm/cam/chem/trop_mozart_aero/oxid/oxid_1.9x2.5_L26_1850-2005_c091123.nc
open_trc_datafile: Opened existing file
/home/meir02/ccsm4_0/inputdata/atm/cam/chem/trop_mozart_aero/oxid/oxid_1.9x2.5_L26_1850-2005_c091123.nc

13
/home/meir02/ccsm4_0/inputdata/atm/cam/chem/trop_mozart_aero/oxid/oxid_1.9x2.5_L26_1850-2005_c091123.nc
trcdata_init: file%has_ps = T
There are no offline tracer sources

airpl_src: NO and CO do not have external source --> no aircraft sources will be applied
chemini: after airpl_src on node 0
srf_emis_inti: n_emis_species = 9
pio_support:io_die:: myrank= 1 : ERROR: ionf_mod.F90:
212 : No such file or directory
rank 1 in job 15 water_56994 caused collective abort of all ranks
exit status of rank 1: killed by signal 9

It seems related to the chem package. I have no clues at all.

Thank you for any advice or suggestion!!

Rui

 

[eaton]

If the DVGM mode of CLM that you wish to use is supported from the CESM scripts then that will be the best way to run it (rather than trying to do so from CAM standalone scripts which will require modification to CAM's configure). You should ask in the CLM forum about how to run this mode.

The differences between cam3 and cam4 physics are mainly the Neale/Richter modifications to the deep convection. In contrast, just about every other parameterization in the physics package changed going from cam4 to cam5. You need to look at the scientific description docs to appreciate these changes.

The error from your most recent post is:

212 : No such file or directory

The next output to the log file from a successful run is

Opened existing file /fis/cgd/cseg/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc 64
trcdata_init: data type: CYCLICAL file: /fis/cgd/cseg/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): using //fis/cgd/cseg/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
open_trc_datafile: //fis/cgd/cseg/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
Opened existing file //fis/cgd/cseg/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc 64
trcdata_init: file%has_ps = F

So I suspect the problem is that this datafile (and probably other similar datasets) are missing. Run build-namelist with the -test option to see what datasets you need to download from the svn repo.

 

[meir02ster@gmail_com]

Hi, Brian,

Thanks for the reply. It's fixed. However, I got a new error message:

(shr_stream_init) Reading file docn.stream.txt
PGFIO-F-209/OPEN/unit=99/'OLD' specified for file which does not exist.
File name = docn.stream.txt
In source file /home/meir02/cesm1_0/models/csm_share/shr/shr_stream_mod.F90, at line number 250
rank 0 in job 22 water_56994 caused collective abort of all ranks
exit status of rank 0: return code 127

It's complaining about this file 'docn.stream.txt', I realized after building CAM5, it creates 7 *in files: atm_in, lnd_in, drv_in ice_in drv_flds_in, docn_in, docn_ocn_in while in CAM4, there are only five : tm_in, lnd_in, drv_in ice_in drv_flds_in,ocn_in.
And in CAM5 docn_ocn_in file, there is such statement: streams = 'docn.stream.txt 0 0 0 '.

So do I need to download this file and specifiy it in the path?

Thank you for the help.

Rui

 

[meir02ster@gmail_com]

And I guess it comes from the differences between build-namelist file in CAM5 and in CAM4. I knew nothing about perl programming and was trying hard to understand part of the problem interest. Hope to get some clues here.

Thanks,
Rui

 

[eaton]

The file docn.stream.txt is also output by build-namelist. If you aren't executing build-namelist from your run directory then this file needs to be copied along with all the other files produced by build-namelist to the run directory. For this reason I find it best to just execute build-namelist from the run directory, and to use the -config argument to specify the location of the config_cache.xml file which is in the build directory.

There is some information about this in the user guide:
http://www.cesm.ucar.edu/models/cesm1.0/cam/docs/users_guide/ug.html#bldnl_default

 

[meir02ster@gmail_com]

Great! It's fixed and now it's running well. However, it takes our machine more than 2 hours to produce one monthly file. For CAM4, I usually get monthly output at every 30mins on this local machine. Is there such a huge difference? I am so surprised.

On the other hand I am trying to familiarize myself with CESM scripts to run the model with more active components. I first try to run it at CAM5 standalone mode. And I got an error message while building it:

here is what I executed: (I changed the environment variables for compiler, netcdf and mpi in test.generic_linux_pgi.build before I execute this file)
"./create_newcase -case /home/meir02/cesm1_0/test -mach generic_linux_pgi -compset FC5 -res f19_g16 -scratchroot /home/meir02/cesm1_0/test -din_loc_root_csmdata /home/meir02/ccsm4_0/inputdata -max_tasks_per_node 8
./configure -case
./test.generic_linux_pgi.build"

Here is the log file of error:
"configure: creating ./config.status
config.status: creating Makefile.conf
Please check the Makefile.conf
Have a nice day!
make[1]: Entering directory `/home/meir02/cesm1_0/test/test/mct/mpeu'
/opt/pgi/linux86-64/10.4/bin/pgcc -c -DFORTRAN_UNDERSCORE_ -DSYSLINUX -DCPRPGI -O get_zeits.c
/opt/pgi/linux86-64/10.4/bin/pgf90 -c -DSYSLINUX -DCPRPGI -O2 -pc 64 m_mpif.F90
PGF90-S-0017-Unable to open include file: mpif.h (m_mpif.F90: 60)
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_integer (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_real (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_double_precision (m_mpif.F90: 68)
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_logical (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_character (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_real4 (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_real8 (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_comm_world (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_comm_null (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_sum (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_prod (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_min (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_max (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_max_error_string (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_status_size (m_mpif.F90: 6
PGF90-W-0155-MODULE PROCEDURE not defined: mpi_any_source (m_mpif.F90: 6
0 inform, 16 warnings, 1 severes, 0 fatal for m_mpif
make[1]: *** [m_mpif.o] Error 2
make[1]: Leaving directory `/home/meir02/cesm1_0/test/test/mct/mpeu'
make: *** [subdirs] Error 2"

Thank you for offering any diagnostic thoughts or solution to this problem.

Rui

 

[meir02ster@gmail_com]

I had some thoughts on this issue. The problem is that it could not find mpif.h. However, I have specified MPI paths in both Macros.generic_linux_pgi file and env_mach_specific.

In Macros.generic_linux_pgi, this is what I modified:

#=====================================================
# GENERIC_USER
# The netcdf and mpi paths need to be set somewhere. In CCSM, this
# is normally done by setting the NETCDF_PATH and MPI_PATH
# variables in this Macros file. But they could also be set in the
# default user paths, by an explicit addition to the local path
# in the env_mach_specific file, or via setting NETCDF_PATH
# and MPI_PATH environment variables in the env_mach_specific file.
# Also, review the relative paths of the INC_, LIB_, and MOD_ paths
# below. Ignore PNETCDF stuff initially, it's not required.

NETCDF_PATH := /usr/netcdf
INC_NETCDF := $(NETCDF_PATH)/include
LIB_NETCDF := $(NETCDF_PATH)/lib
MOD_NETCDF := $(NETCDF_PATH)/include
MPICH_PATH := /usr/local/mpi
INC_MPI := /usr/local/mpi/include
LIB_MPI := /usr/local/mpi/lib
MPI_LIB_NAME :=
PNETCDF_PATH :=
INC_PNETCDF :=
LIB_PNETCDF :=
LAPACK_LIBDIR :=

And in env_mach_specific, I also added:

#--- set env variables for Macros if needed
#setenv NETCDF_PATH something
setenv PGI /opt/pgi
setenv PATH ${PGI}/linux86-64/10.4/bin:/usr/local/mpi/bin:${PATH}
setenv LD_LIBRARY_PATH "${PGI}/linux86-64/10.4/lib"
setenv INC_MPI /usr/local/mpi/include
setenv LIB_MPI /usr/local/mpi/lib
set mpirun = /usr/local/mpi/bin/mpirun
echo "${0}: Set mpirun to $mpirun";
setenv INC_NETCDF /usr/netcdf/include
setenv LIB_NETCDF /usr/netcdf/lib

I guess there must be somewhere that I need to make changes to get it through.
Hope to get more clues here.

Thank you very much!
Rui

 

[eaton]

I don't have time to track this down right now, but the problem is that the command that's being issued to compile m_mpif.F90, i.e.,

/opt/pgi/linux86-64/10.4/bin/pgf90 -c -DSYSLINUX -DCPRPGI -O2 -pc 64 m_mpif.F90

does not have any options for where to look for include files. It should have an option like

-I/usr/local/mpi/include

Try looking for the Makefile that is used to build the mct library, and figure out where it gets its information about include directories from. Look in the file scripts/ccsm_utils/Tools/ccsm_buildexe.csh for information about how mct is built.

 

[meir02ster@gmail_com]

Thanks, Brian. This problem is fixed by adding setenv MPIHEADER -I/usr/local/mpi/include. Now I got another error message while building pio:

"gmake[1]: Entering directory `/home/meir02/cesm1_0/test/test/pio'
mpif90 -c -pc 64 -r8 -byteswapio -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLINUX -DSEQ_ -DFORTRANUNDERSCORE -DNO_SHR_VMATH -i4 -gopt -Mlist -Mextend -byteswapio -Mflushz -Kieee -Ktrap=fp -I/usr/local/mpi/include -DSYSLINUX -DLINUX -DCPRPGI -DSPMD -DHAVE_MPI -DUSEMPIIO -D_NETCDF -D_NOPNETCDF -D_NOUSEMCT -D_USEBOX -I/usr/netcdf/include pio_kinds.F90
f95: 64: No such file or directory
f95: unrecognized option '-pc'
f95: unrecognized option '-r8'
f95: unrecognized option '-byteswapio'
f95: unrecognized option '-Mlist'
f95: unrecognized option '-Mextend'
f95: unrecognized option '-byteswapio'
f95: unrecognized option '-Mflushz'
f95: unrecognized option '-Kieee'
f95: unrecognized option '-Ktrap=fp'
cc1: error: unrecognized command line option "-i4"
gmake[1]: *** [pio_kinds.o] Error 1
gmake[1]: Leaving directory `/home/meir02/cesm1_0/test/test/pio'
gmake: *** [all] Error 2"

So while it was using mpif90 to compile the files, the options are not recognized. Do I need to change to pgf90 compiler. I am not sure if the use of mpif90 in building pio lib is the reason And I do not know where to change it.

Thank you for any help.

Rui

 

[meir02ster@gmail_com]

I identified that this error comes out when the following command in pio.buildlib is being executed under /test/Buildconf

" gmake -f $CASETOOLS/Makefile configure MODEL=pio MACFILE=$CASEROOT/Macros.$MACH"

This points to the Makefile in /test/Tools/ and also the Macros.generic_linux_pgi. I guess I need to change the Macros file for this part:

#=====================================================
# GENERIC_USER
# Options passed to the mct and pio build are set here

ifeq ($(MODEL),mct)
#add arguments for mct configure here
CONFIG_ARGS+= FC="$(FC)" F90="$(FC)" FFLAGS="$(FFLAGS)" --host=Linux
endif

#FFLAGS = 64 -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLINUX -DSEQ_ -DFORTRANUNDERSCORE -DNO_SHR_VMATH -gopt -I/usr/local/mpi/include -DSYSLINUX -DLINUX -DCPRPGI (this is what I added)

ifeq ($(MODEL),pio)
ifneq ($(strip $(PIO_CONFIG_OPTS)),)
CONFIG_ARGS += $(PIO_CONFIG_OPTS)
endif
CONFIG_ARGS += FC="$(FC)" F90="$(FC)" NETCDF_PATH=$(NETCDF_PATH) FFLAGS="$(FFLAGS)" PNETCDF_PATH=$(PNETCDF_PATH) --enable-filesystem-hints lustre --host=Linux
endif
#=====================================================

Then I added one line before it passes to pio to eliminate the options in FFLAGS that is complained above,

FFLAGS = 64 -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLINUX -DSEQ_ -DFORTRANUNDERSCORE -DNO_SHR_VMATH -gopt -I/usr/local/mpi/include -DSYSLINUX -DLINUX -DCPRPGI

Then when I rebuild it, it complains other FFLAGS options pio:

gmake[1]: Entering directory `/home/meir02/cesm1_0/test/test/pio'
mpicc -c -g -O2 -DFORTRAN_UNDERSCORE_ -DFORTRANUNDERSCORE -DSYSLINUX -DLINUX -DCPRPGI -DSPMD -DHAVE_MPI -DUSEMPIIO -D_NETCDF -D_NOPNETCDF -D_NOUSEMCT -D_USEBOX -I/usr/netcdf/include topology.c
mpicc -c -g -O2 -DFORTRAN_UNDERSCORE_ -DFORTRANUNDERSCORE -DSYSLINUX -DLINUX -DCPRPGI -DSPMD -DHAVE_MPI -DUSEMPIIO -D_NETCDF -D_NOPNETCDF -D_NOUSEMCT -D_USEBOX -I/usr/netcdf/include pnetcdfversion.c
mpif90 -c -pc 64 -r8 -byteswapio 64 -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLINUX -DSEQ_ -DFORTRANUNDERSCORE -DNO_SHR_VMATH -gopt -I/usr/local/mpi/include -DSYSLINUX -DLINUX -DCPRPGI -DSYSLINUX -DLINUX -DCPRPGI -DSPMD -DHAVE_MPI -DUSEMPIIO -D_NETCDF -D_NOPNETCDF -D_NOUSEMCT -D_USEBOX -I/usr/netcdf/include pio_kinds.F90
f95: 64: No such file or directory
f95: 64: No such file or directory
f95: unrecognized option '-pc'
f95: unrecognized option '-r8'
f95: unrecognized option '-byteswapio'
f951: error: unrecognised debug output level "opt"
gmake[1]: *** [pio_kinds.o] Error 1
gmake[1]: Leaving directory `/home/meir02/cesm1_0/test/test/pio'
gmake: *** [all] Error 2


Now there are two things I am not sure, do I need to change the mpif90 to pgf90 or delete part of FFLAGS options that are unrecognized. And How to implement those changes in MACROS (or other files if I am wrong)

Thank you!
Rui

 

[eaton]

I think the errors encountered in both the previous posts are related to the mpif90 wrapper. This wrapper seems to be invoking f95 as the Fortran compiler, rather than pgif90 which is what I would expect. The arguments being supplied to the compiler are appropriate for the pgi fortran compiler, yet you're getting messages stating that the options are not recognized. This isn't a problem in the CESM scripts. I assume (based on the other mpi paths that you're using) that your mpif90 is located in /usr/local/mpi/bin/. Are you sure that this mpi was built using pgi compilers?

What output do you get from running the command:

/usr/local/mpi/bin/mpif90 -show

 

[meir02ster@gmail_com]

This is the outupt:

f95 -I/usr/local/mpi/include -I/usr/local/mpi/include -L/usr/local/mpi/lib -L/usr/local/mpi/lib -lmpichf90 -lmpichf90 -lmpich -lopa -lpthread -lpthread -lrt

So it means mpi_wrapper is compiled by f95? However, I only have pgf90 on our local machine. And when I was running CAM5, it does not complain about this. What should I do now?

Thank you very much.

Rui

 

[eaton]

I'm not sure what kind of compiler f95 is. This is a systems issue. You need to talk to your system administrator and find out what fortran compiler was used to build the mpi library. If it wasn't the pgi compiler then you need a special build of the mpi library that does use the pgi fortran compiler.

This is a common problem on linux systems which may support multiple fortran compilers. Generally libraries like mpi and netcdf which have fortran interfaces will need separate builds for each fortran compiler. Then to distinquish them the name of the compiler is part of the name of the directory that contains that build. For example, on the local linux system that I use the location of the mpi library built with pgi is under the directory /usr/local/mpich-1.2.7p1-pgi-hpf-cc-9.3/

 

[meir02ster@gmail_com]

I am the administrator for our local machines. I just installed the MPICH as the installation guide says. So I am not sure why it is using f95 instead of pgf90 as its compiler. I will check the MPI installation again to see if I can fix this. Any advice is appreciated though.

Thank you!
Rui

 

[meir02ster@gmail_com]

Hi, Brian,

I reinstalled MPI by specifying pgf90 as the fortran compiler. Now this problem with CESM1_0 is fixed although I do not understand why the mpi was fine with CAM5 run with CAM scripts before.

Now I got another error message:
CCSM BUILDEXE SCRIPT STARTING
- Build Libraries: mct pio csm_share
Sat Sep 18 17:06:12 CDT 2010 /home/meir02/cesm1_0/test/test/mct/mct.bldlog.100918-170610
Sat Sep 18 17:06:12 CDT 2010 /home/meir02/cesm1_0/test/test/pio/pio.bldlog.100918-170610
Sat Sep 18 17:06:58 CDT 2010 /home/meir02/cesm1_0/test/test/csm_share/csm_share.bldlog.100918-170610
Sat Sep 18 17:07:51 CDT 2010 /home/meir02/cesm1_0/test/test/run/cpl.bldlog.100918-170610
Sat Sep 18 17:07:51 CDT 2010 /home/meir02/cesm1_0/test/test/run/atm.bldlog.100918-170610
Sat Sep 18 17:16:57 CDT 2010 /home/meir02/cesm1_0/test/test/run/lnd.bldlog.100918-170610
Sat Sep 18 17:20:09 CDT 2010 /home/meir02/cesm1_0/test/test/run/ice.bldlog.100918-170610
Sat Sep 18 17:21:23 CDT 2010 /home/meir02/cesm1_0/test/test/run/ocn.bldlog.100918-170610
Sat Sep 18 17:21:34 CDT 2010 /home/meir02/cesm1_0/test/test/run/glc.bldlog.100918-170610
Sat Sep 18 17:21:36 CDT 2010 /home/meir02/cesm1_0/test/test/run/ccsm.bldlog.100918-170610
ERROR: ccsm.buildexe.csh failed, see /home/meir02/cesm1_0/test/test/run/ccsm.bldlog.100918-170610

Here is the log file:
/opt/pgi/linux86-64/10.4/bin/pgf90 -o /home/meir02/cesm1_0/test/test/run/ccsm.exe ccsm_driver.o map_atmatm_mct.o map_atmice_mct.o map_atmlnd_mct.o map_atmocn_mct.o map_glcglc_mct.o map_iceice_mct.o map_iceocn_mct.o map_lndlnd_mct.o map_ocnocn_mct.o map_rofocn_mct.o map_rofrof_mct.o map_snoglc_mct.o map_snosno_mct.o mrg_x2a_mct.o mrg_x2g_mct.o mrg_x2i_mct.o mrg_x2l_mct.o mrg_x2o_mct.o mrg_x2s_mct.o seq_avdata_mod.o seq_diag_mct.o seq_domain_mct.o seq_flux_mct.o seq_frac_mct.o seq_hist_mod.o seq_io_mod.o seq_rearr_mod.o seq_rest_mod.o -L/home/meir02/cesm1_0/test/test/lib -latm -llnd -lice -locn -lglc -L/home/meir02/cesm1_0/test/test/lib -lcsm_share -lmct -lmpeu -lpio -L/usr/netcdf/lib -lnetcdf -L/usr/local/mpi2/lib -lmpi
/usr/bin/ld: cannot find -lmpi
gmake: *** [/home/meir02/cesm1_0/test/test/run/ccsm.exe] Error 2

Is that linker supposed to be -lmpich ? Actually I got some other errors and fixed them before reaching this step. Those errors are all related to MPICH or compiler setting. For example, when it was buiding PIO library. It complains about not finding MPIF.h. I found in the compile command it has no "-I/usr/local/mpi2/include". This was something I met before when it was compiling mct library. I specified MPIHEADER to solve it for MCT. However, I could not find the right env to set to solve for PIO. Then I just delete the PIO directory and rebuild without cleanbuild (to save time I thought). Surprisingly It passed building PIO successfully with mpif90 as the compiler. So I am wondering maybe I solved this problem by chance and in a wrong way?? Is it supposed to use pgf90?? So I am totally confused about how to set the MPI, NETCDF and PGF PATH right and build the model codes and relevant libs with the right option linkers while running CESM1_0. I really wish to learn to solve the problem in the ight way. However, I feel like I am doing a trail-and-error way using 'method of exhuastion' to make the model run. Maybe this is the learning process. Although I feel like I understand things better now from your help while running CAM or CESM, it is still very disappointing to have a feeling that I know something but I do not know enough to solve the problem.

Hope to get some help from you.
Thank you!

Rui

 

[eaton]

Try comparing what you get from the command:

% /usr/local/mpi2/bin/mpif90 -show

with the arguments in your pgf90 command related to linking to mpi, that is:

-L/usr/local/mpi2/lib -lmpi

Normally the link command is where you want to use mpif90 rather than pgf90, because the mpif90 wrapper is what supplies the information about the necessary mpi libraries, and the locations of the libs and include files. The cesm scripts will use mpif90 for both the compilation commands and the final link command. If you've changed the Macros file to use pgf90 as the fortran compiler that might be the problem. You should leave the setting "FC := mpif90". Assuming that your path is set correctly so that the mpif90 you get is the one from your installation of MPI using the pgi compilers, then the mpif90 script will correctly invoke the pgf90 compiler that was used in the build of MPI.

 

[meir02ster@gmail_com]

I executed mpif90 -show and here is the output:

pgf90 -I/usr/local/mpi2/include -I/usr/local/mpi2/include -L/usr/local/mpi2/lib -L/usr/local/mpi2/lib -lmpichf90 -lmpichf90 -lmpich -lopa -lpthread -lrt

So if I want to use pgf90 as FC, I have to specify those links but I do not know how to add them?

And do you mean that mpif90 will invoke pgf90 (used to build MPI) to compile the whole model? I reset FC to mpif90 but it still give me that problem:

-------------------------------------------------------------------------
Building a single executable version of CCSM
-------------------------------------------------------------------------
/home/meir02/cesm1_0/test/Tools/mkSrcfiles > /home/meir02/cesm1_0/test/test/ccsm/obj/Srcfiles
cp -f /home/meir02/cesm1_0/test/test/ccsm/obj/Filepath /home/meir02/cesm1_0/test/test/ccsm/obj/Deppath
/home/meir02/cesm1_0/test/Tools/mkDepends Deppath Srcfiles > /home/meir02/cesm1_0/test/test/ccsm/obj/Depends
/usr/local/mpi2/bin/mpif90 -o /home/meir02/cesm1_0/test/test/run/ccsm.exe ccsm_driver.o map_atmatm_mct.o map_atmice_mct.o map_atmlnd_mct.o map_atmocn_mct.o map_glcglc_mct.o map_iceice_mct.o map_iceocn_mct.o map_lndlnd_mct.o map_ocnocn_mct.o map_rofocn_mct.o map_rofrof_mct.o map_snoglc_mct.o map_snosno_mct.o mrg_x2a_mct.o mrg_x2g_mct.o mrg_x2i_mct.o mrg_x2l_mct.o mrg_x2o_mct.o mrg_x2s_mct.o seq_avdata_mod.o seq_diag_mct.o seq_domain_mct.o seq_flux_mct.o seq_frac_mct.o seq_hist_mod.o seq_io_mod.o seq_rearr_mod.o seq_rest_mod.o -L/home/meir02/cesm1_0/test/test/lib -latm -llnd -lice -locn -lglc -L/home/meir02/cesm1_0/test/test/lib -lcsm_share -lmct -lmpeu -lpio -L/usr/netcdf/lib -lnetcdf -L/usr/local/mpi2/lib -l-lmpich
/usr/bin/ld: cannot find -l-lmpich
gmake: *** [/home/meir02/cesm1_0/test/test/run/ccsm.exe] Error 2

Sorry for so many questions. Thank you!
Rui

 

[eaton]

There's a typo somewhere. It's showing up in your link command. Note the extra "-" character in the name of the library:

-L/usr/local/mpi2/lib -l-lmpich

The result from the "mpif90 -show" command doesn't contain this error.

> And do you mean that mpif90 will invoke pgf90 (used to build MPI) to compile the whole model?

Yes, that is correct. When using mpif90 as the fortran compiler/linker you don't need to manually add the mpi library and include file locations.

I haven't done this so I'm guessing here, but I'd try *not* setting any of the MPI related macros in the Macros.generic_linux_pgi file. Then as long as FC is set to mpif90, and as long as your path is set so that when mpif90 is executed you get the version in /usr/local/mpi2/bin, then you can just rely on mpif90 to provide the necessary information about mpi.