Problem Running CESM in single column

[sriki]

Problem: ./case.submit
MPI_Init: conflicting MPI_COMM_WORLD

./pelayout
Comp NTASKS NTHRDS ROOTPE
CPL : 1/ 1; 0
ATM : 1/ 1; 0
LND : 1/ 1; 0
ICE : 1/ 1; 0
OCN : 1/ 1; 0
ROF : 1/ 1; 0
GLC : 1/ 1; 0
WAV : 1/ 1; 0
ESP : 1/ 1; 0

./preview_run
CASE INFO:
nodes: 1
total tasks: 1
tasks per node: 1
thread count: 1
BATCH INFO:
FOR JOB: case.run
ENV:
module command is /common/Env_Modules/EL7/Modules/3.2.10/bin/modulecmd python load netcdf/4.1.3_no-hdf_pgi16.3_gcc
Setting Environment OMP_STACKSIZE=256M
Setting Environment OMP_NUM_THREADS=1
SUBMIT CMD:
None
FOR JOB: case.st_archive
ENV:
module command is /common/Env_Modules/EL7/Modules/3.2.10/bin/modulecmd python load netcdf/4.1.3_no-hdf_pgi16.3_gcc
Setting Environment OMP_STACKSIZE=256M
Setting Environment OMP_NUM_THREADS=1
SUBMIT CMD:
None
MPIRUN:
/rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.exe >> cesm.log.$LID 2>&1

./xmlquery --listall --file LockedFiles/env_build.xml
setting case file to LockedFiles/env_build.xml
setting case file to LockedFiles/env_build.xml
Results in group build_component_cam
CAM_CONFIG_OPTS: -phys cam6 -dyn eul -scam
CAM_CPPDEFS: -DPLON=1 -DPLAT=1 -DPLEV=32 -DPCNST=33 -DPCOLS=1 -DPSUBCOLS=1 -DN_RAD_CNST=30 -DPTRM=1 -DPTRN=1 -DPTRK=1 -DMODAL_AERO -DMODAL_AERO_4MODE -DCLUBB_SGS -DCLUBB_CAM -DNO_LAPACK_ISNAN -DCLUBB_REAL_TYPE=dp -DHAVE_VPRINTF -DHAVE_TIMES -DHAVE_GETTIMEOFDAY -DHAVE_COMM_F2C -DHAVE_NANOTIME -DBIT64 -DHAVE_SLASHPROC
CAM_DYCORE: eul
CLM_CPPDEFS: UNSET
Results in group build_component_cice
CICE_AUTO_DECOMP: TRUE
CICE_BLCKX: 1
CICE_BLCKY: 1
CICE_CONFIG_OPTS: -phys cice5
CICE_CPPDEFS: -DCESMCOUPLED -Dncdf -DNUMIN=11 -DNUMAX=99 -DNICECAT=1 -DNXGLOB=1 -DNYGLOB=1 -DNTRAERO=0 -DNTRISO=0 -DNBGCLYR=0 -DNICELYR=8 -DNSNWLYR=3 -DTRAGE=1 -DTRFY=1 -DTRLVL=1 -DTRPND=1 -DTRBRI=0 -DTRBGCS=0 -DBLCKX=1 -DBLCKY=1 -DMXBLCKS=1
CICE_DECOMPSETTING: square-ice
CICE_DECOMPTYPE: cartesian
CICE_MODE: prescribed
CICE_MXBLCKS: 1
Results in group build_component_clm
CLM_CONFIG_OPTS: -phys clm5_0
CLM_USE_PETSC: FALSE
Results in group build_def
CALENDAR: NO_LEAP
CIME_OUTPUT_ROOT: /rhome/sriki/cesm/scratch
COMP_INTERFACE: mct
DEBUG: FALSE
EXEROOT: /rhome/sriki/cesm/scratch/$CASE/bld
FORCE_BUILD_SMP: FALSE
SMP_PRESENT: FALSE
USE_ALBANY: FALSE
USE_ESMF_LIB: FALSE
USE_MOAB: FALSE
USE_PETSC: FALSE
USE_TRILINOS: FALSE
Results in group build_derived
INCROOT: /rhome/sriki/cesm/scratch/$CASE/bld/lib/include
LIBROOT: /rhome/sriki/cesm/scratch/$CASE/bld/lib
OBJROOT: /rhome/sriki/cesm/scratch/$CASE/bld
PERL5LIB:
SHAREDLIBROOT: /rhome/sriki/cesm/scratch/$CASE/bld
Results in group build_grid
ATM_GRID: T42
ATM_NX: 128
ATM_NY: 64
GLC_GRID: null
GLC_NX: 0
GLC_NY: 0
GRID: a%T42_l%T42_oi%T42_r%null_g%null_w%null_m%gx1v7
ICE_GRID: T42
ICE_NCAT: 1
ICE_NX: 128
ICE_NY: 64
LND_GRID: T42
LND_NX: 128
LND_NY: 64
MASK_GRID: gx1v7
OCN_GRID: T42
OCN_NX: 128
OCN_NY: 64
ROF_GRID: null
ROF_NX: 0
ROF_NY: 0
WAV_GRID: null
WAV_NX: 0
WAV_NY: 0
Results in group build_macros
COMPILER: pgi
MPILIB: mpi-serial
OS: LINUX
PIO_CONFIG_OPTS:
PIO_VERSION: 1
Results in group build_status
BUILD_COMPLETE: TRUE
BUILD_STATUS: 0
NINST_BUILD: 0
NINST_VALUE: c1a1l1i1o1r1g1w1e1
SMP_BUILD: c0a0l0i0o0r0g0w0e0
SMP_VALUE: c0a0l0i0o0r0g0w0e0

./check_case
Creating component namelists
Calling /nas/rstor/sriki/CESM/components/cam//cime_config/buildnml
CAM namelist copy: file1 /nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/Buildconf/camconf/atm_in file2 /rhome/sriki/cesm/scratch/test_scam_mpace/run/atm_in
Calling /nas/rstor/sriki/CESM/components/clm//cime_config/buildnml
WARNING: CLM is starting up from a cold state
Calling /nas/rstor/sriki/CESM/components/cice//cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/data_comps/docn/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/srof/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/sglc/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/swav/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/sesp/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/drivers/mct/cime_config/buildnml
Finished creating component namelists
check_case OK

./case.build --debug
…………..
………….
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam,
M ./cime
modified sandbox, on cime5.6.19
./components/cam
clean sandbox, on cam1/release_tags/cam_cesm2_1_rel_29/components/cam
./components/cice
clean sandbox, on cice5_cesm2_1_1_20190321
./components/cism
clean sandbox, on release-cesm2.0.04
./components/cism/source_cism
clean sandbox, on release-cism2.1.03
./components/clm
clean sandbox, on release-clm5.0.25
./components/clm/src/fates
clean sandbox, on fates_s1.21.0_a7.0.0_br_rev2
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_180611
./components/mosart
clean sandbox, on release-cesm2.0.03
./components/pop
clean sandbox, on pop2_cesm2_1_rel_n06
./components/pop/externals/CVMix
clean sandbox, on v0.93-beta
./components/pop/externals/MARBL
clean sandbox, on cesm2.1-n00
./components/rtm
clean sandbox, on release-cesm2.0.02
./components/ww3
clean sandbox, on ww3_181001
RUN: /usr/bin/xmllint --format --output /nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/env_build.xml -
Time spent not building: 2.832367 sec
Time spent building: 17.452500 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

./check_input_data
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/clm.input_data_list'
Model clm no file specified for finidat

 

[jedwards]

What problem are you trying to report?

 

[sriki]

When I start executing, I get an error in the log file

Problem: ./case.submit
run command is /rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.exe >> cesm.log.$LID 2>&1
ERROR: RUN FAIL: Command ' /rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /rhome/sriki/cesm/scratch/test_scam_mpace/run/cesm.log.200226-093621

more /rhome/sriki/cesm/scratch/test_scam_mpace/run/cesm.log.200226-093621
MPI_Init: conflicting MPI_COMM_WORLD

 

[jedwards]

Never seen this one before - can you please attach your cesm.bldlog file from the case build directory.

 

[sriki]

cat cesm.bldlog.200224-144621

Building a single executable version of target coupled model
/nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/Tools/mkSrcfiles
cp -f Filepath Deppath
/nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/Tools/mkDepends Deppath Srcfiles > Depends
%file_modules
cplcomp_exchange_mod.F90 => seq_map_type_mod.mod component_type_mod.mod seq_diag_mct.mod
prep_lnd_mod.F90 => map_glc2lnd_mod.mod seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
prep_aoflux_mod.F90 => seq_map_type_mod.mod prep_ocn_mod.mod prep_atm_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
cime_driver.F90 => cime_comp_mod.mod
prep_glc_mod.F90 => map_glc2lnd_mod.mod shr_assert_mod.mod seq_map_type_mod.mod map_lnd2glc_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
seq_map_type_mod.F90 =>
map_lnd2glc_mod.F90 => shr_assert_mod.mod seq_map_type_mod.mod seq_map_mod.mod
prep_atm_mod.F90 => seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
component_mod.F90 => seq_map_type_mod.mod seq_domain_mct.mod cplcomp_exchange_mod.mod prep_ice_mod.mod prep_glc_mod.mod prep_ocn_mod.mod component_type_mod.mod t_drv_timers_mod.mod prep_lnd_mod.mod seq_diag_mct.mod seq_map_mod.mod
prep_rof_mod.F90 => shr_assert_mod.mod seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod map_lnd2rof_irrig_mod.mod prep_lnd_mod.mod seq_map_mod.mod
map_lnd2rof_irrig_mod.F90 => shr_assert_mod.mod seq_map_type_mod.mod seq_map_mod.mod
cime_comp_mod.F90 => rof_comp_mct.mod seq_map_type_mod.mod prep_aoflux_mod.mod seq_domain_mct.mod seq_hist_mod.mod cplcomp_exchange_mod.mod esp_comp_mct.mod prep_ice_mod.mod atm_comp_mct.mod prep_glc_mod.mod seq_io_mod.mod prep_ocn_mod.mod prep_atm_mod.mod component_type_mod.mod t_drv_timers_mod.mod wav_comp_mct.mod component_mod.mod seq_flux_mct.mod prep_lnd_mod.mod seq_frac_mct.mod prep_wav_mod.mod seq_rest_mod.mod prep_rof_mod.mod ocn_comp_mct.mod seq_diag_mct.mod glc_comp_mct.mod ice_comp_mct.mod seq_map_mod.mod
seq_domain_mct.F90 => seq_map_type_mod.mod prep_glc_mod.mod prep_ocn_mod.mod prep_atm_mod.mod component_type_mod.mod prep_lnd_mod.mod seq_map_mod.mod
map_glc2lnd_mod.F90 => shr_assert_mod.mod seq_map_type_mod.mod seq_map_mod.mod
mrg_mod.F90 =>
seq_io_mod.F90 => component_type_mod.mod
seq_frac_mct.F90 => seq_map_type_mod.mod prep_ice_mod.mod prep_glc_mod.mod prep_ocn_mod.mod component_type_mod.mod prep_atm_mod.mod prep_lnd_mod.mod prep_rof_mod.mod seq_map_mod.mod
seq_flux_mct.F90 => component_type_mod.mod
seq_hist_mod.F90 => prep_aoflux_mod.mod seq_io_mod.mod prep_ocn_mod.mod component_type_mod.mod prep_atm_mod.mod
seq_map_mod.F90 => seq_map_type_mod.mod component_type_mod.mod
t_driver_timers_mod.F90 =>
seq_diag_mct.F90 => component_type_mod.mod
prep_wav_mod.F90 => seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
seq_rest_mod.F90 => prep_aoflux_mod.mod prep_glc_mod.mod seq_io_mod.mod prep_ocn_mod.mod component_type_mod.mod prep_rof_mod.mod seq_diag_mct.mod
component_type_mod.F90 => seq_map_type_mod.mod
prep_ice_mod.F90 => seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
prep_ocn_mod.F90 => seq_map_type_mod.mod component_type_mod.mod t_drv_timers_mod.mod seq_map_mod.mod
%file_includes
cplcomp_exchange_mod.F90 => mpif.h
prep_lnd_mod.F90 =>
prep_aoflux_mod.F90 =>
cime_driver.F90 =>
prep_glc_mod.F90 => shr_assert.h
seq_map_type_mod.F90 =>
map_lnd2glc_mod.F90 => shr_assert.h
prep_atm_mod.F90 =>
component_mod.F90 => mpif.h
prep_rof_mod.F90 => shr_assert.h
map_lnd2rof_irrig_mod.F90 => shr_assert.h
cime_comp_mod.F90 => mpif.h
seq_domain_mct.F90 => mpif.h
map_glc2lnd_mod.F90 => shr_assert.h
mrg_mod.F90 =>
seq_io_mod.F90 =>
seq_frac_mct.F90 =>
seq_flux_mct.F90 =>
seq_hist_mod.F90 =>
seq_map_mod.F90 =>
t_driver_timers_mod.F90 =>
seq_diag_mct.F90 =>
prep_wav_mod.F90 =>
seq_rest_mod.F90 =>
component_type_mod.F90 =>
prep_ice_mod.F90 =>
prep_ocn_mod.F90 =>
@check_includes
mpif.h mpif.h mpif.h shr_assert.h shr_assert.h shr_assert.h shr_assert.h shr_assert.h mpif.h
/nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/Tools/mkSrcfiles

 

[jedwards]

I don't see the link step here. I think that this is a retry log.
Do ./case.build --clean-all
./case.build
and then send the log.
Thanks

 

[sriki]

./case.build --clean-all
cleaning directory /rhome/sriki/cesm/scratch/test_scam_mpace/bld
[sriki@matrix test_scam_mpace]$ ./case.build
Building case in directory /nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace
sharedlib_only is False
model_only is False
Generating component namelists as part of build
Creating component namelists
Calling /nas/rstor/sriki/CESM/components/cam//cime_config/buildnml
...calling cam buildcpp to set build time options
CAM namelist copy: file1 /nas/rstor/sriki/CESM/cime/scripts/test_scam_mpace/Buildconf/camconf/atm_in file2 /rhome/sriki/cesm/scratch/test_scam_mpace/run/atm_in
Calling /nas/rstor/sriki/CESM/components/clm//cime_config/buildnml
WARNING: CLM is starting up from a cold state
Calling /nas/rstor/sriki/CESM/components/cice//cime_config/buildnml
...buildnml calling cice buildcpp to set build time options
Calling /nas/rstor/sriki/CESM/cime/src/components/data_comps/docn/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/srof/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/sglc/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/swav/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/components/stub_comps/sesp/cime_config/buildnml
Calling /nas/rstor/sriki/CESM/cime/src/drivers/mct/cime_config/buildnml
Finished creating component namelists
Building mpi-serial with output to file /rhome/sriki/cesm/scratch/test_scam_mpace/bld/mpi-serial.bldlog.200226-101734
Calling /nas/rstor/sriki/CESM/cime/src/build_scripts/buildlib.mpi-serial
Building gptl with output to file /rhome/sriki/cesm/scratch/test_scam_mpace/bld/gptl.bldlog.200226-101734
Calling /nas/rstor/sriki/CESM/cime/src/build_scripts/buildlib.gptl
Building mct with output to file /rhome/sriki/cesm/scratch/test_scam_mpace/bld/mct.bldlog.200226-101734
Calling /nas/rstor/sriki/CESM/cime/src/build_scripts/buildlib.mct
Building pio with output to file /rhome/sriki/cesm/scratch/test_scam_mpace/bld/pio.bldlog.200226-101734
Calling /nas/rstor/sriki/CESM/cime/src/build_scripts/buildlib.pio
Building csm_share with output to file /rhome/sriki/cesm/scratch/test_scam_mpace/bld/csm_share.bldlog.200226-101734
Calling /nas/rstor/sriki/CESM/cime/src/build_scripts/buildlib.csm_share
- Building clm4_5/clm5_0 Library
Building lnd with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/lnd.bldlog.200226-101734
clm built in 176.482466 seconds
Building atm with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/atm.bldlog.200226-101734
Building ice with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/ice.bldlog.200226-101734
Building ocn with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/ocn.bldlog.200226-101734
Building rof with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/rof.bldlog.200226-101734
Building glc with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/glc.bldlog.200226-101734
Building wav with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/wav.bldlog.200226-101734
Building esp with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/esp.bldlog.200226-101734
swav built in 2.609129 seconds
sglc built in 2.610837 seconds
srof built in 2.674324 seconds
sesp built in 2.668397 seconds
docn built in 4.454428 seconds
cam built in 193.234174 seconds
cice built in 193.233709 seconds
Building cesm with output to /rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.bldlog.200226-101734
Time spent not building: 2.911195 sec
Time spent building: 524.153681 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

cat /rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.bldlog.200226-101734


(File attached)

 

[jedwards]

It looks like you are attempting to use a netcdf library built for mpi with a serial build of the model.
/common/pkgs/netcdf/v4.4-f4.4.4-cxx4.2_mpi-pgi16.3_gcc_EL7//lib
You need to use a serial build of netcdf and of hdf5.

 

[sriki]

I recompiled with serial netcdf4,hdf5,hdf4. It now begins to run and then errors out.

test case: test_scam_mpace-- I have not made any changes to the code

ERROR: NetCDF: Attribute not found
NetCDF: Invalid dimension ID or name


cesm.log file is attached

I downloaded the input data cam3.1_64x128_T42_datasets.tar

-rw-r----- 1 sriki sriki 260945920 Apr 8 2005 cam3.1_64x128_T42_datasets.tar
drwxr-x--- 3 sriki sriki 3 Jan 13 14:05 lnd
drwxr-x--- 3 sriki sriki 3 Jan 13 14:05 share
drwxr-x--- 4 sriki sriki 4 Jan 13 14:06 atm
drwxr-x--- 3 sriki sriki 3 Jan 13 14:09 ocn

Is this the correct input tar file for the test_scam_mpace?

 

[jedwards]

Check the other output logs for the name of the file it is having trouble parsing. Also I am going to move this thread to the cam forum.

 

[sriki]

I do not see any error msg in other log files.
Am I using correct input data files for this test case?

-rw-r----- 1 sriki sriki 31003 Feb 27 11:10 cesm.log.200227-111037
-rw-r----- 1 sriki sriki 58255 Feb 27 11:10 cpl.log.200227-111037
-rw-r----- 1 sriki sriki 393915 Feb 27 11:10 atm.log.200227-111037
-rw-r----- 1 sriki sriki 3363 Feb 27 11:10 ocn.log.200227-111037
-rw-r----- 1 sriki sriki 86862 Feb 27 11:10 lnd.log.200227-111037
-rw-r----- 1 sriki sriki 10967 Feb 27 11:10 ice.log.200227-111037
-rw-r----- 1 sriki sriki 66668 Feb 27 11:10 test_scam_mpace.cam.h0.2004-10-05-07171.nc (no data written)
ncdump -h test_scam_mpace.cam.h0.2004-10-05-07171.nc
netcdf test_scam_mpace.cam.h0.2004-10-05-07171 {
dimensions:
lat = 1 ;
lon = 1 ;
time = UNLIMITED ; // (0 currently)
nbnd = 2 ;
chars = 8 ;
lev = 32 ;
ilev = 33 ;

I've cut-n-paste last 10 lines of each log file below:

tail -10 cpl.log.200227-111037
(prep_atm_merge) x2a%Faxx_evap = + lfrac*l2x%Fall_evap + ifrac*i2x%Faii_evap + ofrac*xao%Faox_evap
(prep_atm_merge) x2a%Fall_flxdst1 = = l2x%Fall_flxdst1 = lfrac*l2x%Fall_flxdst1
(prep_atm_merge) x2a%Fall_flxdst2 = = l2x%Fall_flxdst2 = lfrac*l2x%Fall_flxdst2
(prep_atm_merge) x2a%Fall_flxdst3 = = l2x%Fall_flxdst3 = lfrac*l2x%Fall_flxdst3
(prep_atm_merge) x2a%Fall_flxdst4 = = l2x%Fall_flxdst4 = lfrac*l2x%Fall_flxdst4
(seq_mct_drv) : Calling atm_init_mct phase 2
(component_init_cc:mct) : Initialize component atm
(seq_mct_drv) : Model initialization complete
[sriki@matrix run]$ tail -10 atm.log.200227-111037
WSHIST: nhfil( 1 )=test_scam_mpace.cam.h0.2004-10-05-07171.nc
Opening netcdf history file test_scam_mpace.cam.h0.2004-10-05-07171.nc
Opened file test_scam_mpace.cam.h0.2004-10-05-07171.nc to write 2818048
H_DEFINE: Successfully opened netcdf file
Creating new decomp: 1!32!1!!1!1!32!!d6!i2!
Creating new decomp: 1!1!!1!1!!d6!i2!
Creating new decomp: 1!33!1!!1!1!33!!d6!i2!
Creating new decomp: 1!32!1!!1!1!32!!d6!i1!
nstep, te 1 0.25321354140468302E+10 0.25319962133052936E+10 -0.57000269599935382E-02 0.10000756250000000E+06
[sriki@matrix run]$ tail -10 ocn.log.200227-111037
(shr_strdata_print) ----------------------------------------------------------
(docn_comp_init) initialize gsmaps
(docn_comp_init) copy domains
(docn_comp_init) allocate AVs
(shr_dmodel_readstrm) open : /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc
(shr_dmodel_readstrm) file lb: /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc 9
(shr_dmodel_readstrm) file ub: /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc 10
(docn_comp_run) ocn: model date 20041005 7171s
(docn_comp_init) docn_comp_init done
(docn_comp_init) docn_comp_init done
[sriki@matrix run]$ tail -10 lnd.log.200227-111037
(GETFIL): using
/rhome/sriki/cesm/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_64x128_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc
Opened existing file
/rhome/sriki/cesm/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_64x128_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc
2818048
Successfully read monthly vegetation data for
month 10
clm: completed timestep 0
clm: completed timestep 1
[sriki@matrix run]$ tail -10 ice.log.200227-111037
(shr_strdata_print) mapread ( 1) = NOT_SET
(shr_strdata_print) mapwrit ( 1) = NOT_SET
(shr_strdata_print) tintalgo( 1) = linear
(shr_strdata_print) readmode( 1) = single
(shr_strdata_print)
(shr_strdata_print) nvectors = 0
(shr_strdata_print) ----------------------------------------------------------
(shr_dmodel_readstrm) open : /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc
(shr_dmodel_readstrm) file lb: /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc 9
(shr_dmodel_readstrm) file ub: /rhome/sriki/cesm/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_2000climo_c180511.nc 10

 

[John Truesdale]

Hi:

You're right there's absolutely nothing there to go on. The model just looks like it stops cold after 1 timestep. Sometimes there will be an error message a little higher up in the atm.log file, can you send the entire log file. I would also suggest another run after recompiling the model with debug options on. Hopefully, that will give us more information. From inside your case directory type
./xmlchange --id DEBUG --val "TRUE"
./*.build --clean-all
./*.build
./*.submit

jt

 

[cacraig]

If after following John's advice your debug run still doesn't give an error message, we will need additional information from you on your run's specifics. Please see:
https://xenforo.cgd.ucar.edu/cesm/threads/information-to-include-in-help-requests.4974/ (This link is at the very top of the main Forum's page)

Also, if you are not aware of it, there are details on how to run SCAM in the CAM6 User's Guide at:

4. Atmospheric configurations (compsets) — camdoc documentation

ncar.github.io

 

[sriki]

Followed the instructions given by John:
./xmlchange --id DEBUG --val "TRUE"
./*.build --clean-all
./*.build
./*.submit

Attached all log files from the run

 

[John Truesdale]

Thanks Sriki. I'm not sure what is behind the lack of error messages in the logs which makes your case a little bit of a challenge.

We've had very bad luck with finding stable versions of PGI releases that work with CESM. That's an issue we don't see with other compilers and something that could be at the root of your problem. It could also be a mismatch in the libraries you are using for compiling and linking. Getting to the bottom of these issues can be tedious and long. Do you have other compilers to work with intel, nag, or gnu? We may be able to get rid of the problem by just using any other compiler. Intel seems to be a very good option if you have that available.

Could you send me the output of typing

modules avail

Also could you let me know the version of code you are using? Is it the most recent release of CESM? It certainly would help to be using an up to date version of CESM. And would you send along the README.case file in your case directory so I can see how you are configuring the run.

Lastly I would just ask that you check that scam isn't running in a directory where you are hitting a quota as it is writing out its model history? I sometimes see logs being truncated by the OS when hitting a filesystem quota.

jt

 

[sriki]

John,
Thank you!

module list
Currently Loaded Modulefiles:
1) null 3) mpich/3.2-pgi16.3(default) 5) hdf5/v1.8.17_mpi-pgi16.3_gcc(default)
2) pgi/16.3(default) 4) hdf4/v2.11_mpi-pgi16.3_gcc(default) 6) netcdf/4.4.0_mpi-pgi16.3_gcc(default)

Tag
Tag name: cesm2.1.1-rc.05

README.case
2020-02-27 10:56:31: ./create_newcase --case test_scam_mpace --compset FSCAM --res T42_T42 --user-mods-dir ../../components/cam/cime_config/usermods_dirs/scam_mpace --mach centos7-linux
---------------------------------------------------
2020-02-27 10:56:31: Compset longname is 2000_CAM60%SCAM_CLM50%SP_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV
---------------------------------------------------
2020-02-27 10:56:31: Compset specification file is /nas/rstor/sriki/CESM/components/cam//cime_config/config_compsets.xml
---------------------------------------------------
2020-02-27 10:56:31: Pes specification file is /nas/rstor/sriki/CESM/components/cam//cime_config/config_pes.xml
---------------------------------------------------
2020-02-27 10:56:31: Forcing is 1972-2004
---------------------------------------------------
2020-02-27 10:56:31: Using None coupler instances
---------------------------------------------------
2020-02-27 10:56:31: Component ATM is CAM cam6 physics:
---------------------------------------------------
2020-02-27 10:56:31: ATM_GRID is T42
---------------------------------------------------
2020-02-27 10:56:31: This component includes user_mods /nas/rstor/sriki/CESM/components/cam/cime_config/usermods_dirs/scam_mandatory
---------------------------------------------------
2020-02-27 10:56:31: Component LND is clm5.0:Satellite phenology:
---------------------------------------------------
2020-02-27 10:56:31: LND_GRID is T42
---------------------------------------------------
2020-02-27 10:56:31: Component ICE is Sea ICE (cice) model version 5 :prescribed cice
---------------------------------------------------
2020-02-27 10:56:31: ICE_GRID is T42
---------------------------------------------------
2020-02-27 10:56:31: Component OCN is DOCN prescribed ocean mode
---------------------------------------------------
2020-02-27 10:56:31: OCN_GRID is T42
---------------------------------------------------
2020-02-27 10:56:31: Component ROF is Stub river component
---------------------------------------------------
2020-02-27 10:56:31: ROF_GRID is null
---------------------------------------------------
2020-02-27 10:56:31: Component GLC is Stub glacier (land ice) component
---------------------------------------------------
2020-02-27 10:56:31: GLC_GRID is null
---------------------------------------------------
2020-02-27 10:56:31: Component WAV is Stub wave component
---------------------------------------------------
2020-02-27 10:56:31: WAV_GRID is null
---------------------------------------------------
2020-02-27 10:56:31: ESP_GRID is None
---------------------------------------------------

I also tried submitting a slurm job to run the executable : with 1 hr time limit & 2 GB RAM but it keeps running until it hits 1hr limit with netcdf file not growing
66668 Mar 2 09:17 test_scam_mpace.cam.h0.2004-10-05-07171.nc & also other log files
#SBATCH -p shared
#SBATCH --ntasks 1
### total run time estimate (D-HH:MM)
#SBATCH -t 0-01:00
### memory (GB per CPU)
#SBATCH --mem-per-cpu=2G
#SBATCH --mail-type=END,FAIL
### Ouput files
#SBATCH -o slurm-%j.out # STDOUT
#SBATCH -e slurm-%j.err # STDERR
ulimit -a unlimited
/rhome/sriki/cesm/scratch/test_scam_mpace/bld/cesm.exe

No disk quota enabled

 

[John Truesdale]

Thanks Sriki: I downloaded your cesm version and gave it a go on our machines, first using our PGI compiler 19.0.2 which died with a segfault and then same code with gnu (8.2.0) and that worked. I'm assuming intel works as well but I will run your case with that too. If you have another compiler you can try I think that would be best.

Typing 'module avail' should list what compilers are available to you.

jt

 

[sriki]

John,

There's another version of the PGI compiler (19.1) but I don't have the other supporting libraries hdf4/hdf5/netcdf4 built with the same compiler. GNU compilers are 4.8.5. Not sure if it will work with GNU compilers

gfortran --version
GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Copyright (C) 2015 Free Software Foundation, Inc.
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=Red Hat Bugzilla Main Page --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)

 

[John Truesdale]

Hi Sriki:

I mistyped the PGI version I tried with your release version, it was PGI 19.9 not 19.0.2.

We also had GNU 4.8.5 and I gave it a shot with your version of CESM and there were compiler errors for the land component that uses f2003 compliant Fortran (automatic allocation of strings). I assume there would be many more errors than the few routines I saw since our framework uses more modern Fortran features. You would have to use a later version of GNU. Even still I think the end goal of getting something running would be easier if you worked on getting an updated version of GNU installed along with compiling the other libraries (netcdf etc) with GNU than trying to go with PGI.

jt

 

[sriki]

we loaded gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3).

What version of netcdf4/hdf5 are you using?