Main menu

Navigation

cesm1_2_2_CAMChem Running Issue - NetCDF: Invalid dimension ID or name

22 posts / 0 new
Last post
vineetm@...
cesm1_2_2_CAMChem Running Issue - NetCDF: Invalid dimension ID or name

Hi,

I have compiled the application successfully but getting errors while running the application.

PLEASE REFER THE ATTACHMENT FOR THE SAME FORMATTED MAIL FOR BETTER VISUALIZATION (Was not able to format this correctly)

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here are the configurations I have used for building the application:

1) gcc-7.2.0

    binutils-2.31

    coreutils-8.30

    perl-5.10.1

2) pgi-18.4

3) zlib-1.2.11

4) mpich-3.2.1

5) curl-7.61.1

6) szip-2.1.1

7) hdf5-1.10.4 (parallel build)

8) parallel-netcdf-1.10.0

9) netcdf-4.6.1

10) netcdf-fortran-4.4.4

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Note: In CESM code, I have performed the following changes:

1) Updated pio SVN repo link

In the top level SVN_EXTERNAL_DIRECTORIES change:

-models/utils/pio                   http://parallelio.googlecode.com/svn/trunk_tags/pio1_8_12/pio

+models/utils/pio                   https://github.com/NCAR/ParallelIO.git/tags/pio1_8_12/pio

In the tools/cprnc/SVN_EXTERNAL_DIRECTORIES change:

-genf90 http://parallelio.googlecode.com/svn/genf90/trunk_tags/genf90_140121

+genf90 https://github.com/PARALLELIO/genf90/tags/genf90_140121

 

After each change run:

svn propset svn:externals -F SVN_EXTERNAL_DIRECTORIES .

svn update

 2) Patched FindNETCDF.cmake (Reference: https://bb.cgd.ucar.edu/sites/default/files/vbtodrupal/find_netcdf_patch...)

3) Change all the references of variables with dimension "PIO_MAX_VAR_DIMS" from static to dynamic allocation.

This was done since the variables with dimension "PIO_MAX_VAR_DIMS" produced error with static allocation. (Without the change I had to use -mcmodel=medium flag which i didn't want)

#Following are the list of files:

cesm1_2_2_CAMChem/models/utils/pio/piodarray.F90

cesm1_2_2_CAMChem/models/utils/pio/piodarray.F90.in

cesm1_2_2_CAMChem/models/utils/pio/pionfwrite_mod.F90

cesm1_2_2_CAMChem/models/utils/pio/pio_types.F90

cesm1_2_2_CAMChem/models/utils/pio/pionfget_mod.F90.in

cesm1_2_2_CAMChem/models/utils/pio/pionfget_mod.F90

cesm1_2_2_CAMChem/models/utils/pio/pionfread_mod.F90.in

cesm1_2_2_CAMChem/models/utils/pio/pionfread_mod.F90

cesm1_2_2_CAMChem/models/utils/pio/pionfwrite_mod.F90.in

cesm1_2_2_CAMChem/models/utils/pio/pio_msg_callbacks.F90

cesm1_2_2_CAMChem/models/atm/cam/src/physics/cam/radae.F90

cesm1_2_2_CAMChem/models/atm/cam/src/utils/cam_dom/ocn_comp.F90

 

cesm1_2_2_CAMChem/models/atm/cam/src/control/ncdio_atm.F90

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Following is the Macros file:


#

# Makefile Macros generated from /scratch/vineetm/CESM/shared/cesm1_2_2_CAMChem/scripts/ccsm_utils/Machines/config_compilers.xml using

# COMPILER=pgi

# OS=LINUX

# MACH=userdefined

#

CPPDEFS+= -DFORTRANUNDERSCORE -DNO_SHR_VMATH -DNO_R16   -DLINUX -DCPRPGI

SLIBS+=-L/scratch/vineetm/CESM/shared/deps/lib -lnetcdff -lnetcdf -lpnetcdf -lhdf5_hl -lhdf5 -lhdf5hl_fortran -lhdf5_fortran -lcurl -lsz -lz

# USERDEFINED $(shell $(NETCDF_PATH)/bin/nc-config --flibs)

CFLAGS:= -O0 -gopt -Mlist -time -fPIC -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

CONFIG_ARGS:=

CXX_LINKER:=CXX

ESMF_LIBDIR:=

FC_AUTO_R8:= -r8

FFLAGS:=  -O0 -i4 -gopt -Mlist -time -Mextend -byteswapio -Mflushz -Kieee  -fPIC -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

FIXEDFLAGS:= -Mfixed

FREEFLAGS:= -Mfree

LDFLAGS:= -time -Wl,--allow-multiple-definition

MPICC:= mpicc

MPICXX:= mpicxx

MPIFC:= mpifort

MPI_LIB_NAME:=

MPI_PATH:=/scratch/vineetm/CESM/shared/deps

NETCDF_PATH:=/scratch/vineetm/CESM/shared/deps

PNETCDF_PATH:=/scratch/vineetm/CESM/shared/deps

SCC:= pgcc

SCXX:= pgc++

SFC:= pgf95

SUPPORTS_CXX:=FALSE

ifeq ($(DEBUG), TRUE)

   FFLAGS += -O0 -g -Ktrap=fp -Mbounds -Kieee

endif

ifeq ($(compile_threaded), false)

   LDFLAGS += -nomp

   CFLAGS += -nomp

   FFLAGS += -nomp

endif

ifeq ($(compile_threaded), true)

   LDFLAGS += -mp

   CFLAGS += -mp

   FFLAGS += -mp

endif

ifeq ($(MODEL), dwav)

   FFLAGS += -Mnovect

endif

ifeq ($(MODEL), dlnd)

   FFLAGS += -Mnovect

endif

ifeq ($(MODEL), drof)

   FFLAGS += -Mnovect

endif

ifeq ($(MODEL), pop2)

   CPPDEFS += -D_USE_FLOW_CONTROL

endif

ifeq ($(MODEL), dice)

   FFLAGS += -Mnovect

endif

ifeq ($(MODEL), docn)

   FFLAGS += -Mnovect

endif

ifeq ($(MODEL), datm)

   FFLAGS += -Mnovect

endif

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Following is the env_mach_specific file:


#! /bin/csh -f

 

# -------------------------------------------------------------------------

# USERDEFINED

# Edit this file to add module load or other paths needed for the build

# and run on the system.  Can also include general env settings for machine.

# Some samples are below

# -------------------------------------------------------------------------

 

#source /opt/modules/default/init/csh

#if ( $COMPILER == "pgi" ) then

#  module load pgi

#endif

#module load netcdf

 

limit coredumpsize unlimited

limit stacksize unlimited

setenv CESM_PREFIX /scratch/vineetm/CESM/shared

 

#GNU

setenv PATH $CESM_PREFIX/gnu/bin:$PATH

setenv LD_LIBRARY_PATH $CESM_PREFIX/gnu/lib:$CESM_PREFIX/gnu/lib64:$LD_LIBRARY_PATH

setenv LD_RUN_PATH $CESM_PREFIX/gnu/lib:$CESM_PREFIX/gnu/lib64

setenv MANPATH $CESM_PREFIX/gnu/share/man:$MANPATH

 

#PGI environment

setenv pgihome $CESM_PREFIX/pgi

setenv pgidir $pgihome/linux86-64/18.4

setenv PGI $pgihome

setenv CC $pgidir/bin/pgcc

setenv FC $pgidir/bin/pgfortran

setenv F90 $pgidir/bin/pgf90

setenv F77 $pgidir/bin/pgf77

setenv CPP "$pgidir/bin/pgcc -Mcpp"

setenv CXX $pgidir/bin/pgc++

setenv PATH $pgidir/bin:$PATH

setenv MANPATH $pgidir/man:$MANPATH

setenv LD_LIBRARY_PATH $pgidir/lib:$LD_LIBRARY_PATH

 

#PGI openmpi environment

setenv pgilibdir $pgihome/linux86-64/2018/mpi/openmpi-2.1.2

setenv PGI_OPTL_LIB_DIRS "$pgilibdir/lib"

setenv PGI_OPTL_INCLUDE_DIRS "$pgilibdir/include"

setenv PATH $pgilibdir/bin:$PATH

setenv MANPATH $pgilibdir/share/man:$MANPATH

setenv LD_LIBRARY_PATH $pgilibdir/lib:$LD_LIBRARY_PATH

 

#DEPS

setenv HDF5_ROOT $CESM_PREFIX/deps

setenv PATH $CESM_PREFIX/deps/bin:$PATH

setenv LD_LIBRARY_PATH $CESM_PREFIX/deps/lib:$LD_LIBRARY_PATH

setenv MANPATH $CESM_PREFIX/deps/share/man:$MANPATH

setenv PKG_CONFIG_PATH $CESM_PREFIX/deps/lib/pkgconfig

setenv LD_RUN_PATH $CESM_PREFIX/deps/lib:$LD_RUN_PATH

 

setenv LANG C

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Following is the case that i am setting up: COMPSET=B_1850_CN RES=0.9x1.25_gx1v6 MACH=userdefined CCSMROOT=/scratch/vineetm/CESM/shared/cesm1_2_2_CAMChem CASE=SCASE0 CASEROOT=/scratch/vineetm/CESM/shared/MYCASES EXEROOT=/scratch/vineetm/CESM/shared/MYCASES/SCASE0/EXEC RUNDIR=/scratch/vineetm/CESM/shared/MYCASES/SCASE0/EXEC/run  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  Following is the error that I am getting while running: (seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =   128) ( nthreads =  1) (seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =   128) ( nthreads =  1) seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0   127     1 ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =   128) ( nthreads =  1) (seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =   128) ( nthreads =  1) (seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =   128) ( nthreads =  1) (seq_comm_printcomms)     1     0   128     1  GLOBAL: (seq_comm_printcomms)     2     0   128     1  CPL: (seq_comm_printcomms)     3     0   128     1  ALLATMID: (seq_comm_printcomms)     4     0   128     1  ALLLNDID: (seq_comm_printcomms)     5     0   128     1  ALLOCNID: (seq_comm_printcomms)     6     0   128     1  ALLICEID: (seq_comm_printcomms)     7     0   128     1  ALLGLCID: (seq_comm_printcomms)     8     0   128     1  ALLROFID: (seq_comm_printcomms)     9     0   128     1  ALLWAVID: (seq_comm_printcomms)    10     0   128     1  CPLALLATMID: (seq_comm_printcomms)    11     0   128     1  CPLALLLNDID: (seq_comm_printcomms)    12     0   128     1  CPLALLOCNID: (seq_comm_printcomms)    13     0   128     1  CPLALLICEID: (seq_comm_printcomms)    14     0   128     1  CPLALLGLCID: (seq_comm_printcomms)    15     0   128     1  CPLALLROFID: (seq_comm_printcomms)    16     0   128     1  CPLALLWAVID: (seq_comm_printcomms)    17     0   128     1  ATM: (seq_comm_printcomms)    18     0   128     1  CPLATM: (seq_comm_printcomms)    19     0   128     1  LND: (seq_comm_printcomms)    20     0   128     1  CPLLND: (seq_comm_printcomms)    21     0   128     1  OCN: (seq_comm_printcomms)    22     0   128     1  CPLOCN: (seq_comm_printcomms)    23     0   128     1  ICE: (seq_comm_printcomms)    24     0   128     1  CPLICE: (seq_comm_printcomms)    25     0   128     1  GLC: (seq_comm_printcomms)    26     0   128     1  CPLGLC: (seq_comm_printcomms)    27     0   128     1  ROF: (seq_comm_printcomms)    28     0   128     1  CPLROF: (seq_comm_printcomms)    29     0   128     1  WAV: (seq_comm_printcomms)    30     0   128     1  CPLWAV: (t_initf) Read in prof_inparm namelist from: drv_in 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1019.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.51 8 MB memory   alloc in MB is             8.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1019.02 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1019.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1019.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1050.15 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1027.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1027.01 seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.79 8 MB memory   alloc in MB is             8.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.51 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1050.15 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.53 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1027.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.53 8 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1030.04   8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1049.63 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1019.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.79 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.02 8 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1026.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1030.04 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.53 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1049.63 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1050.15 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1030.54   8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50   8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1029.53 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1027.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1026.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1030.54 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1026.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1023.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1030.04 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1022.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1031.05 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1021.01 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1052.26 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1053.32 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.73 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1028.52 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1051.20 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1024.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1026.51 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.50 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1020.02 8 MB memory   alloc in MB is             8.00 8 MB memory dealloc in MB is             0.00 Memory block size conversion in bytes is          1025.00 seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in seq_flds_mod: seq_flds_a2x_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diag seq_flds_mod: seq_flds_a2x_fluxes= Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphi                                                                                                                                                                                                                                                 dry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4 seq_flds_mod: seq_flds_x2a_states= Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustar seq_flds_mod: seq_flds_x2a_fluxes= Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4 seq_flds_mod: seq_flds_l2x_states= Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10 seq_flds_mod: seq_flds_l2x_fluxes= Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_rofice seq_flds_mod: seq_flds_x2l_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diag seq_flds_mod: seq_flds_x2l_fluxes= Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_oc                                                                                                                                                                                                                                                 phodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_flood seq_flds_mod: seq_flds_i2x_states= Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifrac seq_flds_mod: seq_flds_i2x_fluxes= Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_salt seq_flds_mod: seq_flds_x2i_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdy seq_flds_mod: seq_flds_x2i_fluxes= Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Fax                                                                                                                                                                                                                                                  a_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_q seq_flds_mod: seq_flds_o2x_states= So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepth seq_flds_mod: seq_flds_o2x_fluxes= Fioo_q seq_flds_mod: seq_flds_x2o_states= Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes seq_flds_mod: seq_flds_x2o_fluxes= Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:                                                                                                                                                                                                                                                 Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fio                                                                                                                                                                                                                                                 i_salt:Forr_roff:Forr_ioff seq_flds_mod: seq_flds_s2x_states=   seq_flds_mod: seq_flds_s2x_fluxes=   seq_flds_mod: seq_flds_x2s_states=   seq_flds_mod: seq_flds_x2s_fluxes=   seq_flds_mod: seq_flds_g2x_states=   seq_flds_mod: seq_flds_g2x_fluxes=   seq_flds_mod: seq_flds_x2g_states=   seq_flds_mod: seq_flds_x2g_fluxes=   seq_flds_mod: seq_flds_xao_states= So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustar seq_flds_mod: seq_flds_xao_albedo= So_avsdr:So_anidr:So_avsdf:So_anidf seq_flds_mod: seq_flds_r2x_states= Slrr_volr seq_flds_mod: seq_flds_r2x_fluxes= Forr_roff:Forr_ioff:Flrr_flood seq_flds_mod: seq_flds_x2r_states=   seq_flds_mod: seq_flds_x2r_fluxes= Flrl_rofliq:Flrl_rofice seq_flds_mod: seq_flds_w2x_states= Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes seq_flds_mod: seq_flds_w2x_fluxes=   seq_flds_mod: seq_flds_x2w_states= Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepth seq_flds_mod: seq_flds_x2w_fluxes=             128 pes participating in computation  -----------------------------------  TASK#  NAME   0  cn00   1  cn00   2  cn00   3  cn00   4  cn00   5  cn00   6  cn00   7  cn00   8  cn00   9  cn00  10  cn00  11  cn00  12  cn00  13  cn00  14  cn00  15  cn00  16  cn00  17  cn00  18  cn00  19  cn00  20  cn00  21  cn00  22  cn00  23  cn00  24  cn00  25  cn00  26  cn00  27  cn00  28  cn01  29  cn01  30  cn01  31  cn01  32  cn01  33  cn01  34  cn01  35  cn01  36  cn01  37  cn01  38  cn01  39  cn01  40  cn01  41  cn01  42  cn01  43  cn01  44  cn01  45  cn01  46  cn01  47  cn01  48  cn01  49  cn01  50  cn01  51  cn01  52  cn01  53  cn01  54  cn01  55  cn01  56  cn02  57  cn02  58  cn02  59  cn02  60  cn02  61  cn02  62  cn02  63  cn02  64  cn02  65  cn02  66  cn02  67  cn02  68  cn02  69  cn02  70  cn02  71  cn02  72  cn02  73  cn02  74  cn02  75  cn02  76  cn02  77  cn02  78  cn02  79  cn02  80  cn02  81  cn02  82  cn02  83  cn02  84  cn03  85  cn03  86  cn03  87  cn03  88  cn03  89  cn03  90  cn03  91  cn03  92  cn03  93  cn03  94  cn03  95  cn03  96  cn03  97  cn03  98  cn03  99  cn03 100  cn03 101  cn03 102  cn03 103  cn03 104  cn03 105  cn03 106  cn03 107  cn03 108  cn03 109  cn03 110  cn03 111  cn03 112  cn04 113  cn04 114  cn04 115  cn04 116  cn04 117  cn04 118  cn04 119  cn04 120  cn04 121  cn04 122  cn04 123  cn04 124  cn04 125  cn04 126  cn04 127  cn04  Opened existing file b40.1850.track1.1deg.006.cam.i.0863-01-01-00000.nc         65536  Opened existing file  /scratch/vineetm/CESM/DATA/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc         131072  NetCDF: Invalid dimension ID or name   =================================================================================== =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES =   PID 78992 RUNNING AT cn00 =   EXIT CODE: 9 =   CLEANING UP REMAINING PROCESSES =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:2@cn02] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed [proxy:0:2@cn02] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:2@cn02] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event [proxy:0:3@cn03] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed [proxy:0:3@cn03] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:3@cn03] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event [proxy:0:4@cn04] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed [proxy:0:4@cn04] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:4@cn04] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event [proxy:0:1@cn01] srun: error: cn03: task 3: Exited with exit code 7 srun: error: cn02: task 2: Exited with exit code 7 srun: error: cn01: task 1: Exited with exit code 7 HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed [proxy:0:1@cn01] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@cn01] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event srun: error: cn04: task 4: Exited with exit code 7 [mpiexec@cn00] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@cn00] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@cn00] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@cn00] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion Wed Nov 14 16:22:34 IST 2018 -- CSM EXECUTION HAS FINISHED ls: No match. 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

The dependencies have been well tested. Not sure what the problem is. Please let me know what should be the changes required.

 

Thanks,

Vineet More

Attachment: 
jedwards

The problem isn't evident in the fies you have provided.   You might try setting DEBUG=TRUE and recompiliing to see if you can get further information.

vineetm@...

Hello jedwards,

Thanks for the quick reply. Please let me know how to set DEBUG=TRUE.

The steps that I performed were:

1) ./cesm_setup -clean

2) ./xmlchange -file env_build.xml -id DEBUG -val TRUE (Refer : http://www.cesm.ucar.edu/models/cesm1.0/clm/models/lnd/clm/doc/UsersGuid...)

3) And finally <CASE>.build (without modifying the Macros and env_mach_specific file since the contents were same)

 

But I did't get any change in the output. Please guide me how to set the DEBUG flag correctly.

I have also attached the output log, although no changes were observed.

 

Thanks,

Vineet More

jedwards

RUn <CASE>.clean_build before you run build.   This issue looks like you may be running out of memory - you might try a simpler case (An F case and the B without chem) and see if those work.

vineetm@...

Hi jedwards,

 

According to what I observed, the "NetCDF: Invalid dimension ID or name" doesn't seem to be an issue.

I set up a new case with the following compset and resolution:

export COMPSET=F1850CNCHM

export RES=f02_t12

export MACH=userdefined

export CCSMROOT=$CESM_PREFIX/cesm1_2_2_CAMChem

export CASE=SCASE1

export CASEROOT=$CESM_PREFIX/MYCASES

export EXEROOT=$CESM_PREFIX/MYCASES/$CASE/EXEC

export RUNDIR=$CESM_PREFIX/MYCASES/$CASE/EXEC/run

 

And this is what I observed on monitoring the mpi processes using "top" command:

1) The model runs for a while (approx 3-5 minutes)

2) Then suddenly the memory of each process increases.

3) Until (for my cluster), it reached to 41... GB per process and the application crashed, since that much amount of memory is not available.

 

So the issue doesn't seem to be related to any incorrect installation of netcdf or incorrect build of the apllication.

Just one thing more, please suggest me the smallest resolution and related compset so that I can test atleast 1 successful run of the application.

I am not a domain expert, thus, having trouble in deciding the models. It would be nice if you suggest one.

Note: The cluster is having 12 nodes with approx 128GB per node.

 

Thanks,

Vineet More

 

jedwards

RES=f02_t12 is an extremely high resolution grid and will not be suitable for your system.  I would try f19_g16 first and f09_g16 if that is successful.  

vineetm@...

Hi jedwards,

This is what I have tried:

Created a new case with: COMPSET=F1850CNCHM & RES=f19_g16

And what I observed was that while having "mpirun -np 64 -map-by ppr:16:node $EXEROOT/cesm.exe" (doing this with interactive access with the correct files sourced. I just get access to the nodes using srun, then i enter into csh, then I source env_mach_specific and then I source <CASE>.run to observe whats happening), it didn't create 16 procs per node. It created 28 per node. And the same issue as previously mentioned occured. The processes try to use approx 40gb each, the RAM runs out of memory and the application terminates.

So let me know if the COMPSET & RES still to high ( please suggest smaller versions of both) or whether I need to run the problem on a larger cluster?

Also, I am in progress of creating a case with COMPSET=F1850CNCHM & RES=f09_g16, but not sure whether this will run.

[Update : Tried with COMPSET=F1850CNCHM & RES=f09_g16, observed same issue. Out of RAM]

Thanks,

Vineet More

jedwards

Is cesm 2 an option for you?    Can you try f19_g16? 

vineetm@...

Sure, will port cesm 2 on our system, if that is an option. Just want to be sure that, can I do everything in cesm 2 that was possible in cesm1.2.2camchem ?

vineetm@...

Hi,

Well, I didn't shift to cesm 2. Was experimenting with cesm_1_2_2 cam chem.

Created a case using:

create_newcase -case <path> -res T31_g37 -compset F1850CNCHM -mach <mach>

This is a way too much small case I hope.

 

And this is the error [ Specifically0: ALLOCATE: 8589934588 bytes requested; not enough memory] that I am getting.

Can you please suggest what is going wrong here? Or what kind of memory does it require more? Stacksize, heap size? Or is there any alternative?

 

 

(seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =    16) ( nthreads =  1)

(seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0    15     1 ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =    16) ( nthreads =  1)

(seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =    16) ( nthreads =  1)

(seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =    16) ( nthreads =  1)

(seq_comm_printcomms)     1     0    16     1  GLOBAL:

(seq_comm_printcomms)     2     0    16     1  CPL:

(seq_comm_printcomms)     3     0    16     1  ALLATMID:

(seq_comm_printcomms)     4     0    16     1  ALLLNDID:

(seq_comm_printcomms)     5     0    16     1  ALLOCNID:

(seq_comm_printcomms)     6     0    16     1  ALLICEID:

(seq_comm_printcomms)     7     0    16     1  ALLGLCID:

(seq_comm_printcomms)     8     0    16     1  ALLROFID:

(seq_comm_printcomms)     9     0    16     1  ALLWAVID:

(seq_comm_printcomms)    10     0    16     1  CPLALLATMID:

(seq_comm_printcomms)    11     0    16     1  CPLALLLNDID:

(seq_comm_printcomms)    12     0    16     1  CPLALLOCNID:

(seq_comm_printcomms)    13     0    16     1  CPLALLICEID:

(seq_comm_printcomms)    14     0    16     1  CPLALLGLCID:

(seq_comm_printcomms)    15     0    16     1  CPLALLROFID:

(seq_comm_printcomms)    16     0    16     1  CPLALLWAVID:

(seq_comm_printcomms)    17     0    16     1  ATM:

(seq_comm_printcomms)    18     0    16     1  CPLATM:

(seq_comm_printcomms)    19     0    16     1  LND:

(seq_comm_printcomms)    20     0    16     1  CPLLND:

(seq_comm_printcomms)    21     0    16     1  OCN:

(seq_comm_printcomms)    22     0    16     1  CPLOCN:

(seq_comm_printcomms)    23     0    16     1  ICE:

(seq_comm_printcomms)    24     0    16     1  CPLICE:

(seq_comm_printcomms)    25     0    16     1  GLC:

(seq_comm_printcomms)    26     0    16     1  CPLGLC:

(seq_comm_printcomms)    27     0    16     1  ROF:

(seq_comm_printcomms)    28     0    16     1  CPLROF:

(seq_comm_printcomms)    29     0    16     1  WAV:

(seq_comm_printcomms)    30     0    16     1  CPLWAV:

 (t_initf) Read in prof_inparm namelist from: drv_in

8 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

Memory block size conversion in bytes is          1023.50

8 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

Memory block size conversion in bytes is          1023.50

8 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

8 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1023.508 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

Memory block size conversion in bytes is          1023.50

 

8 MB memory   alloc in MB is             8.00

Memory block size conversion in bytes is          1023.50

8 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

Memory block size conversion in bytes is          1023.50

 

 

8 MB memory dealloc in MB is             0.00

8 MB memory dealloc in MB is             0.008 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

8 MB memory   alloc in MB is             8.00

8 MB memory   alloc in MB is             8.00Memory block size conversion in bytes is          1023.50

Memory block size conversion in bytes is          1023.50

8 MB memory dealloc in MB is             0.008 MB memory   alloc in MB is             8.00

Memory block size conversion in bytes is          1023.508 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

 

8 MB memory dealloc in MB is             0.00

 

 

Memory block size conversion in bytes is          1023.508 MB memory dealloc in MB is             0.00

 

8 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1023.50

 

Memory block size conversion in bytes is          1023.50

 

Memory block size conversion in bytes is          1023.50Memory block size conversion in bytes is          1023.50

 

 

seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in

seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in

8 MB memory   alloc in MB is             8.00

8 MB memory dealloc in MB is             0.00

Memory block size conversion in bytes is          1023.50

seq_flds_mod: seq_flds_a2x_states=

Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diag

seq_flds_mod: seq_flds_a2x_fluxes=

Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4

seq_flds_mod: seq_flds_x2a_states=

Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustar:Sl_dd001:Sl_dd002:Sl_dd003:Sl_dd004:Sl_dd005:Sl_dd006:Sl_dd007:Sl_dd008:Sl_dd009:Sl_dd010

seq_flds_mod: seq_flds_x2a_fluxes=

Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Fall_voc001

seq_flds_mod: seq_flds_l2x_states=

Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10:Sl_dd001:Sl_dd002:Sl_dd003:Sl_dd004:Sl_dd005:Sl_dd006:Sl_dd007:Sl_dd008:Sl_dd009:Sl_dd010

seq_flds_mod: seq_flds_l2x_fluxes=

Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_rofice:Fall_voc001

seq_flds_mod: seq_flds_x2l_states=

Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diag

seq_flds_mod: seq_flds_x2l_fluxes=

Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_flood

seq_flds_mod: seq_flds_i2x_states=

Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifrac

seq_flds_mod: seq_flds_i2x_fluxes=

Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_salt

seq_flds_mod: seq_flds_x2i_states=

Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdy

seq_flds_mod: seq_flds_x2i_fluxes=

Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_q

seq_flds_mod: seq_flds_o2x_states=

So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepth

seq_flds_mod: seq_flds_o2x_fluxes=

Fioo_q

seq_flds_mod: seq_flds_x2o_states=

Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes

seq_flds_mod: seq_flds_x2o_fluxes=

Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fioi_salt:Forr_roff:Forr_ioff

seq_flds_mod: seq_flds_s2x_states=

 

seq_flds_mod: seq_flds_s2x_fluxes=

 

seq_flds_mod: seq_flds_x2s_states=

 

seq_flds_mod: seq_flds_x2s_fluxes=

 

seq_flds_mod: seq_flds_g2x_states=

 

seq_flds_mod: seq_flds_g2x_fluxes=

 

seq_flds_mod: seq_flds_x2g_states=

 

seq_flds_mod: seq_flds_x2g_fluxes=

 

seq_flds_mod: seq_flds_xao_states=

So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustar

seq_flds_mod: seq_flds_xao_albedo=

So_avsdr:So_anidr:So_avsdf:So_anidf

seq_flds_mod: seq_flds_r2x_states=

Slrr_volr

seq_flds_mod: seq_flds_r2x_fluxes=

Forr_roff:Forr_ioff:Flrr_flood

seq_flds_mod: seq_flds_x2r_states=

 

seq_flds_mod: seq_flds_x2r_fluxes=

Flrl_rofliq:Flrl_rofice

seq_flds_mod: seq_flds_w2x_states=

Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes

seq_flds_mod: seq_flds_w2x_fluxes=

 

seq_flds_mod: seq_flds_x2w_states=

Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepth

seq_flds_mod: seq_flds_x2w_fluxes=

 

           16 pes participating in computation

 -----------------------------------

 TASK#  NAME

  0  cn003

  1  cn177

  2  cn178

  3  cn183

  4  cn190

  5  cn194

  6  cn195

  7  cn196

  8  cn198

  9  cn202

 10  cn203

 11  cn207

 12  cn208

 13  gpu140

 14  cn227

 15  cn237

 Opened existing file

 /home/cas/faculty/dilipganguly/cesm/inputdata/atm/cam/inic/gaus/cami_0000-01-01_48x96_L26_c091218.nc

             0

 Opened existing file

 /home/cas/faculty/dilipganguly/cesm/inputdata/atm/cam/topo/USGS-gtopo30_48x96_c050520.nc

             1

 Divergence damper for spectral dycore NOT invoked

 Divergence damper for spectral dycore NOT invoked Time filter coefficient (EPS)                      0.060     Divergence damper for spectral dycore NOT invoked

 Time filter coefficient (EPS)                      0.060

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

Divergence damper for spectral dycore NOT invoked

 

Divergence damper for spectral dycore NOT invokedDivergence damper for spectral dycore NOT invoked  Divergence damper for spectral dycore NOT invoked Divergence damper for spectral dycore NOT invoked

 Time filter coefficient (EPS)                      0.060

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

Divergence damper for spectral dycore NOT invoked   Divergence damper for spectral dycore NOT invoked

 Time filter coefficient (EPS)                      0.060

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 Number of levels Courant limiter applied               5

 Divergence damper for spectral dycore NOT invoked

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 Number of levels Courant limiter applied               5

 Dynamics Subcycling                                    1

 

 Time filter coefficient (EPS)                      0.060 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 

Divergence damper for spectral dycore NOT invokedDivergence damper for spectral dycore NOT invoked

 Number of levels Courant limiter applied               5

 Dynamics Subcycling                                    1

 

Divergence damper for spectral dycore NOT invokedDivergence damper for spectral dycore NOT invoked Dynamics Subcycling                                    1

 Time filter coefficient (EPS)                      0.060

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 Number of levels Courant limiter applied               5

 Dynamics Subcycling                                    1

 Time filter coefficient (EPS)                      0.060

 

 Time filter coefficient (EPS)                      0.060 Time filter coefficient (EPS)                      0.060

 

 Time filter coefficient (EPS)                      0.060 Time filter coefficient (EPS)                      0.060

 

 

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 

 Time filter coefficient (EPS)                      0.060 Time filter coefficient (EPS)                      0.060

 

 Time filter coefficient (EPS)                      0.060 Time filter coefficient (EPS)                      0.060 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 

 

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 Number of levels Courant limiter applied               5

 

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06

 

 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL2 Horizontal diffusion coefficient (DIF2)   0.250E+06 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 

 

 Number of levels Courant limiter applied               5 Dynamics Subcycling                                    1

 

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17

 

 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 DEL4 Horizontal diffusion coefficient (DIF4)   0.200E+17 Number of levels Courant limiter applied               5

 

 Number of levels Courant limiter applied               5 Number of levels Courant limiter applied               5

 

 Number of levels Courant limiter applied               5 Number of levels Courant limiter applied               5

 

 

 Dynamics Subcycling                                    1

 

 Number of levels Courant limiter applied               5 Number of levels Courant limiter applied               5

 

 Number of levels Courant limiter applied               5 Number of levels Courant limiter applied               5 Dynamics Subcycling                                    1

 Dynamics Subcycling                                    1 Dynamics Subcycling                                    1

 

 Dynamics Subcycling                                    1 Dynamics Subcycling                                    1

 

 

 

 

 Dynamics Subcycling                                    1 Dynamics Subcycling                                    1

 

 Dynamics Subcycling                                    1 Dynamics Subcycling                                    1

 

 

 

0: ALLOCATE: 8589934588 bytes requested; not enough memory

0: ALLOCATE: 8589934588 bytes requested; not enough memory

 

===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   PID 3265 RUNNING AT cn003.hpc.iitd.ac.in

=   EXIT CODE: 9

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

[proxy:0:3@cn183] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:3@cn183] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:3@cn183] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:6@cn195] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:6@cn195] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:6@cn195] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:10@cn203] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:10@cn203] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:10@cn203] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:1@cn177] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:1@cn177] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:1@cn177] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:7@cn196] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:7@cn196] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:7@cn196] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:13@gpu140] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:13@gpu140] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:13@gpu140] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:4@cn190] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:4@cn190] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:4@cn190] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:12@cn208] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:12@cn208] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:12@cn208] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:14@cn227] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:14@cn227] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:14@cn227] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:15@cn237] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:15@cn237] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:15@cn237] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:2@cn178] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:2@cn178] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:2@cn178] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:9@cn202] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:9@cn202] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:9@cn202] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[proxy:0:8@cn198] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed

[proxy:0:8@cn198] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:8@cn198] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

[mpiexec@cn003] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

[mpiexec@cn003] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion

[mpiexec@cn003] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion

[mpiexec@cn003] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

 

Thanks,

Vineet

 

jedwards
No, the model doesn't have that kind of memory requirement, especially not at T31_g37. This allocation error would seem to indicate a problem in your compiler or otherwise in the local system somehow. You can dig in and try to debug it or look into using a different compiler version or vendor.
vineetm@...

Hello,

I just changed the compiler and the cesm version.
Now I am using intel compilers with cesm 2. I have resolved the issues related to my architecture.

Currencly I am again facing an error related to netcdf. Please let me know whether the netcdf version (or pnetcdf) is having issues or am I making a mistake.

I have attached the logs along with debug info.

Thanks,

Vineet

jedwards

It looks like the last file opened by cam is corrupted or in a format not recognized by your netcdf.   Look in the atm.log to determine what that file is, then use ncdump -k to determine the format of that file, you might try removing the file so that it will be downloaded again, often this will solve the problem

vineetm@...

Thanks for the fast reply jedwards.

I tried what you told, perhaps netcdf is not configured correctly I suppose.

I also deleted the file so that it is downloaded again and verified. Still having the same issue.

I have attached a screenshot for the info along with the generated logs. Please let me know if the file is itself in wrong format or my netcdf installation is faulty. I am sorry I am taking this issue to a totally different discussion. Please also let me know if commenting on this thread is ok.

Also, let me know if I should go with a different case for testing the installation.

Also I checked the other files downloaded in my DATA directory and found out that many files are showing the same message. I insist you to also check the Additional Info.txt file in the atttachments.

Should any flag be enabled in netcdf to read some specific files?

 

Thanks,

Vineet

jedwards

The error is definitly that it cannot open that file, but I don't see the reason.  Check the md5sum of the file, it should be:

08f4ca5459849edaceb992738df523cb 

vineetm@...

Thanks jedwards, the file was indeed not correctly downloaded.

Well now everything is going all well.

Can you please let me know approximately how much average time will the below model take to complete.

./create_newcase --case /scratch/vineetm/CESM_NEW/CASE_DEBUG --compset B1850 --res f19_g17 --mach ported_mach

Because now its been aroung 5 and a half days the model is still running on 6 nodes 8 processes each (with the default settings.)

Thanks

vineetm@...

Hi Jedwards,

I have managed to get the cesm1_2_2_CAMChem up and running.

The case that is created is using the command below:
create_newcase -case /scratch/cc/vfaculty/rsoniya.vfaculty/CASES/CASE1 -res T31_g37 -compset F1850CNCHM -mach <my_mach>

The model runs fine and stops in two cases:

 

1) When using PIO_TYPENAME = netcdf : Here some pio error occurs.  (Refer log : CASE1.o762972 and CASE1.o762972_log)

Netcdf seems to be build correctly. Below is the log:

[rsoniya.vfaculty@login02 CASE1]$ nc-config --all

 

This netCDF 4.3.3.1 has been built with the following features:

 

  --cc        -> mpicc

  --cflags    ->  -I/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/include -I/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/include

  --libs      -> -L/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/lib -lnetcdf

 

  --has-c++   -> no

  --cxx       ->

  --has-c++4  -> no

  --cxx4      ->

 

  --fc        -> mpifort

  --fflags    -> -I/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/include

  --flibs     -> -L/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/lib -lnetcdff -L/scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/lib -lnetcdf -lnetcdf

  --has-f90   -> no

 

  --has-dap   -> yes

  --has-nc2   -> yes

  --has-nc4   -> yes

  --has-hdf5  -> yes

  --has-hdf4  -> no

  --has-pnetcdf-> yes

 

  --prefix    -> /scratch/cc/vfaculty/rsoniya.vfaculty/install/deps

  --includedir-> /scratch/cc/vfaculty/rsoniya.vfaculty/install/deps/include

  --version   -> netCDF 4.3.3.1

 

2) When using PIO_TYPENAME = pnetcdf. (Refer log : CASE1.o763026 and CASE1.o763026_log - these are with debug)

 

Sorry, for stretching this thread too much. But it would be helpful.

Thanks,

Vineet More

jedwards

You are trying to build a T31_g37 case but are using 1.9x2.5 degree boundary files.   This is not a supported combination.   With a few exceptions the dimentions of the boundary files must coorespond to the dimensions of the model. 

vineetm@...

Ok, thanks for the information.

Will try with the basic compset and resolution and let you know.

Thanks,

Vineet More

vineetm@...

Hi Jedwards,

I changed my case to the default for testing:

create_newcase -case /scratch/vineetm/CESM/CASES/CASE1 -res f19_g16 -compset B1850CN -mach <my_mach> 

But still i am getting the same error:

Opened existing file b40.1850.track1.2deg.003.cam.i.0501-01-01-00000.nc

        65536

 Opened existing file

 /scratch/vineetm/CESM/INPUT_DATA/atm/cam/topo/USGS-gtopo30_1.9x2.5_remap_c050602.nc

        131072

 NetCDF: Invalid dimension ID or name

 NetCDF: Invalid argument

 pio_support::pio_die:: myrank=           -1 : ERROR: nf_mod.F90:         1051

 : NetCDF: Invalid argument

 

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1


I referred to this thread: https://bb.cgd.ucar.edu/error-nfmodf90-mpiabort, to check if I had any problems in pio. But the netcdf library seems to be detected correctly. Ihave attached the pio build log for the same.

I am not sure whether hdf5 is causing the issue or not since when pio tries to find libhdf5, it detects /usr/lib64/libhdf5.so instead of my /scratch/vineetm/CESM/install/deps/lib/libhdf5.a (since the .so is preferred)

For this I made a wild try to have  /scratch/vineetm/CESM/install/deps/lib/libhdf5.a in the SLIBS, but the same thing was observed.

Please suggest me a change.

Thanks,

Vineet More

 
jedwards

You got cesm2 working - why go back?   I don't see any problems in what you've posted.   

md5sum 

atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc

e395d2b64ca378646591312014a365fe  

 

 

vineetm@...

Hi jedwards,

Actually, some users on our cluster have explicitly asked for cesm1_2_2CAMChem so we have to keep the same along with cesm 2.

 

Regarding the file :

atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc

It is not present in the folder atm/cam/topo/


Only one file is present and that is:

atm/cam/topo/USGS-gtopo30_1.9x2.5_remap_c050602.nc

And it seems correct. As this one and all the other .nc files in the INPUT_DATA directory were verified using ncdump -k.

 

Please suggest me a solution or just le me know if there is any other compset+res that I can test to conclude that the application is up and running.

Thanks,

Vineet More

Log in or register to post comments

Who's new

  • g.mairiam@...
  • 408251463@...
  • 1161561668@...
  • 645505358@...
  • lbqnj@...