WACCM(X) atm initialisation problem

[ansgar_schanz@iap_unibe_ch]

Hi there,

I am trying to install WACCM(X) on our local cluster during the last weeks. I use the following compilers and software:

compiler: intel 10.0.026
netcdf: 3.6.3
mpich2: 1.4.1-1p1
cesm: 1_0_4
compsets: F_2000_WACCM and F_2000_WACCMX
grid: f19_g16

The compilation is successfull but some warnings during the csm_share build appear (similar warning appears with GNU compilers)

#
# cesm1_0_4/models/utils/timing/gptl.c(2164): warning #167: argument of type "int (*)(const char **, const char **)" is incompatible with parameter of $
# qsort( sort[t], count[t], sizeof(char*), cmp );
#

Wenn I try to run it dies during initialisation of the atm component. I try to give you a breif insight of my run logs for the WACCM and the WACCMX compset.

--- end of cpl.log.* for WACCM & WACCMX ---

(seq_timemgr_clockPrint) Prev Time = 00001201 00000
(seq_timemgr_clockPrint) Next Time = 99991201 00000
(seq_timemgr_clockPrint) Intervl yms = 9999 0 0
(seq_timemgr_clockPrint) Alarm = 7 seq_timemgr_alarm_tprof
(seq_timemgr_clockPrint) Prev Time = 00001201 00000
(seq_timemgr_clockPrint) Next Time = 99991201 00000
(seq_timemgr_clockPrint) Intervl yms = 9999 0 0


(seq_mct_drv) : Initialize each component: atm, lnd, ocn, and ice
(seq_mct_drv) : Initialize atm component

---

--- end of ccsm.log.* for WACCMX ---

(seq_io_init) pio_numtasks = 1
(t_initf) Read in prof_inparm namelist from: drv_in
1 pes participating in computation
-----------------------------------
TASK# NAME
0 submit01
forrtl: warning (402): fort: (1): In call to PARINIT, an array temporary was created for argument #2

forrtl: warning (402): fort: (1): In call to PAR_XSUM, an array temporary was created for argument #2

[ ... this warning (402) occours a several times until finally the MPI abort is called]

forrtl: warning (402): fort: (1): In call to PAR_XSUM, an array temporary was created for argument #2

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

---

--- end of ccsm.log.* for WACCMX---

(seq_io_init) pio_numtasks = 1
(t_initf) Read in prof_inparm namelist from: drv_in
1 pes participating in computation
-----------------------------------
TASK# NAME
0 submit01
forrtl: warning (402): fort: (1): In call to PARINIT, an array temporary was created for argument #2

pio_support::pio_die:: myrank= -1 : ERROR: alloc_check_1d_int:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
ccsm.exe 00000000024FF5D5 Unknown Unknown Unknown
ccsm.exe 00000000024FAF00 Unknown Unknown Unknown
ccsm.exe 0000000002537FB8 Unknown Unknown Unknown
ccsm.exe 00000000024FB433 Unknown Unknown Unknown
ccsm.exe 00000000024436EF Unknown Unknown Unknown
ccsm.exe 00000000005C28B7 cam_pio_utils_mp_ 446 cam_pio_utils.F90
ccsm.exe 00000000013FB726 ncdio_atm_mp_infl 561 ncdio_atm.F90
ccsm.exe 0000000000F0BBC0 inidat_mp_read_in 259 inidat.F90
ccsm.exe 0000000000892D24 startup_initialco 135 startup_initialconds.F90
ccsm.exe 0000000000611FE8 inital_mp_cam_ini 108 inital.F90
ccsm.exe 0000000000528BCA cam_comp_mp_cam_i 147 cam_comp.F90
ccsm.exe 000000000052521E atm_comp_mct_mp_a 272 atm_comp_mct.F90
ccsm.exe 0000000000420084 ccsm_comp_mod_mp_ 684 ccsm_comp_mod.F90
ccsm.exe 00000000004235B4 MAIN__ 90 ccsm_driver.F90
ccsm.exe 000000000040E7CE Unknown Unknown Unknown
libc.so.6 00007FF35823509D Unknown Unknown Unknown
ccsm.exe 000000000040E6E9 Unknown Unknown Unknown

---

--- end of atm.log.* for WACCM ---

Creating new decomp: 260200960144
Creating new decomp: 3602006600950144
Creating new decomp: 3602006600960144
Warning: Not reading O2_1S from IC file.
O2_1S initialized by "chem_init_cnst"
Warning: Not reading O2_1D from IC file.
O2_1D initialized by "chem_init_cnst"

************************************************************************

INIDAT: Globally averaged geopotential height = 231.8451198518 meters

************************************************************************

physics_types: phys_state allocation error = 41
ENDRUN:physics_types: failed to allocate physics_state array

--- end of atm.log.* for WACCMX ---

ch4 volume mixing ratio = -1.00000000000000
n2o volume mixing ratio = -1.00000000000000
f11 volume mixing ratio = -1.00000000000000
f12 volume mixing ratio = -1.00000000000000
ENDRUN:pbuf_allocate: ERROR: allocate failed for FRACIS

---

Do you have an idea why it dies? Is this about the warnings? I also tried it with cesm1_0_3, openmpi, gnu compilers... in all possible combinations without any success. It always dies within the initialisation of the atm componnet.
If you need any further informations as compilerflags or machopts & macros please let my know...

Ans

 

[santos]

This looks to me like you are simply running out of memory. Can you check the memory usage on this machine? It might be necessary to use a lower resolution grid (like f45_f45) or to use more resources on your cluster.

I'd also like to point out that the appropriate grids to use for F compsets are the ones that look like f*_f*, not f*_g* (so use f19_f19 instead of f19_g16). Some of our documentation might have had the wrong grid listed in the last release, which was my mistake. f19_g16 is used for runs using an active ocean, like the B compsets. However, an F case should still run with f19_g16, so I don't think that this is the source of the problem.

 

[ansgar_schanz@iap_unibe_ch]

The memory was the problem! Fixed it! Thanks...

Ans