Main menu

Navigation

CESM 1.0.4 run failed when initializing lnd component

2 posts / 0 new
Last post
yaozhu@...
CESM 1.0.4 run failed when initializing lnd component

I am trying to run a CESM 1.0.4 case on trestles.sdsc.edu. The cases I created is  --compset B -res f19_g16. I can build it successfully using pgi/10.5 + mvapich2/1.5.lpl + netcdf/4.1.1. I then ran the case under DEBUG mode, and using only 4 MPI processes. It failed with the following content in log files. Although running under DEBUG mode, but no useful debugging information is shown. I have no idea why the MPI processes termiated unexpectdely.

 

In cpl.log

(seq_mct_drv) : Initialize each component: atm, lnd, ocn, and ice

(seq_mct_drv) : Initialize atm component

(seq_mct_drv) : Initialize lnd component

 

In lnd.log

Attempting to read ldomain from fsurdat

/oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/surfdata/surfdata_1.9x2.5_simyr2000_c091005.nc

 

And the content of ccsm.log is as

1 (seq_io_init) pio init parameters: before nml read
2 (seq_io_init) pio_stride = -99
3 (seq_io_init) pio_root = -99
4 (seq_io_init) pio_typename = nothing
5 (seq_io_init) pio_numtasks = -99
6 (seq_io_init) pio_debug_level = 0
7 pio_async_interface = F
8 (seq_io_init) pio init parameters: after nml read
9 (seq_io_init) pio_stride = -1
10 (seq_io_init) pio_root = 1
11 (seq_io_init) pio_typename = netcdf
12 (seq_io_init) pio_numtasks = -1
13 (seq_io_init) pio init parameters:
14 (seq_io_init) pio_stride = 4
15 (seq_io_init) pio_root = 1
16 (seq_io_init) pio_typename = NETCDF
17 (seq_io_init) pio_numtasks = 1
18 (seq_io_init) pio_debug_level = 0
19 pio_async_interface = F
20 (seq_comm_setcomm) initialize ID ( 7 GLOBAL ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
21 (seq_comm_setcomm) initialize ID ( 2 ATM ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
22 (seq_comm_setcomm) initialize ID ( 1 LND ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
23 (seq_comm_setcomm) initialize ID ( 4 ICE ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
24 (seq_comm_setcomm) initialize ID ( 5 GLC ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
25 (seq_comm_setcomm) initialize ID ( 3 OCN ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
26 (seq_comm_setcomm) initialize ID ( 6 CPL ) pelist = 0 3 1 ( npes = 4) ( nthreads = 1)
27 (seq_comm_joincomm) initialize ID ( 8 CPLATM ) join IDs = 6 2 ( npes = 4) ( nthreads = 1)
28 (seq_comm_joincomm) initialize ID ( 9 CPLLND ) join IDs = 6 1 ( npes = 4) ( nthreads = 1)
29 (seq_comm_joincomm) initialize ID ( 10 CPLICE ) join IDs = 6 4 ( npes = 4) ( nthreads = 1)
30 (seq_comm_joincomm) initialize ID ( 11 CPLOCN ) join IDs = 6 3 ( npes = 4) ( nthreads = 1)
31 (seq_comm_joincomm) initialize ID ( 12 CPLGLC ) join IDs = 6 5 ( npes = 4) ( nthreads = 1)
32
33 (seq_comm_printcomms) ID layout : global pes vs local pe for each ID
34 gpe LND ATM OCN ICE GLC CPL GLOBAL CPLATM CPLLND CPLICE CPLOCN CPLGLC nthrds
35 --- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------
36 0 : 0 0 0 0 0 0 0 0 0 0 0 0 1
37 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1
38 2 : 2 2 2 2 2 2 2 2 2 2 2 2 1
39 3 : 3 3 3 3 3 3 3 3 3 3 3 3 1
40
41 (seq_io_init) pio init parameters for CPL:
42 (seq_io_init) pio_stride = 4
43 (seq_io_init) pio_root = 1
44 pio iotype is netcdf
45 (seq_io_init) pio_iotype = 6
46 (seq_io_init) pio_numtasks = 1
47 (seq_io_init) pio init parameters for ATM:
48 (seq_io_init) pio_stride = 4
49 (seq_io_init) pio_root = 1
50 pio iotype is netcdf
51 (seq_io_init) pio_iotype = 6
52 (seq_io_init) pio_numtasks = 1
53 (seq_io_init) pio init parameters for ICE:
54 (seq_io_init) pio_stride = 4
55 (seq_io_init) pio_root = 1

56 pio iotype is netcdf
57 (seq_io_init) pio_iotype = 6
58 (seq_io_init) pio_numtasks = 1
59 (seq_io_init) pio init parameters for OCN:
60 (seq_io_init) pio_stride = 4
61 (seq_io_init) pio_root = 0
62 pio iotype is netcdf
63 (seq_io_init) pio_iotype = 6
64 (seq_io_init) pio_numtasks = 1
65 (seq_io_init) pio init parameters for LND:
66 (seq_io_init) pio_stride = 4
67 (seq_io_init) pio_root = 1
68 pio iotype is netcdf
69 (seq_io_init) pio_iotype = 6
70 (seq_io_init) pio_numtasks = 1
71 (seq_io_init) pio init parameters for GLC:
72 (seq_io_init) pio_stride = 4
73 (seq_io_init) pio_root = 1
74 pio iotype is netcdf
75 (seq_io_init) pio_iotype = 6
76 (seq_io_init) pio_numtasks = 1
77 (t_initf) Read in prof_inparm namelist from: drv_in
78 8 MB memory alloc in MB is 8.008 MB memory alloc in MB is 8.00
79
80 8 MB memory dealloc in MB is 0.00
81 8 MB memory dealloc in MB is 0.00
82 Memory block size conversion in bytes is 4092.00
83 Memory block size conversion in bytes is 4092.00
84 8 MB memory alloc in MB is 8.00
85 8 MB memory dealloc in MB is 0.00
86 Memory block size conversion in bytes is 4092.00
87 4 pes participating in computation
88 -----------------------------------
89 TASK# NAME
90 0 trestles-12-26.local
91 1 trestles-12-26.local
92 2 trestles-12-26.local
93 3 trestles-12-26.local
94 Opened existing file
95 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/inic/fv/cami_0000-01-01_1.9x2.5_L26_c070408.nc
96 14
97 Opened existing file
98 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/topo/USGS-gtopo30_1.9x2.5_remap_c050602.nc
99 15
100 Opened existing file
101 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/sulfate_camrt_c080918.nc
102 16
103 Opened existing file
104 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/dust1_camrt_c080918.nc
105 16
106 Opened existing file
107 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/dust2_camrt_c080918.nc
108 16
109 Opened existing file
110 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/dust3_camrt_c080918.nc

111 16
112 Opened existing file
113 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/dust4_camrt_c080918.nc
114 16
115 Opened existing file
116 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/bcpho_camrt_c080918.nc
117 16
118 Opened existing file
119 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/bcphi_camrt_c080918.nc
120 16
121 Opened existing file
122 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/ocpho_camrt_c080918.nc
123 16
124 Opened existing file
125 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/ocphi_camrt_c080918.nc
126 16
127 Opened existing file
128 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/ssam_camrt_c080918.nc
129 16
130 Opened existing file
131 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/physprops/sscm_camrt_c080918.nc
132 16
133 Opened existing file
134 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/ozone/ozone_1.9x2.5_L26_2000clim_c091112.nc
135 16
136 Opened existing file
137 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_2000clim_c091112.nc
138 17
139 Opened existing file
140 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1849-2006_1.9x2.5_c090803.nc
141 18
142 Opened existing file
143 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/rad/abs_ems_factors_fastvx.c030508.nc
144 19
145 Opened existing file
146 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc
147 19
148
149
150 4 pes participating in computation for CLM
151
152 -----------------------------------
153
154 NODE# NAME
155 ( 0) trestles-12-26.local
156 ( 1) trestles-12-26.local
157 ( 2) trestles-12-26.local
158 ( 3) trestles-12-26.local
159 Opened existing file
160 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/griddata/griddata_1.9x2.5_060404.nc
161 19
162 Opened existing file
163 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/griddata/fracdata_1.9x2.5_gx1v6_c090206.nc
164 20
165 Opened existing file

166 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/surfdata/surfdata_1.9x2.5_simyr2000_c091005.nc
167 19
168
169 proc= 1 beg atmcell = 1417 end atmcell = 2832
170 total atmcells per proc = 1416
171 proc= 1 atm ngseg = 408 atm nlseg = 99
172 proc= 1 nclumps = 1
173
174 proc= 1 beg atmcell = 1417 end atmcell = 2832
175 total atmcells per proc = 1416
176 proc= 1 atm ngseg = 408 atm nlseg = 99
177 proc= 1 nclumps = 1
178
179 proc= 2 beg atmcell = 2833 end atmcell = 4248
180 total atmcells per proc = 1416
181 proc= 2 atm ngseg = 408 atm nlseg = 108
182 proc= 2 nclumps = 1
183 Opened existing file
184 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/griddata/griddata_1.9x2.5_060404.nc
185 19
186 Opened existing file
187 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/griddata/fracdata_1.9x2.5_gx1v6_c090206.nc
188 19
189 Opened existing file
190 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/surfdata/surfdata_1.9x2.5_simyr2000_c091005.nc
191 19
192 Opened existing file
193 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/surfdata/surfdata_1.9x2.5_simyr2000_c091005.nc
194 19
195 Opened existing file
196 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/pftdata/pft-physiology.c110425.nc
197 19
198 Opened existing file
199 /oasis/projects/nsf/prd137/ccsmuser/tgdata/cesm1_0_4/inputdata/lnd/clm2/surfdata/surfdata_1.9x2.5_simyr2000_c091005.nc
200 19
201 Exit code -5 signaled from trestles-12-26
202 MPI process (rank: 0) terminated unexpectedly on trestles-12-26.local

Yao Zhu

jedwards

Just a guess but I suspect that you ran out of memory.  

CESM Software Engineer

Log in or register to post comments

Who's new

  • lwheele@...
  • afroberts@...
  • richard.allard@...
  • tracy@...
  • 498749303@...