Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

The error of executing cesm_setup when setting the compset to B_2000_CAM5

J

jindachao@gmail_com

New Member
Hello everyone,

I am now running CESM1.2.2, I can run CAM case (setting compset to F_2000_CAM5) no problem. But if I set the compset to B_2000_CAM5, when execute the ./cesm_setup, the model will be break off for "ERROR: pop2.buildnml.csh failed" (the specific information will be enclosed). I found the same error in our forum, but I only found somebody said I can "try a new perl build". But I don't know how to try a new perl build. BTW, as the suggestion, I search my perl version is v5.10.1 which is not older than v5.9.

So any suggestion which can hlep me fix this problem is appreciated.

./cesm_setup
##########################################
Macros script already created ...skipping
Machine/Decomp/Pes configuration has already been done ...skipping
Running preview_namelist script
infile is /fs3/jindc/cesm1_2_2/scripts/cesm-test/Buildconf/cplconf/cesm_namelist
CAM writing dry deposition namelist to drv_flds_in
CAM writing namelist to atm_in
CLM configure done.
CLM adding use_case 2000_control defaults for var sim_year with val 2000
CLM adding use_case 2000_control defaults for var sim_year_range with val constant
CLM adding use_case 2000_control defaults for var use_case_desc with val Conditions to simulate 2000 land-use
CICE configure done.
SYNOPSIS
generate_pop_decomp.pl [options]
OPTIONS
-res (or -r) Horizontal resolution (gx1v6 etc.). (default gx1v6))
-nproc (or -n) Number of processors to use. (required)
-thrds (or -t) Number of threads per processor (default 1)
-output (or -o) Either output: all, maxblocks, bsize-x, bsize-y, or decomptype
(default: all)
EXAMPLES

generate_pop_decomp.pl -res gx1v6 -nproc 80 -output maxblocks

will return a single value -- the optimum max number of blocks to use.

if: Expression Syntax.
ERROR: pop2.buildnml.csh failed
ERROR: /fs3/jindc/cesm1_2_2/scripts/cesm-test/preview_namelists failed: 25344

Best regards,
Dachao
 
J

jedwards

CSEG and Liaisons
Staff member
It looks as if the arguments provided to generate_pop_decomp.pl are not correct.   In your case directory edit file Buildconf/pop2.buildnml.csh andadd a x to the top line: #! /bin/csh -fx Then run again, this will echo each command as it is run.  Probably an environment or xml variable is not resolved correctly.                         
 
J

jedwards

CSEG and Liaisons
Staff member
It looks as if the arguments provided to generate_pop_decomp.pl are not correct.   In your case directory edit file Buildconf/pop2.buildnml.csh andadd a x to the top line: #! /bin/csh -fx Then run again, this will echo each command as it is run.  Probably an environment or xml variable is not resolved correctly.                         
 
J

jedwards

CSEG and Liaisons
Staff member
It looks as if the arguments provided to generate_pop_decomp.pl are not correct.   In your case directory edit file Buildconf/pop2.buildnml.csh andadd a x to the top line: #! /bin/csh -fx Then run again, this will echo each command as it is run.  Probably an environment or xml variable is not resolved correctly.                         
 
J

jindachao@gmail_com

New Member
Dear jedwards, Thanks for you suggestion, as your suggestion, I deleted the "" in the command of "  set config = `env UTILROOT=$UTILROOT ./generate_pop_decomp.pl -res $OCN_GRID  -nproc $ntasks -thrds $NTHRDS_OCN -output all`" in Buildconf/pop2.buildnml.csh, and had fixed the problem.

But when executed the case.submit, I met a new problem:
#####################################
check_case OK
-------------------------------------------------------------------------
 CESM BUILDNML SCRIPT STARTING
 - To prestage restarts, untar a restart.tar file into /fs3/jindc/cesm1_2_2/scripts/cesm-control
 infile is /fs3/jindc/cesm1_2_2/scripts/cesm-control/Buildconf/cplconf/cesm_namelist
CAM writing dry deposition namelist to drv_flds_in
CAM writing namelist to atm_in
CLM configure done.
CLM adding use_case 2000_control defaults for var sim_year with val 2000
CLM adding use_case 2000_control defaults for var sim_year_range with val constant
CLM adding use_case 2000_control defaults for var use_case_desc with val Conditions to simulate 2000 land-use
CICE configure done.
POP2 build-namelist: ocn_grid is gx1v6
POP2 build-namelist: ocn_tracer_modules are  iage
 CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
 CESM PRESTAGE SCRIPT STARTING
 - Case input data directory, DIN_LOC_ROOT, is /fs3/jindc/cesm_inputdata
 - Checking the existence of input datasets in DIN_LOC_ROOT
 CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Thu Aug 28 17:27:03 JST 2014 -- CSM EXECUTION BEGINS HERE
Thu Aug 28 17:27:06 JST 2014 -- CSM EXECUTION HAS FINISHED
Model did not complete - see /fs3/jindc/cesm1_2_2/scripts/cesm-control/cesm.log.140828-172536
ERROR: cesm_submit problem sourcing templar

#############################################################
And the error information in cesm.log.140828-172536 is show below:

#############################################################(seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =     8) ( nthreads =  1)
(seq_comm_printcomms)     1     0     8     1  GLOBAL:
(seq_comm_printcomms)     2     0     8     1  CPL:
(seq_comm_printcomms)     3     0     8     1  ALLATMID:
(seq_comm_printcomms)     4     0     8     1  ALLLNDID:
(seq_comm_printcomms)     5     0     8     1  ALLOCNID:
(seq_comm_printcomms)     6     0     8     1  ALLICEID:
(seq_comm_printcomms)     7     0     4     1  ALLGLCID:
(seq_comm_printcomms)     8     0     4     1  ALLROFID:
(seq_comm_printcomms)     9     0     8     1  ALLWAVID:
(seq_comm_printcomms)    10     0     8     1  CPLALLATMID:
(seq_comm_printcomms)    11     0     8     1  CPLALLLNDID:
(seq_comm_printcomms)    12     0     8     1  CPLALLOCNID:
(seq_comm_printcomms)    13     0     8     1  CPLALLICEID:
(seq_comm_printcomms)    14     0     8     1  CPLALLGLCID:
(seq_comm_printcomms)    15     0     8     1  CPLALLROFID:
(seq_comm_printcomms)    16     0     8     1  CPLALLWAVID:
(seq_comm_printcomms)    17     0     8     1  ATM:
(seq_comm_printcomms)    18     0     8     1  CPLATM:
(seq_comm_printcomms)    19     0     8     1  LND:
(seq_comm_printcomms)    20     0     8     1  CPLLND:
(seq_comm_printcomms)    21     0     8     1  OCN:
(seq_comm_printcomms)    22     0     8     1  CPLOCN:
(seq_comm_printcomms)    23     0     8     1  ICE:
(seq_comm_printcomms)    24     0     8     1  CPLICE:
(seq_comm_printcomms)    25     0     4     1  GLC:
(seq_comm_printcomms)    26     0     8     1  CPLGLC:
(seq_comm_printcomms)    27     0     4     1  ROF:
(seq_comm_printcomms)    28     0     8     1  CPLROF:
(seq_comm_printcomms)    29     0     8     1  WAV:
(seq_comm_printcomms)    30     0     8     1  CPLWAV:
 (t_initf) Read in prof_inparm namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in
seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: seq_flds_a2x_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_a2x_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4
seq_flds_mod: seq_flds_x2a_states=
Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustar
seq_flds_mod: seq_flds_x2a_fluxes=
Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4
seq_flds_mod: seq_flds_l2x_states=
Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10
seq_flds_mod: seq_flds_l2x_fluxes=
Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_x2l_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_x2l_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_flood
seq_flds_mod: seq_flds_i2x_states=
Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifrac
seq_flds_mod: seq_flds_i2x_fluxes=
Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_salt
seq_flds_mod: seq_flds_x2i_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdy
seq_flds_mod: seq_flds_x2i_fluxes=
Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_q
seq_flds_mod: seq_flds_o2x_states=
So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepth
seq_flds_mod: seq_flds_o2x_fluxes=
Fioo_q
seq_flds_mod: seq_flds_x2o_states=
Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_x2o_fluxes=
Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fioi_salt:Forr_roff:Forr_ioff
seq_flds_mod: seq_flds_s2x_states=

seq_flds_mod: seq_flds_s2x_fluxes=

seq_flds_mod: seq_flds_x2s_states=

seq_flds_mod: seq_flds_x2s_fluxes=

seq_flds_mod: seq_flds_g2x_states=

seq_flds_mod: seq_flds_g2x_fluxes=

seq_flds_mod: seq_flds_x2g_states=

seq_flds_mod: seq_flds_x2g_fluxes=

seq_flds_mod: seq_flds_xao_states=
So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustar
seq_flds_mod: seq_flds_xao_albedo=
So_avsdr:So_anidr:So_avsdf:So_anidf
seq_flds_mod: seq_flds_r2x_states=
Slrr_volr
seq_flds_mod: seq_flds_r2x_fluxes=
Forr_roff:Forr_ioff:Flrr_flood
seq_flds_mod: seq_flds_x2r_states=

seq_flds_mod: seq_flds_x2r_fluxes=
Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_w2x_states=
Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_w2x_fluxes=

seq_flds_mod: seq_flds_x2w_states=
Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepth
seq_flds_mod: seq_flds_x2w_fluxes=

           8 pes participating in computation
 -----------------------------------
 TASK#  NAME
  0  arc-env
  1  arc-env
  2  arc-env
  3  arc-env
  4  arc-env
  5  arc-env
  6  arc-env
  7  arc-env
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c10
 0618.nc       65536
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc
      131072
 NetCDF: Invalid dimension ID or name
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E439B  Unknown               Unknown  Unknown
libpthread.so.0    00002B7D47D458F0  Unknown               Unknown  Unknown
libc.so.6          00002B7D47FDB3EB  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A51FEDF  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B7D4F334DAB  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B7D4ECF54D1  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A50A2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D47171796  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D471A729C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B7D4789464D  Unknown               Unknown  Unknown
cesm.exe           000000000221930F  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B7D47F71C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AD9897EA8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AD99079A313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AD98BFAF2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C16796  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C4C29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AD98933964D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AD989A16C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B34CF70C8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B34D66BC245  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B34D1ED12EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B34D6CF507F  Unknown               Unknown  Unknown
libmpi.so.1        00002B34CEB62676  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B34CF25848B  Unknown               Unknown  Unknown
cesm.exe           00000000021F5443  pionfread_mod_mp_         145  pionfread_mod.F90.in
cesm.exe           00000000021E82E4  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B34CF938C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AF7C658B8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AF7CD53B226  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AF7C8D502EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59B7796  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59ED29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AF7C60DA64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AF7C67B7C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B441444F8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B441B3FF20F  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B4416C142EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B441BA39BBC  Unknown               Unknown  Unknown
libmpi.so.1        00002B44138A83FD  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B4413F9BAB8  Unknown               Unknown  Unknown
cesm.exe           00000000022190BE  pio_spmd_utils_mp         297  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B441467BC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002ADB963348F0  Unknown               Unknown  Unknown
libopen-pal.so.6   00002ADB98AF9252  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB95760796  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB9579629C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002ADB95E8364D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002ADB96560C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AC9F23018F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AC9F92B1313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AC9F4AC62EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F172D796  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F176329C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AC9F1E5064D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AC9F252DC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B9BD1BFD8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B9BD8BAD234  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B9BD43C22EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD1029796  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD105F29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B9BD174C64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B9BD1E29C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[6166,1],1]
  Exit code:    174
--------------------------------------------------------------------------
#########################################
So, how could I solve this problem?

Thanks! Best wishes!
Dachao
 
 
J

jindachao@gmail_com

New Member
Dear jedwards, Thanks for you suggestion, as your suggestion, I deleted the "" in the command of "  set config = `env UTILROOT=$UTILROOT ./generate_pop_decomp.pl -res $OCN_GRID  -nproc $ntasks -thrds $NTHRDS_OCN -output all`" in Buildconf/pop2.buildnml.csh, and had fixed the problem.

But when executed the case.submit, I met a new problem:
#####################################
check_case OK
-------------------------------------------------------------------------
 CESM BUILDNML SCRIPT STARTING
 - To prestage restarts, untar a restart.tar file into /fs3/jindc/cesm1_2_2/scripts/cesm-control
 infile is /fs3/jindc/cesm1_2_2/scripts/cesm-control/Buildconf/cplconf/cesm_namelist
CAM writing dry deposition namelist to drv_flds_in
CAM writing namelist to atm_in
CLM configure done.
CLM adding use_case 2000_control defaults for var sim_year with val 2000
CLM adding use_case 2000_control defaults for var sim_year_range with val constant
CLM adding use_case 2000_control defaults for var use_case_desc with val Conditions to simulate 2000 land-use
CICE configure done.
POP2 build-namelist: ocn_grid is gx1v6
POP2 build-namelist: ocn_tracer_modules are  iage
 CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
 CESM PRESTAGE SCRIPT STARTING
 - Case input data directory, DIN_LOC_ROOT, is /fs3/jindc/cesm_inputdata
 - Checking the existence of input datasets in DIN_LOC_ROOT
 CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Thu Aug 28 17:27:03 JST 2014 -- CSM EXECUTION BEGINS HERE
Thu Aug 28 17:27:06 JST 2014 -- CSM EXECUTION HAS FINISHED
Model did not complete - see /fs3/jindc/cesm1_2_2/scripts/cesm-control/cesm.log.140828-172536
ERROR: cesm_submit problem sourcing templar

#############################################################
And the error information in cesm.log.140828-172536 is show below:

#############################################################(seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =     8) ( nthreads =  1)
(seq_comm_printcomms)     1     0     8     1  GLOBAL:
(seq_comm_printcomms)     2     0     8     1  CPL:
(seq_comm_printcomms)     3     0     8     1  ALLATMID:
(seq_comm_printcomms)     4     0     8     1  ALLLNDID:
(seq_comm_printcomms)     5     0     8     1  ALLOCNID:
(seq_comm_printcomms)     6     0     8     1  ALLICEID:
(seq_comm_printcomms)     7     0     4     1  ALLGLCID:
(seq_comm_printcomms)     8     0     4     1  ALLROFID:
(seq_comm_printcomms)     9     0     8     1  ALLWAVID:
(seq_comm_printcomms)    10     0     8     1  CPLALLATMID:
(seq_comm_printcomms)    11     0     8     1  CPLALLLNDID:
(seq_comm_printcomms)    12     0     8     1  CPLALLOCNID:
(seq_comm_printcomms)    13     0     8     1  CPLALLICEID:
(seq_comm_printcomms)    14     0     8     1  CPLALLGLCID:
(seq_comm_printcomms)    15     0     8     1  CPLALLROFID:
(seq_comm_printcomms)    16     0     8     1  CPLALLWAVID:
(seq_comm_printcomms)    17     0     8     1  ATM:
(seq_comm_printcomms)    18     0     8     1  CPLATM:
(seq_comm_printcomms)    19     0     8     1  LND:
(seq_comm_printcomms)    20     0     8     1  CPLLND:
(seq_comm_printcomms)    21     0     8     1  OCN:
(seq_comm_printcomms)    22     0     8     1  CPLOCN:
(seq_comm_printcomms)    23     0     8     1  ICE:
(seq_comm_printcomms)    24     0     8     1  CPLICE:
(seq_comm_printcomms)    25     0     4     1  GLC:
(seq_comm_printcomms)    26     0     8     1  CPLGLC:
(seq_comm_printcomms)    27     0     4     1  ROF:
(seq_comm_printcomms)    28     0     8     1  CPLROF:
(seq_comm_printcomms)    29     0     8     1  WAV:
(seq_comm_printcomms)    30     0     8     1  CPLWAV:
 (t_initf) Read in prof_inparm namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in
seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: seq_flds_a2x_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_a2x_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4
seq_flds_mod: seq_flds_x2a_states=
Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustar
seq_flds_mod: seq_flds_x2a_fluxes=
Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4
seq_flds_mod: seq_flds_l2x_states=
Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10
seq_flds_mod: seq_flds_l2x_fluxes=
Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_x2l_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_x2l_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_flood
seq_flds_mod: seq_flds_i2x_states=
Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifrac
seq_flds_mod: seq_flds_i2x_fluxes=
Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_salt
seq_flds_mod: seq_flds_x2i_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdy
seq_flds_mod: seq_flds_x2i_fluxes=
Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_q
seq_flds_mod: seq_flds_o2x_states=
So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepth
seq_flds_mod: seq_flds_o2x_fluxes=
Fioo_q
seq_flds_mod: seq_flds_x2o_states=
Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_x2o_fluxes=
Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fioi_salt:Forr_roff:Forr_ioff
seq_flds_mod: seq_flds_s2x_states=

seq_flds_mod: seq_flds_s2x_fluxes=

seq_flds_mod: seq_flds_x2s_states=

seq_flds_mod: seq_flds_x2s_fluxes=

seq_flds_mod: seq_flds_g2x_states=

seq_flds_mod: seq_flds_g2x_fluxes=

seq_flds_mod: seq_flds_x2g_states=

seq_flds_mod: seq_flds_x2g_fluxes=

seq_flds_mod: seq_flds_xao_states=
So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustar
seq_flds_mod: seq_flds_xao_albedo=
So_avsdr:So_anidr:So_avsdf:So_anidf
seq_flds_mod: seq_flds_r2x_states=
Slrr_volr
seq_flds_mod: seq_flds_r2x_fluxes=
Forr_roff:Forr_ioff:Flrr_flood
seq_flds_mod: seq_flds_x2r_states=

seq_flds_mod: seq_flds_x2r_fluxes=
Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_w2x_states=
Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_w2x_fluxes=

seq_flds_mod: seq_flds_x2w_states=
Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepth
seq_flds_mod: seq_flds_x2w_fluxes=

           8 pes participating in computation
 -----------------------------------
 TASK#  NAME
  0  arc-env
  1  arc-env
  2  arc-env
  3  arc-env
  4  arc-env
  5  arc-env
  6  arc-env
  7  arc-env
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c10
 0618.nc       65536
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc
      131072
 NetCDF: Invalid dimension ID or name
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E439B  Unknown               Unknown  Unknown
libpthread.so.0    00002B7D47D458F0  Unknown               Unknown  Unknown
libc.so.6          00002B7D47FDB3EB  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A51FEDF  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B7D4F334DAB  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B7D4ECF54D1  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A50A2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D47171796  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D471A729C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B7D4789464D  Unknown               Unknown  Unknown
cesm.exe           000000000221930F  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B7D47F71C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AD9897EA8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AD99079A313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AD98BFAF2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C16796  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C4C29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AD98933964D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AD989A16C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B34CF70C8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B34D66BC245  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B34D1ED12EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B34D6CF507F  Unknown               Unknown  Unknown
libmpi.so.1        00002B34CEB62676  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B34CF25848B  Unknown               Unknown  Unknown
cesm.exe           00000000021F5443  pionfread_mod_mp_         145  pionfread_mod.F90.in
cesm.exe           00000000021E82E4  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B34CF938C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AF7C658B8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AF7CD53B226  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AF7C8D502EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59B7796  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59ED29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AF7C60DA64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AF7C67B7C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B441444F8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B441B3FF20F  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B4416C142EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B441BA39BBC  Unknown               Unknown  Unknown
libmpi.so.1        00002B44138A83FD  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B4413F9BAB8  Unknown               Unknown  Unknown
cesm.exe           00000000022190BE  pio_spmd_utils_mp         297  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B441467BC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002ADB963348F0  Unknown               Unknown  Unknown
libopen-pal.so.6   00002ADB98AF9252  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB95760796  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB9579629C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002ADB95E8364D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002ADB96560C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AC9F23018F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AC9F92B1313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AC9F4AC62EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F172D796  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F176329C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AC9F1E5064D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AC9F252DC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B9BD1BFD8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B9BD8BAD234  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B9BD43C22EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD1029796  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD105F29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B9BD174C64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B9BD1E29C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[6166,1],1]
  Exit code:    174
--------------------------------------------------------------------------
#########################################
So, how could I solve this problem?

Thanks! Best wishes!
Dachao
 
 
J

jindachao@gmail_com

New Member
Dear jedwards, Thanks for you suggestion, as your suggestion, I deleted the "" in the command of "  set config = `env UTILROOT=$UTILROOT ./generate_pop_decomp.pl -res $OCN_GRID  -nproc $ntasks -thrds $NTHRDS_OCN -output all`" in Buildconf/pop2.buildnml.csh, and had fixed the problem.

But when executed the case.submit, I met a new problem:
#####################################
check_case OK
-------------------------------------------------------------------------
 CESM BUILDNML SCRIPT STARTING
 - To prestage restarts, untar a restart.tar file into /fs3/jindc/cesm1_2_2/scripts/cesm-control
 infile is /fs3/jindc/cesm1_2_2/scripts/cesm-control/Buildconf/cplconf/cesm_namelist
CAM writing dry deposition namelist to drv_flds_in
CAM writing namelist to atm_in
CLM configure done.
CLM adding use_case 2000_control defaults for var sim_year with val 2000
CLM adding use_case 2000_control defaults for var sim_year_range with val constant
CLM adding use_case 2000_control defaults for var use_case_desc with val Conditions to simulate 2000 land-use
CICE configure done.
POP2 build-namelist: ocn_grid is gx1v6
POP2 build-namelist: ocn_tracer_modules are  iage
 CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
 CESM PRESTAGE SCRIPT STARTING
 - Case input data directory, DIN_LOC_ROOT, is /fs3/jindc/cesm_inputdata
 - Checking the existence of input datasets in DIN_LOC_ROOT
 CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Thu Aug 28 17:27:03 JST 2014 -- CSM EXECUTION BEGINS HERE
Thu Aug 28 17:27:06 JST 2014 -- CSM EXECUTION HAS FINISHED
Model did not complete - see /fs3/jindc/cesm1_2_2/scripts/cesm-control/cesm.log.140828-172536
ERROR: cesm_submit problem sourcing templar

#############################################################
And the error information in cesm.log.140828-172536 is show below:

#############################################################(seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0     3     1 ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =     4) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =     8) ( nthreads =  1)
(seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =     8) ( nthreads =  1)
(seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)
(seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =     8) ( nthreads =  1)
(seq_comm_printcomms)     1     0     8     1  GLOBAL:
(seq_comm_printcomms)     2     0     8     1  CPL:
(seq_comm_printcomms)     3     0     8     1  ALLATMID:
(seq_comm_printcomms)     4     0     8     1  ALLLNDID:
(seq_comm_printcomms)     5     0     8     1  ALLOCNID:
(seq_comm_printcomms)     6     0     8     1  ALLICEID:
(seq_comm_printcomms)     7     0     4     1  ALLGLCID:
(seq_comm_printcomms)     8     0     4     1  ALLROFID:
(seq_comm_printcomms)     9     0     8     1  ALLWAVID:
(seq_comm_printcomms)    10     0     8     1  CPLALLATMID:
(seq_comm_printcomms)    11     0     8     1  CPLALLLNDID:
(seq_comm_printcomms)    12     0     8     1  CPLALLOCNID:
(seq_comm_printcomms)    13     0     8     1  CPLALLICEID:
(seq_comm_printcomms)    14     0     8     1  CPLALLGLCID:
(seq_comm_printcomms)    15     0     8     1  CPLALLROFID:
(seq_comm_printcomms)    16     0     8     1  CPLALLWAVID:
(seq_comm_printcomms)    17     0     8     1  ATM:
(seq_comm_printcomms)    18     0     8     1  CPLATM:
(seq_comm_printcomms)    19     0     8     1  LND:
(seq_comm_printcomms)    20     0     8     1  CPLLND:
(seq_comm_printcomms)    21     0     8     1  OCN:
(seq_comm_printcomms)    22     0     8     1  CPLOCN:
(seq_comm_printcomms)    23     0     8     1  ICE:
(seq_comm_printcomms)    24     0     8     1  CPLICE:
(seq_comm_printcomms)    25     0     4     1  GLC:
(seq_comm_printcomms)    26     0     8     1  CPLGLC:
(seq_comm_printcomms)    27     0     4     1  ROF:
(seq_comm_printcomms)    28     0     8     1  CPLROF:
(seq_comm_printcomms)    29     0     8     1  WAV:
(seq_comm_printcomms)    30     0     8     1  CPLWAV:
 (t_initf) Read in prof_inparm namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in
seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
8 MB memory   alloc in MB is             8.00
8 MB memory dealloc in MB is             0.00
Memory block size conversion in bytes is          1024.50
seq_flds_mod: seq_flds_a2x_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_a2x_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4
seq_flds_mod: seq_flds_x2a_states=
Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustar
seq_flds_mod: seq_flds_x2a_fluxes=
Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4
seq_flds_mod: seq_flds_l2x_states=
Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10
seq_flds_mod: seq_flds_l2x_fluxes=
Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_x2l_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diag
seq_flds_mod: seq_flds_x2l_fluxes=
Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_flood
seq_flds_mod: seq_flds_i2x_states=
Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifrac
seq_flds_mod: seq_flds_i2x_fluxes=
Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_salt
seq_flds_mod: seq_flds_x2i_states=
Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdy
seq_flds_mod: seq_flds_x2i_fluxes=
Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_q
seq_flds_mod: seq_flds_o2x_states=
So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepth
seq_flds_mod: seq_flds_o2x_fluxes=
Fioo_q
seq_flds_mod: seq_flds_x2o_states=
Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_x2o_fluxes=
Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fioi_salt:Forr_roff:Forr_ioff
seq_flds_mod: seq_flds_s2x_states=

seq_flds_mod: seq_flds_s2x_fluxes=

seq_flds_mod: seq_flds_x2s_states=

seq_flds_mod: seq_flds_x2s_fluxes=

seq_flds_mod: seq_flds_g2x_states=

seq_flds_mod: seq_flds_g2x_fluxes=

seq_flds_mod: seq_flds_x2g_states=

seq_flds_mod: seq_flds_x2g_fluxes=

seq_flds_mod: seq_flds_xao_states=
So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustar
seq_flds_mod: seq_flds_xao_albedo=
So_avsdr:So_anidr:So_avsdf:So_anidf
seq_flds_mod: seq_flds_r2x_states=
Slrr_volr
seq_flds_mod: seq_flds_r2x_fluxes=
Forr_roff:Forr_ioff:Flrr_flood
seq_flds_mod: seq_flds_x2r_states=

seq_flds_mod: seq_flds_x2r_fluxes=
Flrl_rofliq:Flrl_rofice
seq_flds_mod: seq_flds_w2x_states=
Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokes
seq_flds_mod: seq_flds_w2x_fluxes=

seq_flds_mod: seq_flds_x2w_states=
Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepth
seq_flds_mod: seq_flds_x2w_fluxes=

           8 pes participating in computation
 -----------------------------------
 TASK#  NAME
  0  arc-env
  1  arc-env
  2  arc-env
  3  arc-env
  4  arc-env
  5  arc-env
  6  arc-env
  7  arc-env
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c10
 0618.nc       65536
 Opened existing file
 /fs3/jindc/cesm_inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc
      131072
 NetCDF: Invalid dimension ID or name
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E439B  Unknown               Unknown  Unknown
libpthread.so.0    00002B7D47D458F0  Unknown               Unknown  Unknown
libc.so.6          00002B7D47FDB3EB  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A51FEDF  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B7D4F334DAB  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B7D4ECF54D1  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B7D4A50A2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D47171796  Unknown               Unknown  Unknown
libmpi.so.1        00002B7D471A729C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B7D4789464D  Unknown               Unknown  Unknown
cesm.exe           000000000221930F  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B7D47F71C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AD9897EA8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AD99079A313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AD98BFAF2EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C16796  Unknown               Unknown  Unknown
libmpi.so.1        00002AD988C4C29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AD98933964D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AD989A16C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B34CF70C8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B34D66BC245  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B34D1ED12EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B34D6CF507F  Unknown               Unknown  Unknown
libmpi.so.1        00002B34CEB62676  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B34CF25848B  Unknown               Unknown  Unknown
cesm.exe           00000000021F5443  pionfread_mod_mp_         145  pionfread_mod.F90.in
cesm.exe           00000000021E82E4  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B34CF938C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AF7C658B8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AF7CD53B226  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AF7C8D502EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59B7796  Unknown               Unknown  Unknown
libmpi.so.1        00002AF7C59ED29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AF7C60DA64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AF7C67B7C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B441444F8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B441B3FF20F  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B4416C142EB  Unknown               Unknown  Unknown
mca_pml_ob1.so     00002B441BA39BBC  Unknown               Unknown  Unknown
libmpi.so.1        00002B44138A83FD  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B4413F9BAB8  Unknown               Unknown  Unknown
cesm.exe           00000000022190BE  pio_spmd_utils_mp         297  pio_spmd_utils.F90.in
cesm.exe           0000000002212A3B  box_rearrange_mp_        1372  box_rearrange.F90.in
cesm.exe           000000000220A23D  box_rearrange_mp_        1029  box_rearrange.F90.in
cesm.exe           000000000210FBA6  piolib_mod_mp_pio        1166  piolib_mod.F90
cesm.exe           00000000004E0411  cam_pio_utils_mp_         449  cam_pio_utils.F90
cesm.exe           00000000006A9035  ncdio_atm_mp_infl         279  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B441467BC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002ADB963348F0  Unknown               Unknown  Unknown
libopen-pal.so.6   00002ADB98AF9252  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB95760796  Unknown               Unknown  Unknown
libmpi.so.1        00002ADB9579629C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002ADB95E8364D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002ADB96560C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002AC9F23018F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002AC9F92B1313  Unknown               Unknown  Unknown
libopen-pal.so.6   00002AC9F4AC62EB  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F172D796  Unknown               Unknown  Unknown
libmpi.so.1        00002AC9F176329C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002AC9F1E5064D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002AC9F252DC4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source            
cesm.exe           00000000023B6D19  Unknown               Unknown  Unknown
cesm.exe           00000000023B55EE  Unknown               Unknown  Unknown
cesm.exe           000000000234E862  Unknown               Unknown  Unknown
cesm.exe           00000000022DA953  Unknown               Unknown  Unknown
cesm.exe           00000000022E4369  Unknown               Unknown  Unknown
libpthread.so.0    00002B9BD1BFD8F0  Unknown               Unknown  Unknown
mca_btl_vader.so   00002B9BD8BAD234  Unknown               Unknown  Unknown
libopen-pal.so.6   00002B9BD43C22EB  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD1029796  Unknown               Unknown  Unknown
libmpi.so.1        00002B9BD105F29C  Unknown               Unknown  Unknown
libmpi_mpifh.so.2  00002B9BD174C64D  Unknown               Unknown  Unknown
cesm.exe           0000000002217CBF  pio_spmd_utils_mp         345  pio_spmd_utils.F90.in
cesm.exe           000000000220625C  box_rearrange_mp_         571  box_rearrange.F90.in
cesm.exe           00000000021A611A  rearrange_mp_rear         108  rearrange.F90.in
cesm.exe           00000000021E8EA9  piodarray_mp_read         390  piodarray.F90.in
cesm.exe           00000000006A90DE  ncdio_atm_mp_infl         287  ncdio_atm.F90
cesm.exe           000000000121005A  inidat_mp_read_in         230  inidat.F90
cesm.exe           0000000000F5FCE2  startup_initialco          54  startup_initialconds.F90
cesm.exe           0000000000522964  inital_mp_cam_ini          51  inital.F90
cesm.exe           00000000004A9491  cam_comp_mp_cam_i         164  cam_comp.F90
cesm.exe           00000000004A53B0  atm_comp_mct_mp_a         276  atm_comp_mct.F90
cesm.exe           0000000000432BDB  ccsm_comp_mod_mp_        1058  ccsm_comp_mod.F90
cesm.exe           00000000004353D6  MAIN__                     90  ccsm_driver.F90
cesm.exe           0000000000411E46  Unknown               Unknown  Unknown
libc.so.6          00002B9BD1E29C4D  Unknown               Unknown  Unknown
cesm.exe           0000000000411D39  Unknown               Unknown  Unknown
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[6166,1],1]
  Exit code:    174
--------------------------------------------------------------------------
#########################################
So, how could I solve this problem?

Thanks! Best wishes!
Dachao
 
 
J

jedwards

CSEG and Liaisons
Staff member
The pio errors are because at least one process aborted before the others got to the collective.   You are going to need to use a debugger or print statements in the code to figure out where the process is aborting and why.
 
J

jedwards

CSEG and Liaisons
Staff member
The pio errors are because at least one process aborted before the others got to the collective.   You are going to need to use a debugger or print statements in the code to figure out where the process is aborting and why.
 
J

jedwards

CSEG and Liaisons
Staff member
The pio errors are because at least one process aborted before the others got to the collective.   You are going to need to use a debugger or print statements in the code to figure out where the process is aborting and why.
 
You must log in or register to reply here.
Top