cesm1.2.2 aborted after 2 steps

[wadewei]

Hi everyone,

I am stuck with a strange problem when running a LGM run. Judging from the logs, it seems to have run two steps before aborting, approximately one minute after it was submitted. Even with PIO_DEBUG_LEVEL=1 INFO_DBUG=2, I am not getting much cues from the error messages.

Any help is appreciated.

Thanks,
Wade



-----------cesm.log-----------
/home/wade/cesm1_2_2/models/utils/pio/pionfwrite_mod.F90.in 123
/home/wade/cesm1_2_2/models/utils/pio/pionfwrite_mod.F90.in 127
1
pionfwrite_mod::write_nfdarray_double: File%iosystem%comp_rank: 25
: relaying IOBUF for write size= 1 1 1
1 0 0 1 32767
piodarray 480 NAME : IAM: 26 UseRearranger: T
0 0 0
piodarray 480 NAME : IAM: 26 UseRearranger: T
0 0 0
piodarray 480 NAME : IAM: 26 UseRearranger: T
0 0 0
piodarray 480 NAME : IAM: 27 UseRearranger: T
0 0 0
piodarray 480 NAME : IAM: 27 UseRearranger: T
0 0 0
piodarray 480 NAME : IAM: 27 UseRearranger: T
0 0 0
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cesm.exe 00000000016C8DB5 Unknown Unknown Unknown
cesm.exe 00000000016C69D7 Unknown Unknown Unknown
cesm.exe 0000000001677194 Unknown Unknown Unknown
cesm.exe 0000000001676FA6 Unknown Unknown Unknown
cesm.exe 00000000015F6F26 Unknown Unknown Unknown
cesm.exe 0000000001600950 Unknown Unknown Unknown
libpthread.so.0 00002B29F32966D0 Unknown Unknown Unknown
cesm.exe 00000000016D92B5 Unknown Unknown Unknown

Stack trace terminated abnormally.


-----------atm.log-----------
WSHIST: writing time sample 0 to h-file 2 DATE=0001/01/01 NCSEC= 0

WSHIST: nhfil( 2 )=
test_sb_F_oci_in_high_lat_sst_glacier_MEC.cam.h1.0001-01-01-00000.nc
Opening netcdf history file
test_sb_F_oci_in_high_lat_sst_glacier_MEC.cam.h1.0001-01-01-00000.nc
H_DEFINE: Successfully opened netcdf file


-----------lnd.log-----------
Successfully initialized the land model
begin initial run at:
nstep= 0 year= 1 month= 1 day=
1 seconds= 0

************************************************************

Attempting to read monthly vegetation data .....
nstep = 0 month = 1 day = 1
(GETFIL): attempting to find local file
surfdata_1.9x2.5_wade3_glacier_c220105_mec.nc
(GETFIL): using
/home/wade/snowball/data/ocn_in_high_lat/surfdata_1.9x2.5_wade3_glacier_c220105
_mec.nc
Successfully read monthly vegetation data for
month 1

dtime_sync= 1800 dtime_clm= 1800 mod = 0
(lnd_init_mct) :Atmospheric input is from a prognostic model
BalanceCheck: soil balance error nstep = 0 point = 3665 imbalance = -0.000000 W/m2
clm2: completed timestep 0
BalanceCheck: soil balance error nstep = 1 point = 3655 imbalance = 0.000000 W/m2
clm2: completed timestep 1

 

[mlevy]

Can you run in debug mode? To enable debug, you will need to rebuild the model... so you want to run

$ ./case.build --clean-all
$ ./xmlchange DEBUG=TRUE
$ ./case.build

And then submit as usual.