Error while running CESM1.2.2: forrtl error (72)

[blumer@climate_unibe_ch]

 Dear allI am trying to run a simulation on CESM1.2.2 with the compset B_1850-2000_BGC_BPRP, intel-compiler and resolution 0.9x1.25_gx1v6 on a Cray XC30 machine. The model compiles fine but when running it, it crashes with the following fortran runtime library error:forrtl: error (72): floating overflowWhereas the code crashes in./models/ocn/pop2/source/tavg.F90 at line 2930: TAVG_BUF_2D(:,:,block,bufloc) + dtavg*ARRAY.However, I can't find the reason for the crash and don't know how the solve it... Has anyone got an advice on this?Thank you very much and best wishes.Sandro 

 

[jedwards]

What compiler version are you using?   Have you tried running in DEBUG mode to see if you get more information?

 

[blumer@climate_unibe_ch]

Hi,
Thank you for the response, I use:- intel compiler version 14.0.2.- cmake 3.0.1- cray-mpich 7.0.4- cray-netcdf-hdf5parallel 4.3.2. Yes, I tried to run it in the debug-mode, but the first error which occurred is the "forrtl: error (72): floating overflow": offset:            0 len:            0
 offset:            0 len:            0
        26          -1
 offset:            0 len:            0
 offset:            0 len:            0
 piodarray::read_darray_nf_double: {comp,io}_rank:            0           0
 offset:            0 len:       122880
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source            
cesm.exe           00000000076BF829  Unknown               Unknown  Unknown
cesm.exe           00000000076BE0FE  Unknown               Unknown  Unknown
cesm.exe           000000000766F8D2  Unknown               Unknown  Unknown
cesm.exe           00000000075FB7F3  Unknown               Unknown  Unknown
cesm.exe           000000000760577B  Unknown               Unknown  Unknown
cesm.exe           00000000071F71E0  Unknown               Unknown  Unknown
cesm.exe           000000000531C584  tavg_mp_accumulat        2930  tavg.F90
cesm.exe           0000000006360D60  ecosys_mod_mp_eco        7001  ecosys_mod.F90
cesm.exe           0000000005D4CD75  passive_tracers_m        1216  passive_tracers.F90
cesm.exe           000000000528B31E  step_mod_mp_step_         193  step_mod.F90
cesm.exe           0000000005010C4A  ocn_comp_mct_mp_o         632  ocn_comp_mct.F90
cesm.exe           000000000049E4D6  ccsm_comp_mod_mp_        4046  ccsm_comp_mod.F90
cesm.exe           00000000004AF4FE  MAIN__                     91  ccsm_driver.F90
cesm.exe           0000000000401056  Unknown               Unknown  Unknown
cesm.exe           00000000076D4BF1  Unknown               Unknown  Unknown
cesm.exe           0000000000400F25  Unknown               Unknown  UnknownKind regards

 

[jedwards]

Try the following change in ecosys_parms.F90, < indicates your code > indicates the replacement 59c59,61<         grazee_ind                            ! which grazee category does autotroph belong to--->           grazee_ind                            , &! which grazee category does autotroph belong to>           Chl_ind, C_ind, Fe_ind,             & ! tracer indices for Chl, C, Fe content>           Si_ind, CaCO3_ind                ! tracer indices for Si, CaCO3 content61,66c63,66<         Chl_ind, C_ind, Fe_ind,             & ! tracer indices for Chl, C, Fe content<         Si_ind, CaCO3_ind,                  & ! tracer indices for Si, CaCO3 content<         kFe, kPO4, kDOP, kNO3, kNH4, kSiO3, & ! nutrient uptake half-sat constants<         Qp,                                 & ! P/C ratio<         gQfe_0, gQfe_min,                   & ! initial and minimum fe/C ratio<         alphaPI,                            & ! init slope of P_I curve (GD98) (mmol C m^2/(mg Chl W sec))--->           kFe, kPO4, kDOP, kNO3, kNH4, kSiO3, & ! nutrient uptake half-sat constants>           Qp,                                 & ! P/C ratio>           gQfe_0, gQfe_min,                   & ! initial and minimum fe/C ratio >           alphaPI,                            & ! init slope of P_I curve (GD98) (mmol C m^2/(mg Chl W sec))

 

[blumer@climate_unibe_ch]

Hi,
After carrying out the above changes, the model still crashes and (altough in debug mode) I only get the following message:ionf_mod.F90         252 CFILE closing :      -131072
 ionf_mod.F90         252 CFILE closing :      -131072
 piolib_mod.f90        2828 close:       131072
 ionf_mod.F90         252 CFILE closing :       131072
 ionf_mod.F90         262           0
 ionf_mod.F90         264           0


Thank you in advance for further support... Best wishes   

 

[jedwards]

It looks like you turned on the PIO_DEBUG flag as well as the DEBUG flag?   Try with PIO_DEBUG=0, attach the cesm.log to your reply should it crash again.

 

[blumer@climate_unibe_ch]

Yes, it crashed again. Please find the log-file attached.

 

[blumer@climate_unibe_ch]

Dear all,I am still struggling with the problem above. Does anyone have further suggestions? I am grateful for every advice.  Thanks and best wishessandro

 

[jedwards]

The error indicates that you are out of memory - try increasing the pe-layout.  

 

[urs_beyerle@env_ethz_ch]

It looks like POP is running out of memory. Since you are using intel 14.x compiler, you most probably hit the following bug (memory leak in pop2)http://bugs.cgd.ucar.edu/show_bug.cgi?id=2121Cheers, Urs 

 

[blumer@climate_unibe_ch]

Yes, with this modifications it works properly! Thanks for the advice.Best regardsSandro