Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM Error as Segmentation fault at particular step

Dear CESM Users,
I am running the cesm1.2.0 for a compset equvalant to F compset for RCP85 emission scenario. I created the case using the following command lines.create_newcase -case HadGEM2_RCP85_r3i1p1_9x125_2006-2099 -res f09_f09 -user_compset RCP8_CAM5_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV -mach IITD -mpilib mpich -compiler pgi I am supplying the boundary conditons created using CMIP5 HadGEM2-ES output ts and sic and cesm boundary conditons file (sst_HadOIBl_bc_0.9x1.25_1850_2013_c140701.nc). I had added the model climatological bias(from obs where ss_HadOIB is treated as obs)  for present day period (1970-1990) to the future ts and sice from of HadGEM2-ES model output of RCP85.I started the case from 2006 to 2099, model runs sucessful upto 2090-12-31 but it stuck at 2091-01 (in January month). cesm log file gives the following error message.-------------------------------------------------------------------------------------------------------------------------CALEDDY: Warning, CL with zero TKE, i, kt, kb             2            5
           17
 CALEDDY: Warning, CL with zero TKE, i, kt, kb             2            5
           27
 CALEDDY: Warning, CL with zero TKE, i, kt, kb             2            5
           27
 imp_sol: Time step   1.8000000000000E+03 failed to converge @ (lchnk,lev,col,nstep) =   4938    19     2******
 imp_sol : @ (lchnk,lev,col) =          4938           19            2  failed
             1  times
 CALEDDY: Warning, CL with zero TKE, i, kt, kb             2            5
           20
 QNEG3 from chemistry/num_a2:m= 21 lat/lchnk=   4323 Min. mixing ratio violated at   15 points.  Reset to  1.0E-36 Worst =-1.5E-08 at i,k=   2 20
 QNEG3 from chemistry/num_a2:m= 21 lat/lchnk=   4324 Min. mixing ratio violated at   16 points.  Reset to  1.0E-36 Worst =-1.5E-08 at i,k=   2 20
--------------------------------------------------------------------------
WARNING: A process refused to die despite all the efforts!
This process may still be running and/or consuming resources.

Host: cn126
PID:  12104

--------------------------------------------------------------------------
[cn126:12078] 23 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[cn126:12078] 1 more process has sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 143 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 1 more process has sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 214 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 96 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 24 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 1 more process has sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 143 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 120 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 72 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 1 more process has sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 48 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 71 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 24 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
[cn126:12078] 1 more process has sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
--------------------------------------------------------------------------
mpirun noticed that process rank 124 with PID 0 on node cn162.hpc.iitd.ac.in exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[cn126:12078] 23 more processes have sent help message help-orte-odls-base.txt / orte-odls-base:could-not-kill
---------------------------------------------------------------------------I am attaching the full log file here the snapshot of 2091-01-15 boundary condition SST and model original TS values here. I am also attaching the logfile. Please look suggest source of error.Regards,  
 
Top