Segmentation fault problem after Divergence damping step in aqua-planet simulation.

[rakeshtejak@gmail_com]

Hi,I have set up CAM5.3 standalone model for aqua planet experiment by using cam3_5_1 physics with disabled chemistry. I have changed the initial conditionand ran it for one month. But the CAM execution got stopped after divergence damping step with the following error messase: field_atm,lmerge, imerge, xmerge, omerge= Fall_flxdst4 F T T T(seq_mct_drv) : Calling atm_init_mct phase 2(seq_mct_drv) : Initialize atm component phase 2 ATMFV subcycling - nv, n2, nsplit, dt = 2 1 4225.0000000000000Divergence damping: use 2nd order damping[compute-0-6:06096] *** Process received signal ***[compute-0-6:06096] Signal: Segmentation fault (11)[compute-0-6:06096] Signal code: Address not mapped (1)[compute-0-6:06096] Failing at address: 0x7f2b6cef3038[compute-0-6:06097] *** Process received signal ***[compute-0-6:06097] Signal: Segmentation fault (11)[compute-0-6:06097] Signal code: Address not mapped (1)[compute-0-6:06097] Failing at address: 0x7f0f7e697620[compute-0-6:06096] *** End of error message ***[compute-0-6:06097] *** End of error message ***--------------------------------------------------------------------------mpiexec noticed that process rank 0 with PID 6096 on node compute-0-6.local exited on signal 11 (Segmentation fault).-------------------------------------------------------------------------- It would be great if you could help me to resolve this problemPlease find the attached log file.  Thank you. 

 

[jedwards]

First you should check that the model runs okay without your modifications.  

 

[rakeshtejak@gmail_com]

Model is running fine without any modifications. The problem is only when i changed the initial condition. Thank you

 

[eaton]

Have you tried running with debug options on (-debug option to configure)?  I'm not seeing anything in the log file that's indicating the problem.  You need to find where the segfault is originating. 

 

[rakeshtejak@gmail_com]

I Have also tried the "-debug" option while configuring. As i tried to check the root of this segmentation fault problem. I couldn't get any clear idea about it.

 

[eaton]

Try adding "state_debug_checks=.true." to your namelist.  This may catch a problem resulting from bad data in the IC file.Another suggestion is to get help from a local expert to generate a traceback or core file which can be used to determine where the segfault originates.  That's the best option when there is not sufficient information in the log files to understand the problem.

 

[olson]

Since the run is failing immediately at startup, it could very well be some field in your initial file contains a very unrealistic value or values.  Plot the fields from the initial file to see if they look reasonable. Alsolook for "missing" values.  Print maximum and minimums for each field as well.  In other words, make sure the initialfile looks satisfactory before trying to debug the model.  If you don't find anything obviously wrong with your initial file,you could also check if it fails for other configurations of the model besides aquaplanet.