No output Netcdf file generated CESM2.1.3

[pritam_physics@gmail_com]

I am running a QPC6 aquaplanet in cesmv2.1.3. the model runs for stipulated time without error but does not generate any output netcdf data. The cesm.log and atm.log show that it cannot find some NETCDF input attribute and variables. Showing below the tail of the cesm.log file and also attached.

calcsize j,iq,jac, lsfrm,lstoo 2 5 2 19 23
GPTLprint_memusage: Using bytesperblock=4096
size=1284.4 MB rss=373.6 MB share=46.5 MB text=29.7 MB datastack=0.0 MB
size=1332.4 MB rss=373.7 MB share=46.6 MB text=29.7 MB datastack=0.0 MB
....
size=1332.4 MB rss=373.7 MB share=46.6 MB text=29.7 MB datastack=0.0 MB
aprun: Apid 942830: Caught signal Terminated, sending to application
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 0000000001CD9A54 Unknown Unknown Unknown
libpthread-2.26.s 00002AAAB4B502D0 Unknown Unknown Unknown
cesm.exe 00000000013FF3E6 modal_aero_wateru 545 modal_aero_wateruptake.F90
cesm.exe 000000000101E820 aero_model_mp_aer 1082 aero_model.F90
cesm.exe 00000000006E389A physpkg_mp_tphysb 2206 physpkg.F90
cesm.exe 00000000006DC868 physpkg_mp_phys_r 1054 physpkg.F90
cesm.exe 00000000004FD6C6 cam_comp_mp_cam_r 258 cam_comp.F90
cesm.exe 00000000004EDDEA atm_comp_mct_mp_a 454 atm_comp_mct.F90
cesm.exe 0000000000434830 component_mod_mp_ 728 component_mod.F90
cesm.exe 000000000041A474 cime_comp_mod_mp_ 3465 cime_comp_mod.F90
cesm.exe 000000000043447C MAIN__ 125 cime_driver.F90
cesm.exe 00000000004167D2 Unknown Unknown Unknown
libc-2.26.so 00002AAAB50B834A __libc_start_main Unknown Unknown
cesm.exe 00000000004166EA Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 0000000001CD9A54 Unknown Unknown Unknown
...

 

[jedwards]

This is not an IO problem - the model is blowing up in microphysics. I am moving your post to the CAM forum.

 

[cacraig]

There is no immediate answer to what exactly may be causing the program to crash. The backtraces indicate that each node was in a different location and it's not apparent which might be the culprit.

One thing to check is to make sure you are not running out of disk space.

If you have not reached your quota on disk space, I would suggest that you repeat the run, but have it stop on timestep 1258 (the last timestep which appears to have run successfully for you). This will give you netCDF files which you can examine and see if the values all look reasonable. It will also give you restart files which will allow you to rapidly get to the place which is giving you an error and allow you to debug the problem more quickly.

 

[pritam_physics@gmail_com]

Thank you for your reply, I will check those. I recently re ran the model with increased number of cores from earlier 96 to now 240 (10 nodes*24). It is now generating the output netcdf files. This somehow has worked for me as I was trying out different things. Though the 'netcdf variable/attribute not found' issue still persists. even though it does not give any fatal errors. You may suggest if you have any comment.

 

[cacraig]

I am glad that you got the job working.

The "NetCDF: Variable not found" and "NetCDF: Attribute not found" messages are a side effect of NetCDF and are expected in every run, even runs which are 100% correct. They are not indicative of a problem in the run.