Segmentation fault on ccsm_init()

[joy_merwin@gmail_com]

Hi,
Im trying to run CAM 5 in standalone mode following the instructions given in the CESM tutorial page. I tried this both on an intel cluster and an Ubuntu PC (intel i7, 16 GB RAM).I was successfully able to compile the serial version with the following configure statement../CESM/cesm1_2_0/models/atm/cam/bld/configure -fc ifort -cc icc -fflags '-shared-intel -mcmodel=medium -heap-arrays' -dyn fv -hgrid 10x15 -ice sice -lnd slnd -rof srof -nosmp -nospmd -chem none -ldflags '-limf -lm'

I then built the namelist using  ../CESM/cesm1_2_0/models/atm/cam/bld/build-namelist -config ../cam_test/config_cache.xml -csmdata ../cam_data/ -test

Running then gives a segmentation fault. Tried all the normal things, ulimit -s unlimited and adding the mcmodel=medium  and heap_arrays argument to ifort. The error it gives isforrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source            
cam                0000000000F08AA6  Unknown               Unknown  Unknown
cam                0000000000F015D4  Unknown               Unknown  Unknown
cam                0000000000EC16A0  Unknown               Unknown  Unknown
cam                0000000000545A31  Unknown               Unknown  Unknown
cam                000000000063053F  Unknown               Unknown  Unknown
cam                0000000000E8F571  Unknown               Unknown  Unknown
cam                00000000004A7128  Unknown               Unknown  Unknown
cam                000000000044C525  Unknown               Unknown  Unknown
cam                0000000000517F71  Unknown               Unknown  Unknown
cam                000000000053FCCC  Unknown               Unknown  Unknown
cam                0000000000409D2C  Unknown               Unknown  Unknown
libc.so.6          0000003D3541D994  Unknown               Unknown  Unknown
cam                0000000000409C39  Unknown               Unknown  Unknown

The model dies as soon as it exits ccsm_init(), which indicates it is some kind of memory issue (i.e, it prints a debug message put at the end ccsm_init() but does not print the same debug message put just after the place where ccsm_init() is called), but I'm all out of tricks to try out. The same error cropped up on the cluster as well  as an aside, the build_namelist script omits to mention that I need certain aerosol files for the radiation code. It crashed when it could not find these files, had to manually identify which they were and download them to their proper locations.
Any pointers/help would be greatly appreciated! TIA,Joy

 

[eaton]

The configuration you are building is not supported, and I'm suspcious that it will not work.  You have removed the land and sea ice, but not specified another component which could supply values on those parts of the globe.  I recommend starting from a supported configuration.  Based on what you've already specified maybe a cam4 aquaplanet run would be a good place to start.../CESM/cesm1_2_0/models/atm/cam/bld/configure -fc ifort -cc icc -dyn fv -hgrid 10x15 -phys cam4 -ocn aquaplanet -nosmp -nospmdIf you run build-namelist with the -test option it will check that all the boundary datasets required by the run are on a local disk, and will report any that are not.

 

[joy_merwin@gmail_com]

I marked the previous comment as spam while hitting reply, please excuse me!
Thanks for the tip, it worked. I then used the default ice and lnd components (cice and clm).It compiles and runs well.My previous configure script arguments were based on the assumption that for a standalone CAM run one would not need to include cice and clm, which would make it more of an earth system model rather than an atmosphere model. Obviously my assumptions were faulty.About the build-namelist, well, I did run it with -test which gave me a list of files to download. However, it omitted to mention the aerosol files, which I found it needed only after
 the model ran and crashed. Even with the new configuration, the -test option does not even list the files while going through the required files, which I find strange.
So, I'm assuming that the current configuration (with CICE and CLM) actually only does some sort of prescribed climatology and no interactive models? I'm building from the configure
 script in the atm/cam/bld folder. Thanks again for all the help,Joy

 

[eaton]

Standalone CAM is an overloaded term.  It generally implies that you are using a "standalone" script to build/run CAM rather than using the CESM scripts.  But the standard configuration for running CAM, by which I mean what you get by specifying just the dycore and the resolution to configure, is the same as what you get from using a CESM "FC5" compset, i.e., CAM w/ the "cam5" physics package,  CLM (fully active land), DOCN (prescribed SST and ice fraction), and CICE in thermodynamic mode.If I run the configure command: "configure  -dyn fv -hgrid 10x15 -phys cam5", and then run build-namelist just specify the location of the input datasets and the -test option, I get the follow output for the aerosol files:OK -- found ext_frc_specifier for SO2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so2_elev_2000_c090726.nc
OK -- found ext_frc_specifier for bc_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_bc_elev_2000_c090726.nc
OK -- found ext_frc_specifier for num_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a1_elev_2000_c090726.nc
OK -- found ext_frc_specifier for num_a2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a2_elev_2000_c090726.nc
OK -- found ext_frc_specifier for pom_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_oc_elev_2000_c090726.nc
OK -- found ext_frc_specifier for so4_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a1_elev_2000_c090726.nc
OK -- found ext_frc_specifier for so4_a2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a2_elev_2000_c090726.nc
OK -- found srf_emis_specifier for DMS = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
OK -- found srf_emis_specifier for SO2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so2_surf_2000_c090726.nc
OK -- found srf_emis_specifier for SOAG = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_soag_1.5_surf_2000_c100217.nc
OK -- found srf_emis_specifier for bc_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_bc_surf_2000_c090726.nc
OK -- found srf_emis_specifier for num_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a1_surf_2000_c090726.nc
OK -- found srf_emis_specifier for num_a2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_num_a2_surf_2000_c090726.nc
OK -- found srf_emis_specifier for pom_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_oc_surf_2000_c090726.nc
OK -- found srf_emis_specifier for so4_a1 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a1_surf_2000_c090726.nc
OK -- found srf_emis_specifier for so4_a2 = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mozart_aero/emis/ar5_mam3_so4_a2_surf_2000_c090726.nc
OK -- found mode_defs for so4_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
OK -- found mode_defs for pom_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/ocpho_rrtmg_c101112.nc
OK -- found mode_defs for soa_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc
OK -- found mode_defs for bc_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc
OK -- found mode_defs for dst_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/dust4_rrtmg_c090521.nc
OK -- found mode_defs for ncl_a1 = /fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
OK -- found mode_defs for so4_a2 = /fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
OK -- found mode_defs for soa_a2 = /fs/cgd/csm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc
OK -- found mode_defs for ncl_a2 = /fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
OK -- found mode_defs for dst_a3 = /fs/cgd/csm/inputdata/atm/cam/physprops/dust4_rrtmg_c090521.nc
OK -- found mode_defs for ncl_a3 = /fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
OK -- found mode_defs for so4_a3 = /fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
OK -- found rad_climate for mam3_mode1 = /fs/cgd/csm/inputdata/atm/cam/physprops/mam3_mode1_rrtmg_c110318.nc
OK -- found rad_climate for mam3_mode2 = /fs/cgd/csm/inputdata/atm/cam/physprops/mam3_mode2_rrtmg_c110318.nc
OK -- found rad_climate for mam3_mode3 = /fs/cgd/csm/inputdata/atm/cam/physprops/mam3_mode3_rrtmg_c110318.nc

That includes all the surface emissions for the prognostic modal aerosols and all the optics files for those aerosols.  If instead I had run configure with the additional argument "-chem none" then the build-namelist command in test mode would give me the following dataset needed for the prescribed modal aerosols:OK -- found prescribed_aero_file = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mam/aero/mam3_1.9x2.5_L30_2000clim_c130319.nc
OK -- found prescribed_aero_datapath = /fs/cgd/csm/inputdata/atm/cam/chem/trop_mam/aeroalong with the same datasets as listed above for the optical properties of the aerosol modes and species.The thing thats important to realize is that the configure command writes a config_cache.xml file which is read by build-namelist in order to determine how CAM was configured which enables it to provide reasonable default values for the namelist.  So be sure to always run build-namelist after configure so that the generated namelists are consistent with the build.

 

[joy_merwin@gmail_com]

That was probably the issue. Will keep that in mind for future reference. Thanks!