Problem running cam4 with waccm_mozart

[rddixon@wisc_edu]

I recently compiled CESM 1.0 on a single processor AMD 64 machine running Ubuntu with ifort and icc with the intention of running CAM 4.0 in standalone mode. I was able to successfully run CAM, however, when I configure it to use waccm_mozart I get a segmentation fault while the model is initializing:

chem_surfvals_init: ghg surface values are fixed as follows
co2 volume mixing ratio = 3.788125000000000E-004
ch4 volume mixing ratio = -1.00000000000000
n2o volume mixing ratio = -1.00000000000000
f11 volume mixing ratio = -1.00000000000000
f12 volume mixing ratio = -1.00000000000000
INITIALIZE_RADBUFFER: ntoplw = 15 pressure: 0.655682550000000
Creating new decomp: 262046072
Creating new decomp: 362066045072
Creating new decomp: 362066046072
Warning: Not reading O2_1S from IC file.
O2_1S initialized by "chem_init_cnst"
Warning: Not reading O2_1D from IC file.
O2_1D initialized by "chem_init_cnst"
Warning: Not reading AOA1 from IC file.
AGE-OF-AIR CONSTITUENTS: INITIALIZING AOA1 62
AOA1 initialized by "aoa_tracers_init_cnst"
Warning: Not reading AOA2 from IC file.
AGE-OF-AIR CONSTITUENTS: INITIALIZING AOA2 63
AOA2 initialized by "aoa_tracers_init_cnst"
Warning: Not reading HORZ from IC file.
AGE-OF-AIR CONSTITUENTS: INITIALIZING HORZ 64
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cam 000000000050F221 Unknown Unknown Unknown
cam 000000000093C66B Unknown Unknown Unknown
cam 0000000000937092 Unknown Unknown Unknown
cam 0000000000F46AD3 Unknown Unknown Unknown
cam 0000000000948127 Unknown Unknown Unknown
cam 000000000055B212 Unknown Unknown Unknown
cam 00000000005390CB Unknown Unknown Unknown
cam 00000000005A772F Unknown Unknown Unknown
cam 000000000040424C Unknown Unknown Unknown
libc.so.6 00002AF9099FDABD Unknown Unknown Unknown
cam 0000000000404149 Unknown Unknown Unknown​

During my normal CAM run it was able to pass this without error. Could my problem be that I have values of -1 for several of my surface volume mixing ratios? I figured that they were changed because they were no longer fixed values. It also appears that there might be a missing IC file or fields missing in the IC file, although building the namelist checks for input files and has given me the OK for all of them.

My configure and build namelist commands are as follows:

$camcfg/configure -cam_bld /home/rddixon/cesm1_0/camrun_c/bld
-cam_exedir /home/rddixon/cesm1_0/camrun_c
-cam_exe cam
-cc icc
-fc ifort
-nc_inc $HOME/netcdf/include
-nc_lib $HOME/netcdf/lib
-phys cam4
-chem waccm_mozart
-dyn fv
-hgrid 4x5
-nospmd -nosmp -test -v > &! configure.out

$camcfg/build-namelist -case camrun_c -v -runtype startup -test
-config /home/rddixon/cesm1_0/camrun_c/bld/config_cache.xml
-namelist "&camexp stop_option='nmonths', stop_n=2 /"
-use_case waccm_2000_cam4 > & ! namelist.out​

If anyone has any suggestions or thoughts, your advice and input would be greatly appreciated!

 

[rddixon@wisc_edu]

It ends up that my problem was running the age of air tracer package. After configuring the model with -no-age_of_air_trcs, the model was built and could be run altering the associated output fields from -use_case waccm_2000_cam4. The removed fields were AOA1, AOA2, HORZ, VERT, from fincl1 in the atm_in namelist file. My run is not quite finished yet, but it seems to be running without errors.

Anyone hazard a guess why my compiler/processor can't seem to handle the age of air tracer routine?

After reading the thread by cfletche (http://bb.cgd.ucar.edu/showthread.php?t=1508) I am still troubled by the fact that the values of the chem_surfvals_init for so many species are initialized to -1. It would be easy enough to input values for these fields in the atm_in namelist, but I am worried that perhaps I am missing a input file that would read these values in. I plan on trying to build the model with the chemistry package waccm_ghg next and see if the age of air tracer works with that build and if it possibly suggests additional input files to help my build.

 

[mmills@ucar_edu]

The problem is that you did not have the AOA fields in your IC files. WACCM4 should be built by choosing a WACCM compset and configuring CESM using the create_newcase and configure scripts as described in the CESM user guide. When that is done, an appropriate WACCM IC file will be put in your namelist and downloaded via subversion. You seem to be configuring outside of these scripts, which is causing problems.

The GHG (surfvals) file is only used in WACCM to get CO2 for the Fomichev code in the mesosphere. Therefore the -1 values for other species may be irrelevant.

 

[mmills@ucar_edu]

Here is the relevant log from a normal constant year 2000 (F_2000_WACCM compset) WACCM run:

chem_surfvals_init: ghg surface values are fixed as follows
   co2 volume mixing ratio =  0.368900000000000022E-03
   ch4 volume mixing ratio =  -1.00000000000000000
   n2o volume mixing ratio =  -1.00000000000000000
   f11 volume mixing ratio =  -1.00000000000000000
   f12 volume mixing ratio =  -1.00000000000000000

So the output you are seeing is normal. CO2 is the only relevant quantity in the GHG file; the others should be ignored.

A transient (i.e. B_1850-2000_WACCM_CN) compset will print only the CO2 values in the log file:

chem_surfvals_set: ncdate=  19750706  co2vmr= 0.331177391825636760E-03

The CO2 values in this file are only used for the Fomichev code in mesosphere, not for the lower boundary conditions of the chemical tracers. There is a separate LBC file that has the correct LBCs for CH4, N2O, F11, F12, CO2 and all of the other species with specified mixing ratio LBCs.

 

[rddixon@wisc_edu]

Thanks for all your input. At home I run CAM in standalone mode (because I had difficulties porting the coupled CESM to my machine, just to ask it to run the F dataset), but in order to download the datasets without fuss, I use the CESM script create_newcase, configure it for the same resolution and compset, then use ./check_input_data -export to download all the data that build-namelist checks for with the -test flag.

The output from build-namelist is as follows:

Setting CAM configuration script directory to /home/rddixon/cesm1_0/models/atm/cam/bld
build-namelist was invoked with the commandline:
/home/rddixon/cesm1_0/models/atm/cam/bld/build-namelist -case camrun_c -v -runtype startup -test -config /home/rddixon/cesm1_0/camrun_c/bld/config_cache.xml -namelist &camexp stop_option='nmonths', stop_n=2 / -use_case waccm_2000_cam4
Using CAM configuration cache file /home/rddixon/cesm1_0/camrun_c/bld/config_cache.xml
CCSM inputdata root directory: /home/rddixon/cesm1_0/inputdata
Using namelist definition file /home/rddixon/cesm1_0/models/atm/cam/bld/namelist_files/namelist_definition.xml
Using namelist defaults file /home/rddixon/cesm1_0/models/atm/cam/bld/namelist_files/namelist_defaults_cam.xml
Using namelist use case file /home/rddixon/cesm1_0/models/atm/cam/bld/namelist_files/use_cases/waccm_2000_cam4.xml
Writing namelist files for cam standalone scripts
Writing CICE namelist to ./ice_in
Writing DOCN namelist to ./docn_ocn_in
Writing DOCN stream file to ./docn.stream.txt
Writing CLM namelist to ./lnd_in
Writing driver namelist to ./drv_in
Writing dry deposition namelist to ./drv_flds_in
Writing ocean component namelist to ./docn_in
Writing CAM namelist to ./atm_in
Checking whether input datasets exist locally...
OK -- found sulf_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/sulf/sulfate.M1.nc
OK -- found tgcm_ubc_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/ub/tgcm_ubc.2000.c100203.nc
OK -- found depvel_lnd_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart/dvel/regrid_vegetation.nc
OK -- found xs_long_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/temp_prs_GT200nm_jpl06_c080930.nc
OK -- found sad_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/sulf/SAD_SULF_1849-2100_1.9x2.5_c090817.nc
OK -- found rsf_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc
OK -- found xs_short_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/xs_short_jpl06_c080930.nc
OK -- found electron_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/electron.dat
OK -- found photon_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/photon.dat
OK -- found euvac_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/euvac.nc
OK -- found flbc_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/lb/LBC_1765-2005_1.9x2.5_CMIP5_za_c091204.nc
OK -- found xs_coef_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/effxstex.txt
OK -- found clim_soilw_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart/dvel/clim_soilw.nc
OK -- found solar_parms_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/wa_smax_c100517.nc
OK -- found airpl_emis_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart/emis/emissions.aircraft.T42LR.nc
OK -- found euvacdat_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/phot/euvac.dat
OK -- found snoe_ubc_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/ub/snoe_eof.nc
OK -- found season_wes_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc
OK -- found prescribed_aero_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850-2005_c091112.nc
OK -- found prescribed_aero_datapath = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart_aero/aero
OK -- found solar_data_file = /home/rddixon/cesm1_0/inputdata/atm/cam/solar/spectral_irradiance_Lean_1610-2009_ann_c100405.nc
OK -- found ncdata = /home/rddixon/cesm1_0/inputdata/atm/waccm/ic/wa3548_4x_refb1.3.cam2.i.2000-01-01-00000.nc
OK -- found efield_lflux_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/efld/coeff_lflux.dat
OK -- found bnd_topo = /home/rddixon/cesm1_0/inputdata/atm/cam/topo/USGS-gtopo30_4x5_remap_c050520.nc
OK -- found qbo_forcing_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/qbo/qbocyclic28months.nc
OK -- found absems_data = /home/rddixon/cesm1_0/inputdata/atm/cam/rad/abs_ems_factors_fastvx.c030508.nc
OK -- found efield_hflux_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/efld/coeff_hflux.dat
OK -- found efield_wei96_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/efld/wei96.cofcnts
OK -- found bndtvs = /home/rddixon/cesm1_0/inputdata/atm/cam/sst/sst_HadOIBl_bc_4x5_clim_c061031.nc
OK -- found focndomain = /home/rddixon/cesm1_0/inputdata/atm/cam/ocnfrac/domain.camocn.4x5_USGS_070807.nc
OK -- found aerodep_flx_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1849-2006_1.9x2.5_c090803.nc
OK -- found aerodep_flx_datapath = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart_aero/aero
OK -- found tropopause_climo_file = /home/rddixon/cesm1_0/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc
OK -- found gw_drag_file = /home/rddixon/cesm1_0/inputdata/atm/waccm/gw/newmfspectra40_dc25.nc
OK -- found fpftcon = /home/rddixon/cesm1_0/inputdata/lnd/clm2/pftdata/pft-physiology.c100226
OK -- found fsnowaging = /home/rddixon/cesm1_0/inputdata/lnd/clm2/snicardata/snicar_drdt_bst_fit_60_c070416.nc
OK -- found fatmlndfrc = /home/rddixon/cesm1_0/inputdata/lnd/clm2/griddata/fracdata_4x5_USGS_070110.nc
OK -- found fsnowoptics = /home/rddixon/cesm1_0/inputdata/lnd/clm2/snicardata/snicar_optics_5bnd_c090915.nc
OK -- found fsurdat = /home/rddixon/cesm1_0/inputdata/lnd/clm2/surfdata/surfdata_4x5_simyr2000_c090928.nc
OK -- found fatmgrid = /home/rddixon/cesm1_0/inputdata/lnd/clm2/griddata/griddata_4x5_060404.nc
OK -- found srf_emis_specifier for CH2O -> /home/rddixon/cesm1_0/inputdata/atm/cam/chem/1850-2000_emis/IPCC_emissions_houw_CH2O_1850-2000_1.9x2.5.c090728.nc
OK -- found srf_emis_specifier for CO -> /home/rddixon/cesm1_0/inputdata/atm/cam/chem/1850-2000_emis/IPCC_emissions_houw_CO_1850-2000_1.9x2.5.c090728.nc
OK -- found srf_emis_specifier for NO -> /home/rddixon/cesm1_0/inputdata/atm/cam/chem/1850-2000_emis/IPCC_emissions_houw_NOx_1850-2000_1.9x2.5.c090728.nc
OK -- found rad_climate for D_sulf:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/sulfate_camrt_c080918.nc
OK -- found rad_climate for D_dust1:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/dust1_camrt_c080918.nc
OK -- found rad_climate for D_dust2:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/dust2_camrt_c080918.nc
OK -- found rad_climate for D_dust3:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/dust3_camrt_c080918.nc
OK -- found rad_climate for D_dust4:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/dust4_camrt_c080918.nc
OK -- found rad_climate for D_bcar1:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/bcpho_camrt_c080918.nc
OK -- found rad_climate for D_bcar2:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/bcphi_camrt_c080918.nc
OK -- found rad_climate for D_ocar1:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/ocpho_camrt_c080918.nc
OK -- found rad_climate for D_ocar2:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/ocphi_camrt_c080918.nc
OK -- found rad_climate for D_SSLTA:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/ssam_camrt_c080918.nc
OK -- found rad_climate for D_SSLTC:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/sscm_camrt_c080918.nc
OK -- found rad_climate for D_H2SO4M:/home/rddixon/cesm1_0/inputdata/atm/cam/physprops/sulfuricacid_cam3_c080918.nc

I guess my big question is why isn't the build-namelist checking for an IC AOA tracer file? I'm not to concerned about it since I have worked around it by turning AOA off (and doing my larger runs on bluefire), but thought it might be a possible bug/issue that others might have. Thanks again for your help!

 

[mmills@ucar_edu]

AOA would be one of hundreds of species that could be in an IC file. The scripts that build a namelist can check whether all of the input files listed in the namelist exist where they are supposed to, but they do not look inside each file to make sure it has all of the species required for your setup. I imagine such a mechanism would take a very long time to execute!