error return from ESMF_TimeSet for set_time_float_from_date on 'aerosoldep' file

[strandwg]

This is a fully-coupled run using an 'aerosoldep' file that I created to repeat 1850 conditions but with a model time start year of 3050. The structure of the 'aerosoldep' file is identical to any one of the standard RCP input datasets, and the 'time' and 'date' variables are entirely correct.I noticed a similar error reported here in 2010: https://bb.cgd.ucar.edu/node/1001811 And I've done a thorough check of the new 'aerosoldep' file. Thanks.

 

[eaton]

What CESM model version are you using?

 

[ahu]

The model version I used is cesm_1.0.5

 

[eaton]

What compset are you running?How did you modify the namelist?Which aerosoldep file was modified, and what was the modification.Please post the log output that contains the error messages. 

 

[ahu]

I run in B comset (fully coupled run) which extens the RCP8.5 run beyound 2300, then the forcing is reversed to 1850. Gary created the forcings files. It was fine when the forcings go from RCP8.5 2300 level to 1850. Then we got these error message (if you need everyting from this atm log file, please let e know): open_trc_datafile:
 /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300ext/SourceMods/src.cam/ozone_rcp85_v1_1.9x2.5_L26_1995-3505_c130514.nc
 trcdata_init: file%has_ps =   T
 aero is prescribed in :aero_rcp85_v2_1.9x2.5_L26_1995-3505_c130514.nc
 trcdata_init: data type: INTERP_MISSING_MONTHS file: aero_rcp85_v2_1.9x2.5_L26_1995-3505_c130514.nc
 (GETFIL): attempting to find local file
 aero_rcp85_v2_1.9x2.5_L26_1995-3505_c130514.nc
 (GETFIL): using
 /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300ext/SourceMods/src.cam/aero_rcp85_v2_1.9x2.5_L26_1995-3505_c130514.nc
 open_trc_datafile:
 /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300ext/SourceMods/src.cam/aero_rcp85_v2_1.9x2.5_L26_1995-3505_c130514.nc
 trcdata_init: file%has_ps =   T
 aero dep fluxes are prescribed in :aerosoldep_rcp8.5_monthly_1849-3505_1.9x2.5_c130514.nc
 trcdata_init: data type: INTERP_MISSING_MONTHS file: aerosoldep_rcp8.5_monthly_1849-3505_1.9x2.5_c130514.nc
 (GETFIL): attempting to find local file
 aerosoldep_rcp8.5_monthly_1849-3505_1.9x2.5_c130514.nc
 (GETFIL): using
 /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300ext/SourceMods/src.cam/aerosoldep_rcp8.5_monthly_1849-3505_1.9x2.5_c130514.nc
 open_trc_datafile:
 /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300ext/SourceMods/src.cam/aerosoldep_rcp8.5_monthly_1849-3505_1.9x2.5_c130514.nc
 set_time_float_from_date: error return from ESMF_TimeSet for set_time_float_from_date
 ENDRUN:CHKRC

 

[eaton]

The same code is used to interpolate the ozone, aerosol, and aerosol deposition files.  Since it appears the initialization was successful for the ozone and aerosol files the first thing to check is that the date and datesec variables are correct for the new time slices in the aerosoldep file.  I know Gary said he's checked the file, but I can't think of any other source of this problem.  Perhaps a second set of eyes on it is needed.  If you give me read permission for the ozone, aerosol, and aerosoldep files I'll take a look.

 

[eaton]

I looked at the extended files and didn't see a problem.  So I ran a short standalone CAM test using cesm1_0_5.  (I did the run on yellowstone using the files that Gary copied to /glade).configure  -dyn fv -hgrid 1.9x2.5 -phys cam4

build-namelist   -ignore_ic_year
    -namelist "&atm_in start_type='startup' start_ymd=30510101
         stop_option='nsteps' stop_n=9 restart_option='none' 
solar_data_file='$datadir/SOLAR_TSI_Lean_1610-3500_annual_c140212.nc'
scenario_ghg='RAMPED'
bndtvghg='$datadir/ghg_rcp85_1765-3500_c140212.nc'
prescribed_aero_file='$datadir/aero_rcp85_v2_1.9x2.5_L26_1995-3500_c140212.nc'
prescribed_aero_type='INTERP_MISSING_MONTHS'
aerodep_flx_file='$datadir/aerosoldep_rcp8.5_monthly_1849-3500_1.9x2.5_c140212.nc'
prescribed_ozone_file='$datadir/ozone_rcp85_v1_1.9x2.5_L26_1995-3500_c140212.nc'
prescribed_ozone_name='O3'
prescribed_ozone_type='INTERP_MISSING_MONTHS'  /"

This run was successful.  Note that I started at 30510101 rather than 30500101 since some of the new datasets have the first timesample at 30500116.So it looks like something in your code base is not consistent with cesm1_0_5. 

 

[ahu]

Now Gary add an extra year for all forcing files which is located at/project/projectdirs/ccp/people/ahu/ccsm4_forcings/ and can be seen from hopper at nersc. Now the CCSM log file show a message as:Rank 86 [Thu Feb 27 21:23:47 2014] [c9-3c1s5n0] application called MPI_Abort(MPI_COMM_WORL
D, 1) - process 86
 column cbalance error =    8.9406967163085938E-008        30421
 begcb       =    -420699735.1155245    
 endcb       =    -600922468.0455700    
 delta store =    -180222732.9300455    
 input mass  =     0.000000000000000    
 output mass =     180222732.9300454    
 net flux    =    -180222732.9300454    
 nee         =     180222732.9300454    
 gpp         =     0.000000000000000    
 er          =     180222732.9300454    
 col_fire_closs         =     0.000000000000000    
 col_hrv_xsmrpool_to_atm =     0.000000000000000    
 dwt_closs         =     0.000000000000000    
 product_closs         =     0.000000000000000    
 ENDRUN: called without a message string
_pmii_daemon(SIGCHLD): [NID 04820] [c9-3c1s5n0] [Thu Feb 27 21:23:48 2014] PE 86 exit sign
al Aborted
[NID 04820] 2014-02-27 13:23:48 Apid 26393508: initiated application termination
Application 26393508 exit codes: 134
Application 26393508 exit signals: Killed
Application 26393508 resources: utime ~1439s, stime ~2s, Rss ~192672, inblocks ~11067453,
outblocks ~47998335I have looked other log files and could not find any error message. If possible, you can look at my run diretory on hopper:/scratch/scratchdirs/ahu/b40.rcp8.5.2300ext/runThere is a core file, but I don;t know how to look at it. From other log files, it seems that the initialization of the model is OK. Thanks! 

 

[eaton]

using gdb on your core file shows:#5  0x000000000046939d in abortutils::endrun (msg="") at /project/projectdirs/ccsm1/collections/cesm1_0_beta23/models/atm/cam/src/utils/abortutils.F90:36
#6  0x00000000008be2d1 in cnbalancecheckmod::cbalancecheck (lbc=-40912, ubc=-40912, num_soilc=-40912, filter_soilc=...)
    at /project/projectdirs/ccsm1/collections/cesm1_0_beta23/models/lnd/clm/src/biogeochem/CNBalanceCheckMod.F90:254
#7  0x00000000009bcc10 in clm_driver::clm_drv (doalb=4294967295, nextsw_cday=1.8125, declinp1=-0.40193890144331151, declin=-0.40196795685562658, rstwr=.FALSE.,
    nlend=.FALSE., rdate=
../../gdb/exec.c:614: internal-error: failed internal consistency check
A problem internal to GDB has been detected,
further debugging may prove unreliable.

I'm not sure why gdb is unhappy.  However, the location of the stack traceback is consistent with the messages in the log files.The ccsm.exe executable in this directory is dated 22 August 2012.  That seems like a potential problem to me.  I'd recommend rebuilding the executable to be sure it's consistent with the current programming environment. At any rate the problem is not associated with the extended datasets.  

 

[ahu]

I have rebuilt the model and run it from year 3050. The model failed in the same manner. Then, I backed up from 3050 to 3049 which is the year I had last successful run before the extention of the forcing data. But the model still failed and gave the similar error message. Then I tried to build the model for an 1850 control run. In this run, I got an error message ofENDRUN:SPMDINIT_DYN: less than 3 latitudes per subdomainThese runs are done at hopper. You can find the run directories:/scratch/scratchdirs/ahu/b40.rcp8.5.2300extcon/run --- new 1850 control run/scratch/scratchdirs/ahu/b40.rcp8.5.2300ext/run --- original runI wonder whether this is related to the way the model set up now and changes of the hopper eviroment.

 

[eaton]

The FV 1.9x2.5 grid has 96 latitudes and since it must have at least 3 latitudes in each subdomain the max number of latitude subdomains is 32.  From your atm log output I see that 64 latitude subdomains have been requested.  This is controled by the namelist variable npr_yz which is currently set to npr_yz=64,12,12,64.  This is consistent with asking for 768 tasks for the atm.  Try setting up the PE layout to ask for only 384 tasks which should give a npr_yz setting of 32,12,12,32.

 

[eaton]

I need to have read permissions in your new directory on hopper to have a look at the latest build problem. 

 

[ahu]

Sorry. I have changed the permission now. Thanks!

 

[ahu]

Hi Brian,
I have just created a new case and changed the env_config file and rebuilt the model. I got the same error message as I have reported today. So you can have a look of the error message:Fri Mar 7 15:28:51 PST 2014 /scratch/scratchdirs/ahu/b40.rcp8.5.2300extnew/mct/mct.bldlog.
140307-152825
Copying source to CCSM EXEROOT...
New build of MCT
Running configure...
for OS=CNL MACH=hopp2
cat: Filepath: No such file or directory
cat: Srcfiles: No such file or directory
/global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300extnew/Tools/mkSrcfiles > /scrat
ch/scratchdirs/ahu/b40.rcp8.5.2300extnew/mct/Srcfiles
cp -f /scratch/scratchdirs/ahu/b40.rcp8.5.2300extnew/mct/Filepath /scratch/scratchdirs/ahu
/b40.rcp8.5.2300extnew/mct/Deppath
/global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300extnew/Tools/mkDepends Deppath S
rcfiles > /scratch/scratchdirs/ahu/b40.rcp8.5.2300extnew/mct/Depends
./configure  FC="time ftn" F90="time ftn" FFLAGS=" -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLIN
UX -DSEQ_ -DCATAMOUNT -DFORTRANUNDERSCORE -DNO_SHR_VMATH -D_USE_FLOW_CONTROL -DTHREADED_OM
P -i4 -target=linux -gopt -Mlist -time -Mextend -byteswapio -Mflushz -Kieee -Ktrap=fp   -m
p" --host=Linux 
configure: WARNING: If you wanted to set the --build type, don't use --host.
    If a cross compiler is detected then cross compile mode will be used.
checking for Linux-cc... no
checking for cc... cc
checking for C compiler default output file name... configure: error: C compiler cannot cr
eate executables
See `config.log' for more details.
gmake: *** [configure] Error 77
cp: cannot stat `Makefile.conf': No such file or directory
Makefile:4: Makefile.conf: No such file or directory
make: *** No rule to make target `Makefile.conf'.  Stop.

 

[ahu]

When I create new case. I run my script at/global/project/projectdirs/ccsm1/collections/cesm1_0_beta23/scriptsThe commend I used is./create_newcase -case /global/u2/a/ahu/franklin/ccsm3_0_1_beta27/b40.rcp8.5.2300extnew -res f09_g16 -compset B1850CN -mach hopp2 -skip_rundb

 

[eaton]

I'm not surprised that the MCT build is failing in this old tag.  cesm1_0_beta23 was created on 2011-08-10.  I'd recommend moving to the latest release tag in the 1.0 series, cesm1_0_5.

 

[ahu]

When I am running my experiment, cesm1_0_beta23 is the version I could use on Hopper for CCSM4. So do you know which version I should use on Hopper for CCSM4 run? Thanks!