run time error with MOAR historical forcing in CPLHIST3HrWx mode

[koichi_sakaguchi@pnnl_gov]

Dear all,

I am trying to run off-line CLM with the atmospheric forcing from the CCSM4 historical simulation for CMIP5 ('Mother of All Run'), but my simulation gets stalled at or before reading in the forcing data file. I wonder if anybody else had similar problems.


I followed CLM4 user guide
http://www.cesm.ucar.edu/models/cesm1.0/clm/models/lnd/clm/doc/UsersGuide/x9361.html
(and links from there to another example and data atm model guide)

and created a case like this on bluefire

create_newcase -case ../cases/CNDV_MOAR_test01 -res f09_g16 -compset I_1850-2000_CN -mach bluefire

and edit env_conf.xml file like this







(the first one 'DATM_MODE' is not edited in CLM4 user guide, but I assumed I need to change this as well)

then build and run, but the job reaches the time limit even before executing the first time step.

The last line of the atm log file is
(datm_comp_init) flds_strm = strm_tbot:strm_z:strm_pbot:strm_shum:strm_lwdn


While for a successful off-line simulation with Qian forcing, for example, it usually continues to:
(shr_dmodel_readLBUB) reading file: /cgd/tss/atm_forcing.datm7.Qian.T62.c080727/Solar6Hrly/clmforc.Qian.c2006.T62.Solr.1972-01.nc 1
….


The stream text files seem to have correct file paths:

/glade/data01/CMIP5/CCSM/csm/b40.20th.track1.1deg.012/cpl/hist/b40.20th.track1.1deg.012.cpl.ha2x3h.1850-01.nc

I can see these files from my home directory and can ncdump them.
I have browsed through datm_comp_init and a function called 'mct_aVect_indexRA' might be relevant, but can not locate it in the source code yet.
I noted the bug report 1399 (can't run with 83 or more years of forcing), so using only 50-year period.

I do not see any explicit error messages in other log files either. The land log file was not written yet.
The coupler log file ends with
(seq_mct_drv) : Initialize atm component


The ccsm log file looks like below before it reports exiting status
…..
65:8 MB memory alloc in MB is 8.00
65:8 MB memory dealloc in MB is 0.00
65:Memory block size conversion in bytes is 1020.51
76:8 MB memory alloc in MB is 8.00
76:8 MB memory dealloc in MB is 0.00
76:Memory block size conversion in bytes is 1020.02
May 26 20:58:56 2012 389310 4 7.06 handleTSRegisterTerm(): TS reports task pid on host killed or core dumped
INFO: 0031-656 I/O file STDOUT closed by task 64
2:IB RDMA initialization completed successfully
INFO: 0031-656 I/O file STDERR closed by task 64
INFO: 0031-251 task 64 exited: rc=-127
78:Communication statistics of task 78 is associated with task key: 2254146812_78
101:Communication statistics of task 101 is associated with task key: 2254146812_101
110:Communication statistics of task 110 is associated with task key: 2254146812_110
115:Communication statistics of task 115 is associated with task key: 2254146812_115
116:Communication statistics of task 116 is associated with task key: 2254146812_116
67:Communication statistics of task 67 is associated with task key: 2254146812_67
0:Communication statistics of task 0 is associated with task key: 2254146812_0
6:Communication statistics of task 6 is associated with task key: 2254146812_6
32:Communication statistics of task 32 is associated with task key: 2254146812_32
39:Communication statistics of task 39 is associated with task key: 2254146812_39
50:Communication statistics of task 50 is associated with task key: 2254146812_50
6:
6:Running: ./ccsm.exe
6:Please wait...
6:
6:Memory usage for ./ccsm.exe (task # 0) is: 212592 KB. Exit status: 143. Signal: 0
….


If you have any suggestions, please let me know.

Sincerely,

Koichi

 

[sacks]

Hi Koichi,

I recently experienced a similar problem.

It looks like you want to start with the first year of the MOAR forcing data -- 1850. In that case, I think you should set DATM_CPL_YR_ALIGN to 1850 rather than 1 -- assuming RUN_STARTDATE (in env_conf) is 1850-01-01.

This "align" variable means: which year of your simulation (with "year" meaning the year that appears in the history file date stamps, etc.) corresponds to the first year of forcing data.

The underlying problem is that, when you specify an "align" year of 1, the model is trying to start with a year of forcing data in the middle of the time series. This requires datm to read in many of the earlier files to find the right start file. With the large-block file system on bluefire, this can take a very long time.

Bill

 

[koichi_sakaguchi@pnnl_gov]

Hi Bill,

Thank you so much for your reply. I have not came back to this website to recognize it until today...

I've been trying several things, and finally made my simulations run, and I thought I needed two things. The first is your answer, DATM_CPL_YR_ALIGN to 1850. Now I understand why it needs to be done. At the same time I also applied the bug fix 1339 to shr_stream_mod.F90, but according to your explanation probably I don't need this bug fix.

Thanks again,

Koichi