tp_core.F90 error on customised ncdata

[thibaut_lurton@cnrs-orleans_fr]

Hello to all,I have recently customised the chemistry module on compset FSDWSF in order to take into account CH2Br2 and CHBr3. The pre-processing step went perfectly fine, in fact the model ran ok with my modifications, until I realised that I should also give initial conditions for those two species.So I added two extra variables to my regular ncdata file, which I extracted from the file f2000_stratmam7_alpha03c_chem2_1.9x2.5_L30_0011-01-01-00000_c130328.nc. (I had to pad the upper levels in order to reach lev=88).Now, when I run my case with the modified ncdata file, the run fails and cesm.log ends with:[...](shr_stream_set) size of filename =      1
(shr_stream_set) filename = /scratch/CESM/inputdata/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_clim_c061031.nc

MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
forrtl: severe (408): fort: (3): Subscript #1 of the array QV has value -18 which is less than the lower bound of 1

Image              PC                Routine            Line        Source
cesm.exe           00000000066B8E90  Unknown               Unknown  Unknown
cesm.exe           00000000038ADC16  tp_core_mp_xtpv_          379  tp_core.F90
cesm.exe           00000000038A3410  tp_core_mp_tp2d_          231  tp_core.F90
cesm.exe           000000000389CCC5  tp_core_mp_tp2c_          119  tp_core.F90
cesm.exe           0000000003814ABA  sw_core_mp_d_sw_          893  sw_core.F90
cesm.exe           00000000033CD842  cd_core_                 1110  cd_core.F90
cesm.exe           00000000023C1116  dyn_comp_mp_dyn_r        1822  dyn_comp.F90
cesm.exe           00000000019AADCF  stepon_mp_stepon_         422  stepon.F90
cesm.exe           00000000006F4219  cam_comp_mp_cam_r         250  cam_comp.F90
cesm.exe           00000000006B89D5  atm_comp_mct_mp_a         364  atm_comp_mct.F90
cesm.exe           000000000047A5C4  ccsm_comp_mod_mp_        2525  ccsm_comp_mod.F90
cesm.exe           00000000004C2BDD  MAIN__                     90  ccsm_driver.F90
cesm.exe           000000000041475E  Unknown               Unknown  Unknown
libc.so.6          00002AAAAC2F4D5D  Unknown               Unknown  Unknown
cesm.exe           00000000004145E9  Unknown               Unknown  Unknown
forrtl: severe (408): fort: (3): Subscript #1 of the array QV has value -18 which is less than the lower bound of 1

Image              PC                Routine            Line        Source
cesm.exe           00000000066B8E90  Unknown               Unknown  Unknown
cesm.exe           00000000038ADC16  tp_core_mp_xtpv_          379  tp_core.F90

...skipping one line
cesm.exe           000000000389CCC5  tp_core_mp_tp2c_          119  tp_core.F90
cesm.exe           0000000003814ABA  sw_core_mp_d_sw_          893  sw_core.F90
cesm.exe           00000000033CD842  cd_core_                 1110  cd_core.F90
cesm.exe           00000000023C1116  dyn_comp_mp_dyn_r        1822  dyn_comp.F90
cesm.exe           00000000019AADCF  stepon_mp_stepon_         422  stepon.F90
cesm.exe           00000000006F4219  cam_comp_mp_cam_r         250  cam_comp.F90
cesm.exe           00000000006B89D5  atm_comp_mct_mp_a         364  atm_comp_mct.F90
cesm.exe           000000000047A5C4  ccsm_comp_mod_mp_        2525  ccsm_comp_mod.F90
cesm.exe           00000000004C2BDD  MAIN__                     90  ccsm_driver.F90
cesm.exe           000000000041475E  Unknown               Unknown  Unknown
libc.so.6          00002AAAAC2F4D5D  Unknown               Unknown  Unknown
cesm.exe           00000000004145E9  Unknown               Unknown  Unknown
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[9413,1],0]
  Exit code:    152
Going back to the standard ncdata file yields no error in the run.I was wondering whether that could be related to my using intel14 (as I previously posted on a similar problem, with the same "negative subscript" type of issue)?Thanks.Thibaut.

 

[jedwards]

It certainly looks like the same type of issue - can you try with a differenent compiler and see if it works?   

 

[thibaut_lurton@cnrs-orleans_fr]

Thank you Jim for your quick answer.I've tried with intel15, but it yields the same error.Have you got any idea what would be a possible workaround to this issue? I remember we could fix a similar problem on ice_itd.F90 by removing the explicit array syntax (http://bb.cgd.ucar.edu/error-running-compset-fsdwsf-ert).Many thanks.Thibaut.

 

[jedwards]

Looking at tp_core.F90 line 379.   I see that the issue is in the advection code, apparently the trajectory being calculated exceeds the CFL condition.   Try reducing the dynamics time step try: dtime=300, 

 

[mmills]

I believe the easiest way to reduce the dynamical time step is to increase nsplit, rather than reducing dtime. nsplit needs to be a multiple of nspltrac, so you can try doubling nsplit.  

 

[thibaut_lurton@cnrs-orleans_fr]

 Hello,Many thanks for the suggestions.At the end of the day, the issue apparently stemmed from a bad formatting in my customised IC file (or so I guess): after a lot of tweaking with dtime, nsplit, etc. I eventually re-generated my IC file including the two extra species, and the run performed ok.
It was indeed strange that only a difference in the IC file would entail an advection code misbehaviour.Sorry for having put you on a false lead. Anyway, it was good to get a bit acquainted with those advection parameters, which can obviously be of some importance in certain cases.Many thanks.Thibaut.