Main menu

Navigation

ERROR: remap transport: bad departure points

9 posts / 0 new
Last post
zdliu@...
ERROR: remap transport: bad departure points

Hi,

I am an ocean modeler using cesm, and now I make a B-coupled simulation whose compset is B1850CN, and grid of ocean/ice is tx0.1v2, atm/land is f02. All setups are default of the model cesm1.0.4, and nothing I have changed. I have successfully run it for about 9 years.

Strangely at model date of 0009-02-24 it suddenly crashed and I have found the error infomation which is about the ice model(ERROR: remap transport: bad departure points). But I am not very familiar with the ice model.

Would someone give me some tips to solve the problem? Thanks.

All of the run logs I have uploaded as the Attachment, and I cut some logs and paste as follows:

the last lines of ccsm.log.140512-155037::
****************************************************************************************************
.................
Warning: Departure points out of bounds in remap
my_task, i, j = 317 17 46
dpx, dpy = 3762.16863479234 -4974.00738819010
HTN(i,j), HTN(i+1,j) = 4476.88132996261 4475.23055030556
HTE(i,j), HTE(i,j+1) = 4898.73547254783 4898.68871367705
istep1, my_task, iblk = 2241 317 3
Global block: 2492
Global i and j: 2968 2397
(shr_sys_abort) ERROR: remap transport: bad departure points
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
..................
====================================================================================================

the last lines of ice.log.140512-155037::
****************************************************************************************************
.................
istep1: 2232 idate: 90224 sec: 21600
aero: 1 faero : 36.2462371301263
1.304543495329568E-003
aero: 1 fsoot : 18.4685036007151
1.681745062953533E-003
aero: 1 faero-fsoot : 17.7777335294113
-3.772015676239654E-004
aero: 1 aerotot : 195941.050381871
21.2423627971295
aero: 1 aerotot change: 17.7777328629745
-3.772015697123265E-004
aero: 1 aeromax agg: 2.151841182019741E-006
1.126117094359773E-009
aero: 2 faero : 1583.38457522712
3.23464545706469
aero: 2 fsoot : 935.720086677989
6.33849299129514
aero: 2 faero-fsoot : 647.664488549130
-3.10384753423044
aero: 2 aerotot : 11729048.8982101
86998.9239151817
aero: 2 aerotot change: 647.664475198835
-3.10384753585095
aero: 2 aeromax agg: 8.669209048754921E-006
6.495814630033117E-007
aero: 3 faero : 372234.168491625
679.457009484847
aero: 3 fsoot : 187227.539204398
1351.11710689450
aero: 3 faero-fsoot : 185006.629287227
-671.660097409649
aero: 3 aerotot : 3360342858.96942
18472176.5975063
aero: 3 aerotot change: 185006.626586914
-671.660098429769
aero: 3 aeromax agg: 1.447804554003243E-003
1.234499446153346E-004
Arctic Antarctic
total ice area (km^2) = 1.89671408638049476E+07 3.06020372930500610E+06
total ice extent(km^2) = 2.03685775415963493E+07 4.14215124914498627E+06
total ice volume (m^3) = 5.36174879912741562E+13 6.87703837837442969E+12
total snw volume (m^3) = 5.03336661380871484E+12 2.10218274491342407E+12
tot kinetic energy (J) = 2.31576843008776500E+15 1.27669585439344109E+14
rms ice speed (m/s) = 0.30186307689732444 0.19099007672940616
average albedo = 0.84574880470801961 0.85467644991666492
max ice volume (m) = 39.51896918749236676 14.35145404678220871
max ice speed (m/s) = 17.13251561085255759 1.56739673132420743
max strength (kN/m) = 25703.36089026039189775 376.58030885300729551
----------------------------
arwt rain h2o kg in dt = 2.01653209232571526E+10 1.05292888817998943E+10
arwt snow h2o kg in dt = 1.93805251525475220E+11 4.84196203894528427E+10
arwt evap h2o kg in dt = -1.93293593760930705E+09 -8.52299550231652379E+08
arwt frzl h2o kg in dt = 2.94771741866568542E+11 3.76958373426313477E+10
arwt frsh h2o kg in dt = -5.13754150498331116E+11 3.91798869393167542E+11
arwt ice mass (kg) = 4.91672364879984000E+16 6.30624419296935200E+15
arwt snw mass (kg) = 1.66101098255687600E+15 6.93720305821430000E+14
arwt tot mass (kg) = 5.08282474705552800E+16 6.99996449879078200E+15
arwt tot mass chng(kg) = 1.02056352886400000E+12 -2.96006422332000000E+11
arwt water flux = 1.02056352887602271E+12 -2.96006422329515137E+11
(=rain+snow+evap+frzl-fresh)
water flux error = 2.36535896404646510E-16 3.54982269078543249E-16
----------------------------
arwt atm heat flux (W) = -6.27107818696205125E+14 -5.69447086940087812E+13
arwt ocn heat flux (W) = -3.25224491767778937E+14 -1.74101721668745375E+14
arwt frzl heat flux(W) = 1.09294811400970984E+14 1.39767788013734219E+13
arwt tot energy (J) = -1.77142130912257942E+22 -2.32966259373387284E+21
arwt net heat (J) = -3.70060324496457472E+17 9.28622107560268480E+16
arwt tot energy chng(J)= -3.70056511015092224E+17 9.28626726393610240E+16
arwt heat error = 2.15278056417696244E-10 1.98261900851365468E-10
----------------------------
arwt salt mass (kg) = 1.96668945951993594E+14 2.52249767718774102E+13
arwt salt mass chng(kg)= 2.57670198357372570E+09 -1.21896945666252542E+09
arwt salt flx in dt(kg)= -2.57670198361655378E+09 1.21896945665137744E+09
arwt salt flx error = -2.17767389920237276E-16 -4.41941969756696491E-16
----------------------------
====================================================================================================

Attachment: 

Zedong LIU
OUC & QNLM
Email:zdliu@ouc.edu.cn

dbailey

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:


http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.html

You should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.


Dave

zdliu@...

Thanks.

I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

Zedong LIU
OUC & QNLM
Email:zdliu@ouc.edu.cn

dbailey

I missed that you were running the 0.1-degree domain. I would recommend reducing the thermodynamic timestep by changing the coupling frequency (NCPL_ATM/ICE/LND) to 144, i.e. every ten minutes. I believe the default is 96. Then xndt_dyn = 2 will actually be enough. It's still a 5 minute dynamics timestep, but we've found this is better overall. Note that changing the coupling interval requires a hybrid start and you cannot do a continue or branch run.

Dave

zdliu@...

David, Thank you very much.


I follow your two suggestions and re-run my case.
Firstly, I just adjust the dynamics timestep(xndt_dyn), from default 3.0 to 6.0, and I run for one month, it has integrated past this problem. After that,I set the value back to 3.0. Is it ok to do so? one month is enough? Is it possible that I will continue to encounter this situation again?

Secondly,following your later suggestion,I reduce the thermodynamic timestep by changing the coupling frequency to 144 from the default 96, and also xndt_dyn from default 3.0 to 2(to keep the dynamics timestep unchanged).But now it has an error(as follows) about FV dycore timestep, and I changed some values(NSPLIT and NSPLTRAC) as suggested. I run again and now it looks ok(I just started to calculate, and now it is running, not finished). But whether is there any further problem?

Thanks.


the last lines of atm.log.140514-151753
=========================================
 FV subcycling - nv, n2, nsplit, dt =            2           3           2       50.0000000000000
 ERROR:  Because of loop nesting, FV dycore can't use the specified namelist settings for subcycling
   The original namelist settings were:
   NSPLIT   =           11
   NSPLTRAC =            5
   NSPLTVRM =            2

   NSPLIT needs to be a multiple of NSPLTRAC
     which in turn needs to be a multiple of NSPLTVRM.
   Suggested settings would be:
   NSPLIT   =           12
   NSPLTRAC =            6
   NSPLTVRM =            2

=============================================

 

zedong



Zedong LIU
OUC & QNLM
Email:zdliu@ouc.edu.cn

dbailey

If the first run worked with xndt_dyn=6, then don't bother with the second idea. As I said, you can't change the coupling intervals in a continue run.


Dave

huili7@...

Hi Zedong,

 

I encountered the same problem and solved it by increasing xndt_dyn to 6. Was one month enough for you to integrate past that instability point?

 

Thanks.

zdliu@...

Hi! Hui

At last I runned for 3 months. I ever runned one month for test, it is ok "seemingly".

For insurance purposes, I use 3-months.

Zedong LIU
OUC & QNLM
Email:zdliu@ouc.edu.cn

huili7@...

Hi Zedong,

Thanks for the suggestions!

 

Hui

Log in or register to post comments

Who's new

  • m.kliphuis@...
  • ddc3061993@...
  • hui.ding@...
  • zhouc@...
  • arianna.valmass...