ERROR: remap transport: bad departure points

[zdliu]

Hi,

I am an ocean modeler using cesm, and now I make a B-coupled simulation whose compset is B1850CN, and grid of ocean/ice is tx0.1v2, atm/land is f02. All setups are default of the model cesm1.0.4, and nothing I have changed. I have successfully run it for about 9 years.

Strangely at model date of 0009-02-24 it suddenly crashed and I have found the error infomation which is about the ice model(ERROR: remap transport: bad departure points). But I am not very familiar with the ice model.

Would someone give me some tips to solve the problem? Thanks.



All of the run logs I have uploaded as the Attachment, and I cut some logs and paste as follows:

the last lines of ccsm.log.140512-155037::
****************************************************************************************************
.................
Warning: Departure points out of bounds in remap
my_task, i, j = 317 17 46
dpx, dpy = 3762.16863479234 -4974.00738819010
HTN(i,j), HTN(i+1,j) = 4476.88132996261 4475.23055030556
HTE(i,j), HTE(i,j+1) = 4898.73547254783 4898.68871367705
istep1, my_task, iblk = 2241 317 3
Global block: 2492
Global i and j: 2968 2397
(shr_sys_abort) ERROR: remap transport: bad departure points
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
..................
====================================================================================================


the last lines of ice.log.140512-155037::
****************************************************************************************************
.................
istep1: 2232 idate: 90224 sec: 21600
aero: 1 faero : 36.2462371301263
1.304543495329568E-003
aero: 1 fsoot : 18.4685036007151
1.681745062953533E-003
aero: 1 faero-fsoot : 17.7777335294113
-3.772015676239654E-004
aero: 1 aerotot : 195941.050381871
21.2423627971295
aero: 1 aerotot change: 17.7777328629745
-3.772015697123265E-004
aero: 1 aeromax agg: 2.151841182019741E-006
1.126117094359773E-009
aero: 2 faero : 1583.38457522712
3.23464545706469
aero: 2 fsoot : 935.720086677989
6.33849299129514
aero: 2 faero-fsoot : 647.664488549130
-3.10384753423044
aero: 2 aerotot : 11729048.8982101
86998.9239151817
aero: 2 aerotot change: 647.664475198835
-3.10384753585095
aero: 2 aeromax agg: 8.669209048754921E-006
6.495814630033117E-007
aero: 3 faero : 372234.168491625
679.457009484847
aero: 3 fsoot : 187227.539204398
1351.11710689450
aero: 3 faero-fsoot : 185006.629287227
-671.660097409649
aero: 3 aerotot : 3360342858.96942
18472176.5975063
aero: 3 aerotot change: 185006.626586914
-671.660098429769
aero: 3 aeromax agg: 1.447804554003243E-003
1.234499446153346E-004
Arctic Antarctic
total ice area (km^2) = 1.89671408638049476E+07 3.06020372930500610E+06
total ice extent(km^2) = 2.03685775415963493E+07 4.14215124914498627E+06
total ice volume (m^3) = 5.36174879912741562E+13 6.87703837837442969E+12
total snw volume (m^3) = 5.03336661380871484E+12 2.10218274491342407E+12
tot kinetic energy (J) = 2.31576843008776500E+15 1.27669585439344109E+14
rms ice speed (m/s) = 0.30186307689732444 0.19099007672940616
average albedo = 0.84574880470801961 0.85467644991666492
max ice volume (m) = 39.51896918749236676 14.35145404678220871
max ice speed (m/s) = 17.13251561085255759 1.56739673132420743
max strength (kN/m) = 25703.36089026039189775 376.58030885300729551
----------------------------
arwt rain h2o kg in dt = 2.01653209232571526E+10 1.05292888817998943E+10
arwt snow h2o kg in dt = 1.93805251525475220E+11 4.84196203894528427E+10
arwt evap h2o kg in dt = -1.93293593760930705E+09 -8.52299550231652379E+08
arwt frzl h2o kg in dt = 2.94771741866568542E+11 3.76958373426313477E+10
arwt frsh h2o kg in dt = -5.13754150498331116E+11 3.91798869393167542E+11
arwt ice mass (kg) = 4.91672364879984000E+16 6.30624419296935200E+15
arwt snw mass (kg) = 1.66101098255687600E+15 6.93720305821430000E+14
arwt tot mass (kg) = 5.08282474705552800E+16 6.99996449879078200E+15
arwt tot mass chng(kg) = 1.02056352886400000E+12 -2.96006422332000000E+11
arwt water flux = 1.02056352887602271E+12 -2.96006422329515137E+11
(=rain+snow+evap+frzl-fresh)
water flux error = 2.36535896404646510E-16 3.54982269078543249E-16
----------------------------
arwt atm heat flux (W) = -6.27107818696205125E+14 -5.69447086940087812E+13
arwt ocn heat flux (W) = -3.25224491767778937E+14 -1.74101721668745375E+14
arwt frzl heat flux(W) = 1.09294811400970984E+14 1.39767788013734219E+13
arwt tot energy (J) = -1.77142130912257942E+22 -2.32966259373387284E+21
arwt net heat (J) = -3.70060324496457472E+17 9.28622107560268480E+16
arwt tot energy chng(J)= -3.70056511015092224E+17 9.28626726393610240E+16
arwt heat error = 2.15278056417696244E-10 1.98261900851365468E-10
----------------------------
arwt salt mass (kg) = 1.96668945951993594E+14 2.52249767718774102E+13
arwt salt mass chng(kg)= 2.57670198357372570E+09 -1.21896945666252542E+09
arwt salt flx in dt(kg)= -2.57670198361655378E+09 1.21896945665137744E+09
arwt salt flx error = -2.17767389920237276E-16 -4.41941969756696491E-16
----------------------------
====================================================================================================

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[dbailey]

This is essentially a CFL violation and you can adjust the dynamics timestep. This is similar to an issue discussed in the documentation:
http://www.cesm.ucar.edu/models/cesm1.0/cice/doc/node35.htmlYou should also check the forcing from the atmosphere and ocean components. Turning on frequent coupler history is the best way to diagnose this. See the CESM documentation on how to do this.
Dave

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[zdliu]

Thanks.I will follow your suggestions to restart my simulation.  But how much will I reduce the dynamics timestep? In the ocean model, "decrease the ocean timestep by roughly 20%", what about the ice dynamics timestep? In my case, the default "xndt_dyn = 3.0".

 

[dbailey]

I missed that you were running the 0.1-degree domain. I would recommend reducing the thermodynamic timestep by changing the coupling frequency (NCPL_ATM/ICE/LND) to 144, i.e. every ten minutes. I believe the default is 96. Then xndt_dyn = 2 will actually be enough. It's still a 5 minute dynamics timestep, but we've found this is better overall. Note that changing the coupling interval requires a hybrid start and you cannot do a continue or branch run.Dave

 

[dbailey]

I missed that you were running the 0.1-degree domain. I would recommend reducing the thermodynamic timestep by changing the coupling frequency (NCPL_ATM/ICE/LND) to 144, i.e. every ten minutes. I believe the default is 96. Then xndt_dyn = 2 will actually be enough. It's still a 5 minute dynamics timestep, but we've found this is better overall. Note that changing the coupling interval requires a hybrid start and you cannot do a continue or branch run.Dave

 

[dbailey]

I missed that you were running the 0.1-degree domain. I would recommend reducing the thermodynamic timestep by changing the coupling frequency (NCPL_ATM/ICE/LND) to 144, i.e. every ten minutes. I believe the default is 96. Then xndt_dyn = 2 will actually be enough. It's still a 5 minute dynamics timestep, but we've found this is better overall. Note that changing the coupling interval requires a hybrid start and you cannot do a continue or branch run.Dave