Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

error: ice_transport_remap

S

s_l_bradley-1@tudelft_nl

New Member
Hello,I am running CESM2.1.0 - B case for an LGM settings (f09_g17) compset : 1850_CAM60_CLM50%BGC_CICE_POP2_RTM_CISM2%NOEVOLVE_SWAVI am trying to start the model using restarts from (a)  CAM/CLM/CICE: PI run b.e20.B1850.f09_g17.pi_control.all.299(b) POP (LGM CCSM4) /gpfs/fs1/scratch/erik/b40.lgm21ka.1deg.003/rest/1846-01-01-00000The 'B' case runs succesfully when not using the restart files. However we get an error related to the ice_transport_remap [cesm.log.4983760.chadmin1.190411-070243]88173 716: Warning: Departure points out of bounds in remap88174 716: my_task, i, j =         104          10           888175 716: dpx, dpy =   18853.1040333448        4399.0312862886288176 716: HTN(i,j), HTN(i+1,j) =   15418.6525795818        15277.651612705388177 716: HTE(i,j), HTE(i,j+1) =   55727.0484591510        55918.917355676988178 716: istep1, my_task, iblk =           3         104           288179 716: Global block:         62488180 716: Global i and j:          57         37988181 716: remap transport: bad departure points88182 716: ERROR: remap transport: bad departure points88183 674:Image              PC                Routine            Line        Source88184 674:cesm.exe           000000000343DB6D  Unknown               Unknown  Unknown88185 674:cesm.exe           0000000002BAE022  shr_abort_mod_mp_         114  shr_abort_mod.F9088186 674:cesm.exe           0000000001754524  ice_exit_mp_abort          46  ice_exit.F9088187 674:cesm.exe           000000000199A252  ice_transport_rem         560  ice_transport_remap.F9088188 674:cesm.exe           000000000198D6F4  ice_transport_dri         474  ice_transport_driver.F9088189 674:cesm.exe           0000000001977BFB  ice_step_mod_mp_s        1208  ice_step_mod.F9088190 674:cesm.exe           000000000183F212  cice_runmod_mp_ci         205  CICE_RunMod.F9088191 674:cesm.exe           0000000001745CBA  ice_comp_mct_mp_i         563  ice_comp_mct.F9088192 674:cesm.exe           0000000000425864  component_mod_mp_         728  component_mod.F9088193 674:cesm.exe           000000000040AEBE  cime_comp_mod_mp_        2699  cime_comp_mod.F9088194 674:cesm.exe           000000000042550C  MAIN__                    125  cime_driver.F9088195 674:cesm.exe           0000000000408C9E  Unknown               Unknown  Unknown88196 674:libc-2.19.so       00002AAAB08F9B25  __libc_start_main     Unknown  Unknown88197 674:cesm.exe           0000000000408BA9  Unknown               Unknown  Unknown 88198 672:Image              PC                Routine            Line        Source Following looking at the discussion board, I can see that suggestions for this error are: (a) increase ndtd =2(b) ATM_NCPL = 144This led to error with ATM: with the FV subcycling: atm.log.4983830.chadmin1.190411-072126When using the revised FV setting: (C) user_nl_atm:  fv_nsplit   =            4fv_nspltrac =            2fv_nspltvrm =            2
Trying all these suggesions, leads to a new error (see log with * 4983867.chadmin1.190411-073836) component_mod:check_fields NaN found in ATM instance:    1 field Sa_z 1d global88951 207:  index:    2544188952 513: ERROR:88953 513: component_mod:check_fields NaN found in ATM instance:    1 field Sa_z 1d global88954 513:  index:    2546388955 9: ERROR:88956 9: component_mod:check_fields NaN found in ATM instance:    1 field Sa_z 1d global88957 9:  index:    2575388958 8: ERROR:88959 8: component_mod:check_fields NaN found in ATM instance:    1 field Sa_z 1d global88960 8:  index:    2575188961 257: ERROR: 88962 257: component_mod:check_fields NaN found in ATM instance:    1 field Sa_z 1d global
I am then assuming that the steps we followed to resolve our CICE error where not the correct approach
Do you have any suggestions? I have uploaded all the log files

 
D

dbailey

CSEG and Liaisons
Staff member
Is your land/ocean mask the same as the pre-industrial case? It sounds like you have a mismatch here.Dave
 
D

dbailey

CSEG and Liaisons
Staff member
Is your land/ocean mask the same as the pre-industrial case? It sounds like you have a mismatch here.Dave
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi Dave Sorry, I did not add this to the questions. Yes our run has different land-ocean mask. However when I run with all PI - restarts: a)  CAM/CLM/CICE/POP: PI run b.e20.B1850.f09_g17.pi_control.all.299The model runs - for 5 days - with no problems. This gave me confidence that the change in the land/ocean mask between PI restart files and our new LGM set up did not seem to be a problem.So this was why I was a little confused why when changing the POP to be an LGM restart file - issue arose.  Thanks Sarah 
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi Dave Sorry, I did not add this to the questions. Yes our run has different land-ocean mask. However when I run with all PI - restarts: a)  CAM/CLM/CICE/POP: PI run b.e20.B1850.f09_g17.pi_control.all.299The model runs - for 5 days - with no problems. This gave me confidence that the change in the land/ocean mask between PI restart files and our new LGM set up did not seem to be a problem.So this was why I was a little confused why when changing the POP to be an LGM restart file - issue arose.  Thanks Sarah 
 
D

dbailey

CSEG and Liaisons
Staff member
I'm still confused, but it sounds like your POP restart/initial file is not providing data in an area where your run needs it. This would cause the system to go haywire. Can you get hourly or daily history files before the crash? This would be a way to see where the problem is.Dave
 
D

dbailey

CSEG and Liaisons
Staff member
I'm still confused, but it sounds like your POP restart/initial file is not providing data in an area where your run needs it. This would cause the system to go haywire. Can you get hourly or daily history files before the crash? This would be a way to see where the problem is.Dave
 
D

dbailey

CSEG and Liaisons
Staff member
I don't see the attachment. How many timesteps does it run before this error? It sounds like it is just blowing up from the initialization. You can try dramatically increasing dt_count in POP to 500 or something for a year or so.Dave
 
D

dbailey

CSEG and Liaisons
Staff member
I don't see the attachment. How many timesteps does it run before this error? It sounds like it is just blowing up from the initialization. You can try dramatically increasing dt_count in POP to 500 or something for a year or so.Dave
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi DavidFirst - an update. I revised the FV settings for CAM and then added your great suggestions of setting hourly output we now have an error which is related to POP.This is great news - as we can try to work with this: I have attached the new cesm.log file :POP aborting...88979 863: ERROR: k.e. > 10088980 863:
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi DavidFirst - an update. I revised the FV settings for CAM and then added your great suggestions of setting hourly output we now have an error which is related to POP.This is great news - as we can try to work with this: I have attached the new cesm.log file :POP aborting...88979 863: ERROR: k.e. > 10088980 863:
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi David Is the cesm.log not attached - I have attached again it does not run at all - not even one time stepWe tried making it dt_count=96 and we thought that was already large. Okay - I will try 500!  Let see Sarah 
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi David Is the cesm.log not attached - I have attached again it does not run at all - not even one time stepWe tried making it dt_count=96 and we thought that was already large. Okay - I will try 500!  Let see Sarah 
 
D

dbailey

CSEG and Liaisons
Staff member
If it is crashing on the first step, you might have to initialize differently. Sounds like you might not be able to use the velocities from the POP restart. In this case, you can try a T/S initialization. Dave
 
D

dbailey

CSEG and Liaisons
Staff member
If it is crashing on the first step, you might have to initialize differently. Sounds like you might not be able to use the velocities from the POP restart. In this case, you can try a T/S initialization. Dave
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi David I tried again with dt_count = 500 and unfortunately, that did not resolve the ke> 100 error. In the case notes from the older ccsm4 LGM run - it suggests that the POP restart files can cause this error: k.e > 100 To resovle you should set the velocities.surface pressure etc all to zero and this will resolve  it.Unfortunately that still did not resolve this issue. I am unsure what other method of initialising the T/S: we are using the restart from the CCSM4 LGM run so that we can reduce the amount of spin up time for the ocean for our B run, which we expect will have to be ~ 300 - 500 years.Perhaps we should be changing something else when we read in the T/S from the restart files?  Do you have any other suggestions? Thanks 
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi David I tried again with dt_count = 500 and unfortunately, that did not resolve the ke> 100 error. In the case notes from the older ccsm4 LGM run - it suggests that the POP restart files can cause this error: k.e > 100 To resovle you should set the velocities.surface pressure etc all to zero and this will resolve  it.Unfortunately that still did not resolve this issue. I am unsure what other method of initialising the T/S: we are using the restart from the CCSM4 LGM run so that we can reduce the amount of spin up time for the ocean for our B run, which we expect will have to be ~ 300 - 500 years.Perhaps we should be changing something else when we read in the T/S from the restart files?  Do you have any other suggestions? Thanks 
 
D

dbailey

CSEG and Liaisons
Staff member
Is this a hybrid run? That is, is POP doing a "startup_spunup"? This might be the issue. I'm really not familiar enough with POP to help here.Dave
 
D

dbailey

CSEG and Liaisons
Staff member
Is this a hybrid run? That is, is POP doing a "startup_spunup"? This might be the issue. I'm really not familiar enough with POP to help here.Dave
 
S

s_l_bradley-1@tudelft_nl

New Member
Hi dave Yes it is: thanks for all your help with this: it did resolve the issues with the CICE which was a big hurdle. I have contacted, with Erik Kluzek assiatance someone from the PWG group. As you say - they are more familar with the CCSM4 Pop issues Sarah 
 
You must log in or register to reply here.
Top