Thermodynamic convergence error

[sur23beeb]

I am doing a high-resolution ocean-ice simulation forced by atmospheric reanalysis. The ocean model has the submesoscale parameterization turned on.

resolution: TL319_t13
compset: 2000_DATM%JRA_SLND_CICE_POP2_DROF%JRA_SGLC_SWAV
cesmtag: cesm2.1.4-rc.08

Problem: I am trying out different values for one of the parameters in the submesoscale parameterization. For one setting, I have managed to run the model for > 50 years but it is blowing up after 15 years for a different setting. The error message I get is "vertical thermo error".

Steps taken: I consulted the FAQ for this exact error and am in step 1 of the suggested list of things to do, i.e., see if everything going into the ice model is okay. I am attaching a couple of plots, one showing the SST and identifying the exact location (yellow dot) where the error is occurring. The second plot shows timeseries of various quantities at this location over the last 10 years of the simulation, right until the moment the model blows up. The quantities shown are: (i) SHF (surface heat flux); (ii) SST; (iii) UVEL at the topmost level; and (iv) SUBM_ADV_TEND_TEMP at the topmost level, which is the submesoscale contribution to the temperature tendency. The timeseries do not seem to show anything out of the ordinary for the parameter setting which causes the model to blow up.

Because I am not very familiar with the ice model, my question is do I need to check more quantities before progressing to the next steps given in the FAQ.

Thanks for any help.

 

[dbailey]

As this is a high resolution case you might have to decrease the timestep in the ocean and the sea ice. For the ocean it is dt_count. For the sea ice, this is related to the coupling interval (NCPL_ICE). For a JRA case, NCPL_ICE = NCPL_ATM should be 48 ... this might need to increase. Also, one can subcycle the sea ice dynamics with ndtd or xndyn_dt depending on the version.

 

[sur23beeb]

Also want to add that the water column at that yellow dot is very shallow (KMT = 3).

 

[sur23beeb]

As this is a high resolution case you might have to decrease the timestep in the ocean and the sea ice. For the ocean it is dt_count. For the sea ice, this is related to the coupling interval (NCPL_ICE). For a JRA case, NCPL_ICE = NCPL_ATM should be 48 ... this might need to increase. Also, one can subcycle the sea ice dynamics with ndtd or xndyn_dt depending on the version.

Thanks for these suggestions! Do you recommend starting by decreasing just one of the two timesteps first or decreasing both? The reason I am asking is that these runs are expensive and I am trying to economize my awarded computational allocation.

 

[sur23beeb]

The settings right now are OCN_NCPL = 48 and ATM_NCPL = ICE_NCPL = 144. Is the value of 144 still too low?

 

[dbailey]

Sounds like you should try dt_count in POP first since your point is so shallow. What do the temperature and salinities look like in POP? Looks like your velocity in POP stays well behaved. 144 might still be too low for the sea ice coupling. Also, check the CFL numbers in the ocean log file.

 

[sur23beeb]

1. I increased dt_count from 500 to 750 and restarted around 46 days before the blow up. Still blowing up at the same time.
2. The cfl numbers for meridional, vertical and total advection do not seem too big (attached plot).
3. The vertical profiles at the location where I get the error show a slight inversion in rho (is this significant?)

What is a reasonable increment for ATM_NCPL = ICE_NCPL (from 144)?

It is also possible that dt_count=750 is still too low and needs to be increased even further?

 

[dbailey]

Try dt_count=1500. If it still blows up, try writing frequent coupler history and looking at the fields being passed through the coupler.

In env_run.xml look for:
HISTAVG_OPTION = 'ndays'
HISTAVG_N = 1

Dave