Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Thermodynamic convergence error

sur23beeb

ST
Member
I am doing a high-resolution ocean-ice simulation forced by atmospheric reanalysis. The ocean model has the submesoscale parameterization turned on.

resolution: TL319_t13
compset: 2000_DATM%JRA_SLND_CICE_POP2_DROF%JRA_SGLC_SWAV
cesmtag: cesm2.1.4-rc.08

Problem: I am trying out different values for one of the parameters in the submesoscale parameterization. For one setting, I have managed to run the model for > 50 years but it is blowing up after 15 years for a different setting. The error message I get is "vertical thermo error".

Steps taken: I consulted the FAQ for this exact error and am in step 1 of the suggested list of things to do, i.e., see if everything going into the ice model is okay. I am attaching a couple of plots, one showing the SST and identifying the exact location (yellow dot) where the error is occurring. The second plot shows timeseries of various quantities at this location over the last 10 years of the simulation, right until the moment the model blows up. The quantities shown are: (i) SHF (surface heat flux); (ii) SST; (iii) UVEL at the topmost level; and (iv) SUBM_ADV_TEND_TEMP at the topmost level, which is the submesoscale contribution to the temperature tendency. The timeseries do not seem to show anything out of the ordinary for the parameter setting which causes the model to blow up.

Because I am not very familiar with the ice model, my question is do I need to check more quantities before progressing to the next steps given in the FAQ.

Thanks for any help.
 

Attachments

  • SST_Lf200m_0259_12_blowup.png
    SST_Lf200m_0259_12_blowup.png
    292.5 KB · Views: 10
  • tseries_compare_Lf5kmvs200m_j1872i1280.png
    tseries_compare_Lf5kmvs200m_j1872i1280.png
    520.7 KB · Views: 10

dbailey

CSEG and Liaisons
Staff member
As this is a high resolution case you might have to decrease the timestep in the ocean and the sea ice. For the ocean it is dt_count. For the sea ice, this is related to the coupling interval (NCPL_ICE). For a JRA case, NCPL_ICE = NCPL_ATM should be 48 ... this might need to increase. Also, one can subcycle the sea ice dynamics with ndtd or xndyn_dt depending on the version.
 

sur23beeb

ST
Member
As this is a high resolution case you might have to decrease the timestep in the ocean and the sea ice. For the ocean it is dt_count. For the sea ice, this is related to the coupling interval (NCPL_ICE). For a JRA case, NCPL_ICE = NCPL_ATM should be 48 ... this might need to increase. Also, one can subcycle the sea ice dynamics with ndtd or xndyn_dt depending on the version.
Thanks for these suggestions! Do you recommend starting by decreasing just one of the two timesteps first or decreasing both? The reason I am asking is that these runs are expensive and I am trying to economize my awarded computational allocation.
 

dbailey

CSEG and Liaisons
Staff member
Sounds like you should try dt_count in POP first since your point is so shallow. What do the temperature and salinities look like in POP? Looks like your velocity in POP stays well behaved. 144 might still be too low for the sea ice coupling. Also, check the CFL numbers in the ocean log file.
 

sur23beeb

ST
Member
1. I increased dt_count from 500 to 750 and restarted around 46 days before the blow up. Still blowing up at the same time.
2. The cfl numbers for meridional, vertical and total advection do not seem too big (attached plot).
3. The vertical profiles at the location where I get the error show a slight inversion in rho (is this significant?)

What is a reasonable increment for ATM_NCPL = ICE_NCPL (from 144)?

It is also possible that dt_count=750 is still too low and needs to be increased even further?
 

Attachments

  • cfl.png
    cfl.png
    12.4 KB · Views: 4
  • TSrho_profiles_blowup.png
    TSrho_profiles_blowup.png
    127.7 KB · Views: 4

dbailey

CSEG and Liaisons
Staff member
Try dt_count=1500. If it still blows up, try writing frequent coupler history and looking at the fields being passed through the coupler.

In env_run.xml look for:
HISTAVG_OPTION = 'ndays'
HISTAVG_N = 1

Dave
 
Top