Vertical thermo error in CICE4 running on iCESM1.2

[AKAleksinski]

Hello all,

I am attempting to run an f19 g16 B1850C5CN case using isotope-enabled CESM1.2. After several hundred years of runtime without any significant errors, the model throws this error and ends the run:

dec1074.hsn.de.hpc.ucar.edu 572: Thermo iteration does not converge,istep1, my_task, i, j: 24432
444 2 9
Ice thickness: 0.846030639199860
Snow thickness: 6.370398117613274E-002
dTsf, Tsf_errmax: 8.565832206711910E-012 5.000000000000000E-004
Tsf: 0.000000000000000E+000
fsurf: 3.21764668279478
fcondtop, fcondbot, fswint 3.21764668283471 -1.49006126827805
3.91452417225266
fswsfc, fswthrun 5.64579140084606 2.12067015858148
Flux conservation error = 8.320100164382893E-011
Internal snow absorption:
1.07596141887106
Internal ice absorption:
1.92036111564226 0.428638864597167 0.290315194277395
0.199247578864767
Initial snow temperatures:
-0.351791965936265
Initial ice temperatures:
-0.990754585344572 -1.63607940924510 -1.89381869215702
-1.88787766110751
Final snow temperatures:
-0.341628172857914
Final ice temperatures:
-0.990618701584815 -1.63404292789490 -1.89244174813344
-1.88704214806627
istep1, my_task, iblk = 24432 444 2
Global block: 1706
Global i and j: 201 344
Lat, Lon: 78.8062001807415 59.8503037618985
(shr_sys_abort) ERROR: ice: Vertical thermo error
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping

I have already consulted the FAQ post. Nothing in the history or prior restarts at the listed latitude/longitude looks particularly odd. Increasing nitermax to 200 in a ice_therm_vertical.F90 sourcemod (in both places where it is listed) doesn't appear to affect the issue. And, running a new hybrid version of the case where ATM_NCPL is doubled results in a different set of errors:

dec1859.hsn.de.hpc.ucar.edu 4: error - BalanceCheck: Tracer water balance
snl= -5
water tracer index and time step= 1 5516
-4 snoliq= 1.941693712963299E-008 1.941693716041791E-008
-4 snoice= 2.99142166023053 2.80602817286978
-3 snoliq= 1.737152323057217E-009 1.737152325811415E-009
-3 snoice= 7.40713491169632 6.57434903204052
-2 snoliq= 6.013009652125410E-011 1.410327690799634E-010
-2 snoice= 18.8845287342261 16.0619903259735
-1 snoliq= 8.532056266521636E-013 3.068130358220789E-012
-1 snoice= 49.4709260809410 44.6700551993047
0 snoliq= 4.757380724690075E-015 1.710755739423489E-014
0 snoice= 313.517614165778 306.144805591445
totliq= 43247.5133422503 43247.5133422503
totice= 6045.68501466995 6045.68501466995
---------------------------------------------------------
errh2o= 2.717437819904633E-013 -15.9983834421336
endwb= 6668.49042127817 6662.00259755922
begwb= 6668.17772028401 6677.68723894404
forc_rain= 7.225298800029393E-012 7.225298811523174E-012
forc_snow= 9.273426228064559E-005 9.273426228064560E-005
qflx_irrig= 0.000000000000000E+000 0.000000000000000E+000
qflx_floodc= 0.000000000000000E+000 0.000000000000000E+000
qflx_evap_tot= 4.461101321152192E-006 4.171917109396985E-006
qflx_surf= 0.000000000000000E+000 0.000000000000000E+000
qflx_qrgwl= 0.000000000000000E+000 0.000000000000000E+000
qflx_drain= 1.411780918967786E-006 1.411780918967786E-006
qflx_snwcp_ice= 0.000000000000000E+000 0.000000000000000E+000
ENDWB 6668.49042127817 6662.00259755922
BEGWB 6668.17772028401 6677.68723894404
QFLX_EVAP_TOT 1.605996475614789E-002 1.501890159382915E-002
QFLX_SURF 0.000000000000000E+000 0.000000000dec1859.hsn.de.hpc.ucar.edu 4: MPICH ERROR [Rank 4] [job id 753bff16-5892-4979-9378-389e2ec8f042] [Wed Mar 12 18:35:18 2025] [dec1859] - Abort(1) (rank 4 in comm 400): application called MPI_Abort(comm=0xC4000005, 1) - process 4

column cbalance error = 2.746582031250000E-004 181
begcb = 25815.3412789856
endcb = -54812789444903.5
delta store = -54812789470718.9
input mass = -55992685126557.2
output mass = -1179895655838.31
net flux = -54812789470718.9
nee = 54812789470718.9
gpp = -55992685126557.2
er = -1179895655838.31
col_fire_closs = 0.000000000000000E+000
col_hrv_xsmrpool_to_atm = 0.000000000000000E+000
dwt_closs = 0.000000000000000E+000
product_closs = 0.000000000000000E+000
ENDRUN: called without a message string

This set of errors appears to be linked to iCESM's isotope tracing capability. Does anyone have any other thoughts on how I could solve this issue? I would appreciate any help!

 

[dbailey]

What sort of climate is this? Really warm? Is there still ice left? Maybe try a pertlim in the atmosphere. I haven't looked at these conservation issues in CICE for awhile. What is ICE_NCPL? ATM_NCPL?

 

[AKAleksinski]

This is an Eocene paleoclimate case that experiments with reducing atmospheric pCO2 -- so, quite warm but cooling down. There is ice forming in the Arctic Ocean, which is where CICE appears to have an issue. ATM_NCPL and ICE_NCPL are both set to 48. My control case with high pCO2 did not have any CICE errors.

A new hybrid with a change to pertlim sounds promising. I tried to manually perturb the atmosphere by modifying the horizontal diffusivity variable del2coef, but this ended up not doing anything. I will report back any new developments.