CICE in CESM with custom grid

[paulhall]

CESM version: cesm2_3_alpha17a
Comp Set: 2000_DATM%SIMPLE_SLND_CICE_MOM6_SROF_SGLC_SWAV

I am trying to run a case with MOM6+CICE on a custom grid for a model domain located 214°E-216°E and 75.25°N-75.75°N (~50km x 50km domain, 500m grid spacing). However, I get the following error message in the cesm log file (excerpted):

dec1767.hsn.de.hpc.ucar.edu 25: ERROR: CICE n, mesh_lon , lon, diff_lon = 1 215.9300000000000 -64.1917550934441 -0.79878D+02
dec1767.hsn.de.hpc.ucar.edu 25:
dec1767.hsn.de.hpc.ucar.edu 25: (abort_ice)ABORTED:
dec1767.hsn.de.hpc.ucar.edu 25: (abort_ice) called from ice_mesh_mod.F90
dec1767.hsn.de.hpc.ucar.edu 25: (abort_ice) line number 647
dec1767.hsn.de.hpc.ucar.edu 25: (abort_ice) error = ice_mesh_check:
dec1767.hsn.de.hpc.ucar.edu 25: ERROR: (abort_ice) ice_mesh_check:

The CICE log file includes the following warnings and errors:

(ice_comp_nuopc):(InitializeAdvertise)WARNING: tfrz_option from driver linear_s
alt is overwriting tfrz_option from cice namelist mushy
(ice_comp_nuopc):(InitializeAdvertise)WARNING: atmiter_ from driver 0.10000D-
01 is overwritting atmiter_conv from cice namelist 0.00000D+00
mesh file for cice domain is
/glade/u/home/paulhall/CESM/SASIP/grids/n100/ESMF_mesh_sasip_n100.nc
mask file for cice domain is
/glade/u/home/paulhall/CESM/SASIP/grids/n100/ESMF_mesh_sasip_n100.nc
(ice_read_xyt) read_global 98 1 503316480.000000
503316480.000000 5033164800000.00
(ice_read_global) read_global 99 1 -1.777781880591592E+306
8.515146846555257E+294 -1.777781880583230E+308
(ice_read_global) read_global 99 2 -2.893653090372116E+301
1.180408341142792E+308 Infinity
(ice_read_global) read_global 99 7 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000
(ice_read_global) read_global 99 3 -2.708864630005210E+299
1.221661105937845E+300 9.324806461428189E+301
(ice_read_global) read_global 99 4 -6.106018129814693E-257
2.422516750935218E+270 7.752053602992255E+273

min/max ULON: -1.657941094533100E+303 Infinity
min/max ULAT: -1.018593986527287E+308 4.878819762417483E+296
min/max TLON: -180.000000000000 180.000000000000
min/max TLAT: -89.7889613300025 89.9514162758023

min/max NLON: -180.000000000000 180.000000000000
min/max NLAT: -89.9467305341177 89.8260581488775
min/max ELON: -180.000000000000 180.000000000000
min/max ELAT: -80.8717325355776 87.8815721249592

(abort_ice)ABORTED:
(abort_ice) called from ice_mesh_mod.F90
(abort_ice) line number 647
(abort_ice) error = ice_mesh_check:
ERROR: (abort_ice) ice_mesh_check:

There is clearly an issue with my CICE grid, but I'm not sure what it is. The ULON and ULAT values in the CICE log look problematic, but the netcdf grid files I am using to create the CICE binary grid files look fine to me (and cases without CICE run fine). I was able to create CICE grids and run on a similar model domain in an earlier version of cesm (cesm2_3_beta14). Does anyone have any insight into what I'm doing wrong, or suggestions for documentation on how to use custom CICE grids within CESM? Thanks!

 

[dbailey]

Sorry I missed this one Paul. A couple of questions. What is the long grid name you are using for the case? This check appears to be a disagreement between the mesh file and the binary grid file. Note that the center lats and longs (TLAT / TLON) are computed internally in the CICE model. Actually, now I see that you have undefined values for lats and lons. You should be able to read the netCDF grid directly in CICE (grid_format = 'nc'). I wonder if the binary format is wrong (little endian versus big endian).

 

[paulhall]

@dbailey Thanks for the response! I was running on derecho, if you want to take a look at the grids and meshes I was using. They can be found in /glade/u/home/paulhall/CESM/SASIP/grids/n100. I created the binary cice grid files (cice_hgrid_sasip_n100.ieeer8 and cice_topo_sasip_n100.ieeei4) from a netCDF grid file (cice_sasip_n100.nc) using an ncl script that Gustavo provided to me a while ago (gen_cice_gridfiles.ncl, a copy can be found in the same directory, though it is tailored to be run on our local cluster, where I created the grid files, not on derecho, so the paths in the script aren't meaningful).

Are you suggesting that I can skip the binary files and just use the netCDF grid file directly instead? Other than adding grid_format='nc' to user_nl_cice, do I need to make any other changes? Presumably I would need to include the path to the netCDF file? Would this be the grid_file in user_nl_cice? Do I need to do anything about kmt_file?

Thanks!

 

[dbailey]

Right you should set:

grid_format = 'nc'
kmt_file = path_to_kmt_file
grid_file = path_to_grid_file

That should do it.

 

[paulhall]

@dbailey I think I must be missing something obvious. When I try the following combination:

grid_format = 'nc'
kmt_file = path_to_cice_topo_sasip_n100.ieeei4
grid_file= path_to_cice_sasip_n100.nc

I get an error in the CESM log file (similar in the CICE log file) suggesting it can't open the kmt file:

dec1771.hsn.de.hpc.ucar.edu 0: ERROR:
dec1771.hsn.de.hpc.ucar.edu 0: (abort_ice)(ice_open_nc) ERROR: Cannot open /glade/u/home/paulhall/CESM/SASIP/g
dec1771.hsn.de.hpc.ucar.edu 0: rids/n100/cice_topo_sasip_n100.ieeei4

Do I need the kmt file to be in netCDF format too? If so, what should that file contain? I tried using the same netCDF file (which includes depths), but when I try:

grid_format = 'nc'
kmt_file = path_to_cice_sasip_n100.nc
grid_file= path_to_cice_sasip_n100.nc

I get the following error:

dec1784.hsn.de.hpc.ucar.edu 0: ERROR: (abort_ice)(ice_read_global_nc) ERROR: Cannot find variable kmt

Any idea where I'm going wrong? Thanks!

 

[dbailey]

The kmt_file has the array of KMT values or the level of the bottom. This needs to be in the file as well. CICE uses this for the internal mask.

 

[paulhall]

So is it a matter of including the kmt values in the netCDF grid file and then using that file for both the kmt_file and the grid_file, or should I create a separate, netDCF-formatted kmt file?

 

[paulhall]

Hi @dbailey I followed your suggestion and attempted to use a netCDF-formatted grid file for CICE. I created a single grid file that should contain all of the information necessary for both the grid and the kmt file (/glade/u/home/paulhall/CESM/SASIP/grids/n100/cice_sasip_n100.nc, if it is helpful to take a look). However, I am still getting essentially the same error message when I try to run, e.g. (from CESM logfile):

dec0404.hsn.de.hpc.ucar.edu 27: ERROR: CICE n, mesh_lon , lon, diff_lon = 21 214.0100000000000 -145.0100000000000 -0.98000D+00
dec0404.hsn.de.hpc.ucar.edu 27:
dec0404.hsn.de.hpc.ucar.edu 27: (abort_ice)ABORTED:
dec0404.hsn.de.hpc.ucar.edu 27: (abort_ice) called from ice_mesh_mod.F90
dec0404.hsn.de.hpc.ucar.edu 27: (abort_ice) line number 647
dec0404.hsn.de.hpc.ucar.edu 27: (abort_ice) error = ice_mesh_check:
dec0404.hsn.de.hpc.ucar.edu 27: ERROR: (abort_ice) ice_mesh_check:

In the CICE log file, I now see the following:

(ice_comp_nuopc):(InitializeAdvertise)WARNING: tfrz_option from driver linear_s
alt is overwriting tfrz_option from cice namelist mushy
(ice_comp_nuopc):(InitializeAdvertise)WARNING: atmiter_ from driver 0.10000D-
01 is overwritting atmiter_conv from cice namelist 0.00000D+00
mesh file for cice domain is
/glade/u/home/paulhall/CESM/SASIP/grids/n100/ESMF_mesh_sasip_n100.nc
mask file for cice domain is
/glade/u/home/paulhall/CESM/SASIP/grids/n100/ESMF_mesh_sasip_n100.nc
(ice_read_nc_xy) fid= 131072, lnrec = 1, varname = kmt
(ice_read_nc_xy) min, max, sum = 30.0000000000000
30.0000000000000 300000.000000000 kmt
(ice_read_global_nc) min, max, sum = 1.31336026212573
1.32199964192311 13176.7995202444 ulat
(ice_read_global_nc) min, max, sum = 3.73500459926787
3.76956211845735 37522.8335886261 ulon
(ice_read_global_nc) min, max, sum = 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 angle
(popgrid_nc) CICE will calculate angleT, TLON and TLAT
(ice_read_global_nc) min, max, sum = 54802.1413192928
56664.3124334943 557335700.328688 htn
(ice_read_global_nc) min, max, sum = 55658.5498460485
55658.5498462067 556585498.461050 hte

min/max ULON: 214.000000000000 215.980000000000
min/max ULAT: 75.2500000000000 75.7400000000000
min/max TLON: -145.990000000000 -144.030000000000
min/max TLAT: 75.2475002148935 75.7445415176276

min/max NLON: -145.990000000000 -144.030000000000
min/max NLAT: 75.2500002148603 75.7420418455440
min/max ELON: -146.000000000000 -144.020000000000
min/max ELAT: 75.2475000000000 75.7425000000000

(abort_ice)ABORTED:
(abort_ice) called from ice_mesh_mod.F90
(abort_ice) line number 647
(abort_ice) error = ice_mesh_check:
ERROR: (abort_ice) ice_mesh_check:

So the min/max values for ULON/ULAT, TLON/TLAT, NLON/NLAT, and ELON/ELAT are at least looking more sensible now. I notice that ULON appears to be given in positive °E while the other LON are being given in negative °E. Any idea what this would be, and whether it could be the source of the issue? I have been defining my grids (e.g., MOM6) using positive °E for longitudes. Does this convention not work for CICE?

 

[paulhall]

@dbailey I notice that there was a recent (June 2023) change in how CICE calculates differences between read-in and internally generated longitudes in ice_mesh_mod.F90. I'm wondering if something in this change might be at play? Possibly the order of abs and mod in calculating diff_lon? I'm wondering if they should be reversed?

 

[dbailey]

Good to see your lat / lon values looking more reasonable! I had changed the check recently to account for the crazy longitudes that MOM6 is using. Something like -270 to 90. They do this to move the longitude wrap around into Europe / Africa. Hopefully this is not a problem here. We are generally trying to make sure the internally computed lats and lons are consistent with the mesh values. Was there more information about the problem causing the mesh check abort?

 

[paulhall]

Hi @dbailey No, I'm afraid that's all the relevant information in the log files. If you want to take a look for yourself, please do so. Here are the relevant files for the run with the netCDF-formatted CICE grid:

CESM log: /glade/u/home/paulhall/scratch/si.floe.T62_sasip_n100.resting.011/run/cesm.log.3856458.desched1.240320-103059
CICE log: /glade/u/home/paulhall/scratch/si.floe.T62_sasip_n100.resting.011/run/ice.log.3841433.desched1.240318-203216
CICE grid file: /glade/u/home/paulhall/CESM/SASIP/grids/n100/cice_sasip_n100.nc

I would be interested in getting your take on my CICE grid file. The grid file looks ok to me, and CICE seems to be getting the min/max coordinates right now, but I may be overlooking something.

Is there a preferred convention for longitudes for CICE in CESM? For example: -180 to 180 vs 0 to 360 vs ? I have been using the 0 to 360 approach to define the grid for MOM6, and just stuck with that for the CICE grid. Does CICE prefer -180 to 180?

If the CICE grid file is ok, and the longitude convention isn't an issue, I'm not sure what the problem could be, other than something internal to the CESM/CICE code. Do you have any ideas?

 

[dbailey]

There is more information here:

dec0404.hsn.de.hpc.ucar.edu 27: ERROR: CICE n, mesh_lon , lon, diff_lon = 21 214.0100000000000 -145.01000000
00000 -0.98000D+00

So, in this case 360-145.01 = 214.99, So, this longitude difference is almost a degree. So, your longitudes are off in either the mesh file or the grid file.

 

[paulhall]

Hi @dbailey Right, I noticed that. But I'm not sure where it would be coming from. Is this a difference between the CICE grid file (i.e., the netCDF formatted grid file) and the ESMF mesh, or between the CICE grid file and the coordinates calculated internally by CICE? The ESMF mesh file was generated from the same MOM6 supergrid file that I used to generate the CICE grid file.

 

[dbailey]

The sea ice model, CICE is on the B-grid, so the ULAT and ULON in the grid file are at the corners of the cells. The TLAT, TLON are at the center of the grid cells and are computed internally in the CICE code. The mesh file should have the TLAT, TLON and these need to agree with the internally computed values within 0.1 degree or so. So, perhaps you have created the mesh file with the corner lat / lons?