Deep time paleoclimate: (seq_domain_check_grid) ERROR: incompatible domain grid coordinates

[minminfu]

Hi, I am trying to build some CESM 1.2.2.1 experiments with dramatically different continental configurations following the instructions available at PaleoResources for CESM1.2.

After producing my KMT file, I used the convertPOPT utility from /paleoToolkit/cesm1/cpl_mapping/scrip1.4/grids/convertPOPT.f to create a SCRIPgrid. I used ./gen_cesm_maps.sh to create mapping files, and used ./gen_domain.sh on my mapping file to produce land and ocean domains. The only place I deviate is I had to use convertPOPT (with hard coded file names) instead of myconvertPOPT, because myconvertPOPT was not giving valid output data (I am unsure if this could be a source of error, but the output from convertPOPT seemed reasonable). When I try to launch a simulation with the new domain and mapping files, I persistently get the error:

(seq_domain_check_grid) ERROR: incompatible domain grid coordinates

This is cryptic, so I added some print statements in seq_domain_mct.F90. It seems the error occurs during the ice/ocean grid consistency check, in particular during comparison of the latitude attribute. Furthermore, the data1 and data2 are not even close.

e.g,
10: MJFU n= 31 data1= -68.7069428207174 data2=
10: -85.0038255423153 diff= 16.2968827215979 eps=
10: 1.000000000000000E-002
10: MJFU ATTR IS lat
10: MJFU n= 32 data1= -68.7069428207174 data2=
10: -85.0038255423153 diff= 16.2968827215979 eps=
10: 1.000000000000000E-002

I have set the CICE and DOCN namelists such that they use the exact same domain file, so I am not sure how they can be inconsistent. user_nl_cice also points to the relevant grid and kmt files used to generate the domain file. I was wondering if you have any idea how to resolve this issue or where the problem may originate from. I am at wit’s end, so your help is extremely appreciated! Thank you for your time, it is very much appreciated!

kmt and domain files can be found at
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/domain.lnd.48x96_gx3vSB.nc
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/domain.ocn.gx3vSB.nc
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da

My procedure for using paleotoolkit is documented here
/glade/u/home/mjfu/zwork/Snowball/generate_pop_ensemble.sh
/glade/u/home/mjfu/zwork/Snowball/generate_cpl_ensemble.sh

CESM Log file here:
/glade/scratch/mjfu/e.e12.E1850CN.T31_g37.1367_p-1.0_001_1000ppm_nooht_v9/run/cesm.log.211106-090511

Other namelist changes:

# KEEP CN MODEL OFF
./xmlchange -file env_build.xml -id CLM_CONFIG_OPTS -val "-phys clm4_0"

./xmlchange -file env_run.xml -id ATM_DOMAIN_FILE -val 'domain.lnd.48x96_gx3vSB.nc'
./xmlchange -file env_run.xml -id ATM_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id LND_DOMAIN_FILE -val 'domain.lnd.48x96_gx3vSB.nc'
./xmlchange -file env_run.xml -id LND_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id OCN_DOMAIN_FILE -val 'domain.ocn.gx3vSB.nc'
./xmlchange -file env_run.xml -id OCN_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id ICE_DOMAIN_FILE -val 'domain.ocn.gx3vSB.nc'
./xmlchange -file env_run.xml -id ICE_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'

./xmlchange -file env_run.xml -id OCN2ATM_FMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_gx3vSB_TO_T31_aave.nc'
./xmlchange -file env_run.xml -id OCN2ATM_SMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_gx3vSB_TO_T31_aave.nc'

./xmlchange -file env_run.xml -id ATM2OCN_FMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_aave.nc'
./xmlchange -file env_run.xml -id ATM2OCN_SMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_patc.nc'
./xmlchange -file env_run.xml -id ATM2OCN_VMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_patc.nc'


Best,
Minmin

 

[mlevy]

Hi Minmin,

From poking through your case files, it looks like the grid file and the topography file you are sending to CICE are not the right size:

&grid_nml
 grid_file = '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da'
 grid_format = 'bin'
 grid_type = 'displaced_pole'
 gridcpl_file = 'unknown_gridcpl_file'
 kcatbound =  0
 kmt_file = '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da'
/

Your grid_file and kmt_file are both 4x larger than the default gx3v7 files:

$ ls -l /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/horiz_grid_20030806.ieeer8 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da
-rw-rw-r-- 1 fischer cseg 649600 Apr 14 2020 /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/horiz_grid_20030806.ieeer8
-rw-r--r-- 1 mjfu ncar 2598400 Nov 5 08:29 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da

$ ls -l /glade/p
/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/topography_20100105.ieeei4 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da
-rw-rw-r-- 1 fischer cseg 46400 Feb 4 2020 /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/topography_20100105.ieeei4
-rw-r--r-- 1 mjfu ncar 185600 Nov 5 08:29 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da

Can you regenerate these files, verify that they are the expected size, and then try to run again?

 

[minminfu]

Hi Mike,

This helps a lot! Now the values are close but not exactly the same and the model still fails.

See: /glade/scratch/mjfu/e.e12.E1850CN.T31_g37.1367_p-1.0_001_1000ppm_nooht_v11/run/cesm.log.211109-131425

For instance:
70: MJFU ATTR IS lat
70: MJFU n= 28 data1= 83.4789366693172 data2=
70: 83.4789357817499 diff= 8.875672676822433E-007 eps=
70: 1.000000000000000E-012

Minmin

 

[mlevy]

Hmm, that's interesting. I'm grasping at straws here, but a few possible things that could be going wrong:

1. It looks like mk_grid_1x1_template.csh generates the grid in an iterative process. If you increase the number of iterations, does the final grid get closer to what's in the domain file?
2. Is this a common issue when generating grids? Do you just need to reduce the tolerance to 1e-6?
3. How is the domain file generated? Do you need to make sure the fortran compiler / compile flags from building ns_dipole match whatever other tools are used throughout the chain? And does it also need to match what CESM is built with?

At this point, I need someone from the paleo climate to jump in and offer suggestions for getting over this (hopefully final) hurdle.

 

[minminfu]

Hi! The domain file is generated via a 3-step process from the kmt file.

1. A SCRIPgrid for the new ocean grid is produced from the kmt file via myconvertPOPT (this now works)
located at /glade/u/home/mjfu/zwork/Snowball/paleoToolkit/cesm1/cpl_mapping/scrip1.4/grids
2. The SCRIPgrid is used to create blin, patc, and aave mapping files between various components using gen_cesm_maps.sh
located at /glade/u/home/mjfu/zcesm/tools/mapping/gen_mapping_files/
3. The aave ocean to atmosphere mapping file is then converted to domain files for atm/ocn via ./gen_domain
located at /glade/u/home/mjfu/zcesm/tools/mapping/gen_domain_files/

So it seems quite a few steps occur between kmt → domain file. I also wonder whether compiler flags matter. I tried my best to be consistent throughout the chain, although errors in latitude of a maximum of around 1e-6 continue to occur.

 

[minminfu]

Quick update: if I raise eps in seq_domain_mct.F90 to 1.0e-5_R8 for lat and lon under ' --- checking ocn/ice domains ---', as well as lat,lon, & area under ' --- checking atm/land domains ---', the model successful begins to run. However, given the default eps is 1e-12, I wonder if this will result in large numerical errors that would render the simulation output unreliable. But so far it seems to run!