Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Deep time paleoclimate: (seq_domain_check_grid) ERROR: incompatible domain grid coordinates

minminfu

Member
Hi, I am trying to build some CESM 1.2.2.1 experiments with dramatically different continental configurations following the instructions available at PaleoResources for CESM1.2.

After producing my KMT file, I used the convertPOPT utility from /paleoToolkit/cesm1/cpl_mapping/scrip1.4/grids/convertPOPT.f to create a SCRIPgrid. I used ./gen_cesm_maps.sh to create mapping files, and used ./gen_domain.sh on my mapping file to produce land and ocean domains. The only place I deviate is I had to use convertPOPT (with hard coded file names) instead of myconvertPOPT, because myconvertPOPT was not giving valid output data (I am unsure if this could be a source of error, but the output from convertPOPT seemed reasonable). When I try to launch a simulation with the new domain and mapping files, I persistently get the error:

(seq_domain_check_grid) ERROR: incompatible domain grid coordinates

This is cryptic, so I added some print statements in seq_domain_mct.F90. It seems the error occurs during the ice/ocean grid consistency check, in particular during comparison of the latitude attribute. Furthermore, the data1 and data2 are not even close.

e.g,
10: MJFU n= 31 data1= -68.7069428207174 data2=
10: -85.0038255423153 diff= 16.2968827215979 eps=
10: 1.000000000000000E-002
10: MJFU ATTR IS lat
10: MJFU n= 32 data1= -68.7069428207174 data2=
10: -85.0038255423153 diff= 16.2968827215979 eps=
10: 1.000000000000000E-002

I have set the CICE and DOCN namelists such that they use the exact same domain file, so I am not sure how they can be inconsistent. user_nl_cice also points to the relevant grid and kmt files used to generate the domain file. I was wondering if you have any idea how to resolve this issue or where the problem may originate from. I am at wit’s end, so your help is extremely appreciated! Thank you for your time, it is very much appreciated!

kmt and domain files can be found at
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/domain.lnd.48x96_gx3vSB.nc
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/domain.ocn.gx3vSB.nc
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da
/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da

My procedure for using paleotoolkit is documented here
/glade/u/home/mjfu/zwork/Snowball/generate_pop_ensemble.sh
/glade/u/home/mjfu/zwork/Snowball/generate_cpl_ensemble.sh

CESM Log file here:
/glade/scratch/mjfu/e.e12.E1850CN.T31_g37.1367_p-1.0_001_1000ppm_nooht_v9/run/cesm.log.211106-090511

Other namelist changes:

# KEEP CN MODEL OFF
./xmlchange -file env_build.xml -id CLM_CONFIG_OPTS -val "-phys clm4_0"

./xmlchange -file env_run.xml -id ATM_DOMAIN_FILE -val 'domain.lnd.48x96_gx3vSB.nc'
./xmlchange -file env_run.xml -id ATM_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id LND_DOMAIN_FILE -val 'domain.lnd.48x96_gx3vSB.nc'
./xmlchange -file env_run.xml -id LND_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id OCN_DOMAIN_FILE -val 'domain.ocn.gx3vSB.nc'
./xmlchange -file env_run.xml -id OCN_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'
./xmlchange -file env_run.xml -id ICE_DOMAIN_FILE -val 'domain.ocn.gx3vSB.nc'
./xmlchange -file env_run.xml -id ICE_DOMAIN_PATH -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001'

./xmlchange -file env_run.xml -id OCN2ATM_FMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_gx3vSB_TO_T31_aave.nc'
./xmlchange -file env_run.xml -id OCN2ATM_SMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_gx3vSB_TO_T31_aave.nc'

./xmlchange -file env_run.xml -id ATM2OCN_FMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_aave.nc'
./xmlchange -file env_run.xml -id ATM2OCN_SMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_patc.nc'
./xmlchange -file env_run.xml -id ATM2OCN_VMAPNAME -val '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/map_T31_TO_gx3vSB_patc.nc'


Best,
Minmin
 

mlevy

Michael Levy
CSEG and Liaisons
Staff member
Hi Minmin,

From poking through your case files, it looks like the grid file and the topography file you are sending to CICE are not the right size:

Code:
&grid_nml
 grid_file = '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da'
 grid_format = 'bin'
 grid_type = 'displaced_pole'
 gridcpl_file = 'unknown_gridcpl_file'
 kcatbound =  0
 kmt_file = '/glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da'
/

Your grid_file and kmt_file are both 4x larger than the default gx3v7 files:

Code:
$ ls -l /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/horiz_grid_20030806.ieeer8 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da
-rw-rw-r-- 1 fischer cseg 649600 Apr 14 2020 /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/horiz_grid_20030806.ieeer8
-rw-r--r-- 1 mjfu ncar 2598400 Nov 5 08:29 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/grid.1.pop.da

$ ls -l /glade/p
/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/topography_20100105.ieeei4 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da
-rw-rw-r-- 1 fischer cseg 46400 Feb 4 2020 /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx3v7/grid/topography_20100105.ieeei4
-rw-r--r-- 1 mjfu ncar 185600 Nov 5 08:29 /glade/u/home/mjfu/zwork/Snowball/FINAL_OUTPUT/topo_ens_n12_p-1.0_001/kmt.1.da

Can you regenerate these files, verify that they are the expected size, and then try to run again?
 

minminfu

Member
Hi Mike,

This helps a lot! Now the values are close but not exactly the same and the model still fails.

See: /glade/scratch/mjfu/e.e12.E1850CN.T31_g37.1367_p-1.0_001_1000ppm_nooht_v11/run/cesm.log.211109-131425

For instance:
70: MJFU ATTR IS lat
70: MJFU n= 28 data1= 83.4789366693172 data2=
70: 83.4789357817499 diff= 8.875672676822433E-007 eps=
70: 1.000000000000000E-012

Minmin
 

mlevy

Michael Levy
CSEG and Liaisons
Staff member
Hmm, that's interesting. I'm grasping at straws here, but a few possible things that could be going wrong:

1. It looks like mk_grid_1x1_template.csh generates the grid in an iterative process. If you increase the number of iterations, does the final grid get closer to what's in the domain file?
2. Is this a common issue when generating grids? Do you just need to reduce the tolerance to 1e-6?
3. How is the domain file generated? Do you need to make sure the fortran compiler / compile flags from building ns_dipole match whatever other tools are used throughout the chain? And does it also need to match what CESM is built with?

At this point, I need someone from the paleo climate to jump in and offer suggestions for getting over this (hopefully final) hurdle.
 

minminfu

Member
Hi! The domain file is generated via a 3-step process from the kmt file.

1. A SCRIPgrid for the new ocean grid is produced from the kmt file via myconvertPOPT (this now works)
located at /glade/u/home/mjfu/zwork/Snowball/paleoToolkit/cesm1/cpl_mapping/scrip1.4/grids
2. The SCRIPgrid is used to create blin, patc, and aave mapping files between various components using gen_cesm_maps.sh
located at /glade/u/home/mjfu/zcesm/tools/mapping/gen_mapping_files/
3. The aave ocean to atmosphere mapping file is then converted to domain files for atm/ocn via ./gen_domain
located at /glade/u/home/mjfu/zcesm/tools/mapping/gen_domain_files/

So it seems quite a few steps occur between kmt → domain file. I also wonder whether compiler flags matter. I tried my best to be consistent throughout the chain, although errors in latitude of a maximum of around 1e-6 continue to occur.
 

minminfu

Member
Quick update: if I raise eps in seq_domain_mct.F90 to 1.0e-5_R8 for lat and lon under ' --- checking ocn/ice domains ---', as well as lat,lon, & area under ' --- checking atm/land domains ---', the model successful begins to run. However, given the default eps is 1e-12, I wonder if this will result in large numerical errors that would render the simulation output unreliable. But so far it seems to run!
 
Top