DART+CICE running error

[Young-chan]

Now, I am constructing the sea-ice data assimilation system using the CICE and DART packages.
Two packages were well installed in our machine, and the case was created using the shell script (CESM2_0_setup_ensemble) included in the DART (DART-10.5.3/models/cice/shell_scripts).

When I submitted this case by typing the command "case.submit", the case was quickly terminated.
In the CESM log, the following messages were repeatedly presented.

================================================================================================================
MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 13824 Number of rows in sMat = 18048
012.MCT(MPEU)::die.: from MCT::m_SparseMatrixPlus::initDistributed_()
MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 13824 Number of rows in sMat = 18048
013.MCT(MPEU)::die.: from MCT::m_SparseMatrixPlus::initDistributed_()
MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 13824 Number of rows in sMat = 18048
014.MCT(MPEU)::die.: from MCT::m_SparseMatrixPlus::initDistributed_()
================================================================================================================

As it is the first time for me to run the CICE model with the DART package, I don't know what these messages mean exactly.
The original log file and the shell script are attached here.

If you have some comments to solve this issue, please let me know.

Many thanks,

Youngchan Noh

 

[dbailey]

This does not look like a CICE error. This appears to be a DTEST compset with T62_g16 resolution. You are trying to run multi-instance? This may require help from the DART folks.

 

[Young-chan]

Thank you for your reply.

Yes, I am trying to run the CICE+DART assimilation system with five ensemble members (5 instances) that are just for the test.
In my view, these errors seem to be due to the mismatch between the sea-ice initial condition and the model grid.

How can I contact the DART folks to get some help solving this error issue?

Many thanks,

Youngchan Noh

 

[dbailey]

I have contacted them and hopefully they will reply soon.

 

[raeder]

Hi Yongchan,
I'm happy to hear that you're setting up a CICE+DART experiment.
Do you have any other experience with DART?
Our first recommendation is use the standard CESM setup scripts to setup and run a single instance DTEST experiment.
This will take DART out of the uncertainty.
If that's successful, then do the same for a multi-instance DTEST experiment.
If that is successful, then we need to see what's going wrong with the DART setup script.

The CICE+DART developers are using CESM2_1, and don't have direct experience with 2_0.
Do you need to use 2_0?

I have not used CICE+DART, but your suspicion about the initial condition file led me to look for clues in that direction.
I see in the DART setup script that the ice_ic namelist variable is being set to 'default'.
Have you identified which file is being used?
Is that a good choice for your compset, etc.?

I also see that the run type is 'startup', which means that refcase and refdate will not be used.
I don't know what is used instead, by CESM, but it's likely chosen by the ice_ic='default' setting.
If you do a hybrid or branch case in the future, I believe that you'll need to choose a date that's in the range
of the f.e21.FHIST_BGC.f09_025.CAM6assim.011 case; 2011-01-01... 2020-12-31 (or use a different case).

Kevin

 

[Young-chan]

Thank you for your kind response.

Actually, I have an experiment to run the DART system with the CAM6 as the atmosphere component in the CESM.
In the DART+CAM6 experiment, the satellite-observed radiances were assimilated on the top of the DART in which the conventional observations were assimilated.

Now, I am interested in running the DART with the CICE to see the analysis impact of the sea ice by assimilating the satellite-based sea-ice retrieval.
In South Korea, sea ice predictability in the Arctic seems to be a key factor in winter-time seasonal prediction, so I'd like to update the sea-ice initial condition by assimilating the satellite-retrieved sea-ice product in the DART+CICE. And then the CAM6 coupled with the CICE will be initiated from the updated initial condition (sea ice component from CICE DA and the atmospheric component from CAM DA).

Even though I have handled the DART+CAM6 DA system, the DART+CICE DA system is definitely different.
I respond to your questions as follows.

(1) CESM2_0 or CESM2_1 ?
In the recent version of DART (v10.5.3), the CICE standard setup script (DART-10.5.3/models/cice/shell_scripts) is named "CESM2_0_setup_ensemble or CESM1_5_setup_ensemble".Thus, I understand that the CESM v2.0 or v5.1. is able to be linked with the DART to run the CICE DA. So, I chose CESM v2.0 which is a more recent version than CESM1.5.

(2) Runtype = 'branch' or 'startup'
In the original setup script (CESM2_0_setup_ensemble), the 'RUNTYPE' is defined 'branch', meaning that "the compset are initialized using a consistent set of restart files from a previous run". In the 'branch' run, the predefined information (e.g., refcase, refdate, etc) is used for the run. But, now I don't have any restart files derived from another CICE run. Thus, we defined the 'RUNTYPE' as the 'startup'.
Instead, the 'ice_in' is defined as 'default' as follows:

"ice_ic = 'default' "

According to the CICE user guide, "ice_in = 'default' " initializes the sea ice to 100% concentration where the SST is below the freezing point to a thickness of 2 m in the Northern Hemisphere or 1 m in the Southern Hemisphere. As the SST is automatically defined in the slab ocean mode (SOM), I thought that the sea ice input data is not needed but is automatically defined by the SST derived from the SOM.

As you recommended, I tried to run the DART+CICE with a single instance. Unfortunately, the trial run failed with the input error as follows:

===============================================================================
Model datm missing file domain1 = 'UNSET/UNSET.cpl.ha2x1hi.2005-01.nc'
Cannot download file since it lives outside of the input_data_root '/lustre/ycnoh/cesm_data/inputdata'
Model datm missing file file1 = 'UNSET/UNSET.cpl.ha2x1hi.2005-01.nc'
Cannot download file since it lives outside of the input_data_root '/lustre/ycnoh/cesm_data/inputdata'
Model datm missing file file2 = 'UNSET/UNSET.cpl.ha2x1hi.2005-02.nc'
================================================================================

I tried to download the input data by typing the 'check_input_data --download', but the input data are not available.

Thank you for your help.

Youngchan Noh

 

[raeder]

I thought I recognized your name. Dr. Yonghan Choi told me that you
started working with him earlier this year. He likely knows something
about what I'll describe next. I'm glad to meet you.

If you can use a different starting date, then I see an easy way forward.
The f.e21.FHIST_BGC.f09_025.CAM6assim.011 case is a CAM6+CICE+CLM reanalysis,
which contains both an ensemble of CICE initial conditions at weekly frequency
and an ensemble of DATM, coupler history, "forcing" files, which are
the missing cpl.ha2x... files in your case.
You will feed these to CESM as "stream" files.
The initial files are "spun up" to a state that is consistent
with the 1 degree CAM6 forcing. The sea ice coverage came from
a daily, 1/4 degree AVHRR data set, while the thickness evolved
from the atmospheric forcing, and did not use any sea ice observations.
You can get them from NCAR's Research Data Archive, ds345.0.
They span 2011-2020. This data set is described in detail in a Scientific Reports paper.
I believe that these files can be used in CESM2.0, with the proper choice of resolution.

If your start year must be 2005, then there's a similar reanalysis
which used CAM4 at a 2 degree resolution for 1998-2010 in RDA 199.1.
There are no CICE initial or restart files in that dataset,
and the forcing files are packaged differently, but they have been
used in DART assimilations with CICE, CLM, and POP.
If you can't find CICE restart file(s) for your case date,
then you will need to start the case long before that date
to allow the very crude, default, CICE state to spin up
to what the CICE actually looked like. I don't know how long that takes,
but there are probably CICE papers about that.
It might be discussed on Yongfei Zhang's papers about CICE+DART.

 

[Young-chan]

Yes. Now I am working at the Korea Polar Research Institute with Dr. Yonghan Choi.
I am nice to meet you, too.

As you mentioned, I did setup a different starting date (April 6 2015) and downloaded two input data (CPL and CICE) from the reanalysis (f.e21.FHIST_BGC.f09_025.CAM6assim.011). However, the trial run failed too. The setup shell script and log files are attached here.

In my opinion, this failure seems to be due to the dimension mismatch between the CICE model grid and the CICE input data.
In this run, the CICE grid has the default setup (T62_g16, 320x384), but the CICE input (f.e21.FHIST_BGC.f09_025.CAM6assim.011.cice_0001.r.2015-04-06-00000.nc) has 1440 x 720 (1.4 degrees in the latitude and longitude). The CICE input data (1440 x 720) has too higher horizontal resolution than the model grid (320x384).

In addition, the CICE model needs five category information on the sea ice but the CICE input has just one category of the sea ice. The category mismatch also seems to be one of the reasons why the trial run failed.

Key error messages are shown as follows:
================================================================
Reading forcing_nml
Reading zbgc_nml
Reading zbgc_nml
Reading zbgc_nml
NetCDF: Start+count exceeds dimension bound
pio_support::pio_die:: myrank= -1 : ERROR: pionfread_mod.F90.in:
204 : NetCDF: Start+count exceeds dimension bound
================================================================

Is there any setup to downscale the high-resolution CICE input to the low-resolution model grid?

Many thanks,

Youngchan Noh

 

[Young-chan]

As the f.e21.FHIST_BGC.f09_025.CAM6assim.011.cice_0001.r.2015-04-06-00000.nc has one category of sea-ice, I modified the number of the category from five (default) to one in the CICE setup file (CESM-2.0.1/components/cice/cime_config/buildcpp).
And then I ran the trial run, but different error messages occurred as follows:

cesm.log.7170.esniac00.221129-134651
===========================================================================
MCT::m_AttrVectComms::GSM_scatter_: myID = 0. Invalid input, GSMap%gsize = 55296, lsize(iV) = 18048
shr_mct_sMatReaddnc: Error on scatter of areadst0
000.MCT(MPEU)::die.: from shr_mct_sMatReaddnc()
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 2) - process 0
===========================================================================

According to two log files (cesm.log.7170.esniac00.221129-134651 and cpl.log.7170.esniac00.221129-134651), these errors seem to be related to the CPL processing. However, I cannot understand what these errors mean.

Three log files are attached here.

Many thanks,

Youngchan Noh

 

[raeder]

Hi Younchan,
I'm sorry to hear that using the Reanalysis CICE restart files is not as simple as I hoped.

There's a regridding tool (xesmf; python) that should be useful in this case.
I have no experience with it, so I can't directly help.
There's an example of its use for generating forcing files for CICE here,
but you (or we) will need to adapt it to convert the 1/4 degree CICE files
into the CICE grid you want.
I don't know much about handling (thickness?) categories.
If we're fortunate, the regridding will handle them too.

The regridding tool (in "conservative" mode) may smooth the data,
but it seems that if you're starting from high resolution (1/4 degree)
and interpolating to a lower resolution (g16), then smoothing shouldn't be a problem.

I'll try to look at the log files and other output you sent later this week.

 

[Young-chan]

Hi Kevin,

I found that the errors, above mentioned, are due to two issues as follows:

1. Incorrect definition of model resolution
Even though the CPL data (f.e21.FHIST_BGC.f09_025.CAM6assim.011.cpl_0001.ha2x1d.2015.nc) has f09 resolution (192x288), the default setup (T62_g16) was still used in the shell script, which means the CPL forcing data of 96 x 192 and the CICE data of gx1v6 (320 x 384). So did occur the grid mismatch between the CPL forcing data and the defined model CPL resolution in the shell script.
Now, I defined the correct model resolution of f09_g16 in the shell script.

2. Grid mismatch of CICE data
First, I repeatedly tried to use the NCAR reanalysis CICE data (f.e21.FHIST_BGC.f09_025.CAM6assim.011.cice_0001.r.2015-04-06-00000.nc), but the trial run failed. Considering that the NCAR CICE data has one category of sea ice, I modified the CICE source code to use one sea-ice category rather than five categories. But, the trial run still failed.
Thus, I changed the CICE input data from the NCAR CICE data to the sample historical CICE output (b.e20.B1850.f09_g17.pi_control.all.297.cice.r.0130-01-01-00000.nc) that was obtained by googling. This historical CICE data has the normal CICE resolution (gx1v6, 320 x 384) and five sea-ice categories, even though the CICE date is not matched with the trial run (April 6 2015).

As a result, the CICE test run was normally finished, even though the model outputs are not checked yet.
Thanks to your comments, these issues are solved.

Many thanks,

Youngchan