Coupling CICE6 to ROMS

[Stefan]

Hi, what is my best approach to implement coupling of CICE6 to ROMS ? And is this the right place to ask this question ?

Just in case, has anybody done this already and willing to share the code changes necessary in CICE6 ?
In the past I have played with a coupled ROMS-CICE5 system but I did not implement the coupling so I have no experience with CICE code as such.

After adapting env.machine and cice.setting (gnu compiler, netcdf) CICE6 standalone compiles fine on my machine (Ubuntu 18.04, 8 cores).
Now, as recommended in the CICE Documentation I looked at the already implemented CESM coupling capability that uses MCT. It appears to be activated from cpp option CESMCOUPLED. The driver lives under cicecore/drivers/mct/cesm1/. Eventually I would copy that folder into something like cicecore/drivers/mct/ROMS where the adaptations to ROMS would be made.

But as a first step I try to compile CICE with the CESMCOUPLED flag on (without actually planning to run it)
changes in
cice.settings
setenv ICE_DRVOPT mct/cesm1
in cice.build
setenv ICE_CPPDEFS "-DCESMCOUPLED"

now the first question would be how does CICE find the MCT libraries ?
I downloaded MCT2.1, built the library and I'm now looking for the right place to inform CICE about their existence.

I tried adding the MCT inc and lib path to the cice.build
## List of source code directories (in order of importance).
cat >! Filepath << EOF
local/MCT/lib
local/MCT/include
${ICE_SANDBOX}/cicecore/drivers/${ICE_DRVOPT}
${ICE_SANDBOX}/cicecore/cicedynB/dynamics

the compilation fails as soon as one of the mct module is called.

/cicecore/drivers/mct/cesm1/ice_cpl_indices.F90:3:6:

   use mct_mod

      1

Fatal Error: Can't open module file ‘mct_mod.mod’ for reading at (1): No such file or directory

compilation terminated.

Obviously I need to tell CICE something else about the MCT library.
Apologies for these naive questions but are the mct modules actually compiled with CICE ? Do I need to direct CICE to the build code of MCT?

Any help and leads would be much appreciated.
thank you
Stefan

 

[tcraig]

Just to take a step back, the mct/cesm1 interfaces are used to couple models thru the CESM coupler. MCT is generally part of CESM and is built with CESM. CESM has a coupler that calls into the different components using this mct/cesm1 interface. The mct/cesm1 interface meets the requirements for coupling in CESM1. If you are using a version of CESM to couple ROMS and CICE6, the mct/cesm1 interface may meet your needs.

If you are trying to develop a new coupled system that just includes ROMS and CICE6, then there are a number of issues to consider. It sounds like you are able to run ROMS and CICE6 each as standalone models, but now you want to couple them? If that is your intent, you need to decide how you plan to couple. Will you have an external driver and coupler to call ROMS and CICE6 separately. That driver/coupler might manage the time sequencing, coupling field exchange, and many other features. That would need to be designed and implemented. Or you could start with the CESM driver/coupler, for instance. There are many other ways to move forward. You could have the ROMS standalone model call into CICE6. Or you could have the CICE6 standalone model call into ROMS. In either case, you would need to design an interface to manage the coupling exchange. You could use Oasis3-MCT to implement coupling between standalone CICE6 and ROMS via the Oasis coupling approach. You could implement a coupled system based on the NUOPC/ESMF package. There are other options as well.

The main point is that the coupling interfaces that are provided in CICE6 are for implemented and working coupled systems. In many ways, CICE6 plays little role in implementing or maintaining those interfaces. Those interfaces do not simply "work" for any application. These interfaces are implemented for particular coupled systems after significant efforts in designing the overall coupling. It is possible the existing coupling interfaces in CICE6 could be leveraged for a new coupled system or could be used by reusing an existing coupled system. But you cannot simply compile CICE6 with mct and expect that to magically couple to ROMS with an mct interface added. If you do want to use the cesm1/mct interfaces, you would also need to build a driver/coupler component (or use CESM1).

Instantiating a coupled system in not a simple operation. There are lots of examples and several different infrastructure packages that can aid in doing so. Even in the case where you leverage an existing coupled system, there is often significant work to get the science functioning properly. I'm happy to make some additional recommendations if you clarify a bit whether you are using CESM as a starting point and/or what your expectations and plans are for the system.

 

[Stefan]

Dear Replier (Craig?),

thank you so much for taking the time to reply to my call for help. I really appreciate this!

Setting up this coupled system will require time and a steep learning curve on my part. I have exhausted the “out of the box” approaches with older CICE-ROMS version and for sure I'm not expecting “magic simply work” solutions. There is no other choice than digging in and spend the effort of actually implementing this by myself. So thanks for clarifying my confusion about MCT and the coupling used in CESM.
But clearly CESM is not my starting point unfortunately. CICE and ROMS are. If there is another more suitable forum to deal with CICE6 please point me to it if possible.

1. Choice of a coupler
With no experience but after playing, running and modifying a few of the Oasis3-mct tutorial examples I would choose Oasis3-mct.
- handle ROMS standalone and CICE standalone
- seems well documented, hopefully allows me to understand the details of the strategy and implementation
- allows coupling within model / components (i.e. subcomponents) which could be a powerful tool for future ideas
- building up knowledge/capability how to build Oasis & CICE could allow to couple other ocean models and atmospheric model in the future
(one PhD project on its way once Covid-19 allows ).
So getting up to speed with one coupler software and CICE in a rather comprehensive way is worth the effort.

2. Coupling Strategy
naturally a very uneducated guess and apologies if that is too simplistic
- start both models, two individual binaries on their own set of MPI tasks,
e.g. mpirun -np 100 ./oceanM < roms.in : -np 20 ./cice > trial.out
- each model initializes coupler, submitting some form of model/component ID
- let each model‘s mpi tasks (==rectangular grid tile in ROMS; more sophisticated “blocks” distribute across tasks in CICE ?)
-- request/sort out its local MPI communication details
-- submit its local grid decomposition details to coupler, (i.e. tile corners ?)
-- register its variables and associated IDs with coupler (although this could be done outside the decompositioned part ?)
- once the physics is running, mpi tasks
-- send time stamped field to coupler at predefined intervals, alongside with the associated, preregistered field ID
-- receive variable field from coupler, time stamped and ID’ed
-- this means CICE, coupler and ROMS need to handle time consistent, i.e. equal starting time and model time unit
- let coupler handle
-- the merging of decompositioned sent field
-- spatial interpolation/averaging to receiver grid by some pre computed weighting matrix
-- split up field onto receiver grid decomposition
-- exchange of fields should probably happen every 30 ocean minutes.
- need to ensure that ROMS does not loose much speed by waiting for CICE
- this assumes for example that get_field_from_coupler() function when called in CICE will hold its execution until the coupler has processed the field received from ROMS
- it seems a load leveler which comes with Oasis could be used to balance that

3 Alternative
A different (conceivable?) solution would be to run CICE inside ROMS. But my insights are too limited - I can only think this through as far as the vertical component of CICE. How to advect ice inside ROMS, especially across tile boundaries and to the satisfaction of CICE I can not see being easy. Whatever CICE uses as ice state pencil needs to be submitted to the ROMS internal mp exchanger ... That might mess up CICE's own exchange logic. I fear this is way more work than the first approach.
The beauty of course would be that CICE implicitly sits on the same grid as ROMS and the system maintains its nesting capabilities which is somewhat important for my applications. A disadvantage is that CICE would become difficult to update with new releases once it is torn to pieces to live inside ROMS.

Regardless, all of this is rather speculative. Despite having spent some time browsing the code I have no real insights to how CICE is structured and if the above is even possible, let alone how to code this up. I’m open to leads how to start this project.

Thank you
Stefan

 

[tcraig]

I believe Oasis3-MCT is a good choice for coupling CICE and ROMS as you suggest. In addition, it should be extensible to swap in or add other components in the future. It is used in many coupled systems around the world to couple many sophisticated models. There are many nice features as you point out, especially for implementing new coupled systems from scratch from standalone models. I suggest as a start, you simply couple CICE and ROMS and not create a coupler. If you want to create a coupler, you'll need to develop a separate third component to do the field merging and other operations. Oasis3-MCT is not a separate coupler, it is just coupling infrastructure that is called from components. A better initial approach might be to just couple CICE and ROMS and to do any field merging needed in those components before or after sending/receiving fields. You should be able to couple fields on different mulitple grids (say grid centers and/or corners) and Oasis3-MCT should be able to handle the interpolation. It should also be able to handle any lags, coupling frequency, and time averaging needed.

To implement Oasis3-MCT, you should be able to follow the Oasis3-MCT documentation and add the required calls in CICE6. My recommendation would be to start with the "standalone/cice" code, copy it to a "direct/oasis3-mct" directory, and then add the oasis calls there. Hopefully, oasis calls appear only at that level of code. If we can provide any other guidance, please ask. Finally, if you do implement CICE6 coupled thru Oasis3-MCT, I hope you might consider offering the modified code to the CICE-Consortium in the future. Thanks.

 

[XavierC]

Hi Stefan,
time flew since your last post . Did you get somewhere with OASIS and CICE ?
X.