MPI Invalid Tag Error

What version of the code are you using?
cesm 2.1.5

Have you made any changes to files in the source tree?
No

Describe every step you took leading up to the problem:
Compset: 2000_DATM%CPLHIST_SLND_CICE_DOCN%SOM_SROF_SGLC_SWAV_TEST
Grid: Grid resolution
Number of instances: 20
Compiler: intel, intelmpi

Describe your problem or question:
Hi everyone,

I am trying to run a multi-instance standalone CICE run with data atmosphere and ocean.
The atmospheric forcing is a coupler history output from a fully coupled control simulation.
The ocean forcing is slab ocean model forcing also generated from the fully coupled control simulation.

I have perturbed the atmospheric forcing files to generate a 20-member ensemble run.
When I submit the case, the run fails due to an invalid tag MPI error (the log files and env_mach_pes.xml are attached).
No errors are seen from the component log files, but only in the cesm.log file (even with the DEBUG=TRUE).

Is there a way to solve this issue?

Best,
Hee-Sung

I don't see any obvious reason for the error. You might try using fewer instances - maybe start with 2 and work your way up?

Hi jedwards,

Thanks for the hint.
I tried with a fewer instances and it seems to work well for up to 10 instances!

Best
Hee-Sung

There is a variable in MPI MPI_TAB_UB, you might check the value of that variable - it might explain why tag values of 1060075 and
larger are failing. Maybe if you can find where that mpi_irecv call is we can replace the tag with something that will work in your case.

Hi jedwards,

I managed to get around the problem by setting ./xmlchange MULTI_DRIVER=TRUE
following the instruction on 10. Multi-instance component functionality — CIME cime5.6 documentation

Thanks for all the help!

Best,
Hee-Sung