CAM4 crashes when running with SOM (CESM1 E_1850_CN compset)

[slgs@hawaii_edu]

Hi there,

I have been banging my head against the wall trying to run the CESM1 in the slab ocean model configuration for a couple of weeks now and was hoping someone could help me. I need to run the model using the E_1850_CN compset with a forcing file derived from a coupled CESM1 simulation which was itself run at T31x3 resolution. I am able to get the model to run just fine using the out-of-the-box file, but can't do it with a file I generated myself.

I followed the instructions in the README file at models/ocn/docn/tools/pop_som_frc, and used a modified version of pop_frc_mlt.ncl to create a new forcing file for my simulation. But when I start CESM1 using this file, it fails after about a minute of runtime with an error that appears to come from the CICE module. The following is the output from my ice.log file:


(ice_init_mct) idate from sync clock = 10101
(ice_init_mct) tod from sync clock = 0
(ice_init_mct) resetting idate to match sync clock

istep1: 0 idate: 10101 sec: 0

Warning: Departure points out of bounds in remap
my_task, i, j = 0 25 11
dpx, dpy = -574806.569721255 -574910.258176825
HTN(i,j), HTN(i+1,j) = 168597.487171003 168597.487169934
HTE(i,j), HTE(i,j+1) = 168586.024086546 178692.867991835
istep1, my_task, iblk = 1 0 1
Global block: 1
Global i and j: 24 10
(shr_sys_abort) ERROR: remap transport: bad departure points
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping


so it looks to me like there is some kind of numerical issue with the sea ice propagation. But I don't know how to fix this - all the fields in my forcing file look reasonable, and are on the same order of magnitude as those in the default CESM forcing file pop_frc.1x1d.090130.nc.

Any ideas on what could cause this kind of error would be really helpful, thanks very much!!

 

[dbailey]

I sent this by email as well.

We have been having issues with the bulletin board and are working on an upgrade. Cecile forwarded your email as it seems most relevant to me. This is definitely a CFL instability issue in the sea ice model. Have a look at the SOM forcing file that you generated just using ncview. What do the ocean currents look like? What do the sea surface tilt terms look like? I suspect there is a units issue here. What ocean model simulation did you use to generate the SOM forcing?

Dave

 

[slgs@hawaii_edu]

Hi Dave,

I sent an email earlier with some details of my setup, I hope you received it. I thought I would update the thread here since I have now isolated the problem as an adjustment from the initial conditions from which the model starts - I got it to run for about 4 months without crashing, and the problem is that the Arctic SSTs in my desired forcing simulation are MUCH warmer than the ones in the CESM default configuration, such that all the ice is melting unphysically fast and crashing CICE. So I would like to initialize the model from a different set of restart files where the SST etc. are closer to the mean conditions in the forcing file I would eventually like to use, but don't have another run with the E compset to use to generate these files. Do you know if it's possible to begin a restart simulation branching from a run performed with a different compset? I have a climatologically forced CAM4 run (F compset) which could do the job, but when I try to use it as a starting point for a "hybrid" simulation, the CLM build crashes giving me an error in histFileMod.F90 saying "internal compiler error: internal abort".

Thanks for all your help,

-Sam