Main menu

Navigation

WACCM-X crash at high solar activity - reduce time step?

7 posts / 0 new
Last post
inos@...
WACCM-X crash at high solar activity - reduce time step?

Hello all,

I am running a CESM2.1.0 WACCM-X historical compset (FXHIST) on a HPC facility in the UK. My simulation ran successfully from 1950-01-01 until 1957-12-21, at which point it crashes. Sometimes the following error appears in the cesm and/or atm log file:

SHR_REPROSUM_CALC: Input contains  0.10811E+06 NaNs and  0.00000E+00 INFs on process       0
 ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input

I suspect this error is due to the model becoming unstable due to the very high solar activity level (F10.7>360 sfu) in Dec 1957, which will give very high temperatures and gradients in the upper atmosphere. I am running with the Matthes et al. (2017) CMIP6 solar forcing (Geosci. Model Dev., 10, 2247–2302).

Following advice given elsewhere on the forum, I have tried gradually increasing fv_nsplit and fv_nspltrac and fv_nspltvrm to get past the period of instability. However, even with fv_nsplit = 256, fv_nspltrac = 32, and fv_nspltvrm = 32 I am still getting the same crash. So I am wondering if I might need to reduce the actual model time step to get past the period of high solar activity. But how can I do that? I have tried doubling ATM_NCPL in env_run.xml, but that results in an error about timings being out of sync, so I'm clearly missing something. Can anyone on here offer some advice?

Many thanks,

Ingrid

mmills

Hello Ingrid,

Before attempting to reduce the model time step, have you tried higher vaiues of fv_nspltrac and fv_nspltvrm? Setting them to the same high value as fv_nsplit, such as 128 (or in your case 256), can often be helpful, particularly if the instability involves tracer advection or vertical remapping.

Mike Mills
WACCM Liaison
Atmospheric Chemistry Division
NCAR Foothills Lab
Boulder, Colorado USA

inos@...

Hi Mike,

I've tried running with fv_nsplit = 256, fv_nspltrac = 256, and fv_nspltvrm = 256, but it crashes at the same point again. I do not get a warning about vertical levels crossing either, so it doesn't look to me like that's the problem here. Do you have any other suggestions?

Thanks,

Ingrid

stans

Try increasing osplit (the number of O+ transport timesteps per 5-minute physics time step). It's unlikely that the physics time step itself is the problem.

Are you running at the higher (~1 degree) resolution?  If so, you should try running at the lower (~2 degree) resolution, if you don't absolutely need the high-res output.

inos@...

Hi Stan,

Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.

Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.

Ingrid

joemci@...

Hi Ingrid,

 

You can try setting the following in your user_nl_cam file to see if it helps.  I beliieve this is set to 15 by default.

 dadadj_niter           = 30

 

Joe

inos@...

Hi Joe,

Thanks for your advice. I tried with dadadj_niter = 30, but it still crashed, I think even slightly earlier than it did before. However, I then tried Stan's suggestion above and it looks like this solved the problem.

Ingrid

Log in or register to post comments

Who's new

  • opioronald123@...
  • shimura.tomoya....
  • rack.hun.son@...
  • nathalie.schall...
  • sf870222@...