WACCM-X crash at high solar activity - reduce time step?

[inos@bas_ac_uk]

Hello all,I am running a CESM2.1.0 WACCM-X historical compset (FXHIST) on a HPC facility in the UK. My simulation ran successfully from 1950-01-01 until 1957-12-21, at which point it crashes. Sometimes the following error appears in the cesm and/or atm log file:SHR_REPROSUM_CALC: Input contains  0.10811E+06 NaNs and  0.00000E+00 INFs on process       0
 ERROR: shr_reprosum_calc ERROR: NaNs or INFs in inputI suspect this error is due to the model becoming unstable due to the very high solar activity level (F10.7>360 sfu) in Dec 1957, which will give very high temperatures and gradients in the upper atmosphere. I am running with the Matthes et al. (2017) CMIP6 solar forcing (Geosci. Model Dev., 10, 2247–2302).Following advice given elsewhere on the forum, I have tried gradually increasing fv_nsplit and fv_nspltrac and fv_nspltvrm to get past the period of instability. However, even with fv_nsplit = 256, fv_nspltrac = 32, and fv_nspltvrm = 32 I am still getting the same crash. So I am wondering if I might need to reduce the actual model time step to get past the period of high solar activity. But how can I do that? I have tried doubling ATM_NCPL in env_run.xml, but that results in an error about timings being out of sync, so I'm clearly missing something. Can anyone on here offer some advice?Many thanks,Ingrid

 

[mmills]

Hello Ingrid,Before attempting to reduce the model time step, have you tried higher vaiues of fv_nspltrac and fv_nspltvrm? Setting them to the same high value as fv_nsplit, such as 128 (or in your case 256), can often be helpful, particularly if the instability involves tracer advection or vertical remapping.

 

[mmills]

Hello Ingrid,Before attempting to reduce the model time step, have you tried higher vaiues of fv_nspltrac and fv_nspltvrm? Setting them to the same high value as fv_nsplit, such as 128 (or in your case 256), can often be helpful, particularly if the instability involves tracer advection or vertical remapping.

 

[inos@bas_ac_uk]

Hi Mike,I've tried running with fv_nsplit = 256, fv_nspltrac = 256, and fv_nspltvrm = 256, but it crashes at the same point again. I do not get a warning about vertical levels crossing either, so it doesn't look to me like that's the problem here. Do you have any other suggestions?

Thanks,Ingrid

 

[inos@bas_ac_uk]

Hi Mike,I've tried running with fv_nsplit = 256, fv_nspltrac = 256, and fv_nspltvrm = 256, but it crashes at the same point again. I do not get a warning about vertical levels crossing either, so it doesn't look to me like that's the problem here. Do you have any other suggestions?

Thanks,Ingrid

 

[stans]

Try increasing osplit (the number of O+ transport timesteps per 5-minute physics time step). It's unlikely that the physics time step itself is the problem.Are you running at the higher (~1 degree) resolution?  If so, you should try running at the lower (~2 degree) resolution, if you don't absolutely need the high-res output.

 

[stans]

Try increasing osplit (the number of O+ transport timesteps per 5-minute physics time step). It's unlikely that the physics time step itself is the problem.Are you running at the higher (~1 degree) resolution?  If so, you should try running at the lower (~2 degree) resolution, if you don't absolutely need the high-res output.

 

[joemci@ucar_edu]

Hi Ingrid, You can try setting the following in your user_nl_cam file to see if it helps.  I beliieve this is set to 15 by default. dadadj_niter           = 30 Joe

 

[joemci@ucar_edu]

Hi Ingrid, You can try setting the following in your user_nl_cam file to see if it helps.  I beliieve this is set to 15 by default. dadadj_niter           = 30 Joe

 

[inos@bas_ac_uk]

Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid

 

[inos@bas_ac_uk]

Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid

 

[inos@bas_ac_uk]

Hi Joe,Thanks for your advice. I tried with dadadj_niter = 30, but it still crashed, I think even slightly earlier than it did before. However, I then tried Stan's suggestion above and it looks like this solved the problem. Ingrid

 

[inos@bas_ac_uk]

Hi Joe,Thanks for your advice. I tried with dadadj_niter = 30, but it still crashed, I think even slightly earlier than it did before. However, I then tried Stan's suggestion above and it looks like this solved the problem. Ingrid

 

[ZYue]

Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid

Hi Ingrid, I met the same questions. How can I set this parameter "ionos_xport_nsplit"?

 

[CaliFornia]

Hi Ingrid, I met the same questions. How can I set this parameter "ionos_xport_nsplit"?

Hi ZYue, did you solve it? I grep 'ionos_xport_nsplit' but found nothing under the /run folder

 

[inos@bas_ac_uk]

Hi both,
Sorry for the slow response - I don't get a notification when another comment comes in on an existing thread. But you need to set "ionos_xport_nsplit" in your user_nl_cam file. So add a line there, e.g.: ionos_xport_nsplit=10