Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

WACCM-X crash at high solar activity - reduce time step?

Hello all,I am running a CESM2.1.0 WACCM-X historical compset (FXHIST) on a HPC facility in the UK. My simulation ran successfully from 1950-01-01 until 1957-12-21, at which point it crashes. Sometimes the following error appears in the cesm and/or atm log file:SHR_REPROSUM_CALC: Input contains  0.10811E+06 NaNs and  0.00000E+00 INFs on process       0
 ERROR: shr_reprosum_calc ERROR: NaNs or INFs in inputI suspect this error is due to the model becoming unstable due to the very high solar activity level (F10.7>360 sfu) in Dec 1957, which will give very high temperatures and gradients in the upper atmosphere. I am running with the Matthes et al. (2017) CMIP6 solar forcing (Geosci. Model Dev., 10, 2247–2302).Following advice given elsewhere on the forum, I have tried gradually increasing fv_nsplit and fv_nspltrac and fv_nspltvrm to get past the period of instability. However, even with fv_nsplit = 256, fv_nspltrac = 32, and fv_nspltvrm = 32 I am still getting the same crash. So I am wondering if I might need to reduce the actual model time step to get past the period of high solar activity. But how can I do that? I have tried doubling ATM_NCPL in env_run.xml, but that results in an error about timings being out of sync, so I'm clearly missing something. Can anyone on here offer some advice?Many thanks,Ingrid
 

mmills

CSEG and Liaisons
Staff member
Hello Ingrid,Before attempting to reduce the model time step, have you tried higher vaiues of fv_nspltrac and fv_nspltvrm? Setting them to the same high value as fv_nsplit, such as 128 (or in your case 256), can often be helpful, particularly if the instability involves tracer advection or vertical remapping.
 

mmills

CSEG and Liaisons
Staff member
Hello Ingrid,Before attempting to reduce the model time step, have you tried higher vaiues of fv_nspltrac and fv_nspltvrm? Setting them to the same high value as fv_nsplit, such as 128 (or in your case 256), can often be helpful, particularly if the instability involves tracer advection or vertical remapping.
 
Hi Mike,I've tried running with fv_nsplit = 256, fv_nspltrac = 256, and fv_nspltvrm = 256, but it crashes at the same point again. I do not get a warning about vertical levels crossing either, so it doesn't look to me like that's the problem here. Do you have any other suggestions?

Thanks,Ingrid
 
Hi Mike,I've tried running with fv_nsplit = 256, fv_nspltrac = 256, and fv_nspltvrm = 256, but it crashes at the same point again. I do not get a warning about vertical levels crossing either, so it doesn't look to me like that's the problem here. Do you have any other suggestions?

Thanks,Ingrid
 

stans

New Member
Try increasing osplit (the number of O+ transport timesteps per 5-minute physics time step). It's unlikely that the physics time step itself is the problem.Are you running at the higher (~1 degree) resolution?  If so, you should try running at the lower (~2 degree) resolution, if you don't absolutely need the high-res output.
 

stans

New Member
Try increasing osplit (the number of O+ transport timesteps per 5-minute physics time step). It's unlikely that the physics time step itself is the problem.Are you running at the higher (~1 degree) resolution?  If so, you should try running at the lower (~2 degree) resolution, if you don't absolutely need the high-res output.
 

joemci@ucar_edu

New Member
Hi Ingrid, You can try setting the following in your user_nl_cam file to see if it helps.  I beliieve this is set to 15 by default. dadadj_niter           = 30 Joe
 

joemci@ucar_edu

New Member
Hi Ingrid, You can try setting the following in your user_nl_cam file to see if it helps.  I beliieve this is set to 15 by default. dadadj_niter           = 30 Joe
 
Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid
 
Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid
 
Hi Joe,Thanks for your advice. I tried with dadadj_niter = 30, but it still crashed, I think even slightly earlier than it did before. However, I then tried Stan's suggestion above and it looks like this solved the problem. Ingrid
 
Hi Joe,Thanks for your advice. I tried with dadadj_niter = 30, but it still crashed, I think even slightly earlier than it did before. However, I then tried Stan's suggestion above and it looks like this solved the problem. Ingrid
 

ZYue

ZYue
New Member
Hi Stan,Thanks for the tip! I'm not sure how to increase osplit though. It doesn't seem to be a namelist parameter (at least it's not in atm_in). Do I need to change it in the code itself somewhere? I am running with the 2-degree resolution already.Edit: I found the namelist parameter "ionos_xport_nsplit". I increased it from 5 to 10 and am now running beyond the point where it crashed previously, so it looks like this did the trick.Ingrid
Hi Ingrid, I met the same questions. How can I set this parameter "ionos_xport_nsplit"?
 
Hi both,
Sorry for the slow response - I don't get a notification when another comment comes in on an existing thread. But you need to set "ionos_xport_nsplit" in your user_nl_cam file. So add a line there, e.g.: ionos_xport_nsplit=10
 
Top