Error when changing nlev for ideal-phys model

[kangwanying1992@gmail_com]

Hi,I was trying to add stratosphere to an ideal-physics model (CESM1.2.2, F_IDEAL_PHYS, f19_f19) by "./xmlchange -file env_build.xml -id CAM_CONFIG_OPTS -val "-phys ideal -nlev 66"" and change initial ncdata to a WACCM initial file. Generally for other models, this works well (used this to SPCAM and also change vertical levels in WACCM). This time they first report "Lagrangian levels are crossing", but this is fixed by increasing nsplit and nspltvrm. Then everything seems to be fine, but the model stopped before finishing the first time step. The attachments are log files and the configuration files I used to build the model. atm.log and cpl.log seem not showing any error report. The only difference between these logs and those of successful model is in cesm.log file. It shows: (line 11424)========================= "   0:/glade/p/cesmdata/cseg/tools/bin/hybrid_launch: line 80: 18697 Segmentation fault      /usr/bin/numactl -C $cpuset -N $mynode $@INFO: 0031-251  task 0 exited: rc=-11"=========================Really do not understand what is going on...I am Grateful to any suggestion! Thank you in advance!Wanying

 

[santos]

Can you do the following?> ./IDEAL_strat.clean_build all> ./xmlchange DEBUG=TRUE> ./IDEAL_strat.build> ./IDEAL_strat.submitThis will probably still fail, but it will be interesting to see if there is new output in the log files, or core files in the run directory.

 

[kangwanying1992@gmail_com]

Dear Sean,I have tried debug mode, but no extra information is given in the log files... The attachment are the rundirectory for 'FV ideal-phys 66lev' model and 'SE ideal-phys 66lev'.... Appreciate any suggestion for next step...Best,Wanying 

 

[santos]

I noticed your case directory on yellowstone, and I see that there actually is information in this file:/glade/scratch/wykang/IDEAL_strat512FV/run/coredir.11/core_liteIt is not very obvious, but I can see some information here:#7  0x00000000028c953b in geopk (grid=..., pe=..., delp=..., pk=..., wz=..., hs=..., pt=..., cp=@0x408f651eb851eb85, akap=@0x3fd2492e38a1f3e8, nx=@0x5243444300000004) at /glade/p/ucsu0031/wykang/CESM1_2_2/models/atm/cam/src/dynamics/fv/geopk.F90:100
#8  0x000000000267622c in cd_core (grid=..., nx=@0x5243444300000004, u=..., v=..., pt=..., delp=..., pe=..., pk=..., ns=@0x2, dt=@0x404c200000000000, ptopin=@0x3f3d7e960814a178, umax=@0x4072c00000000000, pi=@0x400921fb54442d18, ae=@0x41584de500000000, cp=@0x408f651eb851eb85, akap=@0x3fd2492e38a1f3e8, iord_c=@0xfffffffefffffffe, jord_c=@0x4fffffffe, iord_d=@0x400000004, jord_d=@0x400000004, ipe=@0x1, om=@0x3f131da947eca4ca, hs=..., cx3=..., cy3=..., mfx=..., mfy=..., delpf=..., uc=..., vc=..., ptc=..., dpt=..., ptk=..., wz3=..., pkc=..., wz=..., hsxy=..., ptxy=..., pkxy=..., pexy=..., pkcc=..., wzc=..., wzxy=..., delpxy=..., pkkp=..., wzkp=..., cx_om=..., cy_om=..., filtcw=@0x0, s_trac=@0xffffffff00000000, mlt=@0x100000000, ncx=@0x0, ncy=@0x0, nmfx=@0x0, nmfy=@0x0, iremote=..., cxtag=..., cytag=..., mfxtag=..., mfytag=..., cxreqs=..., cyreqs=..., mfxreqs=..., mfyreqs=...) at /glade/p/ucsu0031/wykang/CESM1_2_2/models/atm/cam/src/dynamics/fv/cd_core.F90:1249
This says that on line 1249 of cd_core, it calls geopk, and somewhere near line 100 of geopk, the error occurs. Furthermore, the CESM log says "task 11 exited: rc=-8". This is what is printed for SIGFPE, which is a floating point exception.Typically, an crash in cd_core represents a CFL violation in the dycore, i.e. the winds are too fast for the dynamical core to handle. This is not too surprising, because WACCM is only stable with parameterized gravity wave and ion drag enabled. If you are running all the way up to the lower thermosphere with only ideal physics, these processes are not present, so you will probably need to find some other form of drag to control the wind speeds. Otherwise, you will have to lower the lid so that you don't include the entire mesosphere (which might not be enough), or further increase the value of nsplit.

 

[kangwanying1992@gmail_com]

Thank you so much! And all these are super super helpful, otherwise I will be stucked on this forever! Now I know the potential problem that faces me, but still not clear about how to deal with that. If you could help me, I will be more than appreciate.  What I am trying to do is to use  idealized physics (what the physics package do is just relax the state to an equilibrium temperature) simulate troposphere and stratosphere, so there is not real physics in that and I do not really need mesosphere (hopefully a model top at 1pa may be enough)... Do you think I can avoid this if moving down the model top to be 1pa? If the answer is no, then is the WACCM initial condition cause the problem? If so, may I try to interpolate and expolate a CAM initial condition to start up the model? Thank you again!Best regards,Wanying

 

[santos]

I am not exactly sure what's best. I think that the easiest thing to try is to take WACCM initial conditions, and remove all levels past 1 Pa from the file. Then you can try a high nsplit/nspltrac/nspltvrm to spin up the model, and then reduce nsplit once you can do so stably.

 

[kangwanying1992@gmail_com]

Dear Santos,Thank you for your suggestion! I set dtime manually to be 1/10 of default value (also increase CLM_NCPL, ATM_NCPL etc.) and increase nsplit a lot to have the model adapt to IC. Next step, I need to change them back to default value. But when I did a branch run from this initial run, it output every 1/10 days or 1/10 months instead of at whole day counts... That may be caused by my changing dtime from 1800 to 180 in the initial run and changing it back to 1800 in the branch run. Do you have any suggestion on this?1) Can I run a continue run from the initial one? Since I need change user_nl_cam (where I set dtime) and env_run.xml (where I set ATM_NCPL, CLM_NCPL...), do I need to clean build and build again? Is this a continue run?2) Branch run? How can i tell the model the dtime in the two runs are different...3) Hybrid run? Still not sure...My RUNDIR is /glade/scratch/wykang/IDEAL_INITIAL for initial run and /glade/scratch/wykang/IDEAL_strat1pa_CONT for branch run... Thank you so much for answering these silly questions...Best,Wanying

 

[santos]

If you set ATM_NCPL to a higher value, it is not necessary to reduce dtime manually, since this is done automatically when ATM_NCPL is changed. However, this is probably not causing your problem.To answer your questions:1) If you continue (or "restart") a run, that only can happen within your existing case, and you cannot change dtime or any history outputs. I don't think that that is what you want.2) If you do a "branch" run, it's probably not possible to change dtime (or it may not do the right thing), because the restart file may not have all the outputs at the right time.3) You probably don't want to use a "hybrid" run, because that mostly is used to get a good ocean state for B cases, but you are just running ideal physics. It would probably work, but it's unnecessarily complicated for you.4) What you actually want to do is to do a "startup" run, but with an initial condition file from your spun up case.So my recommendation is: - Redo your run with 1/10 dtime, only with the following line in user_nl_cam:        inithist = "ENDOFRUN" - This will create an IC file that has ".cam.i." in the middle of name. - Then create a new case, still as a startup run, but this time, add the name of the file produced in the previous run as the value of "ncdata" in user_nl_cam. - Also, copy the ".cam.i." file from the previous run to the case directory of the new run.Hopefully, the new run should work with the longer dtime.

 

[santos]

Oops, typo. I meant the file should be copied to the run directory of the new case, not to the case directory.