Trouble starting from spun-up initial conditions

[urata@colorado_edu]

Hi,
I'm having trouble running CAM from a spun-up state. I start the model after 2 years of spinning up using the model-generated cam.i.x and clm2.i.x files. The problem that I run in to is that the model crashes after around 125 days. The output file when it crashes looks like:

QNEG4 WARNING from TPHYSAC , lchnk = 14; Max possible LH flx exceeded at 1 points. Worst excess = NaN at i = 17
QNEG3 from vertical diffusion/Q:m= 1 lat/lchnk= 48 Min. mixing ratio violated at 26 points. Reset to 1.0E-12 Worst = NaN at i,k= 8 26
filew still failed NaN 9.741562530802809E-010
filew still failed NaN 2.322888841582044E-044
....(many more filew errors)
then some of these errors:
dp_coupling cant adjust 1 32 14
0.000000000000000E+000 0.000000000000000E+000 5.100744301052514E-013
dp_coupling cant adjust 1 40 6
0.000000000000000E+000 0.000000000000000E+000 5.009632175122562E-013

I have made changes to the code to allow for a different calendar, vapor pressure equations for low temperatures (atm and lnd), and added some custom modules. The model runs fine for long periods of time if I start from a non spun-up state, but when I start with that initial state I am forced to use a small time step due to water conservation issues.

 

[rneale]

Does the model restart ok? This seems like very odd behavior and if the model crashes even with many different initial files then there is something odd in your code that is making the run crash.

 

[urata@colorado_edu]

As far as I can tell, the model restarts okay. I am not sure what I should be looking for in terms of if it is not restarting properly. I run into these issues when I restart with large time steps (600 or 1200 sec), but seem to be okay at 200 sec. The model seems to run for longer with the 600 sec time step than the 1200 sec time step, so it's possible that I just haven't run the model long enough with the 200 sec time step, and that I would eventually run into the same issue though.

 

[rneale]

What resolution are you running at? We have experience with similar problems with resolutions of 0.5 degree (T170 or so) when the very strong polar night jet goes unstable. Also where are you getting the model generated cam.i and clm.i files? Are they from a run you did with the same code base or are they the ones downloaded from the CAM website?

 

[urata@colorado_edu]

I am running at a resolution of 10x15 with the FV core. I am working on a Mars CAM, adapting the Earth physics to Mars, so the model has very low pressures, and very little water in the atmosphere. Since I am working with Mars, the topography varies a lot, and 1 degree is roughly 60 km. I modified the saturation water vapor pressure equations in both the atmosphere and land models to account for the low temperatures, and added a switch to account for the triple point of water.

I began having these problems when I changed the vapor pressure equation in the land model, so I imagine there might be problems with water near the surface, but I am not sure exactly where the problem originates.

Edit:
I don't know if this is related to the above problem, but it seems that the reason why the model is quitting is because of an endrun call in the cfint2 subroutine in the pkg_cld_sediment.F90 module. The message says "interval not found for col i." The filew errors might be related, but the model isn't quitting at exactly that point.