CCSM3 - CAM error "interval not found"

[umerkel@marum_de]

Dear CCSM experts,

I am currently encountering problems with running CCSM3 (T31_gx3v5) after successful building.
The model doesn't seem to work correctly on the first timestep (please also see messages attached below)

In the atmospheric log file, I get the message

interval was not found for col i

(from CAM routine pkg_cld_sediment.F90).

Reducing the timestep as suggested in another thread of this forum didn't help.

Does anyone have another idea what could cause the problem?
I would be very grateful for any advice.
Thank you very much,
Ute


From the general log file:

...
nstep, te 1 3336908873.72217 0.000000000000000E+000 0.000000000000000E+000 98466.0465231863
nstep, te 2 NaN NaN NaN NaN
MPI: MPI_COMM_WORLD rank 45 has terminated without calling MPI_Finalize()
MPI: aborting job


From the atmosphere log file:

...
(CCSMINI): CCSM initialization complete!

COURANT
NSTEP RMSZ RMSD RMST STPS STQ HOR VERT
NSTEP = 0 8.848256296797174E-05 6.001568877021020E-06 252.524 9.84660E+04 2.488706315074846E+01 0.41 0.07
REFERENCE TEMPERATURES FOR SEMI-IMPLICIT SCHEME =
300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000
300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000 300.000
300.000 300.000
GRAVITY WAVE PHASE SPEEDS (M/S) FOR MEAN STATE =
341.754 204.787 129.913 91.097 68.276 53.568 43.265 35.745 29.950 25.411 21.748 18.719
16.190 14.026 12.136 10.473 9.004 7.692 6.497 5.392 4.380 3.464 2.611 1.884
0.720 1.213
GRAVITY WAVE EQUIVALENT DEPTHS (M) FOR MEAN STATE =
11910.418 4276.669 1721.099 846.262 475.379 292.623 190.884 130.299 91.472 65.850 48.231 35.732
26.730 20.063 15.019 11.185 8.268 6.033 4.304 2.965 1.957 1.224 0.695 0.362
0.053 0.150
NSTEP = 1 NaN NaN NaN NaN NaN 0.41 0.00

 

[eaton]

From the output you've posted I'm guessing that the error message from pkg_cld_sediment is a symptom rather than the root of the problem. The log contains NaN for a lot of global average fields before the executions actually stops. I suggest building the executable with compiler options that allow trapping floating point exceptions so that execution stops when this happens rather than continuing to produce NaNs. That should help to trace the problem to its source.