High resolution F-compset deep time configuration

[bblack@berkeley_edu]

Dear CESM bulletin board community:

When I submit my run, it exits with the error appended below from the cesm.log file. The other log files are not very informative, though I have the debugging output (INFO_DBUG) set to the highest level. I have double-checked my SST file, my land and ocean domain files, my surface dataset, and they all seem to be fine. Substituting the default SST file produces the same error. It is possible that this error message is not what is causing the model to fail.

As background, I am trying to get a high degree (0.47x0.63°) case running for the Permian. I already have a 1.9x1.5 simulation running fine, but during the course of adapting the inputs to the higher resolution I have run into this problem.

Any suggestions would be greatly appreciated. In the meantime, my likely next step is to insert print statements into the Fortran code to try to track down the source of the error.

Thanks very much in advance for your help,
Ben

-------------------------------
excerpt from cesm.log file:

1: Opened existing file
1: /glade/p/work/bblack/H2S/hdeg_cshields/lnd/mksurfdata_hdeg/surfdata_384x576_permian_ccsm4_140314.nc 458752
103: Remapping the ITD is not allowed for ncat=1
103: Using the delta function ITD option instead

... (repeated many times)...

103: Remapping the ITD is not allowed for ncat=1
103: Using the delta function ITD option instead

0:CalcWorkPerBlock: Total blocks: 1536 Ice blocks: 1536 IceFree blocks: 0 Land blocks: 0
255:(shr_stream_set) size of filename = 1
255:(shr_stream_set) filename = /glade/p/work/bblack/H2S/hdeg_cshields/docn/sst_b40.t31x3.pt.5C_bc_0.47x0.63_clim_140325.bab.nc

 

[jedwards]

I don't see any particular reason for your failure - yellowstone was having filesystem issues yesterday afternoon that could have caused this...

 

[bblack@berkeley_edu]

Hi Jim,Thanks very much for your comment. I have tried submitting this run a number of times over the past two weeks, with slightly different namelist parameters, and I consistently get the same failure. So unfortunately I don't think it is related to yesterday's Yellowstone filesystem issues.The traceback from one of the coredir files is as follows:Thread 1 (Thread 0x2b1ad91c81a0 (LWP 18396)):#0  0x00002b1ad2a397ad in waitpid () from /lib64/libc.so.6#1  0x00002b1ad29cb889 in do_system () from /lib64/libc.so.6#2  0x00002b1ad29cbbc0 in system () from /lib64/libc.so.6#3  0x00002b1ad94251af in pm_linux_print_coredump () from /opt/ibmhpc/pe1307/base/intel/lib64/libpoe.so#4  0x00002b1ad9423f1a in pm_lwcf_signal_handler () from /opt/ibmhpc/pe1307/base/intel/lib64/libpoe.so#5  #6  0x00000000009b87fd in ytp (va=Cannot access memory at address 0x240#7  TP_CORE::tp2d (va=Cannot access memory at address 0x240#8  0x00000000009b6977 in TP_CORE::tp2c (dh=Cannot access memory at address 0x240#9  0x00000000009aae11 in SW_CORE::d_sw (grid=Cannot access memory at address 0x240#10 0x00000000008c71a7 in cd_core (grid=Cannot access memory at address 0x240#11 0x000000000070a929 in DYN_COMP::dyn_run (ptop=Cannot access memory at address 0x240#12 0x00000000005df951 in STEPON::stepon_run1 (dtime_out=Cannot access memory at address 0x240#13 0x0000000000494dd8 in CAM_COMP::cam_run1 (cam_in=Cannot access memory at address 0x240#14 0x00000000004911b7 in ATM_COMP_MCT::atm_init_mct (eclock=Cannot access memory at address 0x240#15 0x0000000000429bfb in CCSM_COMP_MOD::ccsm_init () at /glade/p/cesm/cseg/collections/cesm1_1_1/models/drv/driver/ccsm_comp_mod.F90:2194#16 0x000000000042f53e in ccsm_driver () at /glade/p/cesm/cseg/collections/cesm1_1_1/models/drv/driver/ccsm_driver.F90:90 #17 0x000000000040e93c in main ()Best,Ben

 

[jedwards]

Ahh - that's new information.   TP_CORE is a typical place that the model will fail if the input parameters are really messed up (eg T< 0K) or the CFL condition is exceeded.    Try experimenting with the following: nspltvrm

Number of vertical re-mapping timesteps per physics timestep.
Nspltrac is partitioned into nspltvrm and nspltrac/nspltvrm,
with the latter being the number of tracer timesteps per
re-mapping timestep, possibly rounded upward; after initialization,
the code quantity nspltrac is redefined to be the number of
tracer timesteps per re-mapping timestep.
Default: 0
<br /><br />        <code class="varname">
          <a>nsplit</a>
        </code>

Number of dynamics timesteps per physics timestep. If zero, a best-estimate
will be automatically calculated.
Default: 0

<code class="varname">
          <a>nspltrac</a>
        </code>

Number of tracer advection timesteps per physics timestep.
Nsplit is partitioned into nspltrac and nsplit/nspltrac,
with the latter being the number of dynamics timesteps per
tracer timestep, possibly rounded upward; after initialization,
the code quantity nsplit is redefined to be the number of
dynamics timesteps per tracer timestep.
Default: 0

 

[bblack@berkeley_edu]

Thanks very much. It does seem possible that the timestepping is the issue. I set dtime = 1200 in both the CAM and CLM namelists. I then get the following message in the log file: FV subcycling - nv, n2, nsplit, dt =            1           2           6   100.000000000000 ERROR:  Because of loop nesting, FV dycore can't use the specified namelist set tings for subcycling   The original namelist settings were:   NSPLIT   =           11   NSPLTRAC =            2    NSPLIT needs to be a multiple of NSPLTRAC   Suggested settings would be:   NSPLIT   =           12    NSPLTRAC =            2 I tried the suggested values of nsplit (=12) and nspltrac (=2), and I also tried several values of nspltvrm. The message I pasted above disappears, but the model continues to exit (seemingly in the same place based on the cesm.log file).