Resubmissions crash when writing out data in 1D vector format

[nick]

As a simple example, I can run an F2000climo case with the following CLM namelists and it successfully gives me 1D output, but upon the first resubmission the model crashes before the first timestep with a seg fault (pasted below too). I don't think I've used the namelists incorrectly and I don't see a github issue on this. Any insight or pointing-out of errors in my ways would be appreciated. I'm using CESM2.1.3. Possibly a related issue here.

hist_dov2xy = .true., .false.
hist_type1d_pertape = ' ', 'PFTS'
hist_fincl2 = 'TG', 'TV'
hist_nhtfrq = 0, -24
hist_mfilt = 1, 30


forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpnetcdf.so.3.0 00002ACB2F008FD2 for__signal_handl Unknown Unknown
libpthread-2.17.s 00002ACB30291630 Unknown Unknown Unknown
cesm.exe 00000000028B4DE4 m_rearranger_mp_r 845 m_Rearranger.F90
cesm.exe 00000000028AA130 m_matattrvectmul_ 514 m_MatAttrVectMul.F90
cesm.exe 000000000213E111 rtmmod_mp_rtmrun_ 1615 RtmMod.F90
cesm.exe 0000000002121290 rof_comp_mct_mp_r 304 rof_comp_mct.F90
cesm.exe 0000000000434407 component_mod_mp_ 728 component_mod.F90
cesm.exe 00000000004188D5 cime_comp_mod_mp_ 2733 cime_comp_mod.F90
cesm.exe 0000000000434047 MAIN__ 125 cime_driver.F90
cesm.exe 00000000004164E2 Unknown Unknown Unknown
libc-2.17.so 00002ACB304C0555 __libc_start_main Unknown Unknown
cesm.exe 00000000004163E9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpnetcdf.so.3.0 00002ACB2F009071 for__signal_handl Unknown Unknown
libpthread-2.17.s 00002ACB30291630 Unknown Unknown Unknown
libiomp5.so 00002ACB2E5AF455 Unknown Unknown Unknown
libiomp5.so 00002ACB2E5AF2A7 Unknown Unknown Unknown
libiomp5.so 00002ACB2E5AEFF2 Unknown Unknown Unknown
libiomp5.so 00002ACB2E5AF7ED Unknown Unknown Unknown
libiomp5.so 00002ACB2E53DDE1 Unknown Unknown Unknown
ld-2.17.so 00002ACB2679F098 Unknown Unknown Unknown
libc-2.17.so 00002ACB304D7CE9 Unknown Unknown Unknown
libc-2.17.so 00002ACB304D7D37 Unknown Unknown Unknown
libpnetcdf.so.3.0 00002ACB2F004555 for__issue_diagno Unknown Unknown
libpnetcdf.so.3.0 00002ACB2F008FD2 for__signal_handl Unknown Unknown
libpthread-2.17.s 00002ACB30291630 Unknown Unknown Unknown
cesm.exe 00000000028B4DE4 m_rearranger_mp_r 845 m_Rearranger.F90
cesm.exe 00000000028AA130 m_matattrvectmul_ 514 m_MatAttrVectMul.F90
cesm.exe 000000000213E111 rtmmod_mp_rtmrun_ 1615 RtmMod.F90
cesm.exe 0000000002121290 rof_comp_mct_mp_r 304 rof_comp_mct.F90
cesm.exe 0000000000434407 component_mod_mp_ 728 component_mod.F90
cesm.exe 00000000004188D5 cime_comp_mod_mp_ 2733 cime_comp_mod.F90
cesm.exe 0000000000434047 MAIN__ 125 cime_driver.F90
cesm.exe 00000000004164E2 Unknown Unknown Unknown
libc-2.17.so 00002ACB304C0555 __libc_start_main Unknown Unknown
cesm.exe 00000000004163E9 Unknown Unknown Unknown

 

[nick]

I seemed to have come across the same or similar issue reported a couple of years ago here. It seems that requesting a variable at a lower level of subgrid hierarchy than it is defined at causes the problem. e.g. asking for a variable to be output at the patch/PFT level when it is only defined at the column level, I think, causes CESM to crash on restart. It's only on restart though so it obviously is doing something right, just not everything right.

I've tested this by requesting TG at the 'PFTS' level, which produces a crash when the model resubmits. But when I request GSSUNLN at the 'PFTS' level the model can restart fine.

 

[nick]

The solution turns out to be simple. If requesting multiple variables on a single history stream that are defined at different subgrid levels, simply set hist_type1d_pertape = ' '. This will output each variable on its native subgrid level. e.g. requesting TG and GSSUNLN on the same file should be done like so:

hist_dov2xy = .true., .false.
hist_type1d_pertape = ' ', ' '
hist_fincl2 = 'TG','GSSUNLN'

Whereas using the following namelist instead of the above results in a crash on resubmission because TG isn't defined at the PFT level.
hist_dov2xy = .true., .false.
hist_type1d_pertape = ' ', 'PFTS'
hist_fincl2 = 'TG','GSSUNLN'