I'm spinning up the snowpack in an I compset, cycling through 20 year streams from a coupled F compset simulation. After 290 years into the simulation, ctsm errors out due to a negative h2osoi_ice occurrence. From the cesm.log:
The code base is a branch off of ctsm1.0.dev079. The log files are here:
/glade/scratch/aherring/ctsm1.0.dev079.se_grids.n02_I2000Clm50SpSpinup_ne0np4.ARCTIC.ne30x4_mt12_1800pes_200412_1979bc-Nx10yrs/run
Ideally, I would branch off this run well before the crash occurred, but with some type of work around in the code so I can complete the spin-up.
Spin-up progress is attached, for context.
Code:
281: ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negativ
281: e
281: Bulk/tracer name = bulk
281: c, lev_top(c) = 36026 0
281: h2osoi_ice_top_orig = 3.707650617013752E-006
281: h2osoi_ice = -5.285485590866834E-019
281: frac_sno_eff = 1.00000000000000
281: qflx_dew_snow*dtime = 0.000000000000000E+000
281: qflx_sub_snow*dtime = 3.707650617014281E-006
281: ENDRUN:
281: ERROR:
281: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negative
281:Image PC Routine Line Source
281:cesm.exe 0000000001AADF9A Unknown Unknown Unknown
281:cesm.exe 00000000011AAA00 shr_abort_mod_mp_ 114 shr_abort_mod.F90
281:cesm.exe 00000000005032AF abortutils_mp_end 50 abortutils.F90
281:cesm.exe 00000000008C2522 snowhydrologymod_ 1190 SnowHydrologyMod.F90
281:cesm.exe 00000000008BF03D snowhydrologymod_ 986 SnowHydrologyMod.F90
281:cesm.exe 000000000081ABD5 hydrologynodraina 287 HydrologyNoDrainageMod.F90
281:cesm.exe 000000000050AD68 clm_driver_mp_clm 778 clm_driver.F90
281:cesm.exe 00000000004F843B lnd_comp_mct_mp_l 458 lnd_comp_mct.F90
281:cesm.exe 00000000004282A4 component_mod_mp_ 737 component_mod.F90
281:cesm.exe 0000000000409DCB cime_comp_mod_mp_ 2615 cime_comp_mod.F90
281:cesm.exe 0000000000427EDC MAIN__ 133 cime_driver.F90
281:cesm.exe 0000000000407CA2 Unknown Unknown Unknown
281:libc.so.6 00002BA1FD7BF6E5 __libc_start_main Unknown Unknown
281:cesm.exe 0000000000407BA9 Unknown Unknown Unknown
281:MPT ERROR: Rank 281(g:281) is aborting with error code 1001.
281: Process ID: 43616, Host: r4i5n11, Program: /glade/scratch/aherring/ctsm1.0.dev079.se_grids.n02_I2000Clm50SpSpinup_ne0np4.ARCTIC.ne30x4_mt12_1800pes_200412_1979bc-Nx10yrs/bld/cesm.exe
281: MPT Version: HPE MPT 2.19 02/23/19 05:30:09
281:
281:MPT: --------stack traceback-------
281:MPT: Attaching to program: /proc/43616/exe, process 43616
281:MPT: done.
The code base is a branch off of ctsm1.0.dev079. The log files are here:
/glade/scratch/aherring/ctsm1.0.dev079.se_grids.n02_I2000Clm50SpSpinup_ne0np4.ARCTIC.ne30x4_mt12_1800pes_200412_1979bc-Nx10yrs/run
Ideally, I would branch off this run well before the crash occurred, but with some type of work around in the code so I can complete the spin-up.
Spin-up progress is attached, for context.