CESM simulation stops at the start of the second month

[jinmuluo]

My CESM code version: CESM3_beta06


Hi,

I run a CESM simulation with CAM-Chem and CLM on, and I did see my case successfully output the first month results, but my case stops at the second month, which is very weird.

Here is the my case location on derecho: /glade/derecho/scratch/jinmuluo/O3_Crop_soil_nox_on/run

And in the CaseStatus, it says

2025-11-11 17:38:54: case.run starting 3589111.desched1
---------------------------------------------------
2025-11-11 17:38:58: model execution starting 3589111.desched1
---------------------------------------------------
2025-11-11 22:30:54: model execution success 3589111.desched1
---------------------------------------------------
2025-11-11 22:30:54: case.run error
ERROR: Model did not complete - see /glade/derecho/scratch/jinmuluo/O3_Crop_soil_nox_on/run/med.log.3589111.desched1.251111-173854

But nothing wrong in the med.log.3589111.desched1.251111-173854, and no obvious error can be detected in the cesm.log

best.

Jinmu

 

[oleson]

I see this in your cesm log:

dec1952.hsn.de.hpc.ucar.edu 1016: forrtl: severe (408): fort: (2): Subscript #1 of the array HISTO has value 21 which is greater than the upper bound of 20
dec1952.hsn.de.hpc.ucar.edu 1016:
dec1952.hsn.de.hpc.ucar.edu 1016: Image PC Routine Line Source
dec1952.hsn.de.hpc.ucar.edu 1016: cesm.exe 00000000092BA9DD histfilemod_mp_hf 3734 histFileMod.F90
dec1952.hsn.de.hpc.ucar.edu 1016: cesm.exe 00000000092D4E18 histfilemod_mp_hi 4259 histFileMod.F90
dec1952.hsn.de.hpc.ucar.edu 1016: cesm.exe 0000000008F349DB clm_driver_mp_clm 1449 clm_driver.F90
dec1952.hsn.de.hpc.ucar.edu 1016: cesm.exe 0000000008E4AF39 lnd_comp_nuopc_mp 899 lnd_comp_nuopc.F90

The lines near 3734 in histFileMod.F90 are:

if (numdims == 1) then
allocate(hist1do(beg1d_out:end1d_out), stat=ier)
if (ier /= 0) then
write(iulog,*) trim(subname),' ERROR: allocation'
call endrun(msg=errMsg(sourcefile, __LINE__))
end if
hist1do(beg1d_out:end1d_out) = histo(beg1d_out:end1d_out,1)
end if

It looks like the model is dying trying to write out the h2 CLM history file. One of your requested history fields must be causing problems. I see from your lnd_in that you are requesting a bunch of history fields at the column level for the h2 file. Sometimes requesting a variable at a given level that doesn't exist can cause unexpected problems. For example, requesting a variable at the column level when it doesn't exist at the column level, e.g., it is a grid level variable only, is a problem.
To troubleshoot, I'd look at the variables you've requested for the h2 file, particularly any new variables you've added.
It looks like the model was successful at writing out the h0 and h1 files...