Issue with setting landuse timeseries on CTSM5.2

[Greta Shum]

What version of the code are you using?
ctsm-5.2.0

Have you made any changes to files in the source tree?
I've made changes to the NTASKS in env_mach_pes.xml and some start/end date changes (see replay.sh), but I've also tried skipping these and get the same result.

Describe every step you took leading up to the problem:
I'm running a 0.5x0.5 resolution, IHistClm50BgcCrop compset case with data atmosphere files from 1901 to 2019 on the VSC HPC, Hortense.
I have a successful version of this case with non-transient surfdata, using fsurdat = '/cesm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_360x720cru_hist_1850_78pfts_c240216.nc' in the user_nl_clm
But, when I change to a transient landuse case with flanduse_timeseries='/cesm/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/landuse.timeseries_360x720cru_SSP2-4.5_1850-2100_78pfts_c240216.nc', it fails, and I get an error in the cesm.log file as shown in the attached cesm.log.10552665.250218-095917_surfdata_esmf:

(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
PIO2 pio_file.c retry NETCDF
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 1998829 on node node5490.dodrio.os exited on signal 9 (Killed).
--------------------------------------------------------------------------
3 total processes killed (some possibly by mpirun during cleanup)
2025-02-17 19:19:51,489 ERROR RunAsyncMPI MainThread _post_exitcode: problem occured with cmd ['mpirun', '-machinefile', ...
''

I've also tried to use the landuse.timeseries in surfdata_map/, but the case fails because of missing PCT_LAKE_MAX (as in this thread) in the fsurfdata_timeseries file, see cesm.log.10566063.250219-111304_surfdata_map.

Describe your problem or question:
I've started trying to generate my own flanduse_timeseries file using /tools/mksurfdata_esmf/ but is this possible on machines other than Derecho? I'm also wondering in general whether there should be be any obvious difference between how I should generate this file and this one on the trunk (inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_360x720cru_hist_1850_78pfts_c240216.nc).

For now arguments are `./gen_mksurfdata_namelist --res 360x720cru --start-year 1901 --end-year 2019 --rawdata-dir /path/to/my/cesm/inputdata --ssp-rcp "SSP3-7.0"`

Thanks in advance for your help!

 

[oleson]

Have you looked at the lnd log? Maybe there is better information about the error there than in the cesm log?

 

[Greta Shum]

Oh right! Attaching the lnd.log file here (for the case attempting to load the esmf sufdata timeseries). I couldn't immediately see a reason for the fail, but it does look like it's reading in the file.

 

[oleson]

Ok, I don't see anything useful in the lnd log file either, and as you say it looks like it is reading in the landuse file. I guess maybe look at the other log files, e.g., the atm log. Have you run with the full set of this atmospheric forcing before?
I think the surface dataset and landuse timeseries files you are using (c240216) should work with ctsm5.2.0.

 

[Greta Shum]

Glad to hear it's not something super obvious! I know this set of datm forcing setup does work with static land use (only `fsurdat = ...`), and I've tried cloning the case that worked and only adding `flanduse_timeseries = ...` to the namelist, and it fails with the same error logs. I've also tried setting up the case with the default atm forcing, and it fails as well in the same way. I'm attaching a .zip of the run directory here.

My new suspicion is this is a memory issue. When I set this up with lower resolution (f19_g17 as opposed to hcru_hcru_mt13) it runs fine. So this might be a machine optimization issue.

Thanks again for any additional insight.

 

[oleson]

I agree that the successful case at f19 might indicate there is a memory problem with hcru_hcru_mt13.
Another thing to check is to make sure the landuse timeseries file is not corrupted in some way. E.g., try doing an ncdump -k on the file.
You could also try running in DEBUG mode to see if you get a better traceback.