running speed difference between spinup and hist

[Fuhow]

Hi everyone,


I’m currently running a CTSM case with a regional grid of 687*315, using regional surfdata and landuse.timeseries file. I found that during the accelerated spin-up for the I2000 case, it took about 1 minute per model day with 416 processes.


However, after finishing spin-up and switching to the hybrid run for the ihist case (initialized from spin-up restart files), with a 16 GB landuse.timeseries dataset spanning 1980–2023, the model now only runs one model day in about 5 minutes.


Is this large difference in performance normal? Any advice or guidance would be greatly appreciated.


Thanks!

 

[slevis]

I do not know exactly how much slower the model should be, but a slowdown is expected for two reasons that I can think of:
- In accel. spin-up the model outputs significantly reduced amounts of history
- In accel. spin-up the model does not engage in transient landuse calculations

 

[Fuhow]

I do not know exactly how much slower the model should be, but a slowdown is expected for two reasons that I can think of:
- In accel. spin-up the model outputs significantly reduced amounts of history
- In accel. spin-up the model does not engage in transient landuse calculations

Dear @slevis

Thanks for your response.

For condition 1, I set the same daily output variable.
For example, during accel, I set：
fire_method = 'nofire'
hist_empty_htapes = .true.
hist_fields_list_file = .false.

hist_fincl1 = 'TOTECOSYSC','TOTSOMC','TOTVEGC','LIVESTEMC','LEAFC','TLAI','GPP','CPOOL','NPP','NEE','GPP','NEP','SOILC_vr','SOILC_HR','TSOI','H2OSOI','ALT','ER','AR','RR','HR'
hist_fincl2 = 'TOTECOSYSC','TOTSOMC','TOTVEGC','LIVESTEMC','LEAFC','TLAI','GPP','CPOOL','NPP','NEE','GPP','NEP','SOILC_vr','SOILC_HR','TSOI','H2OSOI','ALT','ER','AR','RR','HR'

hist_mfilt = 1, 1
hist_nhtfrq = -24, -8760

during IHIST case run, I set:
fire_method = 'nofire'
hist_empty_htapes = .true.
hist_fields_list_file = .false.

hist_fincl1 = 'TOTECOSYSC','TOTSOMC','TOTVEGC','LIVESTEMC','LEAFC','TLAI','GPP','CPOOL','NPP','NEE','GPP','NEP','SOILC_vr','SOILC_HR','TSOI','H2OSOI','ALT','ER','AR','RR','HR'
hist_fincl2 = 'TOTECOSYSC','TOTSOMC','TOTVEGC','LIVESTEMC','LEAFC','TLAI','GPP','CPOOL','NPP','NEE','GPP','NEP','SOILC_vr','SOILC_HR','TSOI','H2OSOI','ALT','ER','AR','RR','HR'

hist_mfilt = 1, 1
hist_nhtfrq = -24, -8760

So, I think this might not be the reason of low run speed.

For condition 2, I dont know whether the yearly landusetimeseries file is readed each time step, is so, the 16G large file must be the reason of low run speed, but if the landusetimeseries file is read at the first time step of the year and it might not be the reason of the low run speed. Meanwhile, I checked the output time between 2 years, for example:
...
01:01 TP_5km_cruj_hist.clm2.h0a.1983-12-30-00000.nc
01:06 TP_5km_cruj_hist.clm2.h0a.1983-12-31-00000.nc
01:10 TP_5km_cruj_hist.clm2.h0a.1984-01-01-00000.nc
01:26 TP_5km_cruj_hist.clm2.h0a.1984-01-02-00000.nc
01:31 TP_5km_cruj_hist.clm2.h0a.1984-01-03-00000.nc
...
it can be seen that the output time between 1984-01-01 (end of 1983) and 1984-01-02 (start of 1984) take 11 min more than other in year output, so I gusse the landusetimeseries is readed at the start of each year, and it might not be the reason of low daily run speed.

Besides, I checked the difference between 2 lnd.log of IHIST and accel, it shows 48 compute orbital parameters betweeen 2 daily output:

(shr_orb_params) ------ Computed Orbital Parameters ------
(shr_orb_params) Eccentricity = 1.671015E-02
(shr_orb_params) Obliquity (deg) = 2.344185E+01
(shr_orb_params) Obliquity (rad) = 4.091374E-01
(shr_orb_params) Long of perh(deg) = 1.026214E+02
(shr_orb_params) Long of perh(rad) = 4.932674E+00
(shr_orb_params) Long at v.e.(rad) = -3.252218E-02

I think this might be the resaon of low daily output speed.

Therefore, I would like to ask you to help check whether my speculation is correct: is the slow runtime at each time step caused by the need to compute orbital information at every step? I have also asked others about the runtime speed of IHSIT, and their experience is that slowdowns typically occur when starting the IHIST simulation using restart files, and the speed usually returns to normal after running for about 20 years, but I am not sure what causes this. If so, is there any way to solve this problem?

 

[oleson]

The landuse timeseries should be read in once per year, this is normal.
Orbital calculations are done every time step in a historical, this is normal.
There are other input streams that are being read in at various time scales in a historical, this is normal.
I would expect a typical historical to be 2-3 times slower than a spinup depending on configuration, while yours is 5 times slower.
You were already quite slow in your spinup, 3.9 model years/wallclock day. You may just be reaching the limits of your system/processors. Maybe there are memory problems also.

I don't have any specific suggestions other than to add more processors if possible and comparing timing files between your two cases to see where they are differing significantly.