Why my simulation speed is so slow? and how to speed up?

[wfc1102@163_com]

The similation is as slow as "0.24 simulated_years/day". The pe_layout can also be found in the cesm_timing log. The NTASKS and ROOTPE are automatically created.
Any suggestions are welcome, thanks ahead.

./create_newcase --compset BHIST --res f09_g17 --mach pi2 --case BHIST
The machine is a user-defined one.

---------------- TIMING PROFILE ---------------------
Case : BHIST2
LID : 32274966.240404-210421
Machine : pi2
Caseroot : ~/cesm2.2.2/cime/scripts/BHIST2
Timeroot : ~/cesm2.2.2/cime/scripts/BHIST2/Tools
User : ioolhl-fc
Curr Date : Fri Apr 5 13:35:58 2024
grid : a%0.9x1.25_l%0.9x1.25_oi%gx1v7_r%r05_g%gland4_w%ww3a_z%null_m%gx1v7
compset : HIST_CAM60_CLM50%BGC-CROP_CICE_POP2%ECO_MOSART_CISM2%NOEVOLVE_WW3_SIAC_SESP_BGC%BDRD
run type : startup, continue_run = FALSE (inittype = TRUE)
stop option : nmonths, stop_n = 2
run length : 59 days (58.958333333333336 for ocean)

component comp_pes root_pe tasks x threads instances (stride)
--------- ------ ------- ------ ------ --------- ------
cpl = cpl 320 0 320 x 1 1 (1 )
atm = cam 320 0 320 x 1 1 (1 )
lnd = clm 160 0 160 x 1 1 (1 )
ice = cice 160 160 160 x 1 1 (1 )
ocn = pop 40 320 40 x 1 1 (1 )
rof = mosart 160 0 160 x 1 1 (1 )
glc = cism 320 0 320 x 1 1 (1 )
wav = ww 80 0 80 x 1 1 (1 )
iac = siac 1 0 1 x 1 1 (1 )
esp = sesp 1 0 1 x 1 1 (1 )

total pes active : 360
mpi tasks per node : 40
pe count for cost estimate : 360

Overall Metrics:
Model Cost: 36238.00 pe-hrs/simulated_year
Model Throughput: 0.24 simulated_years/day

Init Time : 908.126 seconds
Run Time : 58576.493 seconds 992.822 seconds/day
Final Time : 0.130 seconds

Actual Ocn Init Wait Time : 299.460 seconds
Estimated Ocn Init Run Time : 41.174 seconds
Estimated Run Time Correction : 0.000 seconds
(This correction has been applied to the ocean and total run times)

Runs Time in total seconds, seconds/model-day, and model-years/wall-day
CPL Run Time represents time in CPL pes alone, not including time associated with data exchange with other components

TOT Run Time: 58576.493 seconds 992.822 seconds/mday 0.24 myears/wday
CPL Run Time: 138.746 seconds 2.352 seconds/mday 100.66 myears/wday
ATM Run Time: 18246.423 seconds 309.261 seconds/mday 0.77 myears/wday
LND Run Time: 4427.830 seconds 75.048 seconds/mday 3.15 myears/wday
ICE Run Time: 996.873 seconds 16.896 seconds/mday 14.01 myears/wday
OCN Run Time: 58302.177 seconds 988.172 seconds/mday 0.24 myears/wday
ROF Run Time: 207.054 seconds 3.509 seconds/mday 67.45 myears/wday
GLC Run Time: 2.973 seconds 0.050 seconds/mday 4697.62 myears/wday
WAV Run Time: 1134.853 seconds 19.235 seconds/mday 12.31 myears/wday
IAC Run Time: 0.000 seconds 0.000 seconds/mday 0.00 myears/wday
ESP Run Time: 0.000 seconds 0.000 seconds/mday 0.00 myears/wday
CPL COMM Time: 40215.824 seconds 681.624 seconds/mday 0.35 myears/wday
NOTE: min:max driver timers (seconds/day):

 

[wfc1102@163_com]

Attachments may be useful. userdefined machines can be found by MACH="pi2" in three config_* files

The loaded modules are listed as below.

Currently Loaded Modules:
1) miniconda3/22.11.1
2) intel-oneapi-compilers/2021.4.0
3) intel-oneapi-ipp/2021.4.0
4) intel-oneapi-mkl/2021.4.0
5) intel-oneapi-tbb/2021.4.0
6) oneapi/2021.4.0
7) openblas/0.3.7
8) zlib/1.2.13-intel-2021.4.0
9) hdf5/1.14.1-2-intel-2021.4.0
10) parallel-netcdf/1.12.3-intel-2021.4.0
11) netcdf-c/4.9.2-intel-2021.4.0
12) netcdf-fortran/4.5.2-pnetcdf-intel-2021.4.0
13) intel-oneapi-mpi/2021.10.0