Simulation Successful: Timing File Missing

jbuzan_vub

Jonathan R. Buzan
New Member
Please fill in all relevant information below, deleting the red text after you have read it.

Before submitting a help request, please check to see if your question is already answered:
- Search the forums for similar issues
- Check the CIME troubleshooting guide to see if any suggestions there solve your problem
- Check any other relevant CESM documentation



What version of the code are you using?
- CESM staff members will mainly provide answers for supported model versions, as outlined in the CESM support policy, and can only provide limited help for versions that are no longer supported. You may ask questions about unsupported versions, but may need to rely on community answers.
- For CESM2.1.2 onwards run the script ./describe_version from the top level of your CESM clone to find the version
- From older model versions, provide the output from running the following commands from the top level of your CESM clone
> git describe
> ./manage_externals/checkout_externals --status --verbose


ctsm5.3.044

Have you made any changes to files in the source tree?
- Describe any changes (code, xml files, etc.)

Source code changes that add WBGT and additional variables to the HumanIndexMod.




Describe every step you took leading up to the problem:
- Describe every step you took, starting with the create_newcase command and including any changes you made to xml files, user_nl files, etc. Please try to reproduce the problem first using your own instructions.

./create_newcase --case /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/cases/ctsm5.3.044/i.t530.IHistClm50BgcCrop.hcru_hcru_mt13.obsclim-histsoc-default-20CRv3-ERA5-go-2048.010_64pfts_nodrops --res hcru_hcru_mt13 --compset IHistClm50BgcCrop --mach hortense --project '2022_201' --run-unsupported

Input data is modified to match ISIMIP standardization.

I get consistent PIO errors:
cesm log PIO2 pio_file.c retry NETCDF.

We use:
ESMF/8.6.0-foss-2023a
ESMF_PIO=internal.

My guess is the PIO errors are likely related to the outdated version of PIO internal to ESMF 8.6.0. The simulation seems to run slowly. My last simulation was 5 consecutive years at 2048 cores, 2x 64-core AMD Epyc 7763 CPU 2.45 GHz (“Milan” microarchitecture, 128 cores per node) 238 GiB usable RAM (~1.85GB/core), no swap 480 GB SSD local disk.

We are producing ~100Gb of land data per year, largest file is 53Gb. Restart directory is ~112Gb. It took about 17 hours for the 5 years.


If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.

- If the error occurs during the build, please attach the appropriate build log file showing the compilation error message.
- If the error occurs during the run, please attach all log files from the run (cpl.log, cesm.log and all component log files).



Describe your problem or question:


I do not understand why the timing file is not generated. I tried to set

<entry id="SAVE_TIMING" value="TRUE">

yet the timing files are still not written.
 
Solution
@jbuzan_vub the XML variable to use is:

CHECK_TIMING

rather than SAVE_TIMIING which only means to save any timing files in the RUNDIR as well as under the case.

I often forget the details of how these things works, so I do pattern matching to find the list of applicable variables

Bash:
./xmlquery -p TIM

and then use the "--full" option to get the details on how it works and what the list of options are:

Bash:
./xmlquery --full CHECK_TIMING

Or can look over the env_run.xml file or all of the env_*.xml files to do something similar...

erik

Erik Kluzek
CSEG and Liaisons
Staff member
@jbuzan_vub the XML variable to use is:

CHECK_TIMING

rather than SAVE_TIMIING which only means to save any timing files in the RUNDIR as well as under the case.

I often forget the details of how these things works, so I do pattern matching to find the list of applicable variables

Bash:
./xmlquery -p TIM

and then use the "--full" option to get the details on how it works and what the list of options are:

Bash:
./xmlquery --full CHECK_TIMING

Or can look over the env_run.xml file or all of the env_*.xml files to do something similar...
 
Vote Upvote 1 Downvote
Solution

jbuzan_vub

Jonathan R. Buzan
New Member
Hi Erik and Sam,

Thanks for your replies.
I did some investigations. I have an 'out-of-the-box' simulation (except for an update to the meganfactors file):

./create_newcase --case /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan//cases/ctsm5.3.044/I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20 --res hcru_hcru_mt13 --compset IHistClm50BgcCrop --mach hortense --project '2022_201' --run-unsupported
---------------------------------------------------
2026-04-15 10:44:23: Compset longname is HIST_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV_SESP
---------------------------------------------------
2026-04-15 10:44:23: Compset specification file is /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime_config/config_compsets.xml
---------------------------------------------------
2026-04-15 10:44:23: Pes specification file is /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime_config/config_pes.xml
---------------------------------------------------
2026-04-15 10:44:23: Forcing is Historic transient
---------------------------------------------------
2026-04-15 10:44:23: Using 1 coupler instances
---------------------------------------------------
2026-04-15 10:44:23: Component ATM is Data driven ATM GSWP3v1 data set
---------------------------------------------------
2026-04-15 10:44:23: ATM_GRID is 360x720cru
---------------------------------------------------
2026-04-15 10:44:23: Component LND is clm5.0:BGC (vert. resol. CN and methane) with prognostic crop:
---------------------------------------------------
2026-04-15 10:44:23: LND_GRID is 360x720cru
---------------------------------------------------
2026-04-15 10:44:23: Component ROF is MOSART: MOdel for Scale Adaptive River Transport
---------------------------------------------------
2026-04-15 10:44:23: ROF_GRID is r05
---------------------------------------------------
2026-04-15 10:44:23: ESP_GRID is None
---------------------------------------------------
with the run.CASE file:
cat run.I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Generating namelists for /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/cases/ctsm5.3.044/I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20
2026-04-15 11:42:54 atm
Create namelist for component datm
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/components/cdeps/datm/cime_config/buildnml
2026-04-15 11:42:54 lnd
Create namelist for component clm
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime_config/buildnml
2026-04-15 11:42:57 ice
Create namelist for component sice
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime/CIME/non_py/src/components/stub_comps_nuopc/sice/cime_config/buildnml
2026-04-15 11:42:57 ocn
Create namelist for component socn
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime/CIME/non_py/src/components/stub_comps_nuopc/socn/cime_config/buildnml
2026-04-15 11:42:57 rof
Create namelist for component mosart
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/components/mosart//cime_config/buildnml
2026-04-15 11:42:57 glc
Create namelist for component sglc
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
2026-04-15 11:42:57 wav
Create namelist for component swav
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
2026-04-15 11:42:57 esp
Create namelist for component sesp
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
2026-04-15 11:42:57 cpl
Create namelist for component drv
Calling /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/source/ctsm5.3.044/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ROF']
-------------------------------------------------------------------------
- Prestage required restarts into /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan//output/ctsm5.3.044/I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20/run
- Case input data directory (DIN_LOC_ROOT) is /dodrio/scratch/projects/2022_200/project_input/cesm/inputdata
- Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
run command is mympirun /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan//output/ctsm5.3.044/I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20/bld/cesm.exe >> cesm.log.$LID 2>&1
No timing file found in run directory
check for resubmit
dout_s True
mach hortense
resubmit_num 0

Where the CHECK_TIMING is set to true:
$ ./xmlquery --full CHECK_TIMING

Results in group run_flags
CHECK_TIMING: value=TRUE
type: l
valid_values: ['FALSE', 'TRUE']
description: logical to diagnose model timing at the end of the run
file: /dodrio/scratch/projects/2022_200/project_output/bclimate/jbuzan/cases/ctsm5.3.044/I_t530_IHistClm50BgcCrop_hcru_hcru_mt13__test_20/env_run.xml

The consistent thing between all the cases is: No timing file found in run directory

In other simulations i tried turning on SAVE_TIMING in addition to CHECK_TIMING being TRUE. But all the simulations are consistent with the:
No timing file found in run directory
 
Vote Upvote 0 Downvote

jbuzan_vub

Jonathan R. Buzan
New Member
Is it possible that the PIO configuration is the reason for the timing file not being created?
The machine i am working on has ESMF8.6.0 with ESMF_PIO=internal, which is PIO version 2.5, with known issues on Lustre systems?
 
Vote Upvote 0 Downvote
Back
Top