Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

branch_tags/PPE.n12_ctsm5.1.dev030 case runs indefinitely

A

aelkouk

ahmed elkouk
Member
Hello

I am using the PPE experiment to explore influence of model parameters. I setup a regional case using "branch_tags/PPE.n12_ctsm5.1.dev030", however the case keeps running indefinitely (until job is killed). I noticed that adding (hist_empty_htapes = .true. and hist_fincl2 with some vars) solves the issue. Also it seems that including "H2OSOI" in a daily "hist_fincl" leads to the same issue. Could you please check the log files attached and let me know what do you think ?

Thank you!
 

Attachments

  • atm.log.5184439.chadmin1.ib0.cheyenne.ucar.edu.220727-091911.txt
    85.2 KB · Views: 2
  • cesm.log.5184439.chadmin1.ib0.cheyenne.ucar.edu.220727-091911.txt
    58.1 KB · Views: 1
  • cpl.log.5184439.chadmin1.ib0.cheyenne.ucar.edu.220727-091911.txt
    46.4 KB · Views: 0
  • lnd.log.5184439.chadmin1.ib0.cheyenne.ucar.edu.220727-091911.txt
    99.8 KB · Views: 1
A

aelkouk

ahmed elkouk
Member
I should add that only adding hist_empty_htapes = .true. (no output), the model runs as expected. Without that it runs indefinitely.
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Is this your run:

/glade/work/oleson/ctsm_runs/PPEn12CTSM51_I2000Clm50Sp_nldas2_0.125UCRB_003.eval

If so, the corresponding run file (run.PPEn12CTSM51_I2000Clm50Sp_nldas2_0.125UCRB_003.eval.o5184439) indicates that you are running out of wallclock time:

=>> PBS: job killed: walltime 307 exceeded limit 300
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Ok, sorry, I see what you are saying. It's not closing the primary history file and finishing the run.
I don't see anything wrong with your settings. I'll take a closer look....
 
A

aelkouk

ahmed elkouk
Member
Yes please. I like to add that running a standard case e.g., "I1850Clm50Sp_f09_g17" does not seem to have this issue.
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
The problem may be related to this issue which has not been solved:

github.com

Sparse grid simulation with PPE.n12_ctsm5.1.dev030 fails when trying to close h5 file at end of simulation. · Issue #1449 · ESCOMP/CTSM

Brief summary of bug A sparse grid simulation that uses the PPE.n12_ctsm5.1.dev030 tag fails when trying to close the h5 file at the end of the simulation. General bug information CTSM version you ...
github.com github.com

The work-around is to split the higher-frequency output into more files. E.g, try splitting your h0 output into daily files (one file for each day). That seems to work for me.
 
You must log in or register to reply here.
Top