Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Changing history files affects runtime

Status
Not open for further replies.
A

aelkouk

ahmed elkouk
Member
Hi Keith

I have this issue which may be related to the one you helped me with earlier (
branch_tags/PPE.n12_ctsm5.1.dev030 case runs indefinitely). The model works when I save a couple of variables but runs forever when saving all variables. Could you please check when you can.

I have setup the following case to demonstrate the problem: /glade/work/elkoukah/cases/PPEn12CTSM51_I2000Clm50Sp_nldas2_0.125SWUS_001.debug

The run finishes when you uncomment the two lines user_nl_clm. I appreciate your help, thanks so much!
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I'm not sure what is going on.
I cloned your case and the model ran for three days, writing out daily average history files, but it took 2 hours.
I did a one day run and the model finished successfully but that took about 1 hour.
I upped the processors (30 cores instead of 15) but that didn't help.
Even when running with the default history output (monthly average history files) the model ran very slowly.
The timing file indicates that the model is spending all of its time in "clm_drv_io" which is the section of code that finalizes and writes the history files.
Do you have code mods of your own in this branch_tags/PPE.n12_ctsm5.1.dev030 tag?
 
A

aelkouk

ahmed elkouk
Member
Thank you very much for checking up this. No it is the original branch, I didn't do any code modifications.
Do you think the domain file has something to do with this? I have another case which is essentially a subset of that domain, and runs with no problems (/glade/work/elkoukah/cases/PPEn12CTSM51_I2000Clm50Sp_nldas2_0.125UCRB_001.eval).
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Maybe. I guess you could try substituting in your domain file and fsurdat from the eval case into the debug case and see if it is related to that.
 
A

aelkouk

ahmed elkouk
Member
Hi Keith, Thank you so much, I didn't think of this. Yes when I substitute the domain file and fsurdat from the eval case, the debug case does run as expected. I am still trying to find out what is the issue. I'll put an update here if I manage to figure it out.

Best wishes,
 
A

aelkouk

ahmed elkouk
Member
Hi Keith, I have tried multiple experiments (e.g., recreating the domain) unfortunately without success. Note worthy is that I substituted the default domain and fsurdat with the original ones defined in I2000Ctsm50NwpSpNldasGs compset into the debug case, and had the same run time issue. This I think means there should be no problem with the original domain and fsurdat I have been using. Could this be an unknow issue in "branch_tags/PPE.n12_ctsm5.1.dev030" ? Thank you!
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I don't think there is anything wrong with that tag. I had run a global SP simulation at some point and got about 650 years/day performance.
 
Status
Not open for further replies.
Top