Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Slow CAM initialization on derecho?

R

raeder

Member
Since derecho's system maintenance on May 7 my 6-hour CAM runs are taking 3-5 times longer.
Most of that seems to be in the initialization; I measured one case that took 11 minutes before the time stepping started.
compared to 1-2 minutes before the maintenance.
The atm.log file doesn't change for ~3.5 minutes after the line
open_trc_datafile: /glade/campaign/cesm/cesmdata/inputdata/atm/cam/ozone_strataero/ozone_strataero_WACCM_L70_zm5day_18500101-20150103_CMIP6ensAvg_c180923.nc
is printed.
Have you noticed anything like this? Which CESM version are you using?
I'll ask the CISL consultants about it after I have an idea whether it's my problem or our problem.
Kevin
 
F

fischer

CSEG and Liaisons
Staff member
Hi Kevin,

Did you notice if you were experiencing this slow down before the system maintenance?. I've been experiencing really slow
disk access on derecho for the last couple of weeks. I haven't had a chance to test since derecho came back up.

Chris
 
R

raeder

Member
Definitely not as of Apr 25, but now that I dig deeper I see a 6 hour forecast on May 1 that took 14 minutes,
so that falls in the "slow" group. That was before the 5/7 maintenance, so I guess I can't blame that.
 
R

raeder

Member
Mea Trahan (CISL consultant) responded to my issue report with
It seems the file was automatically compressed through our compression policy. You can undo this with:


/usr/lpp/mmfs/bin/mmchattr --compression=no <your-file>


Please let us know if you have a list of files that you would like to exclude in this policy, and if not then hope this will resolve your issue.


More information on file compression can be found here:
NCAR HPC Documentation - Campaign storage file system
File compression
Click to expand...

I pointed out that I don't own the files, so I probably can't permanently decompress them,
and that there's a large community who uses them, so it seems that it would be more efficient
to store them uncompressed than save the disk space and make everyone's jobs wait minutes
to read the files.
 
R

raeder

Member
Mea and Gary Strand are arranging for the files in /glade/campaign/cesm/cesmdata/inputdata to be decompressed.
It may take a few days. I'll test it then to see if this problem is fixed.
 
You must log in or register to reply here.
Top