CLM tool modify_input_files takes too long

[Youwei]

Hi,

I am using a CLM tool, modify_input_files, to generate an idealized surface data file in fv0.23x0.31 grid in CESM tag cesm2_3_beta17.
The fsurdat file in fv0.23 is 23GB (compared to 923MB in fv0.9), and running ./fsurdat_modifier always exceeds walltime.

I have tried on
Derecho 1 node, 128 CPUs, 12 hours (job script below)
and
Casper 4 CPUs, 24 hours

Both runs failed to finish. Is there any suggestion for how I can accelerate this?


#!/bin/bash

#PBS -N fsurdat_modifier
#PBS -l walltime=12:00:00
#PBS -q main
#PBS -l select=1:ncpus=128:mpiprocs=128

module load conda
conda activate ctsm_pylib
./fsurdat_modifier modify_fsurdat_f02_sx066rv1.cfg

 

[slevis]

@Youwei I will move this to the "infrastructure" Forum. Also, did you email help@ucar.edu? HPC experts on the help team will likely have suggestions on how to best use derecho/casper resources for this case.

 

[Youwei]

Thanks! I emailed help@ucar.edu and they suggested me post it on the forum.

 

[Youwei]

Here is some updates. I test this code on Derecho using 1,2,4,and 6 nodes and each run requests 12 hour walltime. I also test it on Casper using 4 and 36 cpu with 256GB memory and request 24 hour walltime. All these job will exceed requested walltime. However, I run the same code on our local cluster. It only takes 10 minutes to finish the job. Although, the reason is unclear but this problem is solved.