Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

modifying batch settings: CESM1 versus CESM2

K

kyle.clem

Kyle Clem
New Member
Hello,

I've successfully ported and built a CESM2.1.1 case on my local cluster and would like to run it (well, submit it). I am familiar with CESM1.2 where all the batch settings could be added/modified to the case.run script, and I could specify the batch submit type with ./xmlchange BATCHSUBMIT=sbatch.

For CESM2.1.1, there is no case.run script, there is no variable BATCHSUBMIT, but moreover, I don't see all the variables I would need to modify in env_batch.xml. Can anybody tell me how I would go about setting up a slurm job for CESM2.1.1?

For example, setting these types of options:

sbatch
srun
#SBATCH --partition=main
#SBATCH --requeue
#SBATCH --job-name=test
#SBATCH --nodes=1
#SBATCH --ntasks=24
#SBATCH --cpus-per-task=1
#SBATCH --mem=124000
#SBATCH --time=01:00:00
#SBATCH --output=/scratch/krc139/cesm1_2_2/cases/test.1/ascii_output/slurm.%N.%j.out
#SBATCH --error=/scratch/krc139/cesm1_2_2/cases/test.1/ascii_output/slurm.%N.%j.err
#SBATCH --export=ALL
 
J

jedwards

CSEG and Liaisons
Staff member
There are several slurm based machines defined in cime/config/cesm/machines look in config_batch.xml and config_machines.xml for examples.
 
K

kyle.clem

Kyle Clem
New Member
Thanks jedwards! But isn't cime/config/cesm/machines/config_batch.xml called during case.setup? How would I modify these batch settings after the case is already built? While it looks like I can set what I need in cime/config/cesm/machines/config_batch.xml, from what I can tell the changeable variables with xmlchange are more limited. Say, for instance, I wanted to play around with the number of nodes, processors, and run time after a case is already built...can that be doen with xmlchange?

Thanks,
Kyle
 
J

jedwards

CSEG and Liaisons
Staff member
Yes, change the number of tasks with
./xmlchange NTASKS_CMP=xxx (where CMP is the component: ATM, LND, etc) or the shortcut to change them all ./xmlchange NTASKS=xxx
Same with NTHRDS and ROOTPE.
when you change any of these you run
./case.setup --reset
./case.build
The case.setup step regenerates the batch scripts. You can view the submit arguments with the command ./preview_run

You can change the wallclock time and queue with JOB_WALLCLOCK_TIME= and JOB_QUEUE=
all of these can be done with xmlchange.
 
K

kyle.clem

Kyle Clem
New Member
Just to double check, running
./case.setup --reset
./case.build
would not mess with the archived and start/restart files, correct? This way I could change the batch settings, but continue a case run where it left off.
Thanks,
Kyle
 
K

kyle.clem

Kyle Clem
New Member
Hi jedwards, other CESM users,

I'm not sure if you are still following this, but do you know where the case.run script is located? Something strange is happening when I envoke case.submit. The run job, and the short term archive job, both submit successfully (as shown below), but the run job doesn't run. It disappears from my queue and a run log is never even generated. Then of course the archive job is left hanging because it is dependent on the run job.

Because I can't find the case.run script, I can't figure out what is wrong. What surprises me is that I do see the case.st_archive script, but I don't see the case.run script anywhere...any idea where this may be or what might be happening?

Thank you,
Kyle

Setting resource.RLIMIT_STACK to -1 from (8388608, -1)
Creating component namelists
Calling /nfs/home/clemky/my_cesm_sandbox/components/cam//cime_config/buildnml
CAM namelist copy: file1 /nfs/home/clemky/CESM2.1.1/cases/F1850_f19_f19/Buildconf/camconf/atm_in file2 /nfs/scratch/clemky/cesm/output/F1850_f19_f19/run/atm_in
Calling /nfs/home/clemky/my_cesm_sandbox/components/clm//cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/components/cice//cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/cime/src/components/data_comps/docn/cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/components/mosart//cime_config/buildnml
Running /nfs/home/clemky/my_cesm_sandbox/components/cism//cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/cime/src/components/stub_comps/swav/cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/cime/src/components/stub_comps/sesp/cime_config/buildnml
Calling /nfs/home/clemky/my_cesm_sandbox/cime/src/drivers/mct/cime_config/buildnml
NOTE: ignoring setting of rof2ocn_liq_rmapname=idmap in seq_maps.rc
NOTE: ignoring setting of rof2ocn_ice_rmapname=idmap in seq_maps.rc
Finished creating component namelists
Checking that inputdata is available as part of case submission
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Loading input file list: 'Buildconf/clm.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/cism.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Check case OK
submit_jobs case.run
Submit job case.run
Submitting job script sbatch .case.run --resubmit
Submitted job id is 324367
Submit job case.st_archive
Submitting job script sbatch --dependency=afterok:324367 case.st_archive --resubmit
Submitted job id is 324368
Submitted job case.run with id 324367
Submitted job case.st_archive with id 324368
 
J

jedwards

CSEG and Liaisons
Staff member
Is there a cesm.log.* file in your run directory? Is there a slurm output and/or error file in your case directory?
To see the commands that will be run you should use ./preview_run. The actual script is .case.run
You really should follow the porting and testing instructions before you try to run a case, many of these errors would be caught earlier in smaller more targeted tests.
 
K

kyle.clem

Kyle Clem
New Member
Dear jedwards,
No, a cesm.log file is never generated, nor output or error files. So it appears the job didn't run at all. A few days ago following building CESM successfully and it did run (though I had not set up the batch settings in config/cesm/machines/config_batch.xml), but crashed with an error. I suspected this was because I had not set up the batch settings. So I set my batch settings in config_batch.xml following the examples there, and now it does not run at all. So this leads me to believe the batch settings are wrong.

My machine uses Slurm, so I set up the following settings below in config_batch.xml. This is designed to use two processors per node across 24 nodes, so a total of 24 processors.

<batch_system MACH="raapoi_clemky" type="slurm">
<batch_submit>sbatch</batch_submit>
<directives>
<directive>--partition=parallel</directive>
<directive>--requeue</directive>
<directive>--job-name=BTEST</directive>
<directive>--nodes=12</directive>
<directive>--ntasks=24</directive>
<directive>--cpus-per-task=1</directive>
<directive>--mem-per-cpu=5G</directive>
<directive>--time=01:00:00</directive>
<directive>--output=/nfs/scratch/$ENV{USER}/cesm/output/ascii_ouput/$CASE.out</directive>
<directive>--error=/nfs/scratch/$ENV{USER}/cesm/output/ascii_ouput/$CASE.err</directive>
</directives>
<queues>
<queue walltimemin="00:00:00" walltimemax="72:00:00" nodemin="1" nodemax="24" default="true">lr3</queue
>
</queues>
</batch_system>

I also manually set these settings in the case directory

./xmlchange BATCH_SYSTEM=slurm,walltime=01:00:00,NTASKS=24,MAX_MPITASKS_PER_NODE=2,MAX_TASKS_PER_NODE=2

The problem lies somewhere in here, I am guessing...
 
J

jedwards

CSEG and Liaisons
Staff member
Did you try ./preview_run? Did you try any of the suggested porting tests? The way you are setting walltime in the above is incorrect, it should be set using the JOB_WALLCLOCK_TIME variable. When you set these lines:
<directive>--output=/nfs/scratch/$ENV{USER}/cesm/output/ascii_ouput/$CASE.out</directive>
<directive>--error=/nfs/scratch/$ENV{USER}/cesm/output/ascii_ouput/$CASE.err</directive>
did you create a directory /nfs/scratch/$ENV{USER}/cesm/output/ascii_ouput/ ? Usually I would not provide a path in these variables and the output would go to the case directory. If you have a non-existent path it could explain why you don't see these files.
 
K

kyle.clem

Kyle Clem
New Member
Hmmm. I did not create the ascii_output directories. I will remove those paths. Here is the output ./preview_run for the F compset I am currently trying to run (I am trying a smaller job using 12 processors on one node):

CASE INFO:
nodes: 1
total tasks: 12
tasks per node: 12
thread count: 1

BATCH INFO:
FOR JOB: case.run
ENV:
module command is /usr/bin/modulecmd python purge
module command is /usr/bin/modulecmd python load config cmake gcc/8.2.0 openmpi/3.1.5
Setting Environment OMP_STACKSIZE=256M
Setting Environment OMP_NUM_THREADS=1
SUBMIT CMD:
sbatch .case.run --resubmit

FOR JOB: case.st_archive
ENV:
module command is /usr/bin/modulecmd python purge
module command is /usr/bin/modulecmd python load config cmake gcc/8.2.0 openmpi/3.1.5
Setting Environment OMP_STACKSIZE=256M
Setting Environment OMP_NUM_THREADS=1
SUBMIT CMD:
sbatch --dependency=afterok:0 case.st_archive --resubmit

MPIRUN:
salloc -n 12 --time=120:00:00 -p parallel mpirun -np 12 /nfs/scratch/clemky/cesm/output/F1850_f19_f19/bld/cesm.exe >> cesm.log.$LID 2>&1
 
K

kyle.clem

Kyle Clem
New Member
The JOB_WALLCLOCK_TIME settings seem okay:

clemky@c03n01:~/CESM2.1.1/cases/F1850_f19_f19$ ./xmlquery -p WALL

Results in group case.run
JOB_WALLCLOCK_TIME: 72:00:00
USER_REQUESTED_WALLTIME:

Results in group case.st_archive
JOB_WALLCLOCK_TIME: 0:20:00
USER_REQUESTED_WALLTIME:
 
J

jedwards

CSEG and Liaisons
Staff member
I was refering to this setting from your earlier post
./xmlchange BATCH_SYSTEM=slurm,walltime=01:00:00,
It's really hard to help if you don't follow instructions.
 
K

kyle.clem

Kyle Clem
New Member
So should I not mess with the walltime variable, i.e. ./xmlchange walltime=01:00:00, since the JOB_WALLCLOCK_TIME variables are okay?
 
K

kyle.clem

Kyle Clem
New Member
Well, I got it to run, I guess what I needed to add to the batch settings was:

<batch_directive>#SBATCH</batch_directive>

So now that the case is running, I have the same error that i reported last week when it ran.

From cesm.log.*

Opened existing file /nfs/scratch/clemky/cesm/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015_2deg/emissions-cmip6_so4_a1_anthro-ene_vertical_1750-2015_1.9x2.5_c20170616.nc 2162688
NetCDF: Variable not found
ERROR timeaddmonths(): MM out of range
ERROR timeaddmonths(): MM out of range
set_time_float_from_date: error return from ESMF_TimeSet for set_time_float_from_date
ERROR: CHKRC

From atm.log.*

open_trc_datafile: /nfs/scratch/clemky/cesm/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015_2deg/emissions-cmip6_so4_a1_anthro-ene_vertical_1750-2015_1.9x2.5_c20170616.nc
set_time_float_from_date: error return from ESMF_TimeSet for set_time_float_from_date
ERROR: CHKRC
 
J

jedwards

CSEG and Liaisons
Staff member
Please provide details to reproduce this new problem, you should probably start a new thread and close this one since it's a different issue.
<batch_directive>#SBATCH</batch_directive> is the default for slurm systems - you should not have needed to add this.
 
K

kyle.clem

Kyle Clem
New Member
Hm, I did not know that. Then it must have been getting rid of the output and error paths, which I had not created. I made a silly troubleshooting mistake by changing both the <batch_directive>#SBATCH</batch_directive> and the output/error paths instead of one at a time.

I opened a thread for my current ESMF_TimeSet error last week and it was passed to the CAM-Chem forum (https://bb.cgd.ucar.edu/cesm/threads/error-timeaddmonths-mm-out-of-range.5152/#post-35892), but I haven't heard anything more on that.

The last I was asked was if USE_ESMF_LIB in env_build.xml is set to TRUE. Mine is set to FALSE (the default) because I don't have a ESMF library specified.

After this I will keep my queries on this error to the other thread.
 
K

kyle.clem

Kyle Clem
New Member
Hi jedwards, all,

I have another query regarding tasks/processors. I would like to run the B1850 compset job on 32 processors. I have UP TO 24 processors per node on my cluster (which I specify with MAX_TASKS_PER_NODE and MAX_MPITASKS_PER_NODE). However, I don't want CESM to only look for nodes with 24 processors available, I simply want it to find the total number of processors/tasks I specify wherever they are available, across multiple nodes. Therefore I do not set how many processors per node in my batch settings.

To run on 32 processors, in my sbatch settings I set --ntasks=32, and before invoking ./case.setup I set ./xmlchange NTASKS=32.

I am therefore expecting the total number of tasks to be 32. However, as shown below, ./preview_run shows me total tasks equal 128, and it will need 6 nodes/24 tasks per node. Why are there 128 tasks instead of 32? And are "nodes" and "tasks per node" hard defined so that the job will only run once is has 6 nodes with a full 24 processors available on each node?

CASE INFO:
nodes: 6
total tasks: 128
tasks per node: 24
thread count: 1
 
K

kyle.clem

Kyle Clem
New Member
Even if I change ./xmlchange NTASKS=2 it still requires 98 total tasks (and 5 nodes because MAX_TASKS_PER_NODE and MAX_MPITASKS_PER_NODE are both 24). If I set NTASKS=1 it still requires 97 total tasks.

So I am wondering 1) if B1850 compset requires a minimum of 97/98 tasks? And 2) if CESM comes up with its own node configuration based on the compset, NTASKS, and the MAX_TASKS_PER_NODE / MAX_MPITASKS_PER_NODE values, and then the resources I allocate from my cluster MUST match the specific node/processor configuration CESM came up with. This seems quite limiting as the amount of nodes CESM comes up with is based on the max tasks per node, therefore using only empty nodes (e.g., if there are 96 total tasks and the maximum number of tasks I can get per node is 24, CESM says I need strictly 4 nodes and 96 processors, which means it needs 4 empty nodes before it will ever run).

This takes significantly more time for my cluster to find the resources. For example, I have 100s of processors available across all of the nodes right now. However there are only a couple nodes that are empty/idling with all 24 processors available, and often times there aren't any completely empty nodes. I am wondering if it is possible to set a certain number of processors/tasks, say 100 as an example, with no number of nodes specified. Just find 100 processors. But when I change NTASKS and the total number of tasks changes, CESM simultaneously changes the number of nodes it needs based on the max values. I will also say that I don't necessarily want to change the MAX_TASKS_PER_NODE and MAX_MPITASKS_PER_NODE values because I still want to max out empty nodes when available.

I hope this makes sense, and would appreciate any feedback you could provide.
Thanks,
Kyle
 
You must log in or register to reply here.
Top