Using different queues for case.run and case.st_archive

[ceppi]

Dear all,

The system on which I run CESM2.1.3 uses Slurm and has two queues: "standard" and "short", the latter being suited for small jobs shorter than 20 minutes.

1) Is there a way to configure CESM2 (presumably using config_batch.xml) so that any jobs with walltime ≤ 20 minutes are automatically submitted to the short queue, rather than the standard one (without having to manually edit the case.st_archive file)?

2) Additionally, for jobs submitted to the short queue, this system requires an extra sbatch option, "--reservation=shortqos". Again, I wondered if there's way to set up config_batch.xml so that any jobs submitted to the short queue will automatically include that option.

My attempt at making these changes in config_batch.xml is copied below, but I was unsuccessful: short jobs (such as case.st_archive, or the scripts_regression tests) are submitted to the standard queue by default; and if change the $JOB_QUEUE variable in env_workflow.xml to specify the short queue, then the extra sbatch option isn't included and the submission fails.

Any help appreciated – thanks.

 <batch_system MACH="archer2" type="slurm" >
    <batch_submit>sbatch</batch_submit>
    <submit_args>
      <arg flag="--time" name="$JOB_WALLCLOCK_TIME"/>
      <arg flag="-q" name="$JOB_QUEUE"/>
      <arg flag="--account" name="$PROJECT"/>
    </submit_args>

    <directives queue="standard">
      <directive>--partition=standard</directive>
      <directive>--qos=standard</directive>
      <directive>--export=none</directive>
    </directives>
    <directives queue="short">
      <directive>--partition=standard</directive>
      <directive>--qos=short</directive>
      <directive>--reservation=shortqos</directive>
      <directive>--export=none</directive>
    </directives>

    <queues>
      <queue walltimemax="24:00:00" nodemin="1" nodemax="2712" default="true" >standard</queue>
      <queue walltimemax="00:20:00" nodemin="1" nodemax="4" >short</queue>
    </queues>
  </batch_system>

 

[jedwards]

If you look at cheyenne as an example we do:
<queue walltimemax="12:00:00" nodemin="1" nodemax="4032">regular</queue>
<queue default="true" walltimemax="06:00:00" jobmin="1" jobmax="18">share</queue>
so that jobs using less than 1/2 a node will run on the share queue. You can do this by moving the default="true" from standard to
short and either setting nodemin=5 for standard or nodemax=1 for short.

 

[ceppi]

Thanks for your very quick reply. I tried what you suggested, but now both case.run and case.st_archive are being assigned to the short queue, this despite the case.run job having a walltime of 6h (and the number of nodes requested > nodemax for the short queue).

 

[jedwards]

Sorry that isn't working for you. I guess I don't have an easy solution. I've opened an issue to address this and make it easier

Easier split of case jobs into different queues · Issue #4120 · ESMCI/cime

The system on which I (the user) run CESM2.1.3 (archer2) uses Slurm and has two queues: "standard" and "short", the latter being suited for small jobs shorter than 20 minutes. Is there a way to con...

github.com

 

[ceppi]

Thank you, appreciate it!

 

[m_mineter@ed_ac_uk]

I hope its right to add to this thread, not open a new one.... you will see why I do this
Archer2 has now gone live with the full system: a choice of 3 partitions and 6 queues.
Running jobs - ARCHER2 User Documentation

Running CESM 2.1.3

With .cime/confg_batch.xml specifying the serial queue as default (partition serial) I have the following:

• Specify "-q short --machine=archer2" in the create-newcase,
  • bad: the .case.run is generated with serial directives (serial is the default queue in .cime/config_batch.xml)
    • the default queue directives are applied to the model and the st_archive job.
  • good: the sbatch commands have the right -q options (I don't specify qos in the config_batch.xml)
    • -q short for the model
    • -q serial for the archive job

I'm working around it by modifying .cime/config/cesm/machines/template.case.run so after the CIME-set batchdirectives, these are overridden by SBATCH commands that work for the short and standard queues:

#!/usr/bin/env python
# Batch system directives
{{ batchdirectives }}
#hardcoded in template
#SBATCH --partition=standard
#SBATCH --exclusive
#SBATCH --export=NONE
# qos is set from -q in the create case: standard or long or....

Also in the same directory, I modified
config_batch.xml, to remove the exclusive directive (its not wanted in the serial queue)

I attach what I have for the .cime/config_batch.xml

a) I hope this might be useful to other Archer2 users browsing the forum
b) I ask the CESM specialists to tell me if I am missing something
c) that -q short does not lead to the correct directives seems to be an observation that is closely related to the github issue above.

 

[raeder]

This is also related, but not necessarily an answer to the queries above.
Feel free to move it, or ask me to move it, if it doesn't belong here.

On cheyenne the default queue for case.st_archive appears to be "regular" instead of "share",
which is a waste of 35 processors for the duration of the job.
I often see a dozen st_archive jobs running or waiting in non"share" queues.

I believe that this is happening, at least partly, because in
scripts/lib/CIME/XML/env_batch.py: def select_best_queue
the loop over the queues (listed in config/cesm/machines/config_batch.xml?)
stops as soon as it finds a queue which satisfies the spec.
Maybe it finds a sufficient queue (regular) rather than the best queue (share)
because regular is earlier in the list and is not excluded by queue_meets_spec.
But I haven't been able to parse all of the python and the data flow in the modules.

 

[jedwards]

@raeder I just tried this and see that the case.st_archive job will go to the share queue.
You can check this in a case without submitting using the preview_run tool. For my case I see:

 ./preview_run
CASE INFO:
  nodes: 1
  total tasks: 36
  tasks per node: 36
  thread count: 1
  ngpus per node: 0

BATCH INFO:
  FOR JOB: case.run
    ENV:
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment MPI_DSM_VERBOSE=true
      Setting Environment MPI_IB_CONGESTED=1
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_USE_ARRAY=None
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment OMP_WAIT_POLICY=PASSIVE
      Setting Environment TMPDIR=/glade/scratch/jedwards
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSINPUTPATH=/glade/work/turuncu/FV3GFS/benchmark-inputs/2012010100/gfs/fcst

    SUBMIT CMD:
      qsub -q regular -l walltime=12:00:00 -A P93300606 -v ARGS_FOR_SCRIPT='--resubmit' .case.run

    MPIRUN (job=case.run):
      mpiexec_mpt -p "%g:"  -np 36  omplace -tm open64 -vv /glade/scratch/jedwards/foofoo/bld/cesm.exe   >> cesm.log.$LID 2>&1

  FOR JOB: case.st_archive
    ENV:
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment MPI_DSM_VERBOSE=true
      Setting Environment MPI_IB_CONGESTED=1
      Setting Environment MPI_TYPE_DEPTH=16
      Setting Environment MPI_USE_ARRAY=None
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=1024M
      Setting Environment OMP_WAIT_POLICY=PASSIVE
      Setting Environment TMPDIR=/glade/scratch/jedwards
      Setting Environment TMPDIR=/glade/scratch/jedwards
      Setting Environment UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
      Setting Environment UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
      Setting Environment UGCSINPUTPATH=/glade/work/turuncu/FV3GFS/benchmark-inputs/2012010100/gfs/fcst

    SUBMIT CMD:
      qsub -q share -l walltime=00:20:00 -A P93300606  -W depend=afterok:0 -v ARGS_FOR_SCRIPT='--resubmit' case.st_archive

You can also see this by querying the JOB_QUEUE:

./xmlquery JOB_QUEUE

Results in group case.run
    JOB_QUEUE: regular

Results in group case.st_archive
    JOB_QUEUE: share

If you are seeing JOB_QUEUE=regular for case.st_archive you might try changing it manually in your case with

./xmlchange JOB_QUEUE=share --subgroup case.st_archive

 

[scottan]

Hello, I've been trying to set up to run CESM on our local supercomputer in Manchester (CSF3) which uses Slurm, and run into similar issues. This is the `config_batch.xml` I'm using:

<batch_system MACH="csf3_hpcpool" type="slurm">
    <batch_submit>sbatch</batch_submit>
    <submit_args>
      <arg flag="-p" name="$JOB_QUEUE"/>
      <arg flag="-t" name="$JOB_WALLCLOCK_TIME"/>
      <arg flag="-A" name="$PROJECT"/>
    </submit_args>
    <directives>
      <directive> -J {{ job_id }}</directive>
      <directive> --time {{ JOB_WALLCLOCK_TIME }} </directive>
      <directive> -o cesm.stdout.%j </directive>
      <directive> -e cesm.stderr.%j </directive>
      <directive> --nodes={{ num_nodes }}</directive>
      <directive> --ntasks-per-node={{ tasks_per_node }}</directive>
      <directive> --cpus-per-task={{ thread_count }}</directive>
    </directives>
    <directives queue="hpcpool">
      <directive> --partition=hpcpool</directive>
    </directives>
    <queues>
      <queue walltimemax="96:00:00" nodemin="4" nodemax="32">hpcpool</queue>
    </queues>
  </batch_system>

When I run create_newcase I can't work out how to control which resources the case.st_archive job uses:

job is case.run USER_REQUESTED_WALLTIME 03:00:00 USER_REQUESTED_QUEUE hpcpool WALLTIME_FORMAT %H:%M:%S
job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
WARNING: No queue on this system met the requirements for this job. Falling back to defaults

Which means the st_archive job ends up requesting 96 hours on 128 cores!

I tried creating a default serial queue based on the discussion above:

  <queues>
    <queue walltimemax="01:00:00" nodemin="1" nodemax="1" default="true">serial</queue>
    <queue walltimemax="96:00:00" nodemin="4" nodemax="32">hpcpool</queue>
  </queues>

But I couldn't get both this and the hpcpool queues to work at the same time - they seem to end up sharing the same number of tasks being requested so one of the jobs fails because I'm requesting too few / too many cores.

The best workaround I've found is to manually change the wallclocktime for the case.st_archive job before running case.submit:

./xmlchange JOB_WALLCLOCK_TIME=00:20:00 --subgroup case.st_archive

Which at least means it's not requesting too much time, even if it's still using more cores than it needs to.

Has anyone found a better way to control the queues and resources requested for the case.run and case.st_archive jobs individually?

 

[jedwards]

Can you post the results of ./preview_run?
For example on derecho I get:

./preview_run
CASE INFO:
  nodes: 4
  total tasks: 512
  tasks per node: 128
  thread count: 1
  ngpus per node: 0

BATCH INFO:
  FOR JOB: case.run
    ENV:
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
      Setting Environment FI_MR_CACHE_MONITOR=memhooks
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=64M

    SUBMIT CMD:
      qsub -q main -l walltime=12:00:00 -A P93300606 -l job_priority=regular -v ARGS_FOR_SCRIPT='--resubmit' /glade/work/jedwards/sandboxes/cesm3_0_alphabranch/cime/scripts/foo/.case.run

    MPIRUN (job=case.run):
      mpibind  --label  --line-buffer  --  /glade/derecho/scratch/jedwards/foo/bld/cesm.exe   >> cesm.log.$LID 2>&1

  FOR JOB: case.st_archive
    ENV:
      Setting Environment ESMF_RUNTIME_PROFILE=ON
      Setting Environment ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
      Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
      Setting Environment FI_MR_CACHE_MONITOR=memhooks
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_STACKSIZE=64M

    SUBMIT CMD:
      qsub -q develop -l walltime=00:20:00 -A P93300606 -l job_priority=regular  -W depend=afterok:0 -v ARGS_FOR_SCRIPT='--resubmit' /glade/work/jedwards/sandboxes/cesm3_0_alphabranch/cime/scripts/foo/case.st_archive

Here is the config_batch.xml file.

 

[scottan]

Hi jedwards,

Sorry for my slow reply, it was just after I went on Christmas break. We have a setup that works and we're happy with now. I'll share below in case it is useful for anyone else, but it would be nice if the settings for st_archive could be controlled in one of the config_*.xml files.

I'm running the setup with this slurm sbatch script. As you can see, I'm overwriting the JOB_QUEUE, CHARGE_ACCOUNT and PROJECT for the st_archive job with xmlchange:

#!/bin/bash  --login
#SBATCH -p serial  # Partition is required. Runs on an Intel hardware.
#SBATCH -t 0-1     # Wallclock limit (days-hours). Required!
                   # Max permitted is 7 days (7-0).
#SBATCH -n 1       # Causes the $SLURM_NTASKS env var to be set (to 1)

# These paths, case name and project need to be set:
export CASE_NAME="<case_name>"
# CASE_PATH - points to the CASE directory
export CASE_PATH="/<path_to_cases>/${CASE_NAME}"
export RUN_DIR="/<path_to_run_dir>/"
export BUILD_PATH="${RUN_DIR}/${CASE_NAME}/bld"
export RUN_PATH="${RUN_DIR}/${CASE_NAME}/run"
# PROJECT - project code to charge to
export PROJECT=<hpc-project-code>
export DEFAULT_ACCOUNT=$(sacctmgr show user $USER format=DefaultAccount -n -P)

# NNODES - number of nodes for each task to run on, 32 cores per node
# Increase to run on more cores
export NNODES=4
export NTASKS=$(($NNODES*32))

cd $CASE_PATH
./case.setup &> "${CASE_PATH}/setup.log"

# Make changes to configs to use correct partitions
./xmlchange JOB_QUEUE=serial,CHARGE_ACCOUNT=$DEFAULT_ACCOUNT,PROJECT=$DEFAULT_ACCOUNT --subgroup case.st_archive
./xmlchange JOB_QUEUE=hpcpool,PROJECT=$PROJECT,NTASKS=$NTASKS --subgroup case.run

# Setup and Build
./case.setup --reset &> "${CASE_PATH}/setup_reset.log"
wait
./case.build --clean-all
wait
./case.build &> "${CASE_PATH}/build.log"

This is the relevant bit of config_batch.xml:

  <batch_system MACH="csf3_hpcpool" type="slurm">
    <batch_submit>sbatch</batch_submit>
    <submit_args>
      <arg flag="-p" name="$JOB_QUEUE"/>
      <arg flag="-t" name="$JOB_WALLCLOCK_TIME"/>
      <arg flag="-A" name="$PROJECT"/>
    </submit_args>
    <directives>
      <directive> -J {{ job_id }}</directive>
      <directive> --time {{ JOB_WALLCLOCK_TIME }} </directive>
      <directive> -o cesm.stdout.%j </directive>
      <directive> -e cesm.stderr.%j </directive>
    </directives>
    <directives queue="serial">
      <directive> --partition=serial</directive>
    </directives>
    <directives queue="hpcpool">
      <directive> --partition=hpcpool</directive>
      <directive> --ntasks={{ total_tasks }}</directive>
    </directives>
    <queues>
      <queue walltimemax="01:00:00" nodemin="1" nodemax="1" default="true">serial</queue>
      <queue walltimemax="96:00:00" nodemin="4" nodemax="32">hpcpool</queue>
    </queues>
  </batch_system>

This is the output of ./preview_run:

CASE INFO:
  nodes: 4
  total tasks: 128
  tasks per node: 32
  thread count: 1

BATCH INFO:
  FOR JOB: case.run
    ENV:
      module command is /usr/bin/modulecmd python purge
      module command is /usr/bin/modulecmd python load apps/gcc/perl/5.42.0 apps/binapps/anaconda3/2023.09 compilers/gcc/8.2.0 tools/gcc/cmake/3.22.2--gcc8.2.0 libs/ucx/1.18.0-gcc8.2.0 libs/gcc/zlib/1.2.13 mpi/gcc/openmpi/4.1.8-gcc-8.2.0 libs/gcc/openssl/1.0.2k libs/gcc/lapack/3.12.1-gcc8.2.0 libs/gcc/hdf5/1.12.3_mpi4.1.8-gcc-8.2.0 libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0 libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0 libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0 libs/gcc/pio/2.6.2_mpi4.1.8-gcc-8.2.0 apps/gcc/esmf/8.8.1_mpi4.1.8-gcc-8.2.0
Setting Environment PERL5LIB=/opt/apps/apps/gcc/perl/5.42.0
      Setting Environment PERL5LIB=/opt/apps/apps/gcc/perl/5.42.0
Setting Environment OMP_NUM_THREADS={{ thread_count }}
      Setting Environment OMP_NUM_THREADS={{ thread_count }}
Setting Environment OMP_PLACES=cores
      Setting Environment OMP_PLACES=cores
Setting Environment OMP_STACKSIZE=256M
      Setting Environment OMP_STACKSIZE=256M
Setting Environment CESMDATAROOT=/mnt/eps01-rds/Zhonghua_Group/UCSD/
      Setting Environment CESMDATAROOT=/mnt/eps01-rds/Zhonghua_Group/UCSD/
Setting Environment NETCDF_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
Setting Environment NETCDF_C_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_C_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
Setting Environment NETCDF_FORTRAN_PATH=/opt/apps/libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_FORTRAN_PATH=/opt/apps/libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0
Setting Environment PNETCDF_PATH=/opt/apps/libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0
      Setting Environment PNETCDF_PATH=/opt/apps/libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0
      Setting Environment OMP_NUM_THREADS={{ thread_count }}

    SUBMIT CMD:
      sbatch -p hpcpool -t 03:00:00 -A hpc-zz-aerosol .case.run --resubmit

    MPIRUN (job=case.run):
      mpirun  /scratch/mbcx6sa2/runs/foo_hpcpool/bld/cesm.exe   >> cesm.log.$LID 2>&1

  FOR JOB: case.st_archive
    ENV:
      module command is /usr/bin/modulecmd python purge
      module command is /usr/bin/modulecmd python load apps/gcc/perl/5.42.0 apps/binapps/anaconda3/2023.09 compilers/gcc/8.2.0 tools/gcc/cmake/3.22.2--gcc8.2.0 libs/ucx/1.18.0-gcc8.2.0 libs/gcc/zlib/1.2.13 mpi/gcc/openmpi/4.1.8-gcc-8.2.0 libs/gcc/openssl/1.0.2k libs/gcc/lapack/3.12.1-gcc8.2.0 libs/gcc/hdf5/1.12.3_mpi4.1.8-gcc-8.2.0 libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0 libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0 libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0 libs/gcc/pio/2.6.2_mpi4.1.8-gcc-8.2.0 apps/gcc/esmf/8.8.1_mpi4.1.8-gcc-8.2.0
Setting Environment PERL5LIB=/opt/apps/apps/gcc/perl/5.42.0
      Setting Environment PERL5LIB=/opt/apps/apps/gcc/perl/5.42.0
Setting Environment OMP_NUM_THREADS={{ thread_count }}
      Setting Environment OMP_NUM_THREADS={{ thread_count }}
Setting Environment OMP_PLACES=cores
      Setting Environment OMP_PLACES=cores
Setting Environment OMP_STACKSIZE=256M
      Setting Environment OMP_STACKSIZE=256M
Setting Environment CESMDATAROOT=/mnt/eps01-rds/Zhonghua_Group/UCSD/
      Setting Environment CESMDATAROOT=/mnt/eps01-rds/Zhonghua_Group/UCSD/
Setting Environment NETCDF_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
Setting Environment NETCDF_C_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_C_PATH=/opt/apps/libs/gcc/netcdf-c/4.9.2_mpi4.1.8-gcc-8.2.0
Setting Environment NETCDF_FORTRAN_PATH=/opt/apps/libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0
      Setting Environment NETCDF_FORTRAN_PATH=/opt/apps/libs/gcc/netcdf-f/4.6.1_mpi4.1.8-gcc-8.2.0
Setting Environment PNETCDF_PATH=/opt/apps/libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0
      Setting Environment PNETCDF_PATH=/opt/apps/libs/gcc/pnetcdf/1.12.3_mpi4.1.8-gcc-8.2.0
      Setting Environment OMP_NUM_THREADS={{ thread_count }}

    SUBMIT CMD:
      sbatch -p serial -t 00:20:00 -A fatpou01  --dependency=afterok:0 case.st_archive --resubmit

 

[jedwards]

You should be able to adjust these fields via the config_workflow.xml file.
For cesm2.1.x this file can be found in cime/config/cesm/machines/config_workflow.xml
For cesm3.0.x it's in ccs_config/machines/config_workflow.xml
In there is a section for case.st_archive:
<job name="case.st_archive">
<template>template.st_archive</template>
<!-- If DOUT_S is true and case.run (or case.test) exits successfully then run st_archive-->
<dependency>case.run or case.test</dependency>
<prereq>$DOUT_S</prereq>
<runtime_parameters>
<task_count>1</task_count>
<tasks_per_node>1</tasks_per_node>
<walltime>0:20:00</walltime>
</runtime_parameters>
</job>

This should allow you to set the default settings you need.

 

[scottan]

Thank you for the reply, it's not letting me edit the fields I need to atm. If I add the following to config_workflow.xml:

      <runtime_parameters MACH="csf3_hpcpool">
        <JOB_QUEUE>serial</JOB_QUEUE>
        <CHARGE_ACCOUNT>fatpou01</CHARGE_ACCOUNT>
        <PROJECT>fatpou01</PROJECT>
      </runtime_parameters>

I get the error:

ERROR: Command: '/opt/apps/apps/binapps/anaconda3/2023.09/bin/xmllint --xinclude --noout --schema /mnt/iusers01/non-fac/dshub/mbcx6sa2/my_cesm_sandbox/cime/config/xml_schemas/config_workflow.xsd /mnt/iusers01/non-fac/dshub/mbcx6sa2/my_cesm_sandbox/cime/config/cesm/machines/config_workflow.xml' failed with error '/mnt/iusers01/non-fac/dshub/mbcx6sa2/my_cesm_sandbox/cime/config/cesm/machines/config_workflow.xml:51: element JOB_QUEUE: Schemas validity error: Element 'JOB_QUEUE': This element is not expected. Expected is one of ( task_count, tasks_per_node, walltime ).

Many thanks,
Scott

 

[jedwards]

You should use the fields as defined in config_workflow.xsd for runtime_parameters, these are:
<xs:element minOccurs="0" ref="task_count"/>
<xs:element minOccurs="0" ref="tasks_per_node"/>
<xs:element minOccurs="0" ref="walltime"/>

Properly setting these should cause the st_archive to choose the correct queue from
config_batch.xml