CIME porting problems

[tcraig]

I am generating the following errors when I try create_newcase in a port of a recent version of CIME/CIME_CONFIG in E3SM (560611d1c67ee Feb 3, 2021). I am porting to a DOD machine called onyx, and this is what I have observed,
- I ported a version of E3SM from a couple years ago to onyx and that still works fine, even today.
- I am using the same machine settings in this version of E3SM.
- The current version of E3SM sets up a case fine on cori-haswell.
- When I try to setup a case on onyx with the cori-haswell setup, I get exactly the same error. This was just a test to see if the scripts would work.
All this suggests to me that some aspect of the software requirements in the current version of E3SM are not being met on onyx. But they are on cori and in an older version of E3SM on onyx. The error message I'm getting in not particularly insightful. Does anyone have any idea. Do I need a newer version of some xml software or something on onyx?

Traceback (most recent call last):
File "./create_newcase", line 243, in <module>
_main_func(__doc__)
File "./create_newcase", line 238, in _main_func
extra_machines_dir=extra_machines_dir)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/case/case.py", line 1664, in create
extra_machines_dir=extra_machines_dir)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/case/case.py", line 1125, in configure
extra_machines_dir=extra_machines_dir)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/XML/batch.py", line 32, in __init__
GenericXML.__init__(self, infile, schema=schema)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 64, in __init__
self.read(infile, schema)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 102, in read
self.read_fd(fd)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 126, in read_fd
self.tree = ET.parse(fd)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1659, in feed
self._raiseerror(v)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 292, column 115

 

[tcraig]

Just some other info. I have tried using python2 and python3 and the error persists. cori uses python2 fine.

 

[jedwards]

Tony,

This suggests that the xml file config_batch.xml has a syntax error. Did you just copy this file from the older version of the code?
At some point config_batch.xml was split into config_batch.xml and config_workflow.xml and your older version of the file may not be compatible with the
more recent code base. Did you run scripts_regression_tests.py on this system prior to trying to create a case? In most cases this should detect the error.

 

[tcraig]

Thanks @jedwards! How can you tell it's config_batch? I will look at the split between batch and workflow. Also, where is scripts_regression_test.py? Is there some documentation about how to use that tool? Thanks again.

 

[tcraig]

I found a cut/paste error in config_batch and that fixed things. Thanks again @jedwards. I think the challenge with these scripts/syntax is always finding the problems that arise or even having any idea where to look.

 

[jedwards]

Here is how I figured out the problem was in config_batch.xml:
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/Tools/../../scripts/lib/CIME/XML/batch.py", line 32, in __init__

scripts_regression_tests.py is in cime/scripts/tests/

and the users guide is in Case Control System Part 1: Basic Usage — CIME master documentation
this section in particular: 6. Porting and validating CIME on a new platform — CIME master documentation

 

[tcraig]

I continue to struggle. I am not able to submit the job and get some errors when I run case.setup (which seem to go away when I run it twice in a row). I attach the output from scripts_regression_tests.py. The issue I see when I run case.setup, case.submit, and the test script is very similar to

File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/case/case_setup.py", line 254, in <lambda>
functor = lambda: _case_setup_impl(self, caseroot, clean=clean, test_mode=test_mode, reset=reset, keep=keep)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/case/case_setup.py", line 203, in _case_setup_impl
env_batch.make_all_batch_files(case)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/XML/env_batch.py", line 941, in make_all_batch_files
self.make_batch_script(input_batch_script, job, case)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/XML/env_batch.py", line 194, in make_batch_script
overrides = self.get_job_overrides(job, case)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/XML/env_batch.py", line 189, in get_job_overrides
overrides["mpirun"] = case.get_mpirun_cmd(job=job, overrides=overrides)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/case/case.py", line 1467, in get_mpirun_cmd
aprun_args, num_nodes = get_aprun_cmd_for_case(self, run_exe, overrides=overrides)[0:2]
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/aprun.py", line 141, in get_aprun_cmd_for_case
run_exe)
File "/p/home/apcraig/E3SM-dc-ARRM-bigrid/cime/scripts/tests/../lib/CIME/aprun.py", line 49, in _get_aprun_cmd_for_case_impl
tt = rootpe + (ntask - 1) * pstrid + 1
TypeError: unsupported operand type(s) for -: 'str' and 'int'

with the "tt" error as the starting point. I don't believe I have implemented the "tt" computation anywhere in my port. But maybe I need to define a variable somewhere that I'm not? Any ideas?