Porting issues with CESM2.2, multiple errors in scripts_regression_tests.py

[plichtig]

Hello,
(I tried to post this before, but I think it was rejected, probably because I attached the files wrongly. If that is not the case, and this is a duplicate, please let me know and delete this. Thank you!)

I am attempting to port CESM2.2 to a local linux machine with intel and gnu compilers. If I am not wrong, ./manage_externals/checkout_externals seems to be successful, but I get multiple errors when running scripts_regression_tests.py, many of those with TestCreateNewCase. All referenced directories in my create_machine.xml (see attatched file) should be writable, unless I am missing any (I have seen here that it is often the cause for this type of errors). I am using intel-oneapi compilers 2021, and openmpi, as default compilers.
I am attaching here my configuration files in $HOME/.cime and the output of the scripts regression file, as well as the output.
I will be very thankful for any suggestions.




As indicated in the instructions for requesting help, I put here the output of ./describe_version

------------------------------------------------------------------------
git describe:
cesm2.2.0-0-g332937b
------------------------------------------------------------------------

------------------------------------------------------------------------
git status:
Not currently on any branch.
Untracked files:
(use "git add <file>..." to include in what will be committed)
describe_version.txt

nothing added to commit but untracked files present (use "git add" to track)
------------------------------------------------------------------------

------------------------------------------------------------------------
manage_externals status:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CAM.cfg
Processing externals description file : .gitmodules
Processing submodules description file : .gitmodules
Processing externals description file : ../Externals_cime.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, cice, cdeps, fox, cime, cmeps, cism, source_cism, clm, fates, ptclm, fms, mom, mosart, pop, cvmix, marbl, rtm, ww3,
M ./cime
modified sandbox, on cime5.8.32
HEAD detached at cime5.8.32
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: src/drivers/mct/cime_config/config_component.xml

Untracked files:
(use "git add <file>..." to include in what will be committed)
scripts/Tools/case.submitc
scripts/tests/log_scripts_createNewCase
scripts/tests/log_scripts_regression
scripts/tests/scripts_regression_tests.log
scripts/tests/scripts_regression_tests.log2
scripts/tests/scripts_regression_tests.log3
scripts/tests/scripts_regression_tests.log4
scripts/tests/scripts_regression_tests_config.log
src/drivers/mct/cime_config/config_component.xml.backup

no changes added to commit (use "git add" and/or "git commit -a")

e-o ./cime/src/drivers/nuopc/
-, not checked out -->

./components/cam
clean sandbox, on cam_cesm2_2_rel_02
HEAD detached at cam_cesm2_2_rel_02
nothing to commit, working tree clean

./components/cam/chem_proc
clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04

./components/cam/src/atmos_phys
clean sandbox, on version0_00_007
HEAD detached at version0_00_007
nothing to commit, working tree clean

./components/cam/src/dynamics/fv3/atmos_cubed_sphere
clean sandbox, on fv3_cesm.04
HEAD detached at fv3_cesm.04
nothing to commit, working tree clean

./components/cam/src/physics/carma/base
clean sandbox, on carma/release_tags/carma3_49_rel

./components/cam/src/physics/clubb
clean sandbox, on clubb_release_b76a124_20200220_c20200320
HEAD detached at clubb_release_b76a124_20200220_c20200320
nothing to commit, working tree clean

./components/cam/src/physics/cosp2/src
clean sandbox, on CFMIP/COSPv2.0/tags/v2.1.4cesm/src

./components/cam/src/physics/pumas
clean sandbox, on pumas_cam-release_v1.3
HEAD detached at pumas_cam-release_v1.3
nothing to commit, working tree clean

./components/cam/src/physics/silhs
clean sandbox, on silhs_clubb_release_b76a124_20200220_c20200320
HEAD detached at silhs_clubb_release_b76a124_20200220_c20200320
nothing to commit, working tree clean

./components/cdeps
clean sandbox, on 45b7a85823da1d187fe500d1be6e96bbcd1dff2e
HEAD detached at 45b7a85
nothing to commit, working tree clean

./components/cdeps/fox
clean sandbox, on 7b9488446b193192dd3f0378541e71099cb4e8a8
HEAD detached at 7b94884
nothing to commit, working tree clean

./components/cice
clean sandbox, on cice5_20200430
HEAD detached at cice5_20200430
nothing to commit, working tree clean

./components/cism
clean sandbox, on cism2_1_69
HEAD detached at cism2_1_69
nothing to commit, working tree clean

./components/cism/source_cism
clean sandbox, on f1a88d6bbe3bb5e2e8817f91aed6de87227f4bb7
HEAD detached at f1a88d6b
nothing to commit, working tree clean

./components/clm
clean sandbox, on release-cesm2.2.01
HEAD detached at release-cesm2.2.01
nothing to commit, working tree clean

./components/clm/src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
HEAD detached at sci.1.30.0_api.8.0.0
nothing to commit, working tree clean

./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_20200902
HEAD detached at PTCLM2_20200902
nothing to commit, working tree clean

e-o ./components/mom
-, not checked out --> mi_20200908

./components/mosart
clean sandbox, on mosart1_0_37
HEAD detached at mosart1_0_37
nothing to commit, working tree clean

./components/pop
clean sandbox, on cesm_pop_2_1_20200730
HEAD detached at cesm_pop_2_1_20200730
nothing to commit, working tree clean

./components/pop/externals/CVMix
clean sandbox, on v0.98-beta
HEAD detached at v0.98-beta
nothing to commit, working tree clean

./components/pop/externals/MARBL
clean sandbox, on cesm2.2-n00
HEAD detached at marbl0.39.1
nothing to commit, working tree clean

./components/rtm
clean sandbox, on rtm1_0_72
HEAD detached at rtm1_0_72
nothing to commit, working tree clean

./components/ww3
clean sandbox, on ww3_200710
HEAD detached at ww3_200710
nothing to commit, working tree clean

e-o ./libraries/FMS
-, not checked out --> fi_20200609

 

[jedwards]

It looks like you are using a newer version of python and finding some incompatible code in cesm2.2
I recommend using cesm2.3 if possilble, or if you must use 2.2, please try with a python 3.7.

 

[plichtig]

Thank you very much. I am still having the same problem, but I think I might be referencing python the wrong way in my config_machines.xml (there are a few versions installed in the system), so I contacted my system administrators to ask them about that. If this does not solve the problem, I might have to ask here again.
I searched for cesm2.3 (I would happily use it), but I do not seem to be able to find a release tag? Is it just the default version if I clone the repo without any tag specific tag?

 

[jedwards]

We do not have release tags in cesm2.3 yet, the latest beta is cesm2_3_beta08
if you clone the repo without any tag, you are on our most recent development.
Beta tags are pretty carefully tested, I recommend that.

 

[plichtig]

Hello. I would think I corrected for that, yet I keep having problems (even though the failed tests went down from 100+ to 70+). It seems that pylint find some sort of mistake, but I have not been able to solve that. I have also tried directly running a case (following the musica tutorial with FCnudged) in the hope that the problem was just something with the tests, but it is not.
I also tried cesm2.3, without any better results (the script, which is in CIME/lib/tests, just hangs), so for now I am sticking to cesm2.2 just to follow the docs to the T. I upload here my config files and my logfile again (even though the config files have barely changed). The module "cesm2", for now, loads a python3.7.11, which I installed with miniconda3. What you can see as "levante23" there is actually the same machine, configured for the other version of CESM, but I have also tried the same thing deleting that. I have not been able to understand the cause for the AttributionError I get, I guess I might still be referencing something the wrong way?
Thank you very much!

 

[jedwards]

The pylint is warning about requirements not in place when this code was written, you may be able to downgrade pylint to get those to pass. All of the J_TestCreateNewcase tests are passing so that's a good thing. We can look at a set of tests specific to the FCnudged compset you want to run
Try for example: create_test ERS.f19_g17.FCnudged

 

[plichtig]

I am still failing. In the case of FCnudged, the error message has something about the timeaddmonths being out of range, so I decided to run the prealpha tests suggested in the porting docs of CIME. Most of them had at least 1 failure. Here I concatenated all of the TestStatus of prealpha, and in a separated compressed file, the TestStatus, TestStatus.log and cesm.log of ERS.f19_g17.FCnudged.
On a separate note, I tried downgrading pylint to 1.9.2, but the results were worse (every single pylint test failed, so I guess that something with my downgrade went horribly wrong) so I can't comment on that.
I was thinking that maybe the fact that in the previous regression test (see file in my previous message) the following had also failed could provide some clue? Is there any way I can make sure if the problem lies within as
test_cime_case_test_custom_project (__main__.K_TestCimeCase) ... FAIL
test_bless_test_results (__main__.Q_TestBlessTestResults) ... FAIL
test_run_restart (__main__.T_TestRunRestart) ... FAIL
test_user_concurrent_mods (__main__.Y_TestUserConcurrentMods) ... FAIL
test_full_system (__main__.Z_FullSystemTest) ... FAIL

I have tried reading the longer message, but I still cannot identify the cause.

 

[jedwards]

193: ERROR timeaddmonths(): MM out of range
193: set_time_float_from_date: error return from ESMF_TimeSet for set_time_float_fro
193: m_date

 

[jedwards]

Sorry, the TestStatus file doesn't tell us anything other than some tests failed. Many fail in the submit phase
which may indicate a problem downloading inputdata. Other failures include COMPARE_base_rest which may
indicate that cprnc isn't working. Did you build the cprnc tool and verify it's functionality?

 

[plichtig]

There were probably some issues with cprnc, so I compiled it on its own. However, nothing really changed in the output.

I attempted the same with cesm2.3, and I get the error
ERROR: ESMFMKFILE not found None

I found that this Compiling CTSM for WRF with LILAC on other HPC than Cheyenne · Issue #1498 · ESCOMP/CTSM should be the answer to that. However, I don't see what the path should be.
I am sorry for the basic questions, but I am running out of ideas. Both CESM2.2 and CESM2.3 would be fine for me, as long as I can get one of them to run. I am expected to test MUSICA with a specific configuration, and according to the meeting we had with NCAR researchers, I have just been told to port the most recent version that is out.
If there is anything else I can try, please let me know.
I will contact my sysadmin about the problems for downloading data.

 

[plichtig]

OK, it seems to be working now. Here are the main steps I took in case it might be of use to somebody. I don't expect any response to this, I am just posting in case it might be useful.
a) Installing python 3.7 with the https://repo.anaconda.com/miniconda/Miniconda3-py37_4.11.0-Linux-x86_64.sh installer, and not adding any other package to my conda installation.
b) After every single case.setup, even if all the inputdata is already downloaded, running ./check_input_data. It seems that there might be some problem in my system (maybe my python installation) when pointing to the right data. Checking if some data needs to be manually downloaded, because the automatic download feature seems to fail sometimes, at least in my system

That's about it. If anybody else has trouble porting it to Levante (the new machine of DKRZ, in Germany), feel free to DM me.
Thanks to jedwards for the help, I wouldn't have been able to identify the problems without it!
Best
Pablo

 

[jedwards]

Hi Pablo, Great to hear that you've figured it out. If you find any change that would be useful to the community, please consider opening a PR in our github repository.

 

[xiangli]

193: ERROR timeaddmonths(): MM out of range
193: set_time_float_from_date: error return from ESMF_TimeSet for set_time_float_fro
193: m_date

Hi Jim,

I met similar error messages when running a BHIST restart simulation:



Is it related to the input or restart files?

It would be greatly appreciated if you could provide some suggestions!

Thanks,
Xiang