Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

case.build --skip-provenance-check

M

marsh

Member
Hello,

I am not sure what has changed recently, but it seems --skip-provenance-check is now needed to build the model. We found this on both cheyenne and the University of Leeds HPC systems. This is using CESM2.1.3. Using this argument does seem to work, but perhaps this is hiding some problem we should be aware about that might be affecting the validity of our simulations? Do let us know if so or if there's an update we need to make,

thanks,

Dan
 
J

jedwards

CSEG and Liaisons
Staff member
Hi Dan,

This is the first we've heard of this problem - can you please tell us how to reproduce it on cheyenne?
 
F

fischer

CSEG and Liaisons
Staff member
I was able to build a B1850 f09_g17 case on Cheyenne, without needing --skip-provenance-check.
 
M

marsh

Member
Hi both,

All seems to go well during the build until just after cam is built:

cam built in 207.131009 seconds
Building cesm with output to /glade/scratch/marsh/b.e21.BWma1850.f19_g17.0.1pc_o2.ubc.001/bld/cesm.bldlog.200406-112418
ERROR: Error gathering provenance information from manage_externals.

manage_externals error message:
ERROR:root:SVN returned invalid XML message

manage_externals output:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CAM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: cam, chem_proc,
ERROR: SVN returned invalid XML message

To solve this, either:

(1) Find and fix the problem: From /glade/u/home/marsh/cesm/release_cesm2_1_3, try to get this command to work:
manage_externals/checkout_externals --status --verbose --no-logging

(2) If you don't need provenance information, rebuild with --skip-provenance-check
 
M

marsh

Member
Following the instructions:

base) marsh@cheyenne4:~/work/cesm/cases/b.e21.BWma1850.f19_g17.0.1pc_o2.ubc.001> cd /glade/u/home/marsh/cesm/release_cesm2_1_3
(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3> ./manage_externals/checkout_externals --status --verbose --no-logging
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CAM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: cam, chem_proc, ERROR:root:SVN returned invalid XML message

ERROR: SVN returned invalid XML message
(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3>
 
J

jedwards

CSEG and Liaisons
Staff member
If I go to your sandbox on cheyenne I can reproduce the problem, but I can't if I checkout a new cesm2.1.3 sandbox.
Can you try removing the cam component from your sandbox and replacing it by running (from the cesm directory)
./manage_externals/checkout_externals cam
 
M

marsh

Member
Hi Jim,

That works and I can now build without the skip-provenance-check (see output below). So I tried to implement it on the University of Leeds HPC system since we are getting the same error there. And ran into an issue of a 'locked database'. To summarize on that system,
./manage_externals/checkout_externals --status --verbose --no-logging gave same "SVN returned invalid XML message"
Following removing components/cam and manage_externals/checkout_externals cam I now get and error getting cam/chem_proc:

Checking out externals: chem_proc, ERROR:root:Command '['svn', 'checkout', '--quiet', 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel', '/home/home01/phydrm/cesm2/release_cesm2_1_3/components/cam/chem_proc']' returned non-zero exit status 1.
ERROR:root:Failed with output:
svn: E200033: database is locked, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;'

I also did a completely fresh attempt to download CESM2.1.3 and get the same locked database error.

Dan

cheyenne log:

(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3> cd components/
(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3/components> rm -rf cam
(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3/components> cd ../
(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3> ./manage_externals/checkout_externals cam
Processing externals description file : Externals.cfg
Checking status of externals: cam,
Checking out externals: cam,
Processing externals description file : Externals_CAM.cfg
Checking out externals: chem_proc, carma, clubb, cosp2,

(base) marsh@cheyenne4:~/cesm/release_cesm2_1_3> ./manage_externals/checkout_externals --status --verbose --no-logging
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CAM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: cam, chem_proc, carma, clubb, cosp2, cice, cime, cism, source_cism, clm, fates, ptclm, mosart, pop, cvmix, marbl, rtm, ww3,
./cime
clean sandbox, on cime5.6.32
./components/cam
clean sandbox, on cam_cesm2_1_rel_41
./components/cam/chem_proc
clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel
./components/cam/src/physics/carma/base
clean sandbox, on carma/release_tags/carma3_49_rel
./components/cam/src/physics/clubb
clean sandbox, on vendor_clubb_r8099_n03
./components/cam/src/physics/cosp2/src
clean sandbox, on CFMIP/COSPv2.0/tags/v2.1.4cesm/src
./components/cice
clean sandbox, on cice5_cesm2_1_1_20190321
./components/cism
clean sandbox, on cism-release-cesm2.1.2_02
./components/cism/source_cism
clean sandbox, on release-cism2.1.03
./components/clm
clean sandbox, on release-clm5.0.30
./components/clm/src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
./components/clm/tools/PTCLM
clean sandbox, on PTCLM2_20200121
./components/mosart
clean sandbox, on release-cesm2.0.04
./components/pop
clean sandbox, on pop2_cesm2_1_rel_n09
./components/pop/externals/CVMix
clean sandbox, on v0.93-beta
./components/pop/externals/MARBL
clean sandbox, on cesm2.1-n00
./components/rtm
clean sandbox, on release-cesm2.0.04
./components/ww3
clean sandbox, on ww3_181001
 
M

marsh

Member
OK, I am pretty sure the issue could be with doing a CESM code checkout on one machine and building on another, where the two machines share a common login directory. At NCAR it is casper and cheyenne. At Leeds it is arc3 and arc4.

Over the weekend I had done a clean local install of the latest release to ensure I was working with the released code (and not sandbox or developer tag) to mirror the version at Leeds. However I had done it while being logged in on casper. When I built on cheyenne it gave the SVN error that was avoided with the --skip-provenance-check

Wiping out the cam directory and rechecking it out while on cheyenne fixed it. Now, when doing

./manage_externals/checkout_externals --status --verbose --no-logging

I get no errors on cheyenne but the same old "Checking status of externals: cam, chem_proc, ERROR:root:SVN returned invalid XML message" error when I execute it on casper. It seems only the chem / chem_proc external causes the problem.

On the Leeds machines the latest release was ported to arc3 first, so when building on the new machine (arc4) we get the same problem. It does seem to be that this is a recent issue - building on arc4 didn't need the --skip-provenace-check argument until recently. For now, I think we will need to keep using this argument when building on the new machine.

It is not clear to me why this only seems to affect the chem_proc part of the code - perhaps it is because it is one of the few coming from https://svn-ccsm-models.cgd.ucar.edu rather than ESCOMP?
 
F

fischer

CSEG and Liaisons
Staff member
Hi Dan,
I believe part of the problem is the different versions of svn being used. On casper it's 1.7.14, and on cheyenne it's 1.8.19. My guess is chem_proc tag
being used was probably created by yet another version of svn. So this is creating problems.
 
F

fischer

CSEG and Liaisons
Staff member
When I try doing an svn info in chem_proc I get the following error message.

[fischer@casper04 chem_proc]$ svn info
svn: E155021: This client is too old to work with the working copy at
'/glade/u/home/fischer/code/cesm2.1.3/components/cam/chem_proc' (format 31).
You need to get a newer Subversion client. For more details, see
Apache Subversion FAQ
 
A

alvgomezc

Álvaro Gustavo Gómez C.
New Member
Hi, I'm having a similiar error when building the case, but when using the --skip-provenance-check flag I manage to avoid it. The output of
./manage_externals/checkout_externals --status --verbose --no-logging is the following:

Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CAM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, clubb, carma, ERROR:root:SVN returned invalid XML message

ERROR: SVN returned invalid XML message

I'm using CESM 2.1.3.

Thanks in advance :)
 
A

alvgomezc

Álvaro Gustavo Gómez C.
New Member
I don't know how and why but i solved it by correcting de locales variables (with LC_ALL) in my local machine.
 
H

Hemraj

Hemraj Bhattarai
Member
jedwards said:
Try the following:
svn ls https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags
Click to expand...
Hi,
My model was working ok, but of a sudden, I encountered exactly the same problem discussed above. I also tried as suggested but the problem exists the same.

(base) [s1155149263@chpc-sandbox cesm2_1_3]$ svn ls https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags
chem_proc3_7_02/
chem_proc4_0_02/
chem_proc5_0_03_rel/
chem_proc5_0_04/


I also tried as suggested in the error message and that did't help too:

(base) [s1155149263@chpc-sandbox cesm2_1_3]$ manage_externals/checkout_externals --status --verbose --no-logging
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CAM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: cam, chem_proc, ERROR:root:SVN returned invalid XML message
ERROR: SVN returned invalid XML message

I am not sure how this error occurred the way to resolve this issue. Could you please suggest how to deal with this problem?
Many thanks.

Hemraj
 
You must log in or register to reply here.
Top