Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

git instead of svn ./manage_externals/checkout_externals ?

Jbuzan

Jonathan R. Buzan
Member
Hi DiscussCESM users:

I am trying to get CESM installed on Eiger so I can run benchmarking.

Is there a workaround using GitHub instead of svn?
I am working on a new cluster, Eiger (part of Alps CSCS.ch), which has a similar architecture to Derecho. In all my years working on clusters, this is the first time svn is not part of the machine. The cluster support for Eiger has stated that I need to install svn myself, but svn that has many dependencies, so this is a larger task than anticipated.

For functions, like, manage_externals, and data downloading, is there a GitHub workaround?


Cheers,
-Jonathan
 

jedwards

CSEG and Liaisons
Staff member
You didn't say what version of cesm you are working with but github no longer supports an svn interface so as of cesm2.1.5 and cesm2.2.2 and cesm2.3beta16 we do not either. For inputdata we offer alternative interfaces through ftp and wget. As of the cesm3_0_beta01 tag released last week
we no longer use manage_externals. I don't think that we have explicitly tested on a machine without svn but I don't believe that it is a requirement for cesm.
 

Jbuzan

Jonathan R. Buzan
Member
Hi Jim,

Okay, that's interesting. Because I was using cesm2.2.2 following the instructions from:

./manage_externals/checkout_externals


Processing externals description file : Externals.cfg


Processing externals description file : Externals_CAM.cfg


Processing externals description file : .gitmodules


Processing submodules description file : .gitmodules


Processing externals description file : ../Externals_cime.cfg


Processing externals description file : Externals_CISM.cfg


Processing externals description file : Externals_CLM.cfg


Processing externals description file : Externals_POP.cfg


Checking status of externals: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, cice, cdeps, fox, cime, cmeps, cism, source_cism, clm, fates, ptclm, fms, mom, mosart, pop, cvmix, marbl, rtm, ww3,


Checking out externals: cam, cice, cdeps, cime, cism, clm, mosart, pop, rtm, ww3,


Processing externals description file : Externals_CAM.cfg


Checking out externals: chem_proc, ERROR:root:[Errno 2] No such file or directory: 'svn'


ERROR:root:In directory


/users/jbuzan/cesm2_2_2/my_cesm_sandbox


Command execution failed. Does the executable exist?:


svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam/chem_proc








ERROR: In directory


/users/jbuzan/cesm2_2_2/my_cesm_sandbox


Command execution failed. Does the executable exist?:


svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam/chem_proc
 

Attachments

  • svn_manageexternals.png
    svn_manageexternals.png
    83.2 KB · Views: 7

jedwards

CSEG and Liaisons
Staff member
Oh I'm sorry - you are correct. The chem_proc and carma repos in the cam tag do still require svn. But those tags have been duplicated in git and you can replace with:

Code:
[chem_proc]
local_path = chem_proc
protocol = git
repo_url = https://github.com/ESCOMP/CHEM_PREPROCESSOR.git
tag = chem_proc5_0_04
required = True

[carma]
local_path = src/physics/carma/base
protocol = git
repo_url = https://github.com/ESCOMP/CARMA_base.git
tag = carma3_49
required = True
 

Jbuzan

Jonathan R. Buzan
Member
Hi Jim,

I am trying to execute the command. What am running into is blowing up on CAM. I commit the changes to External_CAM.cfg, but the command still stops on the same step.


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:44:12] $ ./manage_externals/checkout_externals


Processing externals description file : Externals.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox)


Processing externals description file : ../Externals_cime.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/cime)


Checking local status of required & optional components: cam, cice, cdeps, cime, cmeps, cism, clm, fms, mom, mosart, pop, rtm, ww3,


Checking out externals: cime, Processing externals description file : ../Externals_cime.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/cime)


Checking out externals:


cam, Processing externals description file : Externals_CAM.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam)


Checking out externals: chem_proc, ERROR:root:[Errno 2] No such file or directory: 'svn'


ERROR:root:In directory


/users/jbuzan/cesm2_2_2/my_cesm_sandbox


Command execution failed. Does the executable exist?:


svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam/chem_proc








ERROR: In directory


/users/jbuzan/cesm2_2_2/my_cesm_sandbox


Command execution failed. Does the executable exist?:


svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam/chem_proc





jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:47:45] $ cd components/cam/


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox/components/cam [07:48:35] $ nano Externals_CAM.cfg


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox/components/cam [07:49:00] $ cd ../../


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:49:03] $ ./manage_externals/checkout_externals


Processing externals description file : Externals.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox)


Processing externals description file : Externals_CAM.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam)


Processing externals description file : ../Externals_cime.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/cime)


Checking local status of required & optional components: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, cice, cdeps, cime, cmeps, cism, clm, fms, mom, mosart, pop, rtm, ww3,


./cime


e-o ./cime/src/drivers/nuopc/


M ./components/cam


e- ./components/cam/chem_proc


e- ./components/cam/src/atmos_phys


e- ./components/cam/src/dynamics/fv3/atmos_cubed_sphere


e- ./components/cam/src/physics/carma/base


e- ./components/cam/src/physics/clubb


e- ./components/cam/src/physics/cosp2/src


e- ./components/cam/src/physics/pumas


e- ./components/cam/src/physics/silhs


e- ./components/cdeps


e- ./components/cice


e- ./components/cism


e- ./components/clm


e-o ./components/mom


e- ./components/mosart


e- ./components/pop


e- ./components/rtm


e- ./components/ww3


e-o ./libraries/FMS


----------------------------------------------------------------------


The external repositories labeled with 'M' above are not in a clean state.


The following are four options for how to proceed:


(1) Go into each external that is not in a clean state and issue either a 'git status' or


an 'svn status' command (depending on whether the external is managed by git or


svn). Either revert or commit your changes so that all externals are in a clean


state. (To revert changes in git, follow the instructions given when you run 'git


status'.) (Note, though, that it is okay to have untracked files in your working


directory.) Then rerun checkout_externals.


(2) Alternatively, you do not have to rely on checkout_externals. Instead, you can manually


update out-of-sync externals (labeled with 's' above) as described in the


configuration file Externals.cfg. (For example, run 'git fetch' and 'git checkout'


commands to checkout the appropriate tags for each external, as given in


Externals.cfg.)


(3) You can also use checkout_externals to manage most, but not all externals: You can specify


one or more externals to ignore using the '-x' or '--exclude' argument to


checkout_externals. Excluding externals labeled with 'M' will allow checkout_externals to


update the other, non-excluded externals.


(4) As a last resort, if you are confident that there is no work that needs to be saved


from a given external, you can remove that external (via "rm -rf [directory]") and


then rerun the checkout_externals tool. This option is mainly useful as a workaround for


issues with this tool (such as Additions/deletions of an externals submodules can lead to a modified status after running checkout_externals · Issue #157 · ESMCI/manage_externals).


The external repositories labeled with '?' above are not under version


control using the expected protocol. If you are sure you want to switch


protocols, and you don't have any work you need to save from this


directory, then run "rm -rf [directory]" before rerunning the


checkout_externals tool.





----------------------------------------------------------------------


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:49:15] $ cd components/cam/


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox/components/cam [07:49:22] $ git status


Refresh index: 100% (1901/1901), done.


HEAD detached at cam_cesm2_2_rel_09


Changes not staged for commit:


(use "git add <file>..." to update what will be committed)


(use "git restore <file>..." to discard changes in working directory)


modified: Externals_CAM.cfg





no changes added to commit (use "git add" and/or "git commit -a")


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox/components/cam [07:49:29] $ git add Externals_CAM.cfg


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox/components/cam [07:49:46] $ cd ../../


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:49:50] $ ./manage_externals/checkout_externals


Processing externals description file : Externals.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox)


Processing externals description file : Externals_CAM.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/components/cam)


Processing externals description file : ../Externals_cime.cfg (/users/jbuzan/cesm2_2_2/my_cesm_sandbox/cime)


Checking local status of required & optional components: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, cice, cdeps, cime, cmeps, cism, clm, fms, mom, mosart, pop, rtm, ww3,


./cime


e-o ./cime/src/drivers/nuopc/


M ./components/cam


e- ./components/cam/chem_proc


e- ./components/cam/src/atmos_phys


e- ./components/cam/src/dynamics/fv3/atmos_cubed_sphere


e- ./components/cam/src/physics/carma/base


e- ./components/cam/src/physics/clubb


e- ./components/cam/src/physics/cosp2/src


e- ./components/cam/src/physics/pumas


e- ./components/cam/src/physics/silhs


e- ./components/cdeps


e- ./components/cice


e- ./components/cism


e- ./components/clm


e-o ./components/mom


e- ./components/mosart


e- ./components/pop


e- ./components/rtm


e- ./components/ww3


e-o ./libraries/FMS


----------------------------------------------------------------------


The external repositories labeled with 'M' above are not in a clean state.


The following are four options for how to proceed:


(1) Go into each external that is not in a clean state and issue either a 'git status' or


an 'svn status' command (depending on whether the external is managed by git or


svn). Either revert or commit your changes so that all externals are in a clean


state. (To revert changes in git, follow the instructions given when you run 'git


status'.) (Note, though, that it is okay to have untracked files in your working


directory.) Then rerun checkout_externals.


(2) Alternatively, you do not have to rely on checkout_externals. Instead, you can manually


update out-of-sync externals (labeled with 's' above) as described in the


configuration file Externals.cfg. (For example, run 'git fetch' and 'git checkout'


commands to checkout the appropriate tags for each external, as given in


Externals.cfg.)


(3) You can also use checkout_externals to manage most, but not all externals: You can specify


one or more externals to ignore using the '-x' or '--exclude' argument to


checkout_externals. Excluding externals labeled with 'M' will allow checkout_externals to


update the other, non-excluded externals.


(4) As a last resort, if you are confident that there is no work that needs to be saved


from a given external, you can remove that external (via "rm -rf [directory]") and


then rerun the checkout_externals tool. This option is mainly useful as a workaround for


issues with this tool (such as Additions/deletions of an externals submodules can lead to a modified status after running checkout_externals · Issue #157 · ESMCI/manage_externals).


The external repositories labeled with '?' above are not under version


control using the expected protocol. If you are sure you want to switch


protocols, and you don't have any work you need to save from this


directory, then run "rm -rf [directory]" before rerunning the


checkout_externals tool.





----------------------------------------------------------------------


jbuzan@eiger-ln003:~/cesm2_2_2/my_cesm_sandbox [07:50:00] $
 

Jbuzan

Jonathan R. Buzan
Member
This a hack, but I logged into the old machine, Piz Daint, and downloaded the model there. the $HOME directory is shared with Eiger.

$ ./manage_externals/checkout_externals --status
Processing externals description file : Externals.cfg (/users/jbuzan/CESM2_2_2/CESM)
Processing externals description file : Externals_CAM.cfg (/users/jbuzan/CESM2_2_2/CESM/components/cam)
Processing externals description file : .gitmodules (/users/jbuzan/CESM2_2_2/CESM/components/cdeps)
Processing externals description file : ../Externals_cime.cfg (/users/jbuzan/CESM2_2_2/CESM/cime)
Processing externals description file : .gitmodules (/users/jbuzan/CESM2_2_2/CESM/cime/src/drivers/nuopc)
Processing externals description file : Externals_CISM.cfg (/users/jbuzan/CESM2_2_2/CESM/components/cism)
Processing externals description file : Externals_CLM.cfg (/users/jbuzan/CESM2_2_2/CESM/components/clm)
Processing externals description file : Externals_POP.cfg (/users/jbuzan/CESM2_2_2/CESM/components/pop)
Checking local status of required & optional components: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, cice, cdeps, fox, cime, cmeps, nems/lib/parallelio, nems/lib/genf90, cism, source_cism, clm, fates, fms, mom, mosart, pop, cvmix, marbl, rtm, ww3,
./cime
./cime/src/drivers/nuopc/
./cime/src/drivers/nuopc/nems/lib/ParallelIO
./cime/src/drivers/nuopc/nems/lib/genf90
./components/cam
./components/cam/chem_proc
./components/cam/src/atmos_phys
./components/cam/src/dynamics/fv3/atmos_cubed_sphere
./components/cam/src/physics/carma/base
./components/cam/src/physics/clubb
./components/cam/src/physics/cosp2/src
./components/cam/src/physics/pumas
./components/cam/src/physics/silhs
./components/cdeps
./components/cdeps/fox
./components/cice
./components/cism
./components/cism/source_cism
./components/clm
./components/clm/src/fates
e-o ./components/mom
./components/mosart
./components/pop
./components/pop/externals/CVMix
./components/pop/externals/MARBL
./components/rtm
./components/ww3
e-o ./libraries/FMS
 

Attachments

  • described_install.txt
    6.3 KB · Views: 7

Sarin

Sarin
New Member
Oh I'm sorry - you are correct. The chem_proc and carma repos in the cam tag do still require svn. But those tags have been duplicated in git and you can replace with:

Code:
[chem_proc]
local_path = chem_proc
protocol = git
repo_url = https://github.com/ESCOMP/CHEM_PREPROCESSOR.git
tag = chem_proc5_0_04
required = True

[carma]
local_path = src/physics/carma/base
protocol = git
repo_url = https://github.com/ESCOMP/CARMA_base.git
tag = carma3_49
required = True
[/QUOTE]
Dear Jim, 
I am a newbie here, and I was trying to install the 2.2.2 version using the same documentation. However, the same problem arises, possibly due to the issue you mentioned, i.e., chem_proc requires svn. I have tried to edit the Externals.cfg file and committed the changes. However, the error persists.

The commands that I have followed are :
git clone -b release-cesm2.2.2 https://github.com/ESCOMP/CESM.git CESM
cd CESM
vi External.cfg # And added the lines as before
git checkout -b CESM
git add Externals.cfg
git commit -m "Updated config file"

The Error generated was:

Processing externals description file : Externals.cfg (/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm)
Processing externals description file : Externals_CAM.cfg (/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam)
Processing externals description file : ../Externals_cime.cfg (/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/cime)
Checking local status of required & optional components: cam, chem_proc, carma, cosp2, clubb, silhs, pumas, atmos_phys, atmos_cubed_sphere, chem_proc, carma, cice, cdeps, cime, cmeps, cism, clm, fms, mom, mosart, pop, rtm, ww3, 
Checking out externals: 

    ./cime
        clean sandbox, on cime5.8.32.9
Processing externals description file : ../Externals_cime.cfg (/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/cime)
Checking out externals: 


    ./components/cam
        clean sandbox, on cam_cesm2_2_rel_09
Processing externals description file : Externals_CAM.cfg (/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam)
Checking out externals: 

e-  chem_proc
        -, not checked out --> chem_proc5_0_04
    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam/chem_proc
ERROR:root:Command '['svn', 'checkout', '--quiet', 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04', '/home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam/chem_proc']' returned non-zero exit status 1.
ERROR:root:Failed with output:
    svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04'
    svn: E175009: The XML response contains invalid XML
    svn: E130003: Malformed XML: no element found

ERROR: In directory
    /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm
Process did not run successfully; returned status 1:
    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam/chem_proc
See above for output from failed command.

ERROR:root:Failed with output:
    svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04'
    svn: E175009: The XML response contains invalid XML
    svn: E130003: Malformed XML: no element found

ERROR: In directory
    /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm
Process did not run successfully; returned status 1:
    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam/chem_proc
See above for output from failed command.


ERROR: Failed with output:
    svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04'
    svn: E175009: The XML response contains invalid XML
    svn: E130003: Malformed XML: no element found

ERROR: In directory
    /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm
Process did not run successfully; returned status 1:
    svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/tools/proc_atm/chem_proc/release_tags/chem_proc5_0_04 /home/aerolab/Sarin/Downloads/Aerosol_2019-20/cesm/components/cam/chem_proc
See above for output from failed command.

after running ./manage_externals/checkout_externals -v. Could you please direct me as to what the issue is? Thank you again for your consideration


Sincerely,
Sarin
 
Top