Checking out development branch causes a build failure

[raeder]

What version of the code are you using?

> git describe
cesm2_0_alpha10e-1237-g5437da0 This has cesm3.0_alphabranch merged in.


Have you made any changes to files in the source tree
I changed .gitmodules to get some clones from my github forks.
All of the forks were up-to-date with the ESCOMP versions when I ran git-fleximod this morning.

Describe every step you took leading up to the problem:
The goal is to make a small change in each of most of the components of CESM,
which will enable st-archiving of new types of files (from DART).
After reading the relevant parts of instructions for developers I:
1) Forked some repositories, with all of the their branches, to github.com/kdraeder:
2) Selected the development branch for each in the forks:

repos          develop branch
 cesm           master
 cime           master
 CAM            cam_development
 CESM_CICE      main
 CTSM           master
 CMEPS          main
 MOM_interface  main
 mizuRoute      cesm-coupling
 RTM            master
 MOSART         master

3) git clone https://github.com/kdraeder/cesm.git cesm3.0_gh_kdraeder
4) Modified .gitmodules to use the kdraeder versions of those repositories,
but continue to use the default versions of the others:

   [submodule "ccs_config"]
        path = ccs_config
        url =           https://github.com/ESMCI/ccs_config_cesm.git
        fxDONOTUSEurl = https://github.com/ESMCI/ccs_config_cesm.git
        fxtag = ccs_config_cesm1.0.32
        fxrequired = ToplevelRequired

    [submodule "cime"]
        path = cime
        url =           https://github.com/kdraeder/cime
        fxDONOTUSEurl = https://github.com/ESMCI/cime
        fxtag = cime6.1.72
        fxrequired = ToplevelRequired
...

5) git-fleximod update
This leaves all of the clones in a detached HEAD state.
6) Built and ran a B compset case for a day as a test.
7) Set up each target clone (but not cesm) for development work by (tcsh):
(These remote commands are left over from cases in which I used the default .gitmodules)

git remote rename origin upstream
      git remote add origin https://github.com/kdraeder/$repos[$r].git
      git checkout    $dev_br[$r]
      git pull origin $dev_br[$r]
      git checkout -b ${repname}_st-arch_dart

Example status:

      git status -uno
          On branch clm_st-arch_dart
          Changes not staged for commit:
                  modified:   src/fates (new commits)
      git remote -v
          origin   https://github.com/kdraeder/CTSM.git (push)
          upstream https://github.com/kdraeder/CTSM (fetch)

8) case.build --clean-all
9) case.build failed in the CLM build:

/glade/work/raeder/Models/cesm3.0_gh_kdraeder/components/clm/src/utils/clmfates_interfaceMod.F90(2878):
    error #6460:
    This is not a component name that is defined in the encompassing structure.   [COSZEN]
      this%fates(nc)%bc_in(s)%coszen = coszen_col(c)
...

Describe your problem or question:
What's wrong with my `build`s, branches, andor remotes?
Why does it work when those pieces are in a detached HEAD state, but not when I check out development branches?

 

[erik]

@raeder I don't have permissions to see your code on glade, so I can't look at any of this. But, I think there's a couple of things that are problematic with the process you've done as I understand it. A simple thing is that you'll want a remote to the upstream version for each of the modified submodule directories. Once, you have that you can fetch from the upstream for each submodule directory and be able to see tags and use describe.

cd components/clm
git remote add upstream https://github.com/ESCOMP/CTSM.git
git fetch upstream
git describe[
/CODE]

It looks like you might have a script that does this setup for you, so you could add some of the above to the script.

The other part though is that if I understand you're process, you have the latest from the development branch for each submodule. And you should NOT expect that to ever work for a CESM checkout. Maybe in some rare conditions you can do that -- but you shouldn't expect it to. What you should expect to work is a branch for each subcomponent that's based of the version used as specified in the CESM .gitmodules file.

So start with a branch for the submodules you need to update from the version in the CESM alpha branch .gitmodules file.

[URL]https://github.com/ESCOMP/CESM/blob/cesm3.0-alphabranch/.gitmodules[/URL]

I assuming you wanted to use the latest in each submodule to make it easter to be able to bring in your changes to each subcomponent. But, don't worry about that -- it'll be close enough. And if you make a PR to each subcomponent the maintainers will be able to merge it to the latest and bring it into that repository

The thing is -- it's tricky to get the submodules all aligned together in a way that they'll work together. Sometimes changes in one submodule makes it incompatible with another. So never expect the latest will ever work together, start with something that's been tested and known to work.

I think that's what you need to get something working together. Let us know how it goes.

 

[raeder]

@erik Thanks for putting me on the path to CESM git enlightenment!
In case other people are trying to figure out the CESM development work flow,
here's a summary of what I've found.
Please correct as needed.

Background:

• A branch name is just a nickname for the last of a series of commits (called the "tip").
• HEAD is a nickname for the *commit* which is currently active in the clone. It may or may not refer to a branch tip.
• A 'detached HEAD' is detached from all branch tips, but it does point at some other commit, so it's not completely detached, and is associated with a particular version of the code.
• `git checkout` checks out a commit.
  • That commit might be referred to by a branch name, in which case the commit is the most recent in that series (the tip). HEAD will also point to that commit.
  • Or it might be a commit which is not the tip of a branch. HEAD will point to it and be "detached". That code might be fully functional. This commit could be used to start a new branch.
• Any commit can be named with 'tag', which makes it easier to work with.

CESM:

1. `git clone <latest_CESM_URL_from_ESCOMP> local_version` gets a .gitmodules file, which lists a tag for each submodule (cime, components/*, ...).
2. A tag is not necessarily a branch, and may be in the history of multiple branches.
3. Descriptions of the tags (with SHAs) can be found in the ESCOMP/$submodule/tags pages.
4. The set of tags (commits) has passed a bunch of tests to ensure that they work together in some, most, or important configurations. Checking out branches for all of the submodules is not needed or helpful.
5. Each submodule has a development branch, which may be main, master, or some other name (see the lists above).
6. Each development branch should work with the .gitmodule tags of the other submodules, but probably not with the other's development branches.
7. If the changes in one submodule can be tested independently of the changes in other submodules, then they can be tested using this_development_branch + other_submodules_tags. So, for each submodule:
   1. check out the development branch,
   2. test the changes,
   3. add and commit them,
   4. sync the development branch on my fork (on my github) to the latest ESCOMP version,
   5. push the changes to that fork,
   6. issue a pull request from my fork to the ESCOMP repository.
8. If I have inseparable changes to multiple submodules, they should be tested without development branches, and use the .gitmodule tags as the base. These changes, which are not yet on a named branch, can be pushed to a different repository (fork) using the SHA of the commit. Erik said (above) that this would be good enough for the maintainers of the development branches to work with.

 

[erik]

In everything important this is a great summary. And it covers both important things about git behavior and CESM behavior.

On git I would add the difference between a branch and a commit (or detached HEAD state or a tag) is that you can add new commits to a branch -- but NOT to a commit. So if you want to commit changes -- you need to be on a branch. But, note that you can start that branch from any commit or tag, and that's typically what you need to do in order to bring in changes that a fix a problem in a given commit or tag.

On "3", on the command line you can use "git tag" to view all the tags. Or "git log" and search for a tag in the commit list to get the log that goes with it. But, in github.com it's convenient to search for tags there.

On "6" I might clarify that you use the .gitmodule file for the kind of checkout you are working with. There are top level checkouts for: CESM, CTSM, CISM, and CAM for example. If you are starting from a CESM checkout from a CESM tag, start from the submodules tags in the .gitmodule file there. If you update any of those submodules -- you'll need to make sure the model still works. Likewise, if for example starting with a CTSM checkout for a CTSM tag, use the submodules at the top level in the CTSM tag you checked out, and do something similar. Same for CAM and CISM. If you want to get something into: CTSM, CISM, or CAM you should likely start with a checkout from them and start with one of their tags.

Hope that makes sense. I'm glad you figured out the big picture here and were able to nicely outline it. Take care. Hopefully, this enables you to make progress in your work.

 

[raeder]

In everything important this is a great summary. And it covers both important things about git behavior and CESM behavior.

On git I would add the difference between a branch and a commit (or detached HEAD state or a tag) is that you can add new commits to a branch -- but NOT to a commit. So if you want to commit changes -- you need to be on a branch. But, note that you can start that branch from any commit or tag, and that's typically what you need to do in order to bring in changes that a fix a problem in a given commit or tag.

On "3", on the command line you can use "git tag" to view all the tags. Or "git log" and search for a tag in the commit list to get the log that goes with it. But, in github.com it's convenient to search for tags there.

On "6" I might clarify that you use the .gitmodule file for the kind of checkout you are working with. There are top level checkouts for: CESM, CTSM, CISM, and CAM for example. If you are starting from a CESM checkout from a CESM tag, start from the submodules tags in the .gitmodule file there. If you update any of those submodules -- you'll need to make sure the model still works. Likewise, if for example starting with a CTSM checkout for a CTSM tag, use the submodules at the top level in the CTSM tag you checked out, and do something similar. Same for CAM and CISM. If you want to get something into: CTSM, CISM, or CAM you should likely start with a checkout from them and start with one of their tags.

Hope that makes sense. I'm glad you figured out the big picture here and were able to nicely outline it. Take care. Hopefully, this enables you to make progress in your work.

@erik Thanks for putting me on the path to CESM git enlightenment!
In case other people are trying to figure out the CESM development work flow,
here's a summary of what I've found.
Please correct as needed.

Background:

• A branch name is just a nickname for the last of a series of commits (called the "tip").
• HEAD is a nickname for the *commit* which is currently active in the clone. It may or may not refer to a branch tip.
• A 'detached HEAD' is detached from all branch tips, but it does point at some other commit, so it's not completely detached, and is associated with a particular version of the code.
• `git checkout` checks out a commit.
  • That commit might be referred to by a branch name, in which case the commit is the most recent in that series (the tip). HEAD will also point to that commit.
  • Or it might be a commit which is not the tip of a branch. HEAD will point to it and be "detached". That code might be fully functional. This commit could be used to start a new branch.
• Any commit can be named with 'tag', which makes it easier to work with.

CESM:

1. `git clone <latest_CESM_URL_from_ESCOMP> local_version` gets a .gitmodules file, which lists a tag for each submodule (cime, components/*, ...).
2. A tag is not necessarily a branch, and may be in the history of multiple branches.
3. Descriptions of the tags (with SHAs) can be found in the ESCOMP/$submodule/tags pages.
4. The set of tags (commits) has passed a bunch of tests to ensure that they work together in some, most, or important configurations. Checking out branches for all of the submodules is not needed or helpful.
5. Each submodule has a development branch, which may be main, master, or some other name (see the lists above).
6. Each development branch should work with the .gitmodule tags of the other submodules, but probably not with the other's development branches.
7. If the changes in one submodule can be tested independently of the changes in other submodules, then they can be tested using this_development_branch + other_submodules_tags. So, for each submodule:
   1. check out the development branch,
   2. test the changes,
   3. add and commit them,
   4. sync the development branch on my fork (on my github) to the latest ESCOMP version,
   5. push the changes to that fork,
   6. issue a pull request from my fork to the ESCOMP repository.
8. If I have inseparable changes to multiple submodules, they should be tested without development branches, and use the .gitmodule tags as the base. These changes, which are not yet on a named branch, can be pushed to a different repository (fork) using the SHA of the commit. Erik said (above) that this would be good enough for the maintainers of the development branches to work with.

8. CORRECTION: It's possible to commit changes when not on a named branch, but as Erik points out, I'm unable to push the change/commit to my fork, so the result is that I need to have a branch checked out in order push my changes.