How to transplant my model to Derecho

[jinmuluo]

Hi,

We are developing our own model based on CLM5.0, and we want to move our work from Cheyenne to Derecho. So I'm wondering how can I define derecho as a new machine in my current cime tool? Thank you!

Best,

 

[jinmuluo]

I tried to replace the "config_machines.xml" under my code repository with the newest one from CTSM, and I saw the derecho definitions. But when I tried to create a new case, I met such error. Any thoughts about this problem?

ERROR: Command: '/glade/u/apps/opt/conda/envs/npl-2023a/bin/xmllint --xinclude --noout --schema /glade/u/home/jinmuluo/CLM-FANv3/cime/CIME/data/config/xml_schemas/config_machines.xsd /glade/u/home/jinmuluo/CLM-FANv3/ccs_config/machines/config_machines.xml' failed with error '/glade/u/home/jinmuluo/CLM-FANv3/ccs_config/machines/config_machines.xml:427: element MAX_CPUTASKS_PER_GPU_NODE: Schemas validity error : Element 'MAX_CPUTASKS_PER_GPU_NODE': This element is not expected. Expected is one of ( MAX_MPITASKS_PER_NODE, COSTPES_PER_NODE, PROJECT_REQUIRED, mpirun ).

/glade/u/home/jinmuluo/CLM-FANv3/ccs_config/machines/config_machines.xml:1277: element MAX_CPUTASKS_PER_GPU_NODE: Schemas validity error : Element 'MAX_CPUTASKS_PER_GPU_NODE': This element is not expected. Expected is one of ( MAX_MPITASKS_PER_NODE, COSTPES_PER_NODE, PROJECT_REQUIRED, mpirun ).

/glade/u/home/jinmuluo/CLM-FANv3/ccs_config/machines/config_machines.xml:1953: element MAX_CPUTASKS_PER_GPU_NODE: Schemas validity error : Element 'MAX_CPUTASKS_PER_GPU_NODE': This element is not expected. Expected is one of ( MAX_MPITASKS_PER_NODE, COSTPES_PER_NODE, PROJECT_REQUIRED, mpirun ).

/glade/u/home/jinmuluo/CLM-FANv3/ccs_config/machines/config_machines.xml fails to validate' from dir '/glade/u/home/jinmuluo/CLM-FANv3/cime/scripts'

 

[oleson]

There is a document in progress that describes how to do this for CTSM.
However, it is not yet finished and needs to be reviewed, hopefully soon.

 

[jinmuluo]

There is a document in progress that describes how to do this for CTSM.
However, it is not yet finished and needs to be reviewed, hopefully soon.

Thank you for your reply, where we can find this document after it is published?

 

[oleson]

There will be a github discussion item on this site, probably later today or tomorrow:

ESCOMP CTSM · Discussions

Explore the GitHub Discussions forum for ESCOMP CTSM. Discuss code, ask questions & collaborate with the developer community.

github.com

 

[jinmuluo]

There will be a github discussion item on this site, probably later today or tomorrow:

ESCOMP CTSM · Discussions

Explore the GitHub Discussions forum for ESCOMP CTSM. Discuss code, ask questions & collaborate with the developer community.

github.com

Hi Sam @samrabin

I copied the derecho machine definitions from your repository (config_machine.xml, config_batch.xml and derecho.cmake). Now I can successfully create a case, but when I built it, the model told me it couldn't load the module "esmf/8.6.0b04-debug". Do you know how to solve this problem?

Thank you!

 

[erik]

We now have a document and discussion about Derecho and CTSM:

Running CTSM on Derecho · ESCOMP/CTSM · Discussion #2284

For users that are ready to start running CTSM on Derecho we've created this document to help guide you through the process. The simplest way to get started is to update to tag ctsm5.1.dev157 (or l...

github.com

Move to Derecho

docs.google.com

I'll answer your question about modules in the next reply...

 

[erik]

I copied the derecho machine definitions from your repository (config_machine.xml, config_batch.xml and derecho.cmake). Now I can successfully create a case, but when I built it, the model told me it couldn't load the module "esmf/8.6.0b04-debug". Do you know how to solve this problem?

This means there is some trouble in the module initialization, as we use a custom initialization for modules for CESM.

There's a working case I have on Derecho that includes the following line in the .env_mach_specific.csh for the case

source /glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/csh module load cesmdev/1.0 ncarenv/23.09 module purge module load craype intel/2023.2.1 mkl ncarcompilers/1.0.0 cmake mpi-serial/2.3.0 netcdf/4.9.2 parallelio/2.6.2-debug esmf/8.6.0b04-debug

You'll need something like that in your case as well.

The other suggestion now that we have a document out is to follow the instructions there. Either update to ctsm5.1.dev157 or at least update the externals to the ctsm5.1.dev157. The document goes into detail on doing that.

Hopefully this helps. Good luck on moving to Derecho!

 

[jinmuluo]

This means there is some trouble in the module initialization, as we use a custom initialization for modules for CESM.

There's a working case I have on Derecho that includes the following line in the .env_mach_specific.csh for the case

source /glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/csh module load cesmdev/1.0 ncarenv/23.09 module purge module load craype intel/2023.2.1 mkl ncarcompilers/1.0.0 cmake mpi-serial/2.3.0 netcdf/4.9.2 parallelio/2.6.2-debug esmf/8.6.0b04-debug

You'll need something like that in your case as well.

The other suggestion now that we have a document out is to follow the instructions there. Either update to ctsm5.1.dev157 or at least update the externals to the ctsm5.1.dev157. The document goes into detail on doing that.

Hopefully this helps. Good luck on moving to Derecho!

Hi Erik,

Thank you! Unfortunately, I have to manually add the derecho definitions to the machines repository. I tried to use the commands you mentioned to load the packages
source /glade/u/apps/derecho/23.06/spack/opt/spack/lmod/8.7.20/gcc/7.5.0/pdxb/lmod/lmod/init/csh
module load cesmdev/1.0 ncarenv/23.09
module purge
module load craype intel/2023.2.1 mkl ncarcompilers/1.0.0 cmake mpi-serial/2.3.0 netcdf/4.9.2 parallelio/2.6.2-debug esmf/8.6.0b04-debug

but it looks like my system can't recognize the command "setenv". Any way to to define this command?

Best,

 

[oleson]

You may be using a bash shell which doesn't recognize setenv. Try using a c-shell or tcshrc.

 

[jinmuluo]

You may be using a bash shell which doesn't recognize setenv. Try using a c-shell or tcshrc.

Hi Keith,

I forked my code from this repository :
GitHub - juliusvira/ctsm at fan-2019-ctsm

this is my github page

GitHub - jinmuluo/ctsm at ctsm-fanv3-2023

Community Terrestrial Systems Model (includes the Community Land Model of CESM) - GitHub - jinmuluo/ctsm at ctsm-fanv3-2023

github.com

and his code is far behind the CTSM source code. I couldn't fetch the newest tags in its main branch, do you know how to skip his main branch to fetch the tags from ctsm official site?

thank you!

 

[oleson]

I don't know, sorry. Pinging @slevis and @erik here because they are more familiar with these github operations than I am.

 

[erik]

Jinmu

@oleson already gave you a way to handle the shell issue. But, if you are using the commands are just just slightly different and you use the export command rather than setenv. And you use the .env_mach_specifc.sh rather than .env_mach_specific.csh.

But, it sounds like you have that part in control, so let me post about the git issue next...

 

[erik]

On the git issue, I had hoped we had some discussion on this in our github wiki. But, I did not find it. Here's maybe the closest, which talks about github remotes...

CTSM development workflow

Community Terrestrial Systems Model (includes the Community Land Model of CESM) - ESCOMP/CTSM

github.com

The CTSM github tutorials also will talk about this, but there isn't a definite part that just talks about this issue.

Anyway, the answer is that you need to add a remote for the ESCOMP fork for CTSM and you need to pull and fetch from it to get the latest changes.

To add a remote, if you are using ssh keys:


git remote add escomp git@github.com:ESCOMP/CTSM.git

Then to fetch all the changes and tags, and then apply them to the branch

git fetch escomp
git merge release-cesm2.2.03. # (This will require you to resolve conflicts in files due to the merge and fix them, you'll also need to make sure it's all working)
git status
git commit # Only required if there were conflicts that you had to resolve)
./manage_externals/checkout_externals

 

[jinmuluo]

I don't know, sorry. Pinging @slevis and @erik here because they are more familiar with these github operations than I am.

Hi Keith,

I have another question.
After I merged my code with the release-cesm2.2.03 tag, the new cime tool couldn't recognize many namelists I had before. I guess it's because I now have the cime tool designed for the cesm not the ctsm? Could you please tell me what the right cesm namelists for the the ctsm namelists like DATM_YR_ALIGN, DATM_YR_START and DATM_YR_END?

Thank you!

 

[oleson]

I'm not sure I understand the question, but DATM_YR_ALIGN, DATM_YR_START, and DATM_YR_END are the more recent versions and are the result of switching from mct to nuopc. The old versions were DATM_CLMNCEP_YR_ALIGN, DATM_CLMNCEP_YR_START, DATM_CLMNCEP_YR_END.

 

[jinmuluo]

I'm not sure I understand the question, but DATM_YR_ALIGN, DATM_YR_START, and DATM_YR_END are the more recent versions and are the result of switching from mct to nuopc. The old versions were DATM_CLMNCEP_YR_ALIGN, DATM_CLMNCEP_YR_START, DATM_CLMNCEP_YR_END.

Thank you for your explanations! DATM_CLMNCEP_YR_ALIGN, DATM_CLMNCEP_YR_START, DATM_CLMNCEP_YR_END work well in my case! But when I build the case, it has an error like below.


ERROR: Command /glade/u/home/jinmuluo/CLM-FANv3tag01/bld/build-namelist failed rc=255


out=


err=ERROR : CLM build-namelist::CLMBuildNamelist::setup_logic_lnd_frac() : -lnd_frac was set as UNSET in the CTSM build-namelist set it with the env variables: LND_DOMAIN_PATH/LND_DOMAIN_FILE.

Any idea of this error?

my case: /glade/work/jinmuluo/cases/DerechoTest
my setup: /glade/u/home/jinmuluo/LNDATM-IN/CaseSetUpMESOCOM_cesm.sh

Thank you!

 

[oleson]

It is asking you to set LND_DOMAIN_PATH and LND_DOMAIN_FILE (these are variables in env_run.xml). I assume this is occurring because you've set "--res CLM_USRDAT" in your create_newcase command.

 

[jinmuluo]

It is asking you to set LND_DOMAIN_PATH and LND_DOMAIN_FILE (these are variables in env_run.xml). I assume this is occurring because you've set "--res CLM_USRDAT" in your create_newcase command.

Thank you! If my case is a single-site simulation, which file it should point to?

 

[oleson]

Is this the first time you've run single-point? If so, you'd need to create a surface dataset and domain file for the gridcell you want to simulate. Some instructions are provided here:

1.6.3. Running Single Point Configurations — ctsm release-clm5.0 documentation