Issues creating new case on CAM Development Branch

[r_a_khan]

Hi,

I've recently received access to the development branch of CAM in order to run some 58L experiments. I downloaded the branch via:

$ git clone GitHub - JulioTBacmeister/CAM: Community Atmosphere Model newCAM
$ cd newCAM
$ git checkout cam_development
$ ./manage_externals/checkout_externals

I am running on Tetralith, so I have replaced the config_batch, config_machine, and config_compiler files in newCAM/ccs_config/machines/ with the Tetralith configuration files, and now I am creating a case with the F1850 compset and f09_f09_mg17 grid using:
./create_newcase --case .../cases/full_newCAM_1850 --compset F1850 --res f09_f09_mg17 --machine tetralith --project snic2022-1-1 --output-root .../newCAM-out

It seems to work, until I receive the following error:
....
*********************************************************************************************************************************
This compset and grid combination is not scientifically supported, however it is used in 4 tests.
*********************************************************************************************************************************

Using charge_account from config_machines.xml: bolinc
cesm model version found: cam6_3_057-34-g33f1d5c
Batch_system_type is slurm
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
Creating Case directory .../cases/full_newCAM_1850
ERROR: Directory None does not appear to be a valid case directory

Can someone direct me to the where the issue might be? There is apparently a problem with the case directory preventing it from being created but I'm not sure how to fix this.

Thanks,
Ramsha

 

[cacraig]

It looks like your case name has three dots instead of two. I believe that may be preventing your case directory from being created. Try it with two dots, and let us know if you still have an issue.

 

[r_a_khan]

Hi Cheryl, I included the dots in the post only so I wouldn't have to write out the full path, but in the actual script I include the whole path from my home directory. So the output is something like:

Creating Case directory /home/sm_ramkh/scam/cases/full_newCAM_1850
ERROR: Directory None does not appear to be a valid case directory

 

[nusbaume]

Hi Ramsha,

Could you possibly post the config files you modified for Tetralith? I just tried your create_newcase command with Julio's CAM branch on our machine (Cheyenne) and it ran without an issue.

Also, have you been able to get other simulations to run with these new machine config files, or is this your first attempt?

Thanks, and have a great day!

Jesse

 

[r_a_khan]

Hi Jesse,

Thank you for your help, I have attached the config files for Tetralith. As Tetralith does not have specific files for development CAM, I used the config files designed for CESM2.1.0, as listed here:

Building CESM 2.1.0 Tetralith - SNIC Documentation

docs.snic.se

I use these config files successfully for all simulations run using CESM2.1.0 and so CAM6.

Perhaps there is something else that needs to be modified?

Cheers,
Ramsha

 

[nusbaume]

Hi Ramsha,

Thanks for the config files! Sadly it looks like those files aren't the problem, so my next guess is that it is an issue with your (python) environment. What is the result when you type:

python3 --version

Also, do you know if you are running the script in any sort of custom (e.g. conda) environment? If so, can you send me the relevant environment info (e.g. the environment file, or the results of conda list -e)?

Finally, can you try running the create_newcase command again, but this time with the --debug flag as well? That should produce a "create_newcase.log" file, which would also be helpful if you could include it in this thread.

Thanks, and have a great day!

Jesse

 

[r_a_khan]

Hi Jesse,

Thanks for your suggestions, I made some progress! I had not realised the base conda environment activated by default, which I then deactivated and loaded the appropriate modules directly:
Currently Loaded Modules:
1) mpprun/4.2.1 5) Python/3.8.3-anaconda-2020.07-extras-nsc1
2) nsc/.1.1 (H,S) 6) git/2.19.3-nsc1-gcc-system
3) buildenv-intel/2018.u1-bare 7) subversion/1.11.1-nsc1-gcc-2018a-eb
4) netCDF-HDF5/4.6.2-nsc1-parallel-private_hdf5-intel20018a-eb

This allowed the case to be created and set up successfully.

However, I now run into problems for the case build, specifically in the preview_namelist command. Here is the output for that:

$ ./preview_namelists
2022-08-31 10:49:24 atm
Create namelist for component cam
Calling /home/sm_ramkh/newCAM/cime_config/buildnml
...calling cam buildcpp to set build time options
CAM namelist copy: file1 /home/sm_ramkh/scam/work/cases/test_newcam/Buildconf/camconf/atm_in file2 /proj/bolinc/users/sm_ramkh/newCAM-out/test_newcam/run/atm_in
2022-08-31 10:49:24 lnd
Create namelist for component clm
Calling /home/sm_ramkh/newCAM/components/clm//cime_config/buildnml
2022-08-31 10:49:25 ice
Create namelist for component cice
Calling /home/sm_ramkh/newCAM/components/cice//cime_config/buildnml
RUN: /home/sm_ramkh/newCAM/components/cice/bld/generate_cice_decomp.pl -ccsmroot /home/sm_ramkh/newCAM -res 0.9x1.25 -nx 288 -ny 192 -nproc 128 -thrds 1 -output all
FROM: /home/sm_ramkh/scam/work/cases/test_newcam
output: 288 192 9 8 6 sectrobin square-ice

2022-08-31 10:49:25 ocn
Create namelist for component docn
Calling /home/sm_ramkh/newCAM/components/cdeps/docn/cime_config/buildnml
File not found: model_meshfile = /proj/cesm_input-data/inputdata//share/meshes/fv0.9x1.25_141008_polemod_ESMFmesh.nc, will attempt to download in check_input_data phase
File not found: model_maskfile = /proj/cesm_input-data/inputdata//share/meshes/gx1v7_151008_ESMFmesh.nc, will attempt to download in check_input_data phase
docn_mode is prescribed
2022-08-31 10:49:25 rof
Create namelist for component mosart
Calling /home/sm_ramkh/newCAM/components/mosart//cime_config/buildnml
2022-08-31 10:49:25 glc
Create namelist for component sglc
Calling /home/sm_ramkh/newCAM/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
2022-08-31 10:49:25 wav
Create namelist for component swav
Calling /home/sm_ramkh/newCAM/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
2022-08-31 10:49:25 esp
Create namelist for component sesp
Calling /home/sm_ramkh/newCAM/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
2022-08-31 10:49:25 cpl
Create namelist for component drv
Calling /home/sm_ramkh/newCAM/components/cmeps/cime_config/buildnml
ERROR: ESMFMKFILE not found None

I had successfully checked out externals when I initially downloaded the branch, but i tried again just now, and this time I get an error:

$ ./manage_externals/checkout_externals --verbose
Processing externals description file : Externals.cfg
Processing externals description file : Externals.cfg
Processing externals description file : .gitmodules
Processing submodules description file : .gitmodules
Processing externals description file : Externals_CDEPS.cfg
Processing externals description file : .gitmodules
Processing submodules description file : .gitmodules
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_FMS.cfg
Processing externals description file : Externals_CAM.cfg
Checking status of externals: ccs_config, cice5, cice6, cice, icepack, cmeps, cdeps, fox, genf90, cpl7, share, mct, parallelio, cime, cism, source_cism, clm, fates, fms, fms, mosart, rtm, cam, chem_proc, ERROR:root:SVN returned invalid XML message

ERROR: SVN returned invalid XML message

Is there a way I can check what's going wrong? It worked the first time without issues (I had to manually enable some permissions but after that the checkout was successful).

Thank you for your help, I really appreciate it!
Ramsha

 

[r_a_khan]

UPDATE:

To ensure there wasn't an issue with virtual environments, I cloned the branch from scratch, checked out externals successfully, replaced config files for tetralith, and then successfully created and set up a case. However in the case build I encountered the exact same error message.

 

[nusbaume]

Hi Ramsha,

Apologies for the delayed response! Do you know if ESMF is installed on Tetralith? If so then there should be an esmf.mk file associated with it, which you can tell CESM to use by adding the following lines to your Tetralith entry in config_machines.xml:

<environment_variables comp_interface="nuopc">
    <env name="ESMFMKFILE">/Path/to/esmf/makefile/esmf.mk</env>
</environment_variables>

If ESMF is not installed, then you will either need to install it yourself, or have the Tetralith admins install it for you. The ESMF source code, along with instructions, can be found on Github here:

GitHub - esmf-org/esmf at v8.3.0

The Earth System Modeling Framework (ESMF) is a suite of software tools for developing high-performance, multi-component Earth science modeling applications. - GitHub - esmf-org/esmf at v8.3.0

github.com

Hope that helps, and good luck!

Jesse