regional single-point test run

[wvsi3w]

Hello,
I used the documentation of running single point configuration for Alaska with CLM5 CESM 2.1.3 and the case I used has --res f19_g16 --compset I2000Clm50BgcCruGs and runs with my own machine config (that is tested and has worked with other compset).

I used this link info to make the case and after it downloaded all of the required input data I submitted the case successfully.

I have to mention that in some parts I had to manually download some domain and surface_map data using the following command:
wget https://svn-ccsm-inputdata.cgd.ucar...clm/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc -P /home/XXXXX/projects/YYYYY/XXXXX/inputdata/share/domains/ --no-check-certificate

But after submission, it failed with 1 min of running with the following message which appears in the log file multiple times:
MPI_ABORT was invoked on rank 55 in communicator MPI_COMM_WORLD with errorcode 1001. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

I searched the forum and couldn't find the reason why I am having this error. I appreciate it if you let me know your opinion.

 

[oleson]

That's a pretty general error. Have you looked carefully at all of the log files to see if there is more information?

 

[wvsi3w]

That's a pretty general error. Have you looked carefully at all of the log files to see if there is more information?

Thanks for your response. Oh, I understand. I also see the below error in the land log file:
ERROR: ERROR in decompInitMod.F90 at line 169

I have attached all of the log files. I don't see any other error in them.

 

[oleson]

It looks like you have more processors assigned to the job (128) than the number of grid cells that you have (85). You can reduce the number of tasks/processors in env_mach_pes.xml.

 

[wvsi3w]

Thanks for your helpful answer.
I have tested the model with reduced number of task/processors in my env_mach_pes file. resubmitted the case and for the first time I used 40 instead of 64 and the land log file showed the following error:

Attempting to read GLACIER_REGION...
(GETFIL): attempting to find local file
surfdata_1.9x2.5_hist_16pfts_Irrig_CMIP6_simyr2000_c190304.nc
(GETFIL): using
/home/meisam/projects/def-hbeltram/meisam/inputdata/lnd/clm2/surfdata_map/relea
se-clm5.0.18/surfdata_1.9x2.5_hist_16pfts_Irrig_CMIP6_simyr2000_c190304.nc
ncd_getiodesc ERROR in vsize 13824 156
88
ERROR:
ERROR in /home/meisam/my_cesm_sandbox/components/clm/src/main/ncdio_pio.F90.in
at line 2388

for the third time I tested it with 4 tasks and mpi tasks per node (instead of 40 which I used for the previous run), and in the land log file it shows the following error:

Attempting to read GLACIER_REGION...
(GETFIL): attempting to find local file
surfdata_1.9x2.5_hist_16pfts_Irrig_CMIP6_simyr2000_c190304.nc
(GETFIL): using
/home/meisam/projects/def-hbeltram/meisam/inputdata/lnd/clm2/surfdata_map/relea
se-clm5.0.18/surfdata_1.9x2.5_hist_16pfts_Irrig_CMIP6_simyr2000_c190304.nc
ncd_getiodesc ERROR in vsize 13824 156
88
ERROR:
ERROR in /home/meisam/my_cesm_sandbox/components/clm/src/main/ncdio_pio.F90.in
at line 2388

I don't know why it shows the same error for two different numbers of tasks per node (and different mpi tasks per node) I have checked the input data which is stated in that error message and I can find it there, I don't know if it has an issue or it can not process it.

In the first error, it said the Number of processes (128) exceeds the number of land grid cells(85); how to set the env_mach_pes variables to fit that odd number? I use any number for the MAX TASKS PER NODE and MAX MPI TASKS PER NODE and it doubles it, which in the first run these two variables were 64 and the number of processes was 128. For instance, here I used 4:

<group id="mach_pes_last">
    <entry id="COST_PES" value="8">
      <type>integer</type>
      <desc>pes or cores used relative to MAX_MPITASKS_PER_NODE for accounting (0 means TOTALPES is valid)</desc>
    </entry>
    <entry id="TOTALPES" value="8">
      <type>integer</type>
      <desc>total number of physical cores used (setup automatically - DO NOT EDIT)</desc>
    </entry>
    <entry id="MAX_TASKS_PER_NODE" value="4">
      <type>integer</type>
      <desc>maximum number of tasks/ threads allowed per node </desc>
    </entry>
    <entry id="MAX_MPITASKS_PER_NODE" value="4">
      <type>integer</type>
      <desc>pes or cores per node for mpitasks </desc>
    </entry>

 

[wvsi3w]

I did all of the steps again from the beginning and it shows the same error using 4 processors. The log files are attached.

 

[slevis]

This still seems like a task geometry issue to me.

What if you try creating the case with
--mpilib mpi-serial
and running on 1 cpu?

If that doesn't work, then @erik may have a suggestion and/or you might ask a system admin for the machine that you are using.

 

[oleson]

It looks like you are using a 2deg surface dataset (144x96 = 13824) instead of the Alaska surface dataset (13x12 = 156)? Your domain file is 13x12.

 

[oleson]

In looking at this in more detail, it doesn't look like this example is supported by recent versions of the model (e.g., release-cesm2.1.3), sorry. The surface dataset for Alaska doesn't come out of the box anymore and the dataset that I found doesn't have GLACIER_REGION on it which is now required.

 

[wvsi3w]

In looking at this in more detail, it doesn't look like this example is supported by recent versions of the model (e.g., release-cesm2.1.3), sorry. The surface dataset for Alaska doesn't come out of the box anymore and the dataset that I found doesn't have GLACIER_REGION on it which is now required.

Thanks for your clarification. Yes, I am using the 2.1.3 version of the model. I used the steps in the link, and I realized there were some other input data that it couldn't download; I downloaded the following manually:

surfdata_1.9x2.5_hist_16pfts_Irrig_CMIP6_simyr2000_c190304.nc
surfdata_13x12pt_f19_alaskaUSA_simyr2000.nc
domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc

So is there a supported list of regional single-point tests for this version?

One other question (less related to the topic of this thread): I couldn't find step-by-step videos of setting up, configuring, and running the model. I believe if we had a series of videos explaining these steps for most of the discussed topics in the forum, it could save a lot of time and effort for everyone using this model. I have read and watched all of the tutorials available but most of them include slides and none of them show the real environment and the steps in real time. This video and other videos in this channel could be examples of what I (and probably many other students) have in mind.

For instance, for CESM we can have one video for each of the following topics:
-the porting process which is the major issue for people with zero knowledge of the model (like me) could be solved with one single video explaining how to do that on some other cluster using different configurations.
-quick run steps and solving some typical errors in it.
-changing the namelists for a desired simulation. Also changing other XML files for that desired simulation.
-running single point + regional study + using user defined input data + using different user defined "compset and res" +...
-coupling the model
-etc

 

[oleson]

In cime/scripts, you can run ./query_config --grids to get the available grids. For release-cesm2.1.3, I see "5x5_amazon" which is a regional grid over the amazon. There are also several supported single-point grids, e.g., 1x1_brazil.
It's a little tricky to find a compset and resolution combination that works, you may need several tries. This worked for me:

./create_newcase --compset 2000_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_MOSART_SGLC_SWAV --res 5x5_amazon --case I2000Clm50Sp_5x5_amazon --run-unsupported

I'm not sure if you've seen the most recent CESM tutorial material, which might be better than the previous material which had videos with slides:

Welcome to the CESM Tutorial — CESM Tutorial

 

[wvsi3w]

Hello everyone,
I forgot to respond to this, I actually did try that Amazon region you suggested and it worked, but there is a new thing I found.

The reason I had the initial problem with the Alaska test run was my model configuration (which was a nightmare doing it in Canadian systems, but we did it and in this link, we discussed it earlier).

recently I tested the Alaska case again (according to these steps) but it failed after submission and after downloading almost everything this message appeared which I guess is related to some missing input data on the server:

ERROR from ftp server, trying next server
Loading input file list: 'Buildconf/clm.input_data_list'
  Model clm missing file fatmlndfrc = '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc'
Trying to download file: 'share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' to path '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' using FTP protocol.
ERROR from ftp server, trying next server
Loading input file list: 'Buildconf/cpl.input_data_list'
Checking server https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata with protocol svn
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol svn with user  and passwd
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
  Model datm missing file domainfile = '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc'
Trying to download file: 'share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' to path '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' using SVN protocol.
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' doesn't exist

Loading input file list: 'Buildconf/clm.input_data_list'
  Model clm missing file fatmlndfrc = '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc'
Trying to download file: 'share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' to path '/home/XXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' using SVN protocol.
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/share/domains/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc' doesn't exist

Loading input file list: 'Buildconf/cpl.input_data_list'
Checking server None with protocol None
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol None not enabled
ERROR: Could not find all inputdata on any server

I thought I should mention it because every step worked and it failed to download the above-mentioned files (maybe due to their inaccessibility).

The steps I took are:

./create_newcase -case $CASES2_1_3/my_userdataset_test -res CLM_USRDAT -compset I2000Clm50BgcCruGs --machine beluga --compiler intel --mpilib intelmpi --walltime 02:00:00 --run-unsupported

cd to the case path

#setenv doesnt work so we need to do export
export GRIDNAME=13x12pt_f19_alaskaUSA
export LMASK=gx1v6

./xmlchange CLM_USRDAT_NAME=$GRIDNAME,CLM_BLDNML_OPTS="-mask $LMASK"

./xmlchange ATM_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc

./xmlchange LND_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc

# if there is no file regarding Alaska:
wget https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/surfdata_map/surfdata_13x12pt_f19_alaskaUSA_simyr2000.nc -P ~/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/lnd/clm2/surfdata_map/ --no-check-certificate

./case.setup

#to use my own soil layer structure
emacs user_nl_clm
soil_layerstruct= '23SL_3.5m_D500'

&clm_inparm
    hist_empty_htapes = .true.
    hist_fincl1 = 'TSOI', 'TSOI_ICE'
/

fsurdat = '/home/XXXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/lnd/clm2/surfdata_map/surfdata_13x12pt_f19_alaskaUSA_simyr2000.nc'

./case.build
./case.submit

 

[slevis]

My sense is that you're asking for files that do not exist in the svn repository. If these files exist locally on your machine, then you should put their paths together with their names, otherwise the model will not know to look for them on your machine.

 

[wvsi3w]

My sense is that you're asking for files that do not exist in the svn repository. If these files exist locally on your machine, then you should put their paths together with their names, otherwise the model will not know to look for them on your machine.

Thank you very much for the response,
I thought that the error message was related to the data not being accessible, sorry, my mistake.

The reason I got lost in the examples of this link (1.6.3. Running Single Point Configurations — ctsm release-clm5.0 documentation) is that it starts with Alaska and then finishes with ./case.setup then jumps into another example (1x1_boulderCO). Also in the Alaska example, it doesn't say anything about the domain file (the error I encountered), which I was mistakenly thinking since Alaska is one of the supported single points it will download it itself.

So, I think I should make a domain file for Alaska by using the instructions in the next example (1.6.3. Running Single Point Configurations — ctsm release-clm5.0 documentation) ???

 

[wvsi3w]

out of curiosity (and apologies for trivial questions), is this the same path I should take to run any other single-point simulations (I provided the steps in Q form below)? Let's say I want to simulate this point (lat 75, lon 270), and with biogeochemistry(and GSWP3 forcing),

Q1: so first I should create the case like this:

./create_newcase --case my_userdataset_test --res CLM_USRDAT --compset IHistClm50BgcCrop--machine beluga --compiler intel --mpilib intelmpi --walltime 02:00:00 --run-unsupported

Q2: Or as Keith once suggested for single-points for cold start runs maybe I should use the long name (HIST_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV)

Q3: then set the GRIDNAME and MAKS as desired, in my case, the GRIDNAME would be 1x1_lat75_lon270 (or there are strict naming instructions?) LMASK=nomask (I think nomask only applies to where we have a grid cell near the ocean (water), so if our grid is like 1x1_boulderCO, which is inland, no need for a mask) (?)

Q4: Update XML Files

./xmlchange CLM_USRDAT_NAME=$GRIDNAME,CLM_BLDNML_OPTS="-mask $LMASK"
./xmlchange ATM_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc
./xmlchange LND_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc

Q5: Create the SCRIP Grid and Mapping Files

cd $CTSMROOT/tools/mkmapdata
./mknoocnmap.pl -p 75,270 -n $GRIDNAME
export MAPFILE=`pwd`/map_${GRIDNAME}_noocean_to_${GRIDNAME}_nomask_aave_da_${CDATE}.nc
cd ../mkmapgrids/
export GRIDFILE=../mkmapgrids/SCRIPgrid_${GRIDNAME}_nomask_${CDATE}.nc
cd ../mkmapdata/
./mkmapdata.sh -r $GRIDNAME -f $GRIDFILE -t regional

Q6: Create Domain Files:

cd ../../../../tools/mapping/gen_domain_files/src
../../../scripts/ccsm_utils/Machines/configure -mach beluga -compiler intel
gmake
cd ..
export OCNDOM=domain.ocn_noocean.nc
export ATMDOM=domain.lnd.${GRIDNAME}_noocean.nc
./gen_domain -m $MAPFILE -o $OCNDOM -l $ATMDOM

Q7:
./case.setup

Modify the user_nl_clm as I need:

emacs user_nl_clm
soil_layerstruct= '23SL_3.5m_D500'

&clm_inparm
    hist_empty_htapes = .true.
    hist_fincl1 = 'TSOI', 'TSOI_ICE'
/

fsurdat = '/home/XXXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/lnd/clm2/surfdata_map/surfdata_XXXX.nc

./case.build

./case.submit

and last question (Q8) if anyone wants to change the forcing (from GSWP3 to something else) the steps taken in this thread (Run CLM with my own atmosphere forcing data) are enough to know how to start?

 

[wvsi3w]

Thank you very much for the response,
I thought that the error message was related to the data not being accessible, sorry, my mistake.

The reason I got lost in the examples of this link (1.6.3. Running Single Point Configurations — ctsm release-clm5.0 documentation) is that it starts with Alaska and then finishes with ./case.setup then jumps into another example (1x1_boulderCO). Also in the Alaska example, it doesn't say anything about the domain file (the error I encountered), which I was mistakenly thinking since Alaska is one of the supported single points it will download it itself.

So, I think I should make a domain file for Alaska by using the instructions in the next example (1.6.3. Running Single Point Configurations — ctsm release-clm5.0 documentation) ???

I am really embarrassed to say that I fixed this with this line below:

wget https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/datm7/domain.clm/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc -P ~/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/ --no-check-certificate

the domain file was in another link (mentioned above) and I only needed to download it. please ignore the message I replied to.

The case is submitted now and I hope it runs as we want.

 

[wvsi3w]

I am really embarrassed to say that I fixed this with this line below:

wget https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/datm7/domain.clm/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc -P ~/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/ --no-check-certificate

the domain file was in another link (mentioned above) and I only needed to download it. please ignore the message I replied to.

The case is submitted now and I hope it runs as we want.

It failed with this error message:

ERROR: GLACIER_REGION NOT on surfdata fileERROR in glcBehaviorMod.F90 at line

So I think based on this thread, I think the reason might be due to the mask (g16 vs g17) OR due to the user_nl_clm that I edited (mentioned above in previous messages). I don't know which.

 

[slevis]

It may help to list everything that you have changed along the way to figure out why you get this error. I assume that you do not get the error when you try a default case "out of the box."

 

[wvsi3w]

Hello @slevis and thank you for your response,
I have written what you requested below but the most important part is the question I have written at the end:

The modification that I have done was the soil layer structure to be 500m depth (which I have tested before with global and single-point test runs) and I only have added that to the user_nl_clm file (plus the variables I would like to have in output directory which are TSOI and TSOI_ICE and the fsurdat path):

./create_newcase -case $CASES2_1_3/my_userdataset_test -res CLM_USRDAT -compset I2000Clm50BgcCruGs --machine beluga --compiler intel --mpilib intelmpi --walltime 02:00:00 --run-unsupported

cd to the case path

#setenv doesnt work so we need to do export
export GRIDNAME=13x12pt_f19_alaskaUSA
export LMASK=gx1v6

./xmlchange CLM_USRDAT_NAME=$GRIDNAME,CLM_BLDNML_OPTS="-mask $LMASK"

./xmlchange ATM_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc

./xmlchange LND_DOMAIN_FILE=domain.lnd.${GRIDNAME}_$LMASK.nc

# if there is no file regarding Alaska:
wget https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/surfdata_map/surfdata_13x12pt_f19_alaskaUSA_simyr2000.nc -P ~/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/lnd/clm2/surfdata_map/ --no-check-certificate

./case.setup

#to use my own soil layer structure
emacs user_nl_clm
soil_layerstruct= '23SL_3.5m_D500'

&clm_inparm
    hist_empty_htapes = .true.
    hist_fincl1 = 'TSOI', 'TSOI_ICE'
/

fsurdat = '/home/XXXX/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/lnd/clm2/surfdata_map/surfdata_13x12pt_f19_alaskaUSA_simyr2000.nc'

./case.build

#then the domain file I found on the website and downloaded it
wget https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/datm7/domain.clm/domain.lnd.13x12pt_f19_alaskaUSA_gx1v6.nc -P ~/scratch/cime_INPUT_DATA/cesm2_1_3/inputdata/share/domains/ --no-check-certificate

./case.submit

#and it did run without issues but failed with this error:
ERROR: GLACIER_REGION NOT on surfdata fileERROR in glcBehaviorMod.F90 at line

Then I ran it without those changes in the user namelist file and only the fsurdat path is there in it and it failed with the same message (so I don't think those lines in user_nl_clm were related to the error) so is it related to the fact that gx1v6 is not supported for this anymore and the this Alaska was for previous versions of the model? or maybe there was something I missed from this link guidance?:



In case this error is related to the fact that the Alaska thing is not supported, I have some questions (because I was trying to run a single-point simulation for a random grid (in northern Canada) and I actually did it with PTS_MODE but since PTS_MODE is not ideal for lengthy runs and it doesn't have restart option (as described here) I aimed to do it with CLM_USRDAT_NAME method):


- In the first step where we set the res to "--res CLM_USRDAT" we then set only the mask to "LMASK=gx1v6" and "GRIDNAME=13x12pt_f19_alaskaUSA " so I guess since this is an out of box test and its domain, name, and input data is already available, the model understands the resolution easily and knows what to do (as it was defined for it beforehand) but let's say I want to try it for another region (like Lat75 Lon270) and I want to try it with 1x1 res then how should I set these res and mask and grid name? is there a strict convention for naming and setting them up (beside the description in single-point link)? I also know in this case I should make a domain file, but I need to understand these things clearly. Sorry, but the examples in this documentation link are not clear to me and create questions, moreover by reading other threads I found that most people already knew how to do these steps (that for me as a person with 0 knowledge of any model is a big deal) and makes my questions trivial, so here is my only hope to ask.
Thank you for your support.

P.S. The version of CESM is 2.1.3 which I am using.

 

[oleson]

My post from 7/13/23 in this same thread noted:

"In looking at this in more detail, it doesn't look like this example is supported by recent versions of the model (e.g., release-cesm2.1.3), sorry. The surface dataset for Alaska doesn't come out of the box anymore and the dataset that I found doesn't have GLACIER_REGION on it which is now required."

If you want to run a single-point or regional simulations, then you'll need to create datasets (domain files, surface datasets, etc.) for them, following instructions given here:

1.3.4. Creating input for surface dataset generation — ctsm release-clm5.0 documentation

Your post on 8/27/24 seemed to indicate that you were on this path to create a single point domain file and surface dataset, but you've now gone back to trying to run the Alaska example using existing datasets that are outdated. I think you should try creating the single point domain file and surface dataset and then we can help you to run the model with them. You should update to release-cesm2.1.5 however, as that is the supported version of the release.