Running T42_T42 Grid Resolution on Derecho

skramer · Jun 7, 2024

Hello,

I am trying to run an experiment using the F2000climo reference case with a T42_T42 resolution instead of f09_f09_mg17 resolution, i.e. I want a low-resolution equivalent version of my previous experiment. I created a new case like so:
./create_newcase -case /glade/u/home/skramer/casefiles/2000_docn_LR_ctrl -compset=2000_CAM60_CLM50%BGC-CROP_CICE%PRES_DOCN%DOM_MOSART_CISM2%NOEVOLVE_SWAV -res=T42_T42 -mach=derecho -project=UCUB0114 --run-unsupported
which was successful. The only issue is when I went into my case 2000_docn_LR_ctrl there are no user_nl_* files. Does anyone know why these files are missing when I try to create a case with the T42_T42 lower resolution, I have never had this issue creating a case with the f09_f09_mg17 resolution. Has anyone else tried running an experiment on Derecho with the T42_T42 resolution and ran into any other issues on top of this that I may expect to run into?

Thank you for any input you can provide,

Sydney Kramer

jedwards · Jun 13, 2024

Hi Sydney - the low resolution grid that you want to try is f19_f19_mg37.

skramer · Jun 13, 2024

Hi Jim,

I would like to use T42_T42.

jedwards · Jun 13, 2024

T42 is not a supported grid for the experiment you want to do.

skramer · Jun 13, 2024

I am confused because T42_T42 is listed as a defined grid resolution for CESM2.2.0 but I don't see any component sets that "scientifically support" a T42_T42 grid resolution. Can you only run the component sets with their "scientifically supported" grid resolutions?

skramer · Jul 4, 2024

jedwards said:
T42 is not a supported grid for the experiment you want to do.

Hi Jim,

I have still not found an answer for this issue. I see there are many compsets: CESM2.2.0 Component Sets Definitions that have no listed scientifically supported grids. Does this mean these compsets aren't available at all on Derecho? I am still trying to run an experiment as similar as possible to the F2000climo compset with a lower resolution grid and am not finding any options.

jedwards · Jul 4, 2024

- the low resolution grid that you want to try is f19_f19_mg37.

skramer · Jul 4, 2024

jedwards said:
- the low resolution grid that you want to try is f19_f19_mg37.

f19_f19_mg17or f25_f25_mg17 if that is the lowest available but T42_T42 is the best for my experiment

skramer · Jul 12, 2024

jedwards said:
- the low resolution grid that you want to try is f19_f19_mg37.

Hi Jim,

Following up on this again since it is vital I figure out the low resolution equivalent experiments soon for revisions. Is there any way on Derecho to run something as similar as possible to the F200climo compset with either a f19_f19_mg17, f25_f25_mg17, or T42_T42 grid resolution?

skramer · Aug 1, 2024

jedwards said:
- the low resolution grid that you want to try is f19_f19_mg37.

I've now tried both the f19 and T42 grids but user_nl files were still not created and I'm not sure where to go from here.

jedwards · Aug 6, 2024

I just tried
./create_newcase --case foo --compset F2000climo --res f19_f19_mg17

I'm not seeing any issues:

Code:

 ./create_newcase --case sydney --compset F2000climo --res f19_f19_mg17
Compset longname is 2000_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV
Compset specification file is /glade/work/jedwards/sandboxes/cesm2.2/components/cam//cime_config/config_compsets.xml
Automatically adding SIAC to compset
Automatically adding SESP to compset
Warning: missing non-idmap ROF2OCN_FMAPNAME for rof_grid, r05 and ocn_grid 1.9x2.5
Compset forcing is 1972-2004
ATM component is CAM cam6 physics:
LND component is clm5.0:Satellite phenology:
ICE component is Sea ICE (cice) model version 5 :prescribed cice
OCN component is DOCN   prescribed ocean mode
ROF component is MOSART: MOdel for Scale Adaptive River Transport
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
IAC component is Stub iac component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /glade/work/jedwards/sandboxes/cesm2.2/components/cam//cime_config/config_pes.xml
Compset specific settings: name is SSTICE_DATA_FILENAME and value is $DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_2000climo_c180511.nc
Compset specific settings: name is SSTICE_GRID_FILENAME and value is $DIN_LOC_ROOT/atm/cam/ocnfrac/domain.camocn.1.9x2.5_gx1v6_090403.nc
Machine is derecho
Pes setting: grid match    is a%1.9x2.5
Pes setting: grid          is a%1.9x2.5_l%1.9x2.5_oi%1.9x2.5_r%r05_g%null_w%null_z%null_m%gx1v7
Pes setting: compset       is 2000_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SIAC_SESP
Pes setting: tasks       is {'NTASKS_ATM': -2, 'NTASKS_LND': -2, 'NTASKS_ROF': -2, 'NTASKS_ICE': -2, 'NTASKS_OCN': -2, 'NTASKS_GLC': -2, 'NTASKS_WAV': -2, 'NTASKS_CPL': -2}
Pes setting: threads     is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_GLC': 1, 'NTHRDS_WAV': 1, 'NTHRDS_CPL': 1}
Pes setting: rootpe      is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 0, 'ROOTPE_GLC': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0}
Pes setting: pstrid      is {}
Pes other settings: {}
Pes comments: none
 Compset is: 2000_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SIAC_SESP
 Grid is: a%1.9x2.5_l%1.9x2.5_oi%1.9x2.5_r%r05_g%null_w%null_z%null_m%gx1v7
 Components in compset are: ['cam', 'clm', 'cice', 'docn', 'mosart', 'sglc', 'swav', 'siac', 'sesp']

*********************************************************************************************************************************
This compset and grid combination is not scientifically supported, however it is used in 2 tests.
*********************************************************************************************************************************

Using project from env PROJECT: P93300606
No charge_account info available, using value from PROJECT
cesm model version found: release-cesm2.2.2
Batch_system_type is pbs
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
 Creating Case directory /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/env_case.xml -> /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/LockedFiles
derecho1: ~/sandboxes/cesm2.2/cime/scripts
:) cd sydney/
derecho1: ~/sandboxes/cesm2.2/cime/scripts/sydney
:) ./case.setup
Setting Environment OMP_STACKSIZE=64M
Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
Setting Environment FI_MR_CACHE_MONITOR=memhooks
Setting Environment MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=4:striping_unit=16777216
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/config/cesm/machines/Depends.intel -> /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney
/glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/env_mach_specific.xml already exists, delete to replace
/glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/env_mach_specific.xml already exists, delete to replace
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
Creating batch scripts
Writing case.run script from input template /glade/work/jedwards/sandboxes/cesm2.2/cime/config/cesm/machines/template.case.run
Creating file .case.run
Writing case.st_archive script from input template /glade/work/jedwards/sandboxes/cesm2.2/cime/config/cesm/machines/template.st_archive
Creating file case.st_archive
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/env_mach_pes.xml -> /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/LockedFiles
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/env_batch.xml -> /glade/work/jedwards/sandboxes/cesm2.2/cime/scripts/sydney/LockedFiles
Creating user_nl_xxx files for components and cpl
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/src/drivers/mct/cime_config/user_nl_cpl -> .
copying /glade/work/jedwards/sandboxes/cesm2.2/components/cam//cime_config/user_nl_cam ->
copying /glade/work/jedwards/sandboxes/cesm2.2/components/clm//cime_config/user_nl_clm ->
copying /glade/work/jedwards/sandboxes/cesm2.2/components/cice//cime_config/user_nl_cice ->
copying /glade/work/jedwards/sandboxes/cesm2.2/cime/src/components/data_comps_mct/docn/cime_config/user_nl_docn ->
copying /glade/work/jedwards/sandboxes/cesm2.2/components/mosart//cime_config/user_nl_mosart ->
If an old case build already exists, might want to run 'case.build --clean' before building
You can now run './preview_run' to get more info on how your case will be run

skramer · Aug 7, 2024

Hi Jim,

While I can create the case and successfully build it with the f19_f19_mg17 grid resolution (still not possible for f25_f25_mg17 or T42_T42), it is still not able to be run. The reference case uses <entry id="RUN_REFCASE" value="b.e20.BHIST.f09_g17.20thC.297_01_v2"> which has a f09_f09_mg17 resolution. There is no available reference case (found here: /glade/p/cesmdata/inputdata/cesm2_init) that runs with a f19_f19_mg17 resolution. Because the grids don't match I get the error "application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 125
dec1362.hsn.de.hpc.ucar.edu: rank 128 exited with code 255" seen here /glade/derecho/scratch/skramer/2000_docn_LR_f19res_ctrl/run/cesm.log.5567402.desched1.240807-083504

The built case can be found here: /glade/u/home/skramer/casefiles/2000_docn_LR_f19res_ctrl

The ./create_newcase properly put f19_f19_mg17 resolution files in place for:
<entry id="SSTICE_DATA_FILENAME" value="$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_2000climo_c180511.nc">
<entry id="SSTICE_GRID_FILENAME" value="$DIN_LOC_ROOT/atm/cam/ocnfrac/domain.camocn.1.9x2.5_gx1v6_090403.nc ">
but the rpointer files pull from b.e20.BHIST.f09_g17.20thC.297_01_v2 which has the higher grid resolution.

jedwards · Aug 7, 2024

Where did you get the REF case? It's not defined for this compset and resolution. You would need to interpolate the
ref case to the resolution you want to run at -- I'm not even sure if tools to do that exist.

skramer · Aug 7, 2024

The reference case is the one I used for the previous f09 resolution experiment. All available references cases can be found here: glade/p/cesmdata/inputdata/cesm2_init

I essentially need a reference case that looks like:
b.e20.BHIST.f19_g17.20thC.297_01_v2

Instead of

b.e20.BHIST.f09_g17.20thC.297_01_v2

But this isn’t available.

jedwards · Aug 7, 2024

Try this one - /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020
no guarantees that it will work

skramer · Aug 7, 2024

While building a new case using that recommended reference case I received this error:

Code:

Building case in directory /glade/u/home/skramer/casefiles/2000_docn_LR_ctrl
sharedlib_only is False
model_only is False
Setting Environment OMP_STACKSIZE=64M
Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
Setting Environment FI_MR_CACHE_MONITOR=memhooks
Setting Environment MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=24
Generating component namelists as part of build
Refcase not found in /glade/campaign/cesm/cesmdata/inputdata/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020/2000-01-01, will attempt to download from inputdata
Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp
Setting Environment OMP_STACKSIZE=64M
Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
Setting Environment FI_MR_CACHE_MONITOR=memhooks
Setting Environment MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=24
Client protocol gftp not enabled
Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata/ with protocol wget
Setting Environment OMP_STACKSIZE=64M
Setting Environment FI_CXI_RX_MATCH_MODE=hybrid
Setting Environment FI_MR_CACHE_MONITOR=memhooks
Setting Environment MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=24
Using protocol wget with user anonymous and passwd user@example.edu
Loading input file list: 'Buildconf/refcase.input_data_list'
Model refcase missing file refdir = '/glade/campaign/cesm/cesmdata/inputdata/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020/2000-01-01/'
Trying to download file: 'cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020/2000-01-01' to path '/glade/campaign/cesm/cesmdata/inputdata/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020/2000-01-01' using WGET protocol.
Creating directory /glade/campaign/cesm/cesmdata/inputdata/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020/2000-01-01
Traceback (most recent call last):
File "/glade/u/home/skramer/casefiles/2000_docn_LR_ctrl/./case.build", line 165, in <module>
_main_func(__doc__)
File "/glade/u/home/skramer/casefiles/2000_docn_LR_ctrl/./case.build", line 157, in _main_func
success = build.case_build(caseroot, case=case, sharedlib_only=sharedlib_only,
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 728, in case_build
return run_and_log_case_status(functor, cb, caseroot=caseroot)
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1768, in run_and_log_case_status
rv = func()
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 721, in <lambda>
functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist,
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 660, in _case_build_impl
sharedpath = _build_checks(case, build_threaded, comp_interface,
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 326, in _build_checks
case.create_namelists()
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/case/preview_namelists.py", line 63, in create_namelists
self.stage_refcase()
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 234, in stage_refcase
success = _downloadfromserver(self, input_data_root=input_data_root, data_list_dir=data_list_dir)
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 191, in _downloadfromserver
success = case.check_input_data(protocol=protocol, address=address, download=True,
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 367, in check_input_data
no_files_missing = _download_if_in_repo(server,
File "/glade/u/home/skramer/derecho_cases/cesm2.2.2/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 130, in _download_if_in_repo
os.makedirs(full_path+".tmp")
File "/glade/u/apps/derecho/23.09/opt/._view/ki3gspoeflu225iuymcch2cddghxcain/lib/python3.10/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/glade/u/apps/derecho/23.09/opt/._view/ki3gspoeflu225iuymcch2cddghxcain/lib/python3.10/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/glade/u/apps/derecho/23.09/opt/._view/ki3gspoeflu225iuymcch2cddghxcain/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/glade/campaign/cesm/cesmdata/inputdata/cesm2_0'

I altered the env_run.xml file to have

Code:

<entry id="RUN_REFDIR" value="cesm2_0">
      <type>char</type>
      <desc>                                                                                                                                 
      Reference directory containing RUN_REFCASE data - used for hybrid or branch runs                                                       
    </desc>
    </entry>
    <entry id="RUN_REFCASE" value="b.e20.BHIST.f19_g17.release_cesm2_1_0.020">
      <type>char</type>
      <desc>                                                                                                                                 
      Reference case for hybrid or branch runs

It doesn't appear to be pulling the reference case from the same file path you included, i.e. /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020

but I can't control where it pulls the file path from aside from changing the RUN_REFDIR from cesm2_init to cesm2_0.

jedwards · Aug 7, 2024

set GET_REFCASE=FALSE
and copy the files to your run directory.

skramer · Aug 7, 2024

jedwards said:
set GET_REFCASE=FALSE
and copy the files to your run directory.

I looked further into the suggested /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e20.BHIST.f19_g17.release_cesm2_1_0.020
but this will also not work because if you look at the env_run_xml file the reference case that b.e20.BHIST.f19_g17.release_cesm2_1_0.020 uses is a file that is B1850. I need forcing to be at 2000, that is why I was using the 20thC reference case, which does not exist for the f19 grid resolution. I checked all other /b.e20.BHIST.f19_g17. options in that file path and the rest are CMIP6-piControl which also doesn't match up with what I need.

Running T42_T42 Grid Resolution on Derecho

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

Sydney Kramer

New Member

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member

CSEG and Liaisons

Sydney Kramer

New Member