Background
We are trying to run CESM2 job on Cheyenne, version release-cesm2.1.3, WACCM MA. The goal is to conduct a simulation of stratospheric aerosol geoengineering, branching from a "background" run in 2035. The first year of simulation runs successfully, but every time we submit the second year, the run crashes immediately. The cesm.log (attached below) shows the following error message:
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/dst/dst_source2x2tunedcam6-2x2-0406201
1: 7.nc 3
7:forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Details
The simulation was created using the following command:
./create_clone --clone /glade/work/visioni/cases/b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.SSP245-MA-GAUSS-LOWER-1.0.001 --case /glade/work/walkerl/cases/b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.feedback_60N_60S.001 --cime-output-root /glade/scratch/walkerl
After creating the case, we made three changes:
1. user_nl_cam (attached) was modified; specifically, in the section "ext_forcing_specifier," we modified the SO2 pointers at the bottom of this section.
2. env_run.xml (attached) was modified, with the following changes:
RUN_REFCASE = "b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.001"
RUN_REFDATE = "2035-01-01"
RUN_STARTDATE = "2035-01-01"
3. After copying the necessary restart files and setting up and building the case, case.st_archive (attached) was modified so that after archiving the model output, it would call a Python script to examine the output, determine the next year's geoengineering strategy, and modify user_nl_cam with the new SO2 pointers. We are quite confident that this script is working correctly, as the current pointers in user_nl_cam show the correct pointers as computed after the first year.
Problem
The simulation is set to run for 12 months and not resubmit, and the first year always runs correctly. After the first year, we update env_run.xml:
RUN_REFCASE = "b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.feedback_60N_60S.001" (the current case)
RUN_REFDATE = "2036-01-01"
RUN_STARTDATE = "2036-01-01"
We also confirm that our Python scripts ran correctly, and that user_nl_cam updated the SO2 pointers accordingly. However, after resubmitting, the run crashes immediately, and the log shows the error above. We do not know how to deal with this error, and starting over while cloning from different background ensemble members does not seem to solve the problem. Any insight would be greatly appreciated; please contact me if you need more information. I can be reached at wl644@cornell.edu. Thanks!
We are trying to run CESM2 job on Cheyenne, version release-cesm2.1.3, WACCM MA. The goal is to conduct a simulation of stratospheric aerosol geoengineering, branching from a "background" run in 2035. The first year of simulation runs successfully, but every time we submit the second year, the run crashes immediately. The cesm.log (attached below) shows the following error message:
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/dst/dst_source2x2tunedcam6-2x2-0406201
1: 7.nc 3
7:forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Details
The simulation was created using the following command:
./create_clone --clone /glade/work/visioni/cases/b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.SSP245-MA-GAUSS-LOWER-1.0.001 --case /glade/work/walkerl/cases/b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.feedback_60N_60S.001 --cime-output-root /glade/scratch/walkerl
After creating the case, we made three changes:
1. user_nl_cam (attached) was modified; specifically, in the section "ext_forcing_specifier," we modified the SO2 pointers at the bottom of this section.
2. env_run.xml (attached) was modified, with the following changes:
RUN_REFCASE = "b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.001"
RUN_REFDATE = "2035-01-01"
RUN_STARTDATE = "2035-01-01"
3. After copying the necessary restart files and setting up and building the case, case.st_archive (attached) was modified so that after archiving the model output, it would call a Python script to examine the output, determine the next year's geoengineering strategy, and modify user_nl_cam with the new SO2 pointers. We are quite confident that this script is working correctly, as the current pointers in user_nl_cam show the correct pointers as computed after the first year.
Problem
The simulation is set to run for 12 months and not resubmit, and the first year always runs correctly. After the first year, we update env_run.xml:
RUN_REFCASE = "b.e21.BWSSP245.f09_g17.release-cesm2.1.3.WACCM-MA-1deg.feedback_60N_60S.001" (the current case)
RUN_REFDATE = "2036-01-01"
RUN_STARTDATE = "2036-01-01"
We also confirm that our Python scripts ran correctly, and that user_nl_cam updated the SO2 pointers accordingly. However, after resubmitting, the run crashes immediately, and the log shows the error above. We do not know how to deal with this error, and starting over while cloning from different background ensemble members does not seem to solve the problem. Any insight would be greatly appreciated; please contact me if you need more information. I can be reached at wl644@cornell.edu. Thanks!