Case run fail in Derecho

[dharmendraks841]

Hi

I found the following error:

2024-02-11 22:58:44: model execution starting

---------------------------------------------------

2024-02-11 22:58:53: model execution success

---------------------------------------------------

2024-02-11 22:58:53: case.run error

ERROR: RUN FAIL: Command 'mpiexec --label --line-buffer -n 1152 /glade/derecho/scratch/dksingh/bssp245smbb_hybrid33_buildtest/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed

See log file for details: /glade/derecho/scratch/dksingh/bssp245smbb_hybrid33_buildtest/run/cesm.log.3045822.desched1.240211-225833



I found the following Error in log file



ERROR:

dec0305.hsn.de.hpc.ucar.edu 659: GETFIL: FAILED to get b.e21.BHISTsmbb.f09_g17.LE2-1231.011.cam.i.2033-01-01-000

dec0305.hsn.de.hpc.ucar.edu 659: 00.nc

dec0289.hsn.de.hpc.ucar.edu 84: ERROR:

dec0289.hsn.de.hpc.ucar.edu 84: GETFIL: FAILED to get b.e21.BHISTsmbb.f09_g17.LE2-1231.011.cam.i.2033-01-01-000





Kind regards

Dharmendra

 

[sacks]

From looking in your case, it looks like you have set up a case that uses the pre-existing reference case, b.e21.BHISTsmbb.f09_g17.LE2-1231.011, but then you ran ./xmlchange RUN_REFDATE=2033-01-01. I think that's the source of your problem, because you are asking for a reference / restart year that doesn't exist: that reference case only has a restart set for 2015-01-01.

If you'd like more help with this, can you please describe what you are trying to do and the steps you have taken?

 

[dharmendraks841]

Hi,
Thanks.
I am currently using the B compset to extract 6-hourly datasets for 2033 using the available restart file. I opted for this approach because it is a hybrid method, allowing the use of the reference date and starting date (both should be the same). My objective is to drive the F compset using the variables extracted from the B compset.

Here is an excerpt from my user_nl_cam:
! namelist_var = new_namelist_value
fincl2 = 'U','V','T','PS','Q','QFLX','SHFLX','TAUX','TAUY','SST','U10','LHFLX','PHIS','TS','ICEFRAC'
mfilt = 1,4
nhtfrq = 0,-6

Initially, I successfully ran the B compset, extracting the first set of 10 variables (fincl2 = 'U','V','T','PS','Q','QFLX','SHFLX','TAUX','TAUY','SST'). However, I encountered an error when attempting to include additional variables ('U10','LHFLX','PHIS','TS','ICEFRAC) as specified above.

For a more detailed understanding, you can refer to another case where I had a successful run at the beginning for 10 variables (fincl2 = 'U','V','T','PS','Q','QFLX','SHFLX','TAUX','TAUY','SST').
As follows:
.

dksingh@derecho2:/glade/work/dksingh/code_cesmupdated_derecho/bssp245smbb_hybrid33> ./xmlquery RUN_TYPE,RUN_REFCASE,RUN_REFDATE,GET_REFCASE

Results in group run_begin_stop_restart

RUN_TYPE: hybrid

RUN_REFCASE: b.e21.BSSP245smbb.f09_g17.001

RUN_REFDATE: 2033-01-01

GET_REFCASE: FALSE

~
Kind regards
Dharmendra

 

[sacks]

In the successful case, it looks like you set RUN_REFCASE to b.e21.BSSP245smbb.f09_g17.001 and properly staged the restart file set for 2033 from that case. In the failed case, it looks like you set RUN_REFCASE differently, to b.e21.BHISTsmbb.f09_g17.LE2-1231.011, and there is no 2033 restart file set from that case; you instead staged the restart file set from b.e21.BSSP245smbb.f09_g17.001. Did you mean to set RUN_REFCASE to
b.e21.BSSP245smbb.f09_g17.001 for your new case (currently failing) as well?

 

[dharmendraks841]

Oh! Thanks for pointing this out. The right case name was missed to update/change and mistakenly, the already default case name was continued for run.
I need more help from you as I need 24 variables as follows:
fincl2='U','V','T','PS','Q','QFLX','SHFLX','TAUX','TAUY','SST','U10','LHFLX','PHIS','TS','ICEFRAC','TSREF','QREF','ASDIR','ASDIF','ALDIR','ALDIF','LWUP = ,SOILW'.'SNOWH'
In all cases of the available directory /glade/campaign/cesm/development/cvcwg/cvwg/
for ssp245, I am not getting these (9) variables TSREF','QREF','ASDIR','ASDIF','ALDIR','ALDIF','LWUP', 'SOILW'.'SNOWH'
Do you know some other directory from where I can extract 6 hourly datasets for ssp245 for 2035 met_background?
It was easy to use the rest file and run the B compset, but all required variables are not available in the CESM directory. I searched other out sources, however, not all variables are there also.


Dharmendra

 

[slevis]

My first guess is that you have typos in the list of variables. If you find it difficult to debug the typos, then my usual approach is to add one new variable at-a-time until you get all of them to work.

 

[dharmendraks841]

Thanks!!
"Yes, I may have checked for typos; however, if there were any, how did I successfully extract all 15 variables in the same way?

I searched for these 9 variables, which were not found in all cases within the available directory (/glade/campaign/cesm/development/cvcwg/cvwg/). An example of my case was b.e21.BSSP245smbb.f09_g17.001.

It would be appreciated if anyone could show me the variables that I may have overlooked."

 

[asphilli]

Specifically speaking about the BSSP245 ensemble located here: /glade/campaign/cesm/development/cvcwg/cvwg/b.e21.BSSP245smbb.f09_g17,
the 9 variables you are looking for on 4x-daily timescales ("TSREF','QREF','ASDIR','ASDIF','ALDIR','ALDIF','LWUP', 'SOILW'.'SNOWH') were not saved. I do not know of another SSP245 run at NCAR that has those output variables.

 

[dharmendraks841]

Specifically speaking about the BSSP245 ensemble located here: /glade/campaign/cesm/development/cvcwg/cvwg/b.e21.BSSP245smbb.f09_g17,
the 9 variables you are looking for on 4x-daily timescales ("TSREF','QREF','ASDIR','ASDIF','ALDIR','ALDIF','LWUP', 'SOILW'.'SNOWH') were not saved. I do not know of another SSP245 run at NCAR that has those output variables.

Yes, the available directory of CESM is incomplete for these important variables.

 

[asphilli]

Due to storage constraints we cannot save all the available variables when we run simulations. The person who creates the simulation chooses which variables to save on which timescales. There is a standard set of variables that we typically save for simulations run by our working group, and this list of variables was formed based on feedback from working group members along with available storage.

 

[dharmendraks841]

Due to storage constraints we cannot save all the available variables when we run simulations. The person who creates the simulation chooses which variables to save on which timescales. There is a standard set of variables that we typically save for simulations run by our working group, and this list of variables was formed based on feedback from working group members along with available storage.

Agree