Error submitting PTCLM case clm5.0

[donakanga]

Hi,

I have problems submitting a single point simulation case (/glade/u/home/akanga/ptclm_cases/IHistCLM50sp_20200212). The other steps seemed to run successfully: creating the case, case.setup and case.build (steps followed documented in attached file - and are largely borrowed from CLM/CTSM materials for running single point simulations). case.submit fails with the following error (found in the resulting the lnd.log file). Thank you.



landuse.timeseries_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr1850-2015_274.4_42.3_c
170824.nc
(GETFIL): using
/glade/scratch/akanga/single_point/landuse.timeseries_0.9x1.25_hist_16pfts_Irri
g_CMIP6_simyr1850-2015_274.4_42.3_c170824.nc
Opened existing file
/glade/scratch/akanga/single_point/landuse.timeseries_0.9x1.25_hist_16pfts_Irri
g_CMIP6_simyr1850-2015_274.4_42.3_c170824.nc 524288

dynpft_consistency_checks settings:
&DYNPFT_CONSISTENCY_CHECKS
CHECK_DYNPFT_CONSISTENCY = T
/

dynpft_check_consistency mismatch between PCT_NAT_PFT at initial time and that
obtained from surface dataset
On landuse_timeseries file: 0.000000000000000E+000 5.383352970758751E-002
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.646399914469780
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.233851621709065
6.591493411356704E-002
On surface dataset: 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 1.00000000000000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000

Confirm that the year of your surface dataset
corresponds to the first year of your landuse_timeseries file
(e.g., for a landuse_timeseries file starting at year 1850, which is typical,
you should be using an 1850 surface dataset),
and that your landuse_timeseries file is compatible with the surface dataset.

If you are confident that you are using the correct landuse_timeseries file
and the correct surface dataset, then you can bypass this check by setting:
check_dynpft_consistency = .false.
in user_nl_clm

local gridcell index = 1
global gridcell index = 1
gridcell longitude = 275.000000000000
gridcell latitude = 41.9371727748691
ENDRUN:
ERROR: ERROR in dynpftFileMod.F90 at line 169

 

[oleson]

Thanks for the documentation. It's likely that in your "single_pt" script that you ran, you had set overwrite_single_pft = True, which will overwrite the pft distribution in the surface dataset to be a single pft (#7). This will then be inconsistent with the single point landuse file which is unmodified from the data that was extracted from the global landuse file.
Can you try regenerating your surface dataset with overwrite_single_pft = False, and see if that then works in conjunction with the landuse file?
If you eventually want to use a single pft (or your own pft distribution) for your transient simulation, you'll have to modify the landuse file accordingly by hand to be consistent with the surface dataset. We've identified this as a feature to be added to the tool, at some point.

 

[donakanga]

Thank you, @oleson! I tried regenerating the surface dataset with overwrite_single_pft = False but seemed to get the same error. Should I have done any edits to this "dominant_pft = 7" as well? Cheyenne is currently down. I will try again once it resumes.

 

[oleson]

No, you shouldn't need to change dominant_pft. Are you sure the surface dataset was regenerated? It still seems to have a single pft specified.
The timestamp on the surface dataset file is Jan 28 which makes me think it hasn't been regenerated...

-rw-r--r-- 1 akanga ncar 104034 Jan 28 07:43 /glade/work/akanga/surfdata_0.9x1.25_16pfts_CMIP6_simyr1850_274.4_42.3_c170706.nc

 

[donakanga]

I think this time the surface dataset was successfully regenerated. I subsequently cleaned the case (./case.build --clean) and submitted. However, it still fails to submit although with no errors in the lnd.log. The error is in the cesm.log:

/glade/scratch/akanga/IHistCLM50sp_20200212/run/cesm.log.1101986.chadmin1.ib0.cheyenne.ucar.edu.200302-100537

Here are the red flags

Invalid PIO rearranger comm max pend req (comp2io), 0
Resetting PIO rearranger comm max pend req (comp2io) to 64
PIO rearranger options:
comm type =
p2p



1 pes participating in computation for CLM

-----------------------------------

NODE# NAME
( 0) r6i6n31
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Variable not found
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Variable not found
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name
NetCDF: Variable not found
NetCDF: Variable not found


* Total calving flux (Gt/y) 0.0000000000000000E+00
* Total dmass/dt (Gt/y) -0.3354463259247479E-10
* dmass/dt error (Gt/y) -0.3354463259247479E-10
* Total gr line flux (Gt/y) -0.2897558656989186E+01
* Mean thickness (m) 1770.6067354502413309
* Mean temperature (C) -19.9347059892333647
* Max thickness (m), i, j 3402.6667480468750000 230 366
* Max temperature, i, j, k -2.5815498828887939 297 216 0
* Min temperature, i, j, k -30.3878574371337891 236 451 0
* Max sfc spd (m/yr), i, j 999.9088970657438722 198 100
* Max base spd (m/yr), i, j 354.5326260228117121 304 572


Writing to file IHistCLM50sp_20200212.cism.initial_hist.1901-01-01-00000.nc
at time 0.000000000000000E+000
MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 1 Number of rows in sMat = 55296
000.MCT(MPEU)::die.: from MCT::m_SparseMatrixPlus::initDistributed_()
MPI_Abort: error code = 2
~

 

[oleson]

I think you need to create your case with "stub" runoff and glacier model.
Your longname compset looks like this (see your README.case in your case directory):

HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV

But it needs to be this (SROF instead of MOSART, and SGLC instead of CISM2%NOEVOLVE):

HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_SROF_SGLC_SWAV

I don't think there is a compset shortname for this, so you would just specify it explicitly in your create_newcase command:

--compset HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_SROF_SGLC_SWAV

 

[donakanga]

I created a new case with the suggested changes (documentation attached). Everything runs well, I think, until when it comes to submitting the case. This time round I don't get an error but I also cannot find the lnd.log nor cesm.log in the run directory as it were in the previous case.

Here is the path to the new case: /glade/u/home/akanga/ptclm_cases/IHistCLM50sp_20200302_1

and the case in the scratch directory: /glade/scratch/akanga/IHistCLM50sp_20200302_1

In this case, what should I include as the resolution while creating the new case?

Thank you.

 

[oleson]

f09_g17 resolution is fine since you are changing everything to single point.
I think something might be wrong with the share queue on cheyenne at the moment. I cloned your case and couldn't get it to run either (no errors).
I then switched the queue to regular and it ran fine for 10 years.
I reran a share queue job that I ran yesterday and the job also disappeared from the queue without running.
You might contact CISL help and tell them about your problem.

 

[donakanga]

Thank you, @oleson! I re-submitted earlier this morning and it ran just fine. I am so grateful for your assistance.