Run at 0.5° stalls out

[samrabin]

I'm now able to build and submit a job at 0.5° (previously; thanks Keith and Erik!), but I have a new problem. The job just… stalls out for some reason. After about a minute, there's nothing new written to any of the log files, and I let it go for 12 hours—nothing. Was there something else I should have specified?

The case is here: /glade/u/home/samrabin/cases_ctsm/halfdeg_test

Here is the output of tail for each of the log files.

atm.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

(shr_strdata_readstrm) setting pio descriptor : /glade/p/cesmdata/cseg/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc
(shr_strdata_set_stream_iodesc) setting iodesc for : TOPO with dimlens(1), dimlens(2) =      288       192   variable as time dimension time
(shr_strdata_readstrm) reading file lb: /glade/p/cesmdata/cseg/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc       1
(shr_strdata_readstrm) reading file ub: /glade/p/cesmdata/cseg/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc       1
(shr_strdata_readstrm) opening   : /glade/p/cesmdata/cseg/inputdata/atm/datm7/CO2/fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc
(shr_strdata_readstrm) reading file lb: /glade/p/cesmdata/cseg/inputdata/atm/datm7/CO2/fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc     101
(shr_strdata_readstrm) reading file ub: /glade/p/cesmdata/cseg/inputdata/atm/datm7/CO2/fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc     101
 (datm_datamode_clmncep_advance): tbotmax =    328.485305786133
 (datm_datamode_clmncep_advance): anidrmax =   1.000000000000000E+0303
 atm : model date     18500101           0

cesm.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

0: (t_initf)       profile_add_detail=       F
0: (t_initf)       profile_papi_enable=      F
0:(shr_orb_params) ------ Computed Orbital Parameters ------
0:(shr_orb_params) Eccentricity      =   1.676429E-02
0:(shr_orb_params) Obliquity (deg)   =   2.345928E+01
0:(shr_orb_params) Obliquity (rad)   =   4.094416E-01
0:(shr_orb_params) Long of perh(deg) =   1.003269E+02
0:(shr_orb_params) Long of perh(rad) =   4.892627E+00
0:(shr_orb_params) Long at v.e.(rad) =  -3.290978E-02
0:(shr_orb_params) -----------------------------------------

drv.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

 LND              : pio_rearranger =            2
 LND              : pio_root =            1
 LND              : pio_iotype =            1
 ROF              : pio_numiotasks =            2
 ROF              : pio_stride =           18
 ROF              : pio_rearranger =            2
 ROF              : pio_root =            1
 ROF              : pio_iotype =            1
 8 MB memory   alloc in MB is             8.00\n8 MB memory dealloc in MB is
          7.98\nMemory block size conversion in bytes is          1024.00

lnd.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

(shr_strdata_readstrm) reading file ub: /glade/p/cesmdata/cseg/inputdata/lnd/clm2/paramdata/finundated_inversiondata_0.9x1.25_c170706.nc       1
 (GETFIL): attempting to find local file megan21_emis_factors_78pft_c20161108.nc
 (GETFIL): using
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/emis/megan21_emis_fac
 tors_78pft_c20161108.nc
 (GETFIL): attempting to find local file
 surfdata_360x720cru_hist_78pfts_CMIP6_simyr1700_c210824.nc
 (GETFIL): using
 /glade/u/home/samrabin/inputdata/lnd/clm2/surfdata_map/ctsm5.1.dev052/surfdata_
 360x720cru_hist_78pfts_CMIP6_simyr1700_c210824.nc

med.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

 (Advertise Fields):Fr_wav: Sw_pstokes_y
 (Advertise Fields):To_wav: cpl_scalars
 (Advertise Fields):To_wav: Si_ifrac
 (Advertise Fields):To_wav: So_t
 (Advertise Fields):To_wav: So_u
 (Advertise Fields):To_wav: So_v
 (Advertise Fields):To_wav: So_bldepth
 (Advertise Fields):To_wav: Sa_u
 (Advertise Fields):To_wav: Sa_v
 (Advertise Fields):To_wav: Sa_tbot

rof.log.3846785.chadmin1.ib0.cheyenne.ucar.edu.220420-155659

           6  TOTAL_DISCHARGE_TO_OCEAN_LIQ     A
 There will be a total of            1 MOSART  history tapes

 MOSART History tape            1  write frequency is MONTHLY
 Number of time samples on MOSART history tape            1  is            1
 Output precision on MOSART history tape            1 =           1

 hist_htapes_build Successfully initialized MOSART history files
------------------------------------------------------------
 (Rtmini)  done

 

[samrabin]

Here's what I added to user_nl_clm:

flanduse_timeseries = '/glade/u/home/samrabin/inputdata/lnd/clm2/surfdata_map/ctsm5.1.dev052/landuse.timeseries_360x720cru_hist_78pfts_CMIP6_simyr1700-2020_c210824.nc'
fsurdat = '/glade/u/home/samrabin/inputdata/lnd/clm2/surfdata_map/ctsm5.1.dev052/surfdata_360x720cru_hist_78pfts_CMIP6_simyr1700_c210824.nc'[/ICODE]

 

[oleson]

It looks like you are only using 1 node for this run?

 

[samrabin]

D'oh! That was it, of course.