customized single-point simulation on derecho: case submit errors

[Danqiong.Dai]

Hi,

Last time I tried using customized observational site forcing with CTSM on derecho. Following the previous workflow, I successfully generated domain files and surface data and modified *.xml files. However, when I submit the case (./case.submit), there is an error in CESM.log:
Obtained 10 stack frames.
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(pio_err+0x82) [0x14ae7b7b7892]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(+0x17c8f) [0x14ae7b7bbc8f]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(PIOc_openfile_retry+0x36f) [0x14ae7b7bb33f]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(PIOc_openfile+0x13) [0x14ae7b7b6b13]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpiof.so(piolib_mod_mp_pio_openfile_+0x1a4) [0x14ae7ba0eb34]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x1153306]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x1159d12]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x1149649]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x57ff83]
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/mpi-serial/2.3.0/oneapi/2023.2.1/juyc/lib/libesmf.so(_ZN5ESMCI6FTable12callVFuncPtrEPKcPNS_2VMEPi+0x444) [0x14ae70ca0594]
MPI_Abort: error code = -1

my case path is /glade/derecho/scratch/danqiongd/test_CLM/CTSM/cime/scripts/CAN-SCC, and a note for running my own case is located in /glade/derecho/scratch/danqiongd/test_CLM/note
Any suggestions would be appreciated!

 

[Danqiong.Dai]

I just now modified my forcing data information, and the errors changed in CESM.log:
Abort with message NetCDF: Variable not found in file /glade/derecho/scratch/csgteam/temp/spack/derecho/23.09/builds/spack-stage-parallelio-2.6.2-bpi7h2bnkshocep4hl3drfik4ebj44iu/spack-src/src/clib/pio_nc.c at line 1164
Obtained 10 stack frames.
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(print_trace+0x30) [0x145e2c52e600]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(check_netcdf2+0x20c) [0x145e2c52ebfc]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpioc.so(PIOc_inq_varid+0x23e) [0x145e2c54727e]
/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/mpi-serial/2.3.0/oneapi/2023.2.1/bpi7/lib/libpiof.so(pio_nf_mp_inq_varid_id_+0x8f) [0x145e2c784a5f]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x113b1d0]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x1139ed3]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x1134269]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x57d668]
/glade/derecho/scratch/danqiongd/CAN-SCC/bld/cesm.exe() [0x5804e8]
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/mpi-serial/2.3.0/oneapi/2023.2.1/juyc/lib/libesmf.so(_ZN5ESMCI6FTable12callVFuncPtrEPKcPNS_2VMEPi+0x444) [0x145e21a17594]
forrtl: error (76): Abort trap signal

my customized forcing data is located in /glade/derecho/scratch/danqiongd/wetland_project/forcingdata_CLM/inputdata/1x1_CAN-SCC/CLM1PT_data.

 

[oleson]

Your surface dataset:

"/glade/derecho/scratch/danqiongd/test_CLM/CTSM/tools/mksurfdata_esmf/surfdata_1x1_CAN-SCC_hist_2005_16pfts_c240723.nc"

doesn't seem to have any valid data in it. Most variables have values of zero.

 

[Danqiong.Dai]

Here is my note about creating my surface data:
###create surface data
###get SCRIPgrid data
module load ncl
cd $mydir/CTSM/tools/site_and_region
./mknoocnmap.pl -p 61.3089,238.7016 -n 1x1_CAN-SCC
# it shows "Successfully created grid/mapping files for single-point"
#in $mydir/CTSM/tools/mkmapgrids, three files are constructed (map_1x1_CAN-SCC_noocean_to_1x1_CAN-SCC_nomask_aave_da_240723.nc,SCRIPgrid_1x1_CAN-SCC_nomask_c240723.nc,SCRIPgrid_1x1_CAN-SCC_noocean_c240723.nc)

###get surface data, reference: CTSM/tools/mksurfdata_esmf at master · ESCOMP/CTSM
./gen_mksurfdata_build #because there no tool_bld directory
module unload python
module load conda
./py_env_create
conda activate ctsm_pylib
cd tools/mksurfdata_esmf
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct /glade/derecho/scratch/danqiongd/test_CLM/CTSM/tools/mkmapgrids/SCRIPgrid_1x1_CAN-SCC_nomask_c240723.nc lnd_mesh.nc 0

module load nco
ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc
ncdump -v grid_dims /glade/derecho/scratch/danqiongd/test_CLM/CTSM/tools/mkmapgrids/SCRIPgrid_1x1_CAN-SCC_nomask_c240723.nc (purpose: to get nx and ny)

./gen_mksurfdata_namelist --start-year 2005 --end-year 2005 --model-mesh-nx 1 --model-mesh-ny 1 --nocrop --model-mesh lnd_mesh.nc --res 1x1_CAN-SCC --hires_pft --hires_soitex

./gen_mksurfdata_jobscript_single --number-of-nodes 1 --tasks-per-node 1 --namelist-file surfdata_1x1_CAN-SCC_hist_2005_16pfts_c240723.namelist

qsub -A P****(#my project account) mksurfdata_jobscript_single.sh
result: generate surface data successfully! (surfdata_1x1_CAN-SCC_hist_2005_16pfts_c240723.nc)

 

[oleson]

I've been informed that it is better to use tools/site_and_regional/subset_data instead of mksurfdata_esmf to create single point datasets.

 

[Yuan Sun]

I've been informed that it is better to use tools/site_and_regional/subset_data instead of mksurfdata_esmf to create single point datasets.

Hi Keith,

I tried but tools/site_and_regional/subset_data and tools/mksurfdata_esmf tools to extract single point surface data in CTSM5.2. The mksurfdata_esmf works but the subset_data generated incorrect single point surface data.

The job script is:
export HOME=/work/n02/n02/yuansun
export CESM_ROOT=${HOME}/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_ctsm5.2/tools
export LAT=53.3
export LON=257.2
export LOCATION=MCR
export OUT_DIR=${HOME}/output_analysis/project6/single_point/lnd_default/default_land_data/sample_data
export USER_MODS_DIR=${HOME}/output_analysis/project6/make_surface/
export INPUT=${CESM_ROOT}/cesm_inputdata
cd ${TOOL_SCRIPT}/site_and_regional
./subset_data point --lat ${LAT} --lon ${LON} --site ${LOCATION} --create-surface --outdir ${OUT_DIR} --inputdata-dir ${INPUT} --crop --overwrite

It did generate a surface data 'surfdata_MCR_hist_2000_78pfts_c241015.nc' from lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_2000_78pfts_c240216.nc but some variables are NaN (invalid values).

For example, the parameter is not complete with a missing dimension 'nlevurb'
double EM_IMPROAD(numurbl=3, lsmlat=1, lsmlon=1);
:_FillValue = NaN; // double
:long_name = "emissivity of impervious road";
:units = "unitless";

It would be helpful if you could take an insight.

Best,
Yuan

 

[oleson]

I tried this with ctsm5.3.0 and it seemed to work for me:

./subset_data point --lat 53.3 --lon 257.2 --site NCR --create-surface --crop

EM_IMPROAD wouldn't have a nlevurb dimension since it is an emissivity for the surface.
Are you using the ctsm_pylib environment to create this?

 

[Yuan Sun]

I tried this with ctsm5.3.0 and it seemed to work for me:

./subset_data point --lat 53.3 --lon 257.2 --site NCR --create-surface --crop

EM_IMPROAD wouldn't have a nlevurb dimension since it is an emissivity for the surface.
Are you using the ctsm_pylib environment to create this?

Hi Keith,

Thanks for your response.

Sorry, I misremembered the dimension of EM_IMPROAD.

After I switched to the ctsm_pylib environment, ./subset_data point worked.

The previous error came from another environment with the dask (parallel computing library) for xarray. When I run ./subset_data with dask in the hpc login node, it fails to generate the surface file with multiple tasks. I learned to avoid dask when using ./subset_data.

Best,
Yuan