mksurfdata_ESMF done around glcmec

Hi all

I am using the new tool mksurfdata in CTSM5.2 in Archer2 to generate regional surface data in high-resolution from raw data.
The workflow is
>ncl mkscripgrid.ncl
>${ESMF}/ESMF_Scrip2Unstruct ${GRID} lnd_mesh.nc 0
>ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc
>ncdump -v grid_dims ${GRID} # to get nx and ny: grid_dims = 36, 25
>
./gen_mksurfdata_namelist --start-year ${START_YEAR} --end-year ${END_YEAR} --res ${RES} --model-mesh ${FORCE_MODEL_MESH_FILE} --model-mesh-nx ${FORCE_MODEL_MESH_NX} --model-mesh-ny ${FORCE_MODEL_MESH_NY} --res ${RES} --rawdata-dir ${INPUT_PATH}

>
./gen_mksurfdata_jobscript_single --debug --number-of-nodes ${NUMBER_OF_NODES} --tasks-per-node ${TASKS_PER_NODE} --namelist-file ${OUTPUT}/surfdata.namelist --bld-path ${BLD_PATH} --machine ${MACH} --walltime ${WALLTIME}


The job stopped after
mksrfdata writing LANDFRAC_MKSURFDATA.

The slurm log is:
Attempting to initialize control settings .....

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error is rather simple so I could not target any specific error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error is:
#0 0x151d06a4ed4f in ???
#0 0x14d795ac3d4f in ???
#1 0x435576 in __mkglcmecmod_MOD_mkglcmecinit
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mkglcmecMod.F90:103
#1 0x435576 in __mkglcmecmod_MOD_mkglcmecinit
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mkglcmecMod.F90:103
#2 0x496907 in mksurfdata
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:793
#3 0x49cc33 in main
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:88
#2 0x496907 in mksurfdata
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:793
#3 0x49cc33 in main
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:88
srun: error: nid002519: task 1: Segmentation fault
srun: launch/slurm: _step_signal: Terminating StepId=6535670.0
slurmstepd: error: *** STEP 6535670.0 ON nid002519 CANCELLED AT 2024-05-10T17:04:51 ***
srun: error: nid002519: task 0: Segmentation fault



When I check the source code, the next thing to do is:

! ----------------------------------------------------------------------
! Make glacier multiple elevation classes [pctglcmec,topoglcmec] from [fglacier] dataset
! ----------------------------------------------------------------------
! This call needs to occur after pctgla has been adjusted for the final time
if (fsurdat /= ' ') then
call mkglcmecInit (pioid)
call mkglcmec(mksrf_fglacier_mesh, mksrf_fglacier, mesh_model, pioid, rc=rc)
flush(ndiag)
if (ChkErr(rc,__LINE__,u_FILE_u)) call shr_sys_abort('error in calling mkglcmec')
end if

Shall I generate a glacier mesh (actually I do not know how to generate)? I just had a lnd_mesh.nc and msk_mesh.nc now.

I am not sure if I did not build mksurfdata correctly or ant command step was wrong. Thanks for any comments.

Best,
Yuan

I've moved this to the CTSM Forum. Are you still having problems with this?

oleson said:

I've moved this to the CTSM Forum. Are you still having problems with this?

Click to expand...

Dear Keith,

Thanks for taking an insight. I still have no idea how to solve this error.

Best,
Yuan

I'm not sure what is going on, but you shouldn't need to generate a glacier mesh file. If you make your scrip file available, we can try it here and see if we can figure it out.

oleson said:

I'm not sure what is going on, but you shouldn't need to generate a glacier mesh file. If you make your scrip file available, we can try it here and see if we can figure it out.

Click to expand...

Of course, my scripts include:

1. uk_grid.sh
#!/bin/bash
# Yuan Sun, 2024-5-5, Manchester, UK
# This script is used for make surface region for uk
# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# bash uk_grid.sh
# module load cray-python
# export PYTHONUSERBASE=/work/n02/n02/yuansun/.local
# export PATH=$PYTHONUSERBASE/bin:$PATH
# source /work/n02/n02/yuansun/myvenv/bin/activate
# python -m pip install
# step1: create uk grid
# ref: mksurfdata_esmf tool: missing percent clay and sand
module load ncl/6.6.2
# Set CESM_ROOT and TOOL_SCRIPT
export CESM_ROOT=/work/n02/n02/yuansun/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_2.3dynamicurban/components/clm/tools/
# Set environment variables
S_LAT=48.5 #48.8300
N_LAT=61 #60.8600
W_LON=347 #360-12.6100=347.39
E_LON=5 #4.5900
GRIDFILE=SCRIPgrid_05x05_UK_nomask_c240507.nc
CELL=0.5
# Calculate NY and NX
NY=$(echo "(${N_LAT}-${S_LAT})/${CELL}" | bc)
NX=$(echo "(360-${W_LON}+${E_LON})/${CELL}" | bc)
# Set environment variables for NCL script
export S_LAT
export N_LAT
export W_LON
export E_LON
export GRIDFILE
export NY
export NX
# Change directory
cd ${TOOL_SCRIPT}/mkmapgrids
# Run NCL script
ncl mkscripgrid.ncl
mv ${TOOL_SCRIPT}/mkmapgrids/${GRIDFILE} /work/n02/n02/yuansun/output_analysis/project4/make_surface

2. uk_namelist.sh
#!/bin/bash
# Yuan Sun, 2024-5-6, Manchester, UK
# according to CTSM/tools/mksurfdata_esmf at master · ESCOMP/CTSM
# after build the gen_mksurfdata_build tool
# step3: set up ctsm_pylib environment
# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# bash uk_namelist.sh
# ref: Creating mapfiles by using the ./mkmapdata.sh
export CESM_ROOT=/work/n02/n02/yuansun/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_ctsm5.2/tools
export START_YEAR=2000
export END_YEAR=2000
export RES=05x05_UK
export FORCE_MODEL_MESH_FILE
export ESMF=/work/n02/n02/yuansun/privatemodules_packages/archer2/apps/gcc/esmf/8.6/bin
export OUTPUT=/work/n02/n02/yuansun/output_analysis/project4/make_surface
export GRID=${OUTPUT}/SCRIPgrid_${RES}_nomask_c240507.nc
export FORCE_MODEL_MESH_FILE=${OUTPUT}/lnd_mesh.nc
export FORCE_MODEL_MESH_NX=36
export FORCE_MODEL_MESH_NY=25
export INPUT_PATH=/work/n02/n02/yuansun/cesm/cesm_inputdata/
module load nco/5.1.6
cd ${OUTPUT}
# converting
${ESMF}/ESMF_Scrip2Unstruct ${GRID} lnd_mesh.nc 0
ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc
ncdump -v grid_dims ${GRID} # to get nx and ny: grid_dims = 36, 25
export PATH=/work/n02/n02/yuansun/privatemodules_packages/anaconda/anaconda3/bin:$PATH
source /work/n02/n02/yuansun/privatemodules_packages/anaconda/anaconda3/bin/activate
conda activate ctsm_pylib
cd ${TOOL_SCRIPT}/mksurfdata_esmf
# ./gen_mksurfdata_namelist --help
./gen_mksurfdata_namelist --start-year ${START_YEAR} --end-year ${END_YEAR} --res ${RES} --model-mesh ${FORCE_MODEL_MESH_FILE} --model-mesh-nx ${FORCE_MODEL_MESH_NX} --model-mesh-ny ${FORCE_MODEL_MESH_NY} --res ${RES} --rawdata-dir ${INPUT_PATH}
# --hires_soitex
# Successfully created input namelist file surfdata.namelist
# for --hires_pft you must set both start-year and end-year to 1850 or to 2005
mv ${TOOL_SCRIPT}/mksurfdata_esmf/surfdata.namelist ${OUTPUT}

3. build_mksur_tool.sh
#!/bin/bash
# Yuan Sun, 2024-5-6, Manchester, UK
# according to CTSM/tools/mksurfdata_esmf at master · ESCOMP/CTSM
# after run, it will create a 'tool_bld' folder in the same directory
# step2: build the gen_mksurfdata_build tool
# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# bash build_mksur_tool.sh
module load PrgEnv-gnu
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
export CESM_ROOT=/work/n02/n02/yuansun/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_ctsm5.2/tools/
export MACH=archer2
export FFLAGS="-fallow-argument-mismatch -fallow-invalid-boz"
export FCFLAGS="-fallow-argument-mismatch -fallow-invalid-boz"
export LD_LIBRARY_PATH=$HDF5_DIR/lib:$LD_LIBRARY_PATH
export PATH=$HDF5_DIR/bin:$PATH
export CPATH=$HDF5_DIR/include:$CPATH
export LIBRARY_PATH=$HDF5_DIR/lib:$LIBRARY_PATH
export CFLAGS='-g -Wall'
export LDFLAGS="-L$HDF5_DIR/lib -lhdf5 -lhdf5_hl"
export CPPFLAGS="-I$HDF5_DIR/include"
cd ${TOOL_SCRIPT}/mksurfdata_esmf
./gen_mksurfdata_build --machine ${MACH}

4.download_rawdata.sh
#!/bin/bash
# Yuan Sun, 2024-5-7, Manchester, UK
# according to: Creating mapfiles by using the ./mkmapdata.sh
# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# bash download_rawdata.sh
export CESM_ROOT=/work/n02/n02/yuansun/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_ctsm5.2/tools
export RUNDIR=/work/n02/n02/yuansun/output_analysis/project4/make_surface
# The rundir is the directory containing surfdata.namelist
# This directory should contain .input_data_list and surfdata.namelist, among other files.
# It will download data one by one like:
# Model missing file mksrf_furban = '/work/n02/n02/yuansun/cesm/cesm_inputdata/lnd/clm2/rawdata/gao_oneill_urban/historical/urban_properties_GaoOneil_05deg_ThreeClass_2005_cdf5_c20220910.nc'
cd ${TOOL_SCRIPT}/mksurfdata_esmf
./download_input_data --help # need a case directory
# it will check inputdata and download missing files
./download_input_data --rundir ${RUNDIR}

5. uk_gen.sh:
#!/bin/bash
# Yuan Sun, 2024-5-7, Manchester, UK
# according to: Creating mapfiles by using the ./mkmapdata.sh
# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# bash uk_gen.sh
export CESM_ROOT=/work/n02/n02/yuansun/cesm
export TOOL_SCRIPT=$CESM_ROOT/my_cesm_sandbox_ctsm5.2/tools
export RES=005x005_UK
export OUTPUT=/work/n02/n02/yuansun/output_analysis/project4/make_surface
export NUMBER_OF_NODES=1
export TASKS_PER_NODE=128
export BLD_PATH=${TOOL_SCRIPT}/mksurfdata_esmf/tool_bld
export MACH=archer2
export WALLTIME=01:00:00
cd ${TOOL_SCRIPT}/mksurfdata_esmf
./gen_mksurfdata_jobscript_single --debug --number-of-nodes ${NUMBER_OF_NODES} --tasks-per-node ${TASKS_PER_NODE} --namelist-file ${OUTPUT}/surfdata.namelist --bld-path ${BLD_PATH} --machine ${MACH} --walltime ${WALLTIME}
mv mksurfdata_jobscript_single.sh /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf

6.mksurfdata_jobscript_single.sh
#!/bin/bash
# Edit the batch directives for your batch system
# Below are default batch directives for archer2
#SBATCH --job-name mksurfdata
#SBATCH --cpus-per-task=64
#SBATCH --partition=standard
#SBATCH --qos=standard
#SBATCH --account=n02-duicv
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# cd /work/n02/n02/yuansun/shell_scripts/archive/project4/mksurf
# sbatch mksurfdata_jobscript_single.sh
# squeue -u yuansun
cd /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf
# Remeber to add --hint=nomultithread --distribution=block:block after srun
# This is a batch script to run a set of resolutions for mksurfdata_esmf input namelist
# NOTE: THIS SCRIPT IS AUTOMATICALLY GENERATED SO IN GENERAL YOU SHOULD NOT EDIT it!!
# Run env_mach_specific.sh to control the machine dependent environment including the paths to compilers and libraries external to cime such as netcdf
. /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/tool_bld/.env_mach_specific.sh
if [ $? != 0 ]; then echo "Error running env_mach_specific script"; exit -4; fi
# Edit the mpirun command to use the MPI executable on your system and the arguments it requires
time srun --hint=nomultithread --distribution=block:block /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/tool_bld/mksurfdata < /work/n02/n02/yuansun/output_analysis/project4/make_surface/surfdata.namelist
if [ $? != 0 ]; then echo "Error running for namelist /work/n02/n02/yuansun/output_analysis/project4/make_surface/surfdata.namelist"; exit -4; fi
echo Successfully ran resolution

Thanks for your efforts.

Best,
Yuan

I think the problem is that your grid crosses Greenwich. So your scrip file has longitudes like this:

342.25, 332.75, 323.25, 313.75, 304.25, 294.75, 285.25,
275.75, 266.25, 256.75, 247.25, 237.75, ......

I don't think the capability exists to create regional surface datasets that span Greenwich. You may have to create two separate grids, one to the west of Greenwich and one to the east of Greenwich and then join the surface datasets together manually. @slevis , any other ideas on this?

I think even if you created a global 0.5deg dataset and then subset it, you'd run across this error:


The workaround there is to also create two grids.

oleson said:

I think the problem is that your grid crosses Greenwich. So your scrip file has longitudes like this:

342.25, 332.75, 323.25, 313.75, 304.25, 294.75, 285.25,
275.75, 266.25, 256.75, 247.25, 237.75, ......

I don't think the capability exists to create regional surface datasets that span Greenwich. You may have to create two separate grids, one to the west of Greenwich and one to the east of Greenwich and then join the surface datasets together manually. @slevis , any other ideas on this?

I think even if you created a global 0.5deg dataset and then subset it, you'd run across this error:

subset_data does not function for regional grids that span across Greenwich longitude zero · Issue #2017 · ESCOMP/CTSM

Brief summary of bug When you tell subset_data to give you a region that goes across zero longitude. General bug information CTSM version you are using: ctsm5.1.dev125-192-gec92580a3 Does this bug ...

github.com

The workaround there is to also create two grids.

Click to expand...

Oh, I see. I would have a try!

Thanks for your suggestion, Keith.

Best,
Yuan

Hi Keith,

I adjusted the domain to:
export S_LAT=48.80 #48.8300
export N_LAT=60.90 #60.8600
export W_LON=347.35 #360-12.6100=347.39
export E_LON=360
export LOCATION=UKwest
export CELL=0.05
export GRIDSIZE=005

Unfortunately, the job stopped after 'writing LANDFRAC_MKSURFDATA' as before. However, the surface data is generated anyway.

When visualizing the surface data, I found the data sounds 'normal'. So maybe something wrong with the code around line 797:
Given that I did not generate dynamic data, fsurdat=' ', but the code could not close surface data and the job was hanging .

! ----------------------------------------------------------------------
! Close surface dataset
! ----------------------------------------------------------------------
if (fsurdat /= ' ') then
call pio_closefile(pioid)
if (root_task) then
write(ndiag,*)
write(ndiag,'(a)') 'Successfully created surface data output file = '//trim(fsurdat)
write(ndiag,'(a)') ' This file contains the land model surface data'
write(ndiag,*)
flush(ndiag)
end if
end if

Thanks for any comments.

Best,
Yuan

Oh, I tried the tool in another machine CSF3, and it ran successfully.

Though not sure why Archer2 stopped at 'mksrfdata writing LANDFRAC_MKSURFDATA', interesting.

mksrfdata writing out PCT_URBAN
mksrfdata writing out PCT_GLACIER
mksrfdata writing out PCT_LAKE
mksrfdata writing out PCT_WETLAND
mksrfdata writing out PCT_OCEAN
mksrfdata writing PCT_NATVEG
mksrfdata writing PCT_CROP
mksrfdata writing PCT_NAT_PFT
mksrfdata writing PCT_CFT
mksrfdata writing LANDFRAC_MKSURFDATA
mkglcmecInit writing out GLC_MEC
================================================================================
Attempting to make percent elevation class
and mean elevation for glaciers .....
Input file is /mnt/iusers01/fatpou01/sees01/a16404ys/scratch/Projects/inputdata/lnd/clm2/rawdata/mksrf_glacier_3x3min_simyr2000.c20210708.nc
Input mesh file is /mnt/iusers01/fatpou01/sees01/a16404ys/scratch/Projects/inputdata/lnd/clm2/mappingdata/grids/UNSTRUCTgrid_3x3min_nomask_cdf5_c200129.nc
mkglcmec writing out pct_glc_mec
mkglcmec writing out topo_glc_mec
Successfully made percent elevation class and mean elevation for glaciers
Successfully created surface data output file = surfdata_005x005_UKeast_hist_2000_78pfts_c240521.nc
This file contains the land model surface data
Successfully ran mksurfdata_esmf

Best,
Yuan

oleson said:

I think the problem is that your grid crosses Greenwich. So your scrip file has longitudes like this:

342.25, 332.75, 323.25, 313.75, 304.25, 294.75, 285.25,
275.75, 266.25, 256.75, 247.25, 237.75, ......

I don't think the capability exists to create regional surface datasets that span Greenwich. You may have to create two separate grids, one to the west of Greenwich and one to the east of Greenwich and then join the surface datasets together manually. @slevis , any other ideas on this?

I think even if you created a global 0.5deg dataset and then subset it, you'd run across this error:

subset_data does not function for regional grids that span across Greenwich longitude zero · Issue #2017 · ESCOMP/CTSM

Brief summary of bug When you tell subset_data to give you a region that goes across zero longitude. General bug information CTSM version you are using: ctsm5.1.dev125-192-gec92580a3 Does this bug ...

github.com

The workaround there is to also create two grids.

Click to expand...

Hi，

Just update. The latest version of ESMF8.6.1 solves this issue of spanning Greenwich.

After using ESMF8.6.1 rather than ESMF8.6.0, CTSM5.2's new tool 'mksurfdata_esmf' works well for regions crossing Greenwich.

Best,
Yuan

Yuan Sun said:

Hi all

I am using the new tool mksurfdata in CTSM5.2 in Archer2 to generate regional surface data in high-resolution from raw data.
The workflow is
>ncl mkscripgrid.ncl
>${ESMF}/ESMF_Scrip2Unstruct ${GRID} lnd_mesh.nc 0
>ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc
>ncdump -v grid_dims ${GRID} # to get nx and ny: grid_dims = 36, 25
>
./gen_mksurfdata_namelist --start-year ${START_YEAR} --end-year ${END_YEAR} --res ${RES} --model-mesh ${FORCE_MODEL_MESH_FILE} --model-mesh-nx ${FORCE_MODEL_MESH_NX} --model-mesh-ny ${FORCE_MODEL_MESH_NY} --res ${RES} --rawdata-dir ${INPUT_PATH}

>
./gen_mksurfdata_jobscript_single --debug --number-of-nodes ${NUMBER_OF_NODES} --tasks-per-node ${TASKS_PER_NODE} --namelist-file ${OUTPUT}/surfdata.namelist --bld-path ${BLD_PATH} --machine ${MACH} --walltime ${WALLTIME}


The job stopped after
mksrfdata writing LANDFRAC_MKSURFDATA.

The slurm log is:
Attempting to initialize control settings .....

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error is rather simple so I could not target any specific error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error is:
#0 0x151d06a4ed4f in ???
#0 0x14d795ac3d4f in ???
#1 0x435576 in __mkglcmecmod_MOD_mkglcmecinit
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mkglcmecMod.F90:103
#1 0x435576 in __mkglcmecmod_MOD_mkglcmecinit
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mkglcmecMod.F90:103
#2 0x496907 in mksurfdata
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:793
#3 0x49cc33 in main
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:88
#2 0x496907 in mksurfdata
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:793
#3 0x49cc33 in main
at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mksurfdata.F90:88
srun: error: nid002519: task 1: Segmentation fault
srun: launch/slurm: _step_signal: Terminating StepId=6535670.0
slurmstepd: error: *** STEP 6535670.0 ON nid002519 CANCELLED AT 2024-05-10T17:04:51 ***
srun: error: nid002519: task 0: Segmentation fault



When I check the source code, the next thing to do is:

! ----------------------------------------------------------------------
! Make glacier multiple elevation classes [pctglcmec,topoglcmec] from [fglacier] dataset
! ----------------------------------------------------------------------
! This call needs to occur after pctgla has been adjusted for the final time
if (fsurdat /= ' ') then
call mkglcmecInit (pioid)
call mkglcmec(mksrf_fglacier_mesh, mksrf_fglacier, mesh_model, pioid, rc=rc)
flush(ndiag)
if (ChkErr(rc,__LINE__,u_FILE_u)) call shr_sys_abort('error in calling mkglcmec')
end if

Shall I generate a glacier mesh (actually I do not know how to generate)? I just had a lnd_mesh.nc and msk_mesh.nc now.

I am not sure if I did not build mksurfdata correctly or ant command step was wrong. Thanks for any comments.

Best,
Yuan

Click to expand...

Hi,

Just update the findings about mksurfdata_esmf in Archer2.

As I mentioned, I tried mksurfdata in 2 machines (Archer2 and CSF3). The tool works well in CSF3 but met error 'Program received signal SIGSEGV: Segmentation fault - invalid memory reference.' in Archer2. The backtrace recorded the at /work/n02/n02/yuansun/cesm/my_cesm_sandbox_ctsm5.2/tools/mksurfdata_esmf/src/mkglcmecMod.F90:103.
The code in Line 103, mkglcmecMod is elevclass_o(:) = elevclass(:). Before this line, I added allocate(elevclass_o(size(elevclass))) before elevclass_o(:) = elevclass(:) and rebuild the tool.

Now it works in Archer2. :)

However, I am unsure if I need to deallocate elevclass_o and deallocate elevclass anywhere.

Best,
Yuan

Based on other examples I see in the mksurfdata code, it does seem like that array needs to be allocated. And based on those other examples, it doesn't look like it needs to be deallocated. I'll file an issue on this, thanks.

@Yuan Sun very nice troubleshooting! Thank you for catching this problem.

Hi @Yuan Sun

My professor expected me to use the new tool mksurfdata in CTSM5.2 to generate regional surface data in high-resolution from raw data as well. I assume that I already successfully ported CTSM5.2 to our supercomputing service (Niagara); at least, I was able to run global simulation in Niagara.

While following your scripts shared here (it was very helpful! Thanks a lot), I was stopped at your step 2: build the gen_mksurfdata_build tool. The error is:
./gen_mksurfdata_build: line 150: ./.env_mach_specific.sh: No such file or directory
Error sourcing the env_mach_specific.sh file.

I recalled that there is a .env_mach_specific.sh file within my specific case directory, should I copy this to the tool_bld diectory? Or any steps am i missing? Considering you are pretty skilled in using mksurfdata tool, could you please provide me some guidance to move further?

Sincerely,
Isaac

Isaactian said:

Hi @Yuan Sun

My professor expected me to use the new tool mksurfdata in CTSM5.2 to generate regional surface data in high-resolution from raw data as well. I assume that I already successfully ported CTSM5.2 to our supercomputing service (Niagara); at least, I was able to run global simulation in Niagara.

While following your scripts shared here (it was very helpful! Thanks a lot), I was stopped at your step 2: build the gen_mksurfdata_build tool. The error is:
./gen_mksurfdata_build: line 150: ./.env_mach_specific.sh: No such file or directory
Error sourcing the env_mach_specific.sh file.

I recalled that there is a .env_mach_specific.sh file within my specific case directory, should I copy this to the tool_bld diectory? Or any steps am i missing? Considering you are pretty skilled in using mksurfdata tool, could you please provide me some guidance to move further?

Sincerely,
Isaac

Click to expand...

Hi Isaac,

Yes, you need to copy an .env_mach_specific.sh from a case folder to the tool_bld folder.

Since the old building process already met an error, do delete the existing tool_bld folder for the next build.

Thanks,
Yuan

Hi Yuan,

Thx for the prompt reply! While I used .env_mach_specific.sh from a case folder (the detailed contents are provided below), I met the error: "
The PIO directory for the PIO build is required and was not set in the configure
Make sure a PIO build is provided for with in config_machines". Should I add PIO_path in the .env_mach_specific.sh?

Additionally, I noticed that the .env_mach_specific.sh is expected to provide information of "COMPILER = $COMPILER, MPILIB = $MPILIB, DEBUG = $DEBUG, OS = $OS" as indicated in gen_mksurfdata_build script. I am wondering whether the contents in my current .env_mach_specific.sh are sufficient?

Thanks again!
Isaac

# This file is for user convenience only and is not used by the model
# Changes to this file will be ignored and overwritten
# Changes to the environment should be made in env_mach_specific.xml
# Run ./case.setup --reset to regenerate this file
. /scinet/niagara/software/2022a/lmod/8.5.27/init/sh
module load NiaEnv/2022a perl/5.38.2 cmake/3.22.5 git/2.39.1 subversion/1.10.7 intel/2022u2 intelmpi/2022u2+ucx-1.11.2 hdf5/1.10.9 netcdf/4.9.0 pnetcdf/1.8.1 python/3.11.5 esmf/8.7.0
export OMP_STACKSIZE=512M
export SCINET_NETCDF_MPI_ROOT=/scinet/niagara/software/2022a/opt/intel-2022u2-intelmpi-2022u2+ucx-1.11.2-hdf5-1.10.9/netcdf/4.9.0
export NETCDF_PATH=/scinet/niagara/software/2022a/opt/intel-2022u2-intelmpi-2022u2+ucx-1.11.2-hdf5-1.10.9/netcdf/4.9.0
export PNETCDF_PATH=/scinet/niagara/software/2022a/opt/intel-2022u2-intelmpi-2022u2+ucx-1.11.2/pnetcdf/1.8.1
export ESMFMKFILE=/scinet/niagara/software/2022a/opt/intel-2022u2-intelmpi-2022u2+ucx-1.11.2/esmf/8.7.0/lib/libO/Linux.intel.64.intelmpi.default/esmf.mk

Hi Yuan,

An update here: In fact, I forgot to delete tool_bld folder when re-ran the gen_mksurfdata_build script. Once I deleted it and ./gen_mksurfdata_build with clarifying the correct ported machine name (I made a typo before; I guess that's why the .env_mach_specific.sh was not automatically placed into the tool_bld directory), it can be built successfully.

Thx for your help!
Chuyin

Hi Yuan,

When I moved further, I met an error (ERROR: Makes no sense to have empty read-only file: /gpfs/fs1/home/slevis/cases_FATES/CZ2_acf_off/env_case.xml) at the fourth step, i.e., ./download_input_data --rundir ${RUNDIR}. Do you have any idea why this error occurred?

Thank you!

Isaactian said:

Hi Yuan,

When I moved further, I met an error (ERROR: Makes no sense to have empty read-only file: /gpfs/fs1/home/slevis/cases_FATES/CZ2_acf_off/env_case.xml) at the fourth step, i.e., ./download_input_data --rundir ${RUNDIR}. Do you have any idea why this error occurred?

Thank you!

Click to expand...

Please go to the ctsm/python/ctsm/mksurfdata_download_input_data.py,

I replaced case = Case(os.path.realpath("/home/slevis/cases_FATES/CZ2_acf_off")) by case = Case(os.path.realpath("/work/n02/n02/yuansun/cesm/runs/0project4/test"))

I think any existing run case should work.

Best,
Yuan

Yuan Sun said:

Please go to the ctsm/python/ctsm/mksurfdata_download_input_data.py,

I replaced case = Case(os.path.realpath("/home/slevis/cases_FATES/CZ2_acf_off")) by case = Case(os.path.realpath("/work/n02/n02/yuansun/cesm/runs/0project4/test"))

I think any existing run case should work.

Best,
Yuan

Click to expand...

Thx a lot! Now it's working. However, I found severl weird points:
1. I double-checked the .input_data_list, some data directory seems to keep as the default path (e.g. mksrf_fsoicol = /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/rawdata/CTSM53RawData/globalctsm53histMKSRFDeg025_240709/mksrf_landuse_ctsm53_soilcolor_CLIM.c240709.nc). This would generate a warning when downloading:
Model missing file mksrf_fsoicol = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/rawdata/CTSM53RawData/globalctsm53histMKSRFDeg025_240709/mksrf_landuse_ctsm53_soilcolor_CLIM.c240709.nc'
WARNING: Cannot download file since it lives outside of the input_data_root '/scratch/c/cgf/cytian/CTSM_test/input'

2. When downloading the input data, wget generated several warnings, for example:
(a)
WARNING: wget failed with output: and errput --2024-11-14 10:18:21-- ftp://ftp.cgd.ucar.edu/cesm/inputdata/lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.25x0.25_nomask_cdf5_c200129.nc
=> ‘/scratch/c/cgf/cytian/CTSM_test/input/lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.25x0.25_nomask_cdf5_c200129.nc’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.13.41
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.13.41|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm/inputdata/lnd/clm2/mappingdata/grids ... done.
==> SIZE UNSTRUCTgrid_0.25x0.25_nomask_cdf5_c200129.nc ... done.
==> PASV ... done. ==> RETR UNSTRUCTgrid_0.25x0.25_nomask_cdf5_c200129.nc ...
No such file ‘UNSTRUCTgrid_0.25x0.25_nomask_cdf5_c200129.nc’.

(b)
WARNING: wget failed with output: and errput --2024-11-14 10:18:22-- ftp://ftp.cgd.ucar.edu/cesm/inputdata/lnd/clm2/rawdata/lake_area/mksurf_lake_0.05x0.05_hist_clm5_hydrolakes_2000.cdf5.c20220325.nc
=> ‘/scratch/c/cgf/cytian/CTSM_test/input/lnd/clm2/rawdata/lake_area/mksurf_lake_0.05x0.05_hist_clm5_hydrolakes_2000.cdf5.c20220325.nc’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.13.41
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.13.41|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm/inputdata/lnd/clm2/rawdata/lake_area ...
No such directory ‘cesm/inputdata/lnd/clm2/rawdata/lake_area’.

I am wondering whether you met similar warnings. Additionally, I want to confirm with you that if I already ran a global simulation before (--compset IHistClm50Bgc), which means the needed input data were already downloaded in the {INPUT_PATH}. Now I want to compare the simulation over a certain region with that under high-resolution scale. Under this context, should I still do the fourth step (i.e., downloading the raw data) when attempting to create a regional surface dataset? Apologyize if the question does not make much sense, as I am pretty new in land-surface simulation.