Errors occur in making surface data, and the error is that "ERROR: mapping file for this resolution does NOT exist

[zhangguanhao]

Dear all:
During the process of making surface data, I met an error about mksurfdata.pl:

What version of the code are you using?
I downloaded the CLM5 from "git clone -b release-clm5.0 GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM) clm5.0"

Have you made any changes to files in the source tree?
No.

Describe every step you took leading up to the problem:
I am making a surface dataset for regional simulation, and the workflow are as follows:

cd /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mkmapdata/
./mknoocnmap.pl -p $CENLAT,$CENLON -n $GRIDNAME -dy $DY -ny $NY -dx $DX -nx $NX
./mkmapdata.sh -r $GRIDNAME -f $GRIDFILE -t regional
cd /project/zhangboxuan/zgh/CLM5/clm5.0/cime/tools/mapping/gen_domain_files/
./gen_domain -m $MAPFILE -o $OCNDOM -l $ATMDOM
cd /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mksurfdata_map/
./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -usrname $GRIDNAME -l $CSMDATA -no-crop

However, errors occur in the last line, when I use:
./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -usrname $GRIDNAME -l $CSMDATA -no-crop
I meet "ERROR: mapping file for this resolution does NOT exist (../mkmapdata/map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc)."

If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.

Errors occur in making surface data, and the error is that "ERROR: mapping file for this resolution does NOT exist (../mkmapdata/map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc).".


Describe your problem or question: I have already generated corresponding files in forders "mkmapgrids", "mkmapdata", and "gen_domain_files", and I wanna to ask how can I solve the problem mentioned before.

Thank you for your time and I am looking forward for your feedback.

Guanhao Zhang

 

[oleson]

Since mksurfdata.pl can't find the mapping file, that likely either means it doesn't exist or it has a different name than what is expected. Did the mkmapdata.sh step complete successfully? What are the files it created in the mkmapdata directory? Do they match the file name format as expected (map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc)?
Also, try the mksurfdata.pl without the -usrname $GRIDNAME in the command.

 

[zhangguanhao]

Dear Oleson

Thanks for your time and kind reply.

For your first comment, In think the mkmapdata.sh step has completed successfully. The files in forders "mkmapgrids", "mkmapdata", and "gen_domain_files" are as follows:

(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mkmapgrids
map_Xinjiang_highres1km_300x300_noocean_to_Xinjiang_highres1km_300x300_nomask_aave_da_260310.nc
mkscripgrid.ncl
README
SCRIPgrid_Xinjiang_highres1km_300x300_nomask_c260310
SCRIPgrid_Xinjiang_highres1km_300x300_noocean_c260310.nc


(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mkmapdata
createXMLEntries.pl
map_0.25x0.25_MODIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.5x0.5_AVHRR_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.5x0.5_MODIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.9x1.25_GRDC_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_10x10min_IGBPmergeICESatGIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_10x10min_nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_360x720cru_cruncep_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_GLOBE-Gardner-mergeGIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_GLOBE-Gardner_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_LandScan2004_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_MODISv2_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_MODIS-wCsp_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_USGS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_IGBP-GSDP_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_ISRIC-WISE_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_ORNL-Soil_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
mkmapdata.sh
mknoocnmap.pl
mkunitymap.ncl
mvNimport.sh
PET0.RegridWeightGen.Log
README
regridbatch.sh
regriddav.sh
rmdups.ncl

(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/cime/tools/mapping/gen_domain_files
domain.lnd.Xinjiang_highres1km_300x300_noocean.260310.nc gen_domain src
domain.ocn.noocean.260310.nc INSTALL
domain.ocn.Xinjiang_highres1km_300x300_noocean.260310.nc README

In addition, I also tried remove "-usrname $GRIDNAME" but met the same error:
(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ./mksurfdata.pl -res usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -l $CSMDATA -no-crop
ERROR: mapping file for this resolution does NOT exist (../mkmapdata/map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc).

To further check this problem, I find that mksurfdata.pl has these codes, in that case I guess maybe this problem is caused by queryDefaultNamelist.pl can not recognize the ${hgrid} and ${lmask} correctly? The mentioned codes in "queryDefaultNamelist.pl" are listed as follows:

if ( $opts{'hgrid'} eq "usrspec" ) {
$map{$typ} = $opts{'usr_mapdir'}."/map_${hgrid}_${lmask}_to_${res}_nomask_aave_da_c${mapdate}\.nc";
} else {
$map{$typ} = `$scrdir/../../bld/queryDefaultNamelist.pl $queryfilopts -namelist clmexp -options frm_hgrid=$hgrid,frm_lmask=$lmask,to_hgrid=$res,to_lmask=nomask -var map`;
}
$map{$typ} = trim($map{$typ});
if ( $map{$typ} !~ /[^ ]+/ ) {
die "ERROR: could NOT find a mapping file for this resolution: $res and type: $typ at $hgrid and $lmask.\n";
}
if ( ! defined($opts{'allownofile'}) && ! -f $map{$typ} ) {
die "ERROR: mapping file for this resolution does NOT exist ($map{$typ}).\n";
}
}

Furthermore, in particular, when I download the clm5 by "git clone -b release-clm5.0 GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM) clm5.0" this time, the folder "/clm5.0/tools/" only has "Readme", so I move the files downloaded before (around September, 2023) to "/clm5.0/tools/". I am not sure whether this would affect the data generation.

Thank you again for your patience and time, and I am looking forward to your valuable feedbacks.

Guanhao Zhang

 

[zhangguanhao]

Since mksurfdata.pl can't find the mapping file, that likely either means it doesn't exist or it has a different name than what is expected. Did the mkmapdata.sh step complete successfully? What are the files it created in the mkmapdata directory? Do they match the file name format as expected (map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc)?
Also, try the mksurfdata.pl without the -usrname $GRIDNAME in the command.

Dear Oleson

The tries and understandings are listed here and I am sorry that I forgot to use the function "reply". Hope you can receive my messages successfully.

Dear Oleson

Thanks for your time and kind reply.

For your first comment, In think the mkmapdata.sh step has completed successfully. The files in forders "mkmapgrids", "mkmapdata", and "gen_domain_files" are as follows:

(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mkmapgrids
map_Xinjiang_highres1km_300x300_noocean_to_Xinjiang_highres1km_300x300_nomask_aave_da_260310.nc
mkscripgrid.ncl
README
SCRIPgrid_Xinjiang_highres1km_300x300_nomask_c260310
SCRIPgrid_Xinjiang_highres1km_300x300_noocean_c260310.nc


(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/tools/mkmapdata
createXMLEntries.pl
map_0.25x0.25_MODIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.5x0.5_AVHRR_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.5x0.5_MODIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_0.9x1.25_GRDC_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_10x10min_IGBPmergeICESatGIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_10x10min_nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_360x720cru_cruncep_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_GLOBE-Gardner-mergeGIS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_GLOBE-Gardner_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_LandScan2004_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_MODISv2_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_MODIS-wCsp_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_3x3min_USGS_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_IGBP-GSDP_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_ISRIC-WISE_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_nomask_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
map_5x5min_ORNL-Soil_to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc
mkmapdata.sh
mknoocnmap.pl
mkunitymap.ncl
mvNimport.sh
PET0.RegridWeightGen.Log
README
regridbatch.sh
regriddav.sh
rmdups.ncl

(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ls /project/zhangboxuan/zgh/CLM5/clm5.0/cime/tools/mapping/gen_domain_files
domain.lnd.Xinjiang_highres1km_300x300_noocean.260310.nc gen_domain src
domain.ocn.noocean.260310.nc INSTALL
domain.ocn.Xinjiang_highres1km_300x300_noocean.260310.nc README

In addition, I also tried remove "-usrname $GRIDNAME" but met the same error:
(cesm_py39) [zhangboxuan@swarm02 mksurfdata_map]$ ./mksurfdata.pl -res usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -l $CSMDATA -no-crop
ERROR: mapping file for this resolution does NOT exist (../mkmapdata/map___to_Xinjiang_highres1km_300x300_nomask_aave_da_c260310.nc).

To further check this problem, I find that mksurfdata.pl has these codes, in that case I guess maybe this problem is caused by queryDefaultNamelist.pl can not recognize the ${hgrid} and ${lmask} correctly? The mentioned codes in "queryDefaultNamelist.pl" are listed as follows:

if ( $opts{'hgrid'} eq "usrspec" ) {
$map{$typ} = $opts{'usr_mapdir'}."/map_${hgrid}_${lmask}_to_${res}_nomask_aave_da_c${mapdate}\.nc";
} else {
$map{$typ} = `$scrdir/../../bld/queryDefaultNamelist.pl $queryfilopts -namelist clmexp -options frm_hgrid=$hgrid,frm_lmask=$lmask,to_hgrid=$res,to_lmask=nomask -var map`;
}
$map{$typ} = trim($map{$typ});
if ( $map{$typ} !~ /[^ ]+/ ) {
die "ERROR: could NOT find a mapping file for this resolution: $res and type: $typ at $hgrid and $lmask.\n";
}
if ( ! defined($opts{'allownofile'}) && ! -f $map{$typ} ) {
die "ERROR: mapping file for this resolution does NOT exist ($map{$typ}).\n";
}
}

Furthermore, in particular, when I download the clm5 by "git clone -b release-clm5.0 GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM) clm5.0" this time, the folder "/clm5.0/tools/" only has "Readme", so I move the files downloaded before (around September, 2023) to "/clm5.0/tools/". I am not sure whether this would affect the data generation.

Thank you again for your patience and time, and I am looking forward to your valuable feedbacks.

Guanhao Zhang

 

[oleson]

Do you have this file: bld/namelist_files/namelist_defaults_clm4_5_tools.xml?
That file may not be available when checking out release-clm5.0, so that might be one problem.
The README says:

"IMPORTANT NOTE: Normally tools for creating input datasets for CLM
such as surface dataset creation would be here. As well as post processing
or other utility side line tools. However, we are only supporting
tools for later versions of the code. The tools should all work to create
or process datasets from these versions.

Therefore if you need any tools here checkout:

ctsm5.1.dev158 or later and use it"

So, you could checkout ctsm5.1.dev158 and use those tools. They should be able to create clm5 compatible files.

Alternatively, you could checkout a release version of CESM that has those tools. I have used the tools available in release-cesm2.1.4 successfully.

 

[zhangguanhao]

Do you have this file: bld/namelist_files/namelist_defaults_clm4_5_tools.xml?
That file may not be available when checking out release-clm5.0, so that might be one problem.
The README says:

"IMPORTANT NOTE: Normally tools for creating input datasets for CLM
such as surface dataset creation would be here. As well as post processing
or other utility side line tools. However, we are only supporting
tools for later versions of the code. The tools should all work to create
or process datasets from these versions.

Therefore if you need any tools here checkout:

ctsm5.1.dev158 or later and use it"

So, you could checkout ctsm5.1.dev158 and use those tools. They should be able to create clm5 compatible files.

Alternatively, you could checkout a release version of CESM that has those tools. I have used the tools available in release-cesm2.1.4 successfully.

Dear Oleson

Thank you for your patience and your constructive comments.

In my "bld/namelist_files/" forder, I have the file named "namelist_defaults_clm4_5_tools.xml". After I changed the model to an older version and use "./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -l $CSMDATA -no-crop" without "-usrname $GRIDNAME", the "queryDefaultNamelist.pl" can find the corresponding files.

Best wishes and have a good day.

Guanhao Zhang