Error in in subgridWeightsMod.F90

[LiwenWu]

Dear Scientists,
I want to run a regional IHISTClm50SpGs case using my own landuse_timeseries data, surface data and meterological data. With your patient answer and help, I successfully generated a 0.01 deg surfdata file (2000) using the rawdata and land use data I prepared. I also use dynamic LAI I prepared by setting use_lai_streams = .true.. I successfully ran a I2000Clm50SpGs case for spinup and got the restart file as the initial file for IHISTClm50SpGs case. I have indicated these files in user_nl_clm as below:

---

&clm_inparm
fsurdat='/home/user/cesm/inputdata/lnd/clm2/surfdata_map/surfdata_0.01x0.01_shanxi_16pfts_Irrig_simyr2000_c240401.nc'
flanduse_timeseries ='/home/user/cesm/inputdata/lnd/clm2/landuse.timeseries.nc'
finidat='/home/user/cesm/scratch/shanxispinup7/run/shanxispinup7.clm2.r.0081-01-01-00000.nc'
use_init_interp = .true.
hist_empty_htapes = .false.
use_lai_streams = .true.
hist_mfilt = 12, 30
hist_nhtfrq = 0, -24
hist_fincl1 = 'FPSN', 'H2OSOI ', 'QVEGT', 'QVEGE', 'QSOIL', 'SOILLIQ ', 'QFLX_EVAP_TOT', 'QRUNOFF', 'QOVER'
/

---

then I build and submit the case while I got many errors as below:

---

ENDRUN:
ERROR: ERROR in subgridWeightsMod.F90 at line 703
check_weights ERROR: at g = 430 total PFT weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 516 total PFT weight is
0.910000000000000 active_only = F
check_weights ERROR: at g = 517 total PFT weight is
0.820000000000000 active_only = F
check_weights ERROR: at g = 518 total PFT weight is
0.940000000000000 active_only = F
check_weights ERROR: at g = 519 total PFT weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 1130 total PFT weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 430 total col weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 516 total col weight is
0.910000000000000 active_only = F
check_weights ERROR: at g = 517 total col weight is
0.820000000000000 active_only = F
check_weights ERROR: at g = 518 total col weight is
0.940000000000000 active_only = F
check_weights ERROR: at g = 519 total col weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 1130 total col weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 430 total lunit weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 516 total lunit weight is
0.910000000000000 active_only = F
check_weights ERROR: at g = 517 total lunit weight is
0.820000000000000 active_only = F
check_weights ERROR: at g = 518 total lunit weight is
0.940000000000000 active_only = F
check_weights ERROR: at g = 519 total lunit weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 1130 total lunit weight is
0.980000000000000 active_only = F

If you are seeing this message at the beginning of a run with
use_init_interp = .true. and init_interp_method = "use_finidat_areas",
and you are seeing weights less than 1, then a likely cause is:
For the above-mentioned grid cell(s):
The matching input grid cell had some non-zero-weight subgrid type
that is not present in memory in the new run.

If you are using a ctsm5.2 or later fsurdat file containing
PCT_OCEAN > 0, then you may solve the error by setting
convert_ocean_to_land = .true.

---

As the surfdata I use in this case is absolutely the same as that in the spin up case, and the spin up case has been successfully run. I think the error should not point to the surfdata. And I also checked the pft data in the landuse_timeseries file, I think it should be fine. Do you have any suggestions for this error?
Any advice would be greatly appreciated. Thank you!

 

[oleson]

This post might be helpful if you have init_interp_method='use_finidat_areas' in your lnd_in:

Issues setting up hybrid run

Hey I've been trying to put up a hybrid branch run using CESM2.1.2. and the BHIST compset starting from the restart files of the CMIP6 historical simulations, however every time I try to run this simulation I get the following error...

bb.cgd.ucar.edu

 

[LiwenWu]

Dear Oleson,
Thanks for your timely reply. I checked the settings in lnd_in and found the default setting is init_interp_method='general'. So I think the error is not due to this setting. I really want to know if the error is due to model settings or file issues?
Maybe the problem is from the initial data or surfdata. But the initial data is a restart file from a I2000Clm50SpGs spin-up case (shanxispinup7.clm2.r.0081-01-01-00000.nc). And the surfdata is absolutely the same as the one that spin-up case used which has successfully run.
I made my own landuse.timeseries file based on CTSM default file landuse.timeseries_4x5_hist_16_CMIP6_1850-2015_c230620.nc which contains dynamic lake and urban. I checked the file to confirm the sum of pfts is 100%. And I have one question about the PCT_WETLAND. Since it's not included in landuse.timeseries file. Will it affect the sum of gridcell equall to 100%?
I am very confused about this issue and hope to receive your help. Thank you.

 

[LiwenWu]

Dear Oleson,
Thanks for your timely reply. Please ignore the above post. I checked the settings in lnd_in and found the default setting is init_interp_method='general'. So I think the error is not due to this setting. I really want to know if the error is due to model settings or file issues?
I checked the lnd.log.240806-234136 and found that the error occured after reading the landuse.timeseries file. Maybe the problem is from the landuse.timeseries data. I made my own landuse.timeseries file based on CTSM default file landuse.timeseries_4x5_hist_16_CMIP6_1850-2015_c230620.nc which contains dynamic lake and urban. I checked the file to confirm the sum of pfts is 100%. Do you have any suggestions? And I have one question about the PCT_WETLAND. Since it's not included in landuse.timeseries file. Will it affect the sum of gridcell equall to 100%?
I am very confused about this issue and hope to receive your help. Thank you.

---

Attempting to read landuse.timeseries data .....
(GETFIL): attempting to find local file
landuse.timeseries.nc
(GETFIL): using /home/user/cesm/inputdata/lnd/clm2/landuse.timeseries.nc
Attempting to read landuse.timeseries data .....
(GETFIL): attempting to find local file
landuse.timeseries.nc
(GETFIL): using /home/user/cesm/inputdata/lnd/clm2/landuse.timeseries.nc
Surface Grid Characteristics
longitude points = 606
latitude points = 808
total number of gridcells = 203708
total number of landunits = 452564
total number of columns = 799076
total number of patches = 3650988
total number of cohorts = 0
Decomposition Characteristics
clumps per process = 1


proc = 0 beg gridcell= 1 end gridcell= 1213 gridcells per proc = 1213
proc = 0 beg landunit= 1 end landunit= 2769 landunits per proc = 2769
proc = 0 beg column = 1 end column = 4978 columns per proc = 4978
proc = 0 beg patch = 1 end patch = 21960 patches per proc = 21960
proc = 0 beg cohort = 1 end cohort = 0 coh per proc = 0
proc = 0 nclumps = 1
check_weights ERROR: at g = 241 total PFT weight is
0.960000000000000 active_only = F
check_weights ERROR: at g = 454 total PFT weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 842 total PFT weight is
0.860000000000000 active_only = F
check_weights ERROR: at g = 1011 total PFT weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1022 total PFT weight is
0.990000000000000 active_only = F
check_weights ERROR: at g = 1023 total PFT weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1047 total PFT weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1048 total PFT weight is
0.890000000000000 active_only = F
check_weights ERROR: at g = 1050 total PFT weight is
0.900000000000000 active_only = F
check_weights ERROR: at g = 1051 total PFT weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1066 total PFT weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 1067 total PFT weight is
0.930000000000000 active_only = F
check_weights ERROR: at g = 1070 total PFT weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1073 total PFT weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1074 total PFT weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 241 total col weight is
0.960000000000000 active_only = F
check_weights ERROR: at g = 454 total col weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 842 total col weight is
0.860000000000000 active_only = F
check_weights ERROR: at g = 1011 total col weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1022 total col weight is
0.990000000000000 active_only = F
check_weights ERROR: at g = 1023 total col weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1047 total col weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1048 total col weight is
0.890000000000000 active_only = F
check_weights ERROR: at g = 1050 total col weight is
0.900000000000000 active_only = F
check_weights ERROR: at g = 1051 total col weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1066 total col weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 1067 total col weight is
0.930000000000000 active_only = F
check_weights ERROR: at g = 1070 total col weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1073 total col weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1074 total col weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 241 total lunit weight is
0.960000000000000 active_only = F
check_weights ERROR: at g = 454 total lunit weight is
0.950000000000000 active_only = F
check_weights ERROR: at g = 842 total lunit weight is
0.860000000000000 active_only = F
check_weights ERROR: at g = 1011 total lunit weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1022 total lunit weight is
0.990000000000000 active_only = F
check_weights ERROR: at g = 1023 total lunit weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1047 total lunit weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1048 total lunit weight is
0.890000000000000 active_only = F
check_weights ERROR: at g = 1050 total lunit weight is
0.900000000000000 active_only = F
check_weights ERROR: at g = 1051 total lunit weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1066 total lunit weight is
0.970000000000000 active_only = F
check_weights ERROR: at g = 1067 total lunit weight is
0.930000000000000 active_only = F
check_weights ERROR: at g = 1070 total lunit weight is
0.880000000000000 active_only = F
check_weights ERROR: at g = 1073 total lunit weight is
0.980000000000000 active_only = F
check_weights ERROR: at g = 1074 total lunit weight is
0.950000000000000 active_only = F

If you are seeing this message at the beginning of a run with
use_init_interp = .true. and init_interp_method = "use_finidat_areas",
and you are seeing weights less than 1, then a likely cause is:
For the above-mentioned grid cell(s):
The matching input grid cell had some non-zero-weight subgrid type
that is not present in memory in the new run.

If you are using a ctsm5.2 or later fsurdat file containing
PCT_OCEAN > 0, then you may solve the error by setting
convert_ocean_to_land = .true.

ENDRUN:
ERROR: ERROR in subgridWeightsMod.F90 at line 703

 

[oleson]

Wetlands don't need to be on the landuse file.
One thing to check is that the data on the surface dataset for the year it represents (2000?) should match the data on the landuse timeseries file for that year.
If that looks ok, then make your surface dataset and landuse timeseries files available and I can look at them.

 

[LiwenWu]

Dear Oleson,
Thanks for your patient reply. I checked and confirmed that the surface dataset matches the data on the landuse timeseries file for 2000. I have no idea about how to solve the error. I also checked several gridecells the logfile mentioned and they are all 100 for pft=0. But the logfile shows that they are < 100%. Another question is, my research area Shaanxi Province is an irregular region so I define elementMask=1 in mask_mesh.nc to prevent grids outside Shaanxi Province from participating in model operations. But it seems that some gridcells mentioned in the logfiles seem to be outside Shaanxi Province. Why could this happen? I emailed the surfdata and landuse.timeseries file to you, could you please help me check if there are any issues with the files?
Thanks for your patience and help again.

 

[oleson]

I don't see anything wrong with your files. Have you looked at your lnd_mesh.nc file to see that it has zero's where there is ocean or where you don't want to simulate anything?

 

[LiwenWu]

Dear Oleson,
First of all, thank you very much for helping me check the surface and landuse files. The lnd_mesh.nc was created using ESMF_Scrip2Unstruct tool and elementMask=1 everywhere. I only modified the mask_mesh.nc and domain.lnd.nc file to distinguish the grid in and out Shaanxi Province. What's more, these files have been read and successfully run in the spin up case. Perhaps the error still lies in the landuse.timeseries file? I really have no idea about this...I attached my code and all logfiles below, could you please help me to see if there is anything wrong?
Looking forward to your reply. Thanks for your patience and help again.

 

[oleson]

How do the lnd_mesh.nc and atm_mesh.nc files differ? For regional simulation, we use lnd_mesh.nc for both LND_DOMAIN_MESH and ATM_DOMAIN_MESH.

./xmlchange LND_DOMAIN_MESH=/home/user/xunbin/CTSM-ctsm5.2.005/tools/mksurfdata_esmf/lnd_mesh.nc
./xmlchange ATM_DOMAIN_MESH=/home/user/xunbin/CTSM-ctsm5.2.005/tools/mksurfdata_esmf/atm_mesh.nc

One thing you could try is to create a lnd_mesh.nc file that has 1s over land that you want to simulate and 0s for ocean or land you don't want to simulate, and use that file for LND_DOMAIN_MESH, ATM_DOMAIN_MESH, and MASK_MESH.

I also re-checked your landuse file and I do see gridcells where the landunits don't add up to 100%. I'm checking the following:

PCT_NATVEG + PCT_LAKE + PCT_GLACIER + PCT_WETLAND + PCT_CROP + PCT_URBAN

Note that PCT_URBAN has three landunits so you need to add them together.

 

[LiwenWu]

Dear Oleson,
The resolution of my atmospheric forcing data is 0.1 deg while my surfdata is 0.01 deg, so I created two different mesh file for lnd and atm data. Following your suggestions, I modified lnd_mesh.nc. It is worth noting that the guidance of Setting up (high-res sparse) regional-grid CTSM simulations#1919( Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP CTSM · Discussion #1919) mentioned that The mask in mask_mesh.nc represents the inverse of the land mask, so land points = 0 and non-land points = 1. So the data in my lnd_mesh.nc and mask_mesh.nc files is exactly the opposite. And when I ran the model using the new lnd_mesh.nc file, the same error occured but there are fewer error gridcells reported in the logfile than before. The logfiles are attached below. I also wonder if I can use the 0.01 deg mesh file for 0.1 deg atmospheric forcing data?
As for the landuse data, since there are no variable PCT_NATVEG and PCT_WETLAND in landuse.timeseries file, I would like to know how you obtained the data and summed it up?
I am not familiar enough with CLM. Perhaps my question is shallow, but it has been troubling me for some time. I apologize for bothering you multiple times and thank you for your patient guidance and help.

 

[LiwenWu]

Recently, I have also been trying to use CLM default data and the mksurfdata_ esmf tool to create the landuse.timeseries file, but when I run
./gen_mksurfdata_namelist --start-year 2005 --end-year 2015 --model-mesh-nx 606 --model-mesh-ny 808 --nocrop --model-mesh lnd_mesh.nc --res 0.01x0.01_shanxi, an error occurred:

---

WARNING: run ./download_input_data to try TO OBTAIN MISSING FILES
WARNING: landunit_input_fnam3: /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/rawdata/lake_area/mksurf_lake_0.05x0.05_hist_clm5_hydrolakes_2015.cdf5.c20220325.nc does not exist
WARNING: run ./download_input_data to try TO OBTAIN MISSING FILES
Successfully created input landuse file landuse_timeseries_hist_2000-2015_16pfts.txt
fatal: 不是一个 git 仓库（或者直至挂载点 / 的任何父目录）
停止在文件系统边界（未设置 GIT_DISCOVERY_ACROSS_FILESYSTEM）。
fatal: 不是一个 git 仓库（或者直至挂载点 / 的任何父目录）
停止在文件系统边界（未设置 GIT_DISCOVERY_ACROSS_FILESYSTEM）。
Traceback (most recent call last):
File "/home/user/xunbin/CTSM-ctsm5.2.005/tools/mksurfdata_esmf/../../python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 368, in main
gitdescribe = subprocess.check_output(git_desc_cmd, shell=True).strip()
File "/home/user/python3/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/home/user/python3/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'git -C /home/user/xunbin/CTSM-ctsm5.2.005/tools/mksurfdata_esmf describe' returned non-zero exit status 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./gen_mksurfdata_namelist", line 34, in <module>
main()
File "/home/user/xunbin/CTSM-ctsm5.2.005/tools/mksurfdata_esmf/../../python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 374, in main
gitdescribe = subprocess.check_output("git describe", shell=True).strip()
File "/home/user/python3/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/home/user/python3/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'git describe' returned non-zero exit status 128.

---

Could you please give me some advice?

 

[oleson]

I'm sorry, I made a mistake, I provided the landunit check for the surface dataset, not the landuse timeseries. You are right that there is no PCT_NATVEG on the landuse timeseries. The model figures that out from the other landunits on the landuse timeseries.
Are you using 0.1 atmospheric forcing data? I see this in your atm log file:

/home/user/cesm/inputdata/lmwg/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.2000-01.nc

which is the GSWP3 forcing data.
Maybe @slevis can answer your question about mksurfdata_esmf.

 

[LiwenWu]

Dear Oleson,
After multiple attempts, I think I have found the reason —— The series of variables named PCT_*_MAX must correspond to data from the year 2000. Previously, I set PCT_CROP, PCT_CFT and PCT_LAKE to data from the year 2000, while PCT_NAT_PFT and PCT_URBAN were set to data from the year 2015. I have also tried to change them all to data from 2015, but still got an error message. Only using data from 2000 can function properly. But from my point of view, if the sum of all variables is equal to 100%, then the maximum values of each variable should not be in the same year. So why can these variables only run normally when they correspond to the year 2000? Please correct me if I have a misunderstanding of the PCT_*_MAX variables.
Based on this situation, may I ask if it is feasible for me to run the model using landuse.timeseries file that MAX data corresponding to data from the year 2000? How does the MAX variable affect the model? Does this have a significant impact on the output of the model?
Any advice would be greatly appreciated. Thank you.

 

[oleson]

My understanding is that the PCT_*_MAX variables indicate the maximum PCT for that landunit and for each gridcell over the timeseries that the landuse is specified for. For example, for urban, any gridcell that has PCT_URBAN_MAX greater than zero is run even if at the start of the run or for any year that PCT_URBAN is zero, in order to provide initial conditions for that gridcell when PCT_URBAN becomes greater than zero. So, for example, PCT_URBAN for a given gridcell may be zero in year 2015, but by year 2050, PCT_URBAN becomes greater than 10% and will be initialized reasonably because it will have been running since the start of the simulation. This was done to be more computationally efficient than just running an urban landunit in every gridcell, regardless of whether it is greater than 0% at any point during the simulation as specified by the landuse timeseries.
So I don't see that the MAX data should correspond to year 2000.

 

[LiwenWu]

I see. Thank you for your timely reply, Oleson. Now I get an new error after running for 5 years. Based on your explanation, I believe this error is related to the setting of PCT_*_MAX. The error shows:

---

set_landunit_weight ERROR: Attempt to assign non-zero weight to a non-existent
landunit
g, l, ltype, weight = 189 -9999 5
1.000000000000000E-002
iam = 100: local landunit index = -9999
iam = 100: global landunit index = -9999
iam = 100: global gridcell index = 7
iam = 100: gridcell longitude = 0.0000000
iam = 100: gridcell latitude = 0.0000000
iam = 100: landunit type = 20442
ENDRUN:
ERROR: ERROR in subgridWeightsMod.F90 at line 529
set_landunit_weight ERROR: Attempt to assign non-zero weight to a non-existent
landunit
g, l, ltype, weight = 181 -9999 5
2.000000000000000E-002
iam = 101: local landunit index = -9999
iam = 101: global landunit index = 0
iam = 101: global gridcell index = 0
iam = 101: gridcell longitude = ************
iam = 101: gridcell latitude = 34.1900000
iam = 101: landunit type = 20429
ENDRUN:
ERROR: ERROR in subgridWeightsMod.F90 at line 529
set_landunit_weight ERROR: Attempt to assign non-zero weight to a non-existent
landunit
g, l, ltype, weight = 1053 -9999 5
5.000000000000000E-002
iam = 111: local landunit index = -9999
iam = 111: global landunit index = 0
iam = 111: global gridcell index = 44601
iam = 111: gridcell longitude = 107.1000000
iam = 111: gridcell latitude = 33.4100000
iam = 111: landunit type = 20442
ENDRUN:
ERROR: ERROR in subgridWeightsMod.F90 at line 529

---

In my landuse.timeseries file, I replaced the data from 2000 to 2004 with data from 2000, replaced the data from 2005 to 2009 with data from 2005, and so on. And the PCT_*_MAX were replaced by the data from 2000. So the running in the first five years was normal. And because some grids in the 2005 data exceeded PCT_* _MAX, the above error occurred. May I ask if my understanding is correct?
If I make changes to PCT_*_MAX, the previous error will appear. So what can I do to keep the model running? I would like to know why changing PCT_* _MAX causes the error, and whether it is related to the default settings of the model?

 

[oleson]

Your hypothesis as to the cause of the most recent error you are getting seems reasonable. However, at this point, I can't really answer your specific questions without actually doing the run myself unfortunately. I suggest looking at setting up a transient case out of the box and looking at the surface dataset and landuse timeseries files to see how they are structured and compare that to your own.

 

[Fuhow]

Hello, Liwen Wu.
May I ask have you solve this problem? I met similar errors, firstly it shows with some , then I add PCT_WETLAND to PCT_LAKE, and set PCT_WETLAND to 0，now it shows some errors with total PFT weight, col weight, and lunit weight.



These errors are from /cesm2.2.0/components/clm/src/main/subgridWeightsMod.F90.

 

[Fuhow]

Hello, Liwen Wu.
May I ask have you solve this problem? I met similar errors, firstly it shows with some , then I add PCT_WETLAND to PCT_LAKE, and set PCT_WETLAND to 0，now it shows some errors with total PFT weight, col weight, and lunit weight.

View attachment 5941View attachment 5942

These errors are from /cesm2.2.0/components/clm/src/main/subgridWeightsMod.F90.

Hi, just an update, the error was caused by the inconsistent between PCT_CROP and PCT_CFT.