./case.submit encountered an error and was killed. The application called MPI_Abort(MPI_COMM_WORLD, 1001)

[xli585]

Currently, I am using the BHIST case with a resolution of f09g17. Then, I am running the hybrid simulation from the 2000 restart file and have modified some forcings to be fixed at the value of 2000.
After submitting, the program terminated and reported an error: Aborted (1001) on node 254 (rank 254 in comm 0): The application called MPI_Abort(MPI_COMM_WORLD, 1001)
The error message in lnd.log is: check_weights ERROR: at l = 93 total PFT weight is
0.999972310011532 active_only = F
Is this a problem with the initialization of the clm module? How to solve it?
I added this sentence to the namelists:
&clm_inparm
! finidat = '/share/project/cesm/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c200728.nc'
fsurdat = '/share/project/cesm/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c190214.nc' check_finidat_year_consistency = .false.
use_init_interp = .true.
check_finidat_pct_consistency = .false.

In addition, I attempted to modify the cam, and the fixed forcing is as follows:
&chem_inparm
ext_frc_cycle_yr = 2000
ext_frc_type = 'CYCLICAL'
srf_emis_cycle_yr = 2000
srf_emis_type = 'CYCLICAL'
tracer_cnst_cycle_yr = 2000
tracer_cnst_type = 'CYCLICAL'

When I changed the year, I tried to change it to 2006, but it gave an error and displayed:
/share/project/cesm/inputdata/atm/cam/tracer_cnst/tracer_cnst_halons_3D_L70_184
9-2015_CMIP6ensAvg_c180927.nc
open_trc_datafile: cycle year not found : 2006
ERROR:
open_trc_datafile: cycle year not found /share/project/cesm/inputdata/atm/cam/t
racer_cnst/tracer_cnst_halons_3D_L70_1849-2015_CMIP6ensAvg_c180927.nc

What is the reason for this?

 

[sacks]

Please provide the information in Information to include in help requests so that we can better understand what you are doing. Specifically:
- What is the model version you're using?
- Please document all of the steps you took in setting up the hybrid case, leading up to the error.

It would also help if you described a little more about what you're trying to do: It's possible that we can recommend other ways to accomplish this that may work better.

I can't tell from your initial message if you made changes to the namelist before or after getting this error. i.e., are these namelist settings causing the error, or are these changes you made to try to solve the error? Have you confirmed that you can successfully run the model for your given compset and resolution before changing the various namelist settings?

It's often problematic to mix and match surface datasets and initial conditions from a different case, so that may be the source of the problem here, but I'd like to better understand your run setup.

 

[xli585]

Please provide the information in Information to include in help requests so that we can better understand what you are doing. Specifically:
- What is the model version you're using?
- Please document all of the steps you took in setting up the hybrid case, leading up to the error.

It would also help if you described a little more about what you're trying to do: It's possible that we can recommend other ways to accomplish this that may work better.

I can't tell from your initial message if you made changes to the namelist before or after getting this error. i.e., are these namelist settings causing the error, or are these changes you made to try to solve the error? Have you confirmed that you can successfully run the model for your given compset and resolution before changing the various namelist settings?

It's often problematic to mix and match surface datasets and initial conditions from a different case, so that may be the source of the problem here, but I'd like to better understand your run setup.

Thank you very much for your reply. I am using the CESM2.1.4 version and the BHIST_f09_g17 model to simulate the actual situation of 2006 and running the hybrid simulation from the 2000 restart file conduct this perturb control experiment. For user_nl_cam, the main modifications are as follows: &chem_inparm ext_frc_cycle_yr = 2006
ext_frc_type = 'CYCLICAL'
srf_emis_cycle_yr = 2006
srf_emis_type = 'CYCLICAL'
tracer_cnst_cycle_yr = 2006
tracer_cnst_type = 'CYCLICAL'

&co2_cycle_nl
co2_flag = .true.
co2_readflux_fuel = .false.
co2_readflux_aircraft = .false.

And an error occurred: open_trc_datafile: /share/project/cesm/inputdata/atm/cam/tracer_cnst/tracer_cnst_halons_3D_L70_184
9-2015_CMIP6ensAvg_c180927.nc
open_trc_datafile: cycle year not found : 2006
ERROR:
open_trc_datafile: cycle year not found /share/project/cesm/inputdata/atm/cam/t
racer_cnst/tracer_cnst_halons_3D_L70_1849-2015_CMIP6ensAvg_c180927.nc
but if I change the cycle_yr = 2000, it's ok.!
I checked the data and found that there was an issue with the timeline. The "time" (time) object is 2kB, ranging from 1850-01-16 12:00:00 to 0164-12-16 12:00:00. Therefore, I am re-downloading the emission data.



Then in user_nl_clm, the following changes were made: &clm_inparm
fsurdat = '/share/project/cesm/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c190214.nc' check_finidat_year_consistency = .false.
use_init_interp = .true.

And an error occurred: "check_weights". ERROR: At index l = 93, the total PFT weight is 0.999972310011532, and "active_only" is set to False.

 

[slevis]

I may have missed it in your posts, but I definitely see it in your lnd.log:
INIT_INTERP_METHOD = use_finidat_areas

Also at the end of your lnd.log:
If you are seeing this message at the beginning of a run with
use_init_interp = .true. and init_interp_method = "use_finidat_areas",
and you are seeing weights less than 1, then a likely cause is:
For the above-mentioned grid cell(s):
The matching input grid cell had some non-zero-weight subgrid type
that is not present in memory in the new run.

I would start by setting
init_interp_method = 'general'
in your user_nl_clm

 

[sacks]

Sam's point is a very good one (thanks for jumping in to add this, Sam!). This will hopefully resolve the error you're getting from CLM.

Can you clarify if you're trying to run the model for just one year (i.e., run for 2006 then stop), or if you're trying to run for multiple years, repeatedly looping over year-2006 forcing?

I'm not sure what the standard practice is for running the model for a single, arbitrary year, but I could try to ask some others. But, if you're trying to run for a single year (rather than looping over 2006 forcing), then have you tried simply setting ./xmlchange RUN_STARTDATE="2006-01-01"? I would think that that would prevent the need for some of the other changes you're making here, though I'm not positive. Before I ask around, I'd like to understand if that's something you've tried.

 

[xli585]

I may have missed it in your posts, but I definitely see it in your lnd.log:
INIT_INTERP_METHOD = use_finidat_areas

Also at the end of your lnd.log:
If you are seeing this message at the beginning of a run with
use_init_interp = .true. and init_interp_method = "use_finidat_areas",
and you are seeing weights less than 1, then a likely cause is:
For the above-mentioned grid cell(s):
The matching input grid cell had some non-zero-weight subgrid type
that is not present in memory in the new run.

I would start by setting
init_interp_method = 'general'
in your user_nl_clm

Thank you very much for your help!
Set init_interp_method to 'general' It was successfully carried out.

 

[xli585]

Sam's point is a very good one (thanks for jumping in to add this, Sam!). This will hopefully resolve the error you're getting from CLM.

Can you clarify if you're trying to run the model for just one year (i.e., run for 2006 then stop), or if you're trying to run for multiple years, repeatedly looping over year-2006 forcing?

I'm not sure what the standard practice is for running the model for a single, arbitrary year, but I could try to ask some others. But, if you're trying to run for a single year (rather than looping over 2006 forcing), then have you tried simply setting ./xmlchange RUN_STARTDATE="2006-01-01"? I would think that that would prevent the need for some of the other changes you're making here, though I'm not positive. Before I ask around, I'd like to understand if that's something you've tried.

Thank you very much for your reply. I am currently trying to repeatedly loop over the 'forcing' data for the year 2006 and I plan to read in my own forcing data. The error in the Ind module has been resolved by using init_interp_method = 'general'.

 

[sacks]

I'm glad to hear that Sam's suggestion solved your CLM issue.

For your CAM issue: Have you tried searching the forums for other threads related to this "cycle year not found"? I did a quick search and found quite a few threads discussing this issue. From looking at the file that's generating this issue, I see that it only has data for certain years, and doesn't have data for the year 2006 that you're requesting. I haven't looked closely at the threads discussing this issue, but I think the solution may be to either somehow specify that it should interpolate between years or create a version of this file where you have added 2006 forcing data, e.g., by doing this interpolation yourself. But I'm not familiar with this part of the model and haven't looked carefully through these threads myself.

So, as a first step, please look through the forum threads related to "cycle year not found" to see if any of them resolve your issue, if you haven't already done this. If that still doesn't answer your question, then please post this question to either the CAM or CAM-chem forums, where people are likely to have more experience with this issue.

 

[xli585]

I may have missed it in your posts, but I definitely see it in your lnd.log:
INIT_INTERP_METHOD = use_finidat_areas

Also at the end of your lnd.log:
If you are seeing this message at the beginning of a run with
use_init_interp = .true. and init_interp_method = "use_finidat_areas",
and you are seeing weights less than 1, then a likely cause is:
For the above-mentioned grid cell(s):
The matching input grid cell had some non-zero-weight subgrid type
that is not present in memory in the new run.

I would start by setting
init_interp_method = 'general'
in your user_nl_clm

Regrettably, after I changed init_interp_method to 'general' last time, the simulation ran successfully, but there were no output files. So I checked the lnd.log file and found that it had been running for the first month without any error messages or kill commands, and it kept running continuously. It has been running for approximately two days. Do you know what the reason is? I have placed the log file and the namelist here.

 

[xli585]

I'm glad to hear that Sam's suggestion solved your CLM issue.

For your CAM issue: Have you tried searching the forums for other threads related to this "cycle year not found"? I did a quick search and found quite a few threads discussing this issue. From looking at the file that's generating this issue, I see that it only has data for certain years, and doesn't have data for the year 2006 that you're requesting. I haven't looked closely at the threads discussing this issue, but I think the solution may be to either somehow specify that it should interpolate between years or create a version of this file where you have added 2006 forcing data, e.g., by doing this interpolation yourself. But I'm not familiar with this part of the model and haven't looked carefully through these threads myself.

So, as a first step, please look through the forum threads related to "cycle year not found" to see if any of them resolve your issue, if you haven't already done this. If that still doesn't answer your question, then please post this question to either the CAM or CAM-chem forums, where people are likely to have more experience with this issue.

Thank you for your feedback. Once I resolve the issue with clm, I will move to the CAM or CAM-chem forums. Unfortunately, after I changed init_interp_method to 'general' last time, the simulation ran successfully, but there were no output files. So I checked the lnd.log file and found that it had been running for the first month without any error messages or kill commands, and it kept running for about two days. Do you know what the reason is? I also asked Sam the same question. Best regards.

 

[xli585]

感谢你的反馈。一旦我解决了 clm 的问题，我会转到 CAM 或 CAM-chem 论坛。不幸的是，上次我把 init_interp_method 改成“通用”后，模拟运行成功了，但没有输出文件。于是我检查了 lnd.log 文件，发现它运行了第一个月，没有任何错误信息或停止命令，并且持续运行了大约两天。你知道原因是什么吗？我也问了 Sam 同样的问题。 此致敬意。