Error in entering/exiting spinup when running CESM2.1.3 with compset BC5L45BGC.

[Jerry]

Hi, everyone,
I'm running CESM2.1.3 with the defined compset BC5L45BGC (longname: 2000_CAM50_CLM45%BGC_CICE_POP2_MOSART_SGLC_SWAV).
After submitting the job, the model encountered an error stating "Error in entering/exiting spinup. spinup_state != restart_file_spinup_state, but do not know what to doERROR in SoilBiogeochemNitrogenStateType.F90 at line 618" and crashed.
The attachment contains the script and log files for the model run.
Does anyone know the cause of the error and how to resolve it?

 

[oleson]

That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0

 

[Jerry]

That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0

Yes, I have checked the two notes you mentioned and confirmed that the parameter "spinup_state"

 

[Jerry]

Yes, I have checked the two notes you mentioned and confirmed that the parameter "spinup_state"

That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0

Thanks for your reply.
Yes, I have checked the two notes you mentioned and confirmed that the parameters "spinup_state" in both
namelist and initial file are set to zero. In addition, I have rerun the case. But the error still persists.
Here are the rerun script, lnd namelist, and log files.

 

[slevis]

@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.

 

[Jerry]

@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.

Good suggestions!
Actually, I have already outputted the values of spinup_state and restart_file_spinup_state, and they are 0 and -1070201441, respectively. As you mentioned, I suspect there might be an issue with reading the restart_file, resulting in the abnormal value of restart_file_spinup_state.
However, I find it difficult to trace back and find the source input-files of "restart_file_spinup_state".

 

[Jerry]

@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.

It has been determined that the issue lies with the input files. Our system administrator did not fully download the required input files, i.e., "clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc", resulting in the file being incomplete or corrupted.
However, it is still possible to use the "ncdump" command to view variables in an incomplete file, which led me to think that the input file was ready.
We thank for both your and @oleson's help!

 

[CGL]

Good suggestions!
Actually, I have already outputted the values of spinup_state and restart_file_spinup_state, and they are 0 and -1070201441, respectively. As you mentioned, I suspect there might be an issue with reading the restart_file, resulting in the abnormal value of restart_file_spinup_state.
However, I find it difficult to trace back and find the source input-files of "restart_file_spinup_state".

I got same error report. But the spinup_state of initial file is 2. I fully download the required input files. How can I fix it? Here is cesm log and lnd_in files.

 

[oleson]

Answered in BGC -cn or -fates spinup problem