Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Error in entering/exiting spinup when running CESM2.1.3 with compset BC5L45BGC.

J

Jerry

Cao
New Member
Hi, everyone,
I'm running CESM2.1.3 with the defined compset BC5L45BGC (longname: 2000_CAM50_CLM45%BGC_CICE_POP2_MOSART_SGLC_SWAV).
After submitting the job, the model encountered an error stating "Error in entering/exiting spinup. spinup_state != restart_file_spinup_state, but do not know what to doERROR in SoilBiogeochemNitrogenStateType.F90 at line 618" and crashed.
The attachment contains the script and log files for the model run.
Does anyone know the cause of the error and how to resolve it?
 

Attachments

  • B2000_CTL.sh.txt
    3 KB · Views: 5
  • cesm.log.3537.230603-142957.TXT
    36.9 KB · Views: 2
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0
 
J

Jerry

Cao
New Member
oleson said:
That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0
Click to expand...
Yes, I have checked the two notes you mentioned and confirmed that the parameter "spinup_state"
 
J

Jerry

Cao
New Member
Jerry said:
Yes, I have checked the two notes you mentioned and confirmed that the parameter "spinup_state"
Click to expand...
oleson said:
That runs fine here on our machine.
Can you confirm that the following out of the box settings are present in your case?
1. spinup_state = 0 in your lnd_in
2. The value of spinup_state in the initial file (identified by finidat in your lnd_in) should also be equal to zero:

finidat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc'

e.g., ncdump -v spinup_state /glade/p/cesmdata/cseg/inputdata/lnd/clm2/initdata_map/clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc should give:

spinup_state = 0
Click to expand...
Thanks for your reply.
Yes, I have checked the two notes you mentioned and confirmed that the parameters "spinup_state" in both
namelist and initial file are set to zero. In addition, I have rerun the case. But the error still persists.
Here are the rerun script, lnd namelist, and log files.
 

Attachments

  • cesm.log.3555.230605-113127.txt
    36.8 KB · Views: 2
  • lnd_in.txt
    7.4 KB · Views: 4
  • runscript.sh.txt
    3.1 KB · Views: 3
slevis

slevis

Moderator
@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.
 
J

Jerry

Cao
New Member
slevis said:
@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.
Click to expand...
Good suggestions!
Actually, I have already outputted the values of spinup_state and restart_file_spinup_state, and they are 0 and -1070201441, respectively. As you mentioned, I suspect there might be an issue with reading the restart_file, resulting in the abnormal value of restart_file_spinup_state.
However, I find it difficult to trace back and find the source input-files of "restart_file_spinup_state".
 
J

Jerry

Cao
New Member
slevis said:
@Jerry that's very interesting and puzzling. The error complains that spinup_state is NOT equal to restart_file_spinup_state.

If you're comfortable modifying code, go to the piece of code that generates the error and write out the values of spinup_state and restart_file_spinup_state. Then keep debugging until you find how the values end up different. This will likely be very time consuming.

Otherwise try to figure out what you are doing differently, because @oleson did the same run and it worked. If I had to guess, the model is using a different restart file from the one you think it's using. It often helps to look at the other .log. files (e.g. in this case lnd.log) to see what the model is doing.
Click to expand...
It has been determined that the issue lies with the input files. Our system administrator did not fully download the required input files, i.e., "clmi.BHIST.2000-01-01.0.9x1.25_gx1v7_simyr2000_c181015.nc", resulting in the file being incomplete or corrupted.
However, it is still possible to use the "ncdump" command to view variables in an incomplete file, which led me to think that the input file was ready.
We thank for both your and @oleson's help!
 
C

CGL

CGL
Member
Jerry said:
Good suggestions!
Actually, I have already outputted the values of spinup_state and restart_file_spinup_state, and they are 0 and -1070201441, respectively. As you mentioned, I suspect there might be an issue with reading the restart_file, resulting in the abnormal value of restart_file_spinup_state.
However, I find it difficult to trace back and find the source input-files of "restart_file_spinup_state".
Click to expand...
I got same error report. But the spinup_state of initial file is 2. I fully download the required input files. How can I fix it? Here is cesm log and lnd_in files.
 
You must log in or register to reply here.
Top