Error related to restart files

[James King]

Hi all,

Happy New Year! I have run into an error when trying to re-run a case I had previously run. I set RUN_TYPE to startup, CONTINUE_RUN to FALSE, RESUBMIT to 14 [years], and RUN_STARTDATE to 0001-01-01. The model runs for 1 year then crashes. The error is in the coupler logs as follows:

(seq_infodata_Init) read rpointer file rpointer.drv
(seq_infodata_Init) restart file from rpointer= CAM_chem_SSP1-2.6_2050_maxforest.cpl.r.0002-01-01-00000.nc
(seq_io_read_openfile) ERROR: file invalid
CAM_chem_SSP1-2.6_2050_maxforest.cpl.r.0002-01-01-00000.nc
ERROR: Unknown error submitted to shr_abort_abort.

Now, the restart file 'CAM_chem_SSP1-2.6_2050_maxforest.cpl.r.0002-01-01-00000.nc' does not exist, as the model has for some reason started from the year 0005. This is the last year of a previous run I did with this case. How do I make sure the model starts from 0001-01-01? Do I need to do something with RUN_REFDATE?
Model version is CESM2.2.0, full version info and relevant log files are attached.

Thanks,

James

 

[James King]

Edit - log files would be attached, but I get a server error when trying to do so.

 

[jedwards]

You may need to add a .txt suffix to the filenames to upload them.

 

[James King]

You may need to add a .txt suffix to the filenames to upload them.

They are .txt files but I am unable to upload them. I think there may be some server issues with the forum.

 

[joneill]

Just testing the upload feature here. Thanks for everyone's patience.

 

[jedwards]

@James King please try again.

 

[James King]

Thanks Joseph. Here are the log files relating to this issue.

 

[jedwards]

So this is the log from the first resubmit correct? I don't see any evidence for your statement "the model has for some reason started from the year 0005."
Is the file CAM_chem_SSP1-2.6_2050_maxforest.cpl.r.0002-01-01-00000.nc in your run directory? Is it in the archive directory? Is it possible there was an error at the end of the previous run that you missed?

 

[James King]

Hi,

This is the log from the first resubmit after which the model crashed (i.e. the only logs in the run directory). There is no file called 'CAM_chem_SSP1-2.6_2050_maxforest.cpl.r.0002-01-01-00000.nc' in the run directory, but there is in the archive directory from the previous time I ran this case. Maybe the issue here is that I didn't clean out the archive directory before re-running the case (having made a few minor edits to its configuration)?

Best,

James

 

[jedwards]

The st_archive process should have left that file in the run directory after the first run completed along with copying it to the archive directory.

 

[James King]

The st_archive process should have left that file in the run directory after the first run completed along with copying it to the archive directory.

That's what I thought, but the file is missing. Re your previous point, the last time I ran this case it completed without any errors and archived the output. Do the logs give any hints as to what happened in this instance, or would it be best if I ran the model again having cleaned up all the remaining files generated during previous runs of this case (e.g. those in the run directory)?

 

[jedwards]

The logs from the initial run and the (queuing system) log from the st_archive process may shed some light on the problem. I guess we need to figure out why that file is missing.

 

[James King]

Where would I find the st_archive logs?

 

[jedwards]

It should be in the case directory.

 

[James King]

Here's the st_archive log.

 

[jedwards]

Okay - It looks like some confusion occurred and the restart files for 0002-01-01 were deleted as intermediates.
This is because restarts from year 6 were found in your run directory. I would suggest repeating from the beginning with
a clean run directory. You might consider a test run of a few days or months to be sure before you commit to another long run - that is change STOP_OPTION from nyears to ndays.

 

[James King]

Okay - It looks like some confusion occurred and the restart files for 0002-01-01 were deleted as intermediates.
This is because restarts from year 6 were found in your run directory. I would suggest repeating from the beginning with
a clean run directory. You might consider a test run of a few days or months to be sure before you commit to another long run - that is change STOP_OPTION from nyears to ndays.

Thanks, I will give this a try.