CESM 2.1.3 hybrid, cannot find restart file named *.case.i* file

[Andy_Wai]

Hello,

I am doing CESM 2.1.3 compset 'BHIST' simulation; I want to save the restart file for future use. but I did not find the restart file called ....cam.i...

/archive/ctr.BHIST.001.v213.t2/rest/1940-03-01-00000
$ ll
total 16207468
-rw-rw-r-- 1 AW AW 985228704 Jul 1 14:54 ctr.BHIST.001.v213.t2.cam.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 1760137528 Jul 1 14:55 ctr.BHIST.001.v213.t2.cam.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 32014336 Jul 1 14:55 ctr.BHIST.001.v213.t2.cam.rs.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 241893376 Jul 1 14:54 ctr.BHIST.001.v213.t2.cice.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 191899368 Jul 1 14:55 ctr.BHIST.001.v213.t2.cism.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 268414648 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 7719366215 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 620308 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.rh0.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 240774260 Jul 1 14:55 ctr.BHIST.001.v213.t2.cpl.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 20756476 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 30079520 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 108390 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.rh0.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 5090246656 Jul 1 14:54 ctr.BHIST.001.v213.t2.pop.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 85217 Jul 1 14:54 ctr.BHIST.001.v213.t2.pop.ro.1940-03-01-00000
-rw-rw-r-- 1 AW AW 14496000 Jul 1 14:54 ctr.BHIST.001.v213.t2.ww3.r.1940-03-01-00000
-rw-r--r-- 1 AW AW 48 Jul 1 14:55 rpointer.atm
-rw-r--r-- 1 AW AW 257 Jul 1 14:55 rpointer.drv
-rw-r--r-- 1 AW AW 257 Jul 1 14:55 rpointer.glc
-rw-r--r-- 1 AW AW 257 Jul 1 14:54 rpointer.ice
-rw-r--r-- 1 AW AW 257 Jul 1 14:54 rpointer.lnd
-rw-r--r-- 1 AW AW 48 Jul 1 14:54 rpointer.ocn.ovf
-rw-r--r-- 1 AW AW 65 Jul 1 14:54 rpointer.ocn.restart
-rw-r--r-- 1 AW AW 65 Jul 1 14:54 rpointer.ocn.tavg.5
-rw-r--r-- 1 AW AW 257 Jul 1 14:53 rpointer.rof


(1) I did specifiy " inithist='ENDOFRUN' " in the 'user_nl_cam'; I have no idea why the *.cam.i* did not generate.

Could you help me identify which command I’m missing? I’m trying to figure out how to set the command so that the *.cam.i* files are generated

(2) I upload the user_nl_cam, and the bash script I used to submit the job case.

Thank you very much.

Ad

 

[peverley]

Hi,

The name of the restart file you're looking for will be in rpointer.atm. It should be the ctr.BHIST.001.v213.t2.cam.r.1940-03-01-00000.nc file (rather than an cam.i file).

I hope I understood your question correctly and that helps!
Courtney

 

[Andy_Wai]

Hi,

The name of the restart file you're looking for will be in rpointer.atm. It should be the ctr.BHIST.001.v213.t2.cam.r.1940-03-01-00000.nc file (rather than an cam.i file).

I hope I understood your question correctly and that helps!
Courtney

Hi,


(1) I can succesuflly run the CESM 2.1.3 using the attached 'user_nl_cam' and 'ctr.BHIST.001.v213.t2.sh';

(1.1) the case name is 'ctr.BHIST.001.v213.t2'; and it can generate files (but I can not find *cam.i.*):

================
[AW@grace2 ctr.BHIST.001.v213.t2]$ pwd
/scratch/user/AW/archive/ctr.BHIST.001.v213.t2
[AW@grace2 ctr.BHIST.001.v213.t2]$ ll
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 atm
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 cpl
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 esp
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 glc
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 ice
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 lnd
drwxrwxr-x 2 AW AW 4096 Jul 1 14:56 logs
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 ocn
drwxrwxr-x 3 AW AW 4096 Jul 1 14:56 rest
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 rof
drwxrwxr-x 3 AW AW 4096 Jul 1 14:57 wav
[AW@grace2 ctr.BHIST.001.v213.t2]$ cd rest/
[AW@grace2 rest]$ ll
drwxrwxr-x 2 AW AW 4096 Jul 1 14:57 1940-03-01-00000
[AW@grace2 rest]$ cd 1940-03-01-00000/
[AW@grace2 1940-03-01-00000]$ ll
-rw-rw-r-- 1 AW AW 985228704 Jul 1 14:54 ctr.BHIST.001.v213.t2.cam.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 1760137528 Jul 1 14:55 ctr.BHIST.001.v213.t2.cam.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 32014336 Jul 1 14:55 ctr.BHIST.001.v213.t2.cam.rs.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 241893376 Jul 1 14:54 ctr.BHIST.001.v213.t2.cice.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 191899368 Jul 1 14:55 ctr.BHIST.001.v213.t2.cism.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 268414648 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 7719366215 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 620308 Jul 1 14:54 ctr.BHIST.001.v213.t2.clm2.rh0.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 240774260 Jul 1 14:55 ctr.BHIST.001.v213.t2.cpl.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 20756476 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.h0.1940-02.nc
-rw-rw-r-- 1 AW AW 30079520 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 108390 Jul 1 14:53 ctr.BHIST.001.v213.t2.mosart.rh0.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 5090246656 Jul 1 14:54 ctr.BHIST.001.v213.t2.pop.r.1940-03-01-00000.nc
-rw-rw-r-- 1 AW AW 85217 Jul 1 14:54 ctr.BHIST.001.v213.t2.pop.ro.1940-03-01-00000
-rw-rw-r-- 1 AW AW 14496000 Jul 1 14:54 ctr.BHIST.001.v213.t2.ww3.r.1940-03-01-00000
-rw-r--r-- 1 AW AW 48 Jul 1 14:55 rpointer.atm
-rw-r--r-- 1 AW AW 257 Jul 1 14:55 rpointer.drv
-rw-r--r-- 1 AW AW 257 Jul 1 14:55 rpointer.glc
-rw-r--r-- 1 AW AW 257 Jul 1 14:54 rpointer.ice
-rw-r--r-- 1 AW AW 257 Jul 1 14:54 rpointer.lnd
-rw-r--r-- 1 AW AW 48 Jul 1 14:54 rpointer.ocn.ovf
-rw-r--r-- 1 AW AW 65 Jul 1 14:54 rpointer.ocn.restart
-rw-r--r-- 1 AW AW 65 Jul 1 14:54 rpointer.ocn.tavg.5
-rw-r--r-- 1 AW AW 257 Jul 1 14:53 rpointer.rof
====================

(2) for the (1), its restart file dir is:

/scratch/user/AW/gcm_models/CESM/input_data_dirs/b.e21.BHIST.f09_g17/cesm2_init/b.e21.BHIST.f09_g17.CMIP6-historical.001

[AW@grace2 b.e21.BHIST.f09_g17.CMIP6-historical.001]$ ll
drwxr-xr-x 2 AW AW 4096 May 3 21:40 1940-01-01-00000


(2.1) I copied the generated 'rest/1940-03-01-00000' of (1) to restart file dir in (1): /scratch/user/AW/gcm_models/CESM/input_data_dirs/b.e21.BHIST.f09_g17/cesm2_init/b.e21.BHIST.f09_g17.CMIP6-historical.001/

now,

[AW@grace2 b.e21.BHIST.f09_g17.CMIP6-historical.001]$ ll
drwxr-xr-x 2 AW AW 4096 May 3 21:40 1940-01-01-00000
drwxrwxr-x 2 AW AW 4096 Jul 2 16:09 1940-03-01-00000

(3) then I used the the attached 'user_nl_cam' and 'ctr.BHIST.001.v213.t2.userest.sh' to submit case, this case name is 'ctr.BHIST.001.v213.t2.userest'; but this time the case has errors:

Important note: differences:

in the 'ctr.BHIST.001.v213.t2.sh' :

case_name=''ctr.BHIST.001.v213.t2'
start_date='1940-01-01'

in the 'ctr.BHIST.001.v213.t2.userest.sh' :

case_name=''ctr.BHIST.001.v213.userest.t2'
start_date='1940-03-01'



(3.1) the CaseStatus of the case 'ctr.BHIST.001.v213.t2.userest':

---------------------------------------------------
2025-07-02 16:12:46: case.setup starting
---------------------------------------------------
2025-07-02 16:13:07: case.setup success
---------------------------------------------------
2025-07-02 16:13:08: case.build starting
---------------------------------------------------
2025-07-02 16:32:41: case.build success
---------------------------------------------------
2025-07-02 16:33:15: case.submit starting
---------------------------------------------------
2025-07-02 16:34:17: case.submit success case.run:15684899, case.st_archive:15684900
---------------------------------------------------
2025-07-02 16:34:33: case.run starting
---------------------------------------------------
2025-07-02 16:35:03: model execution starting
---------------------------------------------------
2025-07-02 16:35:16: model execution success
---------------------------------------------------
2025-07-02 16:35:16: case.run error
ERROR: RUN FAIL: Command 'mpirun -np 1440 /scratch/user/AW/gcm_models/CESM/case_outputs_dirs/ctr.BHIST.001.v213.t2.userest/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /scratch/user/AW/gcm_models/CESM/case_outputs_dirs/ctr.BHIST.001.v213.t2.userest/run/cesm.log.15684899.250702-163433
---------------------------------------------------

and the cesm.log.*** of the case 'ctr.BHIST.001.v213.t2.userest': starting from line 394

...
...

GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc
ERROR:
GETFIL: FAILED to get b.e21.BHIST.f09_g17.CMIP6-historical.001.cam.i.1940-03-01
-00000.nc

...
...

(4) see attached files

Thank you very much.

AD

 

[peverley]

Thanks for elaborating!

Can you attach your atm.log file as well (for the case that you expect to be generating a cam.i file)?

 

[Andy_Wai]

Thanks for elaborating!

Can you attach your atm.log file as well (for the case that you expect to be generating a cam.i file)?

Thank you for your reply; I attached the cesm.log file of the 'ctr.BHIST.001.v213.t2' ; I expected the case 'ctr.BHIST.001.v213.t2' would generate the *cam.i.* for the case 'ctr.BHIST.001.v213.t2.userest';

 

[peverley]

A couple further questions:

For your ctr.BHIST.001.v213.t2 case (that you expect should generate a cam.i file, can you
1. Check the run directory to make sure it's not there (and just wasn't archived for some reason)
2. Check run/atm_in and ensure that inithist = 'ENDOFRUN' (i.e. it isn't getting overwritten somehow)

If neither of those help, let me know and i will try to reproduce the behavior you're seeing on my end.

Courtney