On-demand writing of restart files?

[jrvb]

Greetings!

I'm in the process of setting up some large ensemble runs, and I would like to use spare cycles between other jobs. Since I won't know ahead of time how long each ensemble member will be able to run before being preempted, I need to either (a) write fairly frequent restart files, or (b) notify the running CESM code that it should write restart files and shut down ASAP. Doing (a) is unattractive since all of the extra file writes adds a good deal of overhead and slows down the simulation. Does anyone know if there is a way to do (b)?

Thanks!

-Rob

 

[dbailey]

The default configuration for REST_OPTION and REST_N is that they are the same as STOP_OPTION and STOP_N. This means that restarts will be written at the end of the run. There is also the option of REST_DATE to write the restarts on a specific date. So, perhaps I don't understand option (b) here, but it seems like these would do that?

 

[jrvb]

Thanks for the quick response! Sorry I wasn't clear about my option (b). I'm using a compute environment (Google cloud) where the nodes are preemptable, so at any point the machine can be notified that it needs to shut down, at which point I'll have 30 seconds to do whatever clean shutdown I can. For option (b), what I would like is for my shutdown script to be able to signal the already-running CESM process in some fashion (send a UNIX signal, write to a magic file, etc) and tell it to save a restart file based on whatever it's current state happens to be. Is that more clear? Is there any way to do that?

 

[dbailey]

That I am unsure of. @dobbins any thoughts on this?