Job not completing at wallclock time

[rsansom]

I'm currently porting CESM2 to our local hpc at the University of Leeds. I have been having issues with resubmissions. The initialisation job runs, and any resubmit just runs from the beginning again. I can't seem to find any errors in the log files, nor any confirmation that the job finished successfully. It seems that it is perhaps just being cut off at the wallclock time limit without actually wrapping up the run. If I use the immediate resubmission flag, I have the same issue of each run beginning from the start again, instead of picking up where the last one left off. The batch scheduler we use is sge, so I wonder if it's an issue porting with that. Any ideas on where the configuration might be wrong?

I have attached log files, stout and sterr, and my .xml configs.

 

[jedwards]

According to the coupler log your model is not completing and so a second run will start from the begining. I can't be sure but it looks as if you are hanging after day 3. It wrote a timestamp of 2022-01-14 12:09:29. Can you tell me when the wallclock time expired?

 

[rsansom]

Thanks for your reply. So it had JOB_WALLCLOCK_TIME of 30 mins, and I can see model execution starting at 2022-01-14 11:42:25 and my job end time is down as 12:12:22. I can see in the job info that it was killed due to wallclock time. I still wonder if I haven't configured the batch scheduler correctly, so somehow the timings of the run are not being communicated properly, but I'm not sure what exactly the culprit would be.

 

[jedwards]

If you just want to test the resubmit capability there are tests in the scripts_regression_tests.py to do that. If you want to test with the
particular case you are attempting then you need to increase the JOB_WALLCLOCK_TIME and possibly also reduce the run length so that it will have
sufficient time to run to completion.

 

[rsansom]

Ah yes, thank you for your help. I had been misunderstanding the STOP_N parameter...