Coupled CESM2 run terminated on Derecho: timer for "a:PIO:pre_pio_write_nf" had not been started

[ll859]

Hello!

I've been working on conducting fully coupled simulations for PD (e.g., 2006-2011) and the future (e.g., 2089-2094) under different scenarios (SSP245 and 585) in CESM2.1.4 on Derecho.

I successfully conducted simulations for those periods on Cheyenne using CESM2.1.2 and the restart field I found below.

/glade/campaign/collections/cmip/CMIP6/restarts/

e.g., b.e21.BHIST.f09_g17.CMIP6-historical.001/b.e21.BHIST.f09_g17.CMIP6-historical.001.rest.2010-01-01-00000.tar

Now, I am trying to rerun the simulations using CESM2.1.4 with these restart files generated on Cheyenne, but I encountered a series of error messages.

After resolving some of the messages, I can now generate output files for the first month. However, the model run was terminated afterward with the following error message found in the cesm.log file.

Error message:
dec0662.hsn.de.hpc.ucar.edu 0: WARNING: Rearr optional argument is a pio2 feature, ignored in pio1
dec0662.hsn.de.hpc.ucar.edu 0: GPTLstopf thread 0: timer for "a:PIO:pre_pio_write_nf" had not been started.

Does anyone have any suggestions, comments, or insights into what might be happening here?

Many thanks!

Quote Reply

 

[dbailey]

I wonder if you move to the tag release-cesm2.1.5? I'm not sure that 2.1.2 is supported on derecho.

 

[jedwards]

You should use 2.1.5 on derecho. The error message that you are pointing out indicates that a timer end was encountered without a cooresponding timer begin statement. This usually happens when the code has already encountered some other error that causes it to exit. If you point me to your case directory on derecho I can take a closer look.

 

[ll859]

Thank you both so much for your prompt responses!

I've tested CESM2.1.4 but haven't tried CESM2.1.5 yet. Several modifications need to be made to the model to incorporate our changes to some routines. The modified CESM2.1.4 performs well with MERRA2 meteorological fields on Derecho but isn't working for these coupled runs.

I have also tried porting CESM2.1.2 to Derecho and performing the coupled simulations.

Here is the case directory for the test conducted with CESM2.1.2.

/glade/work/longei/Cheyenne/TESTS/MODELS/cases/Future_runs/derechoCAM6DU08_LiGMD_YR2075Hybrid_SSP585_inuse01

I've also attempted to run CESM2.1.4 without making any modifications to the code from our end, and conducted a hybrid run for the future under SSP585. However, the simulation job continues without producing any model outputs.

You can find the case here
/glade/work/longei/Cheyenne/TESTS/MODELS/cases/CESM214_bulk01_FUSSP585_test01

I will try CESM2.1.5 next.

Your time and assistance are greatly appreciated!

 

[ll859]

Here's an update in case it's helpful for others following this thread.

CESM2.1.5 is working with the restart files on Derecho.

I'm currently working on integrating our modifications to the model. It should also work well on Derecho.

Many thanks to both of you, Jim and David! @jedwards @dbailey

​