POP error restarting CESM piControl run

[bmduran]

Hi,

I am trying to do a simple hybrid run using the CESM2 piControl runs. I picked a random year (1021) from the restart files available at /glade/campaign/collections/cmip/CMIP6/restarts/b.e21.B1850.f09_g17.CMIP6-piControl.001/1001-1201/ . The model is able to build and starting running, but quickly I encounter the following error in the POP log files:

Requested sp_Fe_lim_Cweight_avg_100m
(request_tavg_field) FATAL ERROR: requested field unknown
POP aborting...
FATAL ERROR: requested field unknown

I am unsure if this is related to missing information in the restart files I am using, improper setup of my hybrid case, or a conflicting compset (I'm using the standard B1850 setup). Any suggestions would be appreciated!
Case dir: /glade/work/bduran/test-picontrol-branch-3
run dir: /glade/derecho/scratch/bduran/test-picontrol-branch-3

What version of the code are you using?
cesm2.1.5
./create_newcase --case /glade/work/bduran/test-picontrol-branch-3 --compset B1850 --res f09_g17


Describe every step you took leading up to the problem:
./xmlchange GET_REFCASE=FALSE,RUN_REFCASE=b.e21.B1850.f09_g17.CMIP6-piControl.001,RUN_TYPE=hybrid,RUN_REFDATE=1021-01-01
removed the setting of init_interp_method= 'use_finidat_areas' in env_run.xml as per prior guidance
Prestaged restarts into run directory

 

[mlevy]

There is a somewhat complicated procedure for telling POP what diagnostics MARBL will be providing so that POP can add fields to the history files, and it looks like something is going wrong in that step. If you look in /glade/work/bduran/test-picontrol-branch-3/Buildconf/popconf, there are several files involved:

1. gx1v7_tavg_contents is the list of diagnostics that POP will write to files, and it includes all of the MARBL diagnostics (that's good)
2. ecosys_diagnostics is a file that helps differentiate between diagnostics that POP knows how to compute and diagnostics that MARBL will provide; for some reason, the section MARBL-generated diagnostics is empty (that's bad)
3. marbl_diagnostics_operators is just a list of the MARBL-generated diagnostics, and it helps POP define local buffers for storing the MARBL diagnostics. It is also empty, I think because it is generated in the same process as ecosys_diagnostics

So I'm confused about why ecosys_diagnostics is missing MARBL diagnostics that are included in gx1v7_tavg_contents; if you go to your case directory and run ./preview_namelists are their any helpful messages written to the screen? Or do these files get updated?

 

[bmduran]

Hi Michael,

Thanks for your explanation! That does seem slightly complicated.

After I run ./preview_namelists, it appears that these files get updated. Specifically, there is now a list of diagnostics under the MARBL-generated diagnostics section in ecosys_diagnostics (in prior runs this was empty). Output attached below.

However, marbl_diagnostics_operators remains empty.

I'll try to rebuild and submit the case and see if it generates a new error. Thanks for providing some direction!