Timing issue with FCnudged compset

[James King]

Hi all,

I have a couple of FCnudged cases using MUSICA. These are running on Derecho using CESM2.2.0. One of them works, the other doesn't, and I'm not sure why this is. In the case that doesn't run, I get the following error in the atm logs:

FIND_TIMES: ALL data times are after 0.000000000000000E+000
FIND_TIMES: file:
FIND_TIMES: time: 0.000000000000000E+000
ERROR: find_times: all(all_data_times(:) > time)

I have nudging files set up to cover the model start time and run duration. The srf_emis and ext_frc files also have data covering the right period, and both are set to 'INTERP_MISSING_MONTHS', while the FLBC file is set to 'SERIAL' as per the compset default. I've tried running in 'CYCLICAL' and 'SERIAL' modes for the srf_emis and ext_frc files and get the same error each time. I've attached the CAM namelist file for the case that works (which is running using the NCAR-provided CONUS grid) and the case that doesn't (which uses my own MUSICA grid which has been successfully run using different compsets). Any advice on how to resolve this error would be much appreciated.

Best,

James

 

[James King]

I tried running this in debug mode and got this in the cesm.log. Alas it doesn't tell me which is the errant input file. Any ideas?

forrtl: severe (408): fort: (7): Attempt to use pointer CURR_DATA_TIMES when it is not associated with a target

 

[rrbuchholz]

Hi James,
What is the main difference between the two runs? i.e. what did you change in the one that doesn't work?

Usually, the:
FIND_TIMES: ALL data times are after 0.000000000000000E+000
FIND_TIMES: file:
FIND_TIMES: time: 0.000000000000000E+000
ERROR: find_times: all(all_data_times(:) > time)

means the model can't find input data for the times requested to model. What is the file thet the model is trying to open? It should be trying to be loaded right before the FIND_TIMES lines.

Cheers,
Rebecca

 

[James King]

Hi Rebecca,

The atm log just ends at:
READ_NEXT_TRCDATA emiss
FIND_TIMES: ALL data times are after 0.000000000000000E+000
FIND_TIMES: file:
FIND_TIMES: time: 0.000000000000000E+000
ERROR: find_times: all(all_data_times(:) > time)

The cesm log has:
dec0195.hsn.de.hpc.ucar.edu 0: Reading zbgc_nml
dec0195.hsn.de.hpc.ucar.edu 0: MCT::m_Router::initp_: GSMap indices not increasing...Will correct
dec0195.hsn.de.hpc.ucar.edu 0: MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
dec0195.hsn.de.hpc.ucar.edu 0: MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
dec0195.hsn.de.hpc.ucar.edu 0: MCT::m_Router::initp_: GSMap indices not increasing...Will correct
dec0195.hsn.de.hpc.ucar.edu 0: ERROR: find_times: all(all_data_times(:) > time)
dec0212.hsn.de.hpc.ucar.edu 1152: FIND_TIMES: ALL data times are after 0.000000000000000E+000

So it's not clear to me which file is causing the problem. 'zbgc_nml' seems to be something to do with ice but I've not been able to find any files with that name anywhere in the case, bld, or run directories.

The two runs are on different grids so are using different regridded input files.

Thanks,

James

 

[James King]

Update - in the ice namelist for the case which doesn't run, we have this:

&zbgc_nml
bgc_data_dir = "unknown_bgc_data_dir"
bgc_flux_type = "Jin2006"
nit_data_type = "unknown"
phi_snow = 0.5
restart_bgc = .false.
restart_hbrine = .false.
restore_bgc = .false.
sil_data_type = "unknown"
skl_bgc = .false.
tr_bgc_am_sk = .false.
tr_bgc_c_sk = .false.
tr_bgc_chl_sk = .false.
tr_bgc_dms_sk = .false.
tr_bgc_dmspd_sk = .false.
tr_bgc_dmspp_sk = .false.
tr_bgc_sil_sk = .false.
tr_brine = .false.


In the case which does run, we have this:

&zbgc_nml
bgc_data_dir = "unknown_bgc_data_dir"
bgc_flux_type = "Jin2006"
nit_data_type = "unknown"
phi_snow = 0.5
restart_bgc = .false.
restart_hbrine = .false.
restore_bgc = .false.
sil_data_type = "unknown"
skl_bgc = .false.
tr_bgc_am_sk = .false.
tr_bgc_c_sk = .false.
tr_bgc_chl_sk = .false.
tr_bgc_dms_sk = .false.
tr_bgc_dmspd_sk = .false.
tr_bgc_dmspp_sk = .false.
tr_bgc_sil_sk = .false.
tr_brine = .false.

They're exactly the same!

 

[rrbuchholz]

Hi James,

The two runs are on different grids so are using different regridded input files.

My guess is the regridded input files are the issue.
Did you regrid them?
Where are they stored?
Are they available - are the permissions accessible to the model?
Can you send the path to your model runs on scratch?

Best,
Rebecca

 

[James King]

Hi Rebecca,

Yes I did - the emissions input files are in:

/glade/work/jamesking/MUSICA_emissions/UK_JAK_ne30x16/hist/renamed_pre_split/

And the case directory is:

/glade/work/jamesking/derecho_cases/FCHIST_MUSICA_UK_derecho_test

with run dir:

/glade/derecho/scratch/jamesking/FCHIST_MUSICA_UK_derecho_test/run

As far as I can see, the surface emission files do all have data for the year 2010 which is when I'm trying to start the run, though I will double check. The logs show that the model can read and open these files.

Thanks,

James

 

[rrbuchholz]

Hi James,

Hmmm, curious.
I looked in your atm.log file and it seemed to fail right after trying to read the emis_e90.
1. What happens if you comment out the E90 file in the user_nl_cam? The 2010 dates are in that file, but I wonder if it is having issues because of no data for the year before or after. Have you tried INTERP_MISSING_MONTHS for the srf_emis?
2. What file is loaded right after the E90 for the simulation that works? Then I suggest to check that file for the UK simulation.

Hopefully we will find the issue.

Cheers,
Rebecca

 

[James King]

Hi Rebecca,

Thanks for looking into this!

1 - if I comment out the E90 file and use INTERP_MISSING_MONTHS, I get the same error message.

2 - the file in the cam namelist after the E90 one is a GLYALD surface input file. Removing this file doesn't change the error message.

How can you tell from the atm log that the run is failing right after it read in the E90 file? If I could see which file was causing the issue, I might be able to fix this quicker. Can I also infer from this that all the files in the namelist before E90 were OK? These files were made using standard CMIP6 inputs and NCAR-provided scripts, so not sure what the problem is. They all have data for 2010.

Best,

James