POP aborting: Ocean tracer module problem in B1850 run

[Redlichia]

Hello there, I am running fully coupled paleoclimate simulation (use B1850) with CESM1_2_2. The ocean tracer module is set to "iage ecosys" for this simulation. I used the initial field in the paleoclimate that I set up, and the model has worked successfully for 200 years. When I change to 'branch' continue to run the next 200 years, the run has failed within few seconds after I submit the job. The error message I got is mainly from ocean model.
Here is the last few lines from cesm.log:

1734 ------------------------------------------------------------------------
1735
1736 POP aborting...
1737 Error in getting varid for netCDF field
1738
1739 ------------------------------------------------------------------------
1740 ------------------------------------------------------------------------
1741
1742 POP aborting...
1743 Error in getting varid for netCDF field
1744
1745 ------------------------------------------------------------------------
1746 --------------------------------------------------------------------------
1747 MPI_ABORT was invoked on rank 22 in communicator MPI_COMM_WORLD
1748 with errorcode 0.
1749
1750 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
1751 You may or may not see output from other processes, depending on
1752 exactly when Open MPI kills them.
1753 --------------------------------------------------------------------------

I also checked the pop.log file:
1531 ------------------------------------------------------------------------
1532 (passive_tracer_tools:rest_read_tracer_block) reading tracer block from /public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT _boundary_ecosys_g37_20220531.nc
1533 (passive_tracer_tools:read_field_3D) reading PO4_CUR from /public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecos ys_g37_20220531.nc
1534 (open_read_netcdf) nsize = 603
1535 (open_read_netcdf) len(data_file%history) = 256
1536 string too short; not enough room to read history attribute from /public/home/x
1537 dsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc

------------------------------------------------------------------------
So I believe that this run fails because there is no variable "PO4_CUR " in the ocean initial condition file T31_KT_boundary_ecosys_g37_20220531.nc. So I was wondering if anyone has any solution or hint to this problem.
T31_kt_boundary_ecosys_g37_20220531.nc is the replacement file corresponding to palaeoclimate that I interpolated from the default file ECOsys_JAN_IC_Gx3V7_NOV2012_rode 2.nc. But the default file Ecosys_jan_ic_gx3v7_nov2012_cor.nc also does not have the variable 'PO4_CUR'. Now I don't know what went wrong, where to find the variable 'PO4_CUR'.
Attached is my error log file: pop.log and cesm.log.
I hope to get your help. I would really appreciate your help!

Best,Yihui

 

[mlevy]

what is in your user_nl_pop file? For a branch run, you do not want to set init_ecosys_init_file or init_ecosys_option because the ecosystem tracers need to be read from the restart file... this is also a common issue for users who set CONTINUE_RUN=TRUE after initializing the ecosystem variables from a non-default file, and unfortunately we do not have a good mechanism in place to avoid these errors.

 

[Redlichia]

Hello, Michael Levy. Thank you for your timely reply and answer.
Here is the &ecosys_nml from user_nl_pop:
------------------------------------------------------------------------
92 &ecosys_nml
93 init_ecosys_option='ccsm_continue'
94 alk_riv_flux_input%filename='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
95 dfe_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
96 dic_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
97 din_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
98 dip_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
99 doc_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
100 don_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
101 dop_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
102 dsi_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
103 dust_flux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_dst_g37_20220531.nc'
104 fesedflux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_fesedflux_g37_20220531.nc'
105 init_ecosys_init_file = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc'
106 iron_flux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_solFe_g37_20220531.nc'
107 nhy_flux_monthly_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ndep_g37_20220531.nc'
108 nox_flux_monthly_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ndep_g37_20220531.nc'
------------------------------------------------------------------------
I set CONTINUE_RUN=TRUE in env_run.xml. And init_ecosys_option='ccsm_continue' , init_ecosys_init_file = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc', T31_kt_boundary_ecosys_g37_20220531.nc is the replacement file corresponding to palaeoclimate that I interpolated from the default file ECOsys_JAN_IC_Gx3V7_NOV2012_rode 2.nc.
So if i want a 'branch' run to continue this work , I need to comment out these settings in user_nl_pop file ?
Do you have any good solutions to keep the model going? Thank you very much!
Best, Yihui

 

[Redlichia]

what is in your user_nl_pop file? For a branch run, you do not want to set init_ecosys_init_file or init_ecosys_option because the ecosystem tracers need to be read from the restart file... this is also a common issue for users who set CONTINUE_RUN=TRUE after initializing the ecosystem variables from a non-default file, and unfortunately we do not have a good mechanism in place to avoid these errors.

Hello, Michael Levy.
I comment out init_ecosys_init_file non-default set in user_nl_pop file.
And It works! Thank you! Thank you for your answer, it inspired me.
If there is an opportunity in the future, I hope to have further communication and cooperation with you.
Best,Yihui