Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

POP aborting: Ocean tracer module problem in B1850 run

Redlichia

Yihui Chen
New Member
Hello there, I am running fully coupled paleoclimate simulation (use B1850) with CESM1_2_2. The ocean tracer module is set to "iage ecosys" for this simulation. I used the initial field in the paleoclimate that I set up, and the model has worked successfully for 200 years. When I change to 'branch' continue to run the next 200 years, the run has failed within few seconds after I submit the job. The error message I got is mainly from ocean model.
Here is the last few lines from cesm.log:

1734 ------------------------------------------------------------------------
1735
1736 POP aborting...
1737 Error in getting varid for netCDF field
1738
1739 ------------------------------------------------------------------------
1740 ------------------------------------------------------------------------
1741
1742 POP aborting...
1743 Error in getting varid for netCDF field
1744
1745 ------------------------------------------------------------------------
1746 --------------------------------------------------------------------------
1747 MPI_ABORT was invoked on rank 22 in communicator MPI_COMM_WORLD
1748 with errorcode 0.
1749
1750 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
1751 You may or may not see output from other processes, depending on
1752 exactly when Open MPI kills them.
1753 --------------------------------------------------------------------------

I also checked the pop.log file:
1531 ------------------------------------------------------------------------
1532 (passive_tracer_tools:rest_read_tracer_block) reading tracer block from /public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT _boundary_ecosys_g37_20220531.nc
1533 (passive_tracer_tools:read_field_3D) reading PO4_CUR from /public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecos ys_g37_20220531.nc
1534 (open_read_netcdf) nsize = 603
1535 (open_read_netcdf) len(data_file%history) = 256
1536 string too short; not enough room to read history attribute from /public/home/x
1537 dsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc

------------------------------------------------------------------------
So I believe that this run fails because there is no variable "PO4_CUR " in the ocean initial condition file T31_KT_boundary_ecosys_g37_20220531.nc. So I was wondering if anyone has any solution or hint to this problem.
T31_kt_boundary_ecosys_g37_20220531.nc is the replacement file corresponding to palaeoclimate that I interpolated from the default file ECOsys_JAN_IC_Gx3V7_NOV2012_rode 2.nc. But the default file Ecosys_jan_ic_gx3v7_nov2012_cor.nc also does not have the variable 'PO4_CUR'. Now I don't know what went wrong, where to find the variable 'PO4_CUR'.
Attached is my error log file: pop.log and cesm.log.
I hope to get your help. I would really appreciate your help!

Best,Yihui
 

mlevy

Michael Levy
CSEG and Liaisons
Staff member
what is in your user_nl_pop file? For a branch run, you do not want to set init_ecosys_init_file or init_ecosys_option because the ecosystem tracers need to be read from the restart file... this is also a common issue for users who set CONTINUE_RUN=TRUE after initializing the ecosystem variables from a non-default file, and unfortunately we do not have a good mechanism in place to avoid these errors.
 

Redlichia

Yihui Chen
New Member
Hello, Michael Levy. Thank you for your timely reply and answer.
Here is the &ecosys_nml from user_nl_pop:
------------------------------------------------------------------------
92 &ecosys_nml
93 init_ecosys_option='ccsm_continue'
94 alk_riv_flux_input%filename='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
95 dfe_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
96 dic_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
97 din_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
98 dip_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
99 doc_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
100 don_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
101 dop_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
102 dsi_riv_flux_input%filename ='/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_river_nutrient_g37_20220531.nc'
103 dust_flux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_dst_g37_20220531.nc'
104 fesedflux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_fesedflux_g37_20220531.nc'
105 init_ecosys_init_file = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc'
106 iron_flux_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_solFe_g37_20220531.nc'
107 nhy_flux_monthly_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ndep_g37_20220531.nc'
108 nox_flux_monthly_input%filename = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ndep_g37_20220531.nc'
------------------------------------------------------------------------
I set CONTINUE_RUN=TRUE in env_run.xml. And init_ecosys_option='ccsm_continue' , init_ecosys_init_file = '/public/home/xdsc0718/work/paleo/INPUTTOPO_T31g37/66ma/T31_KT_boundary_ecosys_g37_20220531.nc', T31_kt_boundary_ecosys_g37_20220531.nc is the replacement file corresponding to palaeoclimate that I interpolated from the default file ECOsys_JAN_IC_Gx3V7_NOV2012_rode 2.nc.
So if i want a 'branch' run to continue this work , I need to comment out these settings in user_nl_pop file ?
Do you have any good solutions to keep the model going? Thank you very much!
Best, Yihui
 

Redlichia

Yihui Chen
New Member
what is in your user_nl_pop file? For a branch run, you do not want to set init_ecosys_init_file or init_ecosys_option because the ecosystem tracers need to be read from the restart file... this is also a common issue for users who set CONTINUE_RUN=TRUE after initializing the ecosystem variables from a non-default file, and unfortunately we do not have a good mechanism in place to avoid these errors.
Hello, Michael Levy.
I comment out init_ecosys_init_file non-default set in user_nl_pop file.
And It works! Thank you! Thank you for your answer, it inspired me.
If there is an opportunity in the future, I hope to have further communication and cooperation with you.
Best,Yihui
 
Top