POP aborting: Ocean tracer module problem in B1850 run

[yiyi063@email_arizona_edu]

Hello there, I am running fully coupled pre-industrial simulation (B1850) with CESM1_2_2_1. The initial condition I use for this run comes from CESM large ensemble B1850 control run: b.e11.B1850C5CN.f09_g16.005.0402-01-01-0000. The ocean tracer module is set to "iage ecosys" for this simulation. But the run has failed within few seconds after I submit the job. The error message I got is mainly from ocean model. Here is the last few lines from cesm.log:49:POP aborting...
51:------------------------------------------------------------------------
51:
51:POP aborting...
51: Error in getting varid for netCDF field
51:
51:------------------------------------------------------------------------
67: Error in getting varid for netCDF field
67:
67:------------------------------------------------------------------------
49: Error in getting varid for netCDF field
49:
49:------------------------------------------------------------------------
-1:MPT ERROR: MPI_COMM_WORLD rank 126 has terminated without calling MPI_Finalize()
-1:     aborting job

I also checked the pop.log file: (passive_tracer_tools:rest_read_tracer_block) reading tracer block from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading PO4_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading PO4_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading NO3_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading NO3_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading SiO3_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading SiO3_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading NH4_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading NH4_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading Fe_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading Fe_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading O2_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading O2_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DIC_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DIC_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DIC_ALT_CO2_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DIC_ALT_CO2_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading ALK_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading ALK_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOC_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOC_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DON_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DON_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOFe_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOFe_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOP_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOP_OLD from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc
     (passive_tracer_tools:read_field_3D) reading DOPr_CUR from ./b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc

So I believe that this run fails because there is no variable "DOPr_CUR" in the ocean initial condition file b.e11.B1850C5CN.f09_g16.005.pop.r.0402-01-01-00000.nc. So I was wondering if anyone has any solution or hint to this problem. The case folder is :/glade/work/yiyi063/B1850_LENS_nudging_test7 and log files can be found here: /glade/scratch/yiyi063/B1850_LENS_nudging_test7/run with file name:XXX.log.190724-175728
I would really appreciate your help!
Best,Yiyi

 

[klindsay]

Ocean BGC changed considerably from the code base used in the CESM large ensemble (LENS), which was based on CESM 1.1, to CESM 1.2. The versions of ocean BGC in these code bases are not compatible. Could you explain why you want to initialize a CESM 1.2 experiment from a LENS restart file, instead of1) using the LENS codebase with the LENS restart file, or2) using an initial state from CESM 1.2 with CESM 1.2? If you really need to use CESM 1.2 with the LENS restart file, you could setinit_ecosys_option='startup'in user_nl_pop2 for 1 submission to get ocean BGC initialized. (Do not do this on subsequent continuation runs though. If you do, then the model will reset ocean BGC on every submission.) Alternatively, if your science does not rely on ocean BGC, you could turn it off, by removing ecosys from OCN_TRACER_MODULES. Ocean BGC is passive by default in CESM 1.2, so turning it off does not change answers (if you keep the pe layout the same).

 

[klindsay]

Ocean BGC changed considerably from the code base used in the CESM large ensemble (LENS), which was based on CESM 1.1, to CESM 1.2. The versions of ocean BGC in these code bases are not compatible. Could you explain why you want to initialize a CESM 1.2 experiment from a LENS restart file, instead of1) using the LENS codebase with the LENS restart file, or2) using an initial state from CESM 1.2 with CESM 1.2? If you really need to use CESM 1.2 with the LENS restart file, you could setinit_ecosys_option='startup'in user_nl_pop2 for 1 submission to get ocean BGC initialized. (Do not do this on subsequent continuation runs though. If you do, then the model will reset ocean BGC on every submission.) Alternatively, if your science does not rely on ocean BGC, you could turn it off, by removing ecosys from OCN_TRACER_MODULES. Ocean BGC is passive by default in CESM 1.2, so turning it off does not change answers (if you keep the pe layout the same).

 

[yiyi063@email_arizona_edu]

Hi Dr. Lindsay, Thanks for your reply! I am doing a nudging experiment, which releases u,v to the simulated u,v from Large ensemble B1850 control run. The code base for the nudging experiment is set for CESM1.2. That's why I would like to initialize a CESM1.2 run from LENS restart files. Actually the ocean BGC is not necessary for my simulation. So I tried to turn it off by set OCN_TRACER_MODULES=" " or OCN_TRACER_MODULES="iage"  in the env_build.xml. But this time I got error messages as following: From cesm.log file:0:POP aborting...
0: named_field_get_index: name not found, SFLUX_CO2
0:
0:------------------------------------------------------------------------
-1:MPT ERROR: MPI_COMM_WORLD rank 126 has terminated without calling MPI_Finalize()
-1:     aborting job

From ocn.log file:========================================================================
 End of initialization======================================================================== Global Time Averages: 01-02-0001 01:00:00
 VDC_BCK:   0.160397863902526
 VVC_BCK:    1.60397863717241
 (io_pio_init)  create file ./B1850_LENS_nudging_test7.pop.h.once.nc tavg file written: ./B1850_LENS_nudging_test7.pop.h.once.nc

So I was wondering if there is any other setting that I should make changes as well. Thanks again! Best,Yiyi

 

[yiyi063@email_arizona_edu]

Hi Dr. Lindsay, Thanks for your reply! I am doing a nudging experiment, which releases u,v to the simulated u,v from Large ensemble B1850 control run. The code base for the nudging experiment is set for CESM1.2. That's why I would like to initialize a CESM1.2 run from LENS restart files. Actually the ocean BGC is not necessary for my simulation. So I tried to turn it off by set OCN_TRACER_MODULES=" " or OCN_TRACER_MODULES="iage"  in the env_build.xml. But this time I got error messages as following: From cesm.log file:0:POP aborting...
0: named_field_get_index: name not found, SFLUX_CO2
0:
0:------------------------------------------------------------------------
-1:MPT ERROR: MPI_COMM_WORLD rank 126 has terminated without calling MPI_Finalize()
-1:     aborting job

From ocn.log file:========================================================================
 End of initialization======================================================================== Global Time Averages: 01-02-0001 01:00:00
 VDC_BCK:   0.160397863902526
 VVC_BCK:    1.60397863717241
 (io_pio_init)  create file ./B1850_LENS_nudging_test7.pop.h.once.nc tavg file written: ./B1850_LENS_nudging_test7.pop.h.once.nc

So I was wondering if there is any other setting that I should make changes as well. Thanks again! Best,Yiyi

 

[klindsay]

The flux coupler is asking POP for CO2 fluxes, and POP cannot provide them if BGC is turned off.A workaround for this is to set CCSM_BGC to CO2A:./xmlchange CCSM_BGC=CO2ASorry for not previously mentioning this.

 

[klindsay]

The flux coupler is asking POP for CO2 fluxes, and POP cannot provide them if BGC is turned off.A workaround for this is to set CCSM_BGC to CO2A:./xmlchange CCSM_BGC=CO2ASorry for not previously mentioning this.

 

[yiyi063@email_arizona_edu]

It works! Thank you so much! Best,Yiyi 

 

[yiyi063@email_arizona_edu]

It works! Thank you so much! Best,Yiyi 

 

[darrel_k_robertson@nasa_gov]

I seem to have a similar problem. Everything seems to be running fine until in the cesm.log I get

NetCDF: Variable not found
------------------------------------------------------------------------

POP aborting...
Error in getting varid for netCDF field CFC11_CUR

------------------------------------------------------------------------

Last call in the ocn.log is
(passive_tracer_tools:rest_read_tracer_block) reading CFC11_CUR from ./b.e21.BW1850.f09_g17.CMIP6-piControl.001_v2.pop.r.0056-01-01-00000.nc

I tried checking a couple of xml variables and got
> ./xmlquery CFC11_CUR
ERROR: No results found for variable CFC11_CUR
> ./xmlquery CCSM_BGC
CCSM_BGC: CO2A


In case it helps, I'm running 2.1.3 I had a successful run with the nominal test case --res f19_g17 --compset B1850 and am now trying to get --res f09_g17 --compset BWHIST working. Any clues on how to fix this would be much appreciated.

 

[darrel_k_robertson@nasa_gov]

The problem seems to have occurred because I changed RUN_STARTDATE in env_run.xml from 1850 to 2000 When I set it back to 1850 it ran fine.
How do I correctly get the BWHIST compset to start from a relatively recent date? I'd like to start from around 2000 so I can run forward a decade. Do I need to do an entire run starting from 1850 or is there a way to jump to a reasonable spun-up match to modern day?

 

[dengchuangwu]

The problem seems to have occurred because I changed RUN_STARTDATE in env_run.xml from 1850 to 2000 When I set it back to 1850 it ran fine.
How do I correctly get the BWHIST compset to start from a relatively recent date? I'd like to start from around 2000 so I can run forward a decade. Do I need to do an entire run starting from 1850 or is there a way to jump to a reasonable spun-up match to modern day?

Dear sir,
Do you solve it? can you tell me how do you solve it? Thank you very much!

 

[ucas_qs]

Ocean BGC changed considerably from the code base used in the CESM large ensemble (LENS), which was based on CESM 1.1, to CESM 1.2. The versions of ocean BGC in these code bases are not compatible. Could you explain why you want to initialize a CESM 1.2 experiment from a LENS restart file, instead of1) using the LENS codebase with the LENS restart file, or2) using an initial state from CESM 1.2 with CESM 1.2? If you really need to use CESM 1.2 with the LENS restart file, you could setinit_ecosys_option='startup'in user_nl_pop2 for 1 submission to get ocean BGC initialized. (Do not do this on subsequent continuation runs though. If you do, then the model will reset ocean BGC on every submission.) Alternatively, if your science does not rely on ocean BGC, you could turn it off, by removing ecosys from OCN_TRACER_MODULES. Ocean BGC is passive by default in CESM 1.2, so turning it off does not change answers (if you keep the pe layout the same).

Hello, klindsay. I'm running the case BHISTcmip6 with a resolution of f19_ g17, my scientific questions don't need marine ecology, so I run ./xmlchange OCN_TRACER_MODULES= iage and ./xmlchange CCSM_ BGC = CO2A (Reference for this website, https://bb.cgd.ucar.edu/cesm/threads/pop-aborting-ocean-tracer-module-problem-in-b1850-run.4831/). And then, I run ./case.setup --reset and ./case.build ,and submit the case. Then, the model reported an error
POP aborting...
chl_option==model, but model_chlorophyll is not registered
I look at pop_in the run directory, the "ecosys_ on" is .false. So the marine ecology maybe closed. I look for "chl_option" in pop namelist file, and find that it is auto-filled by CESM scripts. So it's useless for me to change it to "file" manually because it will be covered. So how can I solve this problem? Another problem, do you have the spin up files for the BHISTcmip6_f19_g17 that I can use as initial data to start running from 1850 without needing spin up? Looking forward to your reply.

 

[mlevy]

Besides adding ecosys to OCN_TRACER_MODULES, compsets that turn on the ecosystem by default can also change OCN_CHL_TYPE and OCN_CO2_TYPE. When you run

$ ./xmlquery OCN_CHL_TYPE,OCN_CO2_TYPE

are either of those values prognostic? (Judging from your error message, it seems likely that OCN_CHL_TYPE is. I don't know if the error check for CO2 comes before or after chlorophyll.) You want them both to be diagnostic if you do not have the ecosystem enabled in POP:

$ ./xmlchange OCN_CHL_TYPE=diagnostic,OCN_CO2_TYPE=diagnostic

in your case directory. (In some cases, OCN_CO2_TYPE=constant in which case you should NOT change it to diagnostic... but since the BHIST compset uses CAM for the ocean component I believe diagnostic is the proper choice for you.)

edit: first code block should be xmlquery, not xmlchange

 

[jeffhu]

Hi Sir, may I ask you that have you solved this issue? If so, would you mind telling me how to do it? Thanks!

Hello, klindsay. I'm running the case BHISTcmip6 with a resolution of f19_ g17, my scientific questions don't need marine ecology, so I run ./xmlchange OCN_TRACER_MODULES= iage and ./xmlchange CCSM_ BGC = CO2A (Reference for this website, POP aborting: Ocean tracer module problem in B1850 run). And then, I run ./case.setup --reset and ./case.build ,and submit the case. Then, the model reported an error
POP aborting...
chl_option==model, but model_chlorophyll is not registered
I look at pop_in the run directory, the "ecosys_ on" is .false. So the marine ecology maybe closed. I look for "chl_option" in pop namelist file, and find that it is auto-filled by CESM scripts. So it's useless for me to change it to "file" manually because it will be covered. So how can I solve this problem? Another problem, do you have the spin up files for the BHISTcmip6_f19_g17 that I can use as initial data to start running from 1850 without needing spin up? Looking forward to your reply.

 

[klindsay]

Running the command
./xmlchange OCN_CHL_TYPE=diagnostic
from your CASEROOT will change the ocean model to using a satellite derived climatology of surface Chl, instead of attempting to use model predicted Chl.