POP aborting: nt too small for module ECOSYS_DRIVER

[racosta6]

Hi Everyone,

I'm trying to run a cesm1_2_2_1 paleo simulation with "BGC" or "OCN_TRACER_MODULES = iage ecosys ciso" on. This usually runs without BGC but I'm receiving this error from my cesm.log file

467:POP aborting...

411: nt too small for module ECOSYS_DRIVER

and my ocn.log files ending with the lines below.

------------------------------------------------------------------------

ECOSYS_DRIVER ind_begin = 3

ECOSYS_DRIVER ind_end = 36

------------------------------------------------------------------------

(passive_tracer_tools:set_tracer_indices) nt = 30

(passive_tracer_tools:set_tracer_indices) cumulative_nt = 36

Does anyone know how to increase nt since the passive_tracer_tools.F90 file shows

if (cumulative_nt > nt) then
call document(subname, 'nt', nt)
call document(subname, 'cumulative_nt', cumulative_nt)
error_string = 'nt too small for module ' /&
&/ module_string
call exit_POP(sigAbort, error_string)
end if

I appreciate the help in advance!

 

[mlevy]

Can you walk through the process you used to generate the case? The two things I'm interested in are

1. what compset are you using? (Did you modify OCN_TRACER_MODULES by hand? if so, did you do this before or after building?)
2. are any source mods in place?

but it would be helpful to see the whole process in case the issue isn't related to those two topics. Also, if what machine are you using for the run? If it happens to be cheyenne, can you please point me to your case and run directories?

 

[racosta6]

Can you walk through the process you used to generate the case? The two things I'm interested in are

1. what compset are you using? (Did you modify OCN_TRACER_MODULES by hand? if so, did you do this before or after building?)
2. are any source mods in place?

but it would be helpful to see the whole process in case the issue isn't related to those two topics. Also, if what machine are you using for the run? If it happens to be cheyenne, can you please point me to your case and run directories?

Hi Michael,

This is technically the compset I used B_1850_BGC-BDRD but with CAM4 switch to CAM5.

I did execute ./xmlchange -file env_build.xml -id OCN_TRACER_MODULES -val 'iage ecosys ciso'

Here's the list of Source mods:

build-namelist ecosys_ciso_mod.F90 ecosys_mod.F90 ecosys_share.F90 namelist_defaults_pop2.xml ocn.ciso.tavg.csh passive_tracer_tools.F90

co2calc.F90 ecosys_driver.F90 ecosys_parms.F90 moby_mod.F90 namelist_definition_pop2.xml passive_tracers.F90 pop2.buildexe.csh

This run is on cheyenne. I copied the case and run directory here: /glade/work/pacosta/share/4MLevy

 

[mlevy]

So the good news is that I know what the problem is: in your ${CASEROOT}/SourceMods/src.pop2/ directory, you include the pop2.buildexe.csh file which tells the build to increment then number of tracers when using the ciso module... but when you actually run B.MMIOx2_C5_400_BGC.01.build, the model calls ${CASEROOT}/Buildconf/pop2.buildexe.csh which does not contain that extra line. So when you build CESM, the code compiles thinking there are 30 tracers when there are actually 37 (ciso is adding seven more).

The bad news is that it's been years since I've looked at the CESM 1.2 build system and I have no idea what part is responsible for creating ${CASEROOT}/Buildconf/pop2.buildexe.csh, so I don't know what you need to do to pull that file from SourceMods. I would suggest moving the modified file from your case's SourceMods directory to ${CESMROOT}/models/ocn/pop2/bld (I believe that's /glade/work/pacosta/cesm/cesm1_2_2_1/models/ocn/pop2/bld), overwriting the version of the file that already exists. Then there are three possibilities:

1. Rebuilding will copy the updated version into the proper directory (I think this will be the case, judging from the timestamp on ${CASEROOT}/Buildconf/pop2.buildexe.csh)
2. Cleaning up the current build and running ./cesm_setup -clean ; ./cesm_setup before rebuilding will be necessary
3. Setting up a completely new case will be necessary

 

[racosta6]

So the good news is that I know what the problem is: in your ${CASEROOT}/SourceMods/src.pop2/ directory, you include the pop2.buildexe.csh file which tells the build to increment then number of tracers when using the ciso module... but when you actually run B.MMIOx2_C5_400_BGC.01.build, the model calls ${CASEROOT}/Buildconf/pop2.buildexe.csh which does not contain that extra line. So when you build CESM, the code compiles thinking there are 30 tracers when there are actually 37 (ciso is adding seven more).

The bad news is that it's been years since I've looked at the CESM 1.2 build system and I have no idea what part is responsible for creating ${CASEROOT}/Buildconf/pop2.buildexe.csh, so I don't know what you need to do to pull that file from SourceMods. I would suggest moving the modified file from your case's SourceMods directory to ${CESMROOT}/models/ocn/pop2/bld (I believe that's /glade/work/pacosta/cesm/cesm1_2_2_1/models/ocn/pop2/bld), overwriting the version of the file that already exists. Then there are three possibilities:

1. Rebuilding will copy the updated version into the proper directory (I think this will be the case, judging from the timestamp on ${CASEROOT}/Buildconf/pop2.buildexe.csh)
2. Cleaning up the current build and running ./cesm_setup -clean ; ./cesm_setup before rebuilding will be necessary
3. Setting up a completely new case will be necessary

I ended up creating a new case with the modified pop2.buildexe.csh which got passed the error. Thanks for your help on this.

On another error, seems like i'm getting similar errors (Error in getting varid for netCDF field) as this previous discussion (POP aborting: Ocean tracer module problem in B1850 run). I have init_ecosys_init_file = 'same_as_TS' and init_ecosys_option = 'ccsm_hybrid'. Looks like my pop.r file does have the PO4_CUR variable. Should I switch to a startup instead of a hybrid to get my simulation going? i.e. use /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx1v6/ic/ts_PHC2_jan_ic_gx1v6_20090205.ieeer8 instead of my pop.r file?

Here's the error files
/glade/scratch/pacosta/B.MMIOx2_C5_400_BGC.02/run/cesm.log.230106-152008

/glade/scratch/pacosta/B.MMIOx2_C5_400_BGC.02/run/ocn.log.230106-152008

Thank you for your help!

 

[mlevy]

Do you have initial conditions for the ecosystem (including the carbon isotope tracers) on your grid?

The error in the thread that you linked to was a problem with continuing the run when trying to specify startup initial conditions - the system wanted to read ecosystem tracers from the restart file (because it was a continuation run), and in restart files tracers are stored on two time levels (as TRACER_CUR and TRACER_OLD) but the initialization file the user pointed to only contained TRACER. In your case, the error is occurring because the ecosystem tracers aren't in the initial condition file you are pointing to at all.

To fix this, you need a file similar /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_corrO2_Feb2013.nc but with proper initial conditions given your land mask, and then you can add

init_ecosys_init_file = 'your_file.nc'
ciso_init_ecosys_init_file = 'your_file.nc'

to the 1st submission section of your user_nl_pop2 file.

 

[racosta6]

Do you have initial conditions for the ecosystem (including the carbon isotope tracers) on your grid?

The error in the thread that you linked to was a problem with continuing the run when trying to specify startup initial conditions - the system wanted to read ecosystem tracers from the restart file (because it was a continuation run), and in restart files tracers are stored on two time levels (as TRACER_CUR and TRACER_OLD) but the initialization file the user pointed to only contained TRACER. In your case, the error is occurring because the ecosystem tracers aren't in the initial condition file you are pointing to at all.

To fix this, you need a file similar /glade/p/cesmdata/cseg/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_corrO2_Feb2013.nc but with proper initial conditions given your land mask, and then you can add

init_ecosys_init_file = 'your_file.nc'
ciso_init_ecosys_init_file = 'your_file.nc'

to the 1st submission section of your user_nl_pop2 file.

Hi Michael,

I've successfully recreated the ecosys_jan_IC_gx1v6_corrO2_Feb2013.nc file specified to my kmt and the simulation have forward a bit. However, it looks like files still lacks the TRACER_CUR and TRACER_OLD. Here's the most recent error messages:

ocn log
(passive_tracer_tools:rest_read_tracer_block) reading tracer block from /glade/work/pacosta/PaleoBC/acostar/Mod_POP_Eocosys/Init.MIO_Ecosys.pop.h.climo.nc
(passive_tracer_tools:read_field_3D) reading PO4_CUR from /glade/work/pacosta/PaleoBC/acostar/Mod_POP_Eocosys/Init.MIO_Ecosys.pop.h.climo.nc

cesm log
412: Error in getting varid for netCDF field

I'm I thinking I need to recreate my pop.r file to have PO4_CUR and other TRACER_CUR variables in them. Thank you for your help.

 

[mlevy]

Because you are using a hybrid case, the default option is to initialize from restart files (which use TRACER_OLD and TRACER_CUR). You can tell POP to initialize the ecosystem as though it were a startup run, though. I think adding the following to 1st submission section of user_nl_pop2 will do the trick:

init_ecosys_option = 'ccsm_startup'
ciso_init_ecosys_option = 'ccsm_startup'

(make sure it gets commented out when you switch to continuing your run)

 

[racosta6]

Because you are using a hybrid case, the default option is to initialize from restart files (which use TRACER_OLD and TRACER_CUR). You can tell POP to initialize the ecosystem as though it were a startup run, though. I think adding the following to 1st submission section of user_nl_pop2 will do the trick:

init_ecosys_option = 'ccsm_startup'
ciso_init_ecosys_option = 'ccsm_startup'

(make sure it gets commented out when you switch to continuing your run)

Hi Michael,

I seem to have moved past all the initial errors but have met the same fate as other BGC runs (ERROR: bounding bracket for pH solution not found)

(co2calc.F90:drtsafe_row) i_glob = 151 , j_glob = 191 , k =
ERROR: bounding bracket for pH solution not found

I've tried increasing dt_count to 72, increasing the number of nodes for the ocean model, and even setting the ciso tracers to False. I'm at a loss on what to try next.

Also, for the BDRD configuration do I need to initialize my atmosphere/land offline line first? My cam.i is similar to a standard non-BGC run.

Thank you in advance!

 

[Notos]

Hello! I also encountered the same question ERROR: bounding bracket for pH solution not found.
I have tried revised the dt_count, but it doesn't work. Have you solved this problem? Thanks so much!

 

[mlevy]

Hello! I also encountered the same question ERROR: bounding bracket for pH solution not found.
I have tried revised the dt_count, but it doesn't work. Have you solved this problem? Thanks so much!

Can you give more details about the run where you are encountering this error?

1. What version of CESM are you running?
2. What compset are you using?
3. What resolution are you using?
4. What did you change dt_count from / to?
5. Does the error message note the latitude / longitude of the grid cell(s) reporting the error?

 

[Notos]

Hello! The version is cesm1_2_1, and the compset is B_1850_BGC-BDRD. The resolution is f19_g16. I used the revised KMT and mask file of LGM. Compared to original mask file, there are grid cells that changes from ocean to land and no grid cells that changes from land to ocean, which means all of the ocean cells should have the initial values in ecosys_jan_IC_gx1v6_corrO2_Feb2013.nc. I have tried to change dt_count from 23 to 48, but it doesn't work. And the error exit in too many grids, so I think it does exit globally. Thanks!

 

[mlevy]

We recommend increasing dt_count when running future simulations, because the warmer climate reduces sea ice extent and exposes the ocean in smaller grid cells than the pre-industrial and historical simulations. For a B1850 compset, I would not expect ice melt to create this issue (unless you are running a 2x or 4x CO2 experiment).

How long does your model run before it crashes? If it's immediate, perhaps there's a problem with how you are initializing either the ocean or sea ice model?