Attempting to reproduce an old run (cesm2.1.1-exp13) run using a newer version (cesm2.1.5), fails with ERROR: too much NO3 uptake predicted by FUN

[bmartinez]

I am using cesm2.1.5, but I attempting to recreate a case that was originally performed with cesm2.1.1-exp13

My case is located here: /glade/u/home/bmartinez/b.e21.BWHIST.SD.f09_g17.010, and the case I am attempting to duplicate is located here: /glade/work/ssfcst/sg_cases/b.e21.BWHIST.SD.f09_g17.002.nudgedOcn.tidalOutput

I am having difficulty reproducing the run, and I think this might arise from the fact that the source code has changed significantly between the two versions.

The specific issue seems to be related to the land component. I am able to perform a hybrid start using the restart files from the original run, and it runs successfully for several months, but then it fails on a specific day with the following error:

dec1708.hsn.de.hpc.ucar.edu 502: problem with limitations on no3 uptake 0.000000000000000E+000
dec1708.hsn.de.hpc.ucar.edu 502: -1.320284514970937E-011
dec1708.hsn.de.hpc.ucar.edu 502: ENDRUN:
dec1708.hsn.de.hpc.ucar.edu 502: ERROR: too much NO3 uptake predicted by FUN

I found only two threads on the CESM forum about this issue, one of which is here: error message of "too much NH4 uptake predicted by FUN" and here ERROR: too much NH4 uptake predicted by FUN. The first one says to set FUN and a few other things to false in the clm file, which I tried, but it still fails with the error:

dec0475.hsn.de.hpc.ucar.edu 227: ERROR: Carbon or Nitrogen patch negative = 0.000000000000000E+000
dec0475.hsn.de.hpc.ucar.edu 227: -6.07193035621369
dec0475.hsn.de.hpc.ucar.edu 227: ERROR: limits = -60.0000000000000 -6.00000000000000
dec0475.hsn.de.hpc.ucar.edu 227: ENDRUN:
dec0475.hsn.de.hpc.ucar.edu 227: ERROR:
dec0475.hsn.de.hpc.ucar.edu 227: ERROR: carbon or nitrogen state critically negative ERROR in CNPrecisionControl

which seems like the same issue of negative uptake, but coming from somewhere else since FUN is disabled.

I additionally attempted to increase the threshold on sminn_to_plant_fun_nh4_vr(c,j) - smin_nh4_to_plant_vr(c,j) in SoilBiogeochemCompetitionMod.F90, which caused the model to run successfully for a few more days, but then it would fail with the same (or a similar error). [The output files from the runs in which these changes were made are not attached, but let me know if you would like to see those as well]

I was unable to clone the case directly because I do not have write permissions for the original case directory, so I duplicated the case following the README.case file in the original directory (./create_newcase --res f09_g17 --compset BWHIST --case /glade/u/home/bmartinez/b.e21.BWHIST.SD.f09_g17.010 --project P28100036 --run-unsupported), then copying over user_nl files and modifying the relevant paths so that they match the new file system on derecho, and making some modifications to the env_run and env_build:

In env_build.xml:

add '-offline_dyn -nlev 70' to CAM_CONFIG_OPTS
set OCN_TRACER_MODULES to "" (i.e., delete iage cfc sf6 ecosys)

Since some of the files are too large to attach to the forum itself, I uploaded them to google drive here: error files - Google Drive How can I resolve this negative uptake issue?

 

[oleson]

Not sure I can help much here, but does it work if you start at the RUN_STARTDATE of the original case, which seems to be "2020-11-02"?
Since the error is NO3, not NH4, you could try increasing the threshold for NO3:

if ((sminn_to_plant_fun_no3_vr(c,j)-smin_no3_to_plant_vr(c,j)).gt.0.0000000000001_r8)

For the negative carbon/nitrogen error (which appears to be nitrogen-related), try changing nnegcrit to -7.d+0 or maybe even -8.d+0, instead of the default value of -6.d+0, in your user_nl_clm.

Otherwise, maybe you can try running with the default initial file that comes with that compset instead of the reference case.