Trying to duplicate CMIP6 run

[bcash@gmu_edu]

I am trying to duplicate the output from this CMIP6 experiment: /glade/work/cmip6/cases/DECK_2deg/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001
Using the restart files here: /glade/campaign/collections/cmip/CMIP6/restarts/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001

I selected the 0031-01-01 restarts and made a two-year run, but when I compare my month 1 to the 31-01-01 data from (for example): /glade/campaign/collections/cmip/CMIP6/timeseries-cmip6/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001/atm/proc/tseries/month_1/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001.cam.h0.LHFLX.000101-005012.nc the fields are not identical.

I assume this is because I've made a mistake somehow in setting up the experiment and am hoping someone can straighten me out.
My experiment setup is:

./create_newcase --compset B1850 --res f19_g17 --case b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001
./xmlchange RUN_REFCASE=b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001
./xmlchange RUN_REFDIR=/glade/work/bcash/cesm2_init/0031-01-01-00000
./xmlchange RUN_REFDATE=0031-01-01
./xmlchange RUN_STARTDATE=0031-01-01
./xmlchange RUN_TYPE=hybrid

 

[mlevy]

I believe you want to use

./xmlchange RUN_TYPE=branch

rather than hybrid. A branch run requires all components to initialize from restart files - a hybrid run allows some components to start from initial condition files instead, so even all components are initialized from restarts, it will still start the ocean component one coupling interval into the run rather than at the same time as all the other components and I suspect that is causing the differences you are seeing.

Also, I recommend using a different case name for your case than appears in RUN_REFCASE. Perhaps something like b.e21.B1850.f19_g17.CMIP6-piControl-2deg.bcash_run.001?

 

[bcash@gmu_edu]

Thanks! I initially had it set to branch, but ran into an error "Can only set use_init_interp if finidat is set", which seems to be the same problem encountered here? finidat_interp_dest***.nc

Is there another setting I need to change to avoid that issue?

 

[mlevy]

I'm not sure of the answer to that - I'll move this to the CTSM forum because someone there should be able to tell you how to set up the interpolation correctly.

 

[oleson]

I think you would need to remove this setting for CLM_NAMELIST_OPTS in env_run.xml: use_init_interp=.true.

 

[bcash@gmu_edu]

And that should be changed by deleting that entry from env_run.xml? Or just set it to .false. ?

 

[bcash@gmu_edu]

Update: Removing the entry caused a failure when I tried to submit, and setting it to .false. caused a bunch of errors like:
41: check_dim ERROR: mismatch of input dimension 14762 with expected value
41: 14749 for variable landunit

 

[oleson]

That error means that there are a different number of landunits that the model is trying to run with (14762) than the number of landunits that are on the restart file (14749). It looks like you are using the right surface dataset, however, there is a setting in user_nl_clm in the original case directory:

/glade/work/cmip6/cases/DECK_2deg/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001

The setting is:

glacier_region_behavior = 'single_at_atm_topo', 'virtual', 'virtual', 'virtual'

The default for the last entry is 'multiple', which is probably what you are running with.
So add the setting above to your user_nl_clm and try that.

 

[bcash@gmu_edu]

That stopped it from crashing, thanks! I am still not able to duplicate the DECK 2-degree run, possibly because there are other user_nl_* changes that I didn't copy?

This is turning into a more challenging question than I had expected. How do I duplicate an existing run exactly from a set of restart files?

 

[oleson]

Generally, I use create_clone from either my checkout of the code base that was used to run the simulation or from the location of the code base itself.
For example, I cloned your case from your sandbox using:

./create_clone --case /glade/work/oleson/cesm2_1_1_runs/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001 --clone /glade/work/bcash/ssf/my_cesm_sandbox/cime/scripts/b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001 --cime-output-root /glade/scratch/oleson --project PXXXXXXXX

That way, you get all of the user_nl_* changes and SourceMods from the original case.
Then you could try to replicate the start of the case, or switch to a branch and try to replicate later parts of the simulation.
The code base that the original simulation was run from seems to be /gpfs/u/home/cmip6/cesm_tags/release-cesm2.1.1

 

[bcash@gmu_edu]

So clone, switch to branch, which requires changing use_init_interp to .false.?

 

[oleson]

That sounds right. use_init_interp = .true. was required at the beginning of the original run because the initial file (finidat) had a different subgrid structure than the surface dataset that is used. After that point, restarts contained the correct subgrid structure.
You might also consider verifying that you can replicate the beginning of the run for a short period before you start making changes (e.g., switching to a branch).

 

[bcash@gmu_edu]

Cloning the run, building, and submitting without any changes does not recover the saved output. Could there be a change in the Cheyenne environment that is affecting the answer? I'm really at a loss for where this is going wrong.

 

[oleson]

I tried cloning the case and then restarting at 0031-01 using a branch.
I was able to reproduce the TSA variable in the land history file for 0031-01 for this simulation. I haven't checked other component history files.
I compared my cloned case file (baseline.b.e21.B1850.f19_g17.CMIP6-piControl-2deg.clm2.h0.0031-01.nc) to one I extracted from campaign store.

b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001.clm2.h0.TSA.000101-005012.nc

ncks -d time,360,360 b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001.clm2.h0.TSA.000101-005012.nc b.e21.B1850.f19_g17.CMIP6-piControl-2deg.001.clm2.h0.TSA.0031-01.nc

They were identical when I checked using cprnc.

My case is here if you want to compare:

/glade/work/oleson/cesm2_1_1_runs/baseline.b.e21.B1850.f19_g17.CMIP6-piControl-2deg

 

[bcash@gmu_edu]

Thank you very much for running this case! Seeing your result and digging into my run some more I was able to determine my run actually was correct - the problem was actually in my diagnostic. Thanks again!