Initial file longitudes are 0s in large ensemble CAM6

[raeder]

It appears that for 60 instances and more the longitudes
being written to the CAM initial files are all 0s.
Both lon and slon.  This doesn't happen for 40 instances.
Compare /gpfs/fs1/scratch/raeder/
   CAM6_60mem_debug/run/CAM6_60mem_debug.cam_0001.i.2010-07-01-21600.nc
   CAM6_40mem/run/CAM6_40mem.cam_0001.i.2010-07-01-21600.nc
These were built with the same setup script, just varying the
number of instances.

I'm using cesm2_2_alpha01a and cam6_0_002.  I've looked at the
ChangeLog for cam6_0_014, but don't recognize anything related.
Is it worth the trouble to try to replace 002 with 014?

I've printed some extra output; see
   CAM6_60mem_debug/run/atm_0059.log.2569888.chadmin1.180911-031904:
      'hdefine' and 'write_horiz_coord_var',
but those don't show the problem, except that it's more likely
that the problem is in the distributed lon.

I'll be hard-pressed to trace the source of this farther back
in the code, due to the abstraction, layering of structures, etc.,
so I'm hoping that someone with better insight into the handling
of grid variables can guide me.This is holding up an NSC proposal due 2018-9-24,so I don't have much time to fix it. Thanks,Kevin Raeder

 

[eaton]

We've seen a problem with the output of lon values in 1/4 degree adiabatic runs.  Symptoms are different; bad lon values in the h0 file rather than .i., and the values are fill values rather than zeros.  I'm looking at this problem now and crossing fingers that the two problems have the same solution. 

 

[raeder]

Thanks for the quick reply!I can probably work around this for the moment, so I won't work on debugging it, unless I can be useful.I'm available to run tests of whatever fixes you want to try.

 

[raeder]

Brian Eaton discovered that this was caused by a terrible definition of npr_yz (1,1,1,1)in cam/cime_config/buildnml, which defined 1 subdomain, which ran the wholedycore on 1 task.  Changing to the intended defaultbuild-namelist --ntasks $NTASKS_PER_INST_ATMfixed this and the problem with CAM-Chem (and WACCM) hanging when largeensemble is run:http://bb.cgd.ucar.edu/user/login?destination=/cesm20cam-chem-hanging-mpialltoallintSo this issue is resolved.