noise from copying state%pmid / fillw: Q not allocated

[andreasb]

Hi,I have a rather strange problem: Since version 1_1_1, the following statement in (my own module in a _timestep_init subroutine) causes trouble on yellowstone:mypressure(1:ncol,:)= state%pmid(1:ncol,:)If compiled without debugging, mypressure is "almost" correct: there is noise added, as you can see in the attachment in a variable derived from it.
If compiled with debugging, subroutine filew in fv/fill_module crashes because the variable q is not allocated."mypressure" is a regular field created with addfld in the modules _init routine.If the line above is substituted with mypressure(1:ncol,:)=10000._r8 everyting works fine (no crash if compiled with debugging also).Here are further details:No problem in version 1_1:./create_newcase -res 1.9x2.5_gx1v6 -case test1 -compset F_1955-2005_WACCM_CN -mach yellowstone -pecount M -mach_dir /glade/p/cesm/cseg//collections/cesm1_1/scripts/ccsm_utils/Machines_121207Problem starts in 1_1_1 and is still happening in 1_2_0:./create_newcase -res 1.9x2.5_gx1v6 -case test2 -confopts _D_PT -compset F_1955-2005_WACCM_CN -mach yellowstone

Other variables from "state" work fine, in fact I calculate number density with this line:    numdensity(1:ncol,1:pver)= &
          state%pmid(1:ncol,1:pver)/(boltz*state%t(1:ncol,1:pver))*1e-6_r8

where only pmid causes the problem - if substituded with a number this line also works.I can provide more details or create a simple demonstration module if need be. Thanks!Andreas

 

[jedwards]

Perhaps this is a compiler bug  1.1 uses intel/12.1.5 while 1.1.1 and 1.2.0 use intel/13.0.1   you can try the older compiler with the newer cesm by changeing the module load command in your case env_mach_specific You can also try the latest intel/13.1.2 compiler.  

 

[andreasb]

Finally I found the source for this problem (see the noise in the figure attached to original post): if only one thread is used, this problem does not occur. The compiler version did not matter. The problem is reproducable with a very simple module that just copies state%pmid onto another variable:    pressure_with_noise(:ncol,:) =  state%pmid(:ncol,:)

I tested this on yellowstone using 4 nodes. The problem occurs only with NTHRDS_ATM=2 (as is the standard setting in 1.2.1).