A question of gathering field in modifying the code in atmocphere dynamic in CESM

[jie_wong]

I am researching the roles of WWBs in ENSO, using CESM 1.2. But came to some questions when I am trying to modify the model. I am trying to change the wind or pressure field in the interior atmosphere dynamics. When I tried to modify it, the mpi or chunk problem went out, it seems before I want to change the field I should gather all data from each mpi process or xy-decomp grid in dynamic. It confused me how to do this. So I am asking for help on how to gather/scatter data in CAM dynamic.
I found that there are two function 'scatter chunk from field' and 'gather field from chunk'. But it seems these function are defined or used in physics module, in dynamics it was different.

 

[peverley]

Hello,

Could you clarify what you mean by "the mpi or chunk problem"? Is there a specific error you are getting and if so, can you post it here?

Thanks!
Courtney

 

[jie_wong]

Hello Courtney,
What I want to do is modift the variables inside the dynamic, and I think the variables "dyn_in" and "dyn_out" are those. For example, "dyn_out%t3" represents the temperature in the atmosphere, So I want to get that data out and modify it. But when I try to get it out, because the data was distributed to different mpi/processes, so I need to gather it to the masterproc first. But I don't kown how to do it.

Thanks
Jie

 

[peverley]

Thanks for the info, Jie!

I think what you need is to use mpi_gatherv() or mpi_gather() to get the all of the scattered data onto the master process. An example of this being used in the code is in dynamics/se/dyn_grid.F90.

You can find more information on input variables here: MPI_Gatherv

Once you have everything on the master process, you can use "if (masterproc) then" and then do whatever you want to do with the data.

Let me know if you need more help!

Courtney

 

[jie_wong]

Thanks for you help, Courtney!
I know I should use MPI_Gatherv/scatterv to do this modification. But how to use them confused me, and dynamics/se/dyn_grid.F90 you mentioned it seems only MPI_ALLGATHER was used. After look over almost all the codes in CAM, I haven't found any example how to use the MPI_gather or mpi_gatherv. Anyway, the problem puzzles me is how to use these, including input variables and mpi variables.


Thanks
Jie

 

[peverley]

Hi again,

I forgot you were using CESM 1.2. If you look at a newer version of cam (here: GitHub - ESCOMP/CAM at cam_development) you'll find more examples.

Courtney

 

[jie_wong]

Thanks a lot! I have konw the the process flow: first,gather data ; second do the modification; last scatter them. But it also confused me how to use mpi_gather/scatter function. Can you take a example for me? Suppose that I want to get the u wind in the atmosphere (that is put in dyn_in%u3s I think),
so in the function

call mpi_scatterv (sendbuf, sendcnts, displs, sendtype, recvbuf, recvcnt, &
recvtype, root, comm, ier)

Sendbuf is the local variables dyn_in%u3s, shape is (ifirstxy:ilastxy,jfirstxy:jlastxy,1:km), where ifirst... are the mpi locations
while the recvbuf is the global variables, shape is (im,jm,km), where im,jm,km are the dimensions
The sendtype and recvtype are mipr8,comm is mpicom,root is masterprocid.

But I don't kown how to calculate the sendcnts,,displs, and recvcnt.

 

[peverley]

Hi Jie,

Could you elaborate on what modifications you're doing to the data? If you're only gathering the data to modify it and scatter it back, I think you should be able to skip the gathering entirely.

If you definitely need to do the gather/modify/scatter method, I can ask around for examples.

Courtney

 

[jie_wong]

Hello Courtney,

below is how to define the input parameters in the function

call mpi_gatherv (sendbuf, sendcnts, displs, sendtype, recvbuf, recvcnt, &
recvtype, root, comm, ier)

In my code I defined them like this (In my code I take temerature as example, t3xy pointed to dyn_out%t3)

first I put the values to a local field lfield_p

do k= 1,km
do j = jfirstxy,jlastxy
do i = ifirstxy,ilastxy
lfield_p(i,j,k) = t3xy(i,j,k)
enddo
enddo
enddo

and the calculate the gatherv parameters

numsend = size(t3xy)
recvcnt(1)=get_block_gcol_cnt_d(1)*km !km is the vertical dimension
displs(1)=0
do p=2,npes
displs(p)=displs(p-1)+recvcnt(p-1)
recvcnt(p)=get_block_gcol_cnt_d(p)*km
end do

And last, use gatherv

call mpi_gatherv (lfield_p, numsend, mpir8, gfield_p, numrecv, &
displs, mpir8, masterprocid, mpicom,ier)

In order to take over if the gather is right, I wrote it out and compared with the model output files

if (masterproc) then
!! write
open(513,file='modify.txt')
write(513,*)(((gfield_p(i,j,k),i=1,im),j=1,jm),k=1,km)
close(513)
endif

But the result is random, it shows the gather is invalid or wrong

Can you give me some advice on this?

My goal is just need to modify and gather them, but in my opinion I should first gather data and in masterproc I can modify, at last scatter the modified-data.

Thanks for your help again