Main menu

Navigation

Generating a mapping file for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc

12 posts / 0 new
Last post
gbisht@...
Generating a mapping file for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc

Hi,

I'm trying to create surface-dataset for my coustom CLM grid. As an intermediate step, I have to generate a mapping file from UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc (~125 million grid cells) to destination.nc (~352K grid cells). My job on Titan using 160 cores exceeded wallclock of 2 hours. I'm wondering if anyone can provide me an estimate of how long (wallclock and # cores) does it approximately take to generate a mapping file for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc.

Thanks,

-Gautam.

sacks

Hi Gautam,

Apologies for the slow reply here. The issue here is memory. If I remember right, the ESMF regridding tool has some pieces that don't scale in terms of memory use (so throwing more processors at it won't help). If you have access to NCAR's yellowstone machine, I recommend doing the regridding there. If you use the geyser cluster (which is the default in the regridbatch script), you shouldn't have memory problems - I believe that cluster has 1 TB memory per node, much more than is typical on most supercomputers. 

Unfortunately, because of these memory issues in the ESMF tool (which we do not control), we are unable to support this regridding operation on machines other than yellowstone/geyser/caldera at this time.

Bill

Bill Sacks

CESM Software Engineer

gbisht@...

Hi Bill,

Thanks for the tip regarding memory. I was able to successfully create the mapping file by using one thread per NUMA core on Titian.

-Gautam.

yangk2010@...

Hi Gautam,

I'm trying to run single-point cases with CLM4.5. When I used the tools in cesm to create my own surface dataset, a problem puzzled me for a long time,and I still can't solve it. I have to generate a mapping file from UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc, but  it is Unknown file format  on my cluster.

What can I do to let ESMF regridding tool on my cluster to deal with UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc, or other methods.

If you know how to  solve this problem,  please hlep me

Thanks in advance 

Kai Yang

gbisht@...

Hi Kai,

Was ESMF installed on your cluster with netcdf support? Are you able to generate mapping files using ESMF_RegridWeightGen for netcdf datasets other than UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc? 

-Gautam.

yangk2010@...

Hi Gautam,

ESMF is istalled on my cluster  with netcdf support, but netcdf istalled on my cluster  can't open UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc,it warns that "ncdump: UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc: NetCDF: Unknown file format". Except for "1km-merge-10min_HYDRO1K-merge-nomask", all other mapping files can be generated using   ESMF_RegridWeightGen .

 But, I still don't know how to solve it.

Kai Yang

gbisht@...

Hi Kai,

The use of an optional input argument '--64bit_offset' for ESMF_RegridWeightGen may possibly solve your issue. http://www.earthsystemmodeling.org/esmf_releases/last_built/ESMF_refdoc/... mentions that --64bit_offset is needed for a mapping file that is greater than 2GB and I recall that mapping file corresponding to HYDRO1K data are larger than 2GB.

-Gautam.

yangk2010@...

Hi Gautam,

Thank you for your help.

I have installed netcdf with the parameter" --enable-netcdf-4 --enable-netcdf4" again,ESMF is istalled on my cluster  with netcdf support. Now it can generate mapping files  using ESMF_RegridWeightGen for  UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc. But another problem appears:

  Starting weight generation with these inputs:
   Source File:
 /mnt/Disk16T/data/inputdata/lnd/clm2/mappingdata/grids/UGRID_1km-m
 erge-10min_HYDRO1K-merge-nomask_c130402.nc
   Destination File:
 /home/Single-point/cesm1_2_0/models/lnd/clm/tools/shared/mkmapgrids/SCR
 IPgrid_1x1_TTH_nomask_c150103.nc
   Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea

 ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata script

In PET0.RegridWeightGen.Log :

20150103 135558.359 ERROR            PET0 ESMCI_Mesh_F.C:1203 c_esmc_meshcreate() Internal error: Bad condition St9bad_alloc
20150103 135558.510 ERROR            PET0 ESMF_Mesh.F90:682 ESMF_MeshAddElements() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135558.524 ERROR            PET0 ESMF_Mesh.F90:3295 ESMF_MeshCreateFromUnstruct() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.582 ERROR            PET0 ESMF_Mesh.F90:1751 ESMF_MeshCreateFromFile() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.591 ERROR            PET0 ESMF_RegridWeightGen.F90:945 ESMF_RegridWeightGenFile Internal error: Bad condition - Internal subroutine call returned Error

So it may be  memory requiment problem. 

How can you sovle this problem before ?

If you can make it in the other way, I hope you can help me .

Thanks in advance !

Kai Yang

jlchen@...

Hi, yang, I meet the same promble as you described above,that is 

Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea

 ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata script

Do you have solved this problem? 

study

sacks

A few additions to Gautam's helpful comments:

(1) This UGRID file is in netcdf4 format. netcdf utilities built with a netcdf3 version of the library will not be able to recognize this file

(2) In order to regrid this file using ESMF, you can use a command like this one; note that there are a few differences from the standard remapping commands:

 

mpirun.lsf /glade/u/home/peggyl/bin/ESMF_RegridWeightGen -s $source_gridfile -d $dest_gridfile -w $mapname -m conserve --src_type UGRID --src_meshname landmesh -i --netcdf4

I believe you need ESMF.6.2.0 or newer for netcdf4 functionality.

(3) Even assuming you have recent-enough versions of netcdf and esmf, you may very likely run into trouble regridding this file, because it requires huge amounts of memory. For that reason, we are not able to provide official support for doing this. We are still (slowly) working on an alternative solution for users who want to do high-resolution regional runs: namely, creating a 1/8 deg surface dataset that can be subset to your region of interest.

Bill Sacks

CESM Software Engineer

yangk2010@...

Hi Bill,

Thanks for you reply.

I have run into trouble regridding this file:

Starting weight generation with these inputs:
   Source File:

 /mnt/Disk16T/data/inputdata/lnd/clm2/mappingdata/grids/UGRID_1km-m
 erge-10min_HYDRO1K-merge-nomask_c130402.nc
   Destination File:
 /home/Single-point/cesm1_2_0/models/lnd/clm/tools/shared/mkmapgrids/SCR
 IPgrid_1x1_TTH_nomask_c150103.nc
   Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea

 ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata script

In PET0.RegridWeightGen.Log :

20150103 135558.359 ERROR            PET0 ESMCI_Mesh_F.C:1203 c_esmc_meshcreate() Internal error: Bad condition St9bad_alloc
20150103 135558.510 ERROR            PET0 ESMF_Mesh.F90:682 ESMF_MeshAddElements() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135558.524 ERROR            PET0 ESMF_Mesh.F90:3295 ESMF_MeshCreateFromUnstruct() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.582 ERROR            PET0 ESMF_Mesh.F90:1751 ESMF_MeshCreateFromFile() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.591 ERROR            PET0 ESMF_RegridWeightGen.F90:945 ESMF_RegridWeightGenFile Internal error: Bad condition - Internal subroutine call returned Error

 How can I solve this problem, if I want to run CLM4.5 on my cluster ?

  And another problem alse puzzled me for a long time :

   I have created my own atmosphere focing, but it can't be read by the model. The atmosphere focing on my cluster is like this :

    2010-01.nc   2010-03.nc   2010-05.nc   2010-07.nc   2010-09.nc   2010-11.nc
    2010-02.nc   2010-04.nc   2010-06.nc   2010-08.nc   2010-10.nc   2010-12.nc

netcdf \2010-01 {
dimensions:
        scalar = 1 ;
        lon = 1 ;
        lat = 1 ;
        time = UNLIMITED ; // (1488 currently)
variables:
        double time(time) ;
                time:long_name = "observation time" ;
                time:units = "days since 2010-01-01 00:00:00" ;
        double EDGEW(scalar) ;
                EDGEW:long_name = "western edge in atmospheric data" ;
                EDGEW:units = "degrees E" ;
        double EDGEE(scalar) ;
                EDGEE:long_name = "eastern edge in atmospheric data" ;
                EDGEE:units = "degrees E" ;
        double EDGES(scalar) ;
                EDGES:long_name = "southern edge in atmospheric data" ;
                EDGES:units = "degrees N" ;
        double EDGEN(scalar) ;
                EDGEN:long_name = "northern edge in atmospheric data" ;
                EDGEN:units = "degrees N" ;
        double LONGXY(lat, lon) ;
                LONGXY:long_name = "longtitude" ;
                LONGXY:units = "degrees E" ;
        double LATIXY(lat, lon) ;
                LATIXY:long_name = "latitude" ;
                LATIXY:units = "degrees N" ;
        double FSDS(time, lat, lon) ;
                FSDS:long_name = "incident solar(FSDS)" ;
                FSDS:units = "W/m2" ;
        double FLDS(time, lat, lon) ;
                FLDS:long_name = "incident longwave(FLDS)" ;
                FLDS:units = "W/m2" ;
        double PRECTmms(time, lat, lon) ;
                PRECTmms:long_name = "precipitation (PRECTmms)" ;
                PRECTmms:units = "mm/s" ;
        double TBOT(time, lat, lon) ;
                TBOT:long_name = "temperature at the lowest atm level(TBOT)" ;
                TBOT:units = "K" ;
        double WIND(time, lat, lon) ;
                WIND:long_name = "wind at the lowest atm level(WIND)" ;
                WIND:units = "m/s" ;
        double PSRF(time, lat, lon) ;
                PSRF:long_name = "pressure at the lowest atm level(PSRF)" ;
                PSRF:units = "Pa" ;
        double RH(time, lat, lon) ;
                RH:long_name = "relative humidity at the lowest atm level(RH)" ;
                RH:units = "%" ;
        double ZBOT(time, lat, lon) ;
                ZBOT:long_name = "observational height" ;
                ZBOT:units = "m" ;
}

    If you know how to deal with this problem, I hope you can give me a help.

    Thank you in advance.

Kai Yang

sacks

I'm sorry, I don't know what is causing the ESMF regridding problem. If you need further support with the ESMF utilities, you could try contacting the ESMF support group: http://www.earthsystemmodeling.org/users/index.shtml

Keep in mind that you can still run CLM at any supported resolution without needing to use the ESMF tools. These tools are only needed when creating your own, new resolution, which we cannot provide very much support for.

As for your other problem: There is not enough information here to know what is going on. I would suggest posting this as a new topic with some additional information.

Bill Sacks

CESM Software Engineer

Log in or register to post comments

Who's new

  • m.kliphuis@...
  • ddc3061993@...
  • hui.ding@...
  • zhouc@...
  • arianna.valmass...