Generating a mapping file for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc

[gbisht@lbl_gov]

Hi,I'm trying to create surface-dataset for my coustom CLM grid. As an intermediate step, I have to generate a mapping file from UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc (~125 million grid cells) to destination.nc (~352K grid cells). My job on Titan using 160 cores exceeded wallclock of 2 hours. I'm wondering if anyone can provide me an estimate of how long (wallclock and # cores) does it approximately take to generate a mapping file for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc.Thanks,-Gautam.

 

[sacks]

Hi Gautam,Apologies for the slow reply here. The issue here is memory. If I remember right, the ESMF regridding tool has some pieces that don't scale in terms of memory use (so throwing more processors at it won't help). If you have access to NCAR's yellowstone machine, I recommend doing the regridding there. If you use the geyser cluster (which is the default in the regridbatch script), you shouldn't have memory problems - I believe that cluster has 1 TB memory per node, much more than is typical on most supercomputers. Unfortunately, because of these memory issues in the ESMF tool (which we do not control), we are unable to support this regridding operation on machines other than yellowstone/geyser/caldera at this time.Bill

 

[gbisht@lbl_gov]

Hi Bill,Thanks for the tip regarding memory. I was able to successfully create the mapping file by using one thread per NUMA core on Titian.-Gautam.

 

[yangk2010@lzu_edu_cn]

Hi Gautam,I'm trying to run single-point cases with CLM4.5. When I used the tools in cesm to create my own surface dataset, a problem puzzled me for a long time,and I still can't solve it. I have to generate a mapping file from UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc, but  it is Unknown file format  on my cluster.What can I do to let ESMF regridding tool on my cluster to deal with UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc, or other methods.If you know how to  solve this problem,  please hlep meThanks in advance 

 

[gbisht@lbl_gov]

Hi Kai,Was ESMF installed on your cluster with netcdf support? Are you able to generate mapping files using ESMF_RegridWeightGen for netcdf datasets other than UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc? -Gautam.

 

[yangk2010@lzu_edu_cn]

Hi Gautam,ESMF is istalled on my cluster  with netcdf support, but netcdf istalled on my cluster  can't open UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc,it warns that "ncdump: UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc: NetCDF: Unknown file format". Except for "1km-merge-10min_HYDRO1K-merge-nomask", all other mapping files can be generated using   ESMF_RegridWeightGen . But, I still don't know how to solve it.

 

[gbisht@lbl_gov]

Hi Kai,The use of an optional input argument '--64bit_offset' for ESMF_RegridWeightGen may possibly solve your issue. http://www.earthsystemmodeling.org/esmf_releases/last_built/ESMF_refdoc/node3.html mentions that --64bit_offset is needed for a mapping file that is greater than 2GB and I recall that mapping file corresponding to HYDRO1K data are larger than 2GB.-Gautam.

 

[sacks]

A few additions to Gautam's helpful comments:(1) This UGRID file is in netcdf4 format. netcdf utilities built with a netcdf3 version of the library will not be able to recognize this file(2) In order to regrid this file using ESMF, you can use a command like this one; note that there are a few differences from the standard remapping commands: mpirun.lsf /glade/u/home/peggyl/bin/ESMF_RegridWeightGen -s $source_gridfile -d $dest_gridfile -w $mapname -m conserve --src_type UGRID --src_meshname landmesh -i --netcdf4I believe you need ESMF.6.2.0 or newer for netcdf4 functionality.(3) Even assuming you have recent-enough versions of netcdf and esmf, you may very likely run into trouble regridding this file, because it requires huge amounts of memory. For that reason, we are not able to provide official support for doing this. We are still (slowly) working on an alternative solution for users who want to do high-resolution regional runs: namely, creating a 1/8 deg surface dataset that can be subset to your region of interest.

 

[yangk2010@lzu_edu_cn]

Hi Gautam,Thank you for your help.I have installed netcdf with the parameter" --enable-netcdf-4 --enable-netcdf4" again,ESMF is istalled on my cluster  with netcdf support. Now it can generate mapping files  using ESMF_RegridWeightGen for  UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc. But another problem appears:  Starting weight generation with these inputs:
   Source File:
 /mnt/Disk16T/data/inputdata/lnd/clm2/mappingdata/grids/UGRID_1km-m
 erge-10min_HYDRO1K-merge-nomask_c130402.nc
   Destination File:
 /home/Single-point/cesm1_2_0/models/lnd/clm/tools/shared/mkmapgrids/SCR
 IPgrid_1x1_TTH_nomask_c150103.nc
   Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata scriptIn PET0.RegridWeightGen.Log :20150103 135558.359 ERROR            PET0 ESMCI_Mesh_F.C:1203 c_esmc_meshcreate() Internal error: Bad condition St9bad_alloc
20150103 135558.510 ERROR            PET0 ESMF_Mesh.F90:682 ESMF_MeshAddElements() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135558.524 ERROR            PET0 ESMF_Mesh.F90:3295 ESMF_MeshCreateFromUnstruct() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.582 ERROR            PET0 ESMF_Mesh.F90:1751 ESMF_MeshCreateFromFile() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.591 ERROR            PET0 ESMF_RegridWeightGen.F90:945 ESMF_RegridWeightGenFile Internal error: Bad condition - Internal subroutine call returned ErrorSo it may be  memory requiment problem. How can you sovle this problem before ?If you can make it in the other way, I hope you can help me .Thanks in advance !

 

[yangk2010@lzu_edu_cn]

Hi Bill,Thanks for you reply.I have run into trouble regridding this file:Starting weight generation with these inputs:
   Source File:
 /mnt/Disk16T/data/inputdata/lnd/clm2/mappingdata/grids/UGRID_1km-m
 erge-10min_HYDRO1K-merge-nomask_c130402.nc
   Destination File:
 /home/Single-point/cesm1_2_0/models/lnd/clm/tools/shared/mkmapgrids/SCR
 IPgrid_1x1_TTH_nomask_c150103.nc
   Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata scriptIn PET0.RegridWeightGen.Log :20150103 135558.359 ERROR            PET0 ESMCI_Mesh_F.C:1203 c_esmc_meshcreate() Internal error: Bad condition St9bad_alloc
20150103 135558.510 ERROR            PET0 ESMF_Mesh.F90:682 ESMF_MeshAddElements() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135558.524 ERROR            PET0 ESMF_Mesh.F90:3295 ESMF_MeshCreateFromUnstruct() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.582 ERROR            PET0 ESMF_Mesh.F90:1751 ESMF_MeshCreateFromFile() Internal error: Bad condition - Internal subroutine call returned Error
20150103 135559.591 ERROR            PET0 ESMF_RegridWeightGen.F90:945 ESMF_RegridWeightGenFile Internal error: Bad condition - Internal subroutine call returned Error How can I solve this problem, if I want to run CLM4.5 on my cluster ?  And another problem alse puzzled me for a long time :   I have created my own atmosphere focing, but it can't be read by the model. The atmosphere focing on my cluster is like this :    2010-01.nc   2010-03.nc   2010-05.nc   2010-07.nc   2010-09.nc   2010-11.nc
    2010-02.nc   2010-04.nc   2010-06.nc   2010-08.nc   2010-10.nc   2010-12.ncnetcdf 2010-01 {
dimensions:
        scalar = 1 ;
        lon = 1 ;
        lat = 1 ;
        time = UNLIMITED ; // (1488 currently)
variables:
        double time(time) ;
                time:long_name = "observation time" ;
                time:units = "days since 2010-01-01 00:00:00" ;
        double EDGEW(scalar) ;
                EDGEW:long_name = "western edge in atmospheric data" ;
                EDGEW:units = "degrees E" ;
        double EDGEE(scalar) ;
                EDGEE:long_name = "eastern edge in atmospheric data" ;
                EDGEE:units = "degrees E" ;
        double EDGES(scalar) ;
                EDGES:long_name = "southern edge in atmospheric data" ;
                EDGES:units = "degrees N" ;
        double EDGEN(scalar) ;
                EDGEN:long_name = "northern edge in atmospheric data" ;
                EDGEN:units = "degrees N" ;
        double LONGXY(lat, lon) ;
                LONGXY:long_name = "longtitude" ;
                LONGXY:units = "degrees E" ;
        double LATIXY(lat, lon) ;
                LATIXY:long_name = "latitude" ;
                LATIXY:units = "degrees N" ;
        double FSDS(time, lat, lon) ;
                FSDS:long_name = "incident solar(FSDS)" ;
                FSDS:units = "W/m2" ;
        double FLDS(time, lat, lon) ;
                FLDS:long_name = "incident longwave(FLDS)" ;
                FLDS:units = "W/m2" ;
        double PRECTmms(time, lat, lon) ;
                PRECTmms:long_name = "precipitation (PRECTmms)" ;
                PRECTmms:units = "mm/s" ;
        double TBOT(time, lat, lon) ;
                TBOT:long_name = "temperature at the lowest atm level(TBOT)" ;
                TBOT:units = "K" ;
        double WIND(time, lat, lon) ;
                WIND:long_name = "wind at the lowest atm level(WIND)" ;
                WIND:units = "m/s" ;
        double PSRF(time, lat, lon) ;
                PSRF:long_name = "pressure at the lowest atm level(PSRF)" ;
                PSRF:units = "Pa" ;
        double RH(time, lat, lon) ;
                RH:long_name = "relative humidity at the lowest atm level(RH)" ;
                RH:units = "%" ;
        double ZBOT(time, lat, lon) ;
                ZBOT:long_name = "observational height" ;
                ZBOT:units = "m" ;
}    If you know how to deal with this problem, I hope you can give me a help.    Thank you in advance.

 

[sacks]

I'm sorry, I don't know what is causing the ESMF regridding problem. If you need further support with the ESMF utilities, you could try contacting the ESMF support group: http://www.earthsystemmodeling.org/users/index.shtmlKeep in mind that you can still run CLM at any supported resolution without needing to use the ESMF tools. These tools are only needed when creating your own, new resolution, which we cannot provide very much support for.As for your other problem: There is not enough information here to know what is going on. I would suggest posting this as a new topic with some additional information.

 

[jlchen@lzb_ac_cn]

Hi, yang, I meet the same promble as you described above,that is Weight File:
 map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_TTH_nomask_aave_da_c150103.nc
   Source File is in UGRID format, dummy variable: landmesh
   Source Grid is a global grid
   Source Grid is an unstructured grid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is a logically rectangular grid
   Regrid Method: conserve
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Norm Type: dstarea ERROR: Problem on processor            0
 . Please see the PET*.RegridWeightGen.Log files for a traceback.
Error status returned from mkmapdata scriptDo you have solved this problem? 

 

[adrienD]

Hi everyone, I am also trying to generate a mapping file of ugrid_1km-merge-10min_hydro1k-merge-nomask_c130402.nc. I solved the memory problem but I encountered another issue when I run the script. Here is the error message:

ESMF_RegridWeightGen: /tmp/xas/build/netcdf_netcdf_c_4.4.0_default_gcc48/src/netcdf-4.4.0/libsrc4/nc4file.c:2836: NC4_open: Assertion `0' failed.


Any idea?

 

[adrienD]

Problem solved, I am sorry I bring this subject up. My apologies.

 

[koichi_sakaguchi@pnnl_gov]

Hi everyone,

I am having a difficult time creating remapping files for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc. What feature of CLM4.5/5 requires this mapping file? Are there any ways we can turn off the particular feature of CLM4.5/5 requiring this data through configure or namelist options, so that we don't need to create a mapping file for this data?

We are going to use a series of global high-resolution or variable-resolution simulations (e.g., globally 30km, 15km, and/or higher resolutions over certain regions) using CAM6-CLM5 configuration. Generating mapping files for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc makes our workflow quite inefficient...

 

[koichi_sakaguchi@pnnl_gov]

I've read the section about the surface data in CLM4.5 technical note, and a few different model components are dependent on the HYDRO1k data (maximum fractional saturated area for surface runoff, elevation for snow cover fraction, and surface slope for river transport model). However, two of these model components existed in CLM4 as well (snow cover parameterization seems to be renewed in CLM4.5); So then my question is, can we use older topography data (as in CLM4) to generate input variables?

 

[adrienD]

Hi everyone,

I am having a difficult time creating remapping files for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc. What feature of CLM4.5/5 requires this mapping file? Are there any ways we can turn off the particular feature of CLM4.5/5 requiring this data through configure or namelist options, so that we don't need to create a mapping file for this data?

We are going to use a series of global high-resolution or variable-resolution simulations (e.g., globally 30km, 15km, and/or higher resolutions over certain regions) using CAM6-CLM5 configuration. Generating mapping files for UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc makes our workflow quite inefficient...

I think you need this mapping file unfortunately, but I might be wrong. Have you tried to use the Bill's ./master_script_regional.sh script? You can find more info there: https://bb.cgd.ucar.edu/cesm/threads/memory-requirement-for-mkmapdata.2186/