Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

How to convert column to lon-lat grid

Status
Not open for further replies.

Min-Seok Kim

Min-Seok Kim
New Member
Dear all,

I would like to ask you how to change restart file format to history file format.
Especially, I'm struggling to study soil moisture impact on climate.
Especially, I want to see the impact of reduced soil moisture in CESM2.

By the way, CLM history file is lon-lat grid output (for SOILLIQ) but restart file is column type output (for H2OSOI_LIQ).
To see the impact of the reduced soil moisture, I want to change H2OSOI_LIQ in restart file.
And to see the data efficiently, firstly, I want to change it to lon-lat grid format.
How I can do it? and also conversely, how I can convert lon-lat grid to column data ?

Is there any guideline about it?
I appreciate your help in advance.
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Restart files contain some variables that should enable you to map from column level to lat-lon. These variables begin with cols1d_ (e.g., cols1d_lat, cols1d_lon).
 

Min-Seok Kim

Min-Seok Kim
New Member
Restart files contain some variables that should enable you to map from column level to lat-lon. These variables begin with cols1d_ (e.g., cols1d_lat, cols1d_lon).

Dear Dr. Oleson,

Thank you for your comment.
I checked there are quite many variables begin with cos1d: cols1d_lon, cols1d_lat, cols1d_ixy, cols1d_jxy, cols1d_gridcell_index, cols1d_landunit_index, cols1d_ityplun, cols1d_ityp, and cols1d_active.

When I searched 'cols1d_landunit_index' on Google and this forum, I can't get the information about meaning of this kind of variables.
Could you let me know where I can get knowledge of these variables?

I'm sorry for bothering you.
Thank you for your help.

Best regards,
Min-Seok
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
The "long_name" attribute associated with each variable on the restart file will supply the meaning of the variable. For example, the long_name for cols1d_ityplun says "column landunit type (see global attributes)". So this is the landunit type associated with this column. The global attributes can be obtained by doing an ncdump -h on the restart file. For example, ncdump -h on one of my files gives the global attributes shown below. So, if cols1d_ityplun was "1" for a column, then the column is associated with a "vegetated_or_bare_soil" landunit (see ilun_vegetated_or_bare_soil below).

:Conventions = "CF-1.0" ;
:history = "created on 12/13/23 07:17:36" ;
:username = "oleson" ;
:host = "cheyenne" ;
:version = "unknown" ;
:source = "Community Terrestrial Systems Model" ;
:revision_id = "$Id: restFileMod.F90 41292 2012-10-26 13:51:45Z erik $" ;
:case_title = "ctsm51_ctsm51d150_ne30pg3ne30pg3mg17_GSWP3V1_ABCDE_blk_ABCDE_1850AD" ;
:case_id = "ctsm51_ctsm51d150_ne30pg3ne30pg3mg17_GSWP3V1_ABCDE_blk_ABCDE_1850AD" ;
:surface_dataset = "/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_ne30
np4.pg3_SSP5-8.5_78pfts_CMIP6_1850-2100_c230227.nc" ;
:flanduse_timeseries = "" ;
:title = "CLM Restart information" ;
:create_crop_landunit = "true" ;
:irrigate = "false" ;
:created_glacier_mec_landunits = "true" ;
:ipft_not_vegetated = 0 ;
:ipft_needleleaf_evergreen_temperate_tree = 1 ;
:ipft_needleleaf_evergreen_boreal_tree = 2 ;
:ipft_needleleaf_deciduous_boreal_tree = 3 ;
:ipft_broadleaf_evergreen_tropical_tree = 4 ;
:ipft_broadleaf_evergreen_temperate_tree = 5 ;
:ipft_broadleaf_deciduous_tropical_tree = 6 ;
:ipft_broadleaf_deciduous_temperate_tree = 7 ;
:ipft_broadleaf_deciduous_boreal_tree = 8 ;
:ipft_broadleaf_evergreen_shrub = 9 ;
:ipft_broadleaf_deciduous_temperate_shrub = 10 ;
:ipft_broadleaf_deciduous_boreal_shrub = 11 ;
:ipft_c3_arctic_grass = 12 ;
:ipft_c3_non-arctic_grass = 13 ;
:ipft_c4_grass = 14 ;
:ipft_c3_crop = 15 ;
:ipft_c3_irrigated = 16 ;
:ipft_temperate_corn = 17 ;
:ipft_irrigated_temperate_corn = 18 ;
:ipft_spring_wheat = 19 ;
:ipft_irrigated_spring_wheat = 20 ;
:ipft_winter_wheat = 21 ;
:ipft_irrigated_winter_wheat = 22 ;
:ipft_temperate_soybean = 23 ;
:ipft_irrigated_temperate_soybean = 24 ;
:ipft_barley = 25 ;
:ipft_irrigated_barley = 26 ;
:ipft_winter_barley = 27 ;
:ipft_irrigated_winter_barley = 28 ;
:ipft_rye = 29 ;
:ipft_irrigated_rye = 30 ;
:ipft_winter_rye = 31 ;
:ipft_irrigated_winter_rye = 32 ;
:ipft_cassava = 33 ;
:ipft_irrigated_cassava = 34 ;
:ipft_citrus = 35 ;
:ipft_irrigated_citrus = 36 ;
:ipft_cocoa = 37 ;
:ipft_irrigated_cocoa = 38 ;
:ipft_coffee = 39 ;
:ipft_irrigated_coffee = 40 ;
:ipft_cotton = 41 ;
:ipft_irrigated_cotton = 42 ;
:ipft_datepalm = 43 ;
:ipft_irrigated_datepalm = 44 ;
:ipft_foddergrass = 45 ;
:ipft_irrigated_foddergrass = 46 ;
:ipft_grapes = 47 ;
:ipft_irrigated_grapes = 48 ;
:ipft_groundnuts = 49 ;
:ipft_irrigated_groundnuts = 50 ;
:ipft_millet = 51 ;
:ipft_irrigated_millet = 52 ;
:ipft_oilpalm = 53 ;
:ipft_irrigated_oilpalm = 54 ;
:ipft_potatoes = 55 ;
:ipft_irrigated_potatoes = 56 ;
:ipft_pulses = 57 ;
:ipft_irrigated_pulses = 58 ;
:ipft_rapeseed = 59 ;
:ipft_irrigated_rapeseed = 60 ;
:ipft_rice = 61 ;
:ipft_irrigated_rice = 62 ;
:ipft_sorghum = 63 ;
:ipft_irrigated_sorghum = 64 ;
:ipft_sugarbeet = 65 ;
:ipft_irrigated_sugarbeet = 66 ;
:ipft_sugarcane = 67 ;
:ipft_irrigated_sugarcane = 68 ;
:ipft_sunflower = 69 ;
:ipft_irrigated_sunflower = 70 ;
:ipft_miscanthus = 71 ;
:ipft_irrigated_miscanthus = 72 ;
:ipft_switchgrass = 73 ;
:ipft_irrigated_switchgrass = 74 ;
:ipft_tropical_corn = 75 ;
:ipft_irrigated_tropical_corn = 76 ;
:ipft_tropical_soybean = 77 ;
:ipft_irrigated_tropical_soybean = 78 ;
:cft_lb = 15 ;
:cft_ub = 78 ;
:icol_vegetated_or_bare_soil = 1 ;
:icol_crop = 2 ;
:icol_crop_noncompete = "2*100+m, m=cft_lb,cft_ub" ;
:icol_landice = "4*100+m, m=1,glcnec" ;
:icol_deep_lake = 5 ;
:icol_wetland = 6 ;
:icol_urban_roof = 71 ;
:icol_urban_sunwall = 72 ;
:icol_urban_shadewall = 73 ;
:icol_urban_impervious_road = 74 ;
:icol_urban_pervious_road = 75 ;
:ilun_vegetated_or_bare_soil = 1 ;
:ilun_crop = 2 ;
:ilun_UNUSED = 3 ;
:ilun_landice = 4 ;
:ilun_deep_lake = 5 ;
:ilun_wetland = 6 ;
:ilun_urban_tbd = 7 ;
:ilun_urban_hd = 8 ;
:ilun_urban_md = 9 ;
:issue_fixed_783 = 1 ;
:issue_fixed_1140 = 1 ;
 

Min-Seok Kim

Min-Seok Kim
New Member
The "long_name" attribute associated with each variable on the restart file will supply the meaning of the variable. For example, the long_name for cols1d_ityplun says "column landunit type (see global attributes)". So this is the landunit type associated with this column. The global attributes can be obtained by doing an ncdump -h on the restart file. For example, ncdump -h on one of my files gives the global attributes shown below. So, if cols1d_ityplun was "1" for a column, then the column is associated with a "vegetated_or_bare_soil" landunit (see ilun_vegetated_or_bare_soil below).

:Conventions = "CF-1.0" ;
:history = "created on 12/13/23 07:17:36" ;
:username = "oleson" ;
:host = "cheyenne" ;
:version = "unknown" ;
:source = "Community Terrestrial Systems Model" ;
:revision_id = "$Id: restFileMod.F90 41292 2012-10-26 13:51:45Z erik $" ;
:case_title = "ctsm51_ctsm51d150_ne30pg3ne30pg3mg17_GSWP3V1_ABCDE_blk_ABCDE_1850AD" ;
:case_id = "ctsm51_ctsm51d150_ne30pg3ne30pg3mg17_GSWP3V1_ABCDE_blk_ABCDE_1850AD" ;
:surface_dataset = "/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_ne30
np4.pg3_SSP5-8.5_78pfts_CMIP6_1850-2100_c230227.nc" ;
:flanduse_timeseries = "" ;
:title = "CLM Restart information" ;
:create_crop_landunit = "true" ;
:irrigate = "false" ;
:created_glacier_mec_landunits = "true" ;
:ipft_not_vegetated = 0 ;
:ipft_needleleaf_evergreen_temperate_tree = 1 ;
:ipft_needleleaf_evergreen_boreal_tree = 2 ;
:ipft_needleleaf_deciduous_boreal_tree = 3 ;
:ipft_broadleaf_evergreen_tropical_tree = 4 ;
:ipft_broadleaf_evergreen_temperate_tree = 5 ;
:ipft_broadleaf_deciduous_tropical_tree = 6 ;
:ipft_broadleaf_deciduous_temperate_tree = 7 ;
:ipft_broadleaf_deciduous_boreal_tree = 8 ;
:ipft_broadleaf_evergreen_shrub = 9 ;
:ipft_broadleaf_deciduous_temperate_shrub = 10 ;
:ipft_broadleaf_deciduous_boreal_shrub = 11 ;
:ipft_c3_arctic_grass = 12 ;
:ipft_c3_non-arctic_grass = 13 ;
:ipft_c4_grass = 14 ;
:ipft_c3_crop = 15 ;
:ipft_c3_irrigated = 16 ;
:ipft_temperate_corn = 17 ;
:ipft_irrigated_temperate_corn = 18 ;
:ipft_spring_wheat = 19 ;
:ipft_irrigated_spring_wheat = 20 ;
:ipft_winter_wheat = 21 ;
:ipft_irrigated_winter_wheat = 22 ;
:ipft_temperate_soybean = 23 ;
:ipft_irrigated_temperate_soybean = 24 ;
:ipft_barley = 25 ;
:ipft_irrigated_barley = 26 ;
:ipft_winter_barley = 27 ;
:ipft_irrigated_winter_barley = 28 ;
:ipft_rye = 29 ;
:ipft_irrigated_rye = 30 ;
:ipft_winter_rye = 31 ;
:ipft_irrigated_winter_rye = 32 ;
:ipft_cassava = 33 ;
:ipft_irrigated_cassava = 34 ;
:ipft_citrus = 35 ;
:ipft_irrigated_citrus = 36 ;
:ipft_cocoa = 37 ;
:ipft_irrigated_cocoa = 38 ;
:ipft_coffee = 39 ;
:ipft_irrigated_coffee = 40 ;
:ipft_cotton = 41 ;
:ipft_irrigated_cotton = 42 ;
:ipft_datepalm = 43 ;
:ipft_irrigated_datepalm = 44 ;
:ipft_foddergrass = 45 ;
:ipft_irrigated_foddergrass = 46 ;
:ipft_grapes = 47 ;
:ipft_irrigated_grapes = 48 ;
:ipft_groundnuts = 49 ;
:ipft_irrigated_groundnuts = 50 ;
:ipft_millet = 51 ;
:ipft_irrigated_millet = 52 ;
:ipft_oilpalm = 53 ;
:ipft_irrigated_oilpalm = 54 ;
:ipft_potatoes = 55 ;
:ipft_irrigated_potatoes = 56 ;
:ipft_pulses = 57 ;
:ipft_irrigated_pulses = 58 ;
:ipft_rapeseed = 59 ;
:ipft_irrigated_rapeseed = 60 ;
:ipft_rice = 61 ;
:ipft_irrigated_rice = 62 ;
:ipft_sorghum = 63 ;
:ipft_irrigated_sorghum = 64 ;
:ipft_sugarbeet = 65 ;
:ipft_irrigated_sugarbeet = 66 ;
:ipft_sugarcane = 67 ;
:ipft_irrigated_sugarcane = 68 ;
:ipft_sunflower = 69 ;
:ipft_irrigated_sunflower = 70 ;
:ipft_miscanthus = 71 ;
:ipft_irrigated_miscanthus = 72 ;
:ipft_switchgrass = 73 ;
:ipft_irrigated_switchgrass = 74 ;
:ipft_tropical_corn = 75 ;
:ipft_irrigated_tropical_corn = 76 ;
:ipft_tropical_soybean = 77 ;
:ipft_irrigated_tropical_soybean = 78 ;
:cft_lb = 15 ;
:cft_ub = 78 ;
:icol_vegetated_or_bare_soil = 1 ;
:icol_crop = 2 ;
:icol_crop_noncompete = "2*100+m, m=cft_lb,cft_ub" ;
:icol_landice = "4*100+m, m=1,glcnec" ;
:icol_deep_lake = 5 ;
:icol_wetland = 6 ;
:icol_urban_roof = 71 ;
:icol_urban_sunwall = 72 ;
:icol_urban_shadewall = 73 ;
:icol_urban_impervious_road = 74 ;
:icol_urban_pervious_road = 75 ;
:ilun_vegetated_or_bare_soil = 1 ;
:ilun_crop = 2 ;
:ilun_UNUSED = 3 ;
:ilun_landice = 4 ;
:ilun_deep_lake = 5 ;
:ilun_wetland = 6 ;
:ilun_urban_tbd = 7 ;
:ilun_urban_hd = 8 ;
:ilun_urban_md = 9 ;
:issue_fixed_783 = 1 ;
:issue_fixed_1140 = 1 ;

I appreciate for your help, that is exactly the what I wanted to see.

Actually, I wanted to know information about 'cols1d_landunit_index'.
Because I found a python script for converting lon-lat grid file to column format which using landunit index.
It is from 'Soil Moisture Data Set — prescribeSM 0 documentation'
Even though it is for CLM4, I think the method will be similar.
I think cols1d_landunit_index in CLM5 is same with cols1d_itype_lunit in CLM4.

In the file clm_col_to_xy_example.py, cols1d_itype_lunit is identified and used.
If the value of the cols1d_itype_lunit is 1, it seems like that the point is considered as soil.
But when I checked, 2 means crop, 6 means wetland, 7 to 9 means urban regions.
:ltype_vegetated_or_bare_soil = 1 ;
:ltype_crop = 2 ;
:ltype_UNUSED = 3 ;
:ltype_landice = 4 ;
:ltype_deep_lake = 5 ;
:ltype_wetland = 6 ;
:ltype_urban_tbd = 7 ;
:ltype_urban_hd = 8 ;
:ltype_urban_md = 9 ;

I'm confusing why the script only uses value of 1.
In my guess, 2, 6, 7, 8, 9 should be also considered because there can be also soil moisture data.

If I use the script, do you think I should follow the script with converting the data only where cols1d_itype_lunit is 1 ?
Or will it be better to consider the points for crop, wetland, urban together ?

Below is a part of python code for converting.
with nc.Dataset(fN_3D_in) as ncf: # get indices cols1d_ixy = ncf.variables['cols1d_ixy'][:] cols1d_jxy = ncf.variables['cols1d_jxy'][:] cols1d_itype_lunit = ncf.variables['cols1d_itype_lunit'][:] istsoil = 1 # not all the points in SOILLIQ and SOILICE are SM sel_soil = cols1d_itype_lunit == istsoil # python uses 0 based indexing col = cols1d_ixy[sel_soil] - 1 row = cols1d_jxy[sel_soil] - 1
 

Min-Seok Kim

Min-Seok Kim
New Member
This is a clm_col_to_xy_example.py script.

#!/usr/bin/env python # -*- coding: utf-8 -*- # Author: Mathias Hauser # Date: 08.2014 # USAGE: CONVERT clm history files in xy-format (lat, lon) to col format try: import netCDF4 as nc except: # netCDF4 is not available on cscs import scipy.io.netcdf as nc nc.Dataset = nc.netcdf_file import shutil # DEFINE FILE NAMES # example file in column/ 3D form (see README) # this file needs to exist fN_3D_in = 'file_3D_in.nc' # input file in lat-lon/ 4D form # this file needs to exist # this is what we want to transform fN_4D = 'file_4D.nc' # output file: this is the transformed fN_xy # does not need to exist (will be overwritten) fN_3D_out = 'file_3D_out.nc' # START PROGRAM with nc.Dataset(fN_3D_in) as ncf: # get indices cols1d_ixy = ncf.variables['cols1d_ixy'][:] cols1d_jxy = ncf.variables['cols1d_jxy'][:] cols1d_itype_lunit = ncf.variables['cols1d_itype_lunit'][:] istsoil = 1 # not all the points in SOILLIQ and SOILICE are SM sel_soil = cols1d_itype_lunit == istsoil # python uses 0 based indexing col = cols1d_ixy[sel_soil] - 1 row = cols1d_jxy[sel_soil] - 1 # copy col/3D 'example file' to 'output file' shutil.copyfile(fN_3D_in, fN_3D_out) # read from fN_xy (only the first 10 lvl are SM levels) with nc.Dataset(fN_4D) as ncf: SOILLIQ_xy = ncf.variables['SOILLIQ'][:, 0:10, :, :] SOILICE_xy = ncf.variables['SOILICE'][:, 0:10, :, :] # write out the transformed SM with nc.Dataset(fN_3D_out, 'a') as ncf: ncf.variables['SOILLIQ'][:, 0:10, sel_soil] = SOILLIQ_xy[:, :, row, col] ncf.variables['SOILICE'][:, 0:10, sel_soil] = SOILICE_xy[:, :, row, col]
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I'm not familiar with that script at all, sorry. But I think you would want to deal with soil moisture for at least veg and crop. You could do urban as well, but there are three urban landunits and there is soil moisture only for the pervious canyon floor which is generally a small fraction of the urban areas, plus urban areas are quite small in the surface dataset, so it may not matter much in terms of grid cell averages. We don't have wetlands in the model anymore.
 

Min-Seok Kim

Min-Seok Kim
New Member
I'm not familiar with that script at all, sorry. But I think you would want to deal with soil moisture for at least veg and crop. You could do urban as well, but there are three urban landunits and there is soil moisture only for the pervious canyon floor which is generally a small fraction of the urban areas, plus urban areas are quite small in the surface dataset, so it may not matter much in terms of grid cell averages. We don't have wetlands in the model anymore.
Dear Dr. Oleson,

I appreciate your feedback.
I'll have to study about it some more, and your comments should be very helpful in understanding the data structure.

Best regards,
Min-Seok
 
Status
Not open for further replies.
Top