Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Using PNETCDF for CESM

N

nitkbhat@gmail_com

Member
I am using Pnetcdf 1.4.0 for parallel IO. I have built it along with hdf5 1.8.8 and netcdf 4.3.1.1. After successful build, I get the following error when I run the model:  MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fff7ab9288c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12Fatal error in MPI_Group_range_incl: Invalid argument, error stack:MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fffc2096b0c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12Fatal error in MPI_Group_range_incl: Invalid argument, error stack:MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fff6d27888c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12Fatal error in MPI_Group_range_incl: Invalid argument, error stack:MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fff2f1c5c8c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12Fatal error in MPI_Group_range_incl: Invalid argument, error stack:MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fff329e868c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12Fatal error in MPI_Group_range_incl: Invalid argument, error stack:MPI_Group_range_incl(188).........: MPI_Group_range_incl(group=0x88000002, n=1, ranges=0x14c2e2c0, new_group=0x7fffa0a2828c) failedMPIR_Group_check_valid_ranges(324): The 0th element of a range array ends at 15 but must be nonnegative and less than 12     What is the configuration required for PNETCDF usage?               Even the component typenames have been given as pnetcdf. Why am I getting the error?Thanks, 
 
J

jedwards

CSEG and Liaisons
Staff member
What mpi library and version are you using?   What is the CESM version and what case are you trying to run?   What compiler version are you using?   Can you attach the full cesm.log file with your reply.     Thanks,
 
J

jedwards

CSEG and Liaisons
Staff member
What mpi library and version are you using?   What is the CESM version and what case are you trying to run?   What compiler version are you using?   Can you attach the full cesm.log file with your reply.     Thanks,
 
N

nitkbhat@gmail_com

Member
MPI : Intel MPI Version 4.1.3  Build 20140124Case : B component set - f19_g16 resolution - 5 simulation daysCESM version : Cesm 1.2.2 Compiler : Intel compiler 14.0.2.144 Build 20140120 (mpiicc, mpiifort, mpiicpc)zlib: 1.2.8 hdf5: 1.8.8 Netcdf version : 4.3.1.1 Netcdf fortan version: 4.4.0 Pnetcdf : 1.4.0    Please find attached the cesm.log file and the env_run.xml  
 
N

nitkbhat@gmail_com

Member
MPI : Intel MPI Version 4.1.3  Build 20140124Case : B component set - f19_g16 resolution - 5 simulation daysCESM version : Cesm 1.2.2 Compiler : Intel compiler 14.0.2.144 Build 20140120 (mpiicc, mpiifort, mpiicpc)zlib: 1.2.8 hdf5: 1.8.8 Netcdf version : 4.3.1.1 Netcdf fortan version: 4.4.0 Pnetcdf : 1.4.0    Please find attached the cesm.log file and the env_run.xml  
 
J

jedwards

CSEG and Liaisons
Staff member

Set this to false - The async_interface is an experimental feature and is not supported out of the box. 
 
J

jedwards

CSEG and Liaisons
Staff member

Set this to false - The async_interface is an experimental feature and is not supported out of the box. 
 
N

nitkbhat@gmail_com

Member
After setting the variable back correctly, I am able to built with the configuration as mentioned in the previous comment. (with pnetcdf) So, for CESM to use pnetcdf, is mentioning the PNETCDF_PATH in the Macros file, all that is required? Are there any other flags that are required? Should the pio io variables changed to pnetcdf? 
 
N

nitkbhat@gmail_com

Member
After setting the variable back correctly, I am able to built with the configuration as mentioned in the previous comment. (with pnetcdf) So, for CESM to use pnetcdf, is mentioning the PNETCDF_PATH in the Macros file, all that is required? Are there any other flags that are required? Should the pio io variables changed to pnetcdf? 
 
N

nitkbhat@gmail_com

Member
When i set the io variables in env_run.xml, I got the following error. ...........123  node1124  node1125  node1126  node1127  node1 Opened existing file /storage/home/nitin/CESM_NEW/input_data/atm/cam/inic/fv/cami_0000-01-01_1.9x2.5 _L26_c070408.nc          -1 Opened existing file /storage/home/nitin/CESM_NEW/input_data/atm/cam/topo/USGS-gtopo30_1.9x2.5_remap _c050602.nc          -1 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 9application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 17 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 21 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 25 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 81application called MPI_Abort(MPI_COMM_WORLD, 1) - process 29 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 85 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 49application called MPI_Abort(MPI_COMM_WORLD, 1) - process 93 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 33application called MPI_Abort(MPI_COMM_WORLD, 1) - process 57 PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 97 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 37application called MPI_Abort(MPI_COMM_WORLD, 1) - process 61application called MPI_Abort(MPI_COMM_WORLD, 1) - process 89application called MPI_Abort(MPI_COMM_WORLD, 1) - process 105 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 45 PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 109 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 53 PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 41application called MPI_Abort(MPI_COMM_WORLD, 1) - process 101[0:node8] unexpected disconnect completion event from [17:node7]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 0 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 113application called MPI_Abort(MPI_COMM_WORLD, 1) - process 117application called MPI_Abort(MPI_COMM_WORLD, 1) - process 121application called MPI_Abort(MPI_COMM_WORLD, 1) - process 125[16:node7] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 16[96:node2] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0[32:node6] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 32internal ABORT - process 96[48:node5] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 48[80:node3] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 80[64:node4] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0application called MPI_Abort(MPI_COMM_WORLD, 1) - process 65application called MPI_Abort(MPI_COMM_WORLD, 1) - process 69application called MPI_Abort(MPI_COMM_WORLD, 1) - process 73application called MPI_Abort(MPI_COMM_WORLD, 1) - process 77internal ABORT - process 64[112:node1] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 112[95:node3] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 95..........      How to enable PNETCDF in the build? And what exactly is the configuration in env_run.xml for running cesm? (PFA env_run.xml and Macros)  Thanks. 
 
N

nitkbhat@gmail_com

Member
When i set the io variables in env_run.xml, I got the following error. ...........123  node1124  node1125  node1126  node1127  node1 Opened existing file /storage/home/nitin/CESM_NEW/input_data/atm/cam/inic/fv/cami_0000-01-01_1.9x2.5 _L26_c070408.nc          -1 Opened existing file /storage/home/nitin/CESM_NEW/input_data/atm/cam/topo/USGS-gtopo30_1.9x2.5_remap _c050602.nc          -1 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 9application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 17 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 21 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 25 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 81application called MPI_Abort(MPI_COMM_WORLD, 1) - process 29 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 85 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 49application called MPI_Abort(MPI_COMM_WORLD, 1) - process 93 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 :application called MPI_Abort(MPI_COMM_WORLD, 1) - process 33application called MPI_Abort(MPI_COMM_WORLD, 1) - process 57 PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 97 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 37application called MPI_Abort(MPI_COMM_WORLD, 1) - process 61application called MPI_Abort(MPI_COMM_WORLD, 1) - process 89application called MPI_Abort(MPI_COMM_WORLD, 1) - process 105 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 45 PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 109 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 53 PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 41application called MPI_Abort(MPI_COMM_WORLD, 1) - process 101[0:node8] unexpected disconnect completion event from [17:node7]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 0 pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1229 : PNETCDF not enabled in the build PNETCDF not enabled in the buildapplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 113application called MPI_Abort(MPI_COMM_WORLD, 1) - process 117application called MPI_Abort(MPI_COMM_WORLD, 1) - process 121application called MPI_Abort(MPI_COMM_WORLD, 1) - process 125[16:node7] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 16[96:node2] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0[32:node6] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 32internal ABORT - process 96[48:node5] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 48[80:node3] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 80[64:node4] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0application called MPI_Abort(MPI_COMM_WORLD, 1) - process 65application called MPI_Abort(MPI_COMM_WORLD, 1) - process 69application called MPI_Abort(MPI_COMM_WORLD, 1) - process 73application called MPI_Abort(MPI_COMM_WORLD, 1) - process 77internal ABORT - process 64[112:node1] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 112[95:node3] unexpected disconnect completion event from [0:node8]Assertion failed in file ../../dapl_conn_rc.c at line 1179: 0internal ABORT - process 95..........      How to enable PNETCDF in the build? And what exactly is the configuration in env_run.xml for running cesm? (PFA env_run.xml and Macros)  Thanks. 
 
J

jedwards

CSEG and Liaisons
Staff member
This message  "PNETCDF not enabled in the build" indicates that the pio library did not build correctly.   Please run $CASE.clean_build all$CASE.buildand look in the PIO build log to see if PNETCDF was properly detected and built in.  
 
J

jedwards

CSEG and Liaisons
Staff member
This message  "PNETCDF not enabled in the build" indicates that the pio library did not build correctly.   Please run $CASE.clean_build all$CASE.buildand look in the PIO build log to see if PNETCDF was properly detected and built in.  
 
N

nitkbhat@gmail_com

Member
I ran the following. I get successful builds. I don't see any error in the PIO build log. I am not finding anything relating to parallel netcdf.  I think that the IO configuration in env_run.xml should be changed to enable pnetcdf. I am not sure what exactly it has to be changed to. It is mentioned in the following section of the user manual. http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide/x2147.html. In the 5 steps for turning on PNETCDF, can I get help relating to step 5. I don't understand the configuration required for making PIO with parallel netcdf. 
 
N

nitkbhat@gmail_com

Member
I ran the following. I get successful builds. I don't see any error in the PIO build log. I am not finding anything relating to parallel netcdf.  I think that the IO configuration in env_run.xml should be changed to enable pnetcdf. I am not sure what exactly it has to be changed to. It is mentioned in the following section of the user manual. http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide/x2147.html. In the 5 steps for turning on PNETCDF, can I get help relating to step 5. I don't understand the configuration required for making PIO with parallel netcdf. 
 
J

jedwards

CSEG and Liaisons
Staff member
This message indicates that PNETCDF was not linked with your application and nothing you do in env_run.xml is going to change that.   Do a clean build andcheck the pio.bldlog for errors.  
 
J

jedwards

CSEG and Liaisons
Staff member
This message indicates that PNETCDF was not linked with your application and nothing you do in env_run.xml is going to change that.   Do a clean build andcheck the pio.bldlog for errors.  
 
You must log in or register to reply here.
Top