Runtime error in pio, pleiades-san, 1.2.2

[ep2764@columbia_edu]

I am running into a runtime error coming from pio. In the cesm log it shows up as such:
...
...
Opened existing file
/nobackup/mjmills2/ccsmdata/inputdata/atm/waccm/ic/waccm_geos5_2x_88L_2005-01-0
1_c110419.nc -1
Opened existing file
/nobackup/mjmills2/ccsmdata/inputdata/atm/cam/met/USGS-gtopo30_1.9x2.5_phys_geo
s5_c100929.nc -1
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
MPT: Global rank 5 is aborting with error code 1.
Process ID: 5985, Host: r329i5n16, Program: /nobackupp8/epeck/sdwaccm_codrescu_1_2_2/bld/cesm.exe

MPT: --------stack traceback-------
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1229 :
NETCDF not enabled in the build
MPT: Global rank 93 is aborting with error code 1.
Process ID: 94596, Host: r329i6n3, Program: /nobackupp8/epeck/sdwaccm_codrescu_1_2_2/bld/cesm.exe
...
... this repeats for a while followed by other messages
...
...
MPT: #13 0x00007fffffff84fc in ?? ()
MPT: #14 0x00007fffffff8500 in ?? ()
MPT: #15 0x00007fffffff8504 in ?? ()
MPT: #16 0x00007fffffff80a8 in ?? ()
MPT: #17 0x00007fffffff80ac in ?? ()
MPT: #18 0x00000000055b1340 in ?? ()
MPT: #19 0x0000000000000050 in ?? ()
MPT: #20 0x0000000000000050 in ?? ()
MPT: #21 0x0000000000000020 in ?? ()
MPT: #22 0x000000000b7ae440 in ?? ()
MPT: #23 0x0000000000000020 in ?? ()
MPT: #24 0x000000000b7ae440 in ?? ()
MPT: #25 0xffffffee136a6201 in ?? ()
MPT: #26 0x0000000000000004 in ?? ()
MPT: #27 0xffffffee137e0001 in ?? ()
MPT: #28 0x0000001900000018 in ?? ()
MPT: #29 0x08000cd80b7b5b30 in ?? ()
MPT: #30 0x00002aaa00040007 in ?? ()
MPT: #31 0x0849ab9c0131f0b5 in ?? ()
MPT: #32 0x00002aaaac64a598 in mlx4_free_srq_wqe (srq=0x1b68d5c, ind=-46260)
MPT: at src/srq.c:55
MPT: #33 0x00002aaaac648303 in mlx4_poll_cq (ibcq=0x246, ne=1, wc=0x600000000)
MPT: at src/cq.c:370
MPT: #34 0x00002aaaaad5aa69 in ibv_poll_cq (wc=,
MPT: num_entries=, cq=)
MPT: at ../../../../include/verbs_1.2.h:940
MPT: #35 MPI_SGI_ib_progress (wc=,
MPT: num_entries=, cq=)
MPT: at ibdev_multirail.c:4475
MPT: #36 0x00002aaaaad7ca5d in MPI_SGI_progress () at progress.c:215
MPT: #37 0x00002aaaaad87221 in MPI_SGI_request_wait (request=0x7fffffff5ce4,
MPT: status=0x5592a60, set=0x7fffffff5ce0, gen_rc=0x7fffffff5cdc) at req.c:1608
MPT: #38 0x00002aaaaad8f8fd in MPI_SGI_recv (buf=,
MPT: count=, type=,
MPT: des=, tag=,
MPT: comm=, status=0x7fffffff88b0) at sugar.c:40
MPT: #39 0x0000000000000000 in ?? ()
MPT: (gdb) A debugging session is active.
MPT:
MPT: Inferior 1 [process 64775] will be detached.
MPT:
MPT: Quit anyway? (y or n) [answered Y; input not from terminal]
MPT: Detaching from program: /proc/64775/exe, process 64775

MPT: -----stack traceback ends-----

MPT: -----stack traceback ends-----

MPT: -----stack traceback ends-----
MPT: MPI_COMM_WORLD rank 49 has terminated without calling MPI_Finalize()
aborting job


The only thing that says ERROR is related to pio, nf_mod.F90. Any ideas on what is wrong? Thanks!

-Ethan

 

[santos]

It turns out that my previous solution for the CMake module was incomplete, which I only discovered in testing it yesterday. I'm attaching a new patch which I have confirmed should work on pleiades.

 

[santos]

Also, I'm afraid that you will have to rebuild the case from scratch after making this change. I'm not sure that even using "*.clean_build all" and rebuilding will work; it's best to set up a whole new directory. 

 

[aketh_tm@gmail_com]

sir how do i apply the patch just copy paste the whole contents of the file is it , to the new file.

 

[jedwards]

use the unix patch command.

 

[netsanet_kassa@gmail_com]

HI thereIt looks I am having a similar problem here. I have somewhat set up CESM1.2.2 but get the following error while building... CESM BUILDEXE SCRIPT STARTING
rm: No match.
 COMPILER is pgi
 - Build Libraries: mct gptl pio csm_share
Sat Apr 18 17:48:17 PDT 2015 /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/mct.bldlog.150418-174759
Sat Apr 18 17:48:18 PDT 2015 /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/gptl.bldlog.150418-174759
Sat Apr 18 17:48:18 PDT 2015 /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio.bldlog.150418-174759
ERROR: buildlib.pio failed, see /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio.bldlog.150418-174759
ERROR: cat /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio.bldlog.150418-174759

And th elast few lines of the the ERROR file is (where the error starts)gmake[2]: *** [box_rearrange.F90] Error 127
gmake[2]: *** Waiting for unfinished jobs....
/bin/sh: /global/u1/a/alamirew/cesm_all_ver/cesm122/tools/cprnc/genf90/genf90.pl: No such file or directory
gmake[2]: *** [pionfget_mod.F90] Error 127/bin/sh: /global/u1/a/alamirew/cesm_all_ver/cesm122/tools/cprnc/genf90/genf90.pl: No such file or directory

/bin/sh: /global/u1/a/alamirew/cesm_all_ver/cesm122/tools/cprnc/genf90/genf90.pl: No such file or directory
gmake[2]: *** [alloc_mod.F90] Error 127
gmake[2]: *** [pionfput_mod.F90] Error 127
gmake[2]: Leaving directory `/scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio'
gmake[1]: *** [CMakeFiles/pio.dir/all] Error 2
gmake[1]: Leaving directory `/scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio'
gmake: *** [all] Error 2
exit 2
cesm122/scripts> vi /scratch/scratchdirs/alamirew/nino/bld/pgi/mpt/nodebug/nothreads/pio.bldlog.150418-165957

I applied the patch file based on the suggestions(patch >find_netcdf_patch_0.txt) recreated the case but still I get the problem. Can you please help? thanks  

 

[jedwards]

It looks like you are missing a file in your distribution  /global/u1/a/alamirew/cesm_all_ver/cesm122/tools/cprnc/genf90/genf90.plyou might try downloading the source again.