CESM run fails on hopper with ERROR: ionf_mod.F90

[mvangordon@berkeley_edu]

I'm working running CESM/CLM on hopper. I can build successfully, but the run fails with the following error in my $RUNDIR/cesm.log... file:

...
Opened existing file
/project/projectdirs/ccsm1/inputdata/share/domains/domain.lnd.fv0.9x1.25_gx1v6.090309.nc
65536
Rank 1 [Thu Mar 13 23:28:58 2014] [c9-2c2s7n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - proce
ss 1
pio_support::pio_die:: myrank= -1 : ERROR: ionf_mod.F90:
228 : Permission denied
_pmiu_daemon(SIGCHLD): [NID 04785] [c9-2c2s7n3] [Thu Mar 13 23:28:58 2014] PE RANK 1 exit signal Abort
ed
[NID 04785] 2014-03-13 16:28:58 Apid 26957592: initiated application termination
...

I would appreciate any insight on what the issue is or how I should go about addressing it.

Thanks.

 

[jedwards]

Is this a reproducable error?   Sometimes intermitant file system issues will cause it.   If it is repeatable please let us know the details -what is the model version, the case setup, any changes from the default configuration.

 

[mvangordon@berkeley_edu]

It is a reproducable error. Model version is CESM 1.2.1. Set up is as follows:cesm1_2_1/scripts> ./create_newcase -case ~/cesm1_2_1/cases/quickstart_test -mach hopper -res f09_g16 -compset I1850CRUCLM45BGCcases/quickstart_test> ./xmlchange STOP_OPTION=nmonth,STOP_N=1cases/quickstart_test> ./cesm_setup cases/quickstart_test> ./quickstart_test.build cases/quickstart_test> ./quickstart_test.submit

 

[jedwards]

I just tried it twice and didn't have any problems.   Can you access that file with the netcdf ncdump command? 

 

[mvangordon@berkeley_edu]

I can, yes.

 

[adrianne]

I have gotten that error message when I was missing the clm2.rh restarts.

 

[mvangordon@berkeley_edu]

Hi Adrianne,Could you offer some more detail? How/why would I be missing these and where should they be? This is a first quickstart test run without changing any defaults.Thanks.

 

[adrianne]

If it's a first quickstart test, and you haven't changed any defaults, then the model should have given you initial datasets, so restarts (including the clm.rh files) aren't likely to be the problem.

 

[erik]

Hi
It looks like you aren't in the ccsm1 group. I think that might be an issue. You should be able to read the file, but I think sometimes it stilkl wants you to have write permission even on a read...Here's the permissions on that file.-rw-rw-r-- 1 mickelso ccsm1 5532048 Jul 29  2012 /project/projectdirs/ccsm1/inputdata/share/domains/domain.lnd.fv0.9x1.25_gx1v6.090309.ncTry asking system folks at NERSC to add you to the ccsm1 group and see if it works after that.It's also possible it's another file that it's having trouble with. You might need to load it in a debugger to see exactly where it dies and in what file. Good luck. 

 

[mvangordon@berkeley_edu]

Thanks, Erik! Being added to the ccsm1 group fixed the issue.

 

[wagmanbe@utexas_edu]

I had the same error on stampede. The solution was to use ./xmlchange DIN_LOC_ROOT= Create that directory somewhere with plenty of space. When you setup/build, the input files will be sent over. I did chmod 777 on the problematic input file, but not sure if that was necessary. No further issues with permissions.