WACCM hangs whilst writing output.

[marcinkupilas]

I am running cam6_4_028 on Archer2. I am running a WACCM case with regional refinement to 1/8 degree. I have attached my build script, case settings, machine port details (config compilers and machines) and logs from a typical setup. TThe problem I have is the following.

I am trying to run the model for 1 day and write high frequency output every timestep, which is 3 min 45 s.

The model evolves for 15 timesteps, and then hangs. The history file keeps on growing, but doesn't get to the point of closing.

I have tried

- Running on more/less nodes (tried from 8 - 14)
- Changing nsplit parameter (8 - 24)
- Changing max number of entries to history file (mfilt = 384 (1 day), or 16 (1 hour)
- Changing pio type from netcdf to netcdf4p (model doesn't run)

I would appreciate any help.

Best wishes
Marcin

 

[jedwards]

I suggest that you install and use parallel-netcdf. This is the prefered parallel io library for ParallelIO.

 

[jedwards]

Actually your build log shows that you are already have pnetcdf installed - have you
tried using it?
../xmlchange PIO_TYPE_NAME='pnetcdf'

 

[marcinkupilas]

Thanks Jim

I have the following:
./xmlchange PIO_TYPENAME=pnetcdf
ERROR: Did not find pnetcdf in valid values for PIO_TYPENAME: ['netcdf', 'netcdf4p', 'netcdf4c', 'nothing']

I changed PIO_TYPENAME to netcdf4p, then built and submitted the model.

The model failed, and I get lots of the following error in my cesm.log:

Abort with message NetCDF: Invalid argument in file /mnt/lustre/a2fs-work2/work/n02/n02/mmkupilas/tag_cam6_4_028/libraries/parallelio/src/clib/pioc_support.c at line 2176

I have attached the full logs.

Thanks
Marcin

 

[marcinkupilas]

cc @pel

 

[marcinkupilas]

@jedwards @pel

Could you point me to a tag that is known to have a functional Parallel IO that also does WACCM-RR? Do you know if the newest development tag cam6_4_152 has both things working?

Best
Marcin

 

[jedwards]

You need to check and resolve the reason that you cannot build with pnetcdf - it will be in the pio.bldlog file.