Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Post processing question

I have an old CESM1 simulation that was done years ago and never post processed. I would like to do the post processing now. It was a simulation with a B compset but I'm only interested in the ocn, atm, and ice data. The old case directory still exists and the data is in one of my spaces on glade with the files in atm/hist, ice/hist, and ocn (not in hist, just in the ocn directory, though I could make a hist directory and move them there if needed). To do post processing and create timeseries files for this simulation would I just need to do the following steps:
1) Create the post process directory in the old case directory
2) Edit env_timeseries.xml to remove or comment out all <comp_archive_spec name> entries that are for model components other than cice, pop2, cam
3) For pop2 comment out the line <subdir>hist</subdir>, or create that subdirectory and move the files to it
4) Set DOUT_S_ROOT to the directory that has this old simulation

Would that be correct and sufficient? Thank you!
 

strandwg

Moderator
Staff member
I have an old CESM1 simulation that was done years ago and never post processed. I would like to do the post processing now. It was a simulation with a B compset but I'm only interested in the ocn, atm, and ice data. The old case directory still exists and the data is in one of my spaces on glade with the files in atm/hist, ice/hist, and ocn (not in hist, just in the ocn directory, though I could make a hist directory and move them there if needed). To do post processing and create timeseries files for this simulation would I just need to do the following steps:
1) Create the post process directory in the old case directory
2) Edit env_timeseries.xml to remove or comment out all <comp_archive_spec name> entries that are for model components other than cice, pop2, cam
3) For pop2 comment out the line <subdir>hist</subdir>, or create that subdirectory and move the files to it
4) Set DOUT_S_ROOT to the directory that has this old simulation

Would that be correct and sufficient? Thank you!

Those should be all the steps you need to do. I'd recommend moving all the pop.h history files into ocn/hist just to avoid any glitches.
 
Thanks! I just tried this and it didn't seem to work. The only thing in the error log is this one line over and over:
mismatched tag: line 223, column 2

The only file I modified was env_timeseries.xml and I just removed all lines relating to anything not cice, atm, or pop. I did create the hist directory for ocn/hist and moved the pop files there. The lines removed from env_timeseries.xml were everything in <comp_archive_spec name="clm2"> and the same for rtm, mosart, ww3, and cism. I have a copy of the original version saved as env_timeseries_orig.xml.

This is line 223 of env_timeseries.xml, it is the last line of the file:
</config_definition>

This were the pp_config switches used:
./pp_config --set DOUT_S_ROOT=/glade/scratch/srogstad/archive/RCP85CtrlOld
./pp_config --set ATM_GRID=0.9x1.25
./pp_config --set LND_GRID=0.9x1.25
./pp_config --set ICE_GRID=gx1v6
./pp_config --set OCN_GRID=gx1v6
./pp_config --set ICE_NX=320
./pp_config --set ICE_NY=384
./pp_config --set CASE=RCP85CtrlSourceMods

Pop data is in archive/RCP85CtrlOld/ocn/hist and likewise for atm/hist and ice/hist so the paths should be correct.

So maybe something in env_timeseries.xml was removed that needed to stay but I don't see why since none of those files exist. There is no other info in the log except the one line above and this file timeseries.e3807808 is completely empty. Any suggestions are appreciated!
 

strandwg

Moderator
Staff member
The error indicates that your modifications to env_timeseries.xml left an open "<config_definition>" tag.

Where's the env_timeseries.xml file? I'll take a look.
 
Hi again, I was just trying to post process my current run but the timeseries script just crashed. I was hoping to get this data downloaded tomorrow before the maintenance time starts on Monday, but I know that may not be possible. It fails on a pop file with this error:

NetCDF: HDF error
python: nc4hdf.c:922: var_create_dataset: Assertion `dim && dim->hdr.id == var->dimids[d]' failed.
MPT ERROR: Rank 115(g:115) received signal SIGABRT/SIGIOT(6).
Process ID: 16211, Host: r9i0n14, Program: /glade/p/cesm/postprocessing/cesm-env2/bin/python
MPT Version: HPE MPT 2.19 02/23/19 05:30:09
....
....
MPT: #55 0x00002b5ec0f49653 in PyRun_AnyFileExFlags (fp=fp@entry=0xdcb5d0,
MPT: filename=<optimized out>, closeit=closeit@entry=1,
MPT: flags=flags@entry=0x7ffd9dbf4d1c) at Python/pythonrun.c:761
MPT: #56 0x00002b5ec0f5c730 in Py_Main (argc=<optimized out>, argv=<optimized out>)
MPT: at Modules/main.c:641
MPT: #57 0x00002b5ec1b6b6e5 in __libc_start_main ()
MPT: from /glade/u/apps/ch/os/lib64/libc.so.6
MPT: #58 0x0000000000400839 in _start () at ../sysdeps/x86_64/start.S:118
MPT: (gdb) A debugging session is active.
MPT:
MPT: Inferior 1 [process 16211] will be detached.
MPT:
MPT: Quit anyway? (y or n) [answered Y; input not from terminal]
MPT: Detaching from program: /proc/16211/exe, process 16211

MPT: -----stack traceback ends-----
MPT: On host r9i0n14, Program /glade/p/cesm/postprocessing/cesm-env2/bin/python, Rank 115, Process 16211: Dumping core on signal SIGABRT/SIGIOT(6) into directory /glade/p/cesm/postprocessing/cesm-env2/bin
MPT ERROR: MPI_COMM_WORLD rank 115 has terminated without calling MPI_Finalize()
aborting job
MPT: Received signal 6

Any ideas why this happened? Any help is appreciated, thanks!
 

strandwg

Moderator
Staff member
Hi again, I was just trying to post process my current run but the timeseries script just crashed. I was hoping to get this data downloaded tomorrow before the maintenance time starts on Monday, but I know that may not be possible. It fails on a pop file with this error:

NetCDF: HDF error
python: nc4hdf.c:922: var_create_dataset: Assertion `dim && dim->hdr.id == var->dimids[d]' failed.
MPT ERROR: Rank 115(g:115) received signal SIGABRT/SIGIOT(6).
Process ID: 16211, Host: r9i0n14, Program: /glade/p/cesm/postprocessing/cesm-env2/bin/python
MPT Version: HPE MPT 2.19 02/23/19 05:30:09
....
....
MPT: #55 0x00002b5ec0f49653 in PyRun_AnyFileExFlags (fp=fp@entry=0xdcb5d0,
MPT: filename=<optimized out>, closeit=closeit@entry=1,
MPT: flags=flags@entry=0x7ffd9dbf4d1c) at Python/pythonrun.c:761
MPT: #56 0x00002b5ec0f5c730 in Py_Main (argc=<optimized out>, argv=<optimized out>)
MPT: at Modules/main.c:641
MPT: #57 0x00002b5ec1b6b6e5 in __libc_start_main ()
MPT: from /glade/u/apps/ch/os/lib64/libc.so.6
MPT: #58 0x0000000000400839 in _start () at ../sysdeps/x86_64/start.S:118
MPT: (gdb) A debugging session is active.
MPT:
MPT: Inferior 1 [process 16211] will be detached.
MPT:
MPT: Quit anyway? (y or n) [answered Y; input not from terminal]
MPT: Detaching from program: /proc/16211/exe, process 16211

MPT: -----stack traceback ends-----
MPT: On host r9i0n14, Program /glade/p/cesm/postprocessing/cesm-env2/bin/python, Rank 115, Process 16211: Dumping core on signal SIGABRT/SIGIOT(6) into directory /glade/p/cesm/postprocessing/cesm-env2/bin
MPT ERROR: MPI_COMM_WORLD rank 115 has terminated without calling MPI_Finalize()
aborting job
MPT: Received signal 6

Any ideas why this happened? Any help is appreciated, thanks!

Without being able to log in, it's hard to say.
 
Without being able to log in, it's hard to say.
I'm still having trouble with post processing. I never sorted out the issue from above, but I'm planning to run that simulation farther anyways so I put that issue aside. I am currently trying to post process an old case I just moved back to Cheyenne. I am following exactly what worked above for the previous old case I moved over. It has been running for over an hour (longer than the timeseries script has taken in the past) but so far it has not produced any proc directories or postprocess files. I just double checked and the contents of env_postprocess.xml and env_timeseries.xml are set up exactly like with the case that work (though with the paths updated). It isn't normal for it to run for over an hour and not produce anything right? This is the path in case it is useful: /glade/u/home/srogstad/cases/RCP85PertSourceMods/postprocess
Thanks!
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I see this in /glade/u/home/srogstad/cases/RCP85PertSourceMods/postprocess/logs/timeseries.log.20220701-111847:

[Errno -51] NetCDF: Unknown file format: '/glade/scratch/srogstad/archive/RCP85FW/atm/hist/RCP85PertSourceMods.cam.h0.2081-09.nc'

It seems this file has zero-size:

-rw-r--r-- 1 srogstad ncar 0 Jun 30 14:55 /glade/scratch/srogstad/archive/RCP85FW/atm/hist/RCP85PertSourceMods.cam.h0.2081-09.nc
 

strandwg

Moderator
Staff member
I see this in /glade/u/home/srogstad/cases/RCP85PertSourceMods/postprocess/logs/timeseries.log.20220701-111847:

[Errno -51] NetCDF: Unknown file format: '/glade/scratch/srogstad/archive/RCP85FW/atm/hist/RCP85PertSourceMods.cam.h0.2081-09.nc'

It seems this file has zero-size:

-rw-r--r-- 1 srogstad ncar 0 Jun 30 14:55 /glade/scratch/srogstad/archive/RCP85FW/atm/hist/RCP85PertSourceMods.cam.h0.2081-09.nc

I see there are 19 files in that directory that have zero size. They need to be replaced with correct data files or removed before the postprocessing will work.
 
Top