Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

simulation hangs after "opened existing file" <bnd_topo file>

Hello,I am trying to run a simulation using the FSDW compset with resolution f19_f19 using cesm1.2(rel06). I am able to build the simulation without any problems. Preview_namelists shows no errors and check_case is OK. When I try to run the simulation, it starts running fine but then gets hung up at the same point each time. The last few lines of the cesm.log file are: Opened existing file /projects/ccsm/inputdata/atm/waccm/ic/waccm_geos5_2x_88L_2005-01-01_c110419.nc       65536 Opened existing file /projects/ccsm/inputdata/atm/cam/met/USGS-gtopo30_1.9x2.5_phys_geos5_c100929.nc      131072 The last few lines of the atm.log file are: procid          398  assigned            3  latitude values from          91  through           93 procid          399  assigned            3  latitude values from          94  through           96

The last few lines of the cpl.log file are:(seq_timemgr_clockPrint)   Alarm =  7  seq_timemgr_alarm_tprof(seq_timemgr_clockPrint)     Prev Time   = 00001201   00000(seq_timemgr_clockPrint)     Next Time   = 99991201   00000(seq_timemgr_clockPrint)     Intervl yms =     9999       0           0 (seq_mct_drv) : Initialize each component: atm, lnd, rof, ocn, ice, glc, wav(seq_mct_drv) : Initialize atm component ATM  The ice.log, lnd.log, ocn.log, and rof.log files are not created. No history files are written. The simulation does not end in an error; instead it continues to run until the time limit on the job is reached, without ever progressing past "opened existing file ". I have tried changing the bnd_topo file in user_nl_cam to a bnd_topo file that I know works with another compset/resolution (FWSC/ne30_ne30), and still get the same error. I suspect this is a problem with the resolution (f19_f19) because other simulations that I've run with ne30_ne30 have run just fine, and every simulation I've tried to run with the f19_f19 resolution (using various compsets) has failed in this way. I chose to use f19_f19 with FSDW because the FSDW compset uses "-offline_dyn" in env_build.xml, and this setting works with "fv" but not "se" resolution. I'd appreciate any insights anyone might have that might help me progress past this problem! Thanks in advance,Marie 
 
Some additional notes: I am seeing the same problem with compsets (resolutions) FSSOA (f19_f19), FC5PM (f19_f19), FSDCHM (f19_f19), FMOZSOA (f19_f19), F1850W (f19_f19), FW (f19_g16), FWX (f19_g16), FWX1996 (f19_g16), and FC5 (f19_f19). The FC5 compset runs just fine with ne30_ne30, but none of the others work with ne30_ne30 so I was using a finite volume grid with those. For the FSDCHM compset, I tried the following resolutions with the same outcome: f19_f19, f19_g16, f19_g16_r01, and f19_s11. With other resolutions I was unable to build the simulation (f02_f02, f02_g16, f02_t12, f05_f05, f05_g16, f05_t12, f09_f09, f09_g16, f09_g16_rx1, f10_f10, f119_f19_g110, f19_g16_g110, f19_g16_rx1, f25_f25, f45_f45, f45_g37, f45_g37_rx1, g09_g16, g16_g16, ne30_f19_g16, ne20_f19_g16_rx1, ne30_g16, T31_T31, T62_s11).
 
I am running on our local cluster. It has 2.93 GHz Intel Nehalem processor with 2 sockets, 4 cores, RHEL 6 operating system, with Infiniband. 1.5 GB RAM per core (12 GB per node), 264 teraflops. I generally run on 400 cores. I haven't gotten any responses yet but am still hopeful that someone will have some ideas for me! Marie
 
Top