robinweiss@uchicago_edu
New Member
I'm running into an issue with running the pop2 ocean perturbation validation test as described here http://www.cesm.ucar.edu/models/cesm1.0/pop2/validation/index.html.
The issue I'm running into occurs at run time. The model builds without error and when launched, all modules successfully initialize and the simulation begins running. After just a few seconds, the run terminates and the code creates a bunch of core dumps. I have grep'ed the log files extensively, but have not been able to find any error outputs. The run simply dies without any warning.
In an effort to troubleshoot the problem, I am not doing steps 2, 3, and 4 from the port validation procedure (as near as I can tell, these just modify the way the model outputs data). The procedure I'm using to build this case is as follows (my machine is named midway):
./create_newcase -compset C -res T62_gx1v6 -mach midway -case popValidation
cd popValidation
./configure -case
./popValidation.midway.build
./popValidation.midway.submit
I am building with netCDF v4.2 and the intel compiler suite with intel mpi v12.1. For troubleshooting purposes, I have turned off all compiler optimizations. Building the model does not generate any errors or warnings. I also forced CESM to re-download all the necessary input data files to try to rule out that problem.
I had one of my colleagues try this same procedure on a known working system, and the same result was observed (model builds and initializes successfully, but crashes shortly after it starts running). My first guess is that we're running into a numerical instability and the run is blowing up. I attempted to change the time stepping of the simulation by increasing the variable dt_count as was suggested in another post. This does not resolve the issue however.
I would greatly appreciate if someone could try this same build procedure on a different known-good machine and see if the same crash occurs. Please let me know what compiler/options you are using to build. Also, any other suggestions about what knobs need to be turned or buttons pressed to get this port validation procedure working would be greatly appreciated.
Thanks,
Robin
The issue I'm running into occurs at run time. The model builds without error and when launched, all modules successfully initialize and the simulation begins running. After just a few seconds, the run terminates and the code creates a bunch of core dumps. I have grep'ed the log files extensively, but have not been able to find any error outputs. The run simply dies without any warning.
In an effort to troubleshoot the problem, I am not doing steps 2, 3, and 4 from the port validation procedure (as near as I can tell, these just modify the way the model outputs data). The procedure I'm using to build this case is as follows (my machine is named midway):
./create_newcase -compset C -res T62_gx1v6 -mach midway -case popValidation
cd popValidation
./configure -case
./popValidation.midway.build
./popValidation.midway.submit
I am building with netCDF v4.2 and the intel compiler suite with intel mpi v12.1. For troubleshooting purposes, I have turned off all compiler optimizations. Building the model does not generate any errors or warnings. I also forced CESM to re-download all the necessary input data files to try to rule out that problem.
I had one of my colleagues try this same procedure on a known working system, and the same result was observed (model builds and initializes successfully, but crashes shortly after it starts running). My first guess is that we're running into a numerical instability and the run is blowing up. I attempted to change the time stepping of the simulation by increasing the variable dt_count as was suggested in another post. This does not resolve the issue however.
I would greatly appreciate if someone could try this same build procedure on a different known-good machine and see if the same crash occurs. Please let me know what compiler/options you are using to build. Also, any other suggestions about what knobs need to be turned or buttons pressed to get this port validation procedure working would be greatly appreciated.
Thanks,
Robin