What is the change on yellowstone affecting CCSM/CESM?Recent changes to yellowstone have disabled the ability to run many offline scripts interactively on the login nodes on yellowstone. Examples of these for CLM include: PTCLM and mksurfdata_map as well as interpinic. Our instructions to this point tell you to run these scripts and tools interactively on login nodes, but yellowstone will now kill these interactive jobs when they consume too much CPU and/or memory. This can also include even running your $CASE.build for any kind of CESM case.What to Do about it?In the case of scripts or tools you run interactively on login nodes, you now need to submit them to the batch queue. We recommend sending them to the caldera queue (in the case of PTCLM you need to submit to the geyser queue). We also recommending using the "execca" script to submit to caldera and the "execgy" script to submit to geyser. These scripts will be part of your path as a yellowstone user, they simply submit the script or command you list after to the given queue. You can also create an interactive session with caldera or geyser through bsub, and that kind of usage works fine (that's what the "execca" and "execgy" scripts do anyway).For example...execca $CASE.build >& build.log &Will submit the build to the caldera queue.
You can also get help in using the script with the "-help" option. There are also two env variables that can be used DAV_CORES and DAV_PROJECT. DAV_PROJECT sets the project account to use, and DAV_CORES sets the number of processors to use. Neither is required, but can be useful depending on what you are doing.
In the case of your build, you can also get it to go by lowering your "GMAKE_J", for example./xmlchange GMAKE_J=4What does it affect?This is for all versions of CCSM4 and CESM as it is a change to the machine itself.Example list of tools effected:
This was a change in order to improve the performance of the login nodes and prevent the freezing that has often occurred.See the CISL Daily-Bulletin June/4th announcement for some information on this change.http://dailyb.cisl.ucar.edu/bulletins/cisl-continues-pursue-file-system-hangs
You can also get help in using the script with the "-help" option. There are also two env variables that can be used DAV_CORES and DAV_PROJECT. DAV_PROJECT sets the project account to use, and DAV_CORES sets the number of processors to use. Neither is required, but can be useful depending on what you are doing.
In the case of your build, you can also get it to go by lowering your "GMAKE_J", for example./xmlchange GMAKE_J=4What does it affect?This is for all versions of CCSM4 and CESM as it is a change to the machine itself.Example list of tools effected:
- $CASE.build
- PTCLM.py
- mksurfdata_map
- interpinic
- mkmapdata.sh
This was a change in order to improve the performance of the login nodes and prevent the freezing that has often occurred.See the CISL Daily-Bulletin June/4th announcement for some information on this change.http://dailyb.cisl.ucar.edu/bulletins/cisl-continues-pursue-file-system-hangs