Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

porting CCSM3 code from yellowstone to cheyenne

H

heavens

Member
The most progress I have been able to make is to use something like this in the run script:mpiexec_mpt -v    -np $NTASKS[1] omplace $EXEROOT/all/$COMPONENTS[1]  :    -np $NTASKS[2] omplace $EXEROOT/all/$COMPONENTS[2]  :    -np $NTASKS[3] omplace $EXEROOT/all/$COMPONENTS[3] :    -np $NTASKS[4] omplace $EXEROOT/all/$COMPONENTS[4] :    -np $NTASKS[5] omplace $EXEROOT/all/$COMPONENTS[5]   & The challenge is that you may encounter the error, "MPT ERROR: could not run executable. If this is a non-MPT application,you may need to set MPI_SHEPHERD=true." This is deceptive. It is not caused here by "a bad node" as you may find by searching the forums here. The issue is that MCT does not recognize the various CCSM3 executables as valid MPI programs. I have found that this can be partly solved by ensuring that the code is compiled by the MCT versions of the MPI compilers, but I still end up with segmentation faults:"MPT ERROR: Rank 0(g:0) received signal SIGSEGV(11).Process ID: 25767, Host: r6i4n5, Program: /glade2/scratch2/heavens/Isabel1_mapgenerator/cpl/cplMPT Version: SGI MPT 2.15  12/18/16 02:58:06"I'm trying to see if there are any useful hints in the tracebacks.Nicholas HeavensResearch Assistant Professor of Planetary ScienceHampton University
 
H

heavens

Member
The most progress I have been able to make is to use something like this in the run script:mpiexec_mpt -v    -np $NTASKS[1] omplace $EXEROOT/all/$COMPONENTS[1]  :    -np $NTASKS[2] omplace $EXEROOT/all/$COMPONENTS[2]  :    -np $NTASKS[3] omplace $EXEROOT/all/$COMPONENTS[3] :    -np $NTASKS[4] omplace $EXEROOT/all/$COMPONENTS[4] :    -np $NTASKS[5] omplace $EXEROOT/all/$COMPONENTS[5]   & The challenge is that you may encounter the error, "MPT ERROR: could not run executable. If this is a non-MPT application,you may need to set MPI_SHEPHERD=true." This is deceptive. It is not caused here by "a bad node" as you may find by searching the forums here. The issue is that MCT does not recognize the various CCSM3 executables as valid MPI programs. I have found that this can be partly solved by ensuring that the code is compiled by the MCT versions of the MPI compilers, but I still end up with segmentation faults:"MPT ERROR: Rank 0(g:0) received signal SIGSEGV(11).Process ID: 25767, Host: r6i4n5, Program: /glade2/scratch2/heavens/Isabel1_mapgenerator/cpl/cplMPT Version: SGI MPT 2.15  12/18/16 02:58:06"I'm trying to see if there are any useful hints in the tracebacks.Nicholas HeavensResearch Assistant Professor of Planetary ScienceHampton University
 
H

heavens

Member
Dear all,I have managed to successfully port CCSM3 to cheyenne. That is, I have built and run my very kludgey deep time version of CCSM3 for five days. I give no guarantee that any of this is going to work for you. However, I'll give you the key tips.1. Set up the simulation as a yellowstone experiment.2. Bypass the Machine proxies in the Buildnml scripts with your own standard proxy file (attached) in the top level of the case directory.3. Use the modules file attached. It is essential that CCSM3 executables are recognized as MPT applications. If they are not recognized as such, they will say so at run-time and give you a deceptive error message that you need to set MPT_SHEPHERD true. This is unnecessary.4. Use something like the attached run script. The mpiexec_mpt call is especially critical and should give you guidance.5. Copy the CCSM3 /models/utils onto your scratch. You enable its use by changing UTILROOT in env_run. You then need to edit mph.F such that you comment out "include mpif.h" and put "use mpi" BEFORE the implicit statement. Failure to do so will result in segmentation faults associated with the initiation of coupler communication (I'm attaching a typical example (Isabel1_mapgenerator.o5535144.txt ) as well as an example when things work as they should (Isabel1_mapgenerator.o5535470.txt ).)At some point soon, I will need to work out the long-term archiving scripts and attempt a longer integration. I will update you with my progress. Best regards to the CCSM3 community and the NCAR software engineers (who are right that I should have migrated this project to CESM two years ago, but I have my reasons...), Nicholas HeavensResearch Assistant Professor of Planetary Science'Hampton University   
 
H

heavens

Member
Dear all,I have managed to successfully port CCSM3 to cheyenne. That is, I have built and run my very kludgey deep time version of CCSM3 for five days. I give no guarantee that any of this is going to work for you. However, I'll give you the key tips.1. Set up the simulation as a yellowstone experiment.2. Bypass the Machine proxies in the Buildnml scripts with your own standard proxy file (attached) in the top level of the case directory.3. Use the modules file attached. It is essential that CCSM3 executables are recognized as MPT applications. If they are not recognized as such, they will say so at run-time and give you a deceptive error message that you need to set MPT_SHEPHERD true. This is unnecessary.4. Use something like the attached run script. The mpiexec_mpt call is especially critical and should give you guidance.5. Copy the CCSM3 /models/utils onto your scratch. You enable its use by changing UTILROOT in env_run. You then need to edit mph.F such that you comment out "include mpif.h" and put "use mpi" BEFORE the implicit statement. Failure to do so will result in segmentation faults associated with the initiation of coupler communication (I'm attaching a typical example (Isabel1_mapgenerator.o5535144.txt ) as well as an example when things work as they should (Isabel1_mapgenerator.o5535470.txt ).)At some point soon, I will need to work out the long-term archiving scripts and attempt a longer integration. I will update you with my progress. Best regards to the CCSM3 community and the NCAR software engineers (who are right that I should have migrated this project to CESM two years ago, but I have my reasons...), Nicholas HeavensResearch Assistant Professor of Planetary Science'Hampton University   
 
H

heavens

Member
Dear all,I have managed to successfully port CCSM3 to cheyenne. That is, I have built and run my very kludgey deep time version of CCSM3 for five days. I give no guarantee that any of this is going to work for you. However, I'll give you the key tips.1. Set up the simulation as a yellowstone experiment.2. Bypass the Machine proxies in the Buildnml scripts with your own standard proxy file (attached) in the top level of the case directory.3. Use the modules file attached. It is essential that CCSM3 executables are recognized as MPT applications. If they are not recognized as such, they will say so at run-time and give you a deceptive error message that you need to set MPT_SHEPHERD true. This is unnecessary.4. Use something like the attached run script. The mpiexec_mpt call is especially critical and should give you guidance.5. Copy the CCSM3 /models/utils onto your scratch. You enable its use by changing UTILROOT in env_run. You then need to edit mph.F such that you comment out "include mpif.h" and put "use mpi" BEFORE the implicit statement. Failure to do so will result in segmentation faults associated with the initiation of coupler communication (I'm attaching a typical example (Isabel1_mapgenerator.o5535144.txt ) as well as an example when things work as they should (Isabel1_mapgenerator.o5535470.txt ).)At some point soon, I will need to work out the long-term archiving scripts and attempt a longer integration. I will update you with my progress. Best regards to the CCSM3 community and the NCAR software engineers (who are right that I should have migrated this project to CESM two years ago, but I have my reasons...), Nicholas HeavensResearch Assistant Professor of Planetary Science'Hampton University   
 
H

heavens

Member
Dear all,Here are the latest updates on maintaining the CCSM3 port to cheyenne.1. The entire MCT library in the original CCSM3 codebase must be bypassed and updated to the latest version, which is available here: https://github.com/MCSclimate/MCT.git2. Some of the standard input files have been moved. I have been able to find them in: https://svn-ccsm-inputdata.cgd.ucar.edu/ and ftp://128.117.23.220/cesm/inputdata3. The latest version of the compilers are much stricter with catching memory leaks during run time. 4. If you use any of the code attached above, take note that some software versions have been updated.5. In env_run, you should add a further line to run the program successfully:setenv CESMDATAROOT     /glade/p/cesm/csegI am intending to stop all use of CCSM3 at the end of August. Best regards,Nicholas HeavensResearch Assistant Professor of Planetary ScienceHampton University    
 
H

heavens

Member
Dear all,Here are the latest updates on maintaining the CCSM3 port to cheyenne.1. The entire MCT library in the original CCSM3 codebase must be bypassed and updated to the latest version, which is available here: https://github.com/MCSclimate/MCT.git2. Some of the standard input files have been moved. I have been able to find them in: https://svn-ccsm-inputdata.cgd.ucar.edu/ and ftp://128.117.23.220/cesm/inputdata3. The latest version of the compilers are much stricter with catching memory leaks during run time. 4. If you use any of the code attached above, take note that some software versions have been updated.5. In env_run, you should add a further line to run the program successfully:setenv CESMDATAROOT     /glade/p/cesm/csegI am intending to stop all use of CCSM3 at the end of August. Best regards,Nicholas HeavensResearch Assistant Professor of Planetary ScienceHampton University    
 
H

heavens

Member
Dear all,Here are the latest updates on maintaining the CCSM3 port to cheyenne.1. The entire MCT library in the original CCSM3 codebase must be bypassed and updated to the latest version, which is available here: https://github.com/MCSclimate/MCT.git2. Some of the standard input files have been moved. I have been able to find them in: https://svn-ccsm-inputdata.cgd.ucar.edu/ and ftp://128.117.23.220/cesm/inputdata3. The latest version of the compilers are much stricter with catching memory leaks during run time. 4. If you use any of the code attached above, take note that some software versions have been updated.5. In env_run, you should add a further line to run the program successfully:setenv CESMDATAROOT     /glade/p/cesm/csegI am intending to stop all use of CCSM3 at the end of August. Best regards,Nicholas HeavensResearch Assistant Professor of Planetary ScienceHampton University    
 
You must log in or register to reply here.
Top