You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
GPU card recommendation
- Thread starter goudeuk
- Start date
Hi there,
CESM is not set up to run on GPUs, but small configurations can be set up to run on laptop CPUs. I believe the most common way to set up the model locally is to use Docker and download a container... I've asked Brian Dobbins to comment on this thread because he is the one who has set everything up, but you might find the information you are looking for in this thread.
What are you interested in running? (What component set, what resolution, and how long of a simulation?)
~Mike
CESM is not set up to run on GPUs, but small configurations can be set up to run on laptop CPUs. I believe the most common way to set up the model locally is to use Docker and download a container... I've asked Brian Dobbins to comment on this thread because he is the one who has set everything up, but you might find the information you are looking for in this thread.
What are you interested in running? (What component set, what resolution, and how long of a simulation?)
~Mike
That pretty much covers it, but yes, if you're interested in running on your own laptop, you can try the containerized versions - with or without Jupyter, and while Docker is the easiest, it also works with Singularity. But you'll be limited in what you can run by how much memory you have, and how fast it goes by your CPU type -- there isn't any support yet for GPUs. Note that the container should work out of the box for Windows / Mac, and on some Linux distros, but others may need a tweak - that'll get fixed in the not-too-distant future, hopefully.
If you have trouble, let me know, and good luck!
If you have trouble, let me know, and good luck!
Hello
Apologies for my late reply. Please allow me to give you a bit more information. I recently took over the computing needs of another department (earth and planetary sciences) and I had a request for a high spec machine that can run CESM on it. The academic mentioned that CESM can run on both CPU and GPU resources and that's why I asked about GPU in my first post. However, you mentioned that it can only run on CPU so I will let him know that.
Mike, please see the answers to your questions.
Thank you for your advise.
Apologies for my late reply. Please allow me to give you a bit more information. I recently took over the computing needs of another department (earth and planetary sciences) and I had a request for a high spec machine that can run CESM on it. The academic mentioned that CESM can run on both CPU and GPU resources and that's why I asked about GPU in my first post. However, you mentioned that it can only run on CPU so I will let him know that.
Mike, please see the answers to your questions.
- ESM compset: He will be using a custom CESM2 CLM compset with ExoCAM: starting 1850, simplified CLM5 physics, but with ExoCAM instead of CAM. Aquaplanet or landplanet mask, not SP (satellite phenology). He said that this compset is generally less computationally intensive as the land physics are simplified, some modules are turned off (urban heat island, evapotranspiration etc.), and the atmosphere generalised for exoplanet studies.
- Resolution: either 1 or 2 degree grid (simulations of 2 degree grid with 41 vertical levels for my last paper)
- Simulation length: 80-90 model years. I believe this is around 12 ‘wallclock’ hours.
Thank you for your advise.
I'd guess the memory is sufficient, but the processing is almost certainly insufficient to reach the rates you want for a coupled run. I don't have any direct experience with ExoCAM, but the recent paper about it said it takes ~180 processor hours per model year on an Intel Haswell for a 4x5-degree run. So on a 16-core node, that's 11.25 hours per model year, or just over two simulated years per day (SYPD). It's probably more than ~4x slower to go to 2-degree, and another ~4x or so to go to 1-degree. Even accounting for faster processors and more cores, I think it's incredibly unlikely you get close to 160 - 180 SYPD (80-90 years in 12 hours) in a coupled ExoCAM + CLM configuration, and certainly not at 1-degree.
To give some numbers I do have from standard CESM, a CAM4 aquaplanet case at 2-degrees and 26 vertical levels gets ~28 SYPD on our 36-core Intel Broadwell nodes and I can hit ~84 SYPD on a modern, 96-core AMD Zen3 node. Going to 1-degree, it drops to ~5.5 SYPD on our 36-core Broadwell system, and I haven't run it on the AMD node, but would expect a similar scaling.
The land model in these cases is usually quite fast - on an F2000climo case (which has CLM5 w/ SP), the land model achieves ~80 SYPD on 2-degrees, and ~23 SYPD on 1-degree cases on our system. But again, coupling with the atmosphere will slow that down. In case you're curious, backing up my guess about the memory, in these F2000climo runs, using CAM6 at 32 vertical levels, the memory needed is approximately 36GB (1-degree) and 21GB (2-degree). With more vertical levels you'll need slightly more, but 64GB is still likely sufficient.
In short, for coupled ExoCAM + CLM runs, your memory is fine but you likely won't hit the performance goals you want, especially with a single i9. For simplified land-only runs, it likely could reach those rates on a decent server/workstation. If you guys can provide timing files from runs, maybe I could offer a bit more insight, but does that help at all in the meantime?
To give some numbers I do have from standard CESM, a CAM4 aquaplanet case at 2-degrees and 26 vertical levels gets ~28 SYPD on our 36-core Intel Broadwell nodes and I can hit ~84 SYPD on a modern, 96-core AMD Zen3 node. Going to 1-degree, it drops to ~5.5 SYPD on our 36-core Broadwell system, and I haven't run it on the AMD node, but would expect a similar scaling.
The land model in these cases is usually quite fast - on an F2000climo case (which has CLM5 w/ SP), the land model achieves ~80 SYPD on 2-degrees, and ~23 SYPD on 1-degree cases on our system. But again, coupling with the atmosphere will slow that down. In case you're curious, backing up my guess about the memory, in these F2000climo runs, using CAM6 at 32 vertical levels, the memory needed is approximately 36GB (1-degree) and 21GB (2-degree). With more vertical levels you'll need slightly more, but 64GB is still likely sufficient.
In short, for coupled ExoCAM + CLM runs, your memory is fine but you likely won't hit the performance goals you want, especially with a single i9. For simplified land-only runs, it likely could reach those rates on a decent server/workstation. If you guys can provide timing files from runs, maybe I could offer a bit more insight, but does that help at all in the meantime?