Welcome to the new DiscussCESM forum!
We are still working on the website migration, so you may experience downtime during this process.

Existing users, please reset your password before logging in here: https://xenforo.cgd.ucar.edu/cesm/index.php?lost-password/

Containerized CESM for laptops/workstations (Windows/Mac/Linux)

dobbins

Brian Dobbins
CSEG and Liaisons
Staff member
Have you been doing this primarily with a Dockerfile to build the container or some other method? If the former, maybe we can get it under version control for easier collaboration? I was originally planning on doing a from-scratch Singularity recipe file to build my container, and then I just happened to stumble upon this at an opportune time :)

We've got a Github repo here, and collaboration sounds great:
ESCOMP/ESCOMP-Containers

Note that right now, you'll see the CESM directory has quite a few changes from a 'base' CESM install - this is because we wanted to get something out to experiment with, and CESM 2.2 was going through the release process. In the near future, most of those changes -generally focused on providing a 'container' machine config- will be included in CIME, and thus be unnecessary. In short, it'll get a lot less ugly soon, but still, that repo should let you play around.

The plan is to have Singularity recipes in there at some point, too.

We also have the start of a tutorial repo (for Jupyter Notebooks), but that's probably of less interest to you, I imagine.

What kind of passthrough? I have experience using GPU passthrough on Singularity for Nvidia GPUs (tensorflow/deep learning), which typically works out of the box with the --nv flag (although the user might need to be in a "video" group for it to work). If memory serves, that worked on the RHEL7 kernel (3.10), but I haven't tested on an earlier kernel. I don't have experience with radeon or other ASIC passthrough, so I'd be less helpful there. The Singularity team does appear to be relatively responsive though, so if you pass along those details I don't mind looking into it.

Ah, I meant pass-through to the underlying network - the MPICH ABI Compatibility Initiative lets you replace one compatible MPI implementation with another at runtime, and Cray machines, for example, can do this automagically with Shifter to use the native CrayMPI runtime. I think when I saw this, they were using Singularity containers, but Shifter is the key thing. So, for example, I can compile a CESM case in a container (with MPICH, no knowledge of a high-speed network like a Cray Aries or even Infiniband), and run it on that Cray, and have the host-level network used. Here's a neat paper showing some of this from Blue Waters:

Container solutions for HPC Systems: A Case Study of Using Shifter on Blue Waters

On Cheyenne (our system), we have SGI MPT, and there's a compatibility mode that also allows for this.. but required a little bit of work to get going, unlike Shifter's automatic use of it. I haven't looked into this in a while, but it's a really great thing for HPC-like use. And on the 'to do' list to check up on again.

Cheers,
- Brian
 

smoggy

smogger
New Member
@dobbins How can I look at/edit the CESM code that's inside the Docker image that is being used in the Jupyter notebook? Are the files accessible?
 

dobbins

Brian Dobbins
CSEG and Liaisons
Staff member
@dobbins How can I look at/edit the CESM code that's inside the Docker image that is being used in the Jupyter notebook? Are the files accessible?

Good question - if you just want to look, you can change directory to $CESMROOT ( /opt/ncar/cesm2 ) and poke around. By default, we make them owned by 'root', so you can't easily overwrite them. But you can always copy that whole tree to your home directory, too, to have an editable version:

cp -r /opt/ncar/cesm2 ~/cesm2

(And you can also just 'sudo' to root, without a password, inside the container!)

If you're looking to make source-code changes to a case you're building, you generally identify the file you want changed in the main source tree ( in /opt/ncar/cesm2/components/cam, for example ), then put that file in your case's SourceMods/src.cam directory, again using CAM as the example component. We're working on having tutorials on this sort of thing in the future built into the container, but in the meantime, here's the (non-container) tutorial slides / videos that might help, too:


Hope that helps, and let me know if you have more questions! I'm hoping we can develop more tutorials soon. :-)
 

smoggy

smogger
New Member
Hope that helps, and let me know if you have more questions! I'm hoping we can develop more tutorials soon. :-)

@dobbins Thanks. I read the User Guide and figured out how to use the "name_ul_***" files to change variables.

Regarding the output data in the Jupyter tutorial, I see that you've plotted temperature using QuickView('quickstart_case', 'T') and I see a map of Earth with what I'm assume is the temperature plotted. Looking at the definition of the QuickView function, I see "supported_4D = ( 'T', 'U', 'V', )". However 'U' and 'V' produce errors when I try them in the QuickView function.

  1. What are U and V? Why don't they work with QuickView?
  2. What other output variables exist for this QPC4 compset besides temperature? How can I find out the output variables for any compset?
  3. How can I export these output files (*.nc) out of the Docker image and onto my real computing environment?
 

dobbins

Brian Dobbins
CSEG and Liaisons
Staff member
@dobbins Thanks. I read the User Guide and figured out how to use the "name_ul_***" files to change variables.

Regarding the output data in the Jupyter tutorial, I see that you've plotted temperature using QuickView('quickstart_case', 'T') and I see a map of Earth with what I'm assume is the temperature plotted. Looking at the definition of the QuickView function, I see "supported_4D = ( 'T', 'U', 'V', )". However 'U' and 'V' produce errors when I try them in the QuickView function.

  1. What are U and V? Why don't they work with QuickView?
  2. What other output variables exist for this QPC4 compset besides temperature? How can I find out the output variables for any compset?
  3. How can I export these output files (*.nc) out of the Docker image and onto my real computing environment?

Hi @smoggy,

A thousand apologies -- I didn't see this, and was off on some other projects. I imagine you've solved your problems by now? If not, the short answer is the 'QuickView' function isn't very robust yet - it was really just a preview of the sort of thing we can do. The hope is to make it more robust before an actual release, or perhaps replace it with some standardized CESM diagnostics.

I'm surprised you're having errors with U and V, though, at least with the quick start case. That's working fine for me, provided you ran the full month (which would be needed for the 'T' variable too). U and V are the zonal and meridional winds, respectively, so they should be in that file. As for other variables, there are quite a few - you can do an 'ncdump' on the output file like this and see a lot more, including description and dimensionality:

ncdump -h ~/archive/quickstart_case/atm/hist/quickstart_case.cam.h0.0001-01.nc

Finally, to 'export' a file outside of the Docker image, you've got a few options - the easiest is simply to have mounted a directory into the image when you run it (the '-v' option to 'docker run'). If you didn't do that, you can likely still do a 'sftp' or 'rsync' to a different Linux system, since you have networking available in the container. (You can also 'save' a modified container if you didn't mount something and don't have a place to transfer things to, which adds a layer to the container .. but that's less ideal.)

Does that help? Apologies again. I don't know why I didn't get notified of an update to this thread!

Cheers,
- Brian
 

smoggy

smogger
New Member
Hi @smoggy,

A thousand apologies -- I didn't see this, and was off on some other projects. I imagine you've solved your problems by now? If not, the short answer is the 'QuickView' function isn't very robust yet - it was really just a preview of the sort of thing we can do. The hope is to make it more robust before an actual release, or perhaps replace it with some standardized CESM diagnostics.

I'm surprised you're having errors with U and V, though, at least with the quick start case. That's working fine for me, provided you ran the full month (which would be needed for the 'T' variable too). U and V are the zonal and meridional winds, respectively, so they should be in that file. As for other variables, there are quite a few - you can do an 'ncdump' on the output file like this and see a lot more, including description and dimensionality:



Finally, to 'export' a file outside of the Docker image, you've got a few options - the easiest is simply to have mounted a directory into the image when you run it (the '-v' option to 'docker run'). If you didn't do that, you can likely still do a 'sftp' or 'rsync' to a different Linux system, since you have networking available in the container. (You can also 'save' a modified container if you didn't mount something and don't have a place to transfer things to, which adds a layer to the container .. but that's less ideal.)

Does that help? Apologies again. I don't know why I didn't get notified of an update to this thread!

Cheers,
- Brian
@dobbins Thanks, excuse my late reply - holidays and all that.

Regarding exporting, I used:
docker cp NAME:/home/user/archive/quickstart_case/atm/hist ./Documents
where NAME is the name of the currently-running Docker container

I saw on the spreadsheet that fully-coupled b1850c4_tutorial should be available for this Docker setup. However, it fails before the first output data is saved. I look in the logfile and I see:

"Program received signal SIGBUS: Access to an undefined portion of a memory object."

Do you know what this means or how to solve this error?

Regarding the ncdump of the quickcase QPC4 compset, I saw the long description of variables as you stated. 2 questions:
  1. It appears that elevation is measured in pressure instead of meters?
  2. I don't see a variable for sealevel. How do I calculate this? Or is this not available because this is an atmosphere-only compset?
 
Top