Undefined env var

[Yonas]

Hello everyone, I installed & used the containerized CESM, specifically the cesm-2.2:tech preview, on my laptop. Now I'd like to install & use the same version on my organization's HPC, however, I'm having problem with the "./case.setup" step, which says ERROR: Undefined env var 'CESMDATAROOT'.
These are the steps I took:

Step 1: mkdir -p /home/sysusr/projects/cesm
Step 2: export CESMDATAROOT=/home/sysusr/projects/cesm
Step 3: Create 'inputdata', 'archive' and 'cases' directory
mkdir -p $CESMDATAROOT/inputdata
mkdir -p $CESMDATAROOT/archive
mkdir -p $CESMDATAROOT/cases
Step 4: Install containerized version of CESM:
docker run -i --rm -v ${CESMDATAROOT}/inputdata:/home/user/inputdata -v ${CESMDATAROOT}/archive:/home/user/archive -v ${CESMDATAROOT}/cases:/home/user/cases \
-e CESMDATAROOT=${CESMDATAROOT} \
-p 8888:8888 \
-t escomp/cesm-2.2:tech_preview
Step 5: Create a CESM case:
cd /opt/ncar/cesm2/cime/scripts/
./create_newcase --case ${HOME}/cases/cQPC4 --compset QPC4 --res f45_f45_mg37 --machine container --run-unsupported
Step 6: Setting up the case
cd ${HOME}/cases/cQPC4
sudo ./case.setup

Sincerely,

 

[dobbins]

Hi Yonas,

This seems to happen because of the use of 'sudo' when you're running './case.setup', which isn't necessary since you're in your (mounted) case directory anyway. I'll look into why sudo is resetting the exported environment variable, but perhaps that's a workaround for now?

A few other minor things:
1) Since you're not using the version with Jupyter Lab, you can leave off the '-p 8888:8888' bit of your 'docker run' command. That's just for passing local web requests to the Jupyter server in cesm-lab-2.2:tech_preview, and isn't needed for the 'standard' (command-line) version.

2) Depending on what you want, you can change the '-i' flag to 'docker run' to '-it', which will give you full interactive terminal use; eg, you can use text editors, get tab-completion, etc. If you're just scripting this, that's not necessary, but I thought I'd mention it.

3) You don't actually need to switch to the '/opt/ncar/cesm2/cime/scripts' directory in your script; the container automatically adds that to your path, so you can change your 'create_newcase' line to just be:

create_newcase --case ${HOME}/cases/cQPC4 --compset QPC4 --res f45_f45_mg37 --machine container --run-unsupported

(Technically, you don't even need the ${HOME} in there, but having it be explicit is sometimes nice, vs. assuming you're in the home directory.)

Finally, we are also planning on having a 'cluster' version of the containers, although the focus there tends to be more on using Singularity than Docker since most HPC resources don't allow the latter. It includes the Intel compilers, for better performance, and allows multi-node runs. For f45 QPC4 cases it probably doesn't matter, but if you want to scale out it helps. (Note that the current version you're using can run on multiple processors within a node, but setting it up for multi-node runs is tricky and non-performant on IB networks, since it isn't configured for those.) That version is a bit behind due to other focuses right now, but if it's of interest to you, let me know and I'll keep you updated as we get into some external testing.

Hope that helps, and let me know if you have more questions.

- Brian

 

[Yonas]

Hi Yonas,

This seems to happen because of the use of 'sudo' when you're running './case.setup', which isn't necessary since you're in your (mounted) case directory anyway. I'll look into why sudo is resetting the exported environment variable, but perhaps that's a workaround for now?

A few other minor things:
1) Since you're not using the version with Jupyter Lab, you can leave off the '-p 8888:8888' bit of your 'docker run' command. That's just for passing local web requests to the Jupyter server in cesm-lab-2.2:tech_preview, and isn't needed for the 'standard' (command-line) version.

2) Depending on what you want, you can change the '-i' flag to 'docker run' to '-it', which will give you full interactive terminal use; eg, you can use text editors, get tab-completion, etc. If you're just scripting this, that's not necessary, but I thought I'd mention it.

3) You don't actually need to switch to the '/opt/ncar/cesm2/cime/scripts' directory in your script; the container automatically adds that to your path, so you can change your 'create_newcase' line to just be:

create_newcase --case ${HOME}/cases/cQPC4 --compset QPC4 --res f45_f45_mg37 --machine container --run-unsupported

(Technically, you don't even need the ${HOME} in there, but having it be explicit is sometimes nice, vs. assuming you're in the home directory.)

Finally, we are also planning on having a 'cluster' version of the containers, although the focus there tends to be more on using Singularity than Docker since most HPC resources don't allow the latter. It includes the Intel compilers, for better performance, and allows multi-node runs. For f45 QPC4 cases it probably doesn't matter, but if you want to scale out it helps. (Note that the current version you're using can run on multiple processors within a node, but setting it up for multi-node runs is tricky and non-performant on IB networks, since it isn't configured for those.) That version is a bit behind due to other focuses right now, but if it's of interest to you, let me know and I'll keep you updated as we get into some external testing.

Hope that helps, and let me know if you have more questions.

- Brian

Hello, Brian.

Thank you for your assistance!!!; I followed your advice and it now works as it should within the node.

I'd appreciate it if you could help me figure out how to use all of the HPC computing nodes from the mounted container, if that's even possible, which I don't think it is. correct? It will be beneficial if you have the cluster version of the container.

If that is not the case, my only option is to install CESM and all of its dependencies from the source that I was trying to avoid. As you know, the CESM installation procedures are somewhat complicated, that's why I am using your container. In that case, could you please advise me on how can I install CESM using the same compilers, & libraries that you used to build the Docker image? I'd like to replicate the steps you took. Thanks in Advance!

Yonas