Introducing a system instruction in config_batch.xml

[LluisFB]

Dear all,

I am currently porting CESM 2.2.2 into the HPC in my institution. I am new in CESM, but with experience in models and model development (e.g. WRF-ARW).

I am down writing all the steps in a WIKI page CESM Install hydra.

I succeed building the case --res f19_g17 --compset B1850 , but I failed in submitting it.

I got error messages of the type:

new_task: jobid=43629.hydra node=0 task=1
new_task: jobid=43629.hydra node=0 task=2
ERROR: RUN FAIL: Command '/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpirun  -np 768  /home/lluis.fita/cesm/scratch/b.day1.0/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/lluis.fita/cesm/scratch/b.day1.0/run/cesm.log.43629.hydra.240516-1705

I am guessing that this is because in my HPC, called hydra does not have module. Instead we set-up the compilation environment via a shell script created by our IT team called /opt/load-libs.sh. For example, if we want the intel environment we execute:

$ source /opt/load-libs.sh 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

I am wondering if there is a way to systematically introduce it in the config_batch.xml a shell command (source /opt/load-libs.sh 1) in order to be included and executed in all the PBS jobs.

Many thanks in advance,

Lluís

P.S.: Sorry, I can not attach my xml, the changes are found in the WIKI

 

[jedwards]

Hi Lluís,

First I highly recommend cesm2_3 instead of cesm2.2 if at all possible. We support several methods of loading an environment for cesm, your home grown method is not one of them. You could use the xml variable PRERUN_SCRIPT to run your source command prior to each run.

 

[LluisFB]

Thank you for your answer @jedwards

I incorporated the variable prerun_script into the file in the case directory env_run.xml, see more details (searching for it here), as follows:

      <entry id="PRERUN_SCRIPT">
      <type>char</type>
      <desc>External script to be run before model completion</desc>
      <values>
        <value>source /opt/load-libs.sh 1</value>
      </values>

But when I executed it, I got the error message:

$ ./case.submit
$ cat run.b.day1.0.o43637
ERROR: External script source /opt/load-libs.sh 1 not found

I tried different options (see more details in the [CESMInstall WIKI CESM - hydra - Wikicima WIKI]), but nothing worked. The last attempt was to define one script (/home/lluis.fita/intel_env.sh) with the definition of the different paths as follows:

#!/bin/sh
export PATH="/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin:$PATH"
export PATH="/opt/netcdf/netcdf-4/intel/2021.4.0/bin:$PATH"
export PATH="/opt/hdf5/hdf5-1.10.5/intel/2021.4.0/bin:$PATH"
export PATH="/opt/jasper/jasper-version-2.0.33/intel/2021.4.0/bin:$PATH"

export LD_LIBRARY_PATH=/opt/jasper/jasper-version-2.0.33/intel/2021.4.0/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/zlib/zlib-1.2.11/intel/2021.4.0/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/hdf5/hdf5-1.10.5/intel/2021.4.0/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/netcdf/netcdf-4/intel/2021.4.0/lib:$LD_LIBRARY_PATH

But I got the error message:

$ cat run.b.day1.0.o43651
   Running /home/lluis.fita/intel_env.sh 
/bin/sh: 1: Syntax error: Bad fd number
ERROR: /home/lluis.fita/intel_env.sh /share/cesm/expriments/b.day1.0 >& /home/lluis.fita/cesm/scratch/b.day1.0/run/prerun.external.log.43651.hydra.240517-171306  FAILED, cat /home/lluis.fita/cesm/scratch/b.day1.0/run/prerun.external.log.43651.hydra.240517-171306

I tried with bash and csh, but nothing worked. Also I noticed that the file prerun.external.log.43651.hydra.240517-171306 is not being created. In my HPC

$ ls -lrta /bin/sh
lrwxrwxrwx 1 root root 4 Oct 25  2021 /bin/sh -> dash

At least, now the script is trying to be executed, but I am missing something (does CESM uses a preferred $SHELL?)

Thanks,

Lluís