Derecho Lmod Error: Unknown module "craype" blocking case.setup in release-cesm2.1.5 and cesm2.2.2 (F2000climo)

[usmanummuqulthum]

I am attempting to spin up a standalone F2000climo case on NCAR's Derecho supercomputer using a clean checkout of release-cesm2.1.5 and cesm2.2.2 , but the invocation of case.setup is completely blocked during the machine-specific environment configuration phase.

Issue Summary​

CIME attempts to dynamically load the software stack via Lmod wrappers but fails because it is explicitly requesting the module "craype", which is currently unrecognized or deprecated in my active environment. Because the module framework throws a fatal error, the environment handshake fails, aborting case.setup and preventing the compilation process (case.build) from proceeding.

Case Details​

• Model Version: release-cesm2.1.5
• Machine: Derecho (crhtc45.hpc.ucar.edu)
• Compset: F2000climo (Longname: 2000_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_MOSART_CISM2%NOEVOLVE_SWAV)
• Grid Resolution: f09_g17

 

[fischer]

Hi Usman,

I did a clean checkout of release-2.1.5 and ran
./create_newcase --compset F2000climo --res f09_g17 --case /glade/derecho/scratch/fischer/F2000_2.1.5_BB --run-unsupported
./case.setup
./case.build

I was able to build the model without any issues. Can you show me the create_newcase command you used, and attach the software_environment.txt in your case directory. Or point me to your case directory.

Thanks
Chris

 

[usmanummuqulthum]

Hi Chris,

Thanks for looking into this! Regarding the previous build failure, I realized that I mistakenly tried to run that setup on Casper instead of Derecho. Once I switched over to Derecho, that specific issue was resolved and the build went through successfully.

However, I am now running into a different issue on a new case where I changed the compset to a BSSP126CMIP. To help troubleshoot, I have attached my case run logs and other diagnostic documents. You can also find my active case directory at: /glade/u/home/usman/GGW_cases/SSP_SCENERIOS/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_Daily_Run_215
Latest Batch Run Log in that directory: run.cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_Daily_Run_215.o6601236

Scratch Run Directory (for cesm.log and component logs):/glade/derecho/scratch/usman/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_Daily_Run_215/run/

Please let me know if you need me to send over any additional logs or configuration files.

Best,

Usman

 

[fischer]

Hi Usman,

It looks like the number of tasks being used for WW3 is to high. But I see that you set the number of tasks for WW3 to 32. I'm not sure exactly how the ocean and ww3 interact, but I think ww3 might be seeing the number of tasks for the ocn. Also, you didn't change all of the ROOTPE, so this run is using 18 nodes instead of the 10 nodes you expected. What you should do is use the default processors layout for this run. Remove any changes to NTASKS, NTHRDS, and ROOTPE from your script. This will give you a run that uses 9 nodes.

NTASKS: ['CPL:1024', 'ATM:1024', 'LND:512', 'ICE:512', 'OCN:128', 'ROF:512', 'GLC:1024', 'WAV:256', 'ESP:1']
ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:512', 'OCN:1024', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

Chris

 

[usmanummuqulthum]

Hii Chris, Thank you for your earlier response, after resolving the environment module loading issues, the case successfully configured and built, but immediately crashed during runtime execution. The model failed during the wave initialization phase because the default configuration allocated 1,280 processors globally across all components. WAVEWATCH III aborts if its physical grid size (which is only 600 spectral points in this setup) is smaller than the number of processors assigned to it. I have attached my .csh setup script for reference, and the full crash log directory paths on Derecho are listed below.

Log Directory Path: /glade/derecho/scratch/usman/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_001_Run_215/run/

Attached Configuration Files & Primary Logs:

• SSP_02.csh (Attached)
• sofware.environrment.txt (Attached)

 

[fischer]

Hi Usman,

I didn't get a chance to make a run to confirm this. But the updates to the user_nl_clm should be removed. I think the use_init_interp isn't needed for a B compset. Also, setting hist_nhtfrq and hist_mfilt is also possible causing conflicts in the clm namelist file. So try removing these 3 lines from your script.

echo "hist_nhtfrq = 0, -24" >> user_nl_clm
echo "hist_mfilt = 1,30" >> user_nl_clm
echo "use_init_interp = .true." >> user_nl_clm

Chris

 

[fischer]

Hi Usman,

Looking at your run.cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_001_Run_215.o6649413 file in your case directory, I see that you exceeded the wallclock limit. Try setting the wallclock to 04:00:00. That should give you plenty of wall time.

Chris

 

[usmanummuqulthum]

I am running a transient SSP1-2.6 simulation (release-cesm2.1.5) on Derecho using 1152 cores. The model consistently crashes about 1.5 minutes after it starts running.

The high-level error log shows a general failure when launching the model:

Plaintext

ERROR: RUN FAIL: Command 'mpiexec ... cesm.exe' failed<br>
When I looked deeper into the cesm.log file, it looks like all the MPI processes are getting stuck or killed (SIGTERM) during a startup handshake loop inside the CIME driver. Specifically, it seems to fail at an MPI_Bcast call.

Here is the bottom of the stack trace:

Plaintext

shr_mpi_mod_mp_sh 568 shr_mpi_mod.F90<br>seq_infodata_mod_ 2525 seq_infodata_mod.F90<br>component_mod_mp_ 285 component_mod.F90<br>cime_comp_mod_mp_ 1237 cime_comp_mod.F90<br>MAIN__ 114 cime_driver.F90<br>...<br>forrtl: error (78): process killed (SIGTERM)<br>
Case Details:

• Model version: release-cesm2.1.5
• Machine: NCAR Derecho (Intel compiler / cray-mpich)
• Compset: BSSP126cmip6
• Resolution: f09_g17

What I have checked so far:

1. I checked the PBS batch logs, but I didn't see any explicit out-of-memory (OOM) errors.
2. I ran ./check_input_data on a Derecho login node, and it successfully found and copied the reference case restart pointer files (rpointer.*) into my scratch directory.

Does this specific crash in seq_infodata_mod.F90 look familiar to anyone? I would appreciate any advice on whether this is an environment problem or an issue with how the hybrid/branch run is starting up.

Thanks!

 

[usmanummuqulthum]

Hello Chris, thank you for your help so far and here the relevant doc and file path.

Log Directory Path: ~/ glade/derecho/GGW_cases/SSP_SCENERIOS/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_002_Run_215/
/glade/derecho/scratch/usman/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_002_Run_215/run/
Attached Configuration Files & Primary Logs:

• SSP_03.csh (Attached)
• sofware.environrment.txt (Attached)

 

[fischer]

Hi Usman,

It looks like there's an issue with your custom fsurdat file that you're using. I'd suggest starting a new thread in the Land forum to get help with diagnosing your file.

The error message is in your lnd.log.6672098.desched1.260709-052020 file in your run directory.

ERROR: GLACIER_REGION NOT on surfdata file
ERROR in glcBehaviorMod.F90 at line 662

Chris