Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

case.submit mpiexec ERROR

K

kferns

Kenneth F
New Member
Hello!
I'm having some trouble running my case and am hoping someone on the forum can help me.

I am using a virtual Linux machine running Ubuntu 22.04. I followed this guide to set the machine up and port CESM. However I chose to download a beta version as recommended by the visualCaseGen quickstart. Sp I've done all that and fair bit of troubleshooting already to get to the point where I can build a simple test case successfully. But it throws an error when I submit it and I can work out what's wrong.

Shell commands (per the aforementioned guide)-
Code: 
./create_newcase --case ~/cesm_projects/scratch/quickstart_case --compset QPC4 --res f45_f45_mg37 --run-unsupported
./xmlchange STOP_OPTION=ndays,STOP_N=3
./case.setup
./case.build
./case.submit

Here is is output after ./case.submit (no errors before that, as far as I know)-

Code: 
(visualCaseGen) kferns@kferns-VirtualBox:~/cesm_projects/scratch/quickstart_case$ ./case.submit
  2024-06-22 16:31:24 atm
Create namelist for component cam
   Calling /home/kferns/cesm2_3_beta17_gui/components/cam//cime_config/buildnml
CAM namelist copy: file1 /home/kferns/cesm_projects/scratch/quickstart_case/Buildconf/camconf/atm_in file2 /home/kferns/cesm_projects/scratch/quickstart_case/run/atm_in
  2024-06-22 16:31:24 lnd
Create namelist for component slnd
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/slnd/cime_config/buildnml
  2024-06-22 16:31:24 ice
Create namelist for component sice
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sice/cime_config/buildnml
  2024-06-22 16:31:24 ocn
Create namelist for component docn
   Calling /home/kferns/cesm2_3_beta17_gui/components/cdeps/docn/cime_config/buildnml
docn_mode is sst_aquap3
  2024-06-22 16:31:24 rof
Create namelist for component srof
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/srof/cime_config/buildnml
  2024-06-22 16:31:24 glc
Create namelist for component sglc
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2024-06-22 16:31:24 wav
Create namelist for component swav
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2024-06-22 16:31:24 esp
Create namelist for component sesp
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2024-06-22 16:31:24 cpl
Create namelist for component drv
   Calling /home/kferns/cesm2_3_beta17_gui/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'OCN']
Checking that inputdata is available as part of case submission
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Check case OK
submit_jobs case.run
Submit job case.run
Starting job script case.run
Generating namelists for /home/kferns/cesm_projects/scratch/quickstart_case
  2024-06-22 16:31:25 atm
Create namelist for component cam
   Calling /home/kferns/cesm2_3_beta17_gui/components/cam//cime_config/buildnml
CAM namelist copy: file1 /home/kferns/cesm_projects/scratch/quickstart_case/Buildconf/camconf/atm_in file2 /home/kferns/cesm_projects/scratch/quickstart_case/run/atm_in
  2024-06-22 16:31:25 lnd
Create namelist for component slnd
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/slnd/cime_config/buildnml
  2024-06-22 16:31:25 ice
Create namelist for component sice
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sice/cime_config/buildnml
  2024-06-22 16:31:25 ocn
Create namelist for component docn
   Calling /home/kferns/cesm2_3_beta17_gui/components/cdeps/docn/cime_config/buildnml
docn_mode is sst_aquap3
  2024-06-22 16:31:25 rof
Create namelist for component srof
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/srof/cime_config/buildnml
  2024-06-22 16:31:25 glc
Create namelist for component sglc
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2024-06-22 16:31:25 wav
Create namelist for component swav
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2024-06-22 16:31:25 esp
Create namelist for component sesp
   Calling /home/kferns/cesm2_3_beta17_gui/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2024-06-22 16:31:25 cpl
Create namelist for component drv
   Calling /home/kferns/cesm2_3_beta17_gui/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'OCN']
-------------------------------------------------------------------------
 - Prestage required restarts into /home/kferns/cesm_projects/scratch/quickstart_case/run
 - Case input data directory (DIN_LOC_ROOT) is /home/kferns/cesm_projects/inputdata
 - Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
run command is mpiexec  -np 4 --prepend-rank /home/kferns/cesm_projects/scratch/quickstart_case/bld/cesm.exe   >> cesm.log.$LID 2>&1
Exception from case_run: ERROR: RUN FAIL: Command 'mpiexec  -np 4 --prepend-rank /home/kferns/cesm_projects/scratch/quickstart_case/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/kferns/cesm_projects/scratch/quickstart_case/run/cesm.log.240622-163125
Submit job case.st_archive
Starting job script case.st_archive
st_archive starting
moving /home/kferns/cesm_projects/scratch/quickstart_case/run/cesm.log.240622-163125 to /home/kferns/cesm_projects/archive/case/logs/cesm.log.240622-163125
moving /home/kferns/cesm_projects/scratch/quickstart_case/run/drv.log.240622-163125 to /home/kferns/cesm_projects/archive/case/logs/drv.log.240622-163125
Cannot find a quickstart_case.cpl*.r.*.nc file in directory /home/kferns/cesm_projects/scratch/quickstart_case/run
Archiving history files for cam (atm)
Archiving history files for docn (ocn)
Archiving history files for drv (cpl)
Archiving history files for dart (esp)
st_archive completed
Submitted job case.run with id None
Submitted job case.st_archive with id None

I've attached the cesm.log and drv.log but didn't find any useful information from them. I tried running the command that caused the error on it's own and this was the output-
Code: 
(visualCaseGen) kferns@kferns-VirtualBox:~/cesm_projects/scratch/quickstart_case$ mpiexec  -np 4 --prepend-rank /home/kferns/cesm_projects/scratch/quickstart_case/bld/cesm.exe
[0] At line 54 of file /home/kferns/cesm2_3_beta17_gui/components/cmeps/cime_config/../cesm/driver/esmApp.F90
[0] Fortran runtime error: Cannot open file 'drv_in': No such file or directory
[0]
[0] Error termination. Backtrace:
[0] #0  0x7b2383623960 in ???
[0] #1  0x7b23836244d9 in ???
[0] #2  0x7b238362510f in ???
[0] #3  0x7b2383873ee6 in ???
[0] #4  0x7b23838741dd in ???
[0] #5  0x62b38427242e in ???
[0] #6  0x62b3842727be in ???
[0] #7  0x7b2383229d8f in __libc_start_call_main
[0]     at ../sysdeps/nptl/libc_start_call_main.h:58
[0] #8  0x7b2383229e3f in __libc_start_main_impl
[0]     at ../csu/libc-start.c:392
[0] #9  0x62b38425eda4 in ???
[0] #10  0xffffffffffffffff in ???

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 610050 RUNNING AT kferns-VirtualBox
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

So it seems the error is- Fortran runtime error: Cannot open file 'drv_in': No such file or directory
But I am able to find drv_in in /home/kferns/cesm_projects/scratch/quickstart_case/run/ so I can't work what's wrong...

Attached are the log and xml files if needed.
Any help would be appreciated

Thanks,
Kenneth
 

Attachments

  • version_info.txt
    16.1 KB · Views: 1
  • drv.log.240622-163125.txt
    1,019 bytes · Views: 1
  • cesm.log.240622-163125.txt
    695 bytes · Views: 1
  • config_compilers.xml.txt
    916 bytes · Views: 1
  • config_machines.xml.txt
    2.2 KB · Views: 1
  • drv_in.txt
    552 bytes · Views: 0
J

jedwards

CSEG and Liaisons
Staff member
When you run from the command line you need to run from the case run directory. I suspect that's the reason it couldn't find drv_in.
I also suspect that you are blowing out memory - you can try turning on the DEBUG feature with ./xmlchange DEBUG=TRUE to
perhaps get more information in the cesm.log file. Also check that your ulimits are set - especially the stack size needs to be as big as possible. The file config_compilers.xml is no longer used in this version of cesm and the compiler flags are set in ccs_config/machines/cmake_macros
 
K

kferns

Kenneth F
New Member
So I've tried a few things that haven't worked. But before I go in to those details, could you tell me more about setting the compiler flags in ccs_config/machines/cmake_macros- am I expected to create a kferns.cmake (kferns is my machine name)?
 
J

jedwards

CSEG and Liaisons
Staff member
If you need changes that are specific to your machine then yes, you should add them to a new kferns.cmake file.
 
You must log in or register to reply here.
Top