Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CAM-chem run error

James King

James King
Member
Hi all,

I'm trying to set up a run on Cheyenne with a custom compset: HIST_CAM60%CCTS2_CLM50%BGC-CROP_CICE%PRES_DOCN%DO1M_MOSART_CISM2%NOEVOLVE_SWAV.
Running create_newcase, case_setup, and case_build all work fine. The only change I made was to set JOB_WALLCLOCK_TIME to 00:20:00. However, when attempting to run the standard 5 day test, I run into an error:


2021-10-22 07:57:21: model execution error
ERROR: Command: 'mpiexec_mpt -p "%g:" -np 384 omplace -tm open64 /glade/scratch/jamesking/baseline/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed with error '' from dir '/glade/scratch/jamesking/baseline/run'
---------------------------------------------------
2021-10-22 07:57:21: case.run error
ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 384 omplace -tm open64 /glade/scratch/jamesking/baseline/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed


In the cesm log file, the error is:

-1:MPT ERROR: MPI_COMM_WORLD rank 177 has terminated without calling MPI_Finalize()

It looks like the error arises in CAM, as the atm log file ends with:

open_trc_datafile:
/glade/p/cesmdata/cseg/inputdata/atm/cam/tracer_cnst/tracer_cnst_halons_3D_L70_
1849-2015_CMIP6ensAvg_c180927.nc
trcdata_init: file%has_ps = T
O3 is not an invariant
ERROR: tracer_cnst_init


Any ideas as to what might have broken CESM2 on this occasion? Log files in full are attached.

Cheers,

James
 

Attachments

  • atm.log.1136361.chadmin1.ib0.cheyenne.ucar.edu.211022-075704.txt
    142 KB · Views: 3
  • cesm.log.1136361.chadmin1.ib0.cheyenne.ucar.edu.211022-075704.txt
    560.2 KB · Views: 1
  • cpl.log.1136361.chadmin1.ib0.cheyenne.ucar.edu.211022-075704.txt
    48.5 KB · Views: 0

919265375

Yiqian Ni
New Member
open_trc_datafile:
/glade/p/cesmdata/cseg/inputdata/atm/cam/tracer_cnst/tracer_cnst_halons_3D_L70_
1849-2015_CMIP6ensAvg_c180927.nc
trcdata_init: file%has_ps = T
O3 is not an invariant
ERROR: tracer_cnst_init
I also have this error. Have you solved this problem? Would you tell me how to solve this? Thank you very much.
 

James King

James King
Member
Hi! As you can see I posted this a few years ago so off the top of my head I'm afraid I can't remember how I fixed it. In general it seems this error is related to the initialisation of the input files for CAM-Chem, so make sure that your input files include the time periods over which you are running. You might also want to check 'CAM_NML_USE_CASE' to see if this is what you expect for your compset.
 

919265375

Yiqian Ni
New Member
Thank you so much for your prompt response. I've checked my 'CAM_NML_USE_CASE' and changed it to what I expect. Now my case can continue to run!
 
Top