MPT ERROR: MPI_COMM_WORLD rank 8 has terminated without calling MPI_Finalize()

[Jyoti Singh]

I am trying to run ctsm5.1.dev113 in offline mode using ARISE data for two cases to see the impact of ozone on plants. One case is control case in SSP2.45 climate (/glade/work/jyotis/O3_CLM5/O3_cases/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006) and another one is experiment case with ARISE data (/glade/work/jyotis/O3_CLM5/O3_cases/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006_hourlyO3) and atmospheric ozone input (hourly) with ozone damage function turned on. Both cases are failing with ths error message in cesm.log of case "b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006_hourlyO3" is:
.
.
.
-1:MPT ERROR: MPI_COMM_WORLD rank 8 has terminated without calling MPI_Finalize()

-1: aborting job
.
.
.
The error message for the case "b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006" in cesm.log file is:
.
.
.

155:libc-2.22.so 00002B2040CF5A35 __libc_start_main Unknown Unknown


155:cesm.exe 0000000000414F29 Unknown Unknown Unknown


465:MPT ERROR: Rank 465(g:465) is aborting with error code 1001.


465: Process ID: 59152, Host: r5i6n1, Program: /glade/scratch/jyotis/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006/bld/cesm.exe


465: MPT Version: HPE MPT 2.22 03/31/20 15:59:10


465:


465:MPT: --------stack traceback-------
.
.
.

I am not able figure out the reason of failing the simulation. In case "b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006_hourlyO3" the case run folder in scratch

"/glade/scratch/jyotis/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006_hourlyO3/run" contains the history files.

 

[oleson]

From the lnd log:

htapes_fieldlist ERROR: O3UPTAKESUN in fincl( 3 ) for history tape
3 not found
ENDRUN:
ERROR: ERROR in histFileMod.F90 at line 838

 

[Jyoti Singh]

Thanks Keith. I got the reason for the error. The case "/glade/work/jyotis/O3_CLM5/O3_cases/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006" has no atmospheric ozone and hence it did recognize the ozone-related output variables. I am rerunning the modified case.

Could you please also look into the errors related with the case with atmospheric ozone as input (/glade/work/jyotis/O3_CLM5/O3_cases/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006_hourlyO3).

 

[oleson]

It seem to have something to do with this file as indicated in the atm log:

(shr_strdata_readstrm) reading file ub: /glade/work/jyotis/O3_CLM5/SRF_O3_files/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006.cam.h4.O3_SRF.2055010100-2065010100.cdf5.nc 1
(shr_strdata_advance) ERROR: for stream 7
(shr_strdata_advance) ERROR: dday = 27741
(shr_strdata_advance) ERROR: dtime, dtmax, dtmin, dtlimit =
27740.0416666667 27740.0416666667 4.166666666666663E-002
1.50000000000000
(shr_strdata_advance) ERROR: ymdLB, todLB, ymdUB, todUB = 20641231
82800 21410101 0

Looking at your datm.streams.xml, I see:

<stream_info name="preso3.SSP2-4.5">
<taxmode>cycle</taxmode>
<tintalgo>linear</tintalgo>
<readmode>single</readmode>
<mapalgo>bilinear</mapalgo>
<dtlimit>1.5</dtlimit>
<year_first>2015</year_first>
<year_last>2100</year_last>
<year_align>2015</year_align>
<vectors>null</vectors>
<meshfile>/glade/campaign/cesm/cesmdata/inputdata/share/meshes/fv0.9x1.25_141008_polemod_ESMFmesh.nc</meshfile>
<lev_dimname>null</lev_dimname>
<datafiles>
<file>/glade/work/jyotis/O3_CLM5/SRF_O3_files/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006/b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DEFAULT.006.cam.h4.O3_SRF.2055010100-2065010100.cdf5.nc</file>
</datafiles>
<datavars>
<var>O3_SRF Sa_o3</var>
</datavars>
<offset>0</offset>
</stream_info>

If indeed there is only data for 2055 through the end of 2064 as indicated by the file name, then you would need to tell the datm that. I.e., adjust the year_first, year_last, etc variables.

 

[Jyoti Singh]

Thanks, the problem was as you mentioned above, the years. I added yesr_first, yesr_align and year_end in the
<user_nl_datm_streams> for the surface ozone file and it worked.