Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

model crashed with MARBL warning

Status
Not open for further replies.

tangtao2009

Tao Tang
New Member
Hi

I am doing an idealized CO2x4 experiment with a modified soil physics (I used separated soil column instead shared soil column for each PFT) by using B1850 compset. The model stopped after 41 years and 5months. No errors found in the log file of each component, but MARBL WARNING are shown in the cesm.log file, along with the following info:

-1:MPT ERROR: MPI_COMM_WORLD rank 713 has terminated without calling MPI_Finalize()
-1: aborting job


Here is the path for the scratch folder: /glade/scratch/taotang/LUMIP_indv_soil_CO2X4/run

pretty similar to this post: MARBL ERROR

I changed the dt_count to 48 in the user_nl_pop (the default value is 24). The model went well for another 35 years, but extremely slow then, only 11 months in 6 hours limit. Then the simulation stopped again, with the following MARBL WARNING:

-1:MPT ERROR: MPI_COMM_WORLD rank 257 has terminated without calling MPI_Finalize()
-1: aborting job


Here is the path for the scratch folder: /glade/scratch/taotang/LUMIP_indv_soil_CO2X4_branch/run

The paths for the two cases are:

/glade/u/home/taotang/cases/LUMIP_indv_soil_CO2X4/
/glade/u/home/taotang/cases/LUMIP_indv_soil_CO2X4_branch/

I am not sure if I can further modify dt_count to 72, but even if I can, the model would be slower, and may stop again.

I read some posts saying that modifying PE layout might be a choice. However, I tried to do so and modified env_mach_pes.xml file with the following settings:

./xmlchange NTASKS_ATM=1152,NTHRDS_ATM=3,ROOTPE_ATM=0
./xmlchange NTASKS_ICE=360,NTHRDS_ICE=3,ROOTPE_ICE=792
./xmlchange NTASKS_LND=792,NTHRDS_LND=3,ROOTPE_LND=0
./xmlchange NTASKS_ROF=792,NTHRDS_ROF=3,ROOTPE_ROF=0
./xmlchange NTASKS_CPL=1152,NTHRDS_CPL=3,ROOTPE_CPL=0
./xmlchange NTASKS_OCN=512,NTHRDS_OCN=3,ROOTPE_OCN=1152
./xmlchange NTASKS_GLC=1152,NTHRDS_GLC=3,ROOTPE_GLC=0
./xmlchange NTASKS_WAV=28,NTHRDS_WAV=3,ROOTPE_WAV=1664


Unfortunately, the model never got run, queued for a whole day.

Here is the case path: /glade/u/home/taotang/cases/LUMIP_indv_soil_CO2X4_branch2
Really have no idea how to resolve this issue.

Any help or hint is appreciated.
 

erik

Erik Kluzek
CSEG and Liaisons
Staff member
MARBL is the BGC part of the ocean model, so I'm moving this to that forum.
 
Status
Not open for further replies.
Top