MPT ERROR: Rank 0(g:0) is aborting with error code 1001.

[ysjang]

Hi all,
I am having trouble running SCAM (clm5_1+GSWP3, version: cesm2_3_alpha16g) on Cheyenne with a modified plant functional type (100% temperate needle leaf tree). The job runs successfully with default surface data. However, when I submit the case with surface data modified to 100% temperate needle leaf tree or C3 grass, I encounter ERROR messages as follows:

ERROR: ERROR: One or more of the CTSM cap export_1D fields are NaN

MPT ERROR: Rank 0(g:0) is aborting with error code 1001.
Process ID: 29703, Host: r6i6n33, Program: /glade/scratch/jyishin/scam_arm97_100NET/bld/cesm.exe
MPT Version: HPE MPT 2.22 03/31/20 15:59:10

MPT: header=header@entry=0x7ffdbba47b10 "MPT ERROR: Rank 0(g:0) is aborting with error code 1001.\n\tProcess ID: 29703, Host: r6i6n33, Program: /glade/scratch/jyishin/scam_arm97_100NET/bld/cesm.exe\n\tMPT Version: HPE MPT 2.22 03/31/20 15:59:10"...) at sig.c:340

MPT ERROR: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
aborting job


Please let me know if there is any additional information I should provide or suggested documents. Any advice would be greatly appreciated!

Thanks,
Yi-Shin

 

[nusbaume]

Hi Yi-Shin,

Are you trying to do this run on an NCAR machine, such as derecho or cheyenne? If so, could you provide a path to your modified surface data file and your case directory? That should hopefully be enough info to potentially track down what the problem is.

Thanks, and have a great day!

Jesse

 

[ysjang]

Hi Yi-Shin,

Are you trying to do this run on an NCAR machine, such as derecho or cheyenne? If so, could you provide a path to your modified surface data file and your case directory? That should hopefully be enough info to potentially track down what the problem is.

Thanks, and have a great day!

Jesse

Hi Jesse,
Thanks for your response.
I run on Cheyenne, and here is the path for the surface data: /glade/u/home/jyishin/SCAM_surfdata/SCAM_100NET_CL_obsLAI_tp20bt13_IOPsites.nc
I appreciate your patience and look forward to your suggestion.

Cheers,
Yi-Shin

 

[ysjang]

Hi Jesse,
Thanks for your response.
I run on Cheyenne, and here is the path for the surface data: /glade/u/home/jyishin/SCAM_surfdata/SCAM_100NET_CL_obsLAI_tp20bt13_IOPsites.nc
I appreciate your patience and look forward to your suggestion.

Cheers,
Yi-Shin

I apologize for the oversight in providing the link to my cases:

Successful case with modified surface data: /glade/work/jyishin/my_cesm_sandbox/scam_cases/scam_twp06_100NET
Successful case with default surface data: /glade/work/jyishin/my_cesm_sandbox/scam_cases/scam_arm97
Unsuccessful case with modified surface data: /glade/work/jyishin/my_cesm_sandbox/scam_cases/scam_arm97_100NET

Thanks,
Yi-Shin

 

[slevis]

I have not used SCAM, so bear with me. Are you running two locations (twp versus arm) pointing to the same fsurdat? If nothing else differs between these runs, then there may be something wrong in your modified fsurdat, specific to the arm location. I can't tell specifically what is wrong from the error message.

Since twp works, you may want to look for inconsistencies between your changes in the two locations. Alternatively, you may wish to start over for arm and make different changes to see what happens. For example, you could try keeping all the pfts from the default file, and just change their percentages initially. If that still works, then take the next step to put all the area in one pft.

As you, I also would have expected this to work the same in both locations, so you will have to be creative in your troubleshooting, until you find what went wrong...

 

[mdfowler]

Hi Yi-Shin,

Looking into your fsurdat file, it does look like the modifications haven't made it into the ARM site. I see that at TWP there's 100% need leaf temperate tree, but at ARM it looks like the default distribution is still present (~49% grass and 51% crop).

I'm wondering if the setting of zetamaxstable is playing a role as well; if I recall right, the default value is 0.5 but in your namelist it's set to 100. Does a run without that modification work?

Cheers,
Meg

 

[ysjang]

Hi Samuel and Meg,
I appreciate your suggestion!! I am working on it and will reply with the results soon.

Thanks,
Yi-Shin