Issue with running MUSICAv1 on IITM HPC

C

chaitri

Chaitri Roy
New Member
I have ported and build MUSICAv1 with cesm3_0_beta05 on the IITM HPC (supercomputer in India). However, I am unable to complete even a 5-day run with hourly output, despite using 54 nodes (96 processors per node). The model starts writing the first netcdf file, but doesn't finish even after a week. The same model configuration runs successfully on the NCAR HPC with 50 nodes, and the run completed in 2 hours. Could someone help me troubleshoot this issue? I am wondering if the issue is related to node allocation or something else. I have attached the .xml files. Thanks!
 

Attachments

Sort by date Sort by votes
C

chaitri

Chaitri Roy
New Member
Hi Chris,
I am running -res mpasa60-3-India_g17_mg17 --compset FCHIST. The grid mpasa60-3-India_g17_mg17 is created with zoom-in over North India (3-60km).

Chaitri.
 
Vote Upvote 0 Downvote
F

fischer

CSEG and Liaisons
Staff member
Hi Chaitri,

Can you try doing the run without any output. This will help eliminate I/O as the issue.
Also, can you point me to your run on the NCAR HPC if it's still available.

Thanks
Chris
 
Vote Upvote 0 Downvote
C

chaitri

Chaitri Roy
New Member
Hello Chris,

I have submitted jobs using different combinations of nodes and MPI processes, and also modified the output writing frequency and limited the input parameters for testing. Interestingly, the model was able to write the CICE output files(.nc), but no .cam files. However, it is hanging again without reporting any errors.

Yes I have my run in NCAR HPC: /glade/work/chaitrir/MUSICAv1/MPAS/f.e3b01.FCHIST.mpasa_x20.835586.India.v01

Also, how do I run without any output?

Thanks,
Chaitri.
 
Vote Upvote 0 Downvote
You must log in or register to reply here.
Back
Top