Intermittent Problem on Yellowstone with MPI_Wtime calls

[aliceb]

There have been intermittent failures of CESM that can be traced back to a problem calling MPI_Wtime.   We do not use the MPI_Wtime function in the gptl timing library on yellowstone, however some of the CESM component models (notably pop) call MPI_Wtime directly.    Adding the module load command below replaces the MPI_Wtime function with a call to system clock.   If you have seen this problem on yellowstone, add the lines:module use -a /glade/apps/contrib/wtimemodule load wt

to your env_mach_specific file.

 

[matthew_huber@unh_edu]

Hi,  I think I may be having this problem on our cluster at Purdue (conte, which is functionally identical to yellowstone).  I was wondering if you could point me to the specific version of wtime that the module is loading so that we can build it locally? Thanks!-m

 

[matthew_huber@unh_edu]

Hi,  I think I may be having this problem on our cluster at Purdue (conte, which is functionally identical to yellowstone).  I was wondering if you could point me to the specific version of wtime that the module is loading so that we can build it locally? Thanks!-m

 

[matthew_huber@unh_edu]

Hi,  I think I may be having this problem on our cluster at Purdue (conte, which is functionally identical to yellowstone).  I was wondering if you could point me to the specific version of wtime that the module is loading so that we can build it locally? Thanks!-m

 

[matthew_huber@unh_edu]

Hi,  I think I may be having this problem on our cluster at Purdue (conte, which is functionally identical to yellowstone).  I was wondering if you could point me to the specific version of wtime that the module is loading so that we can build it locally? Thanks!-m

 

[jedwards]

Hi Matthew,Our solution is to use gettimeofday and just bypass the mpi implimentation.    #include

double MPI_Wtime(){

    struct timeval tv;
    struct timezone tz;

    double wt;

    gettimeofday(&tv, &tz);

    wt = (double)tv.tv_usec/1000000.;
    return (wt);

}

double mpi_wtime_(){ return MPI_Wtime(); } 

 

[jedwards]

Hi Matthew,Our solution is to use gettimeofday and just bypass the mpi implimentation.    #include

double MPI_Wtime(){

    struct timeval tv;
    struct timezone tz;

    double wt;

    gettimeofday(&tv, &tz);

    wt = (double)tv.tv_usec/1000000.;
    return (wt);

}

double mpi_wtime_(){ return MPI_Wtime(); } 

 

[jedwards]

Hi Matthew,Our solution is to use gettimeofday and just bypass the mpi implimentation.    #include

double MPI_Wtime(){

    struct timeval tv;
    struct timezone tz;

    double wt;

    gettimeofday(&tv, &tz);

    wt = (double)tv.tv_usec/1000000.;
    return (wt);

}

double mpi_wtime_(){ return MPI_Wtime(); } 

 

[jedwards]

Hi Matthew,Our solution is to use gettimeofday and just bypass the mpi implimentation.    #include

double MPI_Wtime(){

    struct timeval tv;
    struct timezone tz;

    double wt;

    gettimeofday(&tv, &tz);

    wt = (double)tv.tv_usec/1000000.;
    return (wt);

}

double mpi_wtime_(){ return MPI_Wtime(); } 

 

[matthew_huber@unh_edu]

Thanks, I will try and see if that solves some of our hanging wait problems.-m

 

[matthew_huber@unh_edu]

Thanks, I will try and see if that solves some of our hanging wait problems.-m

 

[matthew_huber@unh_edu]

Thanks, I will try and see if that solves some of our hanging wait problems.-m

 

[matthew_huber@unh_edu]

Thanks, I will try and see if that solves some of our hanging wait problems.-m