Input boundary SST dataset

[yingli@atmos_colostate_edu]

I tried again, and the run crased again. 7605:ERROR 1 from file /project/sprelcot/build/rcots007a/src/ppe/lapi/Sam.cpp line 8797605:Sam::CheckTimeout TIMEOUT happened 7605:Error received in error handler.

 

[santos]

I think that both cases are a system issue. I had a bit of trouble on yellowstone yesterday; for instance, one of my CESM jobs completed, but for some reason the mpirun command never exited, so eventually it ran over the time limit.

 

[santos]

I think that both cases are a system issue. I had a bit of trouble on yellowstone yesterday; for instance, one of my CESM jobs completed, but for some reason the mpirun command never exited, so eventually it ran over the time limit.

 

[hannay]

The sam.cpp is usually a yellwostone and not a CESM issue.It is usually due to a defective node. It is more likely to hit these at high resolution (as you are using more nodes).

You should report the problem to CISL Support: cislhelp@ucar.edu. You will specify the job number. They will need this to find which node is defective

 

[hannay]

The sam.cpp is usually a yellwostone and not a CESM issue.It is usually due to a defective node. It is more likely to hit these at high resolution (as you are using more nodes).

You should report the problem to CISL Support: cislhelp@ucar.edu. You will specify the job number. They will need this to find which node is defective

 

[yingli@atmos_colostate_edu]

The run is still unsuccesful. I have reported to CISL, and they are looking into it. <br /><br />The logfile also contains many lines that this: <br />2133:Client is created in Unreliable HW mode<br /><br />Does it has anything to do with the failure? <br /><br />BTW, the case directory is <span style="font-size: 10px;">/glade/u/home/yingli/cesm/runs/f.F2000C5.ne120_ne120.test.011</span>

 

[yingli@atmos_colostate_edu]

The run is still unsuccesful. I have reported to CISL, and they are looking into it. <br /><br />The logfile also contains many lines that this: <br />2133:Client is created in Unreliable HW mode<br /><br />Does it has anything to do with the failure? <br /><br />BTW, the case directory is <span style="font-size: 10px;">/glade/u/home/yingli/cesm/runs/f.F2000C5.ne120_ne120.test.011</span>

 

[jedwards]

You need to change your pe-layout so that the total number of tasks is a mutilpe of 15.

 

[jedwards]

You need to change your pe-layout so that the total number of tasks is a mutilpe of 15.

 

[yingli@atmos_colostate_edu]

Could you please let me know the reason to change the number of tasks to be multiple of 15? I have other similar runs and succesfully done, but the total number of tasks are not multiple of 15.Anyway, I did changed those numbers as suggested, but it failed again. 

 

[yingli@atmos_colostate_edu]

Could you please let me know the reason to change the number of tasks to be multiple of 15? I have other similar runs and succesfully done, but the total number of tasks are not multiple of 15.Anyway, I did changed those numbers as suggested, but it failed again. 

 

[jedwards]

We run CESM using 15 tasks per node on yellowstone, if your pe count is not a multiple of 15 you leave a node partially idle, there is some evidence that this causes problems.   I don't see any explanation for your latest run failure, please resubmit.  

 

[jedwards]

We run CESM using 15 tasks per node on yellowstone, if your pe count is not a multiple of 15 you leave a node partially idle, there is some evidence that this causes problems.   I don't see any explanation for your latest run failure, please resubmit.  

 

[yingli@atmos_colostate_edu]

Thanks for your explanation.The resubmitted run failed again. Is there any other possibliblitie besides the system error causing the run failure? 

 

[yingli@atmos_colostate_edu]

Thanks for your explanation.The resubmitted run failed again. Is there any other possibliblitie besides the system error causing the run failure? 

 

[hannay]

It seems you have the error:
4783:ERROR 1 from file /project/sprelcot/build/rcots007a/src/ppe/lapi/Sam.cpp line 879
4783:Sam::CheckTimeout TIMEOUT happenedAs far as I know the sam timeouts are a bug in yellowstone.  What did CISL say about it ?

 

[hannay]

It seems you have the error:
4783:ERROR 1 from file /project/sprelcot/build/rcots007a/src/ppe/lapi/Sam.cpp line 879
4783:Sam::CheckTimeout TIMEOUT happenedAs far as I know the sam timeouts are a bug in yellowstone.  What did CISL say about it ?

 

[yingli@atmos_colostate_edu]

I have reported to CISL, and they are still looking into it. They haven't found anything related to a defective node so far. So I am wondering if there is any other possibilies except the system issue.  

 

[yingli@atmos_colostate_edu]

I have reported to CISL, and they are still looking into it. They haven't found anything related to a defective node so far. So I am wondering if there is any other possibilies except the system issue.  

 

[jedwards]

I haven't found anything in your logs.   You could try runnining with DEBUG=TRUE in env_build.xml to see if you get more information (requires rebuilding)