You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Running CESM on NCAR's Yellowstone
- Thread starter jedwards
- Start date
yafangzhong@wisc_edu
Member
16:Abort(1) on node 16 (rank 16 in comm 1140850688): Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
yafangzhong@wisc_edu
Member
16:Abort(1) on node 16 (rank 16 in comm 1140850688): Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
yafangzhong@wisc_edu
Member
16:Abort(1) on node 16 (rank 16 in comm 1140850688): Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
yafangzhong@wisc_edu
Member
16:Abort(1) on node 16 (rank 16 in comm 1140850688): Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
yafangzhong@wisc_edu
Member
16:Abort(1) on node 16 (rank 16 in comm 1140850688): Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
16:PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x74b1c00, scount=2496, dtype=0x4c000829, rbuf=0x74b1c00, rcnts=0x6b290f0, displs=0x6b291d0, dtype=0x4c000829, root=0, comm=0x84000001) failed
16:MPIR_Gatherv_impl(210):
16:MPIR_Gatherv(104).....:
16:MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x74b1c00 src=0x74b1c00 len=19968
I found another file with the same problem, please get /glade/scratch/jedwards/cesmtests/ccsm3_test/SourceMods/src.cam/history.F90 and build again.
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
I found another file with the same problem, please get /glade/scratch/jedwards/cesmtests/ccsm3_test/SourceMods/src.cam/history.F90 and build again.
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
I found another file with the same problem, please get /glade/scratch/jedwards/cesmtests/ccsm3_test/SourceMods/src.cam/history.F90 and build again.
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
I found another file with the same problem, please get /glade/scratch/jedwards/cesmtests/ccsm3_test/SourceMods/src.cam/history.F90 and build again.
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
I found another file with the same problem, please get /glade/scratch/jedwards/cesmtests/ccsm3_test/SourceMods/src.cam/history.F90 and build again.
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
It turns out that there is another, easier solution.
Set the environment variable:
MP_EUIDEVELOP=min
and all should work fine.
This variable was recently removed from the default environment which explains why the errors only started appearing now.
- Jim
yafangzhong@wisc_edu
Member
It works. Thank you very much.
Yafang
Yafang
yafangzhong@wisc_edu
Member
It works. Thank you very much.
Yafang
Yafang
yafangzhong@wisc_edu
Member
It works. Thank you very much.
Yafang
Yafang
yafangzhong@wisc_edu
Member
It works. Thank you very much.
Yafang
Yafang
yafangzhong@wisc_edu
Member
It works. Thank you very much.
Yafang
Yafang
yafangzhong@wisc_edu
Member
Hi Jim,
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
yafangzhong@wisc_edu
Member
Hi Jim,
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
yafangzhong@wisc_edu
Member
Hi Jim,
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
yafangzhong@wisc_edu
Member
Hi Jim,
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang
The CCSM3 has problems with resubmitting jobs automatically, though the manually submitted continue-run would be fine.
Could you please take a look at /glade/u/home/yafang/test1? There are two stdout files from Mar 1st. ccsm.stdout.891572
was from a manually submitted continue-run, ccsm.stdout.891581 was from the one submitted automatically. env_run was emptied
as a result of the aborted run.
Thanks,
Yafang