I'm running CESM2.1.3 with the --compset G1850ECO --res T62.t12. Since there is no initial field for the ecosy system (MARBL) with the this resolution provided in the website, I interpolated the physical variables from other restart files with lower resolutions. I carefully check the all the fields after interpolation and found all the variable are in the reasonable range without any NaN. However, during the running, the program crashed with the errors in cesm.log:
(Task 541, block 2) Message from (lon, lat) ( 81.975, 72.287), which is global (i,j) (2278, 2304)
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) it = 1
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) x1,f = 0.1000000E-008-0.1686966E-004
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) x2,f = 0.1000000E-006-0.4663282E-003
(Task 541, block 2) MARBL WARNING (marbl_surface_flux_mod:marbl_surface_flux_compute): Warning reported from marbl_co2calc_surface() with flux_alt_co2
ecosys_driver:ecosys_driver_set_interior: NaN in dtracer_module, (i,j,k)=(
1494 , 1105 , 1 )
(lon,lat)=( 39.3500000000000 , -7.72692510078884 )
PO4 0.144610345363617 NaN
NO3 0.549050509929657 NaN
SiO3 3.46367835998535 NaN
NH4 2.596265450119972E-002 NaN
Fe 2.673334965948015E-004 NaN
Lig 1.153798191808164E-003 NaN
O2 209.786178588867 NaN
DIC 1957.42382812500 NaN
DIC_ALT_CO2 1957.42382812500 NaN
ALK 2376.66992187500 NaN
ALK_ALT_CO2 2376.66992187500 NaN
DOC 49.0058708190918 1.728319855685696E-007
DON 4.35348033905029 1.617495927988613E-008
DOP 0.173821091651917 2.532653210415041E-009
DOPr 2.519437111914158E-002 NaN
DONr 1.28296422958374 NaN
DOCr 23.4972362518311 NaN
zooC 0.529732704162598 -1.969501277076526E-007
spChl 4.706111550331116E-002 -5.759960074765008E-007
spC 0.425841957330704 -5.212015702882479E-006
spP 2.835230669006705E-003 -3.470129273726819E-008
spFe 1.123612491937820E-005 -1.375225177697648E-010
spCaCO3 1.042646169662476E-002 -1.276127912637340E-007
diatChl 4.038055893033743E-003 -2.204692064930859E-008
diatC 3.059628047049046E-002 -1.670491954630460E-007
diatP 2.037274825852364E-004 -1.112308437926462E-009
diatFe 3.452960584127140E-007 -1.885242552852529E-012
diatSi 2.669562352821231E-003 -1.457523890706072E-008
diazChl 4.901377949863672E-003 -4.412383088502405E-008
diazC 4.515984281897545E-002 -4.065439819086936E-007
diazP 3.006088081747293E-004 -2.706180251783954E-009
diazFe 1.579724539624294E-006 -1.422120455600523E-011
Dust Flux
4.344853078362572E-012
PAR Column Fraction
1.00000000000000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
Surface Shortwave
241.908259579890 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
Potential Temperature
28.1760933316955
Salinity
34.5737510637689
Pressure
0.496808084874780
Iron Sediment Flux
6.958874272741377E-009
------------------------------------------------------------------------
POP aborting...
Stopping in ecosys_driver:ecosys_driver_set_interior
------------------------------------------------------------------------
Abort(0) on node 180 (rank 180 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 0) - process 180
It seems that the error is resulted from the abnormal NaN value in the river forcing field (the second column for PO4 ~ DOCr). I also check the ocn.log and find that the program is reading the river forcing:
(shr_strdata_print)
(shr_strdata_print) nvectors = 0
(shr_strdata_print) ----------------------------------------------------------
(shr_dmodel_readstrm) open : /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc
(shr_dmodel_readstrm) file lb: /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc 1
(shr_dmodel_readstrm) file ub: /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc 1
I think this may be the cause for this error but I cannot solve it. Any help is wellcome.
Hexin Ye
(Task 541, block 2) Message from (lon, lat) ( 81.975, 72.287), which is global (i,j) (2278, 2304)
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) it = 1
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) x1,f = 0.1000000E-008-0.1686966E-004
(Task 541, block 2) MARBL WARNING (marbl_co2calc_mod:drtsafe): (marbl_co2calc_mod:drtsafe) x2,f = 0.1000000E-006-0.4663282E-003
(Task 541, block 2) MARBL WARNING (marbl_surface_flux_mod:marbl_surface_flux_compute): Warning reported from marbl_co2calc_surface() with flux_alt_co2
ecosys_driver:ecosys_driver_set_interior: NaN in dtracer_module, (i,j,k)=(
1494 , 1105 , 1 )
(lon,lat)=( 39.3500000000000 , -7.72692510078884 )
PO4 0.144610345363617 NaN
NO3 0.549050509929657 NaN
SiO3 3.46367835998535 NaN
NH4 2.596265450119972E-002 NaN
Fe 2.673334965948015E-004 NaN
Lig 1.153798191808164E-003 NaN
O2 209.786178588867 NaN
DIC 1957.42382812500 NaN
DIC_ALT_CO2 1957.42382812500 NaN
ALK 2376.66992187500 NaN
ALK_ALT_CO2 2376.66992187500 NaN
DOC 49.0058708190918 1.728319855685696E-007
DON 4.35348033905029 1.617495927988613E-008
DOP 0.173821091651917 2.532653210415041E-009
DOPr 2.519437111914158E-002 NaN
DONr 1.28296422958374 NaN
DOCr 23.4972362518311 NaN
zooC 0.529732704162598 -1.969501277076526E-007
spChl 4.706111550331116E-002 -5.759960074765008E-007
spC 0.425841957330704 -5.212015702882479E-006
spP 2.835230669006705E-003 -3.470129273726819E-008
spFe 1.123612491937820E-005 -1.375225177697648E-010
spCaCO3 1.042646169662476E-002 -1.276127912637340E-007
diatChl 4.038055893033743E-003 -2.204692064930859E-008
diatC 3.059628047049046E-002 -1.670491954630460E-007
diatP 2.037274825852364E-004 -1.112308437926462E-009
diatFe 3.452960584127140E-007 -1.885242552852529E-012
diatSi 2.669562352821231E-003 -1.457523890706072E-008
diazChl 4.901377949863672E-003 -4.412383088502405E-008
diazC 4.515984281897545E-002 -4.065439819086936E-007
diazP 3.006088081747293E-004 -2.706180251783954E-009
diazFe 1.579724539624294E-006 -1.422120455600523E-011
Dust Flux
4.344853078362572E-012
PAR Column Fraction
1.00000000000000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
Surface Shortwave
241.908259579890 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
Potential Temperature
28.1760933316955
Salinity
34.5737510637689
Pressure
0.496808084874780
Iron Sediment Flux
6.958874272741377E-009
------------------------------------------------------------------------
POP aborting...
Stopping in ecosys_driver:ecosys_driver_set_interior
------------------------------------------------------------------------
Abort(0) on node 180 (rank 180 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 0) - process 180
It seems that the error is resulted from the abnormal NaN value in the river forcing field (the second column for PO4 ~ DOCr). I also check the ocn.log and find that the program is reading the river forcing:
(shr_strdata_print)
(shr_strdata_print) nvectors = 0
(shr_strdata_print) ----------------------------------------------------------
(shr_dmodel_readstrm) open : /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc
(shr_dmodel_readstrm) file lb: /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc 1
(shr_dmodel_readstrm) file ub: /public/home/hongzhanma/hexin.ye/cesm/cesm2.1.3/inputdata/ocn/pop/tx0.1v3/forcing/riv_nut.gnews_gnm.JRA025m_to_tx0.1v3_nnsm_e333r100_190226.20200616.nc 1
I think this may be the cause for this error but I cannot solve it. Any help is wellcome.
Hexin Ye