Please fill in all relevant information below, deleting the red text after you have read it.
Before submitting a help request, please check to see if your question is already answered:
- Search the forums for similar issues
- Check the CIME troubleshooting guide to see if any suggestions there solve your problem
- Check any other relevant CESM documentation
What version of the code are you using?
CESM v2.2.2
Have you made any changes to files in the source tree?
No modifications
Describe every step you took leading up to the problem:
I ran a FW2000climo compset at f19_f19_mg17 resolution
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
(jupyter) [Jakub FW2000climo_2]$ zcat logs/atm.log.1835539.260304-143052.gz | tail -n 100
WSHIST: writing time sample 1 to h-file 2 DATE=0001/02/01 NCSEC= 0
WSHIST: writing time sample 1 to h-file 7 DATE=0001/02/01 NCSEC= 0
WSHIST: writing time sample 1 to h-file 8 DATE=0001/02/01 NCSEC= 0
Creating new decomp: 16!3!!144!96!!d4!i5!
Creating new decomp: 16!280!3!!144!96!280!!d6!i5!
Creating new decomp: 16!3010!3!!144!96!3010!!d6!i5!
Creating new decomp: 16!71!3!!144!96!71!!d4!i5!
Creating new decomp: 16!201!3!!144!96!201!!d6!i5!
(OPNFIL): Successfully opened file ./rpointer.atm on unit= 94
(WRITE_REST_PFILE): successfully wrote local restart pointer file
./rpointer.atm
---------------------------------------
WRAPUP: nf_close( 1 )=FW2000climo_2.cam.h0.0001-02-01-00000.nc
Primary history file
Output at NSTEP = 1488
Number of time samples on this file = 1
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 2 )=FW2000climo_2.cam.h1.0001-01-31-00000.nc
Auxiliary history file number 1
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 7 )=FW2000climo_2.cam.h6.0001-01-31-00000.nc
Auxiliary history file number 6
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 8 )=FW2000climo_2.cam.h7.0001-01-31-00000.nc
Auxiliary history file number 7
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
QNEG3 from D_P_COUPLING:CLDLIQ Min. mixing ratio violated at 5942 points. Worst = -0.8E-07
QNEG3 from D_P_COUPLING:NUMLIQ Min. mixing ratio violated at 929 points. Worst = -0.3E+06
QNEG3 from D_P_COUPLING:NUMICE Min. mixing ratio violated at 341 points. Worst = -0.4E-01
QNEG3 from D_P_COUPLING:NUMRAI Min. mixing ratio violated at 277 points. Worst = -0.1E-09
QNEG3 from D_P_COUPLING:NUMSNO Min. mixing ratio violated at 8 points. Worst = -0.3E-03
QNEG3 from D_P_COUPLING:HORZ Min. mixing ratio violated at 1128 points. Worst = 0.1E+01
QNEG3 from clubb:HORZ Min. mixing ratio violated at 28788 points. Worst = 0.1E+01
QNEG3 from nucleatei:num_a3 Min. mixing ratio violated at 218 points. Worst = 0.5E-05
QNEG3 from aero_model_wetdep:num_a1 Min. mixing ratio violated at 171202130 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a2 Min. mixing ratio violated at 102707787 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a3 Min. mixing ratio violated at 32228713 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a4 Min. mixing ratio violated at 212100391 points. Worst = 0.1E-04
QNEG3 from convtran2:BCARY Min. mixing ratio violated at 2164 points. Worst = -0.2E-11
QNEG3 from convtran2:MTERP Min. mixing ratio violated at 96 points. Worst = -0.6E-10
QNEG3 from convtran2:NO Min. mixing ratio violated at 1145 points. Worst = -0.1E-10
QNEG3 from convtran2:NO2 Min. mixing ratio violated at 1 points. Worst = -0.2E-11
QNEG3 from aoa_tracers:HORZ Min. mixing ratio violated at 16248 points. Worst = 0.1E+01
QNEG3 from chemistry:H2O2 Min. mixing ratio violated at 696935 points. Worst = -0.1E-10
QNEG3 from chemistry:HNO3 Min. mixing ratio violated at 19 points. Worst = -0.3E-11
QNEG3 from chemistry:NH3 Min. mixing ratio violated at 2833460 points. Worst = -0.7E-10
QNEG3 from chemistry:SO2 Min. mixing ratio violated at 406450 points. Worst = -0.1E-11
QNEG3 from chemistry:SOAG0 Min. mixing ratio violated at 77630 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG1 Min. mixing ratio violated at 47703 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG2 Min. mixing ratio violated at 23151 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG3 Min. mixing ratio violated at 8144 points. Worst = -0.3E-10
QNEG3 from chemistry:SOAG4 Min. mixing ratio violated at 1445 points. Worst = -0.6E-10
QNEG3 from vertical diffusion:CH2O Min. mixing ratio violated at 12 points. Worst = -0.7E-10
QNEG3 from vertical diffusion:H2O2 Min. mixing ratio violated at 181743 points. Worst = -0.2E-10
QNEG3 from vertical diffusion:HCOOH Min. mixing ratio violated at 536 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:HNO3 Min. mixing ratio violated at 2765 points. Worst = -0.4E-09
QNEG3 from vertical diffusion:NH4 Min. mixing ratio violated at 18623 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:num_a1 Min. mixing ratio violated at 13852514 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a2 Min. mixing ratio violated at 11269441 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a3 Min. mixing ratio violated at 2439413 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a4 Min. mixing ratio violated at 14487254 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:SO2 Min. mixing ratio violated at 197441 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:HORZ Min. mixing ratio violated at 1724 points. Worst = 0.1E+01
QNEG3 from aero_model_drydep:num_a1 Min. mixing ratio violated at 190850712 points. Worst = 0.1E-06
QNEG3 from aero_model_drydep:num_a2 Min. mixing ratio violated at 155670306 points. Worst = 0.1E-06
QNEG3 from aero_model_drydep:num_a3 Min. mixing ratio violated at 51903256 points. Worst = 0.3E-06
QNEG3 from aero_model_drydep:num_a4 Min. mixing ratio violated at 190589109 points. Worst = 0.1E-06
QNEG3 from Gravity wave drag:CLDLIQ Min. mixing ratio violated at 62 points. Worst = -0.2E-06
QNEG3 from Gravity wave drag:CLDICE Min. mixing ratio violated at 57 points. Worst = -0.1E-06
QNEG3 from Gravity wave drag:RAINQM Min. mixing ratio violated at 244 points. Worst = -0.4E-04
QNEG3 from Gravity wave drag:NO Min. mixing ratio violated at 278 points. Worst = -0.2E-10
QNEG3 from Gravity wave drag:num_a1 Min. mixing ratio violated at 22249647 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a2 Min. mixing ratio violated at 14178675 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a3 Min. mixing ratio violated at 185378 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a4 Min. mixing ratio violated at 26085702 points. Worst = 0.1E-04
nstep, te 1489 0.26111434810664454E+10 0.26111451610506949E+10 0.92893787441525105E-04 0.98524675199021221E+05
--------------------------------------------------------
euvac_set_etf: f107,f107a = 124.480674743652 124.469436645508
--------------------------------------------------------
Number of completed timesteps: 1488
Time step 1489 partially done to provide convectively adjusted and time filtered values for history tape.
******* END OF MODEL RUN *******
(jupyter) [jjs1u25@login6001 FW2000climo_2]$
Describe your problem or question:
I have run B and F compsets without modifications and my simulations seem to blow up into static. Also, occasionally variables such as T are listed as 4D and sometimes as 3D and sometimes variables don't appear. The port to our supercomputer is fine as far as I am aware, this just recently happened for seemingly no reason; I have run these exact compsets without error before.
I would really appreciate some pointers with how to debug this!
Before submitting a help request, please check to see if your question is already answered:
- Search the forums for similar issues
- Check the CIME troubleshooting guide to see if any suggestions there solve your problem
- Check any other relevant CESM documentation
What version of the code are you using?
CESM v2.2.2
Have you made any changes to files in the source tree?
No modifications
Describe every step you took leading up to the problem:
I ran a FW2000climo compset at f19_f19_mg17 resolution
If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
(jupyter) [Jakub FW2000climo_2]$ zcat logs/atm.log.1835539.260304-143052.gz | tail -n 100
WSHIST: writing time sample 1 to h-file 2 DATE=0001/02/01 NCSEC= 0
WSHIST: writing time sample 1 to h-file 7 DATE=0001/02/01 NCSEC= 0
WSHIST: writing time sample 1 to h-file 8 DATE=0001/02/01 NCSEC= 0
Creating new decomp: 16!3!!144!96!!d4!i5!
Creating new decomp: 16!280!3!!144!96!280!!d6!i5!
Creating new decomp: 16!3010!3!!144!96!3010!!d6!i5!
Creating new decomp: 16!71!3!!144!96!71!!d4!i5!
Creating new decomp: 16!201!3!!144!96!201!!d6!i5!
(OPNFIL): Successfully opened file ./rpointer.atm on unit= 94
(WRITE_REST_PFILE): successfully wrote local restart pointer file
./rpointer.atm
---------------------------------------
WRAPUP: nf_close( 1 )=FW2000climo_2.cam.h0.0001-02-01-00000.nc
Primary history file
Output at NSTEP = 1488
Number of time samples on this file = 1
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 2 )=FW2000climo_2.cam.h1.0001-01-31-00000.nc
Auxiliary history file number 1
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 7 )=FW2000climo_2.cam.h6.0001-01-31-00000.nc
Auxiliary history file number 6
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
WRAPUP: nf_close( 8 )=FW2000climo_2.cam.h7.0001-01-31-00000.nc
Auxiliary history file number 7
Output at NSTEP = 1488
Number of time samples on this file = 2
Model Day = 31.00
---------------------------------------
QNEG3 from D_P_COUPLING:CLDLIQ Min. mixing ratio violated at 5942 points. Worst = -0.8E-07
QNEG3 from D_P_COUPLING:NUMLIQ Min. mixing ratio violated at 929 points. Worst = -0.3E+06
QNEG3 from D_P_COUPLING:NUMICE Min. mixing ratio violated at 341 points. Worst = -0.4E-01
QNEG3 from D_P_COUPLING:NUMRAI Min. mixing ratio violated at 277 points. Worst = -0.1E-09
QNEG3 from D_P_COUPLING:NUMSNO Min. mixing ratio violated at 8 points. Worst = -0.3E-03
QNEG3 from D_P_COUPLING:HORZ Min. mixing ratio violated at 1128 points. Worst = 0.1E+01
QNEG3 from clubb:HORZ Min. mixing ratio violated at 28788 points. Worst = 0.1E+01
QNEG3 from nucleatei:num_a3 Min. mixing ratio violated at 218 points. Worst = 0.5E-05
QNEG3 from aero_model_wetdep:num_a1 Min. mixing ratio violated at 171202130 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a2 Min. mixing ratio violated at 102707787 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a3 Min. mixing ratio violated at 32228713 points. Worst = 0.1E-04
QNEG3 from aero_model_wetdep:num_a4 Min. mixing ratio violated at 212100391 points. Worst = 0.1E-04
QNEG3 from convtran2:BCARY Min. mixing ratio violated at 2164 points. Worst = -0.2E-11
QNEG3 from convtran2:MTERP Min. mixing ratio violated at 96 points. Worst = -0.6E-10
QNEG3 from convtran2:NO Min. mixing ratio violated at 1145 points. Worst = -0.1E-10
QNEG3 from convtran2:NO2 Min. mixing ratio violated at 1 points. Worst = -0.2E-11
QNEG3 from aoa_tracers:HORZ Min. mixing ratio violated at 16248 points. Worst = 0.1E+01
QNEG3 from chemistry:H2O2 Min. mixing ratio violated at 696935 points. Worst = -0.1E-10
QNEG3 from chemistry:HNO3 Min. mixing ratio violated at 19 points. Worst = -0.3E-11
QNEG3 from chemistry:NH3 Min. mixing ratio violated at 2833460 points. Worst = -0.7E-10
QNEG3 from chemistry:SO2 Min. mixing ratio violated at 406450 points. Worst = -0.1E-11
QNEG3 from chemistry:SOAG0 Min. mixing ratio violated at 77630 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG1 Min. mixing ratio violated at 47703 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG2 Min. mixing ratio violated at 23151 points. Worst = -0.4E-11
QNEG3 from chemistry:SOAG3 Min. mixing ratio violated at 8144 points. Worst = -0.3E-10
QNEG3 from chemistry:SOAG4 Min. mixing ratio violated at 1445 points. Worst = -0.6E-10
QNEG3 from vertical diffusion:CH2O Min. mixing ratio violated at 12 points. Worst = -0.7E-10
QNEG3 from vertical diffusion:H2O2 Min. mixing ratio violated at 181743 points. Worst = -0.2E-10
QNEG3 from vertical diffusion:HCOOH Min. mixing ratio violated at 536 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:HNO3 Min. mixing ratio violated at 2765 points. Worst = -0.4E-09
QNEG3 from vertical diffusion:NH4 Min. mixing ratio violated at 18623 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:num_a1 Min. mixing ratio violated at 13852514 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a2 Min. mixing ratio violated at 11269441 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a3 Min. mixing ratio violated at 2439413 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:num_a4 Min. mixing ratio violated at 14487254 points. Worst = 0.1E-04
QNEG3 from vertical diffusion:SO2 Min. mixing ratio violated at 197441 points. Worst = -0.1E-10
QNEG3 from vertical diffusion:HORZ Min. mixing ratio violated at 1724 points. Worst = 0.1E+01
QNEG3 from aero_model_drydep:num_a1 Min. mixing ratio violated at 190850712 points. Worst = 0.1E-06
QNEG3 from aero_model_drydep:num_a2 Min. mixing ratio violated at 155670306 points. Worst = 0.1E-06
QNEG3 from aero_model_drydep:num_a3 Min. mixing ratio violated at 51903256 points. Worst = 0.3E-06
QNEG3 from aero_model_drydep:num_a4 Min. mixing ratio violated at 190589109 points. Worst = 0.1E-06
QNEG3 from Gravity wave drag:CLDLIQ Min. mixing ratio violated at 62 points. Worst = -0.2E-06
QNEG3 from Gravity wave drag:CLDICE Min. mixing ratio violated at 57 points. Worst = -0.1E-06
QNEG3 from Gravity wave drag:RAINQM Min. mixing ratio violated at 244 points. Worst = -0.4E-04
QNEG3 from Gravity wave drag:NO Min. mixing ratio violated at 278 points. Worst = -0.2E-10
QNEG3 from Gravity wave drag:num_a1 Min. mixing ratio violated at 22249647 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a2 Min. mixing ratio violated at 14178675 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a3 Min. mixing ratio violated at 185378 points. Worst = 0.1E-04
QNEG3 from Gravity wave drag:num_a4 Min. mixing ratio violated at 26085702 points. Worst = 0.1E-04
nstep, te 1489 0.26111434810664454E+10 0.26111451610506949E+10 0.92893787441525105E-04 0.98524675199021221E+05
--------------------------------------------------------
euvac_set_etf: f107,f107a = 124.480674743652 124.469436645508
--------------------------------------------------------
Number of completed timesteps: 1488
Time step 1489 partially done to provide convectively adjusted and time filtered values for history tape.
******* END OF MODEL RUN *******
(jupyter) [jjs1u25@login6001 FW2000climo_2]$
Describe your problem or question:
I have run B and F compsets without modifications and my simulations seem to blow up into static. Also, occasionally variables such as T are listed as 4D and sometimes as 3D and sometimes variables don't appear. The port to our supercomputer is fine as far as I am aware, this just recently happened for seemingly no reason; I have run these exact compsets without error before.
I would really appreciate some pointers with how to debug this!