Error：component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global

[SelenaChou]

Hi!
As a novice, I use CESM to calculate the radiation forcing when the condition of underlying surface changes.
I choose PC4 as my compset, and use PORT as a tool. I change the namelist according to the manual of PORT. The link is attached: https://wiki.ucar.edu/display/port/PORT
However, after I succeeded in building my case, the submission failed. The log file points out the error:

ERROR:
component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global
index: 1921(the number may vary)

It seems that some value may not be suitable for calculation. But faced with so many files, I have no idea how to find my mistake or make some correction.
Can you give me some advice? Thanks a lot!

With best regards.

Selena

 

[altuntas]

Hi Selena,

Can you provide the additional information listed here:

Information to include in help requests

When you are posting to the forums because you want help with an issue, please include as much of the following information as possible. Before submitting a help request, please check to see if your question is already answered: Search the forums for similar issues Check the CIME...

bb.cgd.ucar.edu

 

[SelenaChou]

Thanks for your kind suggestion!
This time I attach my log file.
Hope for more guidence!

 

[SelenaChou]

Sorry， the cesm log file isn't complete. I post a new one here.

 

[altuntas]

Please provide the Information listed in the link that I posted above. Some of the critical information which can help us better understand or reproduce the issue:

• Document the version of the code are you using:
  • In CESM2.1.2 and later, run the script ./describe_version from the top level of your CESM clone (that is, from the directory containing cime, components, etc.). Either:
    • Copy and paste the full output into your post
    • Redirect the output to a file (./describe_version > version_info.txt) then attach the file to your post.
  • From older model versions that do not have the describe_version script, give the output from running the following commands from the top level of your CESM clone (that is, run these from the directory containing cime, components, etc.):
    • git describe
    • ./manage_externals/checkout_externals --status --verbose
• If you have made any changes to files in the source tree (code, xml files, etc.), please make those changes available (or at least describe them in your help request)
• Describe every step you took, starting with the create_newcase command and including any changes you made to xml files (via xmlchange or hand edits), user_nl files, or any other changes in your case. (Note that the create_newcase command can be found in README.case, and any changes made via xmlchange can be found in CaseStatus.) It is best if you try to reproduce the problem first using your own instructions, to make sure that your instructions are complete and accurate.

 

[SelenaChou]

Sorry, neither ./describe_version nor ./manage_externals/checkout_externals --status --verbose could be executed successfully in my CESM clone.
Steps I took are attached:

I made some changes in the user_nl_cam file:

1. iradsw = 1
2. iradlw = 1
3. iradae = 1
4. ndens=1,1
5. nhtfrq = 0,73
6. mfilt = 1,1
7. avgflag_pertape = 'A','I'
8. rad_data_output = .true.

Thanks for your advice!

 

[altuntas]

I was able to reproduce this error with CESM2.2 on cheyenne. I will let atmosphere liaisons chime in. @hannay ?

 

[SelenaChou]

I was able to reproduce this error with CESM2.2 on cheyenne. I will let atmosphere liaisons chime in. @hannay ?

Thanks a lot for your kindness!!!

 

[wanzi0411@gmail_com]

Hi Selena(@SelenaChou ), Hi Alper(@altuntas ), Hi Cecile(@hannay),

Hope you are well. I am writing to ask whether you might have solved the issue 'Error：component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global'? I met the same problem in my case running within the cesm2.1.3 framework. My case could run after the submission, but it was crashed at a certain point, near the point when the model tries to write values to the June of the first model year. Might you have some clue where to start for the debugging process? Thank you very much!

All the best,
Wenwen

 

[SelenaChou]

Hi Selena(@SelenaChou ), Hi Alper(@altuntas ), Hi Cecile(@hannay),

Hope you are well. I am writing to ask whether you might have solved the issue 'Error：component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global'? I met the same problem in my case running within the cesm2.1.3 framework. My case could run after the submission, but it was crashed at a certain point, near the point when the model tries to write values to the June of the first model year. Might you have some clue where to start for the debugging process? Thank you very much!

All the best,
Wenwen

Hi, wenwen.

Hope my reply isn't too late.

I change my compset from PC4 to 2000_CAM50%PORT_SLND_SICE_SOCN_SROF_SGLC_SWAV on CESM2.

As well, I set as below:

"./xmlchange --file env_build.xml --id CAM_CONFIG_OPTS --val "-rad rrtmg -phys cam5 -chem trop_mozart -cc icc -fc ifort -fc_type intel" in my first case.

This code is stil under examination. Just give you a new idea that we could change the compset and use CAM5 and rrtmg. At least, last bug disappeared. Hope we can keep in touch!

Selena

 

[CallanGwd]

Dear Selena(@SelenaChou),
These days I performed some simulations and I also met the same error "component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global". The version of CESM I used is 2.1.3.

I tried to add the wet deposition of SO4 in wetdep.F90. Firstly, I succeeded to increase the wet deposition of SO4 to 2, 4, and 8 times the original, respectively, and ran two whole years successfully. However, when I tried to further x9 x10 x12 x15, even only x8.5, they were all failed after running different simulation days (maybe one month and a half, or maybe just a few days, according to the multiple. The more I increased, the less it ran). The only differences of the codes between successful cases and failed ones are the wet deposition multiple numbers.

I tried to adjust and optimize my modification in different ways, and also tried other compsets I used successfully before and tried to change the RUN_STARTDATE. Things could be a little bit better and cases could run with longer time. But they would still interrupt sooner or later and cannot run even a single year.

By the way, I also noticed that there are several posts that are related to the bug "component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global" and all of them were found in CESM2.1.3. Thus I wonder if this error is related to the model version?​

So I want to consult you that whether you have some ideas or suggestions these days?

Thank you!

(The original error is:
ERROR:
component_mod:check_fields NaN found in ATM instance: 1 field Sa_z 1d global
index: 34735)

 

[erik]

Looks like this issue still needs to be resolved. It's more relevant under the CAM forum, so I'm moving it there.

 

[erik]

@hannay @cacraig @nusbaume I moved this over from CESM general. This looks like it's a problem in the CESM release CESM2.1.3. I don't see an issue for this in CAM, but I think there should be. @altuntas did reproduce the error in the CESM2.2.0 release.

 

[CallanGwd]

I also found nothing about this problem in the Fortran source code of CAM. However, I tried another very old compset FMOZ (I used before) a month ago, and both so4_wetdep_x10 x15 and even x20 cases ran succesfully! But x2 and x12 cases failed. So it seems that this section of CESM is still not stable enough.