case.submit error

[tylerh]

Hi,

I've been running iCESM for 20 model years now. It was running overnight and stopped working with 3 resubmits left on the run. When I tried to re-submit the case this morning (without making any changes to anything), I got the error message attached. Any help is much appreciated!

Tyler

 

[erik]

It seems to be failing in the CAM namelist generation. Run the case script "./preview_namelists" with the "-v" option to give more hints as to what's going on. There's also a "-d" option that gives even more output.

Do you know of anything that changed in your case directory? Even by accident?

 

[tylerh]

Hi Erik,
Attached is the message I get using ./preview_namelist -v. The model was running just fine and quit in the middle of a submission during the night. The only thing that I touched that prior evening was the env_workflow.xml file. I was trying to figure out where to change the project code so when I resubmitted the project again, it would pull from another project's core hours. In that file, the only thing I touched are the two "PROJECT" ids and the two "CHARGE_ACCOUNT" ids. Otherwise, I haven't edited anything in the case directory since I've gotten the model running (other than making CONTINUE_RUN=True and changing the RESUBMIT number in the env_run.xml file).

 

[erik]

OK, it still isn't giving much. It listed a CAM build-namelist command that it tried to do and failed. There must be more information coming from it. Try mousing that command and run it on the command line (you might have to try some different things to get it to work -- like chdir to the Buildconf/camconf subdirectory for example). I think you might be able to run it outside of preview_nameslists and it might give you some more help.

Since, you didn't change anything on purpose that should be a problem. Everything you mention should be fine to change before a new submission. I suggest looking for two things. One is a syntax error in the xml files that you changed. One way to check that is to run it through xmllint, which most systems have. The other is just to look at the file around the things changed to see if you see a syntax problem. You also should check for something that might have happened by accident. There should be a set of the files in the "Locked" subdirectory, and you can compare to them.

Also note that settings can be case sensitive so CONTINUE_RUN should be "TRUE" rather than "True". Try running the "./check_case" script in addition to "./preview_namelists" it runs the namelists, but also does some other checking.

 

[tylerh]

Hi Erik,
I'm still checking around files to see if I notice anything off, but haven't found anything yet. The project code that I was running the model on appeared to have gone into the negative core hour range when it stopped running and now won't resubmit. Is there any chance that could have anything to do with this issue? I've switched the project code to the other account that has plenty of core hours on it, but it doesn't fix the error.

Also, I ran ./preview_namelists -d. It spits out a ton of information, but nothing seems odd to me until the very end. I'm attaching that to this message. I'm not sure if that gives you any insight, but it didn't lead me to anything?

 

[tylerh]

Hi Erik,
I forgot to add that when I run that command in either Buildconf or the Buildconf/camconf directories, I get the following: