CCSM3 Paleoclimate Run Problem on Yellowstone

[taylor_hughlett@gmail_com]

Hello there,I am trying to create a paleoclimate simulation using CCSM3 on Yellowstone. I know that CCSM3 is not officially supported anymore, but was hoping I could still get some help for my issue so I can at least get the run working.My problem is that, with the addition of restart files that I used on Bluefire with CCSM3, the model appears to hang when running, and eventually times out. It is taking 6+ hours to run a simple 5 days.I'm unsure of what the problem is, and was wondering if anyone may have an idea of what it could be.Please let me know if there is more information that is needed.Thank you!

 

[jedwards]

Please provide the path to your case and run directories on yellowstone. What source did you build from and what if any modifications did you make?

 

[taylor_hughlett@gmail_com]

The path to the case on Yellowstone is:
/glade/p/work/hughlett/b30fw.013
The run directories are located at:
/glade/scratch/hughlett/b30fw.013
This was a case built off of the TraCE21 simulations by Liu et al. (2009). The only modifications that I made to the original model, along with the restart files, was the addition of the Dynamic Global Vegetaion Model and the inclusion of ramped greenhouse gas values.
Please let me know if you require any more information. I greatly appreciate the help.

 

[jedwards]

You mentioned in your first email running for 5 days. But I see no indication from your logs that the model has run past the first timestep. Have you run more than one step on yellowstone? I didn't see anything in your case or run directories that indicated what the problem was. If you get another hang, please run this debug script and point me to the output:

For example, if the job is running on ys0101-ib:
ssh ys0101-ib ~jlewars/bin/timeout_debug.sh -d

 

[taylor_hughlett@gmail_com]

No, this particular case has never run past one step. I submit it for 5 days, and it hangs before it ever makes it past the first timestep.I submitted the job and it hung once again, so I performed the debug script you requested. I do not see any kind of output from the debug script as far as a file is concerned. Can you give me an idea of what the file might be called?

 

[jedwards]

The output from the debug script should go to stdout, just capture that output and post it.  

 

[taylor_hughlett@gmail_com]

I managed to copy and paste the output from my xterm, and have attached it to this post, since it was very long. Please let me know if for some reason you have trouble opening it or recieving it.

 

[nanr]

Hi Taylor - Have you made any progress on running your simulation?  If you are still stuck, can you try to run a CCSM3 simulation out-of-the-box (i.e., with no changes).  It would be nice to eliminate the possibility that the changes you made to add dynamic veg and/or ramped GHG aren't the problem.Best regards -
Nan Rosenbloom (PWG Liaison)

 

[taylor_hughlett@gmail_com]

Hi Nan!I apologize it took me so long to get back to you; I didn't get notification that you had commented.I have run a CCSM3 simulation out of the box with no changes and it runs fine. I also added DGVM to it, and it ran fine. I've gotten the b30c.031 (Yeager simulation) to run through without any changes, but when I try to add in DGVM or BGC_2, it hangs and just times out. I always added them one at a time, and created new cases with each new addition, so I was sure I was getting a clean start.The same problem is occuring with the YD simulation that I'm continuing from the TraCE_21 experiments.Also, I am having an issue where my stderr files are only showing me information starting at about 20%. I can't see the beginning of the document. I'm not sure why they've started doing this, but being able to see the whole file would be a great help as well. Any ideas?

 

[nanr]

Hi Taylor - 
Can you compare your OTB case (which runs successfully) with the Yeager simulation to see where they are different?  Are you trying to recreate the b30.031 control? If so, can you add the yeager/control forcings to your OTB case one-by-one to see if you have more success.nan

 

[taylor_hughlett@gmail_com]

Hi Nan,I've narrowed the problem down to the implementation of both the BGC_2 model and the DGVM model. I've gotten the Yeager simulation to run successfully without either of those, but I require the BGC_2 and DGVM for my dissertation. I've tried to implement one model at a time. The DGVM causes the model to crash, and the BGC_2 causes the model to hang. I'm working to figure out what the problem is with the DGVM, but due to some problems with viewing files, have been unsuccessful so far. I'm working with CISL help to try to fix the file viewing problem.Any ideas on what could be the problem with the DGVM or BGC_2? For both, all I am doing is adding #define DGVM to the clm.buildexe.csh and set OCN_TRACER_MODULES = (iage BGC_2) in env_run.