Syntax and Permission denied error when setting up case under clm4_5_18_r270

[Cathy Li]

Hi,

I am trying to reproduce a case run by Dr. Keith Oleson (case directory at the end) with CLM version clm4_5_18_r270. I was helped by Keith and was able to get the code with compatible externals by running: svn co https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_18_r270 ~/clm4_5_18_r270.
I also changed ~/clm4_5_18_r270/cime/config/cesm/machines/config_machines.xml to match /glade/work/oleson/clm4_5_18_r270/cime/config/cesm/machines/config_machines.xml following his advice (see the first attachment for changes I made).

I set up a case (case directory at the end) with the following command. It failed at first, suggesting I load python 2.7+ version instead of python 3+, which I did, and it worked.
./create_newcase --compset HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV --res f09_g16_gl4 --case test_withfeedback_clm4_5_18_r270_cplh_hist --pesfile /glade/work/oleson/clm4_5_18_r270/components/clm/cime_config/config_pes.xml --project UIUC0056 --run-unsupported

When I tried to run ./case.setup, it gave me the following SyntaxError and a permission denied error (see the second attachment for debug output):

SyntaxError: Missing parentheses in call to 'print'. Did you mean print("\nResults in group %s"%group)?
File "/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist/./xmlquery", line 331
print "\nResults in group %s"%group
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("\nResults in group %s"%group)?
mkdir: cannot create directory ‘/cismIOconf’: Permission denied
ERROR cism.build_usernl: mkdir /cismIOconf failed: 256

The Syntax error seems to suggest Python 3+ was still running. When I tried in my command line though, print "test" was able to run without an error. Is there a way to resolve this without manually updating the code of xmlquery? Any pointer on this is greatly appreciated!

Thanks,
Cathy

------------------------
incl.
Keith's case directory: /glade/work/oleson/clm4_5_18_r270.p_old/cime/scripts/clm50sp_r270_1deg_cplh_Uv1_hist
My case directory: /glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist
svn diff results on changes made to config_machines.xml (attached)
case.setup debug output (attached)

 

[oleson]

I checked out the same code using svn and created the same case you did on cheyenne (I assume you are doing this on cheyenne), without any modifications. I didn't get any errors when I ran ./case.setup and ./preview_namelists. My modules that are loaded are:

Currently Loaded Modules:
1) ncarenv/1.3 2) intel/19.1.1 3) ncarcompilers/0.5.0 4) mpt/2.25 5) netcdf/4.8.1

So I'm not sure what's going on here. Maybe try to configure your modules to be the same as mine?

 

[Cathy Li]

Hi Keith,

Thank you for looking into this. I tried purging the modules, loading the ones you had, and recreated a new case, and I still got the same error. I have attached screenshots of what I have done.

It is probably not the issue but I did try the same thing on a different login node and got the same error.

Did you also get the error below when you did ./create_newcase though?

ERROR: Python 2, minor version 7+ is required, you have 3.9

Cathy

 

[Cathy Li]

I have also tried doing ./create_newcase under your versions of code (/glade/work/oleson/clm4_5_18_r270/cime/scripts and /glade/work/oleson/clm4_5_18_r270_svn/cime/scripts, and the problem persists...

 

[oleson]

No I didn't get that error. I don't actually have a python module loaded. When I do a "which python" I get:

/usr/bin/python

and this is pointing to:

lrwxrwxrwx 1 root root 9 May 12 2022 /usr/bin/python -> python2.7

so that's probably why I didn't get that error.

 

[oleson]

@erik , or @slevis , do you have any ideas on this problem, thanks.

 

[oleson]

Did you try module unload python? And then what do you get when you do "which python"?

 

[Cathy Li]

Hi Keith,

Thanks for the info. I was using a terminal on JupyterHub, and "which python" points to:
lrwxrwxrwx 1 jupyter jupyter 9 May 11 2022 /ncar/usr/jupyterhub.hpc.ucar.edu/jupyterhub-20220511/bin/python -> python3.9
Which should be what was causing the problem.

I then switched to a noninteractive terminal and tried everything below there. "which python" now points the same place, and I did not encounter the same error again.

When I tried creating and setting up a new case under my version of code (/glade/u/home/xinchang/clm4_5_18_r270), I got a different error instead:

xinchang@cheyenne1:~/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5> ./case.setup
Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5/env_mach_specific.xml already exists, delete to replace
Creating batch script case.run
Writing case.run script from input template /glade/u/home/xinchang/clm4_5_18_r270/cime/config/cesm/machines/template.case.run
Writing case.st_archive script from input template /glade/u/home/xinchang/clm4_5_18_r270/cime/config/cesm/machines/template.st_archive
Locking file env_mach_pes.xml
Creating user_nl_xxx files for components and cpl
ERROR: Command: '/glade/u/home/xinchang/clm4_5_18_r270/cime/../components/cism/cime_config/cism.template /glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5' failed with error 'sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `mkdir 2.7.17 (default, Nov 04 2019, 16:08:54) [GCC]'
ERROR cism.build_usernl: mkdir 2.7.17 (default, Nov 04 2019, 16:08:54) [GCC]
/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5/Buildconf/cismIOconf failed: 256' from dir '/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5'

I then tried creating and setting up a new case under /glade/work/oleson/clm4_5_18_r270_svn/cime/scripts (whose config_machines.xml seems to be unmodified), which was successful. The case directory is: ~/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_6. I then modified user_nl_clm and made changed to SourceMods following the case I am trying to replicate, and ran case.build. The build failed. Upon inspecting the build log (glade/scratch/xinchang/test_withfeedback_clm4_5_1 8_r270_cplh_hist_6/bld/cesm.bldlog.231130-144223), I suspect this might be due to changes not made in config_machines.xml.

Then I tried cloning a fresh version of the code (/glade/u/home/xinchang/clm4_5_18_r270_svn), created a new case (~/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_6), successfully ran ./case.setup, then I made changes to config_machines.xml. After changing user_nl_clm and SourceMods like above, I ran case.build, which again failed. The log (/glade/scratch/xinchang/test_withfeedback_clm4_5_18_r270_cplh_hist_7/bld/cesm.bldlog.231130-155032) seems to indicate the same error as above.

Sorry for taking a while to reply; I hope these things I tried may be somewhat helpful in leading to a solution. I will continue working on this and post what I find here.

Thanks so much again for helping me look into this!

Cathy

 

[oleson]

Ok, the python thing makes sense to me now.
I think the problem now is that you need to make the changes to config_machines.xml before you create the case. The env_mach_specific.xml will then be populated with the correct modules.

 

[Cathy Li]

Hi Keith,

Thanks! I did try creating a case after I modified config_machines.xml, and this was the error:

xinchang@cheyenne1:~/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5> ./case.setup
Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5/env_mach_specific.xml already exists, delete to replace
Creating batch script case.run
Writing case.run script from input template /glade/u/home/xinchang/clm4_5_18_r270/cime/config/cesm/machines/template.case.run
Writing case.st_archive script from input template /glade/u/home/xinchang/clm4_5_18_r270/cime/config/cesm/machines/template.st_archive
Locking file env_mach_pes.xml
Creating user_nl_xxx files for components and cpl
ERROR: Command: '/glade/u/home/xinchang/clm4_5_18_r270/cime/../components/cism/cime_config/cism.template /glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5' failed with error 'sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `mkdir 2.7.17 (default, Nov 04 2019, 16:08:54) [GCC]'
ERROR cism.build_usernl: mkdir 2.7.17 (default, Nov 04 2019, 16:08:54) [GCC]
/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5/Buildconf/cismIOconf failed: 256' from dir '/glade/u/home/xinchang/cases/test_withfeedback_clm4_5_18_r270_cplh_hist_5'

I will give it another go and let you know!

 

[Cathy Li]

I tried it again and ./case.setup was successful! This time I created a new case using the code based I cloned on the noninteractive terminal, instead of the one I cloned yesterday on JupyterHub. I suspect the error above was due to something similar to the python error, i.e., the default modules on JupyterHub being different than those on a noninteractive terminal.

Now the model is building; I have a good feeling about it this time. Fingers crossed! I will report back soon.

Sorry for all the hassle and thank you again for spending time on this!

Cathy

 

[oleson]

Great, glad to hear it!

 

[Cathy Li]

Unfortunately case.build still failed, despite my well wishes...
Error log here:/glade/scratch/xinchang/test_withfeedback_clm4_5_18_r270_cplh_hist_8/bld/cesm.bldlog.231130-165531

The error seems the same, although it looks like all correct modules were loaded (i.e., they were loaded based on the modified config_machines.xml).

I will keep digging into the error log and see what I can find!

 

[oleson]

I went back over my notes for this and found that I was able to run a case with this tag in 2019, but at some point there was a pio problem that we couldn't solve. So even though I set up that urban case later using that tag (clm50sp_r270_1deg_cplh_Uv2_BWIDTHVENT_hist), I found I couldn't build, getting the same error you do now. So I moved to clm5.0.dev010. In general, the results weren't bit-for-bit (i.e., the rural variables) with clm4_5_18_r270, but they were for all of the urban variables. So hopefully you could check out clm5.0.dev010 and do your run. See my example case here:
/glade/work/oleson/clm5.0.dev010/cime/scripts/clm50sp_clm5d010_1deg_cplh_Uv2_BWIDTHVENT_hist

 

[Cathy Li]

I see! Thank you so much for identifying the problem. I also narrowed down on the pio problem which was discussed on some older posts on the forum, but the solutions provided were not directly applicable. I'm glad we have a workaround!

Thank you so much for helping me with this throughout the day and even after working hours. I really appreciate it.

Enjoy your evening,
Cathy