case.st_archive unable to load modules and run python scripts

[ebrody]

What version of the code are you using?
CESM 2.1


Have you made any changes to files in the source tree?
I have some source mods, but the issue I'm having is not with the model.


Describe every step you took leading up to the problem:
I ran my CESM case with resubmit>0


If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
errput: sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
sh: module: command not found
ncdump: /glade/derecho/scratch/ebrody/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001/allthetemporarystuff/TREFHT_out.nc: No such file or directory
Traceback (most recent call last):
File "/glade/work/ebrody/cases/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001/controller/main.py", line 99, in <module>
exec(open(maindir+"/driver.py").read())
File "<string>", line 407, in <module>
File "<string>", line 183, in extractvars
File "<string>", line 93, in readfile
ValueError: 'data:\n' is not in list

ERROR conda.cli.main_run:execute(125): `conda run python /glade/work/ebrody/cases/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001/controller/main.py` failed. (See above for error)



see: /glade/work/ebrody/cases/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001/st_archive.b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SP.001.o8742047


Describe your problem or question:

I am running CESM 1 year at a time, and I've added a line in in case.st archive that calls a python scripts which modifies user_nl_cam in preparation for running the next year:

if success:
run_cmd("conda run -n npl-2022b python /glade/work/ebrody/cases/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001/controller/main.py", verbose=True)

My group has been doing this for about 10 years, but we recently started running into issues. For the first year after I run case.submit, this works as intended. However, if I set resubmit>0 in env_run.xml so that it runs successive years automatically, for all years after the first I get the error message pasted in the previous section. I added <command name="load">conda/latest</command> per advice, but it hasn't helped. The case referenced here is:

/glade/work/ebrody/cases/b.e21.BSSP245smbb.f09_g17.MCBss2.5TgYrRamp_SI.001

but I have several other cases that are behaving in the same manner. Any help would be greatly appreciated.

Thanks!

 

[katec]

Hi there. This error is usually associated with the environment not propagating into a shell script. There's information on addressing it in the CISL help pages here: Daily Bulletin | ARC NCAR
But this page is a little old. It might be useful to enter a CISL support ticket if this was previously working and is not any longer. These tickets let CISL know that changes made to the machine are impacting the user's workflows. I'm also going to move this over to the Infrastructure forum because it fits a little better in that area, and you may get a better response from somebody in that community.