Building on cori-knl fails

[onguba]

I cannot build CESM on cori-knl, clone is from yesterday:

Config line
./cime/scripts/create_newcase -case cesm-try5 -compset F2000climo -res f09_f09_m17 -mach cori-knl --run-unsupported --output-root /global/cscratch1/sd/onguba/cesm_scratch

./case.setup (gives an error because of script cism.template which I commented out in cime/scripts/lib/CIME/case/case_setup.py to avoid the error)

./case.build

leads to error:


```
checking host system type... gmake: Leaving directory '/global/cscratch1/sd/onguba/cesm_scratch/cesm-try5/bld/intel/mpt/nodebug/nothreads/mct/mct'
cat: Filepath: No such file or directory
cat: Srcfiles: No such file or directory
Invalid configuration `Linux': machine `Linux-unknown' not recognized
configure: error: /bin/sh /global/u1/o/onguba/cesm-repo/libraries/mct/config.sub Linux failed
gmake: *** [/global/u1/o/onguba/cesm-repo/cesm-try5/Tools/Makefile:617: /global/cscratch1/sd/onguba/cesm_scratch/cesm-try5/bld/intel/mpt/nodebug/nothreads/mct/mct/Makefile.conf] Error 1
ERROR: cat: Filepath: No such file or directory
cat: Srcfiles: No such file or directory
Invalid configuration `Linux': machine `Linux-unknown' not recognized
configure: error: /bin/sh /global/u1/o/onguba/cesm-repo/libraries/mct/config.sub Linux failed
gmake: *** [/global/u1/o/onguba/cesm-repo/cesm-try5/Tools/Makefile:617: /global/cscratch1/sd/onguba/cesm_scratch/cesm-try5/bld/intel/mpt/nodebug/nothreads/mct/mct/Makefile.conf] Error 1
```
I tried with python2 and python3.

 

[jedwards]

Which cesm version do you have? I just tried and didn't have any problems:
./create_test SMS.f09_f09_mg17.F2000climo.cori-knl_intel

 

[onguba]

This one:
```
commit a23d0337c7dd0ae2a9413bb4ecd3e315c0b54427 (HEAD -> master, origin/master, origin/HEAD)
Author: Chris Fischer <fischer@ucar.edu>
Date: Thu Sep 2 17:30:34 2021 -0600
Update for cesm2_3_alpha05d
```
and this for cime
```
commit ee1c2bf1ca7a5f9a5e832dc8d5a9d983571b0141 (HEAD, tag: cime6.0.7)
Author: Chris Fischer <fischer@ucar.edu>
Date: Wed Aug 25 10:48:41 2021 -0600
Update for cime6.0.7
```

I also tried SMS test as above and got the same error. Am I supposed to have certain modules? If not, would you please tell me which version you are running? I don't need the newest, I just need some version that runs.


thanks!

 

[jedwards]

Try cesm2_3_beta03

 

[jedwards]

I also found this to work using the latest cime master, but if you choose that route you need to add argument --driver mct to
your create_newcase command.

 

[onguba]

Thanks -- that worked to make it build, but I think the model gets stuck in init? I tried 2 runs, on 4 and 16 knl nodes, no threads, and both got stuck here (cpl log)
```
(seq_mct_drv) : Initialize each component: atm, lnd, rof, ocn, ice, glc, wav, esp, iac
(component_init_cc:mct) : Initialize component atm
(component_init_cc:mct) : Initialize component lnd
```
and and log from the 16-node run after 22 min:
```
calling set_mindist for gridcell
setting up interpolators for multi-level variables
check_var: variable DZSOI is not on dataset
```

Thanks again!

 

[jedwards]

I think that this is an issue with KNL - you could try cori-haswell

 

[onguba]

I got a script from Walter H. where he had these lines:
./xmlchange DIN_LOC_ROOT="/global/cscratch1/sd/onguba/inputdata_cesm"
./xmlchange DOUT_S=FALSE

These allowed me to run on knl, not sure which one is important.

Thanks!

 

[onguba]

To add, the HSW run was ok without using the 2 lines above.

 

[jedwards]

Those two lines have nothing to do with the success of running on cori.

 

[onguba]

You are right, thanks -- looking at the timers of the successful run, it seems that init took 70 min on knl for 1024 ranks and 200 min for hsw on 512 ranks, so, for the previous runs, I did not wait long enough.