Icepack - macOS Monterey

[simondriscoll]

Hi all, to give a heads up I think (key word) there may be an issue with Icepack on macOS Monterey. Insofar as I can tell. This is due to Icepack running fine on my Mac yesterday, I updated it to Monterey late last night, and attempted to do some runs (and tested the same runs) today. Instead of "ICEPACK COMPLETED SUCCESSFULLY" I now get "ICEPACK run did not complete". I created a new user account on my Mac and re-installed Icepack entirely on this second account (also in Monterey) and performed a test run and created a case and did a run of that, and both times I get "ICEPACK run did not complete". I was fortunate enough to borrow a friend's laptop (also Mac, not updated, I presume BigSur, but not Monterey) and performed the same download steps of Icepack on it, and ran a test and a case just fine. I.e. "ICEPACK COMPLETED SUCCESSFULLY"

I attach two logs files of a case, and test run, respectfully, in case they are useful to the problem. It seems there is some issue in that an enormous number of

"(ridge_ice)Repeat ridging, niter = 1"

lines are printed in the Monterey runs.

Hope it helps/flag it up early.

 

[Philippe Blain]

Hi Simon,

I do not have Monterey so I can't test your exact setup. Just to confirm, you are using the conda setup described at 3.2. Running CICE — CICE documentation ?

If so I would suggest the following:
- re-install conda (I'm not sure if you did that when you created a second user account).
- re-create the "cice" conda environment
- create a new Icepack case and compile it
- try to run the model again

 

[simondriscoll]

Hi Philippe,

I can confirm I'm using the conda setup as in the Icepack documentation (3.2. Running Icepack — Icepack documentation)

I have re-installed conda, recreated the conda environment as per the instructions in 3.2.3.4 etc. ("conda env create -f configuration/scripts/machines/environment.yml") and followed all previous instructions.

I have created a new Icepack case and compiled it. This is successful.

And I also have done this quite a few times with icepack before (and run it on the previous versions of my Mac etc.)

But the run itself, still even as performing afresh all your instructions once more (and even trying again using a brand new git clone of icepack, and new forcing data to make sure) do not run on macOS Monterey.

I attach another log file of the Icepack model not running in Monterey as an example. I can confirm I repeat all the same steps on another friend's Mac and this works. It doesn't work on macOS Monterey.

Thanks for your help.

 

[Philippe Blain]

Hi Simon,

Sorry for being confusing above by mentioning CICE instead of Icepack (I mostly develop CICE so this is an automatism :P)

I was able to get access to a Mac running Monterey (12.0.1) and I was able to replicate the issue. This was on a M1 Mac (new ARM chip), I would be interested to know if the problem you're seeing is on an Intel Mac or also on M1.

The problem is not in the Icepack code, as far as I understand; there is something really fishy going on with shell redirections/file descriptors on Monterey.
I first tried to remove the message your are seeing several times by commenting it:

diff --git a/columnphysics/icepack_mechred.F90 b/columnphysics/icepack_mechred.F90
index 54a7e9a..16ac344 100644
--- a/columnphysics/icepack_mechred.F90
+++ b/columnphysics/icepack_mechred.F90
@@ -430,8 +430,8 @@ subroutine ridge_ice (dt,          ndtd,       &
       !-----------------------------------------------------------------

          if (iterate_ridging) then
-            write(warnstr,*) subname, 'Repeat ridging, niter =', niter
-            call icepack_warnings_add(warnstr)
+           ! write(warnstr,*) subname, 'Repeat ridging, niter =', niter
+           ! call icepack_warnings_add(warnstr)
          else
             exit rdg_iteration
          endif

It's normal that this message appears, what's not normal is that you don't get more output afterwards.

If I run the model myself instead of using the icepack.run script, it succeeds:

cd ~/icepack-dirs/runs/<name-of-your-case>
conda activate icepack
./icepack

This ends with "ICEPACK COMPLETED SUCCESSFULLY" on my end. If however I redirect the model output to a file, like it's done in icepack.run, like this:

./icepack > out.txt

then the last two lines of output are missing from 'out.txt' ("Restart read/written" and ""ICEPACK COMPLETED SUCCESSFULLY") ... I can't explain that at the moment and couldn't find anything on Google.

It's also possible that it's not the OS, but rather the GFortran compiler from the conda environment that is generating wrong code. Knowing if you are on M1 or Intel would help in pinpointing the exact issue, I think.

 

[simondriscoll]

Great, thanks for being so helpful Philippe! I am on an Apple M1 Chip.

 

[Philippe Blain]

OK, so I played with this last night and could also reproduce on an Intel Mac with Monterey, using GFortran installed from Homebrew, but not with the Intel compilers (oneAPI 2021.4). This lead me to the right keywords to google, and I found that this is indeed a bug in macOS 12.0.1. If you're into that kind of details, check out the following links:

- Thread on Fortran discourse: Gfortran code is not proving data in output file macOS Monterey
- GFortran bug report : 102992 – fortran: redirecting standard out to a file does not work on macOS 12.0
- Openradar: rdar://FB9733712: .mod_term_func entries are no longer run (starting from macOS 12.0)

In the meantime, a workaround is to define GFORTRAN_UNBUFFERED_ALL=1 in the environment when running. You can do this in the env.conda_macos file, like this:

diff --git a/configuration/scripts/machines/env.conda_macos b/configuration/scripts/machines/env.conda_macos
index 0d094d5..7661246 100755
--- a/configuration/scripts/machines/env.conda_macos
+++ b/configuration/scripts/machines/env.conda_macos
@@ -26,6 +26,8 @@ endif
 
 endif
 
+setenv GFORTRAN_UNBUFFERED_ALL 1
+
 setenv ICE_MACHINE_ENVNAME conda
 setenv ICE_MACHINE_COMPILER macos
 setenv ICE_MACHINE_MAKE make

I haven't tested it but it should work. Another solution that I tested is piping the model output to cat instead of redirecting it to the log file, like this:

diff --git a/configuration/scripts/icepack.launch.csh b/configuration/scripts/icepack.launch.csh
index af106fc..36641f1 100755
--- a/configuration/scripts/icepack.launch.csh
+++ b/configuration/scripts/icepack.launch.csh
@@ -17,7 +17,7 @@ EOFR
#==========================================
else
cat >> ${jobfile} << EOFR
-./icepack >&! \$ICE_RUNLOG_FILE
+./icepack | cat \$ICE_RUNLOG_FILE
EOFR

endif

Let me know if this works for you :)

 

[Philippe Blain]

Sorry, the last diff should be :

diff --git a/configuration/scripts/icepack.launch.csh b/configuration/scripts/icepack.launch.csh
index af106fc..36641f1 100755
--- a/configuration/scripts/icepack.launch.csh
+++ b/configuration/scripts/icepack.launch.csh
@@ -17,7 +17,7 @@ EOFR
#==========================================
else
cat >> ${jobfile} << EOFR
-./icepack >&! \$ICE_RUNLOG_FILE
+./icepack | cat >&! \$ICE_RUNLOG_FILE
EOFR

endif

 

[simondriscoll]

Dear Philippe,

you are incredibly helpful! I tried setenv GFORTRAN_UNBUFFERED_ALL 1 in configuration/scripts/machines/env.conda_macos and I can confirm that my "ICEPACK run did not complete"'s have turned into "ICEPACK COMPLETED SUCCESSFULLY"'s once again.

Thank you!

 

[pwongpan]

Thank you both. I found this discussion also fixed my problem. I was wondering after "ICEPACK COMPLETED SUCCESSFULLY", how can I quickly plot the results to understand how Icepack work?

In particular, can I use ./timeseries.csh with this case1 outputs?

Thanks again.