Tracking down memory leak in new COSP code

[jshaw35]

Hi all,

I have coupled the COSP satellite simulator in CAM with a separate radiative transfer model. The coupling and outputs are all working, but there appears to be a memory leak in my new code.

When I run CAM with my new code, the memory usage increases at each timestep until the run crashes. I have pasted output from the cesm.log at the bottom of the post showing this. When I run without the new code, the memory size is nearly constant and I don't have any issues.

I am wondering how to track down the memory leak in my new code. I have gone through all of the allocated variables to make sure that they are being deallocated but haven't been able to find the issue. Is there a way to print memory usage more frequently or another more granular way to go about this?

Thanks!
Jonah

Last 7 memusage statements from the cesm.log showing increasing memory usage:
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3056.4 MB rss=1765.8 MBshare=186.6 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3068.4 MB rss=1780.2 MBshare=186.6 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3089.3 MB rss=1800.0 MBshare=186.6 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3136.9 MB rss=1848.4 MBshare=186.6 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3124.4 MB rss=1836.2 MBshare=186.6 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3226.0 MB rss=1910.9 MBshare=161.8 MB text=58.9 MB datastack=0.0 MB
dec0156.hsn.de.hpc.ucar.edu0: sysmem size=3244.9 MB rss=1936.5 MBshare=162.0 MB text=58.9 MB datastack=0.0 MB

 

[peverley]

Hi Jonah,

I'm sorry to hear about your debugging woes!

One way to track memory usage is by calling shr_mem_getusage(...) before and after various calls and then printing out the difference.

You can see an example of how all that is done in $CAM/src/physics/cam/physics_grid.F90

Good luck!
Courtney

 

[jshaw35]

Hi Courtney,

Awesome, thanks for pointing me to the shr_mem_usage(...) call. I will add this in CESM's COSP interface.

The COSP code is built as a separate library that doesn't have access to the "shr_mem_mod" module. So I can't call "shr_mem_usage" in a more granular way in between steps within COSP and my new code. Is there a good way to make a similar call to print the memory usage without "shr_mem_usage"? It doesn't need to be CESM specific and I can run COSP offline completely separately from CESM if that makes things easier. A simple search points me in this direction (Track memory usage in Fortran 90), but I thought I would ask before diving in.

Thanks again!
Jonah