2019 November: Complete LRGB Tutorial of NGC292, The Small Magellanic Cloud by Christian Sasse, Astronomer in Charge of iTelescope.net
2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!
There must be some memory leak in RGB Combine
When I loaded large files into RGB combine, I see the RAM measure go way high up. However, after exiting the RGB Combine window, even cleaning up all files from the list, the memory never goes down. No trivial operation (such as changing black point or saturation on a small file), which must cause garbage collection does not bring the memory count down. I suspect there is some memory leak there.
APP is written in Java. Being a Java software developer myself, I can tell you that when the Java Virtual Machine (JVM), where the code runs, allocates memory, it never frees the memory back to the operating system. It does free the memory internally but that's it. In Java terms there only is a memory leak if the internal memory of the JVM keeps growing and the JVM needs to keep on allocating more and more memory which to my best of knowledge is not the case with APP.
@wvreeven Thanks for your response. I am a long time java developer myself. My report is referring to the APP probably maintaining some reference to memory and not releasing it, thus GC does not reclaim the space during its cycles. It was evident when I removed all frames from the list, added new ones and made a new heavy operation which resulted in an OOM exception (with a 14GiB heap). Removing everything should have released all references, as there is no way for the system to use the old data again. Regardless of whether the JVM releases the memory back to the OS or not, if there was no leak the memory manager would used that pre-allocated memory and not result in an OOM at that point in time, which it did, forcing me to restart the APPlication.
Also, your assertion that the JVM does not give memory back to the system is no correct, it does do that, and the frequency and magnitude of it can be controlled (e.g. https://stackoverflow.com/a/30464183/1027568).
It is also a common misconception that "managed memory" languages, such as java and python have no memory leaks, but they easily do, and care should still be taken ( https://dzone.com/articles/how-memory-leaks-happen-in-java-apps ).
If there are some extra logs I could enable, I can try to recreate this. Also, using https://www.eclipse.org/mat/ can help analyze memory leaks on the jvm rather easily.
Thanks for pointing me to the stackoverflow comment about the JVM returning memory to the system. As is stated there, the JVM does this reluctantly because it is an expensive operation and that's probably why I never noticed it.
And thanks for your valuable comments on Java memory management. I am well aware of OOMs happening and I regularly had to use MAT to inspect heap dumps.
Your original message did not mention the OOM that you mention in your reply to my comment. That's why I replied the way I did. I agree that if there was an OOM then there is a memory leak in APP. Thanks for pointing that out.
Clear skies, Wouter
@wvreeven Oh, silly me, I forgot to mention that OOM exception in my first post!